Manuel MatuzovićI want to test something. Which AI tool would I use to find accessibility issues? In code but also on the rendered website in the browser?
Sebastian Laube@matuzo They are all bad at it, I guess.
Casey (@[email protected])
Attached: 1 image Introducing WebAccessBench, a novel benchmark for AI language models to assess #accessibility quality and WCAG conformance in generated web interfaces under realistic prompting conditions. I did a bit of research and found that LLMs are incredibly bad at basic digital accessibility tasks. You can compare models and read the full white paper at https://conesible.de/wab. Overall data suggests massive implications for society at large, and major discrimination of people with disabilities. #a11y
Matthias Ott@matuzo I’d say it is futile to just pick any of those tools and run prompts against them “blank” without context. But what would be interesting is whether the results are more useful when you give them more context, like Claude with one or two (?) a11y skills + a set of your own instructions.
Phil Sherry@matthiasott @matuzo “But what would be interesting is whether the results are more useful when you give them more context…”
Garbage in, garbage out.
If you don’t write good context for them and then hone it once you’ve seen how they work with it, you’re destined to be one of those people who go on about how these tools are useless. Think of them like a junior dev who needs educating by their senior and you’re always going to get better results—unless you’re not a very good senior.