I built possibly the earliest same-tab tool for getting AI image descriptions in Aug 2022, using a then newly updated Microsoft AI API. I listened to Disabled folks about it. I've payed close attention to the subject since.
My criticisms of @mozilla plans for Firefox built in AI alt text generation aren't because of knee-jerk AI hatred or some ignorance. They are because I understand this to a breadth and depth extremely rare and I want acknowledgement of major issues being ignored 🧵…
Boost? 💜
This issue is complex in several ways contributing to conflict here. Those saying objectors must not understand things are themselves not understanding the roots of objections.
The text of @mozilla announcements on social media talked only about AI generated descriptions accessible via screen reader. This is commonly asked for by those needing alt text, and the above tool was built for exactly that reason: it can be useful to fetch an AI description of an image when unable to see it.
🧵…
The post image was a UI for use when writing alt text, and the linked blog confirmed that this was on the roadmap.
Usage patterns by those writing alt text can be readily predicted, because these tools already exist and are fairly widely used. Current patterns of usage often show a lack of any care to edit or even review generated descriptions before posting images with them as alt text to social media.
Ignored objections from alt text dependent folks call out these descriptions as poor.
🧵…
The detail that this is specifically about social media posts is important, and it is not widely understood that the description needs and process are substantially different from other page types. I think based on existing image post patterns that Firefox AI description write usage would primarily be on social media.
There post context is crucial. The goal is equal access to a post and that context informs what in the image is important to describe. Sometimes those aspects are abstract.
🧵…
Images posted to social media are very diverse in a number of ways. It is evidenced by the images used to test @mozilla description AI being, as far as I could tell, entirely stock art that this diversity is not being accounted for. Test images are often ones unlikely to be posted to social media as is
The descriptions being generated are also assessed based on needs for posting images in places other than social media
The result: assessing project progress does not consider likely usage
🧵…
Criticism of this @mozilla image description AI and others is frequently met with impressive demos, but this repeats a frequent AI advocate fallacy: AI is often impressive but objections are that it is not reliable in many surprising ways. That concern can not be addressed with just a few examples.
Other remote AIs offering more advanced description produce outputs which are substantially more detailed and advanced, but they fail in shared and unique ways. They also chug power and water.
🧵…
There are still more details, but I'll leave it here.
I believe tools for those writing alt text which include image description AI could contribute to web accessibility.
To actually contribute they need to include and deeply understand the needs of Visually Impaired people relying on alt text. Current tools seem to largely prioritize the time and effort of those writing alt text over equal access to the web.
Questions or comments? I am happy to chat here or via email: [email protected]
💜
Important partial retraction: I thank @[email protected] for the correction on the @[email protected] image description AI plans criticized by me in these last few days. I and many others, most of whom I saw celebrating, believed that those plans included providing such tools to all users writing alt text in the web browser. They don't. I retract my criticism of Mozilla, because they are already doing what I was asking. I maintain my criticism of alt writer AI tools, and I think Jamie has validated my concerns.
@filobus @hannah The most useful tool for me when writing Alt text is OCR, and I've come across a good extension: https://webextension.org/listing/ocr.html
Of course, although its recognition rate is really high, it still wants proofreading! It saves a lot of time typing though.
For anything that's not text, only the person posting it will know what they want to express with their image.
The 'OCR - Image Reader' extension is designed to simplify optical character recognition (OCR) processing within your browser. After installation, the extension adds a new button to the toolbar area of your browser. When you press this button, the current window switches to the selection mode, allowing you to select a region on the current page. The extension captures the image of the area, and the internal engine, Tesseract.js, extracts the content for you. This engine is a JavaScript-based OCR that supports over 100 languages. When performing a new job, the extension displays the progress of the OCR extraction in a popup window. If you have multiple jobs, you will get equivalent floating windows. It's worth noting that the extension fetches the appropriate language database from the server once. In future usage, the cached database is used.
@gunchleoc @filobus when I added the OCR cmd to @AltTextCrew@☠️ I did a lot of research into OCR services. Google's advanced OCR API was most powerful.
It doesn't need to know the language to be extracted, instead you give it an image and tells you the language, with support for different languages in the same image and location data on individual letters.
Proof of concept tool taking an image of a text table and OCRing into an HTML <table> using letter position: https://2d-ocr.glitch.me
@jcsteh thanks so much for the thoughtful reply, I'm glad to make these connections.
My concerns are very specifically this tool being easily used by folks adding alt text to images in general web interactions. While I know that would be a long way off, it is the focus of celebratory discussion I've seen. It is based on a few things:
(cont)
@jcsteh (cont)
- It will primarily be used for social media
- The descriptions it generates are targeted at PDF images, with radically different description needs
- When I talk to Visually Impaired folks on social media interacting with AI usage they have unanimously said frequent abuse created alt text that was often basically worthless, the AI being used an LLM of some sort.
- social media alt text writing tools require immense care to constrain abuse
(cont)
@jcsteh (cont)
That last point is important. I foresee a "this is V1, we can go from here" gradual improvement approach, but I believe the complexity gap between this usage iteration and the usage method minimum needed for responsible use in writing social media alt text is large in complex ways, and later making the jump with the feature already live would be very resisted.
I believe releasing any tool to web writers needs immense care prior to design, see none planned, and alarms ignored.
@jcsteh I am glad to hear that other use is not being pursued right now and hope that this functionality is never provided to general browser users, rather than just by user-selected extension, until that tool's results can be assessed by Visually Impaired folks for images representing the spread likely to have the tool used for them
I believe the only current AI capable of good such use is Google's, and via a small specialized API. I'm working on a proof of concept tool to demo some complexity
@jcsteh I recall seeing a direct mention that providing it to those writing alt text on sites was planned, but not finding it now I'll make no claims as to its author being official.
I have seen a pretty universal perception that this feature will be offered thus in the future, and believe that it is irresponsible to not address that celebration with some acknowledgement that doing so takes immense care.
@jcsteh that is so appreciated. I had a major brain surgery a week ago and have been spending a lot of energy on this because I had the perception that this was on the road map and saw little understanding of potential issues or willingness to listen when I tried to explain those issues or when folks relying on alt text raised alarms.
I should really sleep, but I'm happy to chat more later and if anyone internal would like to speak I can be reached at [email protected]
@jcsteh unrelated but I also wanted to link a discussion of a project under the Mozilla umbrella that I think is worth drawing your attention to if you're willing and able to share it where you think could be useful.
Doing new to me frontend work I used the popular https://developer.mozilla.org HTML and CSS docs and noticed a pattern that could impact accessibility results. The below discussion came to an agreed upon high level plan, but cancer has made me unable to enact it.
@hannah Thank you. I've been following this from afar, and your explanation makes sense.
As a sited reader I often enjoy people's alt text descriptions as a kind of Eater Egg, or hint as to the image relevance to them.
I've wondered if UI for alt text writers could help here, and if AI could be a part of this.
I'm imagining that when an image is added to a post it would immediately have a visible default description in an editable textfield.
My hypothesis is that a well constructed UI could strongly invite a poster to edit the generated text and replace or improve on it.
This is a testable proposal, I guess – A given UI could be evaluated for how effectively the poster is prompted to improve the description.
But – it's not something I'm going to be building as it's miles from my experience!
@benjohn Based on my experience w/ populations of social media users using AI for alt text, simply prompting for edits is unlikely to help much for a number of reasons. I believe there is some low hanging fruit possible examples of what more responsible to provide tools might look like and am working on a proof of concept one, we'll see if I can continue
Unfortunately, they require advanced and remote AI to work. I believe Google's advanced specialized API for this is the only possible one ATM
Hannah Kolbeck 🏳️⚧️
Syn Reitan
filobus
GunChleoc
Geoffrey Conley 🏳️🌈🖥️🚲
Janeishly
Jamie Teh
Benjohn