@Cassandrich @Sobri | Zoe (she/her) @Scott Jenson @Phil Dennis-Jordan Also, an image doesn't always need the exact same alt-text whenever it's posted somewhere.

The alt-text must adapt to the context. It must be different according to the context in which an image is posted. Also, it must adapt to the place where it's posted. The same image, even within a very similar context, must have a different alt-text in the Fediverse than on commercial social media or a static website. Lastly, and this ties in with the Fediverse requiring different alt-texts, the audience must be taken into consideration.

Alt-text in metadata can't do either of this. An LLM can't do either of this either unless it's explicitly prompted to do so, and even that is questionable.

Many Mastodon users dream of only pressing a button or not even that, and some AI automagically generates a perfect alt-text for their image. Perfectly accurate with exactly the details required for the context and the intended audience as well as the expected audience, all while following every last image description and alt-text rule out there to a tee.

It's perfectly understandable. Mastodon had begun to feel like child's play when they were suddenly pressured into describing each and every image they post. Worse yet, it seems like over 90% of all Mastodon users do everything on a phone with no access to a hardware keyboard whatsoever. So they have to fumble their alt-texts into a screen keyboard while not even being able to see the image they're describing.

I'm neither on Mastodon nor on a phone. I've got the luxury of having a desktop computer with a hardware keyboard and being able to bllind-type. So I don't have a problem with writing my image descriptions myself with no help from an AI.

In fact, my own original images are all about an extreme niche topic. It's so obscure that no AI will ever be able to describe such images, much less explain them at my level of accuracy and detail. (Explanations go into the post text, by the way, and not into the alt-text, but I always have an additional image description in the post text for my original images anyway.)

I simply know things that no AI will ever know, not ChatGPT and not Claude either, at least not at the point in time when they need that knowledge. And I can see things that will always remain invisible for AIs.

You can develop better models all you want. But they'll never be able to do all that.

#Long #LongPost #CWLong #CWLongPost #FediMeta #FediverseMeta #CWFediMeta #CWFediverseMeta #AltText #AltTextMeta #CWAltTextMeta #ImageDescription #ImageDescriptions #ImageDescriptionMeta #CWImageDescriptionMeta #AI #AIVsHuman #HumanVsAI

@Woochancho @Diego Martínez (Kaeza) 🇺🇾 @🅰🅻🅸🅲🅴  (🌈🦄) Especially whenever humans have advantages over LLMs.

When I describe my own original images, I have two advantages.

One, I know much more about the contents of the image than any AI. That's because my original images always show something from extremely obscure 3-D virtual worlds. On top of that, I may add some extra insider knowledge or explain pop-cultural references in the long description in the post if it helps understand the image and its descriptions.

Two, the LLM can only look at the image with its limited resolution. That's all it has. In contrast, when I describe my images, I don't just look at the images. I look at the real deal in-world with a nearly infinite resolution.

For example, an LLM can only generate a description from a picture of a virtual building. But when I describe it, my avatar is in-world, standing right in front of the building whose picture I'm describing. I can move the avatar around, I can move the camera around, I can zoom in on anything. I can correctly identify that four-pixel blob as a strawberry cocktail wheras the LLM doesn't even notice it's there.

I've actually done two tests using LLaVA. I've fed it two images I had described myself previously to see what happens. It was abysmal. LLaVA hallucinated, it interpreted stuff wrongly and so forth, not to mention that LLaVA's description, even after being prompted to write a detailed description, wasn't nearly as detailed as mine.

In one image, there's an OpenSimWorld beacon placed rather prominently in the scenery. LLaVA completely ignored it. I described what it looks like in about 1,000 characters, and then I explained what it is, what OpenSimWorld is and how it works in another 4,000 characters or so.

It's an illusion that AI will soon catch up with any of this.

Oh, by the way: How is an AI supposed to pinpoint exactly where an image was made if the image shows a place of which multiple absolutely identical copies exist? Or if the image has a neutral background that doesn't even hint at where it was made? I can do that with no problem because I remember where I've made the image.

#Long #LongPost #CWLong #CWLongPost #AltText #AltTextMeta #CWAltTextMeta #ImageDescription #ImageDescriptions #ImageDescriptionMeta #CWImageDescriptionMeta #AI #LLaVA #AIVsHuman #HumanVsAI

@モスケ^^ ❄️🐈🔥🐴 No. Very clearly no.

People keep thinking that AI solves the alt-text problem perfectly. Like, push one button, get a perfect alt-text for your image, send it without having to check it. Or, better yet, don't even push a button, the AI will take care of everything fully automatically.

However, at best, AI-generated alt-text is better than nothing. Oftentimes, AI-generated alt-text is literally worse than nothing.

First of all, AI does not know the context in which an image is posted. But an alt-text should always be written for a specific context because it usually depends on the context what needs to be described at all and on which level of detail.

This means that AI tends to leave out details that may be important while describing details that literally nobody is interested in.

AI can't take your target audience/your actual audience into consideration either. It can't write an alt-text specifically for that audience, fine-tuned for what that audience knows, what it doesn't know and what it needs and/or wants to know.

Worse yet, AI tends to hallucinate. It tends to mention stuff in an image that simply isn't there. It tends to describe elements of an image falsely. You could post a photo of a Yorkshire terrier, and the AI may think it's a cat because it can't distinguish it from a cat in that photo.

Seriously, AI may get even descriptions of simple images of very common things wrong. If you post images with very obscure, very niche content, AI fares even worse because it knows nothing about that very obscure, very niche content.

If you post a screenshot from social media, AI will not necessarily know that it has to transcribe the text in the screenshot 100% verbatim. And just pushing one button or running AI on full-auto, the thing that so many smartphone users are so much craving for, will not prompt it to do so.

If you want good, useful, accurate, sufficiently detailed image descriptions that match both the context of your posts and your audience, you will have to write them yourself.

Trust me. I know from personal experience. I post some of the most obscure niche stuff in the Fediverse. And I've pitted an image-describing AI against my own 100% hand-written image descriptions twice already. The AI failed miserably to even come close to my descriptions in both cases.

#Long #LongPost #CWLong #CWLongPost #AltText #AltTextMeta #CWAltTextMeta #ImageDescription #ImageDescriptions #ImageDescriptionMeta #CWImageDescriptionMeta #AI #AIVsHuman #HumanVsAI

@iolaire This is my personal analysis of the AI-generated image description, quote-posted from my original comment in the thread linked in my first comment:

RE: https://hub.netzgemeinde.eu/display/451d2f06-7746-4227-a043-76a959420c29

(6/6)

#Long #LongPost #CWLong #CWLongPost #QuotePost #QuoteTweet #QuoteToot #QuoteBoost #ImageDescription #ImageDescriptions #ImageDescriptionMeta #CWImageDescriptionMeta #AI #LLaVA #AIVsHuman #HumanVsAI

@iolaire After I have written the long description, distilled the short description from it and posted the image with both, I have asked a LLM AI for a description.

The AI of my choice was LLaVA 1.6: https://llava.hliu.cc/

The prompt was, "Describe the image in detail."

LLaVA took about half a minute to generate this image description:

The image depicts a modern architectural structure with a distinctive design. The building features a large, curved roof that appears to be made of a reflective material, possibly glass or polished metal. The roof is supported by several tall, slender columns that are evenly spaced and rise from the ground to the roof's edge. The structure has a circular emblem on the front, which includes a stylized letter 'M' and a series of concentric circles, suggesting it might be a logo or emblem of some sort.

The building is situated on a landscaped area with a well-maintained lawn and a few trees. There is a paved walkway leading up to the entrance of the building, which is not visible in the image. The sky is clear with a few scattered clouds, indicating fair weather conditions. The overall style of the image is a digital rendering or a photograph of a 3D model, as indicated by the smooth surfaces and the absence of any visible texture or imperfections that would be present in a real-world photograph. There are no visible texts or brands that provide additional context about the building's purpose or location.

(5/6)

#Long #LongPost #CWLong #CWLongPost #ImageDescription #ImageDescriptions #ImageDescriptionMeta #CWImageDescriptionMeta #AI #LLaVA #AIVsHuman #HumanVsAI

@iolaire Allow me to give you an example.

This is the image I'm talking about: https://hub.netzgemeinde.eu/photos/jupiter_rowland/image/b1e7bf9c-07d8-45b6-90bb-f43e27199295 (linked instead of embedded so I don't have to go through the hassle of having to describe it right here right now).

This is the thread in which I've posted the image before, including image descriptions, also including a comment with the AI description and an analysis of the AI description in comparison with my own descriptions: https://hub.netzgemeinde.eu/item/f8ac991d-b64b-4290-be69-28feb51ba2a7 (yes, this is part of the Fediverse; it's on the same Hubzilla channel that I'm commenting from right now).

(2/6)

#Long #LongPost #CWLong #CWLongPost #FediMeta #FediverseMeta #CWFediMeta #CWFediverseMeta #AltText #AltTextMeta #CWAltTextMeta #ImageDescription #ImageDescriptions #ImageDescriptionMeta #CWImageDescriptionMeta #AI #AIVsHuman #HumanVsAI