Mastodawn

I wrote about AI again. On some level I don't now why I do this to myself

https://jenniferplusplus.com/what-is-a-token/

Actually, I wrote most of it months ago, for work. But, it was well received, so I put it on my blog. This is the more generic version.

The short version is that AI is not magic. It's a real phenomenon with real behavior and tradeoffs. I'm deeply tired of *****ALL***** the tradeoffs being handwaved away. And so much imagination fills in for the actual behavior. So I tried to describe how it's built, because that informs how it works, which informs what it actually does. And to be clear, it does things. It's not useless. But that's not the same as being useful, or worthwhile.

Anyway, I already put ~4k words on this in the article, so I'll shut up and let it speak for itself.

What is a token

AI is meant to seem like magic. But there's no such thing as magic. It's all illusion. So, allow me to spoil that illusion for you.

Jennifer++

Show thread

Almost A Fern Feb 26

@jenniferplusplus In the spirit of your closing speculation on better AI dev tools, I worked on an AI code generation project for a while, and the question the clients kept asking us was "can the AI tell us where it's hallucinating?" The "no" answer from the tool's perspective is a big part of why I avoid using them for my own coding, but I'm curious if that's a fundamental limitation of the models, or just the interface on them.

Does the final LLM capture how well-represented in the training data its current prompt is? Would it be possible with current technology to build an LLM that did? Is "well-represented in the training data" a reasonable proxy for reliability of output?

Show thread

Jenniferplusplus Feb 26

@bruceiv No, it's a pretty fundamental limitation of the models. I don't like hallucination as a term for this, because it's not different than any other output of the model. Everything it produces is a "hallucination." Whenever it generates text that's factually correct, it's because there's a strong correlation between the context and modeled vectors that just happen to be factual. Which is a byproduct of humans being highly interested in writing and reading things that are factual.

Show thread

Jenniferplusplus Feb 26

@bruceiv That said, to build a generator you probably have to build a classifier, and then keep going. So it's certainly possible that a model could do that classification, and include that data in some way with the generated output. I can easily image a model that emits the vector similarities it identified in the context (and probably also the response) on a side channel.

I'm sure there are lots of reasons the model vendors don't do that. It would make the models easier to copy. It would diminish the illusion of magic. Those vectors would invite their own explanation, which is likely not a simple design problem. It would expose details about the training data. Probably other things.

Show thread

Jenniferplusplus

@bruceiv As for whether the strength of those correlations is a good proxy for the reliability of the output, mostly no. The training data also includes plenty of fiction, sarcasm, trolling, and common misconceptions. At least, it does the way today's "AI companies" have gathered it. And those qualities are not guaranteed to be discernable through training. In fact, there's a fair bit of evidence that they are indiscernable. That's why google tells people to put glue on their pizza, and why chatgpt will so readily spiral along with people's delusions. They have no concept of reality, or correctness, or truth.