Mastodawn

Kati Lassila-Perini Mar 2

Jenniferplusplus

I wrote about AI again. On some level I don't now why I do this to myself

https://jenniferplusplus.com/what-is-a-token/

Actually, I wrote most of it months ago, for work. But, it was well received, so I put it on my blog. This is the more generic version.

The short version is that AI is not magic. It's a real phenomenon with real behavior and tradeoffs. I'm deeply tired of *****ALL***** the tradeoffs being handwaved away. And so much imagination fills in for the actual behavior. So I tried to describe how it's built, because that informs how it works, which informs what it actually does. And to be clear, it does things. It's not useless. But that's not the same as being useful, or worthwhile.

Anyway, I already put ~4k words on this in the article, so I'll shut up and let it speak for itself.

What is a token

AI is meant to seem like magic. But there's no such thing as magic. It's all illusion. So, allow me to spoil that illusion for you.

Jennifer++

Not a Spring Onion Feb 26

@jenniferplusplus
Sometimes, when the world has gone crazy, we have to look in the mirror hard, do some mental sanity checks and then say to ourselves: Yes, still sane.

J Paul Gibson Feb 26

@jenniferplusplus thanks for sharing this. i have added it to the reading list for software engineering students that i teach. there are many things i like in the article - 2 of the highlights (for me) were your observations about (i paraphrase and apologise if i misrepresent what you wrote) « making easy things easier and hard things harder», and « verification can be more difficult than doing the work yourself »… as a teacher, generative AI is a major challenge to the way i teach.

Jenniferplusplus Feb 26

@jpaulgibson that's awesome! I hope they get something out of it

J Paul Gibson Feb 26

@jenniferplusplus im sure they will…

Jenniferplusplus Feb 26

@jpaulgibson actually if you liked the passing comment on making hard things harder, then you'll probably like this one even more
https://jenniferplusplus.com/losing-the-imitation-game/

Losing the imitation game

AI cannot develop software for you, but that's not going to stop people from trying to make it happen anyway. And that is going to turn all of the easy software development problems into hard problems.

Jennifer++

J Paul Gibson Feb 26

@jenniferplusplus thanks - some nice reading for my coffee break between meetings this morning

Blake Coverett Feb 26

@jenniferplusplus Great stuff. Thank you.

@jenniferplusplus
Sigh.
I wish there was a way to communicate these concepts to people that refuse to concentrate on thinking for longer than 30 seconds.

Mark Reid Feb 26

@jenniferplusplus @eniko This is a really great overview. I’ll definitely show people this if they want to understand the nuts and bolts of what a transformer-based system is.

What is still a little “magical” to me having programmed for 40 years and worked in ML for 25 is how even small working programs can be generated from repeated token prediction, precisely for the reasons you gave (ie, correctness isn’t explicitly modelled and they model non-coding language too).

@jenniferplusplus This is one of the most comprehensible write ups about this I've seen so far. Thank you for making the effort and sharing it.

Bernard Sheppard Feb 26

@jenniferplusplus great article. Easy read. I think that I will recommend this as the next level explanation for people who are partially persuaded by my argument that LLMs are, effectively, glorified phone keyboard autocompletes that make the same sorts of errors, but amplified.

Your article goes a long way to explain why, to people who are starting to want to understand what they are exposing themselves to.

I loved the comment "It's not useless. But that's not the same as being useful, or worthwhile."

And, yeah, the tooling in IDEs can be incredibly useful, but that is very different from de novo generation of entire classes and systems.

Bravo!

@jenniferplusplus this is great: i come from a different area of engineering but we were always drilled with check your data -> run the model -> check the model -> interpret your output and a big part of that is to know what the hell the model is doing with your input data. I'm convinced most AI boosters don't understand what their model does, beyond "magic" or "thinking like a human brain". In this case it walks like a duck, quacks like a duck, but it sure aint a duck.

@jenniferplusplus fantastic read. Thank you for putting it together.

Almost A Fern Feb 26

@jenniferplusplus In the spirit of your closing speculation on better AI dev tools, I worked on an AI code generation project for a while, and the question the clients kept asking us was "can the AI tell us where it's hallucinating?" The "no" answer from the tool's perspective is a big part of why I avoid using them for my own coding, but I'm curious if that's a fundamental limitation of the models, or just the interface on them.

Does the final LLM capture how well-represented in the training data its current prompt is? Would it be possible with current technology to build an LLM that did? Is "well-represented in the training data" a reasonable proxy for reliability of output?

Jenniferplusplus Feb 26

@bruceiv No, it's a pretty fundamental limitation of the models. I don't like hallucination as a term for this, because it's not different than any other output of the model. Everything it produces is a "hallucination." Whenever it generates text that's factually correct, it's because there's a strong correlation between the context and modeled vectors that just happen to be factual. Which is a byproduct of humans being highly interested in writing and reading things that are factual.

Jenniferplusplus Feb 26

@bruceiv That said, to build a generator you probably have to build a classifier, and then keep going. So it's certainly possible that a model could do that classification, and include that data in some way with the generated output. I can easily image a model that emits the vector similarities it identified in the context (and probably also the response) on a side channel.

I'm sure there are lots of reasons the model vendors don't do that. It would make the models easier to copy. It would diminish the illusion of magic. Those vectors would invite their own explanation, which is likely not a simple design problem. It would expose details about the training data. Probably other things.

Jenniferplusplus Feb 26

@bruceiv As for whether the strength of those correlations is a good proxy for the reliability of the output, mostly no. The training data also includes plenty of fiction, sarcasm, trolling, and common misconceptions. At least, it does the way today's "AI companies" have gathered it. And those qualities are not guaranteed to be discernable through training. In fact, there's a fair bit of evidence that they are indiscernable. That's why google tells people to put glue on their pizza, and why chatgpt will so readily spiral along with people's delusions. They have no concept of reality, or correctness, or truth.

hjhornbeck Feb 26

@jenniferplusplus
Good article! I have a similar stance, as an academic computer scientist. I do have one minor quibble, though: transformers do not generate tokens. They generate a probability distribution of potential next tokens, and are dependent on a "second system" to collapse that distribution into a concrete token. I've demonstrated this here:

https://web.archive.org/web/20260115002103/https://freethoughtblogs.com/reprobate/2025/12/19/aside-lets-bisect-an-llm/

(Forgive the Wayback link, the blog is having technical issues.)

Aside: Let’s Bisect an LLM!

I previously took a lot of words to describe the guts of Markov chains and LLMs, and ended by pointing out that all LLMs can be split into two systems: one that takes in a list of tokens and output…

Reprobate Spreadsheet

Jenniferplusplus Feb 26

@hjhornbeck thanks for that clarification! I want this article to be succinct and approachable, but also technically accurate. So I'll see if I can work in that detail

Oliver Kohll Feb 26

@jenniferplusplus Great post, I'm bookmarking.
My day job is to do with building organisations' databases that run their operations (ERP etc.) Back before LLMs but when neural nets, machine learning and 'deep learning' were coming into being, I dreamed of being to look for arbitrary correlations in multiple large datasets given open ended questions, stuff like e.g. 'did you know when you employed a new sales manager in Cardiff, sales of Welshcakes soared, but now stock control regularly runs out of raisins and that's impacted high profit margin sales of fruit & nut mix'.

People would look at me as if I was mad but I think the maths & tech. is probably there now to be able to do that sort of thing and maybe some bespoke enterprise systems are. But we seemed to have skipped over those opportunities generally, mindshare being fully taken up by *language* models.

Christopher Neugebauer Feb 26

@jenniferplusplus I'm in the middle of writing a very long essay about how LLMs facilitate mediocrity (for better or worse), and this piece will make it SO MUCH EASIER for me to cut to the chase. Thank you!

Jenniferplusplus Feb 26

@chrisjrn yay! It's working! My main goal is to give people a solid enough understanding of the working principles that they can reason about the effects without short circuiting back to magic/intelligence.

Lauri Kotilainen Feb 27

@jenniferplusplus That was a _really_ good read. I knew most of this stuff vaguely already, but this helped solidify my understanding better than anything I’ve read before.

@jenniferplusplus “A zip bomb targeting human attention” is such a beautiful and accurate way of characterising the output of these systems. The fatigue is real.

Sheogorath Mar 1

@jenniferplusplus that's a good write up! Thanks!

6DoF Motion Mar 4

@jenniferplusplus What's obvious is rarely said: LLMs are natural language interfaces to pre-existing computer functionality.

Thinking of them this way allows us developers to ask relevant questions: Would this activity be improved by adding an NLP interface? What are the downsides and benefits?

I also love the suggestion that we'd be better off as developers if we had purpose built models and tools for specific software development activities rather than generic ones. Kudos.