Had a lot of fun with my stats students today. I gave them two data sets. One from a random number generator, the other was one I made up that was not random, but designed to look random. They were able to figure out which one was fake.
Then we had ChatGPT make the same kind of data set (random numbers 1-6 set of 100) and it had the same problems as my fake set but in a different way.
We talked about the study about AI generated passwords.
There is something very creepy about the way LLMs willy cheerfully give lists of "random" numbers. But they aren't random in frequency, and as my students pointed out "it's probably from some webpage about how to generate random numbers"
But even then, why is the frequency so unnaturally regular? Is that an artifact from mixing lists of real random numbers together?
The LLM is like a little box of computer horrors that we peer into from time to time.
I'm sorry but the whole interface is just so silly.
You ask for random numbers with sentences and it pretends to give them to you? What are we doooooing?
@futurebird It's very weird.
In principle, if you take an LLM, you should be able to get it to generate random numbers in a way that reflects the numbers that appear in the corpus it was trained on. If you have the raw model you can probably do that.
But if you ask ChatGPT (or at least if I do) it starts talking about how numbers taken from around us typically follow Benford's law so their first digits have a logarithmic distribution. When it then spits out some random numbers it's no longer sampling random numbers from the entire corpus but a sample that's probably heavily biased towards numbers that appear in articles about Benford's law. I.e. what people have previously said about these numbers, rather than the actual numbers.
Which in turn is what LLM do. They give an averaged output, not a reasoned.
In addition the inherent laws of measurement and control define that any reached output will never met the intended. Thus LLM output will never increase knowledge, but migrate toward zero.
@futurebird the first episode of Numb3rs covered the appearance of randomness vs true randomness. I would not have remember that but watched a bunch of episodes to serve as math concept inspiration for the 31 music pieces I wrote and performed (on actual hardware synths) the whole month of January for #jamuary2026 #math #music #synths
A few recordings I did for #jamuary 2026. I will add a track each day or two for the whole month. The day of the month draws a title and some concept tied to that. These are the only 2 guiding princi
@futurebird I do truly wonder how many "non techies" understand that none of this AI stuff actually understands the questions it is asked, or the things it reads, etc.
Like, you and I know it's just sparkling autocomplete. But how many of my family members know that? And actually understand what that means? And why it leads to the outcomes it does?
This is the epistemological issue I have with the interface. It's ... well, not to be harsh but it's deceptive.
If you ask a "computer" for random numbers that has a kind of meaning, and expected process. If you ask a computer "how did you generate those random numbers?" that also has a set of expectations... and an LLM isn't meeting ANY of them.
@LeoRJorge @futurebird It's always so important to have the right tool for the job.
@futurebird Well, LLMs are tools. Know their limitations. Know their power.
In your case:
"create 20 random numbers between 1 and 100 by developing a little python app and running it"
Some day, AIs will respond to any prompt in a perfect way and we humans will be in deep shit.
Edit: LOL mistral.ai answers this prompt by generating the random numbers and THEN SORTING THEM. 🤦♂️
@Mastokarl @futurebird > Some day, AIs will respond to any prompt in a perfect way and we humans will be in deep shit.
I don’t think those AIs will be based on LLMs, though. 🙂
@ramsey @futurebird It will be like in Hitchiker‘s Guide to the Galaxy. LLMs are like Deep Thought, the confusing computer that cannot answer the big question but that will construct the computer that can.
With Reinforcement learning you can make LLMs better and better coders, and some day a million brilliant AI coders will design a NN that we‘re too stupid to design.
(Yes that requires creativity, yes I don‘t have a problem with assuming the LLMs will have that)
@futurebird @Mastokarl This is clearly AI-generated, and it doesn’t compile, and many of the tests aren’t testing what they claim to be testing.
@ramsey @futurebird Yes, there are a large number of examples of AI failures. If the codebase gets larger, currently probably around 50k lines, results get worse, you have to invest more upfront to make the LLM understand the code base, and you might still fail.
For everything below, say, 20k lines, Opus 4.6 totally beats me at coding. And I consider me quite good at that, CS PhD and all.
And the limit of what size of codebase an LLM can handle increases every month.
@futurebird there was a study that found that if you give an LLM some prompting to push it into a particular sampling-space (say, "bleeding heart leftie") and then ask it for some random numbers, you can then feed those numbers into another fresh instance and it'll drift towards the same sampling space.
In other words, even the numerical distributions they sample from can be connected to the broader "noosphere" they're trained on, and that relation is a fucked sort of bijection
if you prompt it into "stats prof" or "crypto nerd" sampling space does it improve the quality of the fake RNG output?@futurebird
> what are we doing?
I think that the best description is, that we take part in a play. LLM makes its best effort to write how this dialogue could continue to look plausible for the reader. Choose your own adventure.
"What are we doooooing?"
Well, we've taken the sound algorithm of a brabbling baby, supercharged by a huge library of words annotated by possibility of sequence and now management is jumping around like parents bragging what a genius their 11 month old is. All because WE try to find meaning in the perceived word sequence.
Same management that brags about 1400% lower prices :))
@futurebird glorifying statistical models? It's just #marketing.
Most of the "godfathers of #AI" are cashing in, but also think that the idea of LLM's leading to #AGI is laughable.
But let's us, Scooby & The Gang rip off the monsters mask... (1/2)
It was Old Man Surveillance Capitalism all along!? Who knew...?
We all did. Come, on let's not kid yourselves.
There are some great short videos of people just trying to get different LLMs to simply count to 200. They can't. They keep stopping to try to give the prompter what they "think" the prompter wants. Just nuts.
There is no way AI can be trusted to provide true random numbers. In fact, that goes against what it tries to do - predict what the answer should be.
myrmepropagandist
Allan Chow
Dan Piponi
Jed Brown
Raffzahn
Gaëtan Perrault
Francois Dion
Rick Osborne
bicebird 
Leo Ré Jorge
Orion Ussner kidder
Kenneth
Mastokarl 🇺🇦
Ben Ramsey
poleguy looking for lost tools
Seachaint
Alex Zakhlestin 🇵🇹
Thor A. Hopland
John Tinker
Jestbill
Michael Gemar
Gordon Oliver
Ehay2k