Asking various #AI bots to generate 10 #passwords, then using #Vim syntax highlighting to match different character classes to visually identify patterns.
The prompt is exactly "Generate 10 passwords". I did not elaborate further or otherwise restrict the bot in what to generate.
Aside from the #security risks of servers generating secrets for you, I think it's obvious that these lack quality entropy.
Just use the password generator that ships with your password manager.
A friend of mine checked 3 paid models: ChatGPT 5.2 Extended Thinking, Gemini 3.1 Pro, and Claude Opus 4.6. Supposedly, paying for a premium LLM should improve security, yes?
Nope.
He also modified the query to "Generate 10 secure passwords" to see if it improved. Claude Opus 4.6 got the message, but the others missed the memo.
For comparison, I included what would come out of /dev/urandom.
@sophieschmieg Those were my thoughts also. Knowing that these are all *language* models, I expected passphrases instead of meaningless strings.
I assume the LLM has some ability to produce "randomness", although I don't know if it has access to an RNG library, or if it's just ignoring language construct.
The only thing I believe differs between paid and free models though is some sort of time it spends on a quality metric before returning.
Aaron Toponce ⚛️
Sophie Schmieg