@kashhill The mental health situation is already bleak. Add to that the medicare cuts, and that a brain worm addled, methyl blue guzzling, dog eating imbecile is in charge of our health People have no where to turn too and when you get into a bad spot, any lifeline will be accepted...

This may be the perfect storm for mental health we have all feared is coming.

Its going to hit teens hard and the way the current admin is demonizing LGBQT+ groups that already have a crazy high self harm rate its going to be devastating

@kashhill
Fascinating story!

Here's another:
☠️

People Are Losing Loved Ones to AI-Fueled Spiritual Fantasies
Self-styled prophets are claiming they have 'awakened' chatbots and accessed the secrets of the universe through ChatGPT
https://archive.ph/eeesj

One thing that always makes me nervous when I read something like this is I have Baldur’s LLMentalist in my head:

“Remember, the effect becomes more powerful when the mark is both intelligent and wants to believe…. If anything, your intelligence will just improve your ability to rationalise your subjective validation and make the effect stronger. When it’s coupled with a genuine desire to believe in the con—that we are on the verge of discovering Artificial General Intelligence—the effect should both be irresistible and powerful once it takes hold.”

I think of myself as intelligent. So when I read something like this guy falling down a rabbit hole with an AI, my heart goes out to him and I think “there but for the grace of God go I.” I mean, I refuse to believe it could not happen to me. I have to try to stay grounded.

@kashhill thank you for looking into this! It seems pretty clear from even the earliest reported interactions what the problem is: this person trusts the LLM. He believes what it writes, at least as if it was a human writing it, and possibly even more than if it was a human. That is baffling to me.. But I guess because these tools have been hyped up so much and come from reputable companies, a non-informed person might spontaneously trust them?

This is the problem and this is what needs to be fixed. People need to know that the LLMs have no notion of truth and none of what they "say" can be trusted.

#LLM #genAI

@kashhill I can't stand the smarmy mockery of human speech that these things output. It's repugnant.

I also notice that it always ends its flattery-vomit with a prompt for the user to continue interacting with it. "Just type 'yes' and I'll dig the rabbit hole a little deeper for you. That's what you want, isn't it?"

@kashhill read this excellent article and must say I feel for the man you wrote about.

To invest that much time, effort, money into something that was not health-affirming and made his sons and friends concerned for his well-being is really heart-breaking!

@kashhill

Great read, thank you for writing and sharing this. It was interesting that a lot of the quotes that you sourced from other people were also playing into the narrative of the LLM vendors. For example:

Amanda Askell, who works on Claude’s behavior at Anthropic, said that in long conversations it can be difficult for chatbots to recognize that they have wandered into absurd territory and course correct

Note the terminology here: 'difficult for chatbots to recognize...'. It's not that this is a difficult thing for an LLM to recognise, it is that recognising things is fundamentally not something that text extrusion machines do. The same with this bit:

A Google spokesman pointed to a corporate page about Gemini, that warns that chatbots “sometimes prioritize generating text that sounds plausible over ensuring accuracy.”

No. They don't sometimes prioritise generating text that sounds plausible over ensuring accuracy, they always generate text that sounds plausible, that is what an LLM is: a machine for generating text from a high-probability space as defined by their training data. By coincidence, this is often factually accurate, but they do not have any way of determining that.

Even some of your own text is anthropomorphising LLMs:

The reason Gemini was able to recognize and break Mr. Brooks’s delusion

Again, no. Gemini didn't recognise the delusion, it was simply that starting with a delusion the highest-probability next text was a report that it was probably delusional.

Including the final paragraph:

“It’s a dangerous machine in the public space with no guardrails,” he said. “People need to know.”

Guardrails are a marketing term (as is 'AI', as @emilymbender has described at length). Fundamentally, these machines are just producing text. The way that they are marketed is to pretend that they are interlocutors for the user. That UI model is the core of this danger. Adding 'guardrails' is the thing that vendors do to avoid addressing the root problem, because if they did address that problem then people would realise that bullshit generators are not reliable tools.

I'd say it's insidious, but that attributes malice to something that can't do that.

That said, the fact that this keeps happening and that the AI companies' responses have been slightly weaker than Nintendo's old "you've been playing for 15 minutes. remember to take breaks :) ok back to the game" messages… be it malicious or massively incompetent, they clearly should not be trusted to make further decisions.