People are shocked to discover they need to fact-check ChatGPT and I'm having the uncomfortable realization that they never fact-checked their uncle's Facebook posts, their friend's medical advice, or literally anything Google's top result told them, and suddenly the last decade makes sense...

@Daojoan in the early 2000s, my mom was sending me nonsense emails and I was replying with snopes links. In the 2010s, that basically changed to her posting nonsense on Facebook.

So many people don't bother fact checking at all.

@stellarsarah @Daojoan My Boomer sister-in-law (deceased in 2018) sent me (and 50 other people) an email of that sort back in 2003ish. I sent a friendly reply-to-all with a link to the Snopes article debunking the email. SIL wrote back a nasty message about how if I weren't such an ugly person I'd have more friends. ?

She was most firmly fundamentalist-conservative, and that was literally the last time I replied to an email from her or attempted to demonstrate facts to her in any way.

@courtcan ah, my mom never said anything negative in response to the Snopes debunking. I did start these corrections by emailing her individually and then eventually just hit reply all with my debunking after a few attempts. Then one of her friends who was in that email chain thought that my email address belonged to his daughter with the same first name and it got all awkward, but she did stop emailing me those kinds of things.

@Daojoan

@stellarsarah @courtcan @Daojoan not just boomers. had a similar reaction from a millennial around 2010ish to me on facebook

@courtcan @stellarsarah @Daojoan
I remember posting a snopes link on my (now former) sister-in-law's Facebook post about how Obama had banned some country song from ever being played on the radio because it was too patriotic or something. (It was a b-side from ten years prior, not so much banned as never on the radio to begin with.)

Her response was "Well I still believe in things like this."

???

@thecrushedviolet @courtcan @stellarsarah @Daojoan Confirmation bias: people just believe whatever sounds good in their ear-holes.
@courtcan @stellarsarah @Daojoan part of how we've gone so wrong as a culture I think is the norm that correcting people when they're wrong about facts is unspeakably rude
@stellarsarah @Daojoan the horror when a former teacher of mine send me some anti-vaxx stuff and when I sent him link to the [german-language skeptic wiki, I forgot the name], he was already converted and immune to any discussion :/
@jollysea @stellarsarah @Daojoan At least he had some immunity!
@pubby we had a lot of discussion afterwards, and he told me his interest in anti-vax began when he was schedueled for a booster and since he has a fear of needles, he searched for alternatives/decided to "do some research". would be funny if i weren't so tragic :/

@jollysea @pubby many of us had those friends fall down those twisted rabbit holes.

In the early days one friend said, "oh I'm not super extreme like the really antivax folks, I'm just concerned about fluoride."

It went downhill from there.

@jollysea @pubby Yep! A significant portion of antivaxxers are just scared of injections (mostly big tough guys).
@Daojoan The Internet was sold so much as an encyclopedia, a place that was all about facts. And it's so difficult to resell something as what it really is after it's been sold as something else.

@eyrea @Daojoan

The internet is a source of misinformation

@Daojoan

I’m still like deer-in-the-headlights frozen since a colleague earlier this week just casually threw into a discussion about a complex, consequential, and costly legal question “well I checked on ChatGPT and it said…” as though that was some kind of mic drop.

I just wasn’t prepared for this. My jaw was on the floor.

Spoiler alert: ChatGPT was 100% wrong, like exactly the inverse of the true fact, of the complex, consequential, and costly legal question. A yes/no question where the answer was ‘no’ and ChatGPT said ‘yes’.

I bet they’ll keep using it though 🤷🏼‍♂️

@MercG @Daojoan
Went through this 6 mo ago, colleague insisted there is a specific API call, we spent nearly half an hour trying this call that wasn't in the official docs but it didn't work. So we asked where did he get this information, "ChatGPT, I thought it was obv. where the pasted text was from".. (no, genius, I don't use it) GAAH so the f*ckn bot just made up an API call that never existed.

@bike4climate @MercG @Daojoan I've seen this and hallucinated npm-packages too.

An the model? It was Gemini 2.5 pro. Not Claude but still...

@nemeciii @bike4climate @MercG @Daojoan
The "best" part is malicious actors creating those hallucinated npm packages, so now they have full control over the AI-infested application and everything it controls! 🙀

@bike4climate

I have reluctantly just to see what it says entered some very basic like entry-level questions regarding the industrial field in which I work, and the LLM “answers” would quite possibly get people injured, or fired, or sued, maybe all three.

@Daojoan

@MercG
@bike4climate @Daojoan

What gets me is, what about the expectation that these things are supposed to, you know . . . work? I get the sense that people are soldiering through using AI despite having these experiences. It's like using AI is the end, not the means.

@bike4climate @MercG @Daojoan Gemini similarly hallucinated functions in a library for me and, after figuring this out, I went back and looked at the prompts that got me to the fake code. It turns out that I basically just really, really wanted those functions to exist so Gemini made them up for me.

@MercG @Daojoan

I hear you. A colleague recently said that he never experienced any „hallucinations“, as a matter of fact, he finds AI extremely useful. Example? ChatGPT gave him information (in no time!) that he could not find anywhere else… I can only assume that the present technologists stayed silent because there is a lot of pressure in the organisation to adopt AI.

@gastarbajterica @Daojoan

There’s a lot of pressure in Jonestown to drink kool-aid.

@MercG
There is a quite simple explanation for this

LLMs are tuned to produce well-formed sentences with fine grammar

Its the "shape" of the word output that most people take as quality measure, it gives them the feeling that these words must come from an old welleducated gentlemen who knows what he writes

@Daojoan

@Daojoan Another case of LRT (liberal reaction time)
@Daojoan To quote PsyCop Alfred Bester from Babylon 5, “Got it in one, Mr. Garibaldi.”
@Daojoan that’s why disinformation is more powerful
@Daojoan the other day my mom brought to me the most bizarre fact I could ever imagine and when asked where did she get the information from, her answer was some YouTuber in his podcast. These podcasts with half baked research gets so much of views and likes that people take it as a credible source.

@Daojoan people rarely fact check things from Fedi, if they agree with them

Why would you be surprised about the rest of those?

Folks tend to have this attitude that people started out decent, then got caught up in lies online and turned into hateful monsters

I'm more of the opinion the lies just gave them ammunition for what they were already thinking

@codebyjeff @Daojoan I really appreciate you stating this. People want to push this narrative that the quality is so much better on fedi, when it’s not, they’re just less likely to encounter those that disagree with them. But, there’s rampant disinformation & misinformation on fedi as anywhere else!
@Daojoan And they don't know how to fact check. How do you fact check when google's top thing is more AI slop?

@ariaflame @Daojoan I love the AI answer in Google.

With almost every answer it shows me how much nonsense AI is fantasizing about.

@Daojoan And at my elementary school, there is a teacher who took the first (incorrect) Google result and taught the children—including mine—something incorrect. It was only about mountains in my immediate area. So it's not that important if you're not familiar with the area...

@Daojoan it´s like people believe what they see in advertisting and take that as information....

also your post reminded me of people who were annoyed when they had to wash hands because of covid. and I was really shocked that this seemed something new to them.

@Daojoan @paul_edwin And these will largely be the same people who deride Wikipedia for being unreliable.
@brianjohnson @Daojoan @paul_edwin The same people who told us not to believe everything we read online now believe everything they read online.

@Daojoan What also worries me is that Chatbots give different answers when you ask the same question in different languages. Even the sources GPT mentioned were different.

There is really no I in AI.

@Brokar @Daojoan Artificial Intelligence begets Actual Stupidity
@Daojoan Did we accidentally forget to teach critical thinking skills?
@elricofmelnibone @Daojoan The right wing was worried this would hurt attendance at their megachurches, so they scared people that "critical thinking skills" meant teaching people to "criticise America".
@elricofmelnibone @Daojoan I nearly wet myself reading this. Did we? Hell, yes. Anything worthwhile takes effort, and critical thinking takes effort. It becomes easier as we learn those skills. I find it helps if the student is cynical to begin with.
@elricofmelnibone @Daojoan Imo it’s the manufactured lifestyle we’re nudged into: time-poor, economically precarious at any income, stuffing ourselves insatiably on content which is as low in usable information or real meaning as our food is low in nutrition (and to extend that metaphor, due to soil conditions, getting less nutritious all the time even with the best of intentions - soil here might be the entropy of comprehensibility in ever-increasing complexity.)

@Daojoan a person very close to me doesn't trust Wikipedia because "anybody can write there" (in spite of it being one of the few big multi-party curated media), but will happily take as truth any random conspiracy theory someone relays on social media, even if it doesn't make any sense at all once you apply a drop of logic to it...

e.g.: https://fakenews.pl/en/environment/rainforest-alliance-atrazine-and-bill-gates-we-verify-the-narratives-around-the-green-frog-logo/

Rainforest Alliance, atrazine, and Bill Gates. We verify the narratives around the green frog logo - Fakenews.pl

The Rainforest Alliance logo on food products has been causing strong emotions among conspiracy theorists for a long time. The organization has been

Fakenews.pl
@skandalfo @Daojoan The answer to this is that ChatGPT is unreliable because "Nobody can avoid writing there".

@Daojoan
Yeah that makes sense.

By the way who is still depending on ChatGPT for search results?

@Daojoan People do this in person ALL the time as well and I hate it so much

Notice when the people around you preface things with "I don't know if this is true, but someone told me that....".

Then notice people who tell you things WITHOUT that preface.

My M-i-L is the latter type. Fortunately she's surrounded by fact-checkers so the things she hears and says aren't usually complete nonsense.

But I did once tell her that I thought it was "morally wrong" to repeat a claim without clarifying whether you knew the claim to be true of your own knowledge.

@Daojoan I still think about the story a colleague (younger than me, so no boomer!) shared on Facebook about a six year old girl supposedly a survivor of Sandy Hook, running the Boston Marathon who was killed in the bombing at the Marathon. Like, you don't even need to fact check that, dude. How could a six year old be running a marathon you have to be 18 to enter? And finishing with adults? And running a marathon at all when she's SIX?! Just use your brain for like ten seconds! Gah!
@Daojoan this was a huge and interesting problem in the reformation.
@Daojoan had a "Do Xr interaction toolkit have an XRPokeInteractive class?" question two hours ago. There is no such thing, but ChatGPT dreamed it up on the spot!
I was looking for the XRPokeFilter class, but the eagerness to invent stuff on the fly unnerves me. I'll review every line of code it provides.
@Daojoan ChatGPT is nothing more than the game of "someone said something wrong on the internet", but as-a-service. Sometimes it finds some legit information, sometimes it takes stuff out of context or screws information from a pdf because columns are difficult. Then it repeatedly claims it's right.
Sane humans won't win against this firehose.
@Daojoan @becomethewaifu Does no one remember those PSAs that said not to trust random bullshit on the TV/Web/whatever without verifying it?

Surely they still publish modernized versions of those, right?