Mastodawn

⟦ˈmartiːn ˈʃtaɪ̯n⟧

Elen Le Foll 🇫🇷 🇬🇧 🇩🇪

I was only made aware of this (frankly awesome) case of LLM poisoning today: https://www.nature.com/articles/d41586-026-01100-y. A researcher made up a disease and published two evidently fake preprints about it (including sentences such as “this entire paper is made up” and “Fifty made-up individuals aged between 20 and 50 years were recruited for the exposure group”), which were almost immediately picked up by LLMs and documented in their output. Worse, actual – supposedly serious – medical papers also started citing the preprints, demonstrating that academics relying on LLMs to do their work is a genuine problem! Not that I had my doubts but, if anyone did, this seems like the perfect demonstration of the problem. Article immediately added to the syllabus of the class I am co-teaching with Iris Ferrazzo on LLMs for Romance Studies/Humanities!

#LLM #GenAI #academia #research #ResearchIntegrity #humanities

Scientists invented a fake disease. AI told people it was real

Bixonimania doesn’t exist except in a clutch of obviously bogus academic papers. So why did AI chatbots warn people about this fictional illness?

Ку 🇧🇬🇪🇺

neobear_computing

@ElenLeFoll the "reasercher"'s name Lazljiv Izgubljenovic to a speaker of most slavic languages would read sort of like "Lying Loser" 😆😆😆

Grimblob May 13

@ElenLeFoll as a lifelong sufferer of Bixonimania (undiagnosed by any medical professional) I find this article offensive. ChatGPT told me that I asked a really excellent question when I asked if I had Bixonimania, so there.

J.J.Fux 1660-1741 May 13

@pineywoozle ‘s #3WordNote May 13

@ElenLeFoll I love this quote about the original article. “Acknowledgments also thanked “Professor Sideshow Bob” and a professor from the Starfleet Academy for access to a lab aboard the USS Enterprise” 🤣 🤣 🤣 #AI #AiSlop

@Pineywoozle @ElenLeFoll

Wow. It couldn't be more obvious, but this dreck is infiltrating *everything.*

Petra van Cronenburg May 13

@Pineywoozle Yes, big cinema ! 🤣🍿 @ElenLeFoll

Markie 🤸May 13

@Pineywoozle @ElenLeFoll what a fun piece of work really, to write made-up scientific articles with all the silly things you want!

@ElenLeFoll The genuine papers citing this is terrifying, are some scientists not reading their own papers any more? 😬

bencourtice May 13

@FediThing @ElenLeFoll people citing papers they have not read/understood sadly is not new

серафими многоꙮчитїи May 13

@ElenLeFoll Chris Morris would be proud:

https://publications.parliament.uk/pa/cm199596/cmhansrd/vo960723/text/60723w10.htm

Meneer de Bruin 🚂🍉🔻May 13

@ElenLeFoll Good. Poison as much as possible, let it eat its own crap and dance while it burns to the ground.

Petra van Cronenburg May 13

@MeneerDeBruin As a professional 100% human writer, I'm indeed interested in how we could use our creativity for that goal! 😁 @ElenLeFoll

Malcolm Herbert May 13

@ElenLeFoll something for you I think @minxdragon

J. Rosenbaum (they/them)May 14

@mherbert @ElenLeFoll I saw that paper! Fascinating, thanks

Graeme 🏴󠁧󠁢󠁳󠁣󠁴󠁿May 13

@ElenLeFoll Yes, saw this at the time - but it's good to remind everyone of what comes out of artificial idiocy LLMs!

nicole mikołajczyk May 13

a Google spokesperson said such results reflected the performance of an earlier model. They added, “We have always been transparent about the limitations of generative AI and provide in-app prompts to encourage users to double-check information. For sensitive matters such as medical advice, Gemini recommends users consult with qualified professionals.”

that would mostly work well if they released 'AI overview' as an opt-in feature instead of forcing it on users who have some trust in Google built over the past decades and don't expect it to suddenly start making stuff up lol

Urban Hermit May 13

@mkljczk @ElenLeFoll
Good point.

Google, if Gemini is as useful as you hope it will be, it is inevitable that it will just come to be known as "Google" and AI answers to direct questions is just a feature of Google search.

Google, if Gemini is unreliable and can not be reliable, why are you letting it tarnish your brand?

posniewski May 14

@mkljczk @ElenLeFoll They say to always check the answers that Gemini provides, but there's no way to do so any more. "Just Google it" used to mean you got some kind of pretty OK answer. Now it provides either an OK answer or absolute lies, and there's no way to tell which is which. The tool I would have used in the past to help find the truth was Google. Now what?

Google squandering its place as the source of the world's information is so disappointing.

@ElenLeFoll that iMac is approaching his 30s. he's earned the right to be a hypochondriac sometimes.

Truth Or Consequences ✅ 🇺🇦May 13

Some do it, DELIBERATELY with a lot of other subjects too. AI is amoral especially if the material it is fed is amoral and we know a lot of those tech guys and politicians ARE amoral.

Truth Or Consequences ✅ 🇺🇦May 13

The obvious end goal of AI is centralized control of information that can be used to bend public opinion, win elections for pedophiles, criminals and set trends.

https://cybernews.com/ai-news/gpt-grok-push-paid-results-flights/

VelteropⓐⓊ🪂🇪🇺🇳🇱🇬🇧🇩🇪May 13

@ElenLeFoll LLMs, like seagulls, swallow anything that's thrown at them. It's known as 'gullibility' for a good reason.

Cat Hicks May 13

@ElenLeFoll I'm sorry, they put up fake preprints and then said other researchers citing these preprints are the problem? Standing up a fake preprint is absurdly unethical

Elen Le Foll 🇫🇷 🇬🇧 🇩🇪May 13

@grimalkina I see your point, but it seems to me that the researchers did a good job of balancing the pros and cons, here. I fear that this experiment was necessary (though it will not suffice) as a wake-up call to some who currently desperately need it (including in academia!). The researchers apparently sought approval from an ethics board before going ahead with this, they specifically chose a "condition" that is not life-threatening, and the two preprints have now been withdrawn.

Cat Hicks May 13

@ElenLeFoll as a former IRB administrator I wouldn't have approved false medical info in the public ecosystem ever, of any kind

Cat Hicks May 13

@ElenLeFoll you damage the entire preprint server doing that

always tired May 14

@grimalkina @ElenLeFoll

I'm not sure. If being fake was made very obvious in many places in that paper, and there has been ethical board approval, what's unethical?

If the "this is actually fake and an experiment about LLMs" were printed only in small letters at the end, it would be different?

Cat Hicks May 15

@project1enigma @ElenLeFoll I wouldn't approve doing something that violates a preprint's conditions of use & that could come in front of patients. Health evidence is considered a highly sensitive category exactly because of the way it feeds into the ecosystem in the way these researchers took delight in doing and it truly does not matter how "trivial" the condition you are making up fake shit about. I don't give a fuck if you hate AI, if you seek to damage patients USING AI you're doing harm.

Cat Hicks May 15

@project1enigma @ElenLeFoll You think people only deserve to not be harmed by fake health info if they are super diligent and read papers? That's not how health research ethics works.

Cat Hicks May 15

@project1enigma @ElenLeFoll I hope they get hit with a lifetime ban, just as arxiv is going to ban anyone who submits fake references that came FROM AI.

@grimalkina @ElenLeFoll Reminds me of when people at the University of Minnesota deliberately submitted buggy patches to the Linux kernel “for science“. So now maintainers default-reject all submissions from an umn.edu email address.

Intentionally buggy commits for fame—and papers

A buggy patch posted to the linux-kernel mailing list in early April was apparently the last s [...]

LWN.net

Elen Le Foll 🇫🇷 🇬🇧 🇩🇪May 15

@rigrig @grimalkina Thanks for sharing, I didn't know about this!

David, a Bostonian in Tokyo.May 13

This is a rerun of the Sokal Hoax.

https://en.wikipedia.org/wiki/Sokal_affair

Like some here, some folks thought the Sokal hoax was ethically problematic, but the postmodernists needed a wake up call, as do the LLM fans.

Really: the LLM idea is the stupidest* thing to come out of Computer Science ever. We need to be embarrassed.

*: Unnecessary explanation: the idea that random text generation has something to do with intelligence is really really stupid.

Sokal affair - Wikipedia

@ElenLeFoll I'm convinced one of the main reasons we die is that we get too old or too sick to manage our own healthcare. We can't do the research to find the right studies, we can't read or understand those studies, and we can't question our providers to be sure they actually understand what's wrong with us.

Once we are dependent on mere employees, the quality of care goes way down, and mistakes get made, or our treatment is just ineffective.

@ElenLeFoll You might enjoy my recent experience with raclette maximalism.

https://exple.tive.org/blarg/2026/05/02/lies-damned-lies-and-stochastics/

Lies, Damned Lies And Stochastics | blarg

The Turtle May 13

@ElenLeFoll i *like it a lot* when people teach AI... wrong stuff. Piss in the well at every opportunity, folks.

It seems like the real underlying problem is the "publish or perish" syndrome, where the value of a researcher is based on how many papers they write, or how often they're reference.

So there's a proliferation of papers, many of which are meaningless, and which no one has time to actually read, being referenced in other papers by other researchers who don't have time to read and evaluate all these other papers.

Christian Schwägerl May 13

@ElenLeFoll The whole thing is like an oil spill.

Quinn Comendant May 13

@ElenLeFoll Unfortunately it’s impossible to test this now that the veracity of the study has been revealed.

Elen Le Foll 🇫🇷 🇬🇧 🇩🇪May 13

@com I was rather hoping that the authors had published an article documenting the LLMs' original outputs prior to the whole thing being exposed, but I cannot find such an article. I do hope that they will publish one soon as it seems important if we are to draw lessons from this. Now there's even a Wikipedia article on the "condition" so it's really no surprise that LLMs are no longer reporting about it as if it were a real condition.

Twotired May 13

@ElenLeFoll This is why doctors recommend that women get screened for bixonimania annually, and men get screened every fifteen seconds.

Shafik Yaghmour May 13

I recently experimented with asking medical information from LLMs and the results were way worse than I expected. Anything not totally common knowledge they got wrong at least half the time. Just checking the sources they provided quickly showed this.

Fardels Bear May 13

It's interesting they chose that particular model of a Mac to illustrate an anthropomorphic computer. I had one, and really loved that quirky design. More of a personality than a piece of technology.

Francis Mangion (M)May 13

🐈‍⬛David Sommerseth May 13

@ElenLeFoll I'm just waiting for a twist in this story. That even this story is made up, with the help of LLMs ...

@ElenLeFoll Reminds me a bit of the paper stating Covid was caused by by Zubats and then that paper got cited by a bunch of conspiracy papers even though 1) Zubats are pokemon, and 2) the Zubat paper credited Bruce Wayne

bookandswordblog May 17

@ElenLeFoll the case of FamilySearch removing the word Nego and references to race from 18th-century manuscripts it scanned and summarized was striking too https://bsky.app/profile/caitlindeangelis.bsky.social/post/3mlrieau4h227

Caitlin G. DeAngelis (@caitlindeangelis.bsky.social)

Recently, FamilySearch digitized and uploaded tons of microfilmed records, including many from 18th-century Massachsuetts. They're using some sort of AI to transcribe/summarize the handwritten documents. I've noticed that the AI strips out references to race and enslavement in 18thc documents.

Bluesky Social

Elen Le Foll 🇫🇷 🇬🇧 🇩🇪May 17

@bookandswordblog Thanks a lot! This example will also make for excellent awareness raising!