DB 🌱💦

A researcher invented a fake eye condition called bixonimania, uploaded two obviously fraudulent papers about it to an academic server, and watched major AI systems present it as real medicine within weeks.

The fake papers thanked Starfleet Academy, cited funding from the Professor Sideshow Bob Foundation and the University of Fellowship of the Ring, and stated mid-paper that the entire thing was made up. Google's Gemini told users it was caused by blue light. Perplexity cited its prevalence at one in 90,000 people.

ChatGPT advised users whether their symptoms matched. The fake research was then cited in a peer-reviewed journal that only retracted it after Nature contacted the publisher.
#AI #AImistakes
https://www.nature.com/articles/d41586-026-01100-y

Scientists invented a fake disease. AI told people it was real

Bixonimania doesn’t exist except in a clutch of obviously bogus academic papers. So why did AI chatbots warn people about this fictional illness?

Apr 11 at 12:09amWeb
Show threadKevin Karhan1d ago

@dbattistella that's not a mistake, but systemic professional misconduct by those that design #AI and don't #PeerReview papers!

Show threadCharles ☭  is a Green1d ago

@kkarhan @dbattistella

One year, virtually every student in my course claimed a device that hadn't yet been invented was used to create some music at a 1950s World's Faire.

Because somebody had uploaded a paper to an academic server that contained the error. As far as I could tell, it hadn't ever been peer reviewed. But it matched a lot of keywords you might use on a google scholar search and so was usually the first result if you want to read about how music was made for the Philips Pavilion.

I suspect it now has a lot of citations, which presumably causes it to come up at the top of results more . . . .

Anyway, the paper *looked* legit. The relationship between those servers and peer review is perhaps an unsolvable problem.

[Aside: this is part of why the amount of time allotted to marking work can be too short. I can't say a cited assertion is wrong unless I'm completely sure, so I have to go find out what year voltage controlled filters were invented and then go double check that's not a wrong claim made by USians assuming they were first.

I can only imagine LLMs have made this so much worse.]

Show threadpunkcoder -> Still Angry <-1d ago
@dbattistella @adamshostack So its production ready… right?!?
Show threadWind (Vējš)1d ago
@dbattistella
/slow clap/
Show threadVelteropⓐⓊ🪂🇪🇺🇳🇱🇬🇧🇩🇪1d ago
@dbattistella A classic, time-honoured way to test – and expose bad – peer review. And now also AI.
Show threadMarkus1d ago

@minkorrekt das ist doch bestimmt auch was für euch für den Podcast? Mich erschreckt besonders "Even more troublingly, other researchers say, the fake papers were then cited in peer-reviewed literature." damit hätte ich nicht gerechnet. Das AI die Krankheit übernimmt überrascht mich hingegen nicht.

@dbattistella

Show threadgeeeero1d ago
@dbattistella Have you seen this, @davidgerard ?
Show threadAlex P Roe 1d ago
@dbattistella Seems remarkably easy to game these systems. Obviously they are not intelligent enough to know when they are being fooled. Along with hallucination, another little issue that needs resolving before they are used for mission critical tasks even if some already are which is worrying.