JaneQApr 11
DB 🌱💦

A researcher invented a fake eye condition called bixonimania, uploaded two obviously fraudulent papers about it to an academic server, and watched major AI systems present it as real medicine within weeks.

The fake papers thanked Starfleet Academy, cited funding from the Professor Sideshow Bob Foundation and the University of Fellowship of the Ring, and stated mid-paper that the entire thing was made up. Google's Gemini told users it was caused by blue light. Perplexity cited its prevalence at one in 90,000 people.

ChatGPT advised users whether their symptoms matched. The fake research was then cited in a peer-reviewed journal that only retracted it after Nature contacted the publisher.
#AI #AImistakes
https://www.nature.com/articles/d41586-026-01100-y

Scientists invented a fake disease. AI told people it was real

Bixonimania doesn’t exist except in a clutch of obviously bogus academic papers. So why did AI chatbots warn people about this fictional illness?

Apr 11 at 12:09amWeb
Show threadpunkcoder -> Still Angry <-Apr 12
@dbattistella @adamshostack So its production ready… right?!?
Show threadVelteropⓐⓊ🪂🇪🇺🇳🇱🇬🇧🇩🇪Apr 12
@dbattistella A classic, time-honoured way to test – and expose bad – peer review. And now also AI.
Show threadMarkusApr 12

@minkorrekt das ist doch bestimmt auch was für euch für den Podcast? Mich erschreckt besonders "Even more troublingly, other researchers say, the fake papers were then cited in peer-reviewed literature." damit hätte ich nicht gerechnet. Das AI die Krankheit übernimmt überrascht mich hingegen nicht.

@dbattistella

Show threadgeeeeroApr 12
@dbattistella Have you seen this, @davidgerard ?
Show threadAlex P Roe Apr 12
@dbattistella Seems remarkably easy to game these systems. Obviously they are not intelligent enough to know when they are being fooled. Along with hallucination, another little issue that needs resolving before they are used for mission critical tasks even if some already are which is worrying.
Show thread2somethingApr 19
@dbattistella

Okay, "serious" question: what do I have to do to get AIs to start telling people that Rapid Onset Guardian Discernment is real, so I can show the AI response to transphobes who also believe AI is intelligent.

RE: https://transfem.social/notes/adxenr2u57c40448
Show threadSparkpaws™ 5000®Apr 20
@dbattistella Wikipedia vandalism but even more fun. Love to trick the stochastic parrots.