"New study reveals that when used to summarize scientific research, generative AI is nearly five times LESS accurate than humans. Many haven't realized, but Gen AI's accuracy problem is worse than initially thought."
https://royalsocietypublishing.org/doi/epdf/10.1098/rsos.241776
@gerrymcgovern We were told hallucinations would decrease, far as I can tell, they have increased.
LLMs are completely unacceptable for any purpose beyond shitposting/brainstorming. No LLM content is fit to show another human being, even with "this is from an LLM" warning, let alone without.
The problem is that "hallucination" is a human term to describe when we NOTICE that an AI is wrong. It's not doing anything different when it hallucinates. By it's nature, all of it's output is a hallucination, it's just we've created a term to delineate when it clashes with what we know is correct.
If you're making an AI that is general purpose, >50% accuracy is unlikely. if you're making an AI to solve a specific problem, you can get shockingly close to 100%.
@TheZeldaZone @gerrymcgovern There's an AI trained to detect cancer in biopsy slides, and it's reached something like 98% accuracy, often finding pre-cancer anomalies that experts in the field can't spot.
If you try to write a chatbot that can answer any question anyone asks, you're trying to solve a "Very Hard" problem. The sort of thing that chipping away at a crumb is a PHD thesis.
AI huckersters will have you believe it's inevitable.
@jpaulgibson @TheZeldaZone @gerrymcgovern Truely! I remember when IBM's Watson was going to help fight hospital mistakes by being a diagnostic tool to assist doctors. Except they were probably a decade too early and so spent billions on tech that didn't work.
AI has made massive leaps in protein-folding and helping to design fusion reactors (we'll see if it's ever actually possible). It could do so much good creating climate saving solutions, rather than burning our world to make memes.
Gerry McGovern
TheZeldaZone🏳️⚧️🎮🎀
Michael Lubert
J Paul Gibson