Gerry McGovernMay 20

"New study reveals that when used to summarize scientific research, generative AI is nearly five times LESS accurate than humans. Many haven't realized, but Gen AI's accuracy problem is worse than initially thought."

https://royalsocietypublishing.org/doi/epdf/10.1098/rsos.241776

Show threadbovazMay 20
@gerrymcgovern I was talking with a colleague who sets up "AI" systems for some of our clients. Stuff like chat bots, for searches in specialized knowledge-bases. They estimate that at least 30% of answers is made up or wrong (their optimistic estimate from the actual reports they receive), to the point of the bot going completely off topic (once recommending a list of VPNs to try to a user who couldn't find a procedure).
Show threadGerry McGovernMay 20
@bovaz How staggeringly bad is that? It's astonishing! And yet this is acceptable? But then, having spent almost 30 years working with support content, it's not terribly surprising. Management don't give a crap about support. It's always the cheapest option.
Show threadScott P ✅May 20

@gerrymcgovern @bovaz

The whole point of these chat bots is to make it harder to bother a real human. They hope that you’ll just get frustrated and go away when you find that asking for help is more of a problem than the one you’re trying to resolve.

Personally, I have never had a useful or successful experience with one, other than the few times that it is set up to escalate to a human.

#techgarbage #enshittification

Show threadAndrocatMay 21

@saprentice Almost all of them have a failsafe to cut to a human agent

Not always easy to find.

@gerrymcgovern @bovaz

Show threadScott P ✅

@androcat @gerrymcgovern @bovaz

I will typically just ask .. “May I chat with a human?” .. if that doesn’t work, I’ll curse at it and leave.

May 21 at 2:27pmToot!
Show threadAndrocatMay 21

@saprentice

I wonder if they are just looking for the keyword "Human". I hate wasting breath on an edifice of dehumanization.

@gerrymcgovern @bovaz