Mastodawn

My fellow journalists, I need you to stop talking about the Turing Test; it was never a good metric. Turing was a computer scientist, not a psychologist. ELIZA passed it, and also was a decent Person-centered therapy bot. Some people actually used it that way, even knowing it was a bot.

A) There are many paths to the top of a mountain, and

B) LLMs and other AIs aren't real girls and boys just because we feel they are.

DressToKILT 3d ago

@quinn I know of humans who can't pass a Turing test so yeah, not a great metric.

Quinn Norton 3d ago

@dresstokilt I think we all fail it pretty often. I still talk to my plants 😁

John Rohde Jensen 3d ago

@quinn It's great fun to make two Elisa bots talk to each other. The result is most certainly NOT intelligent

Quinn Norton 3d ago

@johnrohde that is funny, i saw that back in the day. but then i've also talked to people who have made about as much sense.

@quinn i suppose the question is, do we have a good test to replace the turing test with? is there a good alternative that people should be focused on instead

Ray McCarthy 3d ago

@imaguid @quinn
No, there isn't, because decent computer applications or useful "AI" are each domain specific and need domain specific tests.

txerren 🇵🇸3d ago

@quinn thank you for this

Joe DiMaggio 3d ago

@quinn The Turing Test is not so much a measure of how good your AI is as it is a measure of how easy it is to fool people.

Quinn Norton 3d ago

@joed it's not even just fooling people, people believe things that can help them, and sometimes that means those things *can* help them.

dogzilla 2d ago

@joed @quinn I think the Turing test is more subtle than that. If your only means of communication is remote - like it is between you and me right now - then is the question of whether the entity on the other end a human or computer a distinction without a difference? Turing was testing the edges of computing theory (not computers but computing).

Ray McCarthy 3d ago

@quinn
The Turing test was only an idea. It was never a serious metric to measure AI.

Also a computer program that can merely "chat" but isn't 100% following instructions and is only 40% accurate is only a toy.

Eliza is limited because the responses are built in. So were others inc 1990s Internet ALICE. Scraping mostly un-curated internet content, inc copyright material simply makes it more realistic, not more useful. It's still a toy.
Eliza is still in Linux emacs,

Ray McCarthy 3d ago

@quinn
Turing also thought chess would need AI. It didn't.
He was a clever mathematician and computer technology was still in it's infancy in 1954 when he died (either by accident or suicide, because he'd been experimenting with dangerous chemicals).
Leo Computer (Lyons, 1st commercial) 1951
First full size transistor computer 1955
ForTran 1957
Cobol 1960
1964 1st commercial MOS ICs.

LLMs are not intelligent at all and less useful than late 1980s AI "Expert systems" using curated data.
.

Marianne 2d ago

@quinn yes! My pals did a pod covering this too, may be of interest
https://www.merseysideskeptics.org.uk/podcasts/skeptics-with-a-k/episode-436
#TuringTest #AI

Episode #436 | Skeptics with a K | Merseyside Skeptics Society

<p>Mike asks if the Turing Test is really a meaningful tool for gauging the state of artificial intelligence, and the team has the last few announcements for QED 2025.</p><p>Online tickets for the final QED are still available, <a href="https://qedcon.org/"><b>pick up yours today</b></a>. You can also chat with us on the <a href="https://discord.gg/d5tN8vaFsy"><b>Skeptics in the Pub Discord server</b></a>.</p><p>You can sign up for the Skeptics with a K Patreon at <a href="https://patreon.com/skepticswithak,"><b>https://patreon.com/skepticswithak,</b></a> or to support Merseyside Skeptics as well as the podcast, donate at <a href="https://patreon.com/merseyskeptics"><b>https://patreon.com/merseyskeptics</b></a></p>

grrl_aex 2d ago

"we"? "we" who?

The LLM ai project has ONE purpose: to create clones of human minds in order to render human minds economically obsolete.

Period. That is the project.

And what happens to these economically obsolete humans?

"Let them eat cake."

Eliza never passed the imitation game and wasn't structured to plausibly take it.

I don't think the current language models are quite up to that particular test either (certainly not with the default settings), but they're in the ballpark. It is certainly true that the limits of a purely linguistic test have become transparent of late.

It's a far better test than chess playing, certainly. On the other hand, Goodhart's law applies.

Amorpheus 2d ago

@quinn Most llms will state that they are able to pass the turing test if you ask them. Best follow up question then is: "How are you feeling today?"

minentromaxinfo 2d ago

@quinn The man was a genius but no modern person in tech would still believe it's a good metric to evaluate machine learning algorithms.

Quinn Norton 2d ago

@minentromaxinfo yes, he was a genius in his moment. Amazing man. But imagine if you handed him an iPhone.

minentromaxinfo 2d ago

@quinn If I ever get my hands on a time machine after sending baby Trump to the moon I would love to get Lovelace, Charles Babbage, Turing and Claude Shannon in a room and give them an iPhone.

Quinn Norton 2d ago

@minentromaxinfo fuck ya, invite me

Alison Chaiken 2d ago

@quinn Having failed a Turing Test myself during the Zero1 Art Festival in San Jose in about 2010, I'm not a fan of the Turing Test. Perhaps I'm actually a bot and don't know it. That would make a great Hollywood movie, perhaps written by Margaret Atwood or Philip K. Dick.

Quinn Norton 2d ago

@alison I've made friends with a rock that I'm still convinced was magical.

David, a Bostonian in Tokyo.2d ago

If you read Turing's paper, and think about it, you would realize two things.

One: What everyone thinks the Turing Test is has nothing to do with what Turing proposed

Two: Turing's test was actually subtle: it asks the computer to be empathetic enough with human men and woman to be able to perform as the man in a game where a man is trying to persuade someone he's a woman.

Turing was one of the smartest blokes of the 20th century, unlike our illiterate 21st century AI bros.

Quinn Norton 2d ago

@djl So I have, but it has been a minute, and I do know about the gender part. (and how it's kind of a bit problematic by current standards, but you know, gay man of the time, honestly better than many, though still wildly misogynistic)

Quinn Norton 2d ago

@djl It still doesn't work, because however you try to make it a question about computing it's really gated by human perception, motivation, and imagination. whether this original form of the turing test is passed is as much a matter of how the human slept and whether they have a stomach ache as it is about programmatic mimicry. honestly, we just shouldn't have taken it so seriously. it was a bit of noodling of a very smart man slightly outside his field, and we've made it a Whole Thing.

Quinn Norton 2d ago

@djl i do not blame Turing for this at all, he just said a thing. it wasn't even his main gig. it's a a bunch of bros since who blew it up into whatever the fuck we're doing now, inventing AI god and sad shit like that.

David, a Bostonian in Tokyo.2d ago

By the way, I got into what could have been an ugly fight with Daniel Dennett. He said something stupid (in a reply on a random blog) about the Turing Test being about fooling people and I called him on it, referencing a Yale AI Lab paper by Roger Schank.

Dennett claimed that he was the one who told Roger that idea.

What I didn't say was "Sheesh. If you know what the Turing Test is actually about, why the hell did you say something so dead wrong?"

But I sure thought it.

Quinn Norton 1d ago

@djl Dennett is such a jackass.

David, a Bostonian in Tokyo.1d ago

Being slow, I only recently started to figure that out. We (cog. sci. end of AI) were short of friends back in the day and were guilty of grasping at straws.

I've been reading a bit of Jerry Fodor recently, and he's serious fun. Lots of snark and it's good brain excercise for philosophy tech terms.

Quinn Norton 1d ago

@djl one small benefit of being a woman in tech for decades is that your jackass radar gets sharp as hell.

David, a Bostonian in Tokyo.1d ago

Yep. I have some horror stories from back in the day.

The place: An AI conference.
The speaker: A woman. Ms. S.
The time: The Q&A session.

Q: I'm not really happy with this approach. Do you know the paper by Ms. B?
Ms. S: Yes, but this work improves on that.
Q: No it doesn't. You clearly don't understand that paper. Ms. B. really got it .
Ms. S.: I assure you, I'm well aware of that paper.
Q: No, you clearly don't understand it.
Ms. S: I understand that paper just fine. I wrote it.

Quinn Norton 1d ago

@djl yeeeaah that... Happens.

David, a Bostonian in Tokyo.2d ago

"we just shouldn't have taken it so seriously. "

But my point is very few people took it seriously. Pretty much everyone who comments on the test hasn't read the paper, or didn't read it carefully.

Again: it's not about the computer fooling people, it's about how well the computer plays the role of a man pretending to be a woman.

It may not be a good test, but since we've gone off on a stupid tangent, we haven't thought about the actual idea and don't know if it works/helps.

David, a Bostonian in Tokyo.2d ago

I agree that it hasn't aged well.

And would need to be rethought for our current ideas about gender and identity.

Still, asking the computer to have empathy for people with different gender (or other) identities seems a rather good idea.

People are easily fooled. Human intelligence seems practically designed to be fooled, since anthropormophisation is such a powerful tool/heuristic for dealing with things that really aren't anywhere near human.

So we need something more subtle.

Quinn Norton 2d ago

@djl I think asking computers to have empathy at all is a terrible decision, because it anthropomorphizes a Turing machine and that is always going to end in tears. and probably blood.

David, a Bostonian in Tokyo.2d ago

I get that. But.

AI (done right) is a branch of cognitive science, and how empathy fits in with cognition is a valid question for cog. sci.

Programs informed by cognitive science might handle empathy sensibly. Might. Some day in a distant future when the current round of AI bros have been chased away. And we have figured out the stuff (basic cognition) we failed at so badly in the 70s and 80s.

Will we make the necessary progress in cog. sci. for those ideas? Maybe, maybe not.

SpaceLifeForm 2d ago

The Turing Test was always backwards.

These days, the bots want you to prove you are not a bot.

MidgePhoto 2d ago

@quinn perhaps Turing Test has migrated in meaning with time?
Hoover used to be a trademark.

#SciFi William Gibson's #Turing are a police department controlling and surveilling intelligent machines in #Neuromancer

@quinn I did see someone once suggest that when it was first conceived it was a reasonable metric, but as soon as it escaped containment it became a target, and then Goodhart's Law took over and the rest is history.