Jordan B. L. SmithJan 28, 2023

A conversation with ChatGPT in which I ask it to write a rap about spaghetti without using the letter 'e'.

It gets off to a good start, but then makes careless mistakes — and cheats!

When asked to check its work, it apologizes, but misdiagnoses the issue.

When asked to revise, it replaces ‘stir-fry’ with ‘cook’ — making a legal word choice worse.

Finally I prompt it again, pointing out the first error, and it does well! But in the 2nd stanza, it gets careless (and cheats) again.

Show threadJordan B. L. SmithJan 28, 2023
How was ChatGPT able to write so many lines without an ‘e’? Why did it succeed for most lines, but then make an easily-avoidable error (“staple” could have been “basic” or “mainstay”)? When asked to diagnose, why did it flag the word “stir-fry”, claiming it had an ‘e’? Its outputs made it seem like it “understood” the requirement — it very cheekily deleted the ‘e’s from “noodle” and “sprinkle” — but if it understood, why did it also fail?
Show threadJordan B. L. Smith
When ChatGPT succeeds, it can seem uncanny, and seduce us into viewing it as an interface to a general AI that has vast knowledge about the world, even if it’s only imperfectly accessible. Failures like this remind us that it doesn’t “have” knowledge. It doesn’t “know” that Paris is the capital of France; it has just often seen those words together. Similarly, it doesn’t even “know” that “askew” contains the letter ‘w’; it might seem to, but gives away the game when it tries to explain why:
Jan 28, 2023 at 5:43pmWeb