But I'm not all sure we're sunk just yet.

It seems to me is that what is happening is that AI systems are creeping their way up Bloom's Taxonomy (image under CC license from Vanderbilt University Center for Teaching).

With GPT3 and the likes, they've gone from being good at looking stuff up (level 1, remember) to being able *fake* (level 2, understanding.)

An aside: they don't actually *understand* anything, though this is a separate thread.

The other remarkable thing is the way that these AIs have leap-frogged to the top of Bloom's taxonomy. I think, though my mind could be changed, that it is accurate to say that they are able create original work (but not, mind you, understand what they've created.)

That seems scary because it gets around the sorts of prompts we might have used for online open book exams in the past.

Here I ask the AI to write a fable about a group of #shoebills who succumb to #GoodhartsLaw.

IMO, remarkable.

However, I've been asking GPT3 to answer my old exam questions and I'm finding that it does very poorly on most of them.

(A large fraction of them involve data graphics in some capacity, and so unfortunately I can't test these.)

Where it seems to be falling short is on analyzing and evaluating arguments.

Here, the AI knows the definition of mathiness (from my book no less) but #McKinsey Consulting fooled it with their silly trust question. It even doubles down and defends the equation!

This question, also about mathiness but from a previous year's exam, nicely illustrates what GPT3.5 is good at and what it fails at.

Here I asked the students to come up with a mathiness equation, and then to call BS on it.

The AI is very good at creating. It's mathiness-style formula for BS detection ability is pitch-perfect.

But it totally fails in its critique and misses the core point.