PwnallthethingsOpenAI's "was this text written by an AI" classifier is going to cause so many problems https://openai.com/blog/new-ai-classifier-for-indicating-ai-written-text/
The problem isn't the false negative rate -- saying "this text wasn't written by an AI" when it was -- but the false positive rate -- saying "this text was written by an AI" when it wasn't. And it's *9%* false positive on the claim "this was likely written by an AI"
The chance that someone integrates this into a plagiarism test and ends up failing students, or job applications, or reports, or anything else where there's some text where a false claim of plagiarism is going to be hard to prove but have severely negative consequences to the accused is going to be large, and cause a lot of problems before its worked out
The problem is that the tool fails the fundamental rule of "should I use AI to solve this problem".
AI is safe to use *only* if you meet the following criteria:
1) The output doesn't matter (e.g. writing a story for your kids is safe)
2) Someone qualified assesses the veracity of the output prior to using its decision in a way that could cause harm.
The problem is this tool will *always* fail these two tests (so long as its false positive rate is > 0%).
There are virtually no ways to use the tool that are objectively assessable -- there are no experts who can verify its output without external information that invalidates the point of the tool -- and virtually every case for knowing if text is written by an AI is about making a judgement decision regarding the author where failing it is harmful
It's worse than existing plagiarism tools, which say "this text is probably plagiarized from [this other text]", because there you can go and look at the other text and see if there's context that's missing. Like, maybe you're "plagiarizing" a properly-attributed quote, or the author of the other text is also you.
But here it's just "expensive magic computer brain says this student is a fraud", and administrators are going to assume it's true, and have few ways to independently validate it
The power of AI is its ability to *enhance* humans to make decisions, process information, and automate time/mentally consuming but generally pretty low-stakes tasks (like drafting an email).
But when you step past that into delegating life-changing decisions to an AI, either explicitly (loan company says no) or implicitly (magic box says 12% yes and human operator develops rule of thumb to press "go" if number > 10), you're going to run into a world of Kafkaesque garbage really quickly.
BoredPeter@Pwnallthethings Don’t you think AI has the potential to be less biased than humans? The best part is it is easily testable too. Throw a bunch of test cases at it, in your loan example see the rates of loans for black vs white applicants, or even remove race from the system entirely. Humans have proven they’re extremely biased, just look at US history of redlining. As long as there is an audit trail and we have control of it I don’t see an issue.
calcifer 🧯@BoredPeter @Pwnallthethings AI trained on a corpus of human decisions will amplify the bias in those decisions. This is a Hard Problem.
Ralston Elko 
@BoredPeter @Pwnallthethings
No. Ai is a creation of humans and will amplify their biases. Read Cathy o Neil’s weapons of math destruction for more on this
No. Ai is a creation of humans and will amplify their biases. Read Cathy o Neil’s weapons of math destruction for more on this