@emilymbender Faced with complains about this, the government agency says, “But thousands of people were helped!” Sure, but how many weren’t lied to?

I feel like this is yet another instance of our general tech+automation+capitalism problem of ignoring the heavy cost of false positives. The systems are at a scale that makes manual review expensive, and the companies have zero incentive to discuss it, because the customer isn’t likely to notice most of them.

Police departments love image recognition because companies claim they catch N% of crooks. They never talk about the percentage of innocent people caught.

Fraud finding algorithms found tons of fraud in postal offices in England. Nobody looked at how many times they caught the wrong people until it was far too late, and the company hid it when they found out rather than lose the contract. People went to prison, lost jobs, committed suicide. But hey, the system caught fraud.

Facebook lauds the number of accounts banned for hate speech but never publishes the number banned incorrectly. Like most companies selling “AI”, they don’t actually know that number, it requires expensive manual review they don’t want to do. When I was in that org I was told that every quarter they put it on the “ought to do” list, but it never made the cut. In large part, I suspect, because, because developers and teams are rewarded based on how they increase profit, and Facebook have no way to measure what they lose by banning the wrong person. Getting developers there to work on something that doesn’t result in promotion is hell.