Reid Mar 21
I'm fucking dying here

Apparently they reverse engineered the reasoning behind an ML model trained to find tuberculosis to see how it figured it out and it used the age of the machine that took the MRIs; the actual MRI was just completely ignored
Show threadIván Sánchez OrtegaMar 21
@reiddragon Oh, I assume you know about that 2021 story about ML models applied to photographs of rulers next to a potential melanoma.
When subject to independent reviews, computer vision models for cancer diagnosis have an astonishingly high rate of failure.
https://civio.es/sanidad/2025/07/03/mole-or-cancer-the-algorithm-that-gets-one-in-three-melanomas-wrong-and-erases-patients-with-dark-skin/
Mole or cancer? The algorithm that gets one in three melanomas wrong and erases patients with dark skin

The Basque Country is implementing Quantus Skin in its health clinics after an investment of 1.6 million euros. Specialists criticise the artificial intelligence developed by the Asisa subsidiary due to its “poor” and “dangerous” results. The algorithm has been trained only with data from white patients.

Civio
Show threadReid Mar 21
@IvanSanchez oh yeah, I read about that when it happened but completely forgot about it since. Someone reminded me just earlier about it
Show threadIván Sánchez OrtegaMar 21
@reiddragon I recall another instance, where the high-resoution x-ray machine was used for cases already flagged as potential, whereas routine reviews used the cheaper, blurrier-rez machine. The computer vision algorithm was ultimately trained on the markings at the margins of the x-ray images.
Show threadmike805

@IvanSanchez @reiddragon Military wanted AI to detect tanks in photos.

Procedure: bring in tanks, take photos, remove tanks, take more photos, train AI on photos.

Result: AI can tell morning from early afternoon!

Mar 26 at 3:45pmWeb
Show threadReid 3d ago
@mike805 @IvanSanchez pretty much. You really need to isolate the variance in the data to JUST the thing you're hoping to recognize which nobody training these models bothers with