@davidgerard , not sure if you saw this great post about ML models in radiology?

The boosters love to say ‘look how good AI is in medical diagnosis, therefore LLMs are good’. Only, it turns out (from the article):

while models beat humans on benchmarks, the standardized tests designed to measure AI performance, they struggle to replicate this performance in hospital conditions. Most tools can only diagnose abnormalities that are common in training data, and models often don’t work as well outside of their test conditions

It also highlights a problem that’s actually quite general in medicine: we have far more data about unhealthy people than healthy ones. I was talking to a cardiologist almost ten years ago who was very excited about the data things like the Apple Watch could collect. Apparently they know that a lot of people who have heart attack have arrhythmia, but they have no idea if this is a meaningful correlation. Healthy people tend to have their heart monitored for a minute or less on a visit to a doctor every few years. People with known heart problems wear heart monitors that can record a load of things, so you have very good data on their heart rhythms but no baseline to compare it against.

This is also true for radiology. You really want to do anomaly detection: take a few million scans of healthy people, wait a few years to see if any of them have undiagnosed conditions, and then use that dataset to train a model of what a healthy lung (or whatever) looks like. Then feed new scans to the model, have it flag anomalies, and loop in an expert to figure out what kind of anomaly it is and whether it’s important.

But what you have is a load of very examples of things that are wrong, in very specific ways. And these also have artefacts that are specific to individual devices, so it’s easy for a model to learn that people who are scanned with this class of machine have this condition.

And that’s just the start of the issues they discuss.