Mastodawn

Something a bit worrying to note about using Ai in healthcare.

I’ve had two specialist appointments recently, both using ai to transcribe. Both sent report letters with inaccuracies about my diagnoses and past medical history. Even my GP was like, “huh, that directly contradicts what I put in the referrals.”

I have followed up both and requested amendments (which were done) but if I hadn’t, these inaccuracies could have significantly damaged ongoing care, further treatment or insurance claims.

Human error has always been a factor, but both doctors were clearly using the ai software and assuming what it spat out was correct. They made no other notes during the appointments to cross-reference and double check. This is how Very Bad Things can happen.

Show thread

Tee Rex 🏳️‍🌈🦖Mar 27

Two friends have told me in the last week they had similar issues happen. One had an incorrect diagnosis listed before they had a procedure done. The other noted viral not bacterial infection (although they did at least get the medication they needed). I feel like I’m being a pain in the bum going over everything and requesting corrections, but I’m seeing so many mistakes, to the point where any human reading them would immediately say “that doesn’t even make sense”. I worry for those who don’t or aren’t capable of checking these things. Sure, using ai might save the docs 10 minutes per patient in the ER but is that really worth the risks?

Show thread

Tee Rex 🏳️‍🌈🦖Mar 30

Given the interest in this, and similar experiences shared, I think it’s important for me to say that I am not strictly anti-ai in a medical situation. There are a lot of powerful and effective machine learning tools that have been used in medicine for quite a while before the gen ai boom began. I’ve personally benefited from ai-supported skin cancer screening, for example.

What I believe is essential is some really hard baked in guard rails for this kind of software. At the very lowest end, stringent keyword checks that calculate probability of accuracy and flag any concerns for human professionals to assess before any information gets passed on. The “transcribe and summarise” tools in medical settings need to have a very high level of trustworthiness and very low acceptable margin of error.

I just don’t think the cake is fully cooked yet, but we’re eating it anyway… and maybe getting sick.

Show thread

Phil Stevens

Mar 30

@bloodflowersburning Way back in the '90s I worked at a software company doing medical imaging. Even then one of the products was a computer-aided analysis engine to assist with the reading of mammograms.

The CEO was a radiologist and one of his insights was that even the best docs in the field had a limited shelf life because after the age of 40 their eyesight inevitably declined...just as their level of experience started to kick into high gear. He believed machine detection of subtle features at high resolution was the answer. Pick up on anomalies and flag them for the human expert to assess.

Show thread

Tee Rex 🏳️‍🌈🦖

@phil_stevens Absolutely. And this just highlights how using it as a middleman is a useful function, it should just not ever be where the buck stops. If we really want to argue that ai is a “tool” we need to consider that a tool is only as effective as the individual who uses it, and the collaboration between the two.