I get to see this in action. Doctors want transcription and summarization services because of the challenges they face getting quickly familiar with a patient in a crazy short amount of time. They also want to automate notetaking for rounds, which can be chaotic. Problem is, these tools suck in chaotic situations, and even in relatively normal ones, hallucination abounds.

There will always be a claim of human review, but I know all too well that it's working against the current to have a human reviewer not assume the model got it right. What's more, those safeguards will eventually be seen as cost centers and redundancies—well, at least until the lawsuits.

One other thing. As noted above, these model-generated fields in charts are a) being used as training material for other models, and b) being used as input for other generative tools without human review. The potential for compound errors and model collapse is immense.

@mttaggart I was cynically thinking to myself "and what are the chances that an industry-loving institution like the Ontario government had any conclusion other than 'well we'll just choose to use Good AI and that will be fine', probably 100%" and jumping to the report's conclusions,

• establish KPI targets to measure and track Microsoft Copilot Chat’s adoption
• take actions to increase use of Microsoft Copilot Chat to the targeted rates and usage in the OPS

• educate OPS staff through AI training about the dangers of using non-Microsoft browsers when accessing AI websites

So, yeah, they did an audit showing LLMs are wildly unreliable and . . . concluded they should encourage use of Microsoft LLM products.

Their audit criteria also included "having due regard for economy".

@mttaggart we all want the autodocs we were promised in sci-fi, but genAI is not that.

PS: medicine is already a field suffering heavily from biases. Adding automated bias at scale is gonna literally kill millions more of us.

@mttaggart

The Ford government's approach to slop looks, well... sloppy.

At a time when OPS staff morale is at an all-time low and the relationship of government members and their staff to what was once a model public service remains one of ideological hostility.

@mttaggart

was the machine EVER right?

@mttaggart I liked* the bit where they only gave accuracy of transcription a 4% weight but made-in-Ontario was 30% of the score when determining which tool was best for purpose.

* hated, but appreciated the accidental honesty

@mttaggart I think this technology is good for a casual, routine meeting at a company that's making products where no lives depend on them.

Errors correct themselves.

To use this in diagnostic and medical care is beyond reckless.

Errors in these fields cause suffering and death.

@mttaggart

You know who must love this result? Attorneys for malpractice plaintiffs.

With the chart no longer an authoritative, or even plausible, record of what happened, a malpractice lawsuit will just come down to who can tell the jury a better story.

@mttaggart

This is what testing in production gets you!