Prof. Emily M. Bender(she/her)Arghh - more problematic reporting, this time about robo-therapists.
A thread:
/1
For the first ~1500 words, exactly 0 people with expertise in psychotherapy are quoted.
/2
They talk up the idea that this is effective because people are more willing to open up to a "bot" than a real person. BUT WHAT IS HAPPENING TO THAT DATA?
(This finally comes up 1000 words further down the article.)
/3
The only studies cited are co-authored by the companies selling this crap.
One of the supposedly positive findings is that people form a "therapeutic alliance" with the bots within "just five days". Not sure how that is measured, but also what happens when the bot can't follow through on what a therapeutic alliance is supposed to be?
/4
When the author finally gets around to reporting on what **actual psychologists** have to say, it's introduced with "What do old-school psychoanalysts and therapists make of their new 'colleagues'?"
This frames the bots as human analogous ("colleagues", ugh) and the actual humans with the relevant expertise as behind-the-times ("old school").
/5
“What if – instead of spending an hour seeing a patient, then 15 minutes writing the clinical encounter note – the therapist could spend 30 seconds checking the note AI came up with?” YIKES
This is a completely unrealistic expectation about what goes into verifying that kind of note and sounds like a recipe for overburdening the medical workforce/setting up errors.
/6
What if -- instead of seeing the process of creating clinical documentation as mere busywork -- the tech bros understood it as possibly part of the process of care?
What if -- instead of leading with the 'gee whiz AI' angle -- journalism in this space started with privacy harms, the fact that somehow tech companies get away with pretending healthcare regulation doesn't apply to them, and chatbots urging self-harm?
/fin
p.s. We covered robo-therapy on Mystery AI Hype Theater 3000 back in September, with Hannah Zeavin:
Episode 13: Beware The Robo-Therapist (feat. Hannah Zeavin), June 8 2023 - Mystery AI Hype Theater 3000
Emily and Alex talk to UC Berkeley scholar Hannah Zeavin about the case of the National Eating Disorders Association helpline, which tried to replace human volunteers with a chatbot--and why the datafication and automation of mental health service...
Allan Chow@emilymbender what if I just made people pay me to let them hear what they wanted AT SCALE?
Debora Weber-Wulff@emilymbender
When Joseph Weizenbaum created Eliza in the 1960s, this nonsense was published:
Colby, K. M., Watt, J. B., & Gilbert, J. P. (1966). A Computer Method of Psychotherapy: Preliminary Communication.
In The Journal of Nervous and Mental Disease, 142, S. 148-152.
1/2
“ […] several hundred patients an hour could be handled by a computer system designed for this purpose. The human therapist […] would become a much more efficient man since his efforts would no longer be limited to the one-to-one patient-therapist ratio as now exists.”
This prompted Weizenbaum to write "Computer Power and Human Reason", a book that is soooo important today.
2/2
Bill, organizer of stuff@emilymbender What if, instead of building whole hospitals with lots of complicated machines and different licensure requirements for different people, we just train up a bunch of medical examiners and buy some freezers?
- a nurse who's worked in software for a long time
Oborosaur@emilymbender I am surprised this isn't illegal. Most jurisdictions regulate the practice of psychotherapy (and similar forms of therapy). It would surprise me if all the engineers on this project are licensed to practice. So... Isn't this practicing without a license?
Magnus Ahltorp@oborosaur @emilymbender Since when have tech bros cared about legality? Operating unauthorised taxi services, taking music from The Pirate Bay to seed their streaming service, just to not name two gigantic tech bro corporations.
mav 
As a security guy, you are definitely asking all the right questions. I suspect you will get no good answers, or at least no positive ones. Any company creating these models will at minimum be using feedback to improve them, and almost all of them are essentially just data mining organizations anyway.
As a person who goes to therapy, this all sounds terrifying. Betterhelp and their ilk gig economy-ing therapy was bad enough.
arestelle@emilymbender I know of a team at a mental health tech company whose (male) leadership is considering using gen AI for a note generating tool and I and another (female) eng have _tried_ to impress upon them how harmful that could be and that they should absolutely not. They acted like they're listening but I can't help thinking that it's only for show and they're going to do it anyway. (Gender mentioned bc it'd be even more par for the course given that.)
@emilymbender ...yup, they're doing it anyway. I'd name them if I didn't fear retaliation, because they deserve to be shamed to hell for this.
The Witch of Crow Briar@emilymbender "What if... the techbros understood..." I've found the problem.
kryphos@emilymbender journalism has already been replaced by AI, for all practical purposes. AI Techbros are banging their own drums in the media.
The remaining niches where manual journalism still prevails has little influence on public debate..
Worik@emilymbender What if the therapist wrote the notes with the patient?
David Adam@emilymbender I don't know about psychotherapists but doctors are falling over ourselves to get AI transcription and summarisation into consulting rooms. We (apparently) really, really hate writing notes, and our insurers feel that more words are always better. There's a dozen plugin products on the market which just plug OpenAI APIs together and the biggest CIS developer in Australia just announced native integration of one of these tools.
It's not cheap but people who use it are talking about going home an hour earlier than usual - that's tough to counter.
@zanchey You gave the answer right in your post: The source of this problem is the insurance companies.
@emilymbender sorry, that's medical indemnity insurers, not health insurers (I'm in Australia). We have universal insurance, there are definitely compliance issues with Medicare that are part of the drive for LLMs, but it is really being sold as a convenience/efficiency thing.
@zanchey I see -- it's going to be convenient until it causes errors in patient care, and then if your system is at all good it will start being REALLY INCONVENIENT for the insurers/providers.
@emilymbender yes I suspect there may be a rapid attitude adjustment at some point
It will always be providers taking the fall and never the tech company
It will always be providers taking the fall and never the tech company
Thomas Depierre@zanchey @emilymbender This is a compliance problem too. The whole stuff is rooted in the belief that enough procedure roots out "errors" and "mistakes" which is... really not how this stuff works out. Risk management and "Safety" have yet to realise the limits of their models of the world...
@emilymbender At that point, it doesn't matter because the bills have been sent and we can always recruit another "patient" right?
trdebunked@emilymbender testing and marketing the efficacy of placebos but with more steps.
Rev. Bussy Body of The Church of America MLM and Shooting Club@emilymbender "alliance"
i think we should force everybody to watch this https://www.youtube.com/watch?v=4JoEWdV7tpQ
i think we should force everybody to watch this https://www.youtube.com/watch?v=4JoEWdV7tpQ
Carnivorous UNDERCOVER caterpillar 🐛😎 | Wild Isles - BBC
Manpageman@emilymbender until it turns on them, apparently. How much liability would a software company have to accept to have a therapist trust their tools?
