So the people at Koko— an online mental health intervention laminate that licenses itself out to other platforms to make use of (Tumblr, facebook, etc.) have apparently just admitted that they tested gpt-3 assisted mental health interventions on some 4,000 people in mental health crisis, without said people's informed consent.
See here: http://web.archive.org/web/20230107210632/https://twitter.com/RobertRMorris/status/1611450197707464706
And followup "Clarification" thread here: http://web.archive.org/web/20230107191117/https://twitter.com/RobertRMorris/status/1611775514590740480
Now, at this point, I really shouldn't have to explain to people why this is… very very bad on a number of levels, but to those who maybe still don't get it, and to anyone ever even THINKING of doing anything like this, PLEASE read the "Digital Futures in Mind" Report:
https://automatingmentalhealth.cc/recommendations
This report was funded by the @mozilla foundation, and coordinated by Piers Gooding, with further authorship by Jonah Bossewitch, Lydia X. Z. Brown, Leah Harris, James Horton, Simon Katterl, Keris Myrick, Kelechi Ubozoh, and Alberto Vásquez Encalada (and by way of full disclosure, i wrote the foreword).
It covers the promises, pitfalls, and outright depredations of the Existing interventions in the "AI" assisted mental health space, while also examining and interrogating the proposed use cases and laying bare their implications.
There are several things in the references you should take in, too, but if nothing else PLEASE take the time to read the report itself.
This is a mess that didn't need to happen, and definitely doesn't need to be repeated.