justinethis propublica article, on police officers' and prosecutors' use of linguistic analysis to identify "guilty" 911 callers, is stomach-churning.
https://www.propublica.org/article/911-call-analysis-fbi-police-courts
something i think the article does a really good job of is illustrating how a method becomes empowered and legimitised -- here, grounds for putting people in jail for murder. (there are emails! stunning.)
the method's validity doesn't matter -- the point is, it's been successfully sold as a valid move in a long-standing game of getting convictions.
and yes, the method is junk. the original study does not replicate, per multiple groups of FBI analysts and academic researchers. explains a social psychologist in the article:
"Given the gamut of human emotion, anyone claiming to know the right and wrong way to speak during an emergency has seen too much television."
i've been thinking about how the line between what's considered good science versus gross [application of] pseudoscience is awfully thin. "911 call analysis" feels gross to me in part because it feels familiar. there's a particular way that text analysis is framed in the academic papers in my orbit: in this work we identify linguistic indicators of a fuzzy human thing.
stripped of the scientific trappings, it sounds clickbaity -- "here are the top 10 words you use if you're a []". dressed up, the story has a ton of power -- in NLP, computational social science, and apparently law enforcement.
consider the intersecting beliefs that such a method is at home in --
that there's a certain relation between language and its users,
that a key challenge in administrative interactions is assessing a citizen's worthiness,
that the words people say can and should be used to make inferences and decisions about them, in settings such as the criminal justice system.
so, the call analysis method is junk science. but, had it been good science -- i.e., replicated, run with more data that was shared with other researchers, had more caveats attached to it in publications -- would it have been any better?