today, in attempting to do a close-read of a piece of alleged research to assess how much role a human had in writing it and to what extent the arguments and sources support the key take-aways it suggest
we found a section which purports to describe the oversight that a human performed over an LLM in writing it
and that section has several major non-sequitors that have nothing to do with the subject matter
we're not naming it because the point is not to have a conversation about a specific piece
it's to remind everyone to engage your brains when reading these things. don't be on autopilot, don't let your assessment of plausibility be based on how formal the writing is or anything like that
oh yeah by the way
these newfangled spam generators do tap into several important effects that haven't been a big problem in the past, such as the linguistic thing that the stochastic parrots paper does a great job of explaining (humans assess the credibility of written information by building a mental model of the person who write it, but here, that model is spoofed)
but, also...
@ShadSterling well, because it was highly effective once upon a time. in the days when things had human authors, it was super useful as a way to guess whether the person actually had a point that you just weren't understanding yet, or if they just didn't know what they were talking about.
definitely not entirely reliable, in part because it does get entangled with respectability dynamics that socially mediate who is allowed to speak, and that sort of thing
@ShadSterling plus it just seems to be, fundamentally, built in to the human brain (we are neither a linguist nor a neuroscientist and this is our lay understanding which we're not sure how strong the evidence for it is or isn't)
the task of communicating is the task of taking a mental structure that exists in someone else's brain and building something analogous to it in your own
@ireneista “the task of communicating is the task of taking a mental structure that exists in someone else's brain and building something analogous to it in your own” - I think this is the crux of it.
The thing I thought of was re the theory that the whole reason we have these complex brains and abstract thinking is for social risk assessments.
…
@ireneista But we’re pretty far in to using these brains for external reality and formal logic, with millennia of intentionally crafting knowledge to be independent of individual person-models. And over the last century or two, we’ve become very dependent on technologies built with that kind of non-social knowledge, which I’d guess would be hard to do without a kind of information ingestion that benefits from skipping the person-model-building effort becoming fairly pervasive
…
Oh goddamnit, not having to constantly model other persons is a fucking privilege.
Being safe enough to nurture other kinds of mental processes that support understanding systems fundamentally different from the predictability patterns of people.
So homicidal bigots and machismo enforcers are holding us back in a bigger way than I’d realized.
Irenes (many)
ShadSterling