today, in attempting to do a close-read of a piece of alleged research to assess how much role a human had in writing it and to what extent the arguments and sources support the key take-aways it suggest
we found a section which purports to describe the oversight that a human performed over an LLM in writing it
and that section has several major non-sequitors that have nothing to do with the subject matter
we're not naming it because the point is not to have a conversation about a specific piece
it's to remind everyone to engage your brains when reading these things. don't be on autopilot, don't let your assessment of plausibility be based on how formal the writing is or anything like that
oh yeah by the way
these newfangled spam generators do tap into several important effects that haven't been a big problem in the past, such as the linguistic thing that the stochastic parrots paper does a great job of explaining (humans assess the credibility of written information by building a mental model of the person who write it, but here, that model is spoofed)
but, also...
old-fashioned automation bias is a very strong factor in why people take them seriously, too
wikipedia gives a description we like: "Automation bias is the propensity for humans to favor suggestions from automated decision-making systems and to ignore contradictory information made without automation, even if it is correct."
it's well-studied, going back many decades. people just... assume the machine must have had some reason for saying that, and discount their own judgement.
@ireneista I always thought this was a variant of deferring to the expert - treating the automation as an implementation of their (superior) expert knowledge. But meaning that in contexts where I also have substantial knowledge I can check that against the automation and either catch an error or learn something, not a reason to discount my own judgement.
(Related to why I prefer to use and make automations that record how a conclusion was reached)
@ShadSterling but yeah, fun fact, there are actually laws on the books stemming from the previous round of harmful use of computers in decision-making in the 1970s, which for example require such decisions to include a description of how the conclusion was reached
to our knowledge nobody has attempted to apply these laws to generative ML, which we would very much like to see happen because there's no way the ML can meet that bar
@ShadSterling these laws are the reason that so-called expert systems were invented (these days they're usually called decision trees)
because ie. in medical contexts people wanted to use the automation but they were required to be able to explain it, and using a tree structure means there's a clear rationale
Irenes (many)
ShadSterling