today, in attempting to do a close-read of a piece of alleged research to assess how much role a human had in writing it and to what extent the arguments and sources support the key take-aways it suggest
we found a section which purports to describe the oversight that a human performed over an LLM in writing it
and that section has several major non-sequitors that have nothing to do with the subject matter
we're not naming it because the point is not to have a conversation about a specific piece
it's to remind everyone to engage your brains when reading these things. don't be on autopilot, don't let your assessment of plausibility be based on how formal the writing is or anything like that
oh yeah by the way
these newfangled spam generators do tap into several important effects that haven't been a big problem in the past, such as the linguistic thing that the stochastic parrots paper does a great job of explaining (humans assess the credibility of written information by building a mental model of the person who write it, but here, that model is spoofed)
but, also...
old-fashioned automation bias is a very strong factor in why people take them seriously, too
wikipedia gives a description we like: "Automation bias is the propensity for humans to favor suggestions from automated decision-making systems and to ignore contradictory information made without automation, even if it is correct."
it's well-studied, going back many decades. people just... assume the machine must have had some reason for saying that, and discount their own judgement.
@ireneista Oooh TIL that there’s a name for that bias - thank you!
… would have been helpful to know and read up on when I wrote https://tomrenner.com/posts/400-year-confidence-trick … could have saved a few words (and improved the argument) by having proper definition of the phenomenon I was observing to hand!
In 1623 the German Wilhelm Schickard produced the first known designs for a mechanical calculator. Twenty years later Blaise Pascal produced a machine of an improved design, aiming to help with the large amount of tedious arithmetic required in his role as a tax collector. The interest in mechanical calculation showed no sign of reducing in the subsequent centuries, as generations of people worldwide followed in Pascal and Wilhelm’s footsteps, subscribing to their view that offloading mental energy to a machine would be a relief.
@ireneista yes exactly! I’m always grateful when I find the proper name for something - it helps encapsulate the idea and place it properly in the pantheon of related topics.
I’m somewhat in awe of people who can define and name phenomena in a way that helps me and others reason about them. It’s a sign of really thorough reasoning.
Irenes (many)
Tom Renner