Yesterday I had a number of conversations with people working in the scholarly publishing sphere about what happens when AI chatbots pollute our information environment and then start feeding on this pollution.
As it so often, the case, we didn’t have to wait long to get some hint of the kind of mess we could be looking at.
https://www.theverge.com/2023/3/22/23651564/google-microsoft-bard-bing-chatbots-misinformation
The confluence of this technology with the information ecosystem that we described in our paper from a couple of years could be an epistemic catastrophe.
https://www.pnas.org/doi/10.1073/pnas.2025764118
I’m coming to think that releasing these tools was a reckless act with the potential to generate negative externalities we have barely started to imagine.
The threat isn’t rogue superintelligence. It’s bullshit at unprecedented scale, reflected back upon itself and iteratively amplified.
I’m certainly not saying that the deployment of these systems will suddenly make it impossible to find and build upon trusted and vetted sources of information. Those aren’t going to magically disappear.
My bigger fear is that rather than making a blunder in tying their infotech empires, two automated bullshit generation. Microsoft and Google have correctly anticipated demand. My fear is that people might want what Bing and Bard are selling.
On the other hand, if I dial up the cynicism, just a little bit more, maybe it doesn’t matter much. One view is that by spewing bullshit into the information ecosystem, generative AI is poisoning the well from which it drew life and ensuring that future generations of such technologies will produce garbage.
Another is that the training set was never a pure wellspring. It was already the town cesspool—and even massive quantities of additional bullshit will barely be noticed.
@ct_bergstrom
> the training set was never a pure wellspring. It was already the town cesspool—and even massive quantities of additional bullshit will barely be noticed
This. The Machine Learning text generators have just amplified both existing pain and work on effective medicines for that pain.
@ct_bergstrom The work of Google (and Bing) was supposed to be cutting through the bullshit and guide you out of the swamps of nonsense.
They've apparently given up on that, and now the goal is to keep you stuck, spinning your wheels endlessly in a mire of their creation.
@tob @ct_bergstrom It was initially the work of search engines to cut through the swamp of nonsense and bring you the hidden pearls.
Unfortunately, this is capitalism, and they need to make money.
And the way they monetized is they bring you the nonsense somebody paid to bring you.
Yeah, I think that hand has the right perspective.
We were already there, and this tech has more potential to reduce the effects of #misinformation than to increase it (from current levels).
Auto-fact checkers, #reputation filters, gradient scoring, citation trees, etc. will be the next lineup of new features for #apps & browsers.
@DavePerrino @ct_bergstrom People have become too comfortable with what purposefully paid media like Fox "News" provides them with:
Entertainment.
The problem here: Entertainment neither regulates nor feeds people's lives.
Politics does.
People are free to dislike this fact, but still: Politics directs people's lives, regulating where necessary & tolerating where mandatory.
Even if guys like Musk, Zuckerberg, Bezos & fake billionaires like Trump may disapprove...
#Democracy #DefendDemocracy #DefensiveDemocracy #Information #Education #YourVoiceMatters #BeInformed #BeFree #StrongerTogether
@ct_bergstrom Absolutely. The Waveform Clips chaps had a good chat about this and came to the same conclusions. Skip 9.5 minutes if you're impatient.
@ct_bergstrom I'd be keen to know if this is introducing some people to the idea of (what I think of as) "informationality": basically, is a message mis-, dis- or genuinely informational, a valid "alien" perspective, convincingly plausible entertainment, spitballing, etc?
If so, maybe we have a chance of a cultural shift towards more informative communication? Less off-the-cuff waffling, more citing examples, giving others more time to consider.
Also, is there a term for "informationality"?
@ct_bergstrom If bullshit is what’s produced by someone who doesn’t care whether they’re telling the truth or not (Harry Frankfurter),
then perhaps bullshit generators are what’s produced by someone who doesn’t care whether they cause harm or not.
@ct_bergstrom That’s true of many artists; compare @jwz’s “dadadodo”.
LLMs produce, not just nonsense, but nonsense which wears the clothes of sense, and therefore enables a particular species of harm.
Carl T. Bergstrom
Strypey
John Stritch
Tom Bellin 
hramrach
Jamie Hopkings
mj
DavePerrino ✅
Arena Cops 🇺🇦✌
Alan James
Matt Mascarenhas
Outeast
Jason Colditz
Ben Rosengart
Frank Bennett 🇯🇵
Starry Starry Knight