Earlier I posted about using ChatGPT's propensity to fabricate citations entirely as a short-term strategy for detecting journal submissions and classroom assignments that had been written by machine.
I've been playing with the system for the last couple of hours, and as best as I can tell, ChatGPT now does a much better job than it did when first released at only citing papers that actually exist.
They're not perfect—for example, DOIs can be wrong and some are fabricated—but most are not.
If I'm not just imagining things, it raises an interesting question.
While this constitutes an "improvement" in the technology in some manner of speaking, it's unclear whether this is a desirable development. It strikes me as a step that makes detecting more difficult without confering any significant epistemic improvement.
In other words, if the system has really been adjusted to avoid citing fake papers, this constitutes a deliberate choice to create more persuasive bullshit.
Also to be clear, the system is most definitely bullshitting if not outright lying.
Take, for example, this methods section ChatGPT just generated in response to my prompt "Write an scientific review paper with references about whether sunscreen ingredients are carcinogenic."
Unlike many factual claims about background, this is straight up false.
The system decidedly DID NOT conduct a comprehensive literature search on specifically those sources using specifically these terms.
What I'm seeing here is bullshit in the precise technical sense of generating content designed to be persuasive yet produced without regard for truth or logical coherence.
Moreover this bullshit generation is clearly aided and abetting by ongoing development at OpenAI.
In general I've been pretty pleased with OpenAI's approach compared to e.g. what we saw with Galactica, but this is disturbing.
If they are making piecemeal changes to features such as citation behavior, designed to help the system cover its "tells", then I'd think that documenting these changes publicly as @emilymbender has requested seems like the bare minimum as far as social responsibility is concerned.
stuff like this really makes the "hallucination" description seem flawed, and more like we're talking about a corrupt bureaucracy that's simply papering over evidence of transgressions, rather than addressing the core issue
(that AIs have no intrinsic tether to reality the way humans do, and so AIs have nothing particularly anchoring them to anything that's *real* more than, say, anything that's complete and utter bullshit)
@ali @ct_bergstrom @emilymbender "corrupt bureaucracy motivated only by the formsn looking right" here is a really excellent thought tool for working with opaque models trained on obscure-to-us data.
"What is the least Real Work such a model would have to do to present the appropriate papers? Assume the bureaucrat will do that."
@ali @ct_bergstrom @emilymbender
The CYA games that OpenAI is playing, like penalizing imaginary citations and recognizing transparently racist leading questions, are really just attempts to nudge the Path of Least Work away from the most obvious shortcuts.
But it will almost always be easier to bullshit, or, in the context of this analogy, make a minor error on internal paperwork that allows the bureaucrat to shortcut doing real work
@ali @ct_bergstrom @emilymbender
It's a little weird to see the Chinese Room reappear, but with a different question entirely -- we don't care if the bureaucracy "is thinking"; we care if it's "serving the people".
Functionaries who follow all the rules with the least possible effort _might_ be "serving the people" but, in a corrupt bureaucracy or a bullshitter, it's optimized towards "least effort" instead
I agree.
It wouldn’t make sense for them to block references to nonexistent materials but keep references to existing sources that weren’t actually referenced by the system.
It’s not an information access system.
i call that confabulating and obfuscating
@ct_bergstrom Frankly, the entire thing is rotten to the core. It has no ability to discern fact from fiction, because it has no underlying knowledge, no mental model of the world. It's great at figuring out correct grammar and sounding literate, but it simply cannot do any more.
As a complete outsider, this technology seems to me to be a complete dead end.
@ct_bergstrom And that's what annoys me so much. Teachers and professors need to defend against students using it; literary magazines get flooded with AI-written submissions; news sites are putting out AI-written articles, as if we need more low-quality nonsense or more loss of writing jobs; and who knews how this will be used for disinformation and spam.
As usual, tech companies put out poorly-thought out creations, and the rest of us pay the price.
@ct_bergstrom
That is very disappointing. Because doing literature search regarding a special question should be something #AI die really help and shine.
Without adhering to facts #ChatGPT is just a fancy expensive Lore Ipsum generator 🤬
@ct_bergstrom Both "bullshitting" and "Lying" imply consciousness and agency - in other words that it knows what it's doing. It doesn't. All it can do is predict very accurately what sort of words fit with what sort of other words. The fact that this fools humans into thinking what it produces is plausible says more about us that about it.
I think we need to be very careful about the language we use to describe what ChatGPT does or we will attribute to it much more power than it actually has.
The more that people interact with ChatGPT, and point out that is creating bullshit, it will just come up with new and improved bullshit.
Whiter whites! Fresher smell!
@ct_bergstrom The biggest problem currently facing the internet is the rampant spread of bad information. Google, in particular, has faced growing criticism of its core search product because people are creating so much bullshit designed to confound pagerank.
Facing this growing storm of bullshit threatening their core technology, it's incredible that the amazing breakthrough they've come up with is... an automated system for bullshit creation.
When you're job is pitching to VCs, making the bullshit persuasive is the whole game.
Carl T. Bergstrom
Sebastian Geukes
Michael Gemar
Ali Alkhatib
Jeremy Kahn
Ben Gleeson
Prof. Emily M. Bender(she/her)
Anthony DiPierro
Vitor Py
David Pollak
Dr. Samuel Wein
John Bonomi
Jon Awbrey
Nihl L'Amas
Ofer Kedem
Claudius Link
April
WayofCats
Andrew Perfors
emmatonkin
rob
SpaceLifeForm
Tom Bellin 
EyeInHand