Earlier I posted about using ChatGPT's propensity to fabricate citations entirely as a short-term strategy for detecting journal submissions and classroom assignments that had been written by machine.
I've been playing with the system for the last couple of hours, and as best as I can tell, ChatGPT now does a much better job than it did when first released at only citing papers that actually exist.
They're not perfect—for example, DOIs can be wrong and some are fabricated—but most are not.
If I'm not just imagining things, it raises an interesting question.
While this constitutes an "improvement" in the technology in some manner of speaking, it's unclear whether this is a desirable development. It strikes me as a step that makes detecting more difficult without confering any significant epistemic improvement.
In other words, if the system has really been adjusted to avoid citing fake papers, this constitutes a deliberate choice to create more persuasive bullshit.
Also to be clear, the system is most definitely bullshitting if not outright lying.
Take, for example, this methods section ChatGPT just generated in response to my prompt "Write an scientific review paper with references about whether sunscreen ingredients are carcinogenic."
Unlike many factual claims about background, this is straight up false.
The system decidedly DID NOT conduct a comprehensive literature search on specifically those sources using specifically these terms.
@ct_bergstrom Frankly, the entire thing is rotten to the core. It has no ability to discern fact from fiction, because it has no underlying knowledge, no mental model of the world. It's great at figuring out correct grammar and sounding literate, but it simply cannot do any more.
As a complete outsider, this technology seems to me to be a complete dead end.
@ct_bergstrom And that's what annoys me so much. Teachers and professors need to defend against students using it; literary magazines get flooded with AI-written submissions; news sites are putting out AI-written articles, as if we need more low-quality nonsense or more loss of writing jobs; and who knews how this will be used for disinformation and spam.
As usual, tech companies put out poorly-thought out creations, and the rest of us pay the price.
Carl T. Bergstrom
Ofer Kedem
Michael Gemar