Mastodawn

Carl T. Bergstrom May 12, 2023

This week, Science published a stunningly irresponsible news story entitled "Fake scientific papers are alarmingly common" and claiming that upward of 30% of the scientific literature is fake.

https://www.science.org/content/article/fake-scientific-papers-are-alarmingly-common

Below, the first two paragraphs of the story.

Headline and intro notwithstanding, the story itself later notes that the detector doesn't actually work and flags nearly half of real papers as fake. Does the reporter just not understand that?

h/t @Hoch

Fake scientific papers are alarmingly common

But new tools show promise in tackling growing symptom of academia’s “publish or perish” culture

Show thread

Carl T. Bergstrom May 12, 2023

The numbers from this story are based on a laughable "fake paper detector" that literally consists of the following ONLY. Do the authors:

1) use private (non-institutional) email addresses and/or have a hospital affiliation,

and

2) have no international coauthors.

That's it.

If these criteria are met, the paper is deemed a "potential red-flag fake publication" and counted toward that 30% tally.

Show thread

Carl T. Bergstrom May 12, 2023

Spin notwithstanding, the technical details within preprint itself make it abundantly clear that the method doesn't work.

In a "juiced" test set with as many fake papers as real ones, the indicators that they use have a sensitivity of 86% and a false alarm rate of 44%.

Yes, they flag 44% of the known real papers as fake.

That's not a detector, it's a coinflip.

This should be a profound embarrassment to everyone involved with the preprint and Science story alike.

https://www.medrxiv.org/content/10.1101/2023.05.06.23289563v1.full.pdf

Show thread

Carl T. Bergstrom May 12, 2023

To test their indicator, the authors conjecture that a valid indicator should meet three criteria based on a questionnaire sent to authors:

(i) Authors of fake publications are reluctant to provide critical information as revealed by their response – or non-response – to the questionnaire by the editor,

(ii) the number of fake publications increases steadily over time, and

(iii) journals with a low to medium impact factor are most affected.

There's a huge problem here.

Show thread

RonWodaski

@ct_bergstrom The criteria are just silly. They are trying to use non-content, easily observed criteria without ever answering the question of sufficiency. I would question the very idea of non-content criteria, although perhaps with some careful work one might find some that are useful. The amazing thing here is that this is basically a fake paper about fake papers that suffers from many of the content deficiencies of fake papers...