The numbers from this story are based on a laughable "fake paper detector" that literally consists of the following ONLY. Do the authors:

1) use private (non-institutional) email addresses and/or have a hospital affiliation,

and

2) have no international coauthors.

That's it.

If these criteria are met, the paper is deemed a "potential red-flag fake publication" and counted toward that 30% tally.

Spin notwithstanding, the technical details within preprint itself make it abundantly clear that the method doesn't work.

In a "juiced" test set with as many fake papers as real ones, the indicators that they use have a sensitivity of 86% and a false alarm rate of 44%.

Yes, they flag 44% of the known real papers as fake.

That's not a detector, it's a coinflip.

This should be a profound embarrassment to everyone involved with the preprint and Science story alike.

https://www.medrxiv.org/content/10.1101/2023.05.06.23289563v1.full.pdf

To test their indicator, the authors conjecture that a valid indicator should meet three criteria based on a questionnaire sent to authors:

(i) Authors of fake publications are reluctant to provide critical information as revealed by their response – or non-response – to the questionnaire by the editor,

(ii) the number of fake publications increases steadily over time, and

(iii) journals with a low to medium impact factor are most affected.

There's a huge problem here.

If non US/EU authors are more likely to use non-institutional email addresses, the detector will pick up these authors disproportionally.

And indeed we expect non US/EU authors to

i) have lower response rates, due to language issues not to mention (deserved, apparently!) distrust

ii) make up an increasing fraction of publication share

iii) publish at higher rates in low and mid impact journals

So all three test hypotheses fail to distinguish between fake papers and non US/EU authorship.

And that's the worst part of it.

While I don't want to imply anything about the motivations of the authors, *their paper has racist consequences.*

Their paper implements a detector that they themselves show doesn't work, and that we have every reason to expect disproportionally flags papers from Asia and the Global South—and then concludes that these area contribute the most fake papers.

Figure 3 is a disgrace.

I'm astonished that Science credulously boosted this rubbish, and even more surprised that Gerd Gigerenzer put his name on it.

/fin

@ct_bergstrom

A colleague, who should know better when it comes to data analysis, circulated this article across our internal science network.

I would direct them to your thread above, but work blocks much of the Fediverse (but not Twitter).

This and other misuse of statistics in practice somewhat annoy me, as some directly impacts my work (and I'm in no way a statistician, nor scientist).

@IceNine
> I would direct them to your thread above, but work blocks much of the Fediverse (but not Twitter)

WTF?!? Have you raised a ruckus with them about how ridiculous that is?

@ct_bergstrom

@strypey

It's probably a good thing, largely keeps me off during work. 😆