The "research" is at times risible. Key experimental results appeared suddenly in a manuscript version upon which lab members were given a couple of hours to comment before submission to Nature.

"When the students asked Dias about the stunning new data, they say, he told them he had taken all the resistance and magnetic-susceptibility data before coming to Rochester."

Just nonchalantly sitting on proof of room-temperature superconductivity for a few years, as one does. /2

For the first paper, Nature engaged three referees and there were three rounds of review. One referee was strongly positive, the other two did not support publication. Nature went ahead anyway.

I can't think of a previous black on white example where Nature have admitted allowing impact to override quality, although that's always been the tacit implication of their editorial policy. And this is exactly the result they risk with that policy. /4

@BorisBarbour

For as long as I can remember, they've always made it quite explicit, that their editors reign supreme and reviewers only advise them - and that this goes in bnoth directions.

In the words of now infamous Declan Butler, "peer-review light": the non-peers are making the main decisions and the peers are relegated to the back-seats.

@brembs @BorisBarbour "For as long as I can remember, they've always made it quite explicit, that their editors reign supreme and reviewers only advise them - and that this goes in bnoth directions."

Isn't that how journals started, and how they're supposed to function? The role of reviewers is to advise the editor, not be the editor and make decisions for the journal.

If editors aren't supposed to make their own judgement calls, why have trained scientist experts be editors at all?

@brembs @BorisBarbour Sure, this sometimes gets you the Benveniste affairs of the world... That's what's happened here right? But that's built in to the system, which relies on good science winning out in the end. And it did that here also. So is there really a problem?

Nature's a private company. They're allowed to screw up, and we're allowed to judge the sum of their work and decide if their error rate is *unacceptably* high. Doing peer review is voluntary, we vote with our feet.

@MarkHanson @brembs

Does the policy pass the honesty test: would they publish if they had to post the referee reports alongside, with only a single positive one? I'm guessing no.

I think Rochester and the funders come out of this affair far worse than Nature. But there are plenty of things Nature can improve upon:
- do more to resolve scientific issues between referees before accepting
- bear in mind track records for quality/integrity
- contact all authors in an investigation

@BorisBarbour 100% agree.

Re: "dangerous" - to who?

What sort of error rate should journals be allowed? Shouldn't we just let Nature accept the egg on their face and we all move on?

I guess if I summed my stance: science does *not* have a no-tolerance policy on being wrong. The issue here stems from giving undue weight to being 'published' as being 'true'.

This isn't some failure of the scientific method. As emphasized here, the scientific method doesn't end at publication.

@MarkHanson @BorisBarbour

Key issue is the system itself: publish a paper and pretend it's the ultimate truth on the matter. A system shift is needed to negate that assumption on published papers, and to instead more humbly publish results as the latest take on the matter, correct or not but hopefully constructive and insightful. A first step to that end is to stop using papers as tokens of academic currency weighted by the publication venue and for any evaluators to start reading the papers.

#ScientificPublishing

@albertcardona @MarkHanson @BorisBarbour

Precisely, Albert!

Some of us are old enough to remember the old tagline of Nature "the world's best science and medicine" - pretty much the opposite of what the data say (which may be one reason why they stopped using it) 🤣

I'd guess at this point, 30 years down the debate, most people with some competence probably agree that the system is FUBAR, like Albert says. That kind of concensus has been emerging on the last decade or so.

@albertcardona @MarkHanson @BorisBarbour

The consensus that we eventuelly will need to replace academic journals has only been emerging in the last 2-3 years and mostly here in Europe, more slowly elsewhere.

@brembs @albertcardona @MarkHanson @BorisBarbour
@neuralreckoning

I think a lot of this recognition that we will need to replace academic journals soon has been the recognition that bioRxiv, psyRxiv, and medRxiv have not been the disasters many thought they would be*. I think a lot of people thought that peer review was critical to the success of the enterprise, and therefore we had to put up with the journals because we needed the peer review gatekeeping. However, it has become clear that (within field), labs can mostly do their own peer review.

It is not clear what we can do about science outside field. As a scientist how can I know whether to believe something outside my immediate field. And how should we control what journalists, politicians, and clinicians trust, given that they do not have the training to do their own "in-lab" peer review.

Nevertheless, importantly, now that we have preprint servers and can compare pre- and post-peer review, it is pretty clear that peer review isn't doing much, which gives us the ability to say that the costs (excessive publisher profits, reviewer time costs, etc) are not worth the gains.

* Yes, I know, arXiv has been around for many many years. But people somehow thought biology, psychology, and the other non-physics fields were different. ¯\_(ツ)_/¯

@adredish @brembs @MarkHanson @BorisBarbour @neuralreckoning

Great point on the changing perception on preprints in the biological sciences. For me a published paper is always like a preprint – I read it with an equal amount of scrutiny – so I haven't noticed any difference with before and after the rise of preprints.

On the "outside field" point: I reckon this is an issue already now and has always been. Peer review is not at all a guarantee, as shown time and again for work that many care about (room-temperature superconductivity being the latest example); and a number of still unexamined peer reviewed studies wouldn't pass muster either if anyone bothered to look.

Journalists, unless they are themselves trained in the field, are limited to report what those in the field have commented. Politicians on the other hand are meant to trust at face value the reports from their specialists – the impact forecast presented in executive summary form – and evaluate them against other pressing needs in society to take, precisely, a political decision. Clinicians are perhaps lacking such counselling from specialists (and the void is filled by unscrupulous pharma companies), but in compensation, have considerable training themselves.