Julia StrandMar 15, 2023
Steve Haroz (@sharoz on 🐤)

New blog post!

What would you do as a reviewer if you suspected that the methods described in a paper weren't accurate?
Here, I walk through such a case and explain how access to the data allow me to test my suspicions.

http://steveharoz.com/blog/2023/why-open-data-is-critical-during-review-an-example-case/

#OpenScience #ieeevis

Mar 15, 2023 at 9:06amWeb
Show threadJulia StrandMar 15, 2023
@sharoz Excellent case study, thanks for sharing!
Show threadSteve Haroz (@sharoz on 🐤)Mar 15, 2023
@juliafstrand Thanks Julia!
Show threadJobMar 15, 2023

@sharoz "authors with PhDs in computer science suddenly don’t know how how to send a file"

I don't even need to wear my protanomaly-correcting glasses to see that red flag, wow

Show threadSteve Haroz (@sharoz on 🐤)Mar 15, 2023
@vanderZwan The situation is as messed up as your L cones 🤓
Show threadNoémi K. SchuurmanMar 15, 2023

@sharoz A big problem is most reviewers won't be able to invest such time, because they have to do a bunch of other things and indeed get no time/money/etc for doing review work.

Given such constraints, one may have asked the authors themselves to prove what they said is correct. This should include describing the design/data collection such that it could actually be replicated.

In any case it shouldn't have been so hard to obtain what was needed for the review (privacy concerns aside)

Show threadSteve Haroz (@sharoz on 🐤)Mar 15, 2023

@NKSchuurman Yeah, a point I want to get across in that bulleted list is that the review took only about an hour in total. The major time drain was the ridiculous back and forth with editors and authors. A reviewer doesn't have to check *everything*, but they should be able to check *anything*.

Also, you might be interested in this http://steveharoz.com/blog/2022/reviewing-tip-the-10-minute-data-check/

Reviewing Tip: The 10-Minute Data Check | Steve Haroz's blog

Solving a Rubik's Cube takes skill and time. But checking if at least one face is solved correctly is quick and simple. Science should work the same way.

Show threadNoémi K. SchuurmanMar 15, 2023
@sharoz
The link doesn't work for me. But I'm interested, because I don't think I have ever only spent an hour on a review.
Show threadSteve Haroz (@sharoz on 🐤)Mar 15, 2023

@NKSchuurman

Yeah, an hour is REALLY short. It was only possible because I spotted and verified a flaw so devastating that I didn't even feel the need to read the whole paper.

Does this link work?
http://steveharoz.com/blog/2022/reviewing-tip-the-10-minute-data-check/

Reviewing Tip: The 10-Minute Data Check | Steve Haroz's blog

Solving a Rubik's Cube takes skill and time. But checking if at least one face is solved correctly is quick and simple. Science should work the same way.

Show threadNoémi K. SchuurmanMar 16, 2023

@sharoz It does now! These are generally sensible things to check/useful tips!

I tend to focus mainly on reviewing what my specific expertise is about (and mostly I review methods papers, which is of course a somewhat different setup for reviewing).

Show threadSteve Haroz (@sharoz on 🐤)Mar 16, 2023
@NKSchuurman I think focusing on methods is exactly the right approach. Those checks aim to quickly test other facets like stats errors.
Show threadJohnMasheyMar 17, 2023
@sharoz @NKSchuurman
Good work ... and of course, back and forth with editors often takes a long time.
Here's one that took us (esp. Ari Jokamaki) ~3 years to get a retraction of paper with plagiarism (also, absurdly wrong, but from experience, that's harder to get retractions).
2013 paper, noticed in 2014, retracted in 2014.
https://skepticalscience.com/florides_retracted.html
Back and forth with dates in:
https://skepticalscience.com/f13_elsevier.html
Retraction of Florides et al. (2013)

Skeptical Science
Show threadSteve Haroz (@sharoz on 🐤)Mar 17, 2023

@JohnMashey @NKSchuurman

It's really such a drawn out process. One thing I've tried to do on pubpeer is document the dates of each message to the authors. Here's an example: https://pubpeer.com/publications/E21B673AFD0DA82B00A6C0426E2025

PubPeer - Effectiveness Error: Measuring and Improving RadViz Visual E...

There are comments on PubPeer for publication: Effectiveness Error: Measuring and Improving RadViz Visual Effectiveness (2022)

Show threadJohnMasheyMar 17, 2023
@sharoz @NKSchuurman
Dating messages to authors (editors): good!
BTW, since you're an R guy, you might enjoy the oral history interview I recently got with John Chambers on his history with S & R, how he got to Bell Labs, etc (we overlapped in 1970s/80s, although I didn't know him, had of course seen S memoes.) Transcript & video:
https://www.computerhistory.org/collections/oralhistories/?s=mashey+chambers
Oral History Collection | Computer History Museum

Show threadaudreybrabsMar 15, 2023
@sharoz Great read, thank you!
Show threadrenebekkersMar 15, 2023
@sharoz thanks for writing this up. So much in there that I recognize both from the reviewer and editor perspective... the delays bc of authors not responding, links not working, lack of documentation, results from the data just different from what the manuscript says...
I can't help thinking that we're seeing only the tip of the iceberg. When data and code sharing becomes the default it could melt away
Show threadSteve Haroz (@sharoz on 🐤)Mar 15, 2023
@renebekkers The tip of the iceberg is exactly right. What's frustrating is that a neutral stance on transparency has the practical effect of disincentivizing it. If an open-data paper and a hidden-data paper both make the same glaring error, it's much more likely to be found in the open-data paper. The net result is a literature plagued with unfindable errors and a de facto discouragement of transparency.
Show threadrenebekkersMar 17, 2023
@sharoz Exactly. "Compared to other industries, science has a particularly lax system of quality control. Before roads are built and new toys for kids are allowed to be sold there are safety and health checks of the builders, their materials, their construction plans and manufacturing processes, and ultimately the safety of their products....
Show threadrenebekkersMar 17, 2023
@sharoz ...But when we do science, there is much less of this. We ask volunteers to primarily look at the product. If we buy a car at an authorized dealer, there’s a money back guarantee. But reviewers of scientific papers do not even start the engine of the data and code to check whether the thing actually works. That is not good enough."
From: https://renebekkers.wordpress.com/2022/12/03/ten-meta-science-insights-to-deal-with-the-credibility-crisis-in-the-social-sciences/
Ten Meta Science Insights to Deal With the Credibility Crisis in the Social Sciences

Rene Bekkers
Show threadStefan ZitzMar 16, 2023
@sharoz Very interesting blog post and kudos to you for putting in that effort!
Do you have an idea how to streamline the process
E.g. mandatory journal hosted repos?
Or mandatory #jupyter notebooks for analysis?
Show threadSteve Haroz (@sharoz on 🐤)Mar 16, 2023

@Zitzero

COS TOP level 2 (https://www.cos.io/initiatives/top-guidelines) outlines a journal policy for mandatory transparency.

1. Mandatory experiment code/stimuli + data + analysis code (I prefer RMD, but jupyter is works too)
2. On a public persistent repo (not hosted by the journal) such as http://osf.io
3. If anything can't be shared (e.g. for privacy), the paper needs to share what it can and be very explicit about why ("privacy" is often a BS excuse).

Show threadStefan ZitzMar 16, 2023
@sharoz Thanks for the link and the outline, I didn't know about them 😃
Show threadMal 甄/kalessin/PeriMar 17, 2023
@sharoz amazing. I'm glad there are folks like you in science, holding the line.
Show threadFardels BearMar 17, 2023

@sharoz

I like the photo of the Potemkin village.

Show threadAdrien FoucartMar 17, 2023
@sharoz Yes. I analyzed the results of a bunch of image analysis competitions for my PhD, and there were really two main types : those without enough information to fully check the challenge's conclusions... and those where we found methodological and/or implementation errors. Data and code availability are absolutely necessary to assess the validity of the results.
Show threadAndrew L. HuftonMar 21, 2023

@sharoz As an editor, I fully support this approach!

I have long argued that giving reviewers raw data access is not burdensome and often actually saves time. Great to see this borne out in your blog post.

Show threadMichael KuhnMar 25, 2023
@sharoz it's great that you insisted and caught this! Unfortunately, I think the most likely course of action is that the authors will submit it unchanged to another journal 😑 At least that has been my experience in cases where I rejected a paper due to concerns about the method. (In one case it even came out in the same journal, via the proceedings of a conference)
Show threadSteve Haroz (@sharoz on 🐤)Mar 25, 2023
@biocs I generally have had the same experience. Or the editor fails to understand the seriousness of the confound and just insists the authors mention it in the limitations section.
Show threadKevin RussellMar 26, 2023

@sharoz

Yes, but what is the provenance of the Potemkin village?

Show threadSteve Haroz (@sharoz on 🐤)Mar 26, 2023
@kevinrns Credit should be in the mouseover tooltip (press and hold on a phone). Let me know if it doesn't work.
Show threadKevin RussellMar 26, 2023

@sharoz

The alt text on hover appears to be just "Facades with no actual buildings"

Show threadSteve Haroz (@sharoz on 🐤)Mar 26, 2023
@kevinrns Check the mouseover on the actual blog. (not by my computer right now, or I'd just answer you)