New blog post!
What would you do as a reviewer if you suspected that the methods described in a paper weren't accurate?
Here, I walk through such a case and explain how access to the data allow me to test my suspicions.
http://steveharoz.com/blog/2023/why-open-data-is-critical-during-review-an-example-case/
@sharoz A big problem is most reviewers won't be able to invest such time, because they have to do a bunch of other things and indeed get no time/money/etc for doing review work.
Given such constraints, one may have asked the authors themselves to prove what they said is correct. This should include describing the design/data collection such that it could actually be replicated.
In any case it shouldn't have been so hard to obtain what was needed for the review (privacy concerns aside)
@NKSchuurman Yeah, a point I want to get across in that bulleted list is that the review took only about an hour in total. The major time drain was the ridiculous back and forth with editors and authors. A reviewer doesn't have to check *everything*, but they should be able to check *anything*.
Also, you might be interested in this http://steveharoz.com/blog/2022/reviewing-tip-the-10-minute-data-check/
It's really such a drawn out process. One thing I've tried to do on pubpeer is document the dates of each message to the authors. Here's an example: https://pubpeer.com/publications/E21B673AFD0DA82B00A6C0426E2025
Steve Haroz (@sharoz on 🐤)
Noémi K. Schuurman
JohnMashey