New blog post!

What would you do as a reviewer if you suspected that the methods described in a paper weren't accurate?
Here, I walk through such a case and explain how access to the data allow me to test my suspicions.

http://steveharoz.com/blog/2023/why-open-data-is-critical-during-review-an-example-case/

#OpenScience #ieeevis

@sharoz thanks for writing this up. So much in there that I recognize both from the reviewer and editor perspective... the delays bc of authors not responding, links not working, lack of documentation, results from the data just different from what the manuscript says...
I can't help thinking that we're seeing only the tip of the iceberg. When data and code sharing becomes the default it could melt away
@renebekkers The tip of the iceberg is exactly right. What's frustrating is that a neutral stance on transparency has the practical effect of disincentivizing it. If an open-data paper and a hidden-data paper both make the same glaring error, it's much more likely to be found in the open-data paper. The net result is a literature plagued with unfindable errors and a de facto discouragement of transparency.
@sharoz Exactly. "Compared to other industries, science has a particularly lax system of quality control. Before roads are built and new toys for kids are allowed to be sold there are safety and health checks of the builders, their materials, their construction plans and manufacturing processes, and ultimately the safety of their products....
@sharoz ...But when we do science, there is much less of this. We ask volunteers to primarily look at the product. If we buy a car at an authorized dealer, there’s a money back guarantee. But reviewers of scientific papers do not even start the engine of the data and code to check whether the thing actually works. That is not good enough."
From: https://renebekkers.wordpress.com/2022/12/03/ten-meta-science-insights-to-deal-with-the-credibility-crisis-in-the-social-sciences/
Ten Meta Science Insights to Deal With the Credibility Crisis in the Social Sciences

Rene Bekkers