7/8 📦 Looking to submit your package?

If you have an R package that you believe should undergo peer review, submit it to our open peer review system: https://github.com/ropensci/software-review

Let’s ensure it meets the highest standards of quality and usability! #PeerReviewWeek #OpenSource #SoftwareReview

3/8 🛠️ What makes rOpenSci’s peer review special?

Our process goes beyond traditional academic peer review. It focuses on code quality, documentation, and community standards: https://devguide.ropensci.org/

It’s a collaborative effort, with authors, reviewers and editors growing through the process. #SoftwareReview #OpenScience

@neuralreckoning @adredish @WorldImagining @mschottdorf @brembs
butting in late on this discussion (thanks all for your thoughts) and as usual I want to inject a little @joss model into the conversation since it's a really good, live, successful experiment in a lot of these ideas.

In my experience, none of the "let's let readers provide comment/review if they want" has worked because it's really rare that people want to write the deep-dive that a good review requires

I get this objection, and so I think opportunistic review is just one of many parts of a more functional review system. This should just be trivially possible for any artifact of scholarly work, but it does require shifting what we think 'scholarly publishing' bodies can be. That would require not only moderation of a peer review process, but moderation of a community and place of digital archival-grade discussion - that's more or less what neuromatch.social is an experiment in making. My own experiences with opportunistic review have been really great, with lots of good commentary from domain experts, both supportive and critical, that directly accompanies the work and makes it richer for experts and nonexperts alike.

Really love what @neuralreckoning is saying here:

Here's my suggestion: give authors an easy way to recognise valuable peer reviews that improved a paper by giving them the option to add them as authors.

and think this is another place where JOSS really works. Scientific experiments are different than software in a lot of ways, obviously, but one thing JOSS shows us about structured post-pub peer review is how in addition to raising issues, reviewers can do pull requests and propose contributions to the work. in software this results in automatic credit - you are the author of the commits that were merged - that is strictly positive for both parties: there is no limit on number of contributors, it's super clear what that person did, etc. A really healthy model of review would be one where reviewers could also contribute to the work and were seen as co-contributors.

Experiments have much longer timescales, material restrictions, etc. for making contributions like that, but one could also see that as being part of author responsibility, to make their methods portable/reproducible enough that they could eg. loan some equipment to another lab so they could collaborate and contribute to an experiment. that happens already as collaborations, so you could see how this would be relatively hazy barrier between "peer review" and "working in public."

when i'm reviewing software, I have actually a lot of motivation because i learn a lot from it, and that would be another part - if review was structured in a way to improve the work, then one could imagine a first pass at a review being "methods sharing" - not just evaluating the text, but collaboratively trying to figure out how to make it so the reviewers-as-collaborators could actually recreate the experiment and use that to improve the methods section.

because it's public, i have additional reputational motivation to review, and disincentive to be toxic and rude. Demonstrating a track record of thoughtful feedback and cooperative behavior is super good for being able to look for work - win/win.

The question of legibility to an outside audience is often a matter of process and interface design to me. A marker of "having gone through peer review process x" is meaningful in a way that "being published in venue y" is not, because in the latter case it's an existential question for the work - something can only be in one y and it must be in some y to exist. Going through a voluntary peer review process with a clear and transparent standards process (alongside the complete artifact of the review) is legible both to "outsiders" (assuming the process has some legible documentation, which is not necessarily a given) and insiders. Labels and open processes are v much not in conflict, but when labels are venues they are just qualitatively different.

so all these approaches are complementary to me - opportunistic peer review for everything against a background of collective moderation, structured peer review as a constructive process of trying to get a work to meet some standard with reviewers as both cooperators and adversaries in different roles, but untied to the mere existence of a work.

#PeerReview #ScholComm #SoftwareReview