Code for Publication - Level 1
Take whatever you've got, archive it online, put a link in the paper. This is low cost & provides an extended methods section where the detailed decisions you made can be found by a sufficiently motivated person. This is a major positive even if no one can successfully run the code.
Everyone's code is messy. We understand. Post your code & I will personally fight anyone who complains that it's messy.
When I say archive I mean https://zenodo.org/ or similar
Code for Publication - Level 2 (Level 1+)
Written instructions for rerunning the code to replicate the analysis (ideally in a file called README). E.g., 1) install packages A, B, & C; 2) download data D; 2) Run script E; 3) manually change F; 4) Run script G, etc.
Ideally go through this process yourself on a computer other than the one you did the analysis on to make sure it works. Even better have a friend do this to make sure someone other than you can follow the steps.
Code for Publication - Level 3 (Level 1/2+)
Automate all the steps for rerunning the analysis using a script. This could be a bash script of a script in the language your code is written in. This should include package installation (ideally w/fixed version numbers if the language allows it). Test this yourself on a computer other than the one you did the analysis, have a friend test it, try to test it on multiple operating systems. Make sure the outputs match the paper.
Code for Publication - Level 4+ (Level 3+)
There are lots of extra things you can do to make all of this even better:
* Use a workflow system instead of script for automation
* Provide a container (e.g., Docker) with code and data
* Have your code produce either a documented version of itself or the entire paper using literature programming tools (e.g., notebooks , Rmarkdown) (I see @tpoisot already getting ahead of me here in the replies; listen to Tim, he's awesome)
There's a ton of write ups of related ideas and even more examples of them. Here's just a few for further reading:
* https://kbroman.org/steps2rr/ by @kbroman
* https://doi.org/10.1371/journal.pcbi.1005510 by @gvwilson et al.
If you're interested in research software instead of code supporting analyses (a very different situation) see @tpoisot's very nice piece as a starting point http://doi.org/10.4033/iee.2015.8.8.f (and references there in for more)
@ethanwhite @kbroman @gvwilson @tpoisot This is something I like to call "layered reproducibility" (discussion here: https://discuss.ropensci.org/t/creating-a-package-to-reproduce-an-academic-paper/1210/2).
I sometimes worry that highly formalized build systems (like targets in R) obscure what could be easier to follow scripts for many users. Targets does make you write the code in reusable-ish functions, though.
I like your README map of steps, Tim! We like to print the graph of steps, but it is not as easy to follow: https://github.com/ecohealthalliance/mpx-diagnosis
I’d say using Docker or a similar environment is the best approach, especially if you have external system dependencies. That said, I try to aim for layered reproducibility, such that you maximized the reproducibility of the project within successive tools. Making an R package is a very good way of wrapping up the pieces of the package for re-use, with dependencies specified in the DESCRIPTION. This has its limits, as you mention above. So I prefer to also have Dockerfile a creates the envi...
@ethanwhite @kbroman @gvwilson @tpoisot
I appreciate the sentiment of your hierarchy.
But realistically, anything that isn't effectively a container of some sort (LXC, docker, Apptainer and so on) will not be fully reproducible by anybody ten years from now. If you're using R the timeframe is more like 3-5 years. If you're still using Python 2 you've already lost.
Data, code, and manuscript text for an article characterizing the fungal endophytic communities of Metrosideros excelsa across the city of San Francisco. - GitHub - ZimmermanLab/SF-metrosideros-end...
In #Maneage [1], level 4+ is different:
* in analysis/ we use 'make' for the higher-level workflow, encouraging bash scripts for details;
* in software/ we use 'make' to build all the software with sha512sum checks on the downloads, starting from a minimal unix-like system;
* the makefiles initialize.mk and paper.mk are the workflow for the paper
Fully reproduce:
./project configure
./project make
Example: [2]
[1] https://maneage.org
[2] https://zenodo.org/record/7792910
Cool! :) The main 'tasks' and 'bugs' are coordinated at savannah [1]; we have some loosely organised irc/matrix channels e.g. [2].
To test a real science paper (peer-reviewed), not too heavy computationally, I recommend [3] (a not-quite-final version ran fully from scratch on a pinephone).
[1] https://savannah.nongnu.org/support/?func=additem&group=reproduce
[2] irc: https://libera.chat ##maneage
matrix bridge: #maneage-community:matrix.org
[3] https://arxiv.org/abs/2112.14174 = https://zenodo.org/record/6794222
@ethanwhite something we've done for a paper was to write all the scripts using literate programming, so that at the end the code became the readme, and then every step became its own notebook.
https://github.com/PoisotLab/MetawebTransferLearning
This was not that much extra work, and it also resulted in code that was far more useful to the collaborators not involved in the programming steps. Strongly recommend it.
Ethan White
Noam Ross
Rainer M Krug
Janne Moren
Oliver Stueker
Naupaka Zimmerman
Timothée Poisot
Boud