Rich LucasNewish paper with Julia Rohrer on reciprocal links between personality and religion: https://psyarxiv.com/zt9mg/ Reanalysis of this: http://dx.doi.org/10.1111/jopy.12770. Some fun issues with 1) post-publication critique 2) writing reproducible code & 3) cross-lagged panel model /1
Regarding post-publication critique: I've been asked why we chose this paper to reproduce/critique. It's a good paper with a lot of strengths, but there are (as is usually true) some alternative choices that could have been made. Turns out these alternative choices lead to different conclusions /2
In short, we test variations on three modeling choices, and none of the effects is robust across these different models (figure shows variability in effects). That doesn't mean the original authors' choices were wrong, just that some caution is warranted when interpreting the results /3
More importantly, given the strong reactions that sometime arise when really problematic papers are subjected to post-publication critique, it might be helpful to normalize post-publication critique by doing it regularly for published papers that vary in quality (including some good ones) /4
I also want to thank the original authors for making their own code available, for thoughtful comments on our reanalysis, and for being really great to work with throughout this process. /5
Regarding writing reproducible code, this project had a challenging twist: I couldn't access the version of the data that the original authors analyzed (it uses the SOEP, which, for privacy reasons, has different version for EU and non-EU users). /6
This meant that I had to write the code (which is quite complex given the nature of the project), test it on incomplete data that didn't have all the variables, and then send it to Julia to run. We went back and forth fixing errors. /7
Hopefully this means that the code is now in much better shape to be run by others; but if anyone wants to try it, the code (and instructions for running it) is here: https://github.com/rlucas11/gsoep_religion. Feel free to file an issue on github if something doesn't work. /8
Finally, this is another example of a case where using the standard lag-1 cross-lagged panel model (which was used in the original paper) is probably not appropriate (see: https://journals.sagepub.com/doi/10.1177/25152459231158378) /9
For variables that social scientists study, CLPM (short dash) usually overestimates (in model-implied correlations) how quickly stability coefs decline with longer lags (as seen here; solid = actual). This leads to bias in the lagged effects. Other models (like RICLPM -- long-dash) do better /10
The paper's under review now, but comments and suggestions are welcome! /end