I looked at 325 psychology study replication results today, and the best simple rule I found for telling if a study would replicate is this:

When the original study's p-value was at most 0.01, about 72% of the papers replicated.

Whereas when p>0.01, only 48% replicated.

Others have found similar results.

For instance, @uebernerd has suggested p=0.005 as a cutoff:

https://replicationindex.com/2021/05/10/osc-pm/

Now out in Psychological Medicine: Our paper demonstrating a ‘leapfrog’ trial as a method to accelerate the development and optimization of psychological interventions:

💰https://www.cambridge.org/core/journals/psychological-medicine/article/abs/demonstration-of-a-leapfrog-randomized-controlled-trial-as-a-method-to-accelerate-the-development-and-optimization-of-psychological-interventions/55D66C3E24BC026E92927256EA0D3443
 https://psyarxiv.com/dvzn4/

#OpenScience: #Preregistered, with #openData and #openMaterial.
(Posts taken from @[email protected])

1/

Bad news: Collecting data on #MTurk may essentially boil down to asking a crowd of #bots for their opinion:

Too Good to Be True: Bots and Bad Data From Mechanical Turk
(by Margaret A. Webb & June P. Tangney)
https://journals.sagepub.com/doi/abs/10.1177/17456916221120027

Quotes:

"at best, I had gathered valid data from 14 human beings—2.6% of my participant sample (N = 529)"

"This article is not meant as an empirical assessment of the validity of all MTurk data; rather, it is an illustration of an individual experience. "

Added a bunch of custom program language emojis!

 -- : rstats :

 -- : stata :

 -- : python :

 -- : julia :

I've made a bot that toots preprints from PsyArXiv: https://botsin.space/@PsyArXivBot I liked it on the birdapp, and I missed it here. Hope it is useful for people.

It is fast, but the hyperlink goes to the SHARE meta-data. You need an extra click to go to the actual preprint. If you know how to fix this (I am using IFTTT) let me know.