@thefirstred

"With approximately 15,000 articles published on MTurk in the first 6 months of 2022 alone, the ripple effects of bad MTurk data are enormous: failure to find replications, erroneous effects, lines of research based on false information."

"Too Good to Be True: Bots and Bad Data From Mechanical Turk" by Margaret A. Webb and June P. Tangney 2022 https://journals.sagepub.com/doi/10.1177/17456916221120027

#MTurk #MechanicalTurk #science #ciència

@albertcardona

There’s some important context that’s missing from that (non-peer reviewed) paper:

https://psyarxiv.com/w7qy9

They claim ripple effects of bad MTurk data based on an n = 1 paper where much of the bad data reported was due to poor screening practices (including expecting people on mturk to complete a 45 minute study for $6 when most studies are 5 minutes or less)

@paolo_palma

To their credit, the authors add such a disclosure at the end:

"This article is not meant as an empirical assessment of the validity of all MTurk data; rather, it is an illustration of an individual experience. There is no way of knowing from these data alone what the true bound of validity is for all MTurk samples."

And Amazon did reimburse them.

As for payment: minimum wage is claimed, which likely means $7/hour in the US. If 5-min pays $6, that'd be far above minimum wage.

@albertcardona

The study was 45 minutes long, and information actual handling of the data was sparse. If we assume 200 participants reached completion, it means over 300 did not get compensated but were included in this analysis