Martin ModrákIf I were to say that the primary benefit of randomization (and possibly blinding) is that it makes simple statistical models match the true data generating process quite well, would that be a provocative statement? Or would that be obvious? Is there a good reference for this line of reasoning?
Mark RubinCritical Metascience:
2022 has been a bumper year for what I’d call “critical metascience” - work that takes a step back and offers a critical perspective in the field.
My Top 10 papers of 2022 in this area are, in alphabetical order… 🥁 🧵👉
#OpenScience
#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis
#SociologyofScience
#ScienceofScience
#PhilosophyOfScience
#PhilSci
#PhilScidon
1/12
For #FollowFriday, can I recommend some fairly new additions to our Mastodonian family:
@devezer theoretical metascientist
@itaiyanai co-host of the Night Science podcast
@orlamuldoon professor of social psychology
@qjep The Quarterly Journal of Experimental Psychology
#Metascience
#NightScience
#SocialPsychology
#Psychology
@socialpsych
#MetaResearch
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
#PhilScidon
#AcademicChatter
@academicchatter
#PolPsych
#PoliticalScience
Can we kill theories?
Social psychology consists of “a vast graveyard of undead theories” (Ferguson & Heene, 2012).
“No experimental result can ever kill a theory" (Lakatos, 1976).
#PhilosophyOfScience
#PhilSci
#PhilScidon
#SocialPsychology
@socialpsych
How interested are scientists in the truth?
Mizrahi (2021) analysed the text of scientific papers and found that “scientists use the terms ‘knowledge’ and ‘understanding’ significantly more often than the term ‘truth’ when they talk about the aims or goals of scientific research in their published works."
#PhilosophyOfScience
#PhilSci
#PhilScidon
#MetaScience
#MetaResearch
#SociologyofScience
#ScienceofScience
Replicability and Theory:
“Our results suggest that many of the practices that have been proposed as a means to improve the replicability of psychological research—such as open data and methods…preregistration and Registered Reports…and basing conclusions on Bayesian inference…or p < .005 rather than p < .05…—do indeed improve confidence in replicability among our sample.”
Continued 🙂 🧵👉
#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
#PhilScidon
John S. Wilkins@MarkRubin This is massively simplistic. Hypotheses include the criteria for delineating phenomena in need of explanation, satisfaction criteria for success, disciplinary standards and practices, and taxonomies of subjects under investigation. IMO.
#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
#PhilScidon
#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
#PhilScidon
What’s a hypothesis?
“A hypothesis is not simply a guess about the result of an experiment. It is a proposed explanation that can predict the outcome of an experiment. A hypothesis has two components: (1) an explanation and (2) a prediction. A prediction simply isn’t useful on its own.” (Haroz, 2014)
Blog post: http://steveharoz.com/blog/2014/mysterious-origins-of-hypotheses-in-visualization-and-chi/
#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
#PhilScidon
A “quietist” response to the replication crisis:
“The quietist approach proposes that we should just accept that it is in the nature of science that we get things wrong, and that this is particularly true with sciences in early stages of development.”
Bird (2021). Understanding the replication crisis as a base rate fallacy.
🔒 https://doi.org/10.1093/bjps/axy051
🔓 https://kclpure.kcl.ac.uk/portal/files/95454096/Replication_base_rate_fallacy_REV2.pdf
#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
#PhilScidon
@philosophy
Understanding the Replication Crisis as a Base Rate Fallacy | The British Journal for the Philosophy of Science: Vol 72, No 4
The replication (replicability, reproducibility) crisis in social psychology and clinical medicine arises from the fact that many apparently well-confirmed experimental results are subsequently overturned by studies that aim to replicate the original study. The culprit is widely held to be poor science: questionable research practices, failure to publish negative results, bad incentives, and even fraud. In this article I argue that the high rate of failed replications is consistent with high-quality science. We would expect this outcome if the field of science in question produces a high proportion of false hypotheses prior to testing. If most of the hypotheses under test are false, then there will be many false hypotheses that are apparently supported by the outcomes of well conducted experiments and null hypothesis significance tests with a type-I error rate (α) of 5%. Failure to recognize this is to commit the fallacy of ignoring the base rate. I argue that this is a plausible diagnosis of the replication crisis and examine what lessons we thereby learn for the future conduct of science.
Crypto-positivism:
“A central dimension of our argument is that many of the tenets of positivism are so embedded within Western culture, academia, and the world of education in particular that they are often invisible to researchers and those who consume their research.”
Kincheloe and Tobin (2015): https://doi.org/10.1163/9789463000765_003
#PhilosophyOfScience
#PhilSci
#PhilScidon
@philosophy
#Positivism
