Here's a new paper summarizing methods for evaluating whether the absence of a statistically significant difference (from a NHST) is actually no difference--test of equivalence, confidence interval bounds, likelihood ratios, Bayes factors, and Bayesian estimation. There's nothing new here, but it's a nice readable overview that might be worth citing for some audiences.
https://royalsocietypublishing.org/doi/full/10.1098/rsbl.2025.0506?af=R
#statstab #334 Workflow Techniques for the Robust Use of Bayes Factors
Thoughts: "We outline a Bayes factor workflow that researchers can use to study whether Bayes factors are robust for their individual analysis"
Inferences about hypotheses are ubiquitous in the cognitive sciences. Bayes factors provide one general way to compare different hypotheses by their compatibility with the observed data. Those quantifications can then also be used to choose between hypotheses. While Bayes factors provide an immediate approach to hypothesis testing, they are highly sensitive to details of the data/model assumptions. Moreover it's not clear how straightforwardly this approach can be implemented in practice, and in particular how sensitive it is to the details of the computational implementation. Here, we investigate these questions for Bayes factor analyses in the cognitive sciences. We explain the statistics underlying Bayes factors as a tool for Bayesian inferences and discuss that utility functions are needed for principled decisions on hypotheses. Next, we study how Bayes factors misbehave under different conditions. This includes a study of errors in the estimation of Bayes factors. Importantly, it is unknown whether Bayes factor estimates based on bridge sampling are unbiased for complex analyses. We are the first to use simulation-based calibration as a tool to test the accuracy of Bayes factor estimates. Moreover, we study how stable Bayes factors are against different MCMC draws. We moreover study how Bayes factors depend on variation in the data. We also look at variability of decisions based on Bayes factors and how to optimize decisions using a utility function. We outline a Bayes factor workflow that researchers can use to study whether Bayes factors are robust for their individual analysis, and we illustrate this workflow using an example from the cognitive sciences. We hope that this study will provide a workflow to test the strengths and limitations of Bayes factors as a way to quantify evidence in support of scientific hypotheses. Reproducible code is available from https://osf.io/y354c/.
#statstab #238 Bridging null hypothesis testing and estimation
Thoughts: An overview of the ways you can claim "no effect" under a bayesian framework.
#bayesian #bayesfactors #nullresults #noeffect #equivalencetests #equivalence #jasp #r
#statstab #236 Using Bayes to get the most out of non-significant results
Thoughts: A bayesian way to investigate "no effect": Bayes Factors. Cool guide on how to think about priors (post hoc even).
#priors #bayesfactors #nullresults #equivalence #nhbt
https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2014.00781/full
#statstab #232 Bayesian Interval-Null Testing
Thoughts: @JASPStats has a module for Equivalence Tests that include Bayesian Overlapping and Non-Overlapping Hypothesis Testing.
#equivalencetests #bayesfactors #jasp #noeffect #bayes
https://jasp-stats.org/2020/06/02/frequentist-and-bayesian-equivalence-testing-in-jasp/
This post demonstrates the Equivalence Testing Module, new in JASP 0.12. In traditional hypothesis testing, both frequentist and Bayesian, the null hypothesis is often specified as a point (i.e., there is no effect whatsoever in the population). Consequently, in veryโฆ Continue reading โ
#statstab #196 JASP Bayesian ANOVA
Thoughts: @JASPStats is used by researchers to "add some bayes factors" to their results. But, do you know what those actually reflect? Here is what their team says:
#statstab #166 Using Bayes to get the most out of non-significant results
Thoughts: Not sure how to set more meaningful priors for your Bayes Factors? This paper has a guidance. Great for simple designs.
#bayesian #bayes #bayesfactors #nullresults #nhst
https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2014.00781/full
#statstab #165 Approximate Objective Bayes Factors From P-Values and Sample Size: The 3pโn Rule
Thoughts: p-values can't quantify evidence, but maybe if we apply a transformation they can? Is JAB_01 the future? Debate!
#statstab #148 Bayes Factors and how to use them, via {bayestestR} pkg
Thoughts: A go-to packages for bayesian model inference. Also has a great explanation of BFs, how to use them, and alternatives.
#r #rstats #bayes #BayesFactors #HypothesisTesting
https://easystats.github.io/bayestestR/articles/bayes_factors.html
#statstab #90 Improving the utility of non-significant results [...]
Thoughts: OK overview, but a fairly naive take on what to do with non-sig results. Also, plz don't just report #bayesfactors and call it a day!
#pvalues #NHST #frequentist #education
https://www.sciencedirect.com/science/article/pii/S1747938X23000830
RDN
Dr Mircea Zloteanu ๐บ๐๐