Martin ModrákMar 19
Preprint alert: Simulation-based validation of Bayes Factor computation with @paul_buerkner and Sebastian Stroppel. We bring lessons learned in improving SBC to validation of Bayes factors. The main idea is still the same: simulate data from the models, fit those and see if the inferences are calibrated. 1/
https://arxiv.org/abs/2508.11814
#Bayesian #stats #rstats #SBC
Show threadMartin ModrákMar 19
First, we highlight a simple, previously unused check: if the posterior model probability is p, the prediction should be correct p of the time. This can be checked with known tools for binary model prediction calibration. We also discuss data-averaged posterior (DAP) check and adapting SBC for the task. 2/
Show threadMartin ModrákMar 19
In theory, SBC should be able to discover all possible problems and binary calibration as well as data-averaged posterior may (and do) miss some. In practice, both alternative methods are sometimes more sensitive than SBC for a given simulation budget. We recommend combining all of them. 3/
Show threadMartin ModrákMar 19

We also show how to leverage posterior SBC to check computation when priors are improper and we thus cannot simulate (we take the BayesFactor package as an example).

We tried hard to move beyond toy examples and do our evaluations on problems that mimic actual bugs in Bayes Factor computation (omitted normalization constant, improper use of Savage-Dickey density ratio, mismatch between simulation and models. 4/

Show threadMartin ModrákMar 19
One general lesson is that most often, apparent problems were in the simulation code and not in the actual BF computation (I in fact e-mailed bridgesampling developers after consistently finding problems in their computation but it later proved to be a bug in my sims). 5/
Show threadMartin Modrák
You may have heard of the previous work by Daniel J. Schad and @ShravanVasishth on the topic (https://arxiv.org/pdf/2103.08744, https://psycnet.apa.org/fulltext/2024-47778-001.html) - where they do only the data-average posterior check. In a recent preprint we collaborated to bring the lessons we learned to the models they find of interest https://arxiv.org/pdf/2406.08022 6/
Mar 19 at 10:38amWeb
Show threadMartin ModrákMar 19

We do throw some shade on the Good check, but a tweaked version was just published (https://arxiv.org/pdf/2602.19838) which appears to address some of the major problems we report. We didn't manage to include it in our current simulations but we definitely plan to do a head-to-head comparison at some point.

We thank Nikola Sekulovski and @EJWagenmakers for kindly providing feedback on an early version. They also tipped us on the further developments for the Good check. 7/8

Show threadMartin ModrákMar 19

My older thread on SBC in general is at https://fediscience.org/@modrak_m/109301406944300548

And there is an R package implementing all of the techniques we discuss: https://github.com/hyunjimoon/SBC/

Martin Modrák (@[email protected])

Attached: 2 images The basic idea is that you implement your model twice: beyond a probabilistic program (e.g. in #Stan, #jags, ...) + a sampling algorithm you also need a simulator drawing from the prior distribution - this tends to be easy to implement. You then simulate multiple datasets, fit those with your probabilistic program and compute ranks of the prior parameter values withing the posterior. If you did everything correct, the ranks are uniform. Non-unifomity then signals a problem. 3/

FediScience.org