An illusion of predictability in scientific results: Even experts confuse inferential uncertainty and outcome variability
https://doi.org/10.1073/pnas.2302491120
Visualizing only inferential uncertainty can lead to significant overestimates of treatment effects, even among highly trained experts
Solution: when possible, plot individual data points alongside statistical estimates
How big problem it is that cross-validation (CV) is biased? I briefly discuss some points on this.
0. Unbiasedness has a special role in statistics, and too often there are dichotomous comments that something is not valid or is inferior because it's not unbiased. However, often the non-zero bias is negligible, and by modifying the estimator we may even increase bias but reduce the variance a lot, providing an overall improved performance.
In depth intro to TMB (bit like stan) in #RStats. Detailed, specific application of state-space models.
A lot to take in, but the best video I've seen on the topic.
Heard through my predoc fellow Hadiqa Tahir. https://www.youtube.com/watch?v=V_2Aw_GvzqM
MetaBayesDTA: codeless Bayesian meta-analysis of test accuracy, with or without a gold standard"
New paper by Enzo Cerullo.
Paper: https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-023-01910-y
Background The statistical models developed for meta-analysis of diagnostic test accuracy studies require specialised knowledge to implement. This is especially true since recent guidelines, such as those in Version 2 of the Cochrane Handbook of Systematic Reviews of Diagnostic Test Accuracy, advocate more sophisticated methods than previously. This paper describes a web-based application - MetaBayesDTA - that makes many advanced analysis methods in this area more accessible. Results We created the app using R, the Shiny package and Stan. It allows for a broad array of analyses based on the bivariate model including extensions for subgroup analysis, meta-regression and comparative test accuracy evaluation. It also conducts analyses not assuming a perfect reference standard, including allowing for the use of different reference tests. Conclusions Due to its user-friendliness and broad array of features, MetaBayesDTA should appeal to researchers with varying levels of expertise. We anticipate that the application will encourage higher levels of uptake of more advanced methods, which ultimately should improve the quality of test accuracy reviews.
If your excellent analysis correctly infers that A causally increases B, this fact remains true without reference to the model and can be used in practice.
If your excellent prediction model finds that A predicts B, this is only true within the exact model you fitted.
In contrast, here's the defaults worked out by a few #RStats nerds in a shed.
(I'm aware that if histogram defaults are carried over from S then they probably had a building rather than a shed).
Looking at some disease counts in an atrocity of a spreadsheet so thought I'd do a quick histogram in excel rather than messing around getting it into #RStats.
And this is the best defaults a $2 Trillion company can come up with for one of its core products?
A typical scientific question is, "what is the value of some parameter", say b.
If we don't have enough data to estimate b, we often resort to a broad, stupid question, "is b not 0?".
With enough data to get tiny p values, you don't have to resort to the stupid question. You can just ask "what is the value of b?"
Ice breaker at a work away day tomorrow: "In 3 minutes present two interesting figures from your research".
It would be very easy to accidentally let this consume my brain for a week. But I won't.
#DataViz
I am in the process of benchmarking our NCBI Taxonomy package in #JuliaLang (https://github.com/PoisotLab/NCBITaxonomy.jl), and so it's a good time to tell everyone that I dislike benchmarks.
Not because we don't compare well (our worst speedup so far is 5x, our best is of the order of 10⁵x).
But because benchmarks assume that we all want the same thing, and this thing is speed. This is not true.
Francisco Rodriguez-Sanchez
Aki Vehtari
Timothée Poisot