Mastodawn

Valeriy M., PhD, MBA, CQF Jan 18

Let's discuss how we can innovate beyond traditional methods to ensure our models truly generalize.

#MachineLearning #DataScience #CrossValidation #TimeSeries #SpatialData

Dr Mircea Zloteanu 🌼🐝May 29, 2024

#statstab #103 On the marginal likelihood and cross-validation

Thoughts: Can't say I can follow much of this, so I'll open it up to the #bayesian community for input. Seems important though.

#stats #bayes #likelihood #evidence #crossvalidation

https://doi.org/10.1093/biomet/asz077

On the marginal likelihood and cross-validation

Summary. In Bayesian statistics, the marginal likelihood, also known as the evidence, is used to evaluate model fit as it quantifies the joint probability

OUP Academic

⬆️

6) thankfully, Wager (2020) https://doi.org/10.1080/01621459.2020.1727235 shows that cross-validation is asymptotically consistant for model selection, so while what we're doing gives us poor estimates of generalization error and bad error bars, at least it's valid for model selection.

⬆️

5) Bates et al. (2023) https://doi.org/10.1080/01621459.2023.2197686 propose a nested cross-validation estimator of generalization error that's unbiased and has an unbiased mean squared error estimator. It's computationally quite intensive. I played a bit with it, and my in high-dimensional set ups (large p small n) I got error bars that had indeed good coverage of the generalization error, but were also covering most of the [0, 1] interval, which is less helpful.

⬇️

⬆️

4) in any case, error bars are wrong, because it's impossible to get an unbiased estimator of the mean squared error of an estimator that's based on a single fold of cross-validation, as shown by Bengio & Grandvalet (2004) https://dl.acm.org/doi/10.5555/1005332.1044695

⬇️

No Unbiased Estimator of the Variance of K-Fold Cross-Validation | The Journal of Machine Learning Research

Most machine learning researchers perform quantitative experiments to estimate generalization error and compare the performance of different algorithms (in particular, their proposed algorithm). In order to be able to draw statistically convincing ...

The Journal of Machine Learning Research

⬆️

3) cross-validation estimators are better estimators of *expected test error* (across all possible training sets) than of *generalization error* of a model.

This has been known for a while and even appears in The Elements of Statistical Learning, so I should have known about this much earlier. Bates et al. (2023) https://doi.org/10.1080/01621459.2023.2197686 show why this is for linear models.

⬇️