People like @emtiyaz, Alex Immer, Erik Daxberger, Matthias Bauer, Runa Eschenhagen and Agustinus himself have built a beautifully comprehensive framework to turn deep models approximately into Gaussian processes, and thus transfer all the clean mechanisms associated with GPs to deep learning:

Calibrated, learnable uncertainty; out-of-distribution robustness; architecture-optimization by evidence maximization; multi-task, life-long learning, and the list goes on.

@emtiyaz At the end, I sneak myself on stage on final time to summarize: https://youtu.be/t2CSqdfmKGA

Long story short:

Computation is inference. Whether information is loaded from disk or prepared directly on the GPU is not a fundamental distinction. If we quantify uncertainty everywhere, we can actively control, guide, manage and monitor the use of empirical as well as computational data.