Memory errors in consumer devices such as PCs and phones are not something you hear much about, yet they are probably one of the most common ways these machines fail.
I'll use this thread to explain how this happens, how it affects you and what you can do about it. But I'll also talk about how the industry failed to address it and how we must force them to, for the sake of sustainability. 🧵 1/17
First of all let's talk briefly about how memory works. What you have in your PC or phone is what we call dynamic random access memory. That is memory that stores bits by putting a minuscule amount of charge into vanishingly small capacitors (or not putting it in if we're storing a zero).
These capacitors continuously leak this charge, so it needs to be refreshed periodically - every few milliseconds - which is why it's called "dynamic". 2/17
Diagnosing it is hard. Windows has a memory diagnostic tool which will catch the worst offenders and is easy to use: https://www.microsoft.com/en-us/surface/do-more-with-surface/how-to-use-windows-memory-diagnostic
It's not enough though, some errors can only be caught with more extensive testing. I recommend the open-source memtest86+ (https://memtest.org/) tool or the closed source memtest86 one (https://www.memtest86.com/) 8/17
Now you might wonder: how often does this actually happen? The common wisdom on this topic is that hardware failures are so rare that software bugs will always dwarf them. As I found out this is demonstrably false.
While investigating Firefox crashes I've come to the conclusion that several of the most common issues we were dealing with were likely caused by flaky hardware. This led me to come up with a simple heuristic to detect crashes potentially caused by bit-flips. 10/17
Deploying this heuristic to Mozilla's crash reporting infrastructure has been eye opening: if I take the 10 most common crashes on Windows, 7 are out-of-memory conditions - that is, not bugs - and 3 are likely caused by bad memory.
You've read that right, three out of the ten most common reasons why Firefox crashes on Windows are caused by memory that's gone bad. 11/17
@rlb we've reduced OOM crashes massively in the past few years: https://hacks.mozilla.org/2022/11/improving-firefox-stability-with-this-one-weird-trick/
We already handle gracefully all the failures that we can realistically handle, but for a lot of them there's nothing we can do. That's especially true on non-Windows platforms where allocations never fails. Both on Linux and Android the kernel will kill processes to save memory without informing them or allowing for any type of reaction, so graceful handling is impossible.
Gabriele Svelto [moved]
rlb