Memory errors in consumer devices such as PCs and phones are not something you hear much about, yet they are probably one of the most common ways these machines fail.
I'll use this thread to explain how this happens, how it affects you and what you can do about it. But I'll also talk about how the industry failed to address it and how we must force them to, for the sake of sustainability. 🧵 1/17
First of all let's talk briefly about how memory works. What you have in your PC or phone is what we call dynamic random access memory. That is memory that stores bits by putting a minuscule amount of charge into vanishingly small capacitors (or not putting it in if we're storing a zero).
These capacitors continuously leak this charge, so it needs to be refreshed periodically - every few milliseconds - which is why it's called "dynamic". 2/17
Diagnosing it is hard. Windows has a memory diagnostic tool which will catch the worst offenders and is easy to use: https://www.microsoft.com/en-us/surface/do-more-with-surface/how-to-use-windows-memory-diagnostic
It's not enough though, some errors can only be caught with more extensive testing. I recommend the open-source memtest86+ (https://memtest.org/) tool or the closed source memtest86 one (https://www.memtest86.com/) 8/17
Now you might wonder: how often does this actually happen? The common wisdom on this topic is that hardware failures are so rare that software bugs will always dwarf them. As I found out this is demonstrably false.
While investigating Firefox crashes I've come to the conclusion that several of the most common issues we were dealing with were likely caused by flaky hardware. This led me to come up with a simple heuristic to detect crashes potentially caused by bit-flips. 10/17
Deploying this heuristic to Mozilla's crash reporting infrastructure has been eye opening: if I take the 10 most common crashes on Windows, 7 are out-of-memory conditions - that is, not bugs - and 3 are likely caused by bad memory.
You've read that right, three out of the ten most common reasons why Firefox crashes on Windows are caused by memory that's gone bad. 11/17
Now there's a few things that are worth mentioning: users with bad hardware will be over-represented in this category, their machines will crash far more often than others.
The second thing is that Firefox is exceptionally stable, we've driven down its crash rate by more than >70% in the last few years. But Firefox is also a 30 million-lines-of-code monster. There are bugs in there, but they're less common than hardware failures! 12/17
@gabrielesvelto @willcage IBM designed parity checks everywhere in the System 360, so that the system would quickly stop in the event of hardware failure. ECC was implemented in main memory systems when it was discovered that solid state RAM was subject to transient memory failures from cosmic rays (really.) Early PCs used memory parity checks (which weren't adequate) until Apple abandoned them for cost reasons. Bad mistake.
https://www.bbc.com/future/article/20221011-how-space-weather-causes-computer-errors
@flyingsaceur @ravenonthill @gabrielesvelto @willcage
1976... I understand the point they are making, but it is somewhat disingenuous.
Apple started manufacturing computers targeted at enthusiasts who wanted something affordable. The lack of parity was not exclusive to Apple, few home computers had it. 5 years later when the IBM PC came out it was aimed at businesses and did support parity (and was very expensive).
Having said that, I also bemoan the lack of ECC capable systems daily.
Thanks for the thread, and the interesting data. As the author of another memory-testing tool, I'd also like to point out that memory errors are also quite commonly triggered by things other than DRAM actually going bad. This is one of the reasons that swapping memory sometimes doesn't make the errors go away.
Includes: flaky power supply, overheating CPU or RAM, micro-cracks or scratches in the mainboard, oxidized contacts in memory slots or on modules ...
1/x
... the list goes on. Sometimes I'm amazed anything computer-related works at all.
My tool, by the way, is called `memtester` - not really of use for Windows users, but it works on anything vaguely Posix-like, including Linux, the BSDs, all commercial Unices, and some weird stuff like VxWorks -- and VMS, the weirdest of them all.
It's actually pretty good at finding intermittent errors that memtest86+ misses.
</plug>
So this reminded me of a Linux kernel feature I'd read about some time back where the kernel could be told on boot to avoid curtain bad sections of RAM, so I searched for it and it's there (see Grub option GRUB_BADRAM). But I also discovered that there's now a kernel option ("memtest=") that will do a memory test on boot and automatically not use addresses it detects are bad, which is *really* *cool*.
I think there's a lot that can be done in this vein at the OS level. Consider an OS thread that during times of low activity tests individual pages of RAM and adds failures to the kernel's block list (and also exports it to the userland so it can be saved for the next boot.)
That would let one continue to use failing RAM (mostly) safely for as long as it was still *mostly* okay and keep it out of e-waste longer.
@gabrielesvelto Really depressing that we've reached the physical limits of creating "memory we're confident that actually will store it's value reliably" :(.
We've went from PARITY CHECK 1/2 to "memory works fine without detection or correction" to "oh now not even parity check is enough". In that sense, it's WORSE than 40 years ago :P.
@jbqueru @gabrielesvelto @cr1901
Also many of the machines just had simple software. Something like AppleSoft BASIC was just easier to keep bug-free (ish) than the giant software of today.
The same size/complexity software is more reliable today, but what they actually shipped was much smaller and simpler.
How can memory errors cause the same crash on two different computers unless they both have the same error at the same address? (Which is of course astronomically unlikely.)
I see. A similar phenomenon was observed in initializing bitcoin full nodes, where verifying the full chain (currently 5TB and growing) exposed a lot of memory errors due to hash mismatches.
That happened essentially because everyone was doing the exact same calculations on the exact same data, in the exact same order, and expecting a known result.
I would have expected a lot more randomness in Firefox, but I forgot how much manual memory management it does.
@rlb we've reduced OOM crashes massively in the past few years: https://hacks.mozilla.org/2022/11/improving-firefox-stability-with-this-one-weird-trick/
We already handle gracefully all the failures that we can realistically handle, but for a lot of them there's nothing we can do. That's especially true on non-Windows platforms where allocations never fails. Both on Linux and Android the kernel will kill processes to save memory without informing them or allowing for any type of reaction, so graceful handling is impossible.
@gabrielesvelto FYI, this seems to be a nonnegligible cause of death of Nintendo 3DS units. some units either exhibit strange behavior, corruption in text characters (which turn out to be single bitflips), or just straight up refuse to boot
it's possible to demonstrate these are indeed DRAM* errors by using the boot9strap jailbreak (with ntrboot if not installed beforehand), as these run from SoC-internal SRAM instead of DRAM. booting the OS then typically fails, and it can also be used as a point to run a memtest
problem is that replacing the DRAM chips is *very* difficult. not only would it require BGA rework (because it's so small they couldn't not solder it on the mobo), Nintendo also used epoxy 'underfill' to glue the DRAM chip stuck to the PCB to deter RAM probing attacks (as those were used against the DSi, the 3DS' predecessor), see the "white glue" here: https://giltesa.com/wp-content/uploads/2013/12/Nintendo_3DS_PCB-Top.jpg
*: DRAM is more often called FCRAM on the 3DS because that's the type of DRAM by fujitsu it uses
@gabrielesvelto though, when running such a memtest, in most cases the errors seem to have the same pattern throughout the entire DRAM.
this means it's probably a solderball crack under the DRAM or SoC, rather than actual semiconductor failure, though the latter is also sometimes observed
@gabrielesvelto thanks for an interesting thread!
I read talk, some years ago, about memtest+ not being maintained and actually not working at all for most cases (iirc it was in the context of the Ubuntu live cd boot option)
Do you know how it looks these days?
@gabrielesvelto FWIW, I wrote a "memtest.js" version that runs in the browser. I even got some bad RAM sticks from Mozilla RelEng to verify that it could detect real failures!
Live version still works, too: https://dolske.net/hacks/memtest.js/live/
@gabrielesvelto Of course there are some limitations, since JS (thankfully) doesn't have bare-metal access. But I wanted to see if periodically testing whatever chunks of memory the browser/OS gave out would work well enough.
That is, it can't say "all clear", but it can say "problems found". The idea being to eventually have the browser itself run a small background check, which over time should either detect any bad bits or give confidence that things seem OK.
@gabrielesvelto Alas it was just a side project for fun, so I set it aside after the proof-of-concept.
It seems like an interesting problem space, so I hope you get good results!
Oh, the old code: https://github.com/dolske/memtest.js
(It was also an excuse to play with the then-new asm.js, for the hot bitwise-op loops. That code is lost, but IIRC it wasn't any faster because whatever JIT we used then already did a good job.)
Frederik Seiffert
Gabriele Svelto [moved]
Jon (now at neuromatch.social)
Chris [list of emoji]
Osma A 🇫🇮🇺🇦
Raven Onthill
AN/CRM-114
Louis Gerbarg
💡𝚂𝗆𝖺𝗋𝗍𝗆𝖺𝗇 𝙰𝗉𝗉𝗌📱
Rob Fahrni
C.
Ludovic :Firefox: :FreeBSD:
Tommy Þ
Gorgeous na Shock!
William D. Jones
@
Noah Gibbs
Thomas Steiner 
Colm Ó Donnchadha
Forbearance
Alistair Buxton
rlb
Jfrench
PoroCYon @ 
Wyatt (🏳️⚧️♀?)
kamstrup