In the last week we received ~470000 crash reports, these do not represent all crashes because it's an opt-in system, the real number of crashes will be several times larger. Still, out of these ~25000 crashes have been detected as having a potential bit-flip. That's one crash every twenty potentially caused by bad/flaky memory, it's huge! And because it's a conservative heuristic we're underestimating the real number, it's probably going to be at least twice as much. 2/5

In other words up to 10% of all the crashes Firefox users see are not software bugs, they're caused by hardware defects! If I subtract crashes that are caused by resource exhaustion (such as out-of-memory crashes) this number goes up to around 15%. This is a bit skewed because users with flaky hardware will crash more often than users with functioning machines, but even then this dwarfs all the previous estimates I saw regarding this problem. 3/5

And to reinforce this estimate I've looked at the numbers we got from the users who run the memory tester after having experienced a crash: for every two crashes we think are caused by a bit-flip the memory tester found one genuine hardware issue. Keep in mind that this is not doing an extensive test of all the machine's RAM, it only checks up to 1 GiB of memory and runs for no longer than 3 seconds... and it has found lots of real issues! 4/5

And for the record I'm looking at this mostly on computers and phones, but this affects *every* device. Routers, printers, etc... you name it. That fancy ARM-based MacBook with RAM soldered on the CPU package? We've got plenty of crashes from those, good luck replacing that RAM without super-specialized equipment and an extraordinarily talented technician doing the job. 5/5

@gabrielesvelto My mind was going to cheap low-end hardware, but now you’re throwing expensive Apple Silicon SOC’s in the mix, it’s a bit harder to believe that they suffer from bitflips at the rates your are implying.

@stevenodb @gabrielesvelto low-end hardware might sometimes even be less likely to hit this because it's not even trying to be super fast. High-end hardware chasing the fastest speeds is pushing the limits of stability all the time.

@valpackett @stevenodb @gabrielesvelto yeah on that point, the "meta" for hardware tuning has gone from overvolting and overclocking (to use the margin of error) to undervolting, because the products are already running just about as fast as they can, and you get decent gains by making them run cooler instead of faster
ie, they're already at the limits, and an occasional memory error in 10% of consumer hardware is probably what they're okay with

@izzy @valpackett @gabrielesvelto @astraleureka My AS systems have been rock solid under load, they have literally never crashed, panicked or had crashing applications nor have they been churning out corrupted files. The rates of bit flips people in this thread are prophesizing should have surfaced in the last 5 years in some way that is visible to users. This Firefox report being the first one, strikes me as oddly isolated. 10% in a few billion iPhones iPads and MacBooks is a HUGE amount.