Jernej Simončič �
Gabriele SveltoIf you have an Intel Raptor Lake system and you're in the northern hemisphere, chances are that your machine is crashing more often because of the summer heat. I know because I can literally see which EU countries have been affected by heat waves by looking at the locales of Firefox crash reports coming from Raptor Lake systems.
Ludovic :Firefox: :FreeBSD:@gabrielesvelto wtf
@usul Raptor Lake systems have known timing/voltage issues that get worse with temperature. Things are so bad at this time that we had to disable a bot that was filing crash reports automatically because it was almost only finding crashes from people with affected systems https://bugzilla.mozilla.org/show_bug.cgi?id=1975808
Hubert Figuière@gabrielesvelto @usul but isn't it supposed to be "Designed in California" ?
oh wait, they all have AC.
Fabrice Desré@hub @gabrielesvelto @usul The climate in central California is temperate, not tropical. It's just almost always sunny, but you don't really need an AC.
StellaFoxxie 
@fabrice @hub @gabrielesvelto @usul you definitely need ac like 60%+ of the year if you think 80 degrees is hot lol
@StellaFoxxie @fabrice @hub @gabrielesvelto anything above 40 is too hot. ACs are not the solution they are the problem.
@usul @fabrice @hub @gabrielesvelto this somehow feels relevant for some reason https://www.youtube.com/watch?v=1UXs8TzlGCU
americans .00002 seconds after Europeans complain about heat
Ursidinoj/The Bjornsdottirs@usul @StellaFoxxie @fabrice @hub @gabrielesvelto 1. they are necessary life support 2. i think you're talking at cross purposes, the Americans unwisely use fahrenheit without signalling it because they think the whole world uses fahrenheit when we don't, we celsius #lang_en
@ellenor2000 @usul i feel like the use of "80 degrees" when referring to human habitable air temperature along with the context of "in California" is enough social cues for a reasonable person to understand that yes i'm referring to 80 degrees fahrenheit
@StellaFoxxie that's the thing, neither in F nor in C is 80 a human habitable air temperature (in F it's survivable, for extended times, yes, but not habitable). so specify a unit. #lang_en
janet (she/her)@ellenor2000 @StellaFoxxie i am sitting here at 80°F (which is about 26.6°C)
if this is not "habitable" to you, i dont know what to tell you.
@usul @StellaFoxxie @fabrice @gabrielesvelto when an American say 80 degress, it's unlikely to be above your 40C. It's actually just below 27C. They use a different scale that no one understands.
Geert Uytterhoeven@hub @usul @StellaFoxxie @fabrice @gabrielesvelto Yeah, the 80-equals-40 rule is only valid for alcohol content in liquor.
Earth Notes@geert @hub @usul @StellaFoxxie @fabrice @gabrielesvelto The -40C = -40F is one I exprienced as a little kid in Canada - comfort cooling wasn't a thing!
Radieschen@usul ACs are not part of the problem, they are part of the solution IMO. They are heat pumps and can be used for heating as well as cooling. When used for cooling, solar energy is vastly available. They will be necessary in a hotter climate because disabled and older people are more likely to die in heat waves.
As long as people don't cool down their houses to 20 degrees when it's 35+ outside, I really don't see a problem. Just put some solar panels on the roof or the balcony when getting an AC.
txt.file
The Lack Thereof 
@hub
Naw, that's Apple's tagline.
Intel has an engineering office in Santa Clara, but Raptor Lake was designed by one of the overseas teams.
Naw, that's Apple's tagline.
Intel has an engineering office in Santa Clara, but Raptor Lake was designed by one of the overseas teams.
Admiral SnackbärI know next to nothing about CPU design, but isn't there supposed to be a fan bolted to the CPU?
Isn't said fan supposed to spin up, when heat rises?
I didn't really understand how relatively mild heat can affect CPUs this much.
Leeloo@AdmSnackbar @gabrielesvelto @usul
- A fan cannot cool it below ambient temperature.
- The overheating parts may not be the ones touching the heat sink.
- The fan may be inadequate.
It used to be normal to get crashes if the cooling was inadequate, but for a couple of decades, cpus have been able to throttle down to manage overheating. Apparently Intel made a mistake in this one.
But are we talking about actual overheating here? These CPUs are relatively new, so I'd guess they're not utilized 100% by simply running firefox.
@AdmSnackbar @gabrielesvelto @usul
That argument could point towards the overheating part not being the one that's expected to get hot, and thus not touching the heat sink.
That argument could point towards the overheating part not being the one that's expected to get hot, and thus not touching the heat sink.
@leeloo @AdmSnackbar @usul Windows running updates in the background is sufficient to peg a CPU core to 100% and lead to significant heat dissipation. I'm just seeing the effects on Firefox, not what's causing it.
Stryder Notavi@leeloo @AdmSnackbar @gabrielesvelto @usul Also, apparently +1° Ambient usually translates to +1° System, since the cooling system is using the hotter air to cool itself
@AdmSnackbar @usul there's a lot of moving parts to CPU cooling. For instance a thermal paste is used between the CPU and heatsink. In OEM systems this usually dries up within a couple of years, increasing thermal resistance. So even with the fan at full speed the heatsink might be unable to cool the CPU adequately. Additionally a single core might have significant load and cause a hotspot even if the others are idle.
Manic Pixie Dream Gremlin@gabrielesvelto you are man after my own heart
Ángela Stella MatutinaOh yes, the two seasons:
https://social.treehouse.systems/@angelastella/114774518321937143
Yellow Flag@gabrielesvelto Guess I’ve been lucky so far. My CPU heats up considerably occasionally, yet not a single crash despite it being a Raptor Lake CPU. In fact, I’ve only had two Firefox crashes since 2022 (December 2024 and November 2023 respectively). Quite remarkable compared to what it used to be.
@WPalant this could be useful info for me. What exact model do you have and what microcode version is it running? Could you try running Firefox nightly? It has a new library that we can't roll out to the stable release channel because it triggers Raptor Lake bugs very frequently. I'm investigating workarounds for that so the more I know about these systems the better it is. In particular it seems to me that the higher-clocked versions are far more affected than others.
@gabrielesvelto /proc/cpuinfo says:
vendor_id : GenuineIntel
cpu family : 6
model : 186
model name : 13th Gen Intel(R) > Core(TM) i7-1360P
stepping : 2
microcode : 0x4124
cpu MHz : 1178.637
cache size : 18432 KB
I’ve updated Nightly, haven’t used it in a while. Will try to use it a bit. How often is “very frequently”?
@WPalant it usually happens when loading a page. Try navigating to YouTube's main page, letting it load fully and then reload it a few times in a row. If it hasn't crashed after a couple dozen times it shouldn't crash.
This is the relevant bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1950764
@gabrielesvelto This managed to heat up the CPU to 70° C (at least for a brief time) but no crash.
@WPalant good. Your CPU tops at 5GHz which means it probably still has enough timing headroom, so the issue doesn't manifest itself. The most expensive Alder Lakes run at 5.40GHz and higher, and I suspect that's when things get real bad. Plus they likely heat all the way to 100C during use which compounds the problem by increasing the circuits' resistance (and thus reducing the timing slack).
@gabrielesvelto Good to know. This is a mobile CPU, and I’m quite happy that it won’t get any warmer than 70° C. As it is, this is already uncomfortable enough in summer.
@WPalant I just noticed something else. Your CPU is a model 186, from BLL/RPL-P sub-family. It was a more recent stepping than BLL/RPL-S which is the one affected by the timing issue. It shouldn't have that timing problem at all
LeoBurrI could probably try this out on a Model 183 i7-14700k later on. It's on a Liquid Freezer III and is set to use a max of 175w, and I haven't seen it get over about 70C. Latest Intel Microcode.
@LeoBurr that's our boy. I've got *plenty* of crash reports from that particular model.
@gabrielesvelto Running Firefox 141.07b on Windows 11, Max core Temp of 50C, min core temp of 33C, during this test, with clock rate fluctuating from 4600MHz-5600MHz, did 36 reloads (complete) of a logged-in YouTube. No crashes no issues. Idle core temps ranging from 23C-28C.
@LeoBurr can you try our nightly builds too? Also, did you update your BIOS to get the new microcode or is it loaded by the OS? Reading through Intel`s errata I now realize that affects the system differently. And BTW those are very good temps, if all those systems worked at those temperatures we'd see a lot less problems.
@gabrielesvelto I can try it again later, sure. Microcode is through the BIOS. (Not OS) I can also try a nightly.
I’ve had very good experiences with Raptor lake on HP Z-series workstations that honor the factory PL1 and PL2 wattages on factory new CPUs with the mitigations in place.
The real issue I’m seeing is with gaming motherboards that set PL1 and PL2 to 4095 and folks are using regular cooling in hot climates, so the CPU is running too much power and doing so at too hot a temperature.
The oddity is that these CPUs are both more power efficient at lower clock speeds and faster at single core workloads at those lower clock speeds than contemporary AMD CPUs. But they have to be pushed to absurd wattage to compete on the high end for gaming.
This, coming from someone using a 9950X3D as a primary gaming system. And a M4 Max Mac Studio beside it.
This i7-14700k is a TV gaming PC.
At any rate, I’ll find time over the next 4 days to get you more data. :)
Xer Shadow Tail ☄️🌺🏳️⚧️@gabrielesvelto @WPalant I have a Core i7-1370P as well and have not had any issues with random crashes or system stability. Being mobile, it seems even if I push it that it will ramp up the fan and downclock the CPU if things get past 80C which generally goes well with quickly cooling the CPU. Using it daily as my main system, it is pretty aggressive with cooling.
@XerShadowTail while checking the datasheets I just realized that both your and @WPalant CPUs are from the model 186 line (aka BLL/RPL-P), so they should be free of the timing issue. They have other bugs but not as bad. Raptor Lake model 183 CPUs are the bad ones
@gabrielesvelto @WPalant This is rather promising. I suppose for these CPUs Intel was much more conservative with fancy stuff .
Niko@WPalant @gabrielesvelto yeah i've never had such issues on a 13700K i seem to have gotten extremely lucky
Koopa512 
🚩🏴@niko @WPalant @gabrielesvelto I usef to have those quite often but they all went away when I uodated it (and it'd also crash at about the same freq on my laptop with an i5-2540M)
(I have a 14700KF though)
(I have a 14700KF though)
TeflonTrout 
he/him@gabrielesvelto probably time to upgrade to the new Gravy Lakes
Janik Besendorf@gabrielesvelto Plesse draw a map and share it here!
@besendorf I'd love to but building a visualization from this data is a lot of work and I've had a very long day already 😫
lost media autumn@gabrielesvelto JUST missed it with alder lake 🙏 the last good architecture intel made
Shalien@gabrielesvelto Mozilla telemetrics allow the Mozilla Foundation to locate users. Nice
@shalien crash reports aren't part of telemetry, they're opt-in and explicitly submitted by users. Locale is the language the browser is set to, not the user location. But obviously a user using Firefox in French is most likely to be in France
@gabrielesvelto There is browser.search.region preference being recorded in the crash reports. This should match the user’s country quite well.
@shalien You can look around on https://crash-stats.mozilla.org/ yourself to see what kind of data crash reports contain. Or you can pull up your own reports on about:crashes if you submitted some. E.g. this is a crash report I picked randomly, coming from what appears to be a Romanian Firefox user: https://crash-stats.mozilla.org/report/index/1c5ab5fc-5e1d-4f08-a7b5-5a20b0250707
Orca 🌻 | 🎀 | 🪁 | 🏴🏳️⚧️@gabrielesvelto @shalien
Does that mean many reports from "English (US)" likely aren't actually from US (or even from English as first language country)?
Does that mean many reports from "English (US)" likely aren't actually from US (or even from English as first language country)?
Dan Veditz@Orca @shalien @gabrielesvelto
Yes, the "English (US)" locale is not a useful bucket for this kind of idle amusement.
Jessie Nabein 
@Orca @shalien @gabrielesvelto possibly yes, given English (US) is used in many parts of English Canada for example; as English (CA) has minor but sometimes "annoying" differences. IIUC
ay
Captain of the SS El Faro@gabrielesvelto takes me back to my overclocking days when my system was on the screaming edge and a one degree increase in ambient would make it crash. Of course, I did that, to myself, on purpose. Intel doing that to all their customers is just desperation. No no, Moores law isn't dead. Look, it's faster! As long as summer isn't a thing, that exists, in reality.
Samara@gabrielesvelto I'd assume this is related to the degradation issues that are known in 13th and 14th gen, the heat is exacerbating the problem https://en.wikipedia.org/wiki/Raptor_Lake#Instability_and_degradation_issue
@StrangelySamara yes, though some of these bugs are very repeatable and predictable which is puzzling. They might be timing-related or even genuine consistent CPU bugs (we've seen a fair bit of those too)
MidniteMikeWrites@gabrielesvelto @StrangelySamara Weren't there microcode fixes to improve voltage/timing? Though I also heard that earlier batches might have had physical flaws too.
13th/14th gen was a mess that Intel tried hard to sweep under the rug. Kind of seems they got away with a lot.