Mastodawn

Once my 16 GB of memory were filled up, with a good extra amount saved thanks to compression, macOS just started creating swapfiles. Memory pressure went from the green to the yellow area, things got a little bit sluggish when switching to the oldest tabs, but no major slowdowns or freezes of multiple seconds.

The more tabs were open, the more swap files got created. At some point the disk would fill up though.

(2/4)

With about 200 MB of disk space left, Firefox froze completely, and at the same time a window popped up telling me that Firefox and a few other apps were frozen (not killed!!!) by the system.

The window prompted to either kill the app or resume it. Pressing resume, Firefox continued running smoothly for a few seconds, until it got frozen again. Pressing resume again, I could quickly close a few tabs, and things worked fine again.

(3/4)

All of this without one tab getting unloaded by Firefox, or one app getting killed without user consent.

So OOM conditions *can* be handled without losing people's data. Don't let kernel developers fool you into believing this is an impossible problem to solve.

(4/4)

sirlan <( rawr )Mar 6

@verdre I mean, some time ago I did ended up running a fork bomb on my system, and it was fine, it was really slow and the system monitor wouldn't open. but nothing got killed. and I don't even use swap, just zram

x0 Mar 6

@verdre Do we know how Windows does this? It's got only the one page file which can fill up I think, and IDK if it can dynamically resize.

@x0 Nope, but would be interesting to take a look. I think they are also notoriously bad at this, so wouldn’t expect too much…

Ein Philosophiker Mar 6

@x0 @verdre Windows freezes, does not wake up, shows a black (sic!) screen, complains about "cannot login".

Ein Philosophiker Mar 6

@x0 @verdre I know because node instance went crazy w/o real reboot after a week

MarkAssPandi Mar 6

@verdre Sorry for sounding like reply guy but this is just unfair comparison.
Kernel OOM is last resort thing, meant to prevent complete freezing of system, which both Linux and MacOS have. MacOS can handle this better because it uses also (partially, more on that later) userspace Jetsom (its not rly named that on MacOS but iOS, but its essentially the same system) handler which monitors memory usage and does actions to prevent memory from going critically high so to not trigger the OOM.
There are systems like that on linux, like systemd-oomd, tho it's done more poorly and those also just kill apps, but before they trigger the kernel OOM.
There are also dynamic swap solutions like swapd, which do more or less the exact same thing, but it's not part of the kernel, which means it has to be installed.
I am not saying that ur wrong about the handling of it on Linux (by default), it's mostly done poorly on most if not all distributions by default, and yeah sure distro's could allow to set it up easier so that they also have dynamic swap, but it's not just a Kernel thing and I dislike this "don't let kernel devs fool you".
It's an desktop implementation and usespace issue, not kernel.
Again sorry for long reply but you insulted ppl based on a fallacious comparison so like lol

gnuntoo Mar 6

@verdre Funny, I just watched this video yesterday: https://youtu.be/Qb4v6YaEQa0?si=J1BHZw9g1EkOSQCE

When I was having trouble running out of memory and my entire computer freezing up, I just bought 32gb of ram, but I’m jealous of macOS.

8 GB of RAM wasn't enough? macOS RAM management explained

YouTube

Charly Coste 🇫🇷Mar 7

@verdre I recently came to the same conclusion that the fundamental mistake in how out-of-memory situations are handled under #Linux is that it's based on killing processes rather than pausing them. That leads to doing nothing until it's too late, because killing a process can result in data loss whereas pausing it is relatively harmless (and entirely normal in #Android).

It isn't really a problem with the kernel though. It's the desktop environments that should handle pausing apps when OOM.

1peter10 Mar 7

@verdre I could not agree more, this really needs improvement on Linux.

There's another feature in macOS that saved me a ton of frustration while writing my thesis in 2019. I had forgotten to plug my MacBook Air in, and with LibreOffice maximized for focus, I must have missed notifications about my battery being empty and only noticed when the screen went black. I was afraid of hours of work being lost, but after plugging in and booting up, LibreOffice launched with my document all intact.

@1peter10 Yup, you're talking about hibernation (aka writing memory to disk and then shutting down).

Not exactly cutting edge technology, it even works on most linux devices if you set it up.

Unsurprisingly, Linux distros don't care enough to make hibernate work OOTB, I'm hoping we can figure it out for GNOME OS though.

1peter10 Mar 7

@verdre Yes, it's often not set up. From my experience with having the same happen on Windows 10 (but with dataloss), this disregard may be inspired by Microslop.

⁂ Jnk ∞ 📎19h ago

@1peter10 @verdre I'm pretty sure this is distro-specific, like the swapd thing, because bazzite does this flawlessly along with going to sleep but most distros I tried fail miserably on both tasks. All of this on KDE, so there must be something else going on.

This kind of systems should be standardized. If it exists but no one ships it it's basically worthless.

☃️karolherbst☃️Mar 7

@verdre the good news is: systemd-swap allows creating swapfiles on demand to increase available swap memory.

The bad news: no distribution enables it by default afaik.

@karolherbst @verdre Just thinking: Couldn't #systemd (already handling oom kills faik) expose some interface over Varlink or dbus for Desktops like @gnome or @kde to replicate that functionality?

☃️karolherbst☃️Mar 7

@jzohren @verdre @gnome @kde not the best to answer that, but I don't see why that wouldn't be possible.

The current implementation is entirely in userspace and it could choke on sudden spikes and not being able to create swap files quick. But I'd assume all those issues are fixable given enough motivation.

Jonas Mar 8

@karolherbst systemd-swap is unmaintained though :/

It seems like in the kernel things are in motion when it comes to mm right now. zram just gained compressed writeback, and zram can also set a custom writeback device, so potentially that could be used as a "fake" swapfile.

Marius Gedminas Mar 8

@verdre @karolherbst There's also https://github.com/Tookmund/Swapspace, packaged in Debian as swapspace, also not installed by default.

It worked well enough to completely hide an xdg-desktop-portal-gnome memory leak from me, until I accidentally noticed I had 17 GB of swap files for some reason after running htop.

GitHub - Tookmund/Swapspace: A fork of Jeroen T. Vermeulen's excellent dynamic swap space manager

A fork of Jeroen T. Vermeulen's excellent dynamic swap space manager - Tookmund/Swapspace

GitHub

PointlessOne

Mar 7

@verdre Just a few days ago I accidentally closed a wrong window in Firefox. Thankfully Firefox keeps track of recently closed windows and lets you reopen them. The window had 8k tabs (don't ask). It took some half an hour and 80 GB of swap space but it did reopen the window and lost no tabs. Even tab groups were preserved.

Robert Mader Mar 6

@verdre Not claiming to understand the details here, but it sounds quite feasible to make the oom daemon freeze some memory heavy cgroup instead of killing it. In Wayland compositors like Mutter we'd already detect that a corresponding window is frozen and show a close/wait dialog - doesn't sound *too* difficult to me to hook up some APIs to detect that the process has been frozen for a reason and offer to resume it :/

@rmader yup, I think all the things necessary are quite doable. Systemd could also dynamically create/remove swapfiles based on memory pressure (IIRC this was even discussed at some point).

Probably the reason nobody does is that developers just don’t run into these issues, because they all get the beefiest machines with 64 gigs of ram :/

Daphne Preston-Kendal Mar 6

@verdre @rmader @fogti There is https://github.com/Tookmund/Swapspace but even it has some pretty nasty sharp edges

GitHub - Tookmund/Swapspace: A fork of Jeroen T. Vermeulen's excellent dynamic swap space manager

A fork of Jeroen T. Vermeulen's excellent dynamic swap space manager - Tookmund/Swapspace

GitHub

@dpk @rmader @fogti mmh interesting.. I guess this would be as far as you can get for the dynamic swap files if you want to avoid doing kernel work.

Timothy Roes Mar 7

@verdre I got a similar prompt on Ubuntu when I ran too many apps from a pen drive install (Wait or Force Quit)

Scien Mar 6

@verdre I actually keep running into OOM problems on linux so it's nice for you to point this out. It's handled *utterly* terribly. I've been trying to leverage EarlyOOM and Zram swap and it'shelped a lot, but the default exp is just utterly unacceptable

Tobias Bernard Mar 6

@verdre Who would we need to bribe to get something similar on Linux? @pid_eins? The kernel people? If so, who?

@tbernard @pid_eins All of the above, this kind of stuff needs work all across the stack.

Tobias Bernard Mar 7

@verdre @pid_eins put it on the agenda for the next BTO? 🙃

jon r Mar 6

@tbernard
ZRAM is already available and solves most cases. When more is needed, indeed, dynamic swapfiles sure sound like a possible last resort, when ultimately filling the disk is circumvented, as in Jonas' example.
@verdre @pid_eins

wagStoy Mar 6

@verdre Anytime i complain about how poor Linux's memory usage is someone comes and breathes down my spine about how they managed to keep it down to 200mb by running some window manager.. try that on a windows/mac!!

But thats never what it's about 🥹 swap is _really_ barebones on Linux still. Then you have applications which swap out and never return back to ram, so they *feel* slow when your system was low on memory to begin with. This one used to eat me all the time and all you can really do is hit the swapoff key on your keyboard.

And then... ""sparse"" swap space, so you alloc/dealloc swap space overtime... its just not really possible right now, unless you do some funky stuff like swapon a new segment then swapoff and monitor that over time (no idea if that even works). You probably need some systemd/service doohickey for that right now... at least thats all i can think of.

root42 Mar 6

@verdre you can disable memory overcommitment I think? Never tried that, as I rarely have issues with OOM. But it should give processes an error when they try to malloc, which they then could handle.

@root42 I don’t think anyone in userspace handles malloc() failures, might as well kill apps instead…

Zack Weinberg Mar 6

@root42 @verdre I have not tested this thoroughly, but I believe that a hefty chunk of Linux's problems with swap thrashing are, at root, caused by its default heuristic that allows 20% of RAM to fill up with data that needs to get written to disk, before it actually starts making write() wait for the disk.

20% of RAM nowadays can easily be in the gigabytes. It takes at least a few seconds to flush all of that to even the fastest SSDs. But it takes only milliseconds to fill it up.

Zack Weinberg Mar 6

@root42 @verdre Swap thrashing went away forever for me when I started using these sysctl settings:

vm.overcommit_memory = 2
vm.overcommit_ratio = 95
vm.dirty_background_bytes = 16777216
vm.dirty_background_ratio = 0
vm.dirty_bytes = 67108864
vm.dirty_ratio = 0

which means no overcommit, reserve 5% of swap for root, start background writeback at 16 *megabytes* of dirty data and throttle writers at 64 megs

I'd love to work with someone on a proper study of whether this makes sense for everyone

Zack Weinberg Mar 6

@root42 @verdre I don't tend to have a lot of heavy browser tabs open but I do regularly do big scientific calculations that need swap (for scale, the machine has 32G of RAM and 256G of swap) and it remains usable even when those calculations are grinding away

root42 Mar 6

@zwol I don’t get the browser tabs thing either, rocking some 7-12 year old Macs here running Linux, all with 8G of RAM. But I think the OOM is worse for servers and enterprise applications. If your DB gets killed for example. But then maybe, just maybe, you have other issues. On the desktop this has never hit me. But I don’t push my machines that hard, I guess. My M1 with 16G only goes OOM in macOS due to a Safari bug with patreon.com.

jon r Mar 7

@zwol
This sounds like proper Phoronix material.
@root42 @verdre

Zack Weinberg Mar 7

@yala @root42 @verdre Uh. It has been my impression for many years that Phoronix is in the business of farming ad impressions, and doesn't publish anything worth reading. Is that inaccurate?

jon r Mar 7

@zwol
I can't tell, I'm using an ad blocker. But they get a lot of hardware sent for free for testing, if that is of concern. But I like the overview on happenings in Linux land. I'm not so much into lwm.net and such.
At least Phoronix does extensive hardware testing and is focussed on Linux, not like many other publications, such as Tom's Hardware or so.
@root42 @verdre

Charly Coste 🇫🇷Mar 25

@zwol Turning off overcommit is only viable when there's more than enough RAM and swap for all virtual memory allocations, so it isn't a safe default for everyone. However, in most cases it should be possible to make disabling overcommit viable by adding swap.

@root42 @verdre

Zack Weinberg Mar 25

@Changaco @root42 @verdre Yes, I always provision at least twice as much swap as RAM. I would *like* to also set resource limits that prevent any one process from consuming more than like 90% of total RAM but there currently isn't a good way to do that.

With today's disk sizes, the cost of reserving a few tens of gigs of disk to keep the kernel happy is almost always negligible. I *have* run into space problems with small VMs - but small VMs are where the OOM killer is most dangerous so I cope.

jon r Mar 7

@zwol
Actually I had this with a hard drive that writes 230 MB/s cached and between 1,5 and 9 MB/s constantly.

Even when the transfers were completed, of course they were not done, it took more than ten minutes to unmount the device, initiated right after finishing.
@root42 @verdre

spyke Mar 6

@verdre I here is me running with swapoff. Using swap adds stutters and trashes the ssd.

matzipan Mar 7

@verdre they're not downloading more ram and your Firefox is still working so surely this is happening in the application itself. No?

I'm trying to think of a memory hog you could use to test this out but none come to mind. 😓

@matzipan They are actually downloading more ram, by just using more and more of the SSD as ram, all the way to the point where the SSD runs out of space. :)

matzipan Mar 7

@verdre your screenshot says swap used 0, though. So I’m not sure what you mean.

@matzipan ah yeah, that one’s just a random screenshot where there’s not that much opened yet.

matzipan Mar 7

@verdre but then... Linux has swap too

@matzipan sure, but distros aren’t using it by default, nor do we have a good way to dynamically manage swap. And if memory actually runs out, we just go on a killing spree and kill apps, rather than freezing them and letting the user decide what to do.