Jonas

Doubling down on my point from yesterday that we don't care enough about proper OOM management.

I just played around with a giant zram device just see if we can make way more use of compressed memory.

Turns out yes, suddenly my Firefox on Linux holds 250 tabs in memory without any disk-backed swap.

Kinda makes the point that having a fixed-size zram device is a bad idea, and the kernel should just compress as much memory as possible?

Mar 7 at 11:20amWeb
Show threadJonasMar 7
zramctl shows 20 GB compressed down to 4.7 GB, and this is not testing with flooding memory with zeros, this is my actual Firefox session with tons of different tabs.
Show threadzyxhere💭Mar 7
@verdre hehe thats how zram works actually https://gitlab.postmarketos.org/postmarketOS/pmaports/-/commit/ffb0d171 ;)
main/postmarketos-zram: increase default zram factor/size to 150% (ffb0d171) · Commits · postmarketOS / pmaports · GitLab

This "size"/factor/percentage represents the total amount of **uncompressed** data that is allowed to be written to the zram-backed swap device. It is **not** the amount of RAM to use for...

GitLab
Show threadJonasMar 7

@zyx yes, the size of the zram disk is the max size of the uncompressed data that can then be compressed by the zram module (see https://docs.kernel.org/admin-guide/blockdev/zram.html).

In the case of my firefox tabs here, it seems like a 4:1 compression ratio is achieved, so that would make the case for a zram device as large as 400%.

zram: Compressed RAM-based block devices — The Linux Kernel documentation

Show threadThibMar 7
@verdre a friend once described zram as “the closest thing we have to downloading new ram”
Show threadAdrian VovkMar 7
@verdre There's also zswap, which I think we should try. The downside is that it requires a swapfile or swap partition. The upside is that it's a dynamically sized pool of compressed pages, and it seems to actually be better suited for this task than zram
Show threadJonasMar 7

@AdrianVovk Yup, the concept behind zswap seems to be much more fitting for us.

Still, in my experiments, zswap causes my firefox to be killed way earlier than when using the giant zram (it should be much later instead, as now there's also real disk storage available to write to).

Show threadJonasMar 7

@AdrianVovk My wild guess as to why that's happening is that the kernel isn't smart enough. I'm testing with a 16 GB swapfile in addition to my 16 GB of RAM.

The kernel now fills up the 16 GB of uncompressed RAM, then starts swapping out, keeping track of each byte that it's swapping out. As soon as it swapped out 16 GB (even though zswap didn't write anything back to disk, I checked that), memory pressure rises and the OOM killer kills firefox, when actually there's more memory available.

Show threadJonasMar 7

@AdrianVovk The concept of exposing potentially infinite memory (in case compression ratio is infinite) as "fixed size" swapfiles just seems wrong to me.

On macOS I can keep writing zeros to memory and the kernel just transparently compresses it away.

Show threadAdrian VovkMar 7
@verdre Well macOS also exposes it as swapfiles. When it can no longer compress stuff it just spills over onto disk
Show threadAdrian VovkMar 7

@verdre Have you played with:

- tuning zswap's pool size (it has a limit in terms of maximum memory use!)
- tuning zswap's compressor
- turning on the zswap shrinker, which periodically evicts cold pages to disk
- tuning vm.swapiness and the other related sysctls

Also, you're talking about memory pressure, which is managed by userspace memory killers. Are you sure it's the kernel killing your stuff and not oomd misconfigured to fire too early?

Show threadJonasMar 7

@AdrianVovk Oh yeah it's the kernel OOM killer.

I'm playing around with zswaps tunables right now actually.

I think my hypothesis is confirming though. I'm doing an absolutely cursed experiment right now and using a sparse swap file that's exposed to the kernel via a loopback device, so that I can overcommit available disk space and measure the actual data written to disk. It's now managing to load all my firefox tabs without OOMing, with almost nothing written to disk.

Show threadJonasMar 7

@AdrianVovk Okay!

Achieved the same behavior with zswap as before with zram now: 0 bytes written back to disk, while all my firefox tabs are loaded.

To get there, adjusted zswap config to disable the shrinker (so it doesn't lazy-write back to disk).

Show threadJonasMar 7

@AdrianVovk the compression ratios with zstd are just insane. I managed to make it store as much as 30 GiB compressed in 7 GiB of physical ram.

Sadly seems like at this point zswap chokes, possibly unable to acquire more ram for its compressed pool, and things get super laggy. Apparently zswap doesn’t start writing compressed pages to disk until max_pool_percent is reached (with the shrinker disabled obviously), and if that percentage can’t be reached, things go bad.

Show threadRobert MaderMar 7

@verdre AFAIK distros like #Fedora currently tend to default to something like 50% of ram for the default zram size, while e.g. #postmarketos recently switched to 150%, see https://gitlab.postmarketos.org/postmarketOS/pmaports/-/merge_requests/7754, #pureos to 200% (https://source.puri.sm/Librem5/librem5-base/-/merge_requests/381) and Android and ChromeOS both overcommit as well.

I saw this helping a lot when compiling stuff (like libcamera) on my Pixel3a, which previously aborted because of OOM - that vanished completely.

So this might be a low hanging fruit for various distros.

zram swap: increase default size to 150%, add sysctl tweaks, refactor (!7754) · Merge requests · postmarketOS / pmaports · GitLab

This branch does a couple of things: increases the default zram size to 150% (see the commit for a lot more details),...

GitLab