@_lennart I'm going to hit you with a super wild guess.

Loose or bad SATA cable.

That's some wild shit, good luck with the troubleshooting.

@_lennart

Does the system come back from this state, or is it something you have to reboot to recover from?

Does anything end up in the system logs? If write IO can't happen, that could be problematic.

What storage device(s) are you using? Are you monitoring them with smartctl / smartd? Does it report anything unusual?

[Edit: sorry, I missed that you said it does sometimes come back]

@_lennart

Two guesses:

First guess: nofiles ulimit is probably too low.

ulimit -a will show you what they're set to, for some insane reason some distros set this to like 1024. The idea being that a user shouldn't have more than 1024 open files.

IME on a few systems, snap has exacerbated this because of the way snap does things.

You can set this via/etc/security/limits.conf:

Then sysctl -p to reload them, then see if it happens again. :)

Second guess: Something is actually locking disk access for some reason (long syncs), which you can look at this using something like iostat or iotop are options.

@_lennart What does DMESG have to say when it's happening? I had a Raspberry Pi that did this because there was a DMA bug in the CPU that needed a workaround. Lots of I/O errors at the time.

Also, what's your 'swappiness' set to, and are you seeing swap in-use go up (even if you have plenty of memory available) at the time?

Are you running a distro that defaults to Transparent Hugepages (e.g., Fedora/CentOS/RHEL), with a workload somewhere that's not good with THP (e.g., redis)?

Can you pop a terminal open with 'sudo top' running and see if maybe a particular kernel process is associated with whatever is going on?

@_lennart
The first thing that comes to mind when file writes are slow is that you have a failing drive, or perhaps an older SSD which is behaving badly and in need of a TRIM. It's reasonable for read performance to be fine if the problem is with writing.

Slow drawing of GUI elements suggests that there's a performance bottleneck somewhere -- If you are having I/O troubles then that could be a side effect, but it may also be something completely unrelated.

The whole thing may just be a graphical problem which is slowing down everything else. Disabling the GUI entirely and switching to a text VT to retest may at least give you a direction to look in.

I would start by taking another look at your top / iostat /sar / or whatever report. You already mentioned that memory and CPU are good, but if you are seeing high IOWait times then that would point to a problem even if the read/s and write/s are low. If you have it configured, "sar" can give you detailed reports going as far back for days or weeks, which is helpful for problems which won't stand still when you want them to. The "dmesg" output should include any warnings thrown by the kernel over non-responsive devices or processes, so that's always worth checking on.

Another option is thermal problems. If anything is overheating that could easily lead to throttling which would look (correctly) like a massive slowdown of the entire system. "sensors" may give you some helpful input there if it has been configured, and if nothing else works you should be able to see all of your internal temperatures through BIOS.

Whatever it is, good luck.

@_lennart If there's nothing obvious in the logs and you've tried testing storage and ram health, it might be helpful to collect deeper metrics with sysstat/sar or other tools... assuming that they'll be able to save their output 🤔

Check out https://www.brendangregg.com/Perf/linux_observability_tools.png for what tools to use where.