Yehor 🇺🇦3d ago
This is a disk I/O report for the last 30 days of every #Proxmox node of a cluster. Something happened around March 28 that caused high disk usage, and I can’t figure out what. The replication tasks are failing randomly, and actually, any disk operations are slowed down. Also, there are no significant changes in CPU, RAM, or network usage.
I was hoping to find which LXCs are causing this, but they all have similar disk IO graphs.
Well, shit.
#homelab #ProxmoxCluster #HighDiskUsage #zfs #mystery
Show threadYehor 🇺🇦3d ago

Ok, that was kinda premature and stupid panic.

#Beszel agent was not reporting Disk I/O until the recent update. All agents are updating automatically, and that happened just around March 28-30. After that, the data began to flow.

In fact, I have sudden disk issues only with a single #Proxmox node, and it is clearly visible on the IO pressure stall graph. Spikes before May 30 are backups. Then it went crazy.

#homelab #zfs

Show threadYehor 🇺🇦

Also, this is a single WD Green with #ZFS on it. Yes, I know some #Proxmox guru would hang me on a tree for that, but we got what we got.

#homelab #wdgreen #wd #ssd #Proxmox

Apr 8 at 9:37amWeb
Show threadSchenkl | 🏳️‍🌈🦄3d ago
@yehor so the problem is beszel?
Showing false data or is causing the problem?
Show threadYehor 🇺🇦3d ago
@schenklklopfer I'm not sure, but it seems likely. I'll stop all agents and monitor the IO pressure stall graph from #Proxmox.
Show threadYehor 🇺🇦3d ago
@schenklklopfer So Beszel was not in charge. Looks like it was just a coincidence and the drive is just dying.