God, I "love" Mastodon.

The server's cache filled up the entire HDD, and then from there put PostgreSQL in recovery mode

Honestly, Mastodon's caching sucks so hard, and it has no good way to do sanity checks like to automatically delete files, or make sure the HDD isn't filled up past a point.

I changed the crontabs to be more aggressive, but it appears the 'recommended' solution is to use S3 storage or something like that.

I'm legitimately debating what to do long term with the server ...

@ncommander S3 is fun too because you go into your instance settings and set a media retention period and then it just

ignores it

every few months I go in and run tootctl to delete all the media older than my retention period and my next month's OVH bill is $20 or $30 lower

@ncommander but… it can automatically delete files, and IIRC it's even done by-default since a while ago.
@ncommander but yes, it would be nicer if it could have a capacity setting, instead of days.
@ar @ncommander There are a few more caches worth considering, here is a nice overview: https://ricard.dev/improving-mastodons-disk-usage/
Improving Mastodon's disk usage - Ricard Torres dev

Mastodon's built-in CLI gives you the availability to clean attachments and previews from remote accounts, purging the disk cache. This is fantastic and

Ricard Torres dev
@ar this is setup, but it filled up regardless. Once I got it back up, I changed the caches to be now two days.

@ncommander What's the size of your local storage? Before I added a dozen relay nodes, I could manage just fine with 80 GB, cleaning it up every few days.

Now I do have object storage added of a terabyte, that I still have to clean up at least once every few months. But consolidated (pruned to only keep the last 28 days of cache), it's like 200 GB right now.

@collectifission The server is 160 GiB of storage, about half of which is used for local media, so 80 GB for caches and such.

Object storage may be what I end up doing long term, but I'm legitimately just frustrated at it.

@ncommander I have great respect for anyone setting up and sharing with the collective debugging of these issues. Thank you for your service.