Mastodawn

Did quite some maintenance on burningboard.net Mastodon instance today ..

Updated servers and jails to FreeBSD 15.0-RELEASE-p5

1.
proxy_cache_path (S3 media cache) was already on a ZFS dataset with autosnapshot disabled .. But proxy_temp_path and client_body_temp_path (both nginx temp folders) weren't. Leading to extremely bloated daily ZFS snapshots for the Nginx jail. Changed those and cleaned old snapshots -> 31GB reclaimed space

2.
Cleaned up Database Snapshots and did a pg_repack on prod to reclaim deleted database pages -> 3GB reclaimed space

3.
Fixed issue with backend path over gif tunnel to Elasticsearch (6in6). Turned out to be a MSS issue (clamp-mss to 1140 on the tunnel iface solved it!)

4.
Fixed IPv6 routing issue for Prometheus Monitoring after reboot (rc.d static_routes != ipv6_static_routes)

All in all, our instance is in very good shape again :-)

#mastodon #mastoadmin #socialmedia #fediverse #freebsd #unix @tux

Show thread

Larvitz

1d ago

@tux Running a pretty reasonable stack now ...

FreeBSD 15.0-RELEASE-p5
Mastodon 4.5.8
PostgreSQL 18.3
Valkey 9.0.1
Opensearch (fork of Elastic) 3.2.0

28,4 GB PostgreSQL Database size
162 GB Media Files
4,88 GB Search Index size

#mastodon #mastoadmin

Show thread

bbₜᵤₓᵢ

@Larvitz Nice work! 🔧 The `proxy_temp_path` / `client_body_temp_path` on the wrong ZFS dataset is a classic pitfall – those temp files are short-lived, but when snapshots keep dragging along the deleted blocks, it adds up brutally. 31GB reclaimed just by separating datasets speaks volumes.

Also the MSS clamping to 1140 on the GIF tunnel – was it fragmentation that got silently dropped by Elasticsearch/Opensearch? Or did TCP retransmits just make the whole thing unbearably slow? Would be curious how that manifested.

And `ipv6_static_routes` vs. `static_routes` in `rc.conf` – classic FreeBSD gotcha that makes you curse after every reboot 😅

Solid maintenance session! 👏