ah poop... I forgot to restart sidekiq yesterday when I performed the upgrade so everyone's timeline is like... a day behind.
running it now and it should catch up..... eventually
psa, just updated our mastodon server to v4.1.2. looks like there isn't a ton different, just some security and bugfixes.
ok so hackny.social went down again last night and I'm starting to do a bit more digging into what's going on. it looks like postgres got a SIGKILL around 1a UTC which also coincided with a spike in CPU usage...
my status monitor service worked! yay babashka!!
I still don't know the cause for postgres occasionally shutting down though. maybe I just need to modify the systemd service to always restart upon exit and call it a day.
I took hackny.social's recent postgres failures as an opportunity to play with babashka (a mini clojure implementation meant for scripting) and wrote an uptime monitor that should ping me if the server goes down:
Postgres seems to have died again this morning. I just hit the reset button and everything appears back in working order. Sorry for the outage ya'll.
ok thanks to that EFF guide I registered a designated agent for us which will hopefully never get used.
I want this instance to be around for the long haul so I'm hoping making these (both legal and technical) infrastructural changes will set us up for success.
ok the migration over to scaleway's object storage is nearly complete. you might notice some assets not working tonight but everything should be resolved by tomorrow morning. pls let me know if you notice anything awry!
CODE IS RAPIDLY SCROLLING ACROSS MY SCREEN AS WE SPEAK. IT'S WORKING??
hmm ok we're hovering at 70% disk space usage even after purging media... I guess it's probably wise to preemptively switch to a cloud storage provider for our instance's media hosting. thankfully somebody has already paved the way for doing something similar[^0]. mayyybbeee going to have some downtime later today as I get this setup.
also going to start chronicling my hosting activities under #hnyadmin for posterity.
[^0]: https://stanislas.blog/2019/11/object-storage-migrate-wasabi-to-scaleway-rclone/
steve gattuso