Okay so it seems something broke when I updated Mastodon to v4.5.0, but there may be something else at fault - the server can't communicate outside. I can't even pull the latest code or ping google.com.
Plus, mastodon-streaming has been quietely failing this whole time.
Fucking hell.
Starting the Mastodon update process, all Mast Insaans hold tight
I'll use this thread for any updates.
Ran a `tootctl accounts cull` for the first time in a year, out of 135k remote accounts only 1065 accounts had been deleted. Not bad.
Ran into an issue - the full-text service on this Mastodon website crashed a little under a day after I deployed it, so full-text search simply stopped working.
ElasticSearch takes a lot of memory, close to 1.6GB, which is fine, I can spare that much. I logged in today to see if it had increased, and saw my RAM at half-capacity, which immediately caused alarm. Dunno why it crashed, too, the logs say nothing.
New #MastAdminExperience post:-
Lots of RedownloadMediaWorker jobs failing, as seen in this sidekiq screenshot for our site.
Not sure what to do, this seems to have started since the 27th.
I've FINALLY found why sidekiq queues are getting clogged up.
It's a known issue - https://github.com/mastodon/mastodon/issues/26115
Image of Sidekiq retries attached alongside.
I'll apply the code fix locally and redeploy it, that should fix the issue until they release a patch. Sorry Mast Insaans.
There was a severe drop in the sidekiq queue throughput for the past week, because of which there was a delay in fetching messages. I have no clue why this suddenly happened or why it suddenly got resolved but it has. Weird.
I may have to upgrade the Mast Insaan server next month.
#MastAdminExperience update
Seems the media storage for mastinsaan.in has already reached close to 90GB. I have media cleanup jobs set to run monthly but I suspect they're failing because the server's kinda small and weak - the last time there was any significant reduction was in March, the cleanup jobs probably haven't run properly for 3 months. I'll have to do it manually.
I am stupid, a #MastAdminExperience thread:-
For a long while I hadn't been able to get proper redirects from www.mastinsaan.in to mastinsaan.in working. Despite having neatly cordoned off both into their own server blocks I kept on having a conflict notice by nginx, and I only got certbot to validate the former URL once, then never again because it just got a 404.
...until today.
Upgraded https://mastinsaan.in to Mastodon v4.1.0! The process was comparatively painless but took a while tbh, close to an hour and a half.
Luv Mehta
