Weirder AdminNov 6, 2022
Heads-up: we are continuing to have increasing issues with overloaded Sidekiq queues - expect major delays on emails, notifications, and timelines until we can resolve these issues.
Show threadWeirder AdminNov 6, 2022

Packbats here - update on the server situation.

tl;dr:
- Around the same time as @LouInABox's scheduled maintenance, the Twitter migration started overloading the server.
- Between Lou and @noracodes (thank you, Nora!), we identified the problem and how to address it.
- The instance is currently in emergency clear-the-twelve-hour-backlog mode, so some posts or notifications might get lost overnight.
- Once it's had time to clear, it'll be put in stable mode and we shouldn't have that problem.

Show threadWeirder AdminNov 7, 2022
Status update on the instance:
- As you probably noticed, the settings change from yesterday did not clear the Sidekiq queues.
- The three of us have continued to do research and system configuration changes, most notably:
= Increasing shared_buffers RAM for the postgres database to handle repeated requests better; and
= Splitting Sidekiq into multiple processes, each assigned to different queues.
- Right now, it looks like the queues are actively emptying.
- Updates as events warrant.
Show threadWeirder AdminNov 7, 2022
- tl;dr on what queues means: the thing what means that your notifications timeline is two hours behind and your home timeline is nine hours behind.
- We have about 168k events left to process, down from 194k two hours ago, and have been getting through 490/min on average over the past fifteen minutes.
- We plan to decrease shared_buffers from 2GB to 1GB once things stabilize, and then take care of other smaller issues we noticed.
Show threadrhosyn
@WeirderAdmin zoiks, scoob!
Nov 7, 2022 at 8:19pmWeb