I really really need some help! ⚠️

For the past two days I have been trying to catch up with the mstdn.social timeline but it's only getting worse.

The default queue is pilling up more and more and it seems I cannot start any more workers. Also the I don't get why the threads are not all working per proccess..

Push & pull are on external servers since when I try to run default queue on an external media upload to Wasabi and email sending does not work.

Please, anymone?

@stux try running less processes not more and have them fully loaded
@mdbraber @stux my starting point would be not more than 1 worker per core * CPU; some threads per worker to keep Ruby busy when it has dispatched tasks to other services, but must say I haven’t had to optimise for such situation. Also, perhaps you can play with priorities? More important work sooner but let the daily update mails pile up?

@mdbraber @stux not sure if you were able to get stuff up and running again (not on an instance you control), but double checked with a book Sidekiq in Practice I recently read: thing with threading in ruby is that these do not really work in parallel. There is a GVL; if it is performing some IO task, however, it can release the lock and allow another thread to work. Hence it is better to have more than use more than one thread. But too many will probably block DB connections / IO pools, and push memory consumption over limits.

See also this post by the author of the aforementioned book, @nateberkopec https://www.speedshop.co/2020/05/11/the-ruby-gvl-and-scaling.html

The Practical Effects of the GVL on Scaling in Ruby

MRI Ruby's Global VM Lock: frequently mislabeled, misunderstood and maligned. Does the GVL mean that Ruby has no concurrency story or CaN'T sCaLe? To underst...