Wayne RohretNov 19, 2022
Evan Phoenix

To all mastodon admins:

Hi! I’m Evan Phoenix, the primary author of puma, the ruby webserver that powers mastodon!

Please reply or DM me if you need tuning help! I’ve got no officiation with the mastodon, just want to see you succeed!

(Quick Tip: set WEB_CONCURRENCY to core count * 1.5 and then tune MAX_THREADS. High thread values will see diminishing returns!)

Nov 18, 2022 at 4:54pmWeb
Show threadjuan Nov 18, 2022
@evanphx i don't remember from the top of my head the tuning recommended in the mastodon doc, but that's what I followed, is what's there still valid and good advice?
Show threadEvan PhoenixNov 18, 2022
@jolvera I’ll have to check!
Show threadEvan PhoenixNov 18, 2022
@jolvera So the official tips don't give concrete suggestions about how to tune puma. So it's still correct, just not specific enough!
Show threadjuan Nov 18, 2022
@evanphx is it worth doing a pull request to their doc to add the tunning you'd suggest?
Show threadEvan PhoenixNov 18, 2022

@jolvera Yeah, probably! I want to figure out a way to gather some usage data from installations to provide even better info as well.

Also considering releasing a special puma-mastodon gem that will tweak these settings automatically.

Show threadBenjamin Sonntag-KingNov 18, 2022

@evanphx @jolvera Hi Evan, THANK YOU for Puma  
(here an admin of piaille.fr, a ~6000 users instance at the moment)

be careful that most instances have all services in one machine (postgresql, node, redis, elasticsearch, AND puma (AND sidekiq ;) )
so your automatic computation may not be right.
1.5* core count on a dedicated puma-only machine, sure ;)

Show threadEvan PhoenixNov 18, 2022
@benjamin @jolvera Totes! As @getajobmike pointed out, gotta leave cpu headroom for others! 1 or 0.8 times core is safest.
Show threadchendoNov 21, 2022
@evanphx @jolvera I think the tuning is heavily dependent on both on where people are running puma. On smaller instances, it's likely running on alongside other workloads, and then those values would be different if one was tuning for throughput as opposed to latency. My mantra is to measure before tuning otherwise I'd be just guessing!
Show threadMike Perham Nov 18, 2022
@evanphx Careful with that 1.5 value. Sidekiq needs to run too.
Show threadDJ SundogNov 18, 2022
@getajobmike @evanphx this is the most delightful resource contention I've seen all week. if we can get core contributors from postgres and nginx into the thread we'd be full stack! ​
Show threadEvan PhoenixNov 18, 2022
@getajobmike For sure! Setting WEB_CONCURRENCY to cores * 1.5 can result in puma soaking up 100% of the cpu. So if you're running sidekiq on the same cores, use 1 * cores or even 0.8 * cores is prudent.
Show threadMike P.Nov 18, 2022
@evanphx @getajobmike In a fresh install of Mastodon following the guide, Puma doesn't read .env.production on boot, so if you want to change the number of workers you either need to edit config/puma.rb or the mastodon-web.service to set e.g. Environment="WEB_CONCURRENCY=3".
Show threadMike P.Nov 18, 2022

@evanphx @getajobmike Additionally, since y'all are giving formulas with floating point math, make sure to round up or down to an Integer value to keep Puma happy.

If you're running a small VPS with e.g. one core for a small number of users, consider setting WEB_CONCURRENCY=0 to take Puma out of clustered mode.

Show threadEvan PhoenixNov 18, 2022
@mwp @getajobmike Yup! Not being in cluster mode will save a small amount of memory as well.
Show threadRalf StockmannNov 18, 2022
@evanphx @getajobmike see - this is where it gets tricky. We need some sort of „calculator“ that takes cores, RAM and dedicated servers into account and gives advise _where_ to set the values, how to restart etc. This would be super-helpful for our bunch of not-so-expert admins. And we need it fast, for numbers are rising fast. The official documentation is way to fuzzy regarding tuning. 🙏
Show threadEvan PhoenixNov 18, 2022
@rstockm @getajobmike Yeah, totally agree. I'm going to setup a google form to start gathering data now, that we could use for a calculator.
Show threadRalf StockmannNov 18, 2022
@evanphx @getajobmike brilliant, and just for the record: it‘s beyond awesome that you get in touch with us. And the actual documentation was never meant to handle a situation like this: the basic setup was fine for several years. But now we are not facing a wave, but a brick wall of new users - and need to adapt. Thanks again for your effort! ❤️
Show threadRalf StockmannNov 20, 2022

@evanphx @getajobmike maybe worth a read and how you do fit in:

https://nora.codes/post/scaling-mastodon-in-the-face-of-an-exodus/

Scaling Mastodon in the Face of an Exodus | Nora Codes

Show threadSantiago Nov 19, 2022
@evanphx @getajobmike where is this setting? i can't remember
Show threadJanNov 18, 2022

@getajobmike @evanphx

Hey Evan, Hey Mike nice to meet you. Puma and Sidekiq works like a charm 👍

Show threadEvan PhoenixNov 18, 2022
@jan @getajobmike 364M happy customers served.
Show threadMike Perham Nov 18, 2022
@jan @evanphx Not bad. I have a customer doing 22 billion per day, 250k per sec. 
Show threadMadMike77Nov 18, 2022
@evanphx @ordnung allenfalls 🤷
Show threadhttps://gigatexal.blog -he/himNov 18, 2022
@evanphx would be so cool to see you hanging out in the mastodon irc or discord either way, thanks for helping all the new found new admins!
Show threadCaro Flores HineNov 18, 2022
@donelias ojo al toot de arriba
Show threadJacob ¶. FordNov 18, 2022
@evanphx You are so cool & nice!
Show threadwitchlectical runeterialistNov 18, 2022
@evanphx thank you i definitely won't abuse this offer 
Show threadThomas Fricke (he/him)Nov 18, 2022

@evanphx

Thanks for puma!

Do you know any integration of Jam

https://medevel.com/jam/

into #mastodon

Some people miss something like the spaces of the dead blue bird

@SheDrivesMobility

Show threadEvan PhoenixNov 18, 2022
@thomasfricke @SheDrivesMobility Sorry, don't know much of anything about mastodon the app itself, just happen to be versed in the rails architecture world!
Show threadThomas Fricke (he/him)Nov 18, 2022

@evanphx

BTW Ruby bundler now supported by the government

https://sovereigntechfund.de/bundler_en.html

Enjoy!

Sovereign Tech Fund

Strengthening Digital Infrastructures and Open Source Ecosystems in the Public Interest

Show threadEvan PhoenixNov 18, 2022
@thomasfricke Ah yes! I got wind this was going to happen, wonderful!
Show threadRenaud ChaputNov 18, 2022
@evanphx Would you consider joining the Mastodon Discord (through their Patreon)? There are a number of instance admins there trying to work on scaling their instances and I feel you could help a lot!
Show threadEvan PhoenixNov 18, 2022
@renchap Sure, I'm not currently a patreon so I don't have access, but definitely happy to jump on and help!
Show threadUlfNov 18, 2022
@evanphx fyi @ordnung @leah @rixx
Show thread@ordnungNov 18, 2022
@ulf Bitte nicht Leah und rixx direkt pingen. Wir lesen diesen Account hier und würden wirklich gerne eine Chance haben, unsere privaten Accounts etwas in Ruhe zu nutzen.
Show threadKevin   Nov 18, 2022
@evanphx this is very cool!!!
Show threadKevin   Nov 18, 2022
@evanphx @david
Show threadEddie KayNov 18, 2022
@WEATHERISHAPPENING #fediadmin #mastoadmin tip
Show threadOleg "Ole" R. 🇱🇹🇨🇦🇺🇦Nov 18, 2022
@evanphx @chad
Show threadLeahNov 18, 2022
@evanphx had it exactly there and the processes consumed 2Gb RAM each so we quickly run out of memory on the 32Gb RAM system. We had 10 threads per process.
Show threadEvan PhoenixNov 18, 2022
@leah Yeah, the core count vs memory definitely impacts the math. Did you find a happy balance? What was it?
Show threadLeahNov 19, 2022
@evanphx currently 10 processes with 10 threads are looking good. But we moved the DB and all the sidekiq workers to other servers already. So at the moment only, redis, streaming and puma are on the same system. otherwise we would have run out of memory a long time ago. I think it depends very much on how your setup look how many resources you can allocate to puma. So maybe there isn't a good and simpel ratio.
Show threadLeahNov 19, 2022
@evanphx maybe it is better to look if a puma process consumes 2gb RAM for everyone and if so depend the scaling more on the available RAM than on cores.
Show threadThe Bear LeftistNov 18, 2022
@evanphx Ruby.  
Show threadJohan EmpaNov 18, 2022
@evanphx
Could you perhaps point to any instructions on how to place some puma processes on a separate machine?
Show threadEvan PhoenixNov 18, 2022

@JohanEmpa I can't find one off hand atm. The quick answer is that you'll configure config/database.yml on the separate machine to point to the correct database, then when with new puma's started, you'll edit the nginx config to include the new machine's ip as a valid backend.

Look for "rails cluster setup" guides, they should apply fine.

Show threadJohan EmpaNov 18, 2022
@KuJoe
Here's a starting point for setting up separate machines
Show threadJohan EmpaNov 18, 2022
@evanphx Thank you for these tips 🙂
Show threadRuudNov 18, 2022
@evanphx Hi, I'll certainly look into that.. our instance grew from 140 users 15 days ago, to 65000 users today. Needs some tweaking here and there...
Show threadatomicwritesNov 19, 2022
@ruud @evanphx Oof, my condolences. Awesome that your getting so Mich traffic, but it must be like a hug of death that never stops.
Show threadflauschpolizeiNov 18, 2022
@evanphx Hey @nathan have you seen this?
Show threadBen S.Nov 18, 2022

@evanphx hey @SDF this seems like useful info for the folks tuning the servers.

Edit: Lots of other potentially helpful comments under the post on the original server as well.

Show threadDamonHDNov 18, 2022
@evanphx one for @Floppy also!
Show threadpieceofthepie Nov 18, 2022
@evanphx what kind of calculations should we be doing for a vCPU situation?
Show threadEvan PhoenixNov 18, 2022

@pieceofthepie If puma is the only software running on the system, 1.5 * vCPU for WEB_CONCURRENCY is an easy place to start. If you're running sidekiq / nginx / postgres on the machine too, go for 0.8 * vCPU.

Set MAX_THREADS to 16 and then see how it feels. You'll see a top end for the thread count, and then you'll know you need more WEB_CONCURRENCY or more raw cores.

Show threadYazdog8Nov 18, 2022
@evanphx I don't get how to change servers. It's confusing bb
Show threadTed FaberNov 18, 2022
@evanphx
I don't know that, but I should've. It delights me how many USCLUG veterans built this city.
Show threadMoved to @[email protected]Nov 18, 2022
@evanphx Do you have any tips for running on a low-core-count server? I'm not having any performance issues, but I'd prefer to avoid wasting resources on my dual-core VPS.
Show threadEvan PhoenixNov 18, 2022
@hazel Since you're probably running all the services on one machine, start small. Use puma with WEB_CONCURRENCY = 0 and threads = 16. You'll have to watch the load and adjust it from there. As usage grows, you'll want to start with WEB_CONCURRENCY = 2, and then tweak the thread counts.
Show threadMoved to @[email protected]Nov 19, 2022
@evanphx I'll give that a try, thanks!