Jamie HowarthNov 24, 2022
Chad 

I'll be honest here, as a humble instance admin receiving all kinds of DMs and emails...

The abuse, excessive reports... folks our instance grew from 60 users to 22000 in the span of two weeks. I didn't expect to suddenly commit 6-8 hours of my day to this. Knowing I have other commitments like a day job and other things.

I'm burning the candle on both ends here. Look at this toot timestamp. Please save space for some patience.

Y'all clearly don't remember this image:

#mastoadmin #mstdnca

Nov 24, 2022 at 9:25amWeb
Show threadChad Nov 24, 2022

I do IT for a living. I know you're getting those 500 errors. I know that the instance is slow.

Do I want that for you? Abso-fucking-lutely not. I want mstdn.ca to load for you like the bird app does. Like how your gmail loads. Snippy snappy.

Unless you have explicit experience with Puma and Sidekiq and are willing to get on a call with me, save your abusive DMs and emails.

I'm scouring the internet, consulting with the Mastodon devs and other operators. This is a trial and error process.

Show threadQuokka 🇦🇺Nov 24, 2022
@chad abusive dm's? That sucks. I hope your tooters appreciate your efforts more than that.
Boosted so maybe another couple of admins might see it and be able to pitch in.
Show threadAccretorNov 24, 2022
@Quokka We do. We absolutely do. @chad
Show threadDerick Campbell Nov 24, 2022
@chad thank you for your hard work on this - we DO appreciate it!
Show threadJosef Davies-CoatesNov 24, 2022
@chad I've seen toots by the author of Puma offering to help with this stuff so if I were you I'd reach out to them if you haven't already done so. Good luck! Glad I'm not in your shoes!
Show thread@[email protected]Nov 24, 2022
@jdaviescoates @chad I just found the original toot again, here it is; https://ruby.social/@evanphx/109365852402052564
Evan Phoenix (@[email protected])

To all mastodon admins: Hi! I’m Evan Phoenix, the primary author of puma, the ruby webserver that powers mastodon! Please reply or DM me if you need tuning help! I’ve got no officiation with the mastodon, just want to see you succeed! (Quick Tip: set WEB_CONCURRENCY to core count * 1.5 and then tune MAX_THREADS. High thread values will see diminishing returns!)

Ruby.social
Show threadSuePatNov 24, 2022
@chad Some of us are very grateful for this site and what you are doing Chad. Keep up the good work and thank you.
Show threadValue SubtractedNov 24, 2022
@chad Ugh, just remove those people. You don't need that. Win-win-win.
Show threadEfultonNov 24, 2022
@chad uh oh…CBCYEG is doing a story on Mastodon right now…you’re gonna get an influx of early risers - although she isn’t really cheerleading for y’all, I think “clunky” was a word she used! I’m old, patient with shitty outdated technology and quite happy to intentionally “slow” down and be present when I sign on to enjoy beautiful photos, funny cat pics and the odd political comment. Keep up the good work and ignore the impatient tire kickers.
Show threadHolly 🇨🇦Nov 24, 2022
@MrtniLady @chad Clunky? 😁 I’d rather clunky and surrounded by cool people than super fast and surrounded by whatever Twitter has become.
Show threadEfultonNov 27, 2022
@Yup_Its_Holly @chad OMG LinkedIn….talk about clunky….but the CEO of my company sent me an invite after I skipped out early on the dreadful holiday gathering last night so ya, I’ll deal with Clunky!​​🍸​
Show threadAnneNov 24, 2022
@chad hang in there!
There are far more people who appreciate what you and others are doing than those who send nasty messages. Some people are never happy with anything, sadly.
Show threadPaul Prins 🥞Nov 24, 2022
@chad
Show threadHolly 🇨🇦Nov 24, 2022
@paulprins @chad Hey… how come you have GIF capability and I don’t? 🧐
Show threadPaul Prins 🥞Nov 25, 2022
@Yup_Its_Holly @chad I searched on the giphy site, download the gif, and then manually upload it to a toot. I’m internet old so this feels natural to me.
Show threadHugo Slabbert ⚠️Nov 24, 2022
@chad you've got this, dude. Sorry I don't have Puma or Sidekiq experience to throw your way, but your work is appreciated. It's not much, but I've thrown a small PayPal donation your way for whatever it's worth. Best of luck. #HugOps
Show threadJanis La CouvéeNov 24, 2022
@chad I am so sorry people are giving you grief. Thank you for all your volunteer work to get a Canadian instance up and running.
Show threadSam HulickNov 24, 2022

@chad Hey Chad! Is going serverless an option? Our application is always snappy no matter how many users are online and how much media they're processing.

https://docs.aws.amazon.com/lambda/latest/dg/lambda-ruby.html

Building Lambda functions with Ruby - AWS Lambda

You can run Ruby code in AWS Lambda. Lambda provides runtimes for Ruby that run your code to process events. Your code runs in an environment that includes the AWS SDK for Ruby, with credentials from an AWS Identity and Access Management (IAM) role that you manage.

Show threadChad Nov 24, 2022
@SamHulick I compared costs and going serverless means ~33% higher costs. That and I'm not particularly keen on supporting a union buster.
Show threadhubickNov 24, 2022
@chad @SamHulick I've seen you make this statement a few times. Who is the union buster? I'm confused, because usually self-managed and serverless are different options from the same hosting providers, so your money goes to the same people either way? I know if it were me, I'd be going serverless, 100%. But I'm also a user who'd be happy to cover my own usage/share of the hosting.
Show threadBert JW Regeer 🏳️‍🌈Nov 24, 2022
@hubick @chad @SamHulick Amazon is the union buster. AWS is not the only place to buy hosting.
Show threadSam HulickNov 24, 2022

@bertjwregeer @hubick @chad We bought into AWS long ago, before any of this came to light. Having said that, there are a lot of great folks at AWS that I've gotten to know, and I'm at least happy knowing that my small contribution helps keep them employed and doing what makes them happy.

It's also worth mentioning that not all cloud providers are equal. Some are stronger in some areas than others, it just depends on your technical needs.

Show threadhubickNov 24, 2022
@bertjwregeer @chad @SamHulick I know AWS isn't the only place - please don't be condescending. That's kinda my point, as I feel like there's a variety of serverless providers. RedHat has OpenShift Serverless, and I code on Google AppEngine, where I thought the PaaS stuff was *cheaper* than full containers/VM's.
Show threadBert JW Regeer 🏳️‍🌈Nov 24, 2022

@hubick @chad @SamHulick you asked who was union busting. The grandparent suggested AWS Lambda. You said "usually money goes to the same place"... suggesting you were referring to AWS EC2 and AWS Lambda as that was the topic at hand.

Clearly the English language leaves a lot to be desired in terms of communicating, but that was not meant to be condescending, but you do you. ¯\_(ツ)_/¯

Show threadFansenNov 24, 2022
@chad Thank you … people have been migrating here faster than starlings, and expect Mstdn to be as good as 🐥 WAS in 30 days that took the bird 12 years. All without pay to those trying to do it. I can wait and even if it is not perfect, neither am I.
Show threadKathryn MathiasNov 24, 2022
@chad I'm sorry you're receiving those messages. That's wrong for people to do this 😔
Show threadRyan WildNov 24, 2022
@chad If you're still struggling feel free to give me a yell, we had the same growing pains here (Growing from 100 users to 52K in around 2 weeks) so happy to lend a hand if you need any pointers.
Show threadnicdex 🇨🇦 🇺🇦Nov 24, 2022
@wild1145 @chad Same here. From 100 users to 52k in 2 weeks. I can say it has been stressful. Don't hesitate to ping me too if you need help.
Show threadAnon YmousNov 24, 2022

@chad I am a newbie here and appreciate what you and those helping you have accomplished.

Whiners can either roll up their sleeves and help you or decamp to another village.

I remember when folks had to wait for days to have a response delivered by a mail carrier … perhaps that perspective helps me to be both patient and appreciative.

Great work. And thank you.

Show threadAnn Petersen-KaneNov 24, 2022
@chad You are doing a great job. Ppl who weren't on the bird site back in the day don't know what it's like when something is new and growing. Servers are part tech and part human resource issues and it's not easy having to wear both hats. Kudos to you.
Show threadJennifer BulmanNov 24, 2022
@chad Grateful for you. Keep up the good work. Thank you for being open and welcoming.
Show threadKevin Lloyd Nov 24, 2022
@chad Thank you for all you do, Chad! I certainly hope more donations are coming in with all the demand! They must know the pressures this puts on you (and in general, other admins)!
Show threadAxel FoleyNov 24, 2022
@chad ⚘⚘⚘
Show threadrb2kNov 24, 2022
@chad thanks for the good work!
I had good success switching a production ruby app to a ruby VM compiled with jemalloc (https://github.com/Binaergewitter/serious-bg/pull/390/commits/c877025f2eeeef36c77c371de96a64db1f754222), resulted in ?20%? memory savings, a bit less CPU and was basically a drop in. Not sure if you had looked into that, but if not, it’s a nice optimization without the need for code changes
Switch fly deployment to docker container with jemalloc ruby by rb2k · Pull Request #390 · Binaergewitter/serious-bg

We are running on the free tier of fly.io Running on the free tier of something is VERY much on brand for the podcast and handcrafted CMS with which we fully celebrate mediocrity in hobby projects ...

GitHub
Show threadMark Poppen 🇳🇱🇨🇦Nov 24, 2022
@chad Other than donating, is there anything I can do? I don't know Puma or Sidekiq but I do know we need a new social media platform to replace the reach of Meta and the bird.
Show threadKevin Lloyd Nov 24, 2022
@markpoppen @chad I know Chad's email is being bombarded, but I've offered to help as well. I'd like more info on being part of the Board, or possibly a Moderator. I'm already donating. :)
Show threadMark Poppen 🇳🇱🇨🇦Nov 24, 2022
@kevinlovestech I want to see how much I'll actually use Mastodon. I've noticed I haven't been on social media a lot lately and I'm trying to figure out if that's because I'm just busy or if it's because I've totally lost interest. I think it's the former.
Show threadKevin Lloyd Nov 25, 2022

@markpoppen I'm certianly losing interest in Twitter. I much prefer Mastodon. It's taking time to build things up here, but that's expected, as are the growing pains.

No worries about getting busy. It's happens. :)

Show threadJames NenoNov 24, 2022
@chad Thank you for all your hard work. Everyone complaining needs to chill out and be respectful.
Show threadBeavarah  Nov 24, 2022
@chad oh no! I’m so sorry you’re being abused. I migrated from twitter and I’ve been very happy. I’m fully aware all servers are being overloaded right now and I’m happy to hang tight through the slow periods. Thank you so much for this great Canadian instance! Your work is greatly appreciated 🙏🏻 
Show threadSecular_Girl (Marg)Nov 25, 2022
@chad I am so sorry. I appreciate everything you do. I think many here do! I, myself, am liking this much better than twitter. This is sane, welcoming, friendly. ❤️
Show threadNoelleNov 25, 2022

@chad I'm sorry you are getting ugliness. The huge influx of new members is a strain & this is not a for profit organization

I want to thank you for the work you do. I know it must be enormously increased

Ppl should be thanking everyone here whose lives we tens of 1000s if not millions of new #TwitterRefugees have made more diffucult

Show threadseb - buttons are toys!Nov 25, 2022

@chad Hey Man, I’m one of the admins for sfba.social and the admin for ioc.exchange .

We also scaled up lots during the last two weeks. We have a good handle on it. Happy to meet with you and help.

Show threadLeslieTFJNov 25, 2022
@chad I read you do IT, as in spicy accountant. Lol.
Show threadhannahshouse2 🇺🇸 CaliforniaNov 25, 2022
@chad
I am not on your instance but thank you for doing what you're doing. I hope for patience for you, your instance and for you from your tooters.
Show threadChuck Taggart, Private EyeNov 25, 2022
@chad i’m not a member of your instance but your work is greatly appreciated. And there is zero excuse for people who are sending abusive DMs and emails. None.
Show threadanniemo71Nov 26, 2022

@chad
Shocked to hear about this abuse. Thank you so much for everything you do 👍🏴󠁧󠁢󠁷󠁬󠁳󠁿 Nothing is really urgent for a lot of people, we are just used to instant everything.

I applaud you for what you are doing 👏👏👏

Show threadChad Nov 24, 2022

Changes I implement while you're sleeping (and I should be too) don't yield results until the middle of the day.
Each day is showing incremental improvement. 500 errors were reduced to less than 10% of all requests yesterday. This is significant.
I'm just asking all fellow Mastodonauts to please give me and the mods some space to develop infrastructure and policy that this sudden influx demands. We're working as hard as we can, as complete volunteers.

Again, volunteers.

Show threadChad Nov 24, 2022
As some have done, if you feel like you want to cancel your PayPal subscription during this time of exploding growth, so be it.
I got the email. I don't need your threats in my DMs or in my email.
It's 0230. If I didn't care I'd be in bed right now.
Show threadAccretorNov 24, 2022
@chad I can't do anything about them, but this spurred me to get off my bum and add my PayPal subscription. Thank you.
Show threadBrad Vrolijk 💬Nov 24, 2022
@chad Thank you for your effort! I hope most of us are reasonable and understand that a user increase of ~ 36,500% in two weeks can cause some problems. Thanks again 🇨🇦
Show threadJustin DerrickNov 24, 2022
@chad Forget the haters. I remember the early internet, circa 1993. If stuff was down, you just logged off and came back tomorrow. My speciality in IT is elsewhere, so I doubt I can help in the short term, but I’m actively trying to see if there’s a way I can use my niche skills to help.
Show threadChad Nov 24, 2022
@JustinDerrick I appreciate the words, Justin.
Show threadJeroen van BergenNov 24, 2022
@JustinDerrick @chad I really appreciate the amount of time and energy you and the other admins are putting into this. It just occurred to me this whole Mastodon thing reminds me of my FidoNey days. Back then we had regular in person meet-ups of SysOps and points (people with a ‘mail address at a FidoNet BBS). We knew each other and what it took to keep the infrastructure running. #hugyouradmin
Show threadAccretorNov 24, 2022
@jeroenvanbergen OMG the mention of #FidoNet and #BBS. I hadn't really thought about the fediverse in those terms, but there is a similarity for sure.
Show threadJason St-Cyr Nov 24, 2022
@chad I am sorry that you are getting such hateful messages for the great work you are doing! That's unacceptable.
Show threadMerzedotesNov 24, 2022

@chad
I'm here to stay, Chad.

You are forging a great community and deserve endless support from all of us.

I'm happy to make a donation to make up in some small way for those cancelling their subscriptions.

Show threadJeffGeeBrownNov 24, 2022
@chad I hope more users will donate more money, so you can buy more hardware & software, or hire more help, or whatever else you need and, of course, get more sleep. Distributed admin is the only viable future for social media. We appreciate you. We care.
Show threadIan Collins  Nov 24, 2022
@chad Thank you for everything you're doing to support this instance! I really appreciate all the hard work you're doing behind the scenes in a very challenging time.
Show threadMaarten den BraberNov 24, 2022

@chad that's recognizable. Take a break - people will understand (and if they don't they probably don't understand where they are).

Also: ask for help: grow your devops and moderation team. You don't have to do it all by your self.

Hang in there!