AndrewAug 19, 2018
Eugen Rochko

Extremely sorry about the downtime. Scaling up to multiple machines has revealed a peculiar issue with a scheduled process that's supposed to run every night... once. It was run on every machine instead, putting the database in a deadlock state.

We're back now and I'm going to add a fix before next night 😩

Aug 19, 2018 at 2:43amWeb
Show threadjeff benzosAug 19, 2018
@Gargron you're doing a great job eugen, keep going!
Show threadthe orb (obsolete)Aug 19, 2018
@Gargron hero of the revolution eugen hero of the revolution eugen hero of the revolution eugen
Show threadSoup-Like Gender Homogenate ✅Aug 19, 2018
@Gargron Wild. Thanks for the post-mortem!
Show threadDefund cops, pay teachersAug 19, 2018
@Gargron thanks for all the hard work you put into this!
Show threadhealynAug 19, 2018
@Gargron Thank you for the update. I will die for you, Eugen
Show threadsbrAug 19, 2018
@Gargron have you tried fucking the servers
Show threadChris PoparomaAug 19, 2018
@Gargron Eugen you're doing amazing sweetie
Show threadInsert punchline hereAug 19, 2018
@Gargron guy with the massive influx of new users I'm surprised this site hasn't melted down into radioactive sludge - it's seriously impressive tbqh
Show threadEugen RochkoAug 19, 2018
@TZaple We had the opportunity to handle big traffic before - periodically every few months beginning in April 2017. This wave is much bigger, but the experience is there, and the software is mature enough. This is one weird issue though!
Show threadkenzoAug 19, 2018
@Gargron @TZaple This time there's more servers to take on users
Show threadLethal JortsAug 19, 2018
@Gargron it's fine, i was done posting today anyway
Show threadNate BarhamAug 19, 2018
@Gargron Nice to hear direct from you what was going on. Thanks for the update! 🍻
Show threadTheloniousPunkAug 19, 2018
@Gargron Thank you. We kind of all just showed up at your doorstep.... it's all good ❤️ Don't forget to get some rest.
Show threadJBAug 19, 2018
@Gargron Out of curiosity, and if you are open to answering, what service do you use for hosting your instance?
Show threadEugen RochkoAug 19, 2018
@Jamacanbacn Hetzner
Show threadnosebleetAug 19, 2018
@Gargron we're very thankful for the time you have put into this for us. thank you 🖤
Show threadThomas CannonAug 19, 2018
@Gargron it’s ok! Good luck with the fix!
Show threadmatusAug 19, 2018
@Gargron don't forget to document the process whenever possible
Show thread@[email protected]Aug 19, 2018
@Gargron #HugOps
Show threadDan CookAug 19, 2018
@Gargron out of sheer curiousity, what does this run on?
Show threadkenzoAug 19, 2018
@dcook @Gargron It's written in ruby and runs on Linux
Show threadDan CookAug 19, 2018
@kenzo @Gargron on AWS or something else is more what I'm wondering. Looking for folks to chat infrastructure with but all I really know is AWS. Always trying to figure out how others deal with everything
Show threadkenzoAug 19, 2018

@dcook @Gargron Since he's asking for Patreon $$$ I'd imagine a cloud instance.

Cloud is the more sensible place to put public instances since the traffic abuse.

Show threadEugen RochkoAug 19, 2018
@kenzo @dcook Hey, no, I use Hetzner bare metal, set them all up manually, it's more cost-effective that way, although more work for me.
Show threadkenzoAug 19, 2018
@Gargron @dcook It's still hosted though, right? As opposed to running a farm in-shop.
Show threadEugen RochkoAug 19, 2018
@kenzo @dcook Ah, that'd be colocation and no, I do not do that - no need for now, anyway.
Show threadkenzoAug 19, 2018
@Gargron @dcook Mastodon's largest instance...
Show threadkenzoAug 19, 2018
@Gargron @dcook The magic of ddg search -- I see, root hosted
Show threadLance HaigAug 19, 2018
@Gargron @kenzo @dcook do you use something like Ansible to configure the servers?
Show threadEugen RochkoAug 19, 2018
@Stokvis @kenzo @dcook i wish i was, but no. Definitely need to learn it
Show threadLance HaigAug 19, 2018
@Gargron if you have a setup document I could try to do that for you
Show threadkenzoAug 19, 2018
@dcook @Gargron Someone on IRC just turned me on to trying out NixOS - the packages are a bit behind but you can define a complete OS plus all the configuration/customization in a simple file. Better than Docker in many ways
Show threadSkagit Slough WitchAug 19, 2018
@Gargron proud of you Eugen, you’re doing great work
Show threadcanvoxAug 19, 2018
@Gargron I thought I'd be able to go a day without shouting ORDER YOUR LOCKS as it's a weekend but here i am
Show threadskrrt_vonnegutAug 19, 2018
@Gargron you are awesome
Show threadAM0💗🔪Aug 19, 2018
@Gargron
Everyone loves it because of the common sense policy and it's not controlled by corporate politics. I'm sure no one is going to mind the growing pains so long as that stays the same.
Show threadHELLGORITHMAug 19, 2018
@Gargron remember twitter’s “fail whale”? You’re doing great, keep it up (literally and figuratively)
Show threadRegine HorteurAug 19, 2018
@Gargron Thanks for keeping us informed. Get some rest though (you and your team).
Show threadgarfieldAug 19, 2018
@Gargron get some sleep, king. but tomorrow... we better get that garflud
#GerfloidEmoji #DoIt
Show thread🚩Millicent Ⓥ neo-Dukeist 🏴Aug 19, 2018
@Gargron Bless you for all your hard work helping us refugees from birbsite gain safe passage.
Show threadmoved: @[email protected]Aug 19, 2018
@Gargron thank you for all you do
Show thread➡️ new acct link in bioAug 19, 2018
@Gargron thanks for your hard work ❤️
Show threadKathleen De VereAug 19, 2018
@Gargron this seems like a lot of work. Thanks for doing it.
Show threadLeonardoAug 19, 2018
@Gargron you are doing totally okay. appreciate all the hard work!
Show threadMeat BoyAug 19, 2018
@Gargron thank you!
Show threadElfLordAug 19, 2018
@Gargron I don't know what any of this means but I think you are really acting like Michelangelo right now. In a good way.
Show threadKfir BregerAug 19, 2018
@Gargron thank you. You’re awesome. Software is hard.