Mike ShewardI do feel for the engineers working on whatever is going on over there btw. I worked on a "Self-DDoS" incident about a decade ago. It sucked. I wrote about it in my book "Hands-on Digital Forensics and Incident Response".
Lori@SecureOwl Shows how easy a basic mistake can happen with considerable ramifications. Feel for that guy. I once deleted half of our CEO’s emails. I remained employed kindly enough and was able to fix it, phew.
Aaron Longchamps@SecureOwl as an infrastructure guy myself: oof.
ₛᵤₙdᵣᵤᵢd
Ian Douglas Scott@SecureOwl Really, it would be quite impressive if they managed to *not* cause an issue like this. Or something else. This sort of rate limit isn't a small change. Doing that without breaking things would require a lot of planning and testing, and take a while before it could actually be rolled out.
While I imagine they were basically only given the time to implement the obvious solution, deploy to production, and hope for the best.
Nick Craver@SecureOwl @SwiftOnSecurity I'm happy to report that while running a top 40 website we never DDoSed ourselves, except that one time.
Half Cocked Law@SecureOwl Well good news, Elon only pays people who write code, not documentation.
Tyson, Chicken Rancher 🐓@SecureOwl Brings back faint memories of writing about Nagle delays, zero windows, and other arcane TCP metrics at ExtraHop. On configuring the ADC: “With great power comes great responsibility.”
@tsupasat funnily enough ExtraHop was one of our main troubleshooting tools during this incident ;-)
Also funnily enough I posted this today because a company got bought out by someone who has no clue how to run it ;-) ;-)
Grand Camel@SecureOwl fresh out of college, I nearly caused a "revenue impacting event" with a security group auditing and remediation script at my giant tech company. Script ran for months without problem until one day it removed all administrator accounts from domain admins due to a db glitch. I remediated by hacking into a domain controller.
David E. Smith@SecureOwl it’s always the F5, it’s right up there with DNS
Michael Weiss@SecureOwl every time someone tells me they're being DoSed, I tell them "99% of the time that you have a DoS, it's something you did." It's held up so far.