Mastodawn

Simon Benbow Jul 2, 2023

For the last two days, Elon Musk has been publicly freaking out about "EXTREME levels of data scraping," so added "temporary emergency measures" like blocking logged-out views and adding tight rate limits on viewing tweets. But, apparently noticed first here by @sysop408, a Javascript bug in the Twitter web app is self-DDOSing their servers, sending an endless loop of requests — which seems related to their scraping panic. https://waxy.org/2023/07/twitter-bug-causes-self-ddos-possibly-causing-elon-musks-emergency-blocks-and-rate-limits-its-amateur-hour/

Twitter bug causes self-DDOS tied to Elon Musk's emergency blocks and rate limits: "It's amateur hour" - Waxy.org

An "amateur hour" Javascript bug is self-DDOSing Twitter, sending infinite requests from users related to — or possibly even causing — Elon Musk's "temporary emergency measures" to stop web scraping.

Waxy.org

Ernie Smith Jul 1, 2023

@andybaio @sysop408 My lord. We are dealing with universe brain.

Sheldon Jul 1, 2023

@andybaio it's probably not the cause of their scraping panic and most of these requests are being blocked.

What I don't know is if MY requests were getting blocked after I launched a few hundred unwittingly or if they just completely shut down a service because it was getting bombarded.

Either way, it's not good and even if this glitch had nothing to do with today's problems, it's still stunning that something this awful got to production.

Andy Baio Jul 1, 2023

@sysop408 Thanks for the reply, I agree it's unlikely but at this point, nothing would surprise me over there. I adjusted the headline and ending to make that clearer.

Xabier Ostale Jul 1, 2023

@andybaio @sysop408 What's data scraping anyway?

emmatonkin Jul 1, 2023

@xabitron1 @andybaio @sysop408
Means: 'remote machines that are not authenticated as users are accessing large numbers of resources in a way that suggests they are collecting and archiving resources for their own aims'. You do sometimes get abusive data scraping, for example, there are galaxy-brain machine learners who aggressively scrape resources without a thought to the server.

Competent sysadmins have ways of squashing that kind of thing. Also you can buy services to deal with it.

Willow Brook 💜Jul 1, 2023

@xabitron1 @andybaio @sysop408
Basically when a program reads stuff from a website. It costs money to serve a request, but humans can't consume fast enough for that cost to be an issue. If a program does it, it can become one, depending on what the request is and how much it does that.

Matt Round Jul 1, 2023

@andybaio Did Twitter have anti-DDoS/-scraping protection provided by a third party? Have to wonder if they've dropped a whole load of network safeguards & are largely flying blind

Andy Baio Jul 1, 2023

@mattround The timing of this makes me wonder. https://xoxo.zone/@neilk/110640561120091775

Neil Kandalgaonkar (@[email protected])

@andybaio @[email protected] @[email protected] also noted that Twitter had to exit GCP this weekend. They may have been trying to shed load in advance of a reduction in some capacities, but blamed it on scrapers (misdirection? mistake? who knows) But then inadvertently self-DDoSed at least from some clients. https://mastodon.social/@Pwnallthethings/110640380771469469

XOXO Zone

UberJumper Jul 1, 2023

@andybaio @mattround yup… https://www.businessinsider.com/elon-musk-twitter-stopped-paying-google-cloud-bills-money-platformer-2023-6

Twitter stopped paying its Google Cloud bills under Musk: Platformer

Since buying Twitter, Musk has embarked on drastic cost-cutting measures, including refusing to pay some bills.

Insider

Andy Baio Jul 1, 2023

@UberJumper June 21 https://www.bloomberg.com/news/articles/2023-06-21/twitter-resumes-paying-google-cloud-patching-up-relationship

Twitter Resumes Paying Google Cloud, Patches Up Relationship

Twitter has resumed paying Google Cloud for its services, patching up a relationship that became strained after Elon Musk acquired the social network and stopped paying Google and various other companies.

Bloomberg

UberJumper Jul 1, 2023

@andybaio good info

Neil Kandalgaonkar Jul 1, 2023

@andybaio @sysop408

@Pwnallthethings also noted that Twitter had to exit GCP this weekend.

They may have been trying to shed load in advance of a reduction in some capacities, but blamed it on scrapers (misdirection? mistake? who knows) But then inadvertently self-DDoSed at least from some clients.

https://mastodon.social/@Pwnallthethings/110640380771469469

Andy Baio Jul 1, 2023

@neilk @sysop408 @Pwnallthethings Interesting! I added that in.

Andy Baio Jul 1, 2023

@neilk or maybe not https://www.engadget.com/twitter-has-supposedly-started-paying-its-google-cloud-bill-again-213824844.html

Engadget is part of the Yahoo family of brands

Neil Kandalgaonkar Jul 1, 2023

@andybaio 🤔 I just don’t understand how they would go offline from only the web bug. I think it’s a side effect

Twitter probably has (had?) among the most sophisticated DDoS mitigation in the world and they are many ways to fix the underlying bug

Plus, I doubt many people are even using the website these days

I have personally seen a mitigation and recovery from a self-DDoS, quickly, with vastly less sophisticated tools

Andy Baio Jul 1, 2023

@neilk yeah, I don't think it's just the web bug either, probably a side effect of the blocks or rate limiting rather than the other way around

kkeller Jul 1, 2023

@andybaio but I thought Elon asked for "extreme hardcore", this isn't what he meant?!?

helafella Jul 1, 2023

@andybaio @sysop408 interesting.

sp00pkitt3n Jul 1, 2023

@helafella @andybaio @sysop408 Looking into it

Kiutawe Jul 1, 2023

@andybaio @sysop408 Elon is huge brain so this is not surprising

ninavizz Jul 1, 2023

@andybaio @sysop408 Ohhh... THAT is funny!

sp00pkitt3n Jul 1, 2023

@andybaio @sysop408 Conservatives are the masters of the self own

Rev. GothAlice Jul 1, 2023

@andybaio @sysop408 But… but… don't advertisers /want/ views?

… oh, right.

That would require actually having paying advertisers; fairly sure the smart and well-paying ones have already left.

Like smart users capable of doing so already have, and are continuing to do every time he breaks something different. Almost wrote new, but nah, he just keeps breaking the whole site over and over again.

No shame if the lock-in actually locked you in. Sympathies.

Tim Ellis 🦝Jul 1, 2023

@andybaio Too perfect. Just a real class act all around over there.

Feiryred Jul 1, 2023

@andybaio @sysop408 He's not paid the server bills is what this is.

Craig Mcgee Jul 1, 2023

@andybaio @sysop408 or, the bug was intentional to give him a reason to do all this blocking, to limit people's access to info on twatter. wouldnt put anything past him.

Kuntie Plopkins Jul 1, 2023

@andybaio @sysop408 @snowy2007

Mx. Luna Corbden 🐸Jul 1, 2023

@andybaio @sysop408 @daylightatheism “Let’s save time by testing in production!” –Elon Musk, probably

Negative12DollarBill Jul 1, 2023

@andybaio @sysop408
What’s that quote about not attributing problems to malice when it’s more likely to be stupidity? You really get such a clear example!

Sheldon Jul 1, 2023

@negative12dollarbill that would be Hanlon's Razor and it is absolutely an axiom I live my life by!

Never attribute to malice that which is adequately explained by stupidity.

tellyworth Jul 1, 2023

@sysop408 @andybaio the scraping is coming FROM INSIDE THE HOUSE

Ine Nai Jul 1, 2023

@andybaio 😏 But is it a bug really...? The plot thickens!

Nelson Jul 2, 2023

@andybaio @sysop408 master class of how to calmly lead a $40 billion company. Truly inspirational 👏

Philip Laureano Jul 2, 2023

@andybaio @sysop408

artnetworx Jul 2, 2023

@andybaio @sysop408
That’s nice. I’m done with Twitter. Admittedly, it took Elon’s action today to force quit my habit. I have a feeling if he let Twitter continue as it was when he purchased it, I wouldn’t have sought an alternative.

Ken Jul 2, 2023

@andybaio @sysop408 the complete clown that he is just makes stupid actions and seems Twitter is filling up with ads