Mastodawn

gaytabase 8h ago

Jonathan Corbet

The @lwn web site is currently under the most intense scraper attack I have seen yet. 1.3M unique IP addresses within the last couple of hours, and it's not done yet. The work we have done on defenses appears to be paying off, though; the server is holding up reasonably well — so far.

...just in case anybody wonders why I have a rather dim view of the whole AI industry...

Nuke Them from Orbit 8h ago

> ...just in case anybody wonders why I have a rather dim view of the whole AI industry...

Me too. Because (and this is just another instance of it), the greed of the few destroy life as it was for the many.

warthog9 8h ago

@corbet @lwn if you need more capacity, give a shout, I've a bit i can throw to the cause if you need it

Bart Veldhuizen 🚀7h ago

@corbet @lwn same on @blenderartists right now with up to 3.7M requests/hour. Looks more like a DDoS in our case than AI scraping though (which I suspected at first)

Fazal Majid 4h ago

@BartV @corbet @lwn @blenderartists
Who would have a motive to DDoS LWN or Blender? Other than Microsoft and Adobe, of course.

Most likely those IPs are from residential proxies so you can't do an easy filtering rule like "Block all IPs in AWS/GCP/Azure address spaces". There were revelations last week than half of all Smart TV apps include residential proxy SDKs.

Bradley M. Kühn 🏳️‍🌈47m ago

At SFC, we've been seeing the primary culprit is .cn IP numbers and Zuckerberg.

& I can confirm User-Agent is fiction, at least from those parties. robots.txt of course ignored.

Cc: @BartV @corbet @lwn @blenderartists

Jef Poskanzer 1h ago

@BartV @corbet @lwn @blenderartists What's the difference?

Michael Cook 7h ago

@corbet @lwn I hate sites with only a paywall. I like open or your “open later” model far better.

But I’m surprised more sites haven’t gone paywall only at this point. How is anyone supposed to survive this?

(Longtime subscriber 👍)

Very Human Robot 7h ago

@corbet @lwn
What's the user agent if the scrapers?
Are you using any captcha or cloud armor?

Alexander S. Kunz 6h ago

@StompyRobot on my site it’s 90% residential proxies that camouflage as some Chrome. Not easy to block. Some pass simple automated challenges. @corbet @lwn

Jonathan Corbet 5h ago

@StompyRobot @lwn User agent is whatever random fiction they choose to put in there; there is no useful signal there. We really don't want to inflict captchas or cloudflare or any of that onto our readers, so we've had to find other ways to defend the site.

Chris Samuel 7h ago

@corbet @lwn ouch - fingers crossed for you all!

I wonder if this is related to weather.gov going down earlier (and forecast.weather.gov still being down), or just coincidence?

Nick Silkey 📻 N5ILK 🪓🪵 🪣💧7h ago

@corbet thank you and the @lwn team for your service. ✌️💙

Jernej Simončič �7h ago

@corbet @lwn A few months ago one of my clients' sites was getting 5000 requests per second, about 10-20 requests from a single IP, all residential. Server held up quite well until /var filled up because the logs didn't rotate fast enough (daily rotation, /var is 16 GB and normally has around 13 GB free).

@corbet @lwn Website operators should be especially vocal about this, because too many people who have a positive view of AI companies have no idea..

𝔅icyclet𝓽𝓲𝓷𝓰 5h ago

@corbet @lwn Inkscape also got heated today:

https://floss.social/@doctormo/116825974597784492

Martin Owens :inkscape: (@[email protected])

I'm going to sleep. If anyone is wondering why the #inkscape website is down, it's because it's being crushed by AI bots again, so I've taken it offline. I tried to put in more ip-blocks, and then tried to install Anubis. But I'm not a sysadmin and so this is just taking time to do. Can't do much of it tonight (this morning) so I'm afraid it's staying offline. Edit: Thanks for the help from @[email protected] getting the server back online. It appears to have been Facebook attacking.

FLOSS.social

sergiodj 5h ago

@[email protected] @[email protected] Amazing. I'd be interested in a post describing what you folks did, lessons learned, etc. (Assuming one doesn't exist already!)

Alan Langford 🇨🇦🧤🧊摏 3h ago

@corbet @lwn I am in the process of rolling out the next major version of my WAF and I've connected to the Abuse IP DB, which I now use to short circuit all the rest of the tests if the score is >=75. It's killing about 95% of the incoming traffic, and the WAF is getting about 95% of the rest (largely through ASN-wide blocks; host an AI scraper and you're dead to me unless you're whitelisted.)

FLOSSbOxIN 1h ago

@corbet @lwn
I had these few months back on my sites. Had to send them to hell by blocking out.

Bradley M. Kühn 🏳️‍🌈50m ago

Anyone scrapping to (re)train LLMs is a selfish capitalist who doesn't care who they inconvenience &/or hurt.

We've enough LLMs for the foreseeable future. None are as Free and Open as we'd like, but I'm sure it's not someone trying to build a truly #FOSS LLM that's DDoS'ing #LWN rn.

All new #LLM training should stop *immediately*; continuing now on training is unconscionable. If you work for a company that is still training, I urge you to resign in protest.

Cc: @lwn @Andres4NY
@corbet

bignose 44m ago

Agreed in the main, @bkuhn.

I imagine an obvious response is “if we don't keep putting *new* data into the #LLM, it will fall out of date”. As far as it goes, yes that's true.

To which I'd respond: so what, why is that so urgent? Why should the internet be degraded for that? Not enough to justify the hammering of websites, the bulldozing of consent, the active deception to pass blocks, the refusal to countenance anything except #Hyperscaler interests. Stop it all, now.

Bradley M. Kühn 🏳️‍🌈41m ago

@bignose Even more than whether it is urgent, & even whether or not you're pro or against *using* LLM-gen-AI, the world is still figuring out if these monstrosities they are useful *for* (if anything).

The ballyhoo is clearly wrong, but I also think those who say they are not useful for anything are also wrong.

We (humanity) need at least two years to even begin to understand what we have & what it's for. Let's pause and figure that out without capitalists in the driver's seat.

@corbet @lwn From my work experience I can say that the only remediation at that scale is #Blackholing traffic at #IX-level from all malicious ASNs used for said #DDoS and sending angry #AbuseReport mails every originator and their Upstreams.

- Make it THEIR PROBLEM!

Also let us know of the IP ranges so everyone else can block them as well!