PLOT
andreasdotorgThis is why we can't have nice things.
Weird Mustard (offline)@andreasdotorg Am Ende ist es einfach Musks AI Firma. Deswegen ist man ja zb auch nicht mehr auf Twitter, um die Scheiße nicht zu unterstützen.
Kevin Karhan 
@andreasdotorg if I were @internetarchive I'd limit the amount of traffic and connections #AWS can make to 1 per IPv4 & 1 per IPv6 @ 64kbit/s and automatically abuse-report and temporarily soft-block the source IPs via #blackholing [like any #DDoS] if not the entire #AWS #AS!
Simon Lucy@kkarhan @andreasdotorg @internetarchive
With swarms of serverless harvesters?
In the end you still end up with CIDR blocking.
@simon_lucy @andreasdotorg @internetarchive
Then that's a necessary sacrifice one needs to do.
If #aws doesn't combat #abuse then it's only valid to #DROP [#DontRouteOrPeer] their systems...
And yes, I do yeet hostile networks as an act of self- and mutual ITsec...
https://github.com/greyhat-academy/lists.d/blob/main/blocklists.list.tsv
@kkarhan @andreasdotorg @internetarchive
The point I was making is that IP specific rules aren't sufficient.
@simon_lucy @andreasdotorg @internetarchive OFC you'd have to block all CIDRs associated to the ASN of AWS...
Which is relatively easy considering that said assignments are public...
@kkarhan @andreasdotorg @internetarchive
Yes, and that negates archive.org, so it's a very temporary mitigation. I imagine AWS knows and have begun limiting the customer.
[Yaseenist] CauseOfBSOD 
@andreasdotorg Wonder if there is a way to identify this abusive traffic and respond with garbage data and insults?
Rairii@CauseOfBSOD @andreasdotorg why respond with garbage data and insults when you can respond with a seven or eight figure invoice? (recurring monthly, of course)
of course they can afford it, they can afford to waste their money on AWS...
John Mierau...Because society doesn't prohibit bad actors when they have fat wallets?
it's B! Cavello 🐝@andreasdotorg I haven’t seen info from @internetarchive here on fedi, but on Twitter they shared this update: https://twitter.com/internetarchive/status/1663001853964189697
@andreasdotorg @internetarchive direct link to blog: https://blog.archive.org/2023/05/29/let-us-serve-you-but-dont-bring-us-down/
“Those wanting to use our materials in bulk should start slowly, and ramp up.
Also, if you are starting a large project please contact us at [email protected], we are here to help.
If you find yourself blocked, please don’t just start again, reach out.”
DieMar4@andreasdotorg Kleinanzeigen wäre vielleicht eine Möglichkei Möglichkeit 🤔
Sebastian Lasse@internetarchive
@brewsterkahle
any updates and background appreciated here in this cosy and federated place :)
nocci #GoToSocial
Tito Swineflu@andreasdotorg There will never be valid web traffic coming from an aws-owned IP. I always block them all first thing. Cuts out 99% of the brute force attacks.
Aral Balkan@andreasdotorg Sounds like capitalism working as designed. No doubt another Silicon Valley success story in the making…
sebastian büttrichit is not easy having #good #things in the presence of #capitalism.
Marie Belanger@andreasdotorg too bad there is not a "scowl"button...
Rob Bos@andreasdotorg Oh, man. I have a couple of library archive sites, and I ended up just blocking AWS and Azure entirely because of the constant high-pressure scraping.
Some people just can't keep their scraping down below a reasonable rate limit. Like, I'd be fine if you kept requests under 1000/hour or something but if you're spinning up 50 servers to hoover up thousands of pages each as fast as I can serve them, fuck y'all. Now you get NOTHING.
@andreasdotorg It's also a little ironic because if they'd just ASKED I could fedex them a hard drive or something.
monoxane@andreasdotorg AI companies and ruining good things for everyone, name a better duo 
Alison Creekside
Garth Cummings@andreasdotorg @internetarchive any reason you’re not posting these updates to your Fediverse account?
Carsten@andreasdotorg It is because some people just overdue some stupid thing without realizing the consequences of data hoarding or data leeching.
enmodo ⚛️ 🧬 🇺🇦 🍉@andreasdotorg maybe someone is trying to archive the archive...
TetsuoYtek 🟦Add to the terms and conditions that traffic coming for bots or “non human” interface will be charged based on the traffic generated.
Then send them the bill.
The Doctor@andreasdotorg Wait until they start scraping the Wayback Machine.
IllusionOfMana@andreasdotorg Its stupid how much of this comes from AWS. Amazon really could care less about what happens on its services. My grandfathers HTML only site constantly get blasted with third party scrapers and DDOS. I shouldnt have to pay this much in security for all HTML sites about nature photography.
HistoPol (#HP) 🏴 🇺🇸 🏴They need to filter out #AWS and other corporate mass-retrievers.
10-20 accesses per day or so.
bigiain@HistoPol @andreasdotorg How about as soon as you get past 1000 requests in an hour, you start getting fed back random gibberish instead of actual pages… <smirk>
1000 seems way too high fit me still @andreasdotorg
Jaime Herazo@andreasdotorg
The Librarian gave some details, it went down for a bit after some people hammered it badly twice: https://blog.archive.org/2023/05/29/let-us-serve-you-but-dont-bring-us-down/
The Librarian gave some details, it went down for a bit after some people hammered it badly twice: https://blog.archive.org/2023/05/29/let-us-serve-you-but-dont-bring-us-down/
(((Jann Gobble)))🏳️🌈@andreasdotorg Hey @donmelton ⬆️ This could probably use a boost to an audience larger than mine...thanks in advance! #archive
Don Melton@jann @andreasdotorg Already boosted. 👍
@donmelton @andreasdotorg I should’ve known! 😂
David 🏳️🌈🏳️⚧️NB they/them@andreasdotorg Corporations can take whatever they want but when we do it to them they call it "piracy"
Elias
Demi Marie Obenour@andreasdotorg Has this been reported to AWS?
sunz*@andreasdotorg we can give them hell
potatoxel
pedro 🍉@andreasdotorg maybe they should have asked the AI how to harvest content in a civilized way?
c0c0bird
Guillaume Rossolini@andreasdotorg
I've seen people use the Archive as a CDN, in the past 🤷🏼♀️
I've seen people use the Archive as a CDN, in the past 🤷🏼♀️
Aviancer@andreasdotorg Upsetting to see them effectively DoS the service when they’ve could do it in a much more reasonable manner. archive.org is a treasure for so many historical computer resources that aren’t really available anywhere else anymore.
Craig Nicol@andreasdotorg so, first cloud services were taken out by cyptominers, and now comment providers are getting taken out by AI? And remember the silicon valley startup wise business model was ddos attacks on restaurants so their app was the only place to get a table?
At which point do we start treating VCs as enablers for this drive-by suffocation?
At which point do we start treating VCs as enablers for this drive-by suffocation?
Martin Be
Walker Boh🛡@andreasdotorg guess elmos not the only one feeling the heat...
Michael Porter@andreasdotorg @NikaShilobod Too bad they can’t detect the AI scrapers and limit the source material to Mr. Rogers, Sesame Street, The Friendly Giant, Mr. Dressup… Train those young metal brains properly!