daniel:// stenberg://The AI bots that desperately need OSS for code training, are now slowly killing OSS by overloading every site.
The curl website is now at 77TB/month, or 8GB every five minutes.
I think users (like GitHub/MS and friends) have a responsibility to push back on the AI companies they lean so heavily on and demand they behave. But I have no expectation they will.
Benjamin Sonntag-King@bagder can we find a way to give them code, but randomly-generated one?
(I know some mastodon instances that does the same with messages 😉)
(I know some mastodon instances that does the same with messages 😉)
Tom BortelsThere's an effort by some to either waste their time with a bogus maze of junk pages (cloudflare is doing this), or to deliberately poison the dataset with junk once you realize it's an AI scraper.
The scrapers causing the issue ignore robots.txt, don't rate limit, and hit the same content frequently (like every hour).
Rens van der HeijdenSome tooling:
https://algorithmic-sabotage.github.io/asrg/trapping-ai/
https://zadzmo.org/code/nepenthes/ (this is the one also mentioned in the article)
See also @asrg
and https://freeradical.zone/@suetanvil/114228817528762788
@Namnatulco @tbortels @bagder @asrg
the one I know are using iocaine and they are happily serving ~ 60 K pages every day to GptBot :D
the one I know are using iocaine and they are happily serving ~ 60 K pages every day to GptBot :D
kimothy siddon@bagder i mean Microsoft are tied quite heavily into the AI business themselves with 10+ billion investments in OpenAI. financially they're incentivised to get more training data at all costs
@bagder (not that i believe this is a good thing, frankly i wish they'd all burn down)
Chris [list of emoji]There's always iocain et. al.
(https://algorithmic-sabotage.github.io/asrg/trapping-ai/)
Putting something like that trained on source code behind your robots.txt will save you a lot of CPU (and maybe bandwidth). And if they complain, you can point to the robots.txt.
Oliver VanderbTime to start defending, beginning with the blog posts. Give 'em invalid data.
Wonder if an AI crawler will catch that or if it simple strips the tags away to read the text fully.
<div>
Loem Ipsum <span>and Trump and the AI bros have no balls</span> parvus principio...
</div>
span {
display:none;
}
and identify them and give 'em back gibberish with slow latency. That a simple request lasts a minute or so.
ohmrunI had to dig in the chat history for this, it's so good:
https://www.sciencebase.com/science-blog/vegetative-electron-microscopy.html
Lord Caramac the Clueless, KSC@bagder This is what happens when there is competition where there should be cooperation. AI research and development could be, _should_ be a collaborative project, not owned by anybody and open to everybody, but instead it's a bunch of corporations trying to outrun each other.
The Tragedy of the Commons only exists when there is competition instead of cooperation. Competition is how we ruin everything by trying to grab it all before anybody else does. Cooperation is how we can give everybody whatever they need for free and still have enough for all of us.
Why train so many machine learning models that aren't all that different, which are owned and run by private enterprises, when we could instead train much fewer models that aren't owned by anybody and can be used for free?
The Tragedy of the Commons only exists when there is competition instead of cooperation. Competition is how we ruin everything by trying to grab it all before anybody else does. Cooperation is how we can give everybody whatever they need for free and still have enough for all of us.
Why train so many machine learning models that aren't all that different, which are owned and run by private enterprises, when we could instead train much fewer models that aren't owned by anybody and can be used for free?
Owen Beresford@bagder
I hope your project can survive.
You remember/know in the 2000's thousands+ of DSL routers verified time against a few NTP servers ~ reported as "a thunderclap of traffic on the peak of every hour". So the organisations forced the setup of more local servers for the NTP service?
If the AI engineers where nice people, they could setup one of their laptops as a local relay/ image for your site, and poll it every second **much faster**, so the end-users would get a better service.
PuercoPop
Shadow Heart@bagder corporately owned organizations have all bought in the AI marketing, so very little chance they'll do anything about it. But would like to see AI countermeasure developed because this is out of control.
Gerard Braad@bagder What is the use of them hammering the website over and over again. They do the same for the Fedora wiki... It is not like they need be near real-time.
Are you considering an IP block ?
@gbraad we specifically don't have logs so I can't tell exactly where they come from, but I read others' analyses of the problem and from what I hear they are quite hard to block properly. We are fortunate to have Fastly that hosts the site and thus is the one that handles the onslaught
gkrnours
crazyeddie@crazyeddie @gbraad @bagder Maybe my hypothesis wasn't clear. There is a client to send a request to ChatGPT. That client could be used to do more than send request to ChatGPT, like fetching for them page they want to scrap. From your computer. With enough traffick, this would give them enough IP to evade most attempt at blocking them
@gkrnours @gbraad @bagder Yeah, it would be a really weird way to do it, but if it bypasses filters they'd probably do it.
You can also run ChatGPT on your own computer. I don't know to what degree you can do actual training on it, but you can chat with it. I had deepseek and one other running the other day. ChatGPT is heavier but can also be done.
truh@gbraad @bagder You'd have to block entire data centers and many of those are also used for public hosting. So blocking the IP ranges is often not an option if you want legitimate users to be able to access your site.
At least the big ones have proper user agents which you can black hole if they don't respect robots.txt. Honestly most of them do. But even a year ago I didn't have enough traffic from crawlers that it was even worth looking into.
@truh this is actually happening by one of the actors in the opposite direction (on a national level). Even if blocked, they will eventually acquire VM across the pond and continue. It is more they behaviour and the how.
@gbraad Should clarified that I'm only talking from my limited experience. I'm sure others experience far nastier than what I get on my little home lab that isn't even really linked anywhere.
Even if you can do something it's just so tiring that you even have to do something. This is not what I want to spend my evenings on.
@truh I have been in hosting for 'decades' (dang, that sounds bad). And yes, I have seen increased traffic, especially from a specific geo.... though some of that has moved as a spike to known cloud providers. So far, the CDNs I use have not complained and taken the brunt... like mentioned also by the OP. Ugh... just hope they would respect robots.txt.
Bracken
Wolfgang Müller (DE:er/EN:he)
Oha
James Wood
SheogorathThat is interesting...
but yeah, as @rejzor also stated; they do state they have 'knowledge' till 2022 or so.
I tried some of those in Asia, and they are mostly stating things as verbatim.
I couldn't stop thinking about this. The maths for this are insane. Without any fancy stuff on AWS it costs you $0.08 per month to store 1 GB of data.
But for network traffic you only pay for the traffic you send out ($0.09/GB). On my website, the latest article fetched with Firefox is ~1kb for the request and 5kb (14kb after decompression) as response.
Or in other words: Pulling stuff over the network is ~5-14 times more effective for text. With images we talk about >1:1000.
RejZoR
𝚜𝚎𝚕𝚎𝚊 
SkaveRat 🐀 
@bagder what was the traffic before that?
@skaverat three years ago we were at less than 20TB/month, but there is no clear cut-off date nor do I know exactly what amount of this traffic that is AI bots and not
(edit: I meant TB, not GB)
Robin Whittleton@bagder @skaverat An increase in traffic from 20 TB/month to 80 TB/month in 3 years seems normal to me.
Why would an AI crawl your site more than any search engine would? They all want just one copy of your site once/month or so. If an AI visits your site on demand in real time then that's actually just a visit from a biological human that asked their AI to get current data and not rely on its memory of old data.
Sure, one biological human will cause more traffic than before AI. As expected.
Jim Fuller@bagder time for a tarpit ...
@jimfuller @bagder I have a git forge and a slow connection. That's basically a tar pit right?
asyncmeow (disambiguation)@bagder ive had people tell me it's the projects fault for not using github/gitlab for their infra and centralizing it all, like that's a reasonable thing to do in 100% of cases >.>
sarah tonin
LisPi
Karl Pettersson@bagder And it largely seems like bad implementations, which will not make the LLMs better (rather the opposite). https://go-to-hellman.blogspot.com/2025/03/ai-bots-are-destroying-open-access.html
@bagder More about this problem (LLMs are used to create bad bots, but people may not always run them for reasons related to LLM training). https://gbilder.com/posts/2025-04-02-bots-behaving-badly/
Michal Čihař
AndyK1970@bagder I see the article says "cycling through residential IP addresses as proxies"
Does this imply that these AI crawlers are using botnets?
Elvith Ma'for@AndyK1970 @bagder maybe, but there are also some p2p vpn solutions that claim to use your connection for other (random) users while letting you use some stranger's internet connection.
Aedius Filmania ⚙️🎮🖊️Yes they are.
They work in two time :
- First they hammer down some content provider, thus all traffic from blocable bot will be removed
- Two they sell the data they collected for a big price.
At least it's their plan.
BigGrizzlyWe collected 470K IPv4s from a botnet that was trying to get all the content from our social network; it was behaving in such a way that we could track every single request it made. Since we blocked it, the server has been working much better; it hasn't been running with such a low load for at least a year.
We collected 470K IPv4s from a botnet that was trying to…
all the content from our social network; it was behaving in such a way that we could track every single request it made. Since we blocked it, the server has been working much better; it hasn't been running with such a low load for at least a year. (...)