I think if you’re going to complain that “AI” bot traffic is swamping your website, you need to share numbers.
I saw a collection of anecdotes that included one from a guy who used a lot of swear words describing how angry he was at reading his web access logs. A full 70% of his traffic was AI bots!
Which turned out to be 2 requests per second.
Another site, which hosts static content that other people write, and makes it available for free download, was very angry that one bot downloaded so much of their free data it cost them $5000 in bandwidth in just one month.
That is their business model! Their pricing starts at $150 a month! Every CDN will extrapolate your monthly cost and you can set alert thresholds. Was it not someone’s job to monitor their expenses?
Candles $3,600, my family is dying
Another site that hosts data for over 10,000 projects was sad they had to give “the VM” more RAM. Someone else had to keep their site alive by increasing their pod count from 3 to 6.
I stubbed my toe this morning but I didn’t blog about it. It sucks but it’s part of that bipedal lifestyle I enjoy. Cost of doing business baby
Here’s an actual number instead of an anecdote, from a reliable source that actually works on this problem at scale.
Cloudflare says “AI Crawlers” are “just under 1%” of web traffic.
Jamie McCarthy