Greg RoachJul 31, 2025

Every website I control is being overwhelmed by robots.

It's all the same pattern: one request per IP address and random user-agent strings.

One site had 1.6 million unique IPs in the last week.

What kind of botnet is this?

Current fix:

If request has no cookies and a chrome/firefox/etc. user-agent, then send 4xx response with a cookie, and a body containing meta-refresh tag.

Browsers see the HTML and reload with a cookie.
Robots see the 4xx.

It works - but the robots keep coming...

Show threadDaddanieleJul 31, 2025
@fisharebest unique and in different inentum and from different location ?
Show threadGreg Roach

@Daddaniele -
On my employer's website, they only fetch the home page.

On a genealogy site, they are following links in a calendar widget to fetch every day in history.

Other sites have different patterns.

The IPs are from every country. AFAICT, mostly residential.

Jul 31, 2025 at 4:13pmWeb
Show threadDaddanieleAug 1, 2025
@fisharebest too many to be a legitimate user, too few to be a ddos attempt (which is the only thing I can think of as to why someone would ping the home 3 times a second).
To date it's never happened to me (or I've never noticed) so the premise is: I don't know.
My guess is some form of misconfigured crawler that relies on vpn's.