TeaMar 21, 2025

Cloudflare announces AI Labyrinth, which uses AI-generated content to confuse and waste the resources of AI Crawlers and bots that ignore “no crawl” directives.

https://programming.dev/post/27302847

Cloudflare announces AI Labyrinth, which uses AI-generated content to confuse and waste the resources of AI Crawlers and bots that ignore “no crawl” directives. - programming.dev

Lemmy

Show threadlily33Mar 21, 2025

while allowing legitimate users and verified crawlers to browse normally.

What is a “verified crawler” though? What I worry about is, is it only big companies like Google that are allowed to have them now?

Show threadwingiee
I assume a crawler which adheres to robots.txt
Mar 21, 2025 at 3:18pmWeb
Show threadlily33Mar 21, 2025
I would love to think so. But the word “verified” suggests more.
Show threadkilleronthecornerMar 21, 2025
IP verification is a not uncommon method for commercial crawlers
Googlebot and Other Google Crawler Verification | Google Search Central  |  Documentation  |  Google for Developers

You can check if a web crawler really is Googlebot (or another Google user agent). Follow these steps to verify that Googlebot is the crawler.

Google for Developers