
Introducing pay per crawl: Enabling content owners to charge AI crawlers for access
Pay per crawl is a new feature to allow content creators to charge AI crawlers for access to their content.
The Cloudflare BlogMy take: this was inevitable from someone. Enforcing it across many sites with valuable “content” (there’s a better term but I’m failing to conjure it right now) is tricky. This will obviously drive more people to Cloudflare short term, which… not great for the health/decentralisation of the internet, but hopefully acts as a good PoC that others implement it (like Anubis does for proof of work bot blocking).