Time will tell how this shakes out…… https://blog.cloudflare.com/introducing-pay-per-crawl/
Introducing pay per crawl: Enabling content owners to charge AI crawlers for access

Pay per crawl is a new feature to allow content creators to charge AI crawlers for access to their content.

The Cloudflare Blog
My take: this was inevitable from someone. Enforcing it across many sites with valuable “content” (there’s a better term but I’m failing to conjure it right now) is tricky. This will obviously drive more people to Cloudflare short term, which… not great for the health/decentralisation of the internet, but hopefully acts as a good PoC that others implement it (like Anubis does for proof of work bot blocking).
@arch Yeah I'm going to pitch this to Automattic for wordpress.com blogs

@arch LMAO

I would totally do this.

"Read my blog for free, or pay $25/page for your AI to read it for you." This is praxis.

Enshittify the enshittification machine.

We should also throw ads in there, via a deliberate prompt injection. I would use that to make their AI bot spew paragraphs of the grossest kink content known to fandom.

@soatok would be really funny, I won’t lie. I don’t know if a header to distinguish is sent to the origin, but if it is…

Oooh boy. :3

@arch @soatok Make them pay and then serve them a zip bomb
@zuthal @arch @cadey So I've got some feature requests for Anubis ;3
Use the very fund to feed generated contents to LLM crawler is the only right mo... | Hacker News

@soatok @arch I like the principle, but it does open the door for a future where people will pay through browser extensions to access content
@arch that's given me an idea
Might slap some htaccess rules on my site for specific crawlers to return a custom 402 page with the "fuck you, pay me" dog
@arch dunno which is better, blocking AI or $5 per crawl

@patterfloof @arch $5?

Raise your rates.

Make each page of your blog cost as much as an ISO standard to read.

$5 is rookie numbers. Make it painful on their budget sheet.

@soatok @patterfloof ISO lays out the standard page access price. Which is the one free standard it offers.
@arch @soatok @patterfloof that's a nice start. But I believe that's price for a single person to buy single page. Their thing is "scrape once, access everywhere", so that should be multiplied by, oh, say 1% of their user count.
@viq @arch @patterfloof Now yer thinkin with yer dipstick!
@viq @arch @soatok @patterfloof Don’t like CF? Deploy a reverse proxy that supports authentication based on origin of the request. Charge a fee for credentials. Say X% of their revenue p. a. 🤔
@arch @soatok @patterfloof I thought Ada was the one truly free thing from ISO?
@soatok @patterfloof @arch it should always cost orders of magnitude more to burn the planet down than to not.

@soatok @patterfloof @arch

How about $5, but....
It's a one year licence. A year from the date, they have to pay again or remove the data from their model.
Shuffle and change the pages regularly, so they can't just rescan the entire site.

@patterfloof @arch same thing really, none of them will ever pay

@arch To be honest, with all the AI crawling - which I can't really block with my current setup - I'm really thinking of maybe using Cloudflare for some of my sites. But I'm unsure. I don't trust them. (Not that I even know if it would be possible to add them to my setup anyway.)

Getting money per crawl (e.g. $25) could be funny, though I don't know how I would need to send that to the tax people in Germany.

Does somebody here have any helpful impressions of Cloudflare? Currently, I'm using the DNS of netcup and a simple nginx config for reverse proxying to my actual services (running in Docker). I wouldn't want to stop using netcup's DNS servers.

@SteffoSpieler You can technically delegate specific subdomains to Cloudflare rather than the whole domain, but by its nature Cloudflare needs control of the nameservers.

Tax wise, it would probably be reported as normal income. Obviously not familiar with German tax law so can’t really say how that would work.

I’m biased, so I won’t tell you whether to use it or not ;p

@SteffoSpieler @arch they're a massive tech company

i mean, just take their own word for their morals after their whole "knowingly servicing kiwifarms" https://blog.cloudflare.com/kiwifarms-blocked/

we do not believe that terminating security services is appropriate, even to revolting content. In a law-respecting world, the answer to even illegal content is not to use other illegal means like DDoS attacks to silence it.

we are committed as a security provider to protecting our customers even when they run deeply afoul of popular opinion or even our own morals. The policy we articulated last Wednesday remains our policy. We continue to believe that the best way to relegate cyberattacks to the dustbin of history is to give everyone the tools to prevent them.

Blocking Kiwifarms

We have blocked Kiwifarms. Visitors to any of the Kiwifarms sites that use any of Cloudflare's services will see a Cloudflare block page and a link to this post.

The Cloudflare Blog

@arch Well there goes my educational and research (NOT AI) archiving. I already get blocked by half the websites I try to archive. And I am not going to do the thing where you proxy stuff through residential connections, my bots are kind to servers and honest about being bots.

I guess thus falls the open web, murdered by GenAI.

@arch

What would be interesting if this leads to a wider standard convention for 402s such that setting up and making micro payments for anything online would be simple (finally).

@arch it's not going to stop the Chrome/121 AI skip bots from Vietnam from scraping your site. It'll only stop the ChatGPT and perplexity bots.