I keep seeing webmasters talking about how to block AI scrapers (through user agents and IP blocks) and not enough webmasters talking about the far better option of rigging their site to return complete gibberish or transgender werewolf erotica* when AI scrapers are detected.
*depending on which one you think is funnier to poison the AI models with
@foone iāve been returning a 302 to a 10GB binary file from hetznerās speedtest page, but honestly.. maybe I should?
all of my pages already contain prompt injection (in multiple places, even)
@domi @foone short term thinking! There's always going to be more scrapers who haven't learned their lesson.
If we can poison the well at scale, we can collapse the business model 
...
Who am I kidding?... I'm sure they'll have models selecting which data is "legit" and it will get better at detecting the "transgender werewolf erotica" over time... The stupidest of arms races
Alice Averlongš³ļøāā§ļø
dmi š½ 
Emelia/Emi
LisPi
sunz*
Nic Roland 
Nathanif ($http_user_agent ~ 'GPTBot|ChatGPT\-User|Google\-Extended|CCBot|PerplexityBot|anthropic\-ai|Claude\-Web|ClaudeBot|Amazonbot|FacebookBot|Applebot\-Extended|semrush|barkrowler|PetalBot|meta-externalagent|meta-externalfetcher|facebookexternalhit|facebookcatalog') {
return 308 https://nbg1-speed.hetzner.com/10GB.bin;
}
if ($request_uri ~ 'wp-content|wp-login\.php|wp\-includes') {
return 308 https://nbg1-speed.hetzner.com/10GB.bin;
}
itās a bit opinionated, the list of bots includes not only AI but also other BS. I include this into other files, inside server { }
@foone Yeah, it's definitely something to do if you aren't paying for the bandwidth or AI isn't hammering an app backend to do it.
Like...static site hosted on a free tier somewhere? Yeah, send them down an absolute bottomless pit of nonsense
@foone If you can give me a piece of nginx config of five lines or fewer* I'll be more than happy to feed them a copy of My Immortal or whatever.
* if you need more lines to cover more scrapers, that doesn't really count.
We need a web proxy that will scrape AO3 + Literotica for 100 most recent stories, sort them by length, and just grab the "next 100 words" from each story in the list, until it runs out of words or hits end of list.. and then serve that to scrapers
@ForiamCJ nah, we don't want to stuff their fics into the AI models any more than they're already in there.
I'll just write my own sacrificial transgender werewolf erotica
The 'goal' of stealing a few words from lots of different stories was to give it the illusion of coherent thought/ storytelling (to the parser) while also trying to make is as incoherent as possible. But I respect your position on that.
@foone I now think it's radical and awesome to have it return transgender werewolf erotica because chances are this content would be subversive enough to destroy the imperial colonization of the internet when it makes real transgender werewolves out of the engineers and staff body.
Genius, pure genius. šš
@foone Maybe push My Immortal + Eye of Argon through the dissociated-press algo (50% weight on each).
I wonder if I can make this static site compatible. Perhaps for each post I pre-generate a slop version that sits next to the real post, and I use an .htaccess file to pull from the slop file instead of the real file when the useragent matches?
@foone I decided to just serve AI scrapers a Markov-mangled version of my own blog posts.
I love the idea of poisoning them with specific topics, but honestly, the output of the Dissociated Press algorithm is probably the most effective possible poison I can make!
Technical deets: https://www.brainonfire.net/blog/2024/09/19/poisoning-ai-scrapers/
In the last few months:
- I was looking for recommended soil planting temps for a garlic variety. Gemini came up with a manglement of my Garlic is Weird blog post from last year.
- During a discussion with a friend about farm transparency reports, he put a prompt into one of the paid GPTs asking for an example. It said you should always abbreviate and gave examples, North Little Rock (NLR) and Jacksonville (JV) -- the designations I used for our properties in our transparency report.
- I was arguing with grep and thought my invocation was right, but it wasn't working. Did a google search that returned unrelated Crapoverflow and man pages. Tried a different query. I wouldn't have noticed the AI slop at the top, except it was using a very specific example from Linux for Government (ISBN in my bio). It didn't just pull "read, red, road, rad, reed..." from nothing. It was the example file we used for grep.
Sidenote: grep works better if you point it at the right file.
Funny thing is, the blog for my farm isn't spread that wide. The Linux book was a one-off mainly written to ensure I had a good book to teach from and didn't have my job outsourced to RedHat instructors. I don't think it sold 5k copies.
These aren't top of the list reference materials. They're good, but they aren't widely linked or known; so I wonder how (or if) sources get weighted.
@johntimaeus Yeah, the weighting thing is... a good question. It sure seems like Google trained theirs pretty heavily on reddit, for instance.
Here's something weird I found when poking at ChatGPT, getting it to reproduce something from a git repo. (It actually did a live fetch of a page on my site instead.) But the formatting was all horked up:
Ā« For more detailed documentation, you can check out their (GitHub
)tps://giā(Brain on Fire
)r the protocol description. Ā»
Wonder what's up there.
Might be fun to implement the more advanced version of Dissociated Press that operates on words instead of characters.
Really tempted to make one that operates on syntax trees (operating on HTML, producing valid-ish HTML) but that is *definitely* too deep a rabbit hole for this week.
@varx @foone You might be interested in a small project I hacked together recently, āPoison the WeLLMsā.
https://codeberg.org/MikeCoats/poison-the-wellms
Iāve stuck some examples of its output in my blog here,
I'm currently serving enticing garbage to AI scrapers whose user-agent matches this regex:
GPT|Claude|anthropic|\bcohere\b|\bmeta\b|Google-Extended
Are there others I should include? I based this on a small sample logs, since I don't have access logs turned on as a baseline.
To help preserve a safe Internet for content creators, weāve just launched a brand new āeasy buttonā to block all AI bots. Itās available for all customers, including those on our free tier.
I've been a little stumped on what to do when CCBot comes to scrape my website.
CommonCrawl archives the web, and then people can use those archives for research, or building search engines... or training LLMs.
Some of those are OK with me. Others aren't.
So... do I serve poison to CCBot? Or block it? Or do nothing?
I asked Common Crawl for adviceāare there ways of indicating acceptable use of my scraped website?
Answer: No, not yet, maybe at some point; there are standards being hammered out. But at least they preserve the robots.txt and response headers and such, so a well-behaved consumer of their archive could choose to respect what I've indicated.
This doesn't help with future scrapers, of course.
I think I'll feed Common Crawl garbage for now. Markov output is still mostly search-engine friendly...
e(Ag)le š¦
King Naga Calyo Lucere-Delphi
Sparky
h3mmy 
princess pancake 
ā
elke
Devourer
paramilitary organizer
Gulleko
Albert Cardona
Kawaoneechan
Farbs
For I am CJ 
Filene Taylor
Aaron
lapisliozuli
varx/social
J.M. Hill
John Timaeus
Old Man in the Shoe
Bill Pleinš¶
John Harris
Mike Coats š“ó §ó ¢ó ³ó £ó “ó æšŖšŗšā»ļø