Paul Cantrell1d ago

RE: https://tldr.nettime.org/@tante/116605858023186072

Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because

and •only because•

they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.

Google is announcing — unambiguously and with great fanfare — that they are now fully breaking that already-ragged contract. We should reciprocate.

1/2

Show threadPaul Cantrell1d ago

Quick strategy discussion, for those who understand Google indexing and SEO:

If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?

The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?

2/2

Show threadPaul Cantrell1d ago

Same question as the previous post, except for Wkipedia. What would you like to see them do to send a shot across the bow?

Or…well, it’s Wikipedia. Maybe more like a shot to the hull.

3/2

Show threadPaul Cantrell1d ago
Going with meta noindex for now. My thinking is that this actively tells Google to yank already-crawled content from their index, whereas they might take a robots.txt entry to mean “do not update, but keep showing last fetched.”
Show threadSpring Jo 🥚  🍀1d ago

@inthehands this is a fence-post defense against this, google Will Not Care

just start poisoning the data once you detect that google is the one fetching it, just absolutely fucking destroy their LLM output

Show threadCoen Wesselman1d ago

@ShadowJonathan @inthehands they alsof are pretty involved with the contents of the standard (75% of the authors), so luring the crawler into a pit of crappy data is probably your only way to protest besides avoiding to hand any of your money and attention to them.

Link: https://datatracker.ietf.org/doc/html/rfc9309

RFC 9309: Robots Exclusion Protocol

This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Specifically, it adds definition language for the protocol, instructions for handling errors, and instructions for caching.

IETF Datatracker
Show threadPaul McGuire
@wsslmn @ShadowJonathan @inthehands Visit my crappy data generator at https://ptmcg.pythonanywhere.com/pyrac A Mad-libsian version of the old Racter program. Refresh for new crap content. (Not AI generated, just obnoxious Python.)
May 20 at 10:45pmWeb
Show threadPaul McGuire2h ago
I played with this some more, now cites fictitious books by various authors.