Chris HeilmannApr 7
If you want to ensure your content does not get indexed by big tech LLMs, just keep it in your robots.txt file.
Show threadadison verliceApr 9

@codepo8 then how about we test that theory?
i'm gonna put up (for test purposes) a robots.txt file and i'll DM you the results. and if it's wrong, i'm gonna call you out on it. funny story bro, but that's not ow robots.txt works.

and i'm gonna do an experiment to show this is true since you clearly think it is.
but read this
"The instructions in robots.txt files cannot enforce crawler behavior to your site; it's up to the crawler to obey them. While Googlebot and other respectable web crawlers obey the instructions in a robots.txt file, other crawlers might not."

https://developers.google.com/search/docs/crawling-indexing/robots/intro#:~:text=The%20instructions%20in%20robots.txt,file%2C%20other%20crawlers%20might%20not.

Robots.txt Introduction and Guide | Google Search Central  |  Documentation  |  Google for Developers

Robots.txt is used to manage crawler traffic. Explore this robots.txt introduction guide to learn what robot.txt files are and how to use them.

Google for Developers
Show threadadison verlice
which btw, Google search might but Gemini certainly doesn't
Apr 9 at 11:05pmWeb
Show threadrootfakeApr 10
@adisonverlice I think you missed the joke, the point was that AI crawlers never look at robots.txt because scrapers don't care, so it'd be the perfect place to hide things. (Sarcastically though, In reality they probably scrape that too)
Show threadadison verliceApr 10
@rootfake o I see.
I mean, if you really wanted to, you could ide the contence of the file by requiring auth.
also i only know if it's a joke if there is a content warning in front of the joke.
eitherway, i think i'm gonna conduct the experiment anyways and it would be fun to see it in action.
Show threadrootfakeApr 10
@adisonverlice definitely sounds like a fun one. Might be worth throwing a directory in there that's not listed anywhere else, see if they're scraping robots.txt for targeting data. And yeah, I figured that was likely the case, I have people in my life who have a hard time with jokes (including me, sometimes), and it seemed like that. (Cause taken literally, that joke would be an absolutely absurd statement)
Show threadadison verliceApr 10
@rootfake hmmm. I plan to start it either tomorrow or this afternoon.