Chris HeilmannApr 7
If you want to ensure your content does not get indexed by big tech LLMs, just keep it in your robots.txt file.
Show threadmatt
@codepo8 https://www.businessinsider.com/openai-anthropic-ai-ignore-rule-scraping-web-contect-robotstxt
Not so sure about this....
OpenAI, Anthropic ignore rule that prevents bots scraping web content

OpenAI and Anthropic have said publicly they respect robots.txt. But they are among the biggest tech companies ignoring the rule, BI has learned.

Business Insider
Apr 8 at 6:14amWeb