gallaugherWe need something beyond robots.txt. I want my context search engines indexed. I do not want my content used to train AI. https://searchengineland.com/gptbot-openais-new-web-crawler-430360
Sean Riley@gallaugher Current robots.txt most of these abide
User-agent: Googlebot
User-agent: DataForSeoBot
User-agent: YandexBot
User-agent: SemrushBot
User-agent: AhrefsBot
User-agent: DotBot
User-agent: Baiduspider
User-agent: 360Spider
User-agent: Yisouspider
User-agent: GPTBot
User-agent: bingbot
User-agent: AwarioSmartBot
User-agent: yacybot
Disallow: /
User-agent: PetalBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Sogou web spider
Disallow: /
User-agent: Sogou inst spider
Disallow: /
User-agent: MJ12bot
Disallow: /