Travis 'Nep' SmithI really like the idea of a #noml meta tag in addition to #noindex nofollow, etc.
I think there could be a way added in #robots.txt as well, but honestly, robots.txt was always a solution meant to minimize the amount/scope of pages a robot tries to fetch, whereas a meta-tag / x-header solution is better when you want some but not all uses of page content (i.e. the robot still ought to fetch the page) - so I'm not in a rush to modify robots.txt's spec
https://mastodon.cloud/@Mojeek@mastodon.social/113078163267679409
Mojeek (@[email protected])
"A voluntary code of practice, where you can flag your wishes to search engines using robots.txt, has largely worked, but what can be done in the age of generative AI? We have a proposal, which we ask you to consider and support below. But first we’ll explain how it would work." https://blog.mojeek.com/2023/10/noml-proposal-and-open-letter.html
Mojeek Search Engine
