@artemesia @carnage4life Yes, they own their content. If they don't want their data to be scraped they can use a wildcard and prohibit all bots to do it.

It is their content and they should decide who gets access.

The internet is by default open, so the default should be "everybody can use the content".

Do you have another opinion?

@artemesia @carnage4life I would prefer to have categories like: "SearchEngine", "DataModel", "NewsAggregator" and the like.

robots.txt is not meant for that, but changing the interpretation of the agent string would technically allow for that, similar to using multiple CSS classes for one element back in the day.

@artemesia @carnage4life
You can match all with * or name specific agents like Googlebot.

I believe the white and blacklisting must be done by the content owner, so the robots.txt is the right place.

I also believe that we need categories instead of specific agents, otherwise we get a closed web very fast. Why should ChatGPT be allowed to scrape and Gemini not?

For "deals" in B2B APIs are the correct approach. They are more efficient and accurate.

https://moz.com/learn/seo/robotstxt

@carnage4life We only have so many original documents and texts, and they have biases.

Filtering content for #LLMs only reduces the total amount of training data.

Until someone rewrites our history and knowledge within their worldview in its entirety, humanity will live in its bubble for better or worse.