@tdr @ben it's called the robots.txt clause for a reason.

Also you may want to point @senficon at it, tho "#learning" is not part of that rejection just like #caching cuz otherwise #WebBrowsers would be illegal!

And even then there's a simple solution:

• #paywall / #loginwall or better yet don't put stuff on the internet if you don't want others to find and/or use it!!!