Only like twenty years too late, but hey; The Internet Archive is going to stop honouring robots.txt https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/
Now if only the internet archive would let me fix the list of bugs I sent to the guy who runs it – it’s open source in much the same way as Android; you can get the source but good fuckin’ luck contributing meaningfully from outside
and also if only thirty years of internet history weren’t lost to time because people sniffed for particular user-agents