Sigh. It's go time, folks: https://github.com/twisted/twisted/issues/12676
@glyph have you considered adding a .robots.txt file at the project root, following the same syntax as the one that would be found at the root of a webserver, with content saying bot are not to parse anything on any path?
Kind of the same idea as people using agents md files to say no to bots but using a convention that was made for telling no to bots.
