#Development #Obituaries
Farewell to robots.txt (1994-2025) · “You were too good for this world.” https://ilo.im/167q2b

_____
#SearchEngine #InternetArchive #AI #Content #Website #RobotsTxt #RFC9309 #WebDev #Backend

I’ve made a little something, so I thought I'd share.

Gort is a robots.txt parser and evaluator. It implements RFC 9309.

More details in the ReadMe: https://github.com/pointlessone/gort

#Ruby #rubygem #release #robotstxt #robots_txt #rfc9309

Setting up /robots.txt, not because it helps, but because being crabby in compliance with an RFC is satisfying.

Who has some unsavory ones besides ChatGPT and Twitterbot?

https://rossabaker.com/configs/website/webcrawlers/

#RFC9309 #RobotsTxt