https://en.wikipedia.org/wiki/Colossus:_The_Forbin_Project #UserAgents #RabbitHole #HackerNews #ngated
#Business #Reports
Anthropic details how Claude crawls sites Β· How to block the three separate user agents https://ilo.im/16ax7y
_____
#AI #Claude #Crawlers #UserAgents #RobotsTxt #Content #Website #WebDev #Frontend #Backend
Of the bigger browsers, I think Safari is probably the closest to that right now, for all its flaws. It may be that Waterfox or some other Firefox fork with the AI garbage ripped out of it is better, I haven't delved into that.
It would be nice if somebody made a user agent for the web.
You know, software that actually works on behalf, and in the interests, of the user, rather than the maker.
Chrome has been adware for years now. Edge was actually pretty good while it was a fairly vanilla Chromium fork, but it seems MS is intent on stuffing Copilot into it too.
https://www.w3.org/WAI/UA/work/wiki/Definition_of_User_Agent