Snuck in a quick, but important https://probes.dev feature and added the User-Agent header to be a good web citizen.

`User-Agent: probes (+https://probes.dev)`

#HTTP #UserAgent #Monitoring

User-Agent情報についての分析から分かること
https://qiita.com/melkruri/items/3dd80adf6b6206e54fba?utm_campaign=popular_items&utm_medium=feed&utm_source=popular_items

#qiita #HTTP #userAgent #honeypot

So, with Google announcing "Search is going full-AI, we won't be sending traffic to the original sites any more", someone else pointed out that this eradication of the traditional search-engine compact - we let you crawl our sites to create your index, and you send visitors to our sites when relevant - means that we can, and should, block all of Google's crawlers now. If they're going to just take, take, take and give nothing back, why let them access your content at all?

But this is cute. Besides the fact that Google documents that some of their crawlers ignore robots.txt, there's this bit of fun. On this page (https://developers.google.com/crawling/docs/robots-txt/create-robots-txt), they link to "the Google list of user agents" (https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers).

However, that links to 3 separate pages of them, and *each of those pages explicitly states that is not comprehensive, but only the ones they commonly get questions about*. And of course, none of the "User-triggered fetchers" obey robots.txt, along with some others.

So Google isn't even reporting the full list of user-agents that can be used to stop their crawling.

That is some bullshit.

#Google #crawler #RobotsTxt #UserAgent #bullshit #antisocial #web #search #WebSearch #LLM #AI

Die Bahn und Pünktlichkeit? Kompliziert. Die Bahn und Linux? Noch komplizierter. 🙃

Zum Artikel: https://heise.de/-11300742?wt_mc=sm.red.ho.mastodon.mastodon.md_beitraege.md_beitraege&utm_source=mastodon

#deutschebahn #linux #webentwicklung #useragent #digitaleausgrenzung