http://g.co/kgs/2SuJZW
Director SEO & Audience Development
@tonline
Ströer Digital Publishing GmbH
| Github | https://github.com/VorticonCmdr |
| Bluesky | https://bsky.app/profile/valentin.app |
As part of a time-limited trial, yesterday we removed the www.bbc.co.uk & www.bbc.com robots.txt "block" on *one* GenAI crawler.
Here's a graph of total daily requests from that 1 GenAI crawler by content-type. You'll note, they've retrieved ~460k web pages in ~16 hours, so a mean of ~28,750 web pages per hour. They'll likely pull ~690k pages today, ~32GB egress.
Whilst this is a drop in the ocean versus our other traffic, I can easily see how this could sink a smaller website.
#BILD beklagt sich mal wieder über die Öffentlich Rechtlichen: “Journalistische Distanz? Fehlanzeige!”
Springer-Chef Mathias Döpfner vor der Bundestagswahl: “Kann man noch mehr für die FDP machen? Die sollten 16 Prozent mindestens kriegen.”
this song came out abt 12 hours ago and it’s the only thing i’ve been able to listen to todaydefinitely don’t expect other vids like this but i needed to get...
"The articles will remain in the index for the regular 30-day period." https://developers.google.com/search/docs/crawling-indexing/sitemaps/news-sitemap#:~:text=The%20articles%20will%20remain%20in%20the%20index%20for%20the%20regular%2030%2Dday%20period.
Is this still valid? Which index does the documentation refer to? @johnmu
Neil Craig
Amanda
Carsten Werner
Rowan Merewood