Mastodawn

Related to my WAF header question the other day, AI crawling is also ruining RSS:

https://www.reddit.com/r/rss/comments/1t0jqis/many_sites_are_blocking_request_to_rss_links/

😔

#WebArchiving #WebCrawling #WebAPI #HTTPasAPI #API #RSS #ATOM #FederatedNews

FYI: OpenAI tripled its web crawl after GPT-5 - but ChatGPT users may be declining: New log file analysis of 7 billion OpenAI bot events reveals a 3.5x surge in OAI-SearchBot activity after GPT-5, while ChatGPT user-driven events dropped 28%. https://ppc.land/openai-tripled-its-web-crawl-after-gpt-5-but-chatgpt-users-may-be-declining/ #OpenAI #GPT5 #ChatGPT #AItrends #webcrawling

OpenAI tripled its web crawl after GPT-5 - but ChatGPT users may be declining

New log file analysis of 7 billion OpenAI bot events reveals a 3.5x surge in OAI-SearchBot activity after GPT-5, while ChatGPT user-driven events dropped 28%.

PPC Land

PPC Land Mar 26

FYI: Google-Agent joins the crawler list as AI browsing gets an official identity: Google on March 20 added Google-Agent to its user-triggered fetchers list, formalizing a new user agent for AI systems like Project Mariner that navigate the web on behalf of users. https://ppc.land/google-agent-joins-the-crawler-list-as-ai-browsing-gets-an-official-identity/ #GoogleAgent #AIBrowsing #UserAgent #WebCrawling #ProjectMariner

Google-Agent joins the crawler list as AI browsing gets an official identity

Google on March 20 added Google-Agent to its user-triggered fetchers list, formalizing a new user agent for AI systems like Project Mariner that navigate the web on behalf of users.

PPC Land

PPC Land Mar 15

FYI: Googlebot is not a program - Google engineers finally explain what it really is: Google engineers reveal Googlebot is a misnomer for a central SaaS crawling platform serving dozens of products, with a 15 MB default file size limit and geo-crawling constraints. https://ppc.land/googlebot-is-not-a-program-google-engineers-finally-explain-what-it-really-is/ #Googlebot #SEO #WebCrawling #DigitalMarketing #SaaS

Googlebot is not a program - Google engineers finally explain what it really is

Google engineers reveal Googlebot is a misnomer for a central SaaS crawling platform serving dozens of products, with a 15 MB default file size limit and geo-crawling constraints.

PPC Land

PPC Land Mar 12

Googlebot is not a program - Google engineers finally explain what it really is: Google engineers reveal Googlebot is a misnomer for a central SaaS crawling platform serving dozens of products, with a 15 MB default file size limit and geo-crawling constraints. https://ppc.land/googlebot-is-not-a-program-google-engineers-finally-explain-what-it-really-is/ #Googlebot #SEO #WebCrawling #SaaS #DigitalMarketing

Googlebot is not a program - Google engineers finally explain what it really is

Google engineers reveal Googlebot is a misnomer for a central SaaS crawling platform serving dozens of products, with a 15 MB default file size limit and geo-crawling constraints.

PPC Land

PPC Land Mar 11

FYI: Google's secret crawl logic, finally explained in one page: Google published a new web crawling overview on March 3, 2026, detailing how Googlebot discovers, renders, and manages site access across 30+ years of web indexing. https://ppc.land/googles-secret-crawl-logic-finally-explained-in-one-page/ #Google #SEO #WebCrawling #Googlebot #DigitalMarketing

Google's secret crawl logic, finally explained in one page

Google published a new web crawling overview on March 3, 2026, detailing how Googlebot discovers, renders, and manages site access across 30+ years of web indexing.

PPC Land

Localize The Docs Mar 9

🎉 scrapy-docs-l10n is published!

🚀 Preview:

https://projects.localizethedocs.org/scrapy-docs-l10n

🌐 Crowdin:

https://localizethedocs.crowdin.com/scrapy-docs-l10n

🐙 GitHub:

https://github.com/localizethedocs/scrapy-docs-l10n

#Crowdin #GitHub #Sphinx #Python #Scrapy #WebCrawling #WebScraping

Redirecting...

PPC Land Mar 9

ICYMI: Google's secret crawl logic, finally explained in one page: Google published a new web crawling overview on March 3, 2026, detailing how Googlebot discovers, renders, and manages site access across 30+ years of web indexing. https://ppc.land/googles-secret-crawl-logic-finally-explained-in-one-page/ #Google #SEO #WebCrawling #Googlebot #DigitalMarketing

Google's secret crawl logic, finally explained in one page

Google published a new web crawling overview on March 3, 2026, detailing how Googlebot discovers, renders, and manages site access across 30+ years of web indexing.

PPC Land

PPC Land Mar 8

Google's secret crawl logic, finally explained in one page: Google published a new web crawling overview on March 3, 2026, detailing how Googlebot discovers, renders, and manages site access across 30+ years of web indexing. https://ppc.land/googles-secret-crawl-logic-finally-explained-in-one-page/ #Google #SEO #WebCrawling #Googlebot #DigitalMarketing

Google's secret crawl logic, finally explained in one page

Google published a new web crawling overview on March 3, 2026, detailing how Googlebot discovers, renders, and manages site access across 30+ years of web indexing.

PPC Land

AI Daily Post Feb 26

Smart TVs are now running Bright SDK to silently crawl the web for AI training, using residential proxies to bypass Google policy. The move sparks a compliance backlash and raises privacy concerns for home devices. How will regulators respond, and what does this mean for open‑source AI? Dive into the details. #SmartTV #BrightSDK #WebCrawling #DeviceCompliance

🔗 https://aidailypost.com/news/smart-tvs-using-bright-sdk-crawl-web-ai-amid-compliance-backlash