mvc1095 (they)1d ago
Quo Vadis, Crawlers? Progress and what’s next on safeguarding our infrastructure https://diff.wikimedia.org/2026/03/26/quo-vadis-crawlers-progress-and-whats-next-on-safeguarding-our-infrastructure/ #AI, #AIDataCrawlers, #Crawlers, #Infrastructure, #Knowledge, #KnowledgeAsAService, #Scraping, #ScrapingBots, #WebScraping, #WikimediaFoundation, #WikimediaProjects
Quo Vadis, Crawlers? Progress and what’s next on safeguarding our infrastructure

One year ago, the Wikimedia Foundation reported a significant increase in bot traffic to the Wikimedia projects, largely coming from crawlers who extract content to train generative AI systems. We …

Diff