Liz ProbertApr 2, 2025
How crawlers impact the operations of the Wikimedia projects https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/ #AI, #Crawlers, #Infrastructure, #KnowledgeAsAService, #KnowledgeContent, #Operations, #Scraping, #ScrapingBots, #Traffic, #WikimediaFoundation, #WikimediaProjects
How crawlers impact the operations of the Wikimedia projects

Since the beginning of 2024, the demand for the content created by the Wikimedia volunteer community – especially for the 144 million images, videos, and other files on Wikimedia Commons – has grow…

Diff
Carbon CarrotApr 2, 2025

Wikimedia Infrastructure is being mass-scraped for AI Usage — the content is free, the infrastructure is not. https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/ #AI, #Crawlers, #Infrastructure, #KnowledgeAsAService, #KnowledgeContent, #Operations, #Scraping, #ScrapingBots, #Traffic, #WikimediaFoundation, #WikimediaProjects

(original repost on lobsters: https://lobste.rs/s/autpsf/how_crawlers_impact_operations)

How crawlers impact the operations of the Wikimedia projects

Since the beginning of 2024, the demand for the content created by the Wikimedia volunteer community – especially for the 144 million images, videos, and other files on Wikimedia Commons – has grow…

Diff
Show threadkanmastodonsvenskaJul 30, 2022

@aktivismoEstasMiaLuo @Wraptile #QuoraFail yes indeed. Sometimes I find a site or two blocking the archival of their site.

I mean its bad for them, less publicity, so that Quora does this might mean they are #WelcomingDefeat or they have already #acknowledged their defeat internally and just waiting for the site to get less and less traffic over time.

Well blocking #Tor is 'immature'. #WikimediaProjects all allow Tor, though they never like people editing with any proxy.

#SpellingErrors