The internet has changed a lot in the past 25 years.

[…] crawling operators have resorted to using a shady network of so-called “residential proxies” — companies that sell access to people’s own home or mobile connections, to hide their data extraction among legitimate browsing traffic. […].

A perspective on protecting open internet infrastructure from the threat of non-human traffic https://diff.wikimedia.org/2026/03/26/quo-vadis-crawlers-progress-and-whats-next-on-safeguarding-our-infrastructure/

#Wikipedia #wikipedia25

Quo Vadis, Crawlers? Progress and what’s next on safeguarding our infrastructure

One year ago, the Wikimedia Foundation reported a significant increase in bot traffic to the Wikimedia projects, largely coming from crawlers who extract content to train generative AI systems. We …

Diff