The Wikimedia Foundation, overseeing Wikipedia and other collaborative knowledge platforms, reported a significant 50% increase in bandwidth usage for Wikimedia Commons since January 2024. 📈 Contrary to initial assumptions, this spike isn’t attributed to human users but to automated scrapers aiming to gather data for AI model training. 🤖
Wikimedia Commons, a vast library of freely licensed multimedia, is facing unprecedented traffic from these bots. “Our systems are designed to handle human traffic surges during peak events, but the scale of bot activity is beyond our usual capacity,” the Foundation explained. This scenario not only risks service disruption but also escalates operational costs.
Interestingly, bots account for 65% of the most resource-heavy traffic, despite making up only 35% of total pageviews. This discrepancy arises because bots often access less popular content stored in the core data center, which is more costly to retrieve. “Human users typically focus on trending topics, whereas bots indiscriminately download large volumes of data, including obscure files,” Wikimedia noted.
The Foundation’s site reliability team is now dedicating substantial efforts to mitigate these crawlers’ impact, ensuring seamless access for genuine users. This situation highlights a broader challenge facing the open internet, as AI crawlers increasingly disregard traditional barriers like “robots.txt” files, prompting a tech arms race. Developers and companies, including Cloudflare with its AI Labyrinth tool, are innovating to deter these bots. Yet, the ongoing battle may push more content behind paywalls, altering the web’s open nature. 🌐
