vercel.com
The Rise of the AI Crawler
Vercel examines traffic from crawlers used by AI services.
https://vercel.com/blog/the-rise-of-the-ai-crawler
About This Resource
Vercel examines traffic from crawlers used by AI services. The article discusses observed crawling behavior and its implications for website operators.
Saved Notes
Original notes saved with this link. They may describe an earlier version.
The rise of the AI crawler - Vercel
The article explores the emerging landscape of AI web crawlers, providing a detailed analysis of how different AI assistants like ChatGPT, Claude, and others crawl and interact with web content. Vercel and MERJ conducted an extensive study examining the crawling behaviors of various AI platforms, revealing significant insights into their web interaction patterns[1].
Key Findings
The research uncovered several critical observations about AI crawlers:
- Crawler Scale: AI crawlers are generating substantial web traffic, with GPTBot generating 569 million requests and Claude 370 million requests across Vercel's network, representing about 28% of Googlebot's volume[1].
- JavaScript Rendering Limitations: Most AI crawlers (OpenAI, Anthropic, Meta, ByteDance, Perplexity) do not render JavaScript, meaning they cannot process client-side rendered content[1].
- Content Prioritization: Different AI crawlers show unique content fetching preferences:
- ChatGPT prioritizes HTML content (57.70% of fetches)
- Claude focuses heavily on images (35.17% of fetches)
- Both spend significant time on JavaScript files despite not executing them[1]
- Crawling Inefficiencies: AI crawlers demonstrate notable inefficiencies:
- ChatGPT spends 34.82% of fetches on 404 pages
- Claude shows similar 404 error rates at 34.16%
- High rates of unnecessary redirects and failed page requests[1]
Implications and Recommendations
The study provides recommendations for:
- Site owners wanting to be crawled (prioritize server-side rendering)
- Site owners wanting to block AI crawlers
- AI users (understanding potential content limitations)
The research ultimately highlights that while AI crawlers are rapidly scaling, they still face significant challenges in web content discovery and processing[1].
Significance
This analysis represents an important early exploration of how AI technologies interact with web content, offering unprecedented insights into the emerging AI crawling ecosystem.
Citations: [1] https://vercel.com/blog/the-rise-of-the-ai-crawler