Two kinds of AI crawler
The split between training-style crawlers and user-side fetchers changes how I read a traffic spike.
Something small worth writing down after a morning of staring at crawler traffic.
Most of the "AI bot traffic" numbers I've seen collapse everything into one bucket. The split I've found more useful:
- Training-style crawlers —
ClaudeBot,GPTBot,PerplexityBot. Sweep pages in bulk. Return almost no referral traffic downstream. - User-side fetchers —
Claude-User,ChatGPT-User,OAI-SearchBot. Fire when a person asks their assistant about your site. Correlate with brand visibility inside AI answers.
OpenAI actually makes this distinction explicit in their crawler overview — GPTBot is for training, ChatGPT-User is user-initiated, OAI-SearchBot surfaces sites inside ChatGPT search. Same shape holds for Anthropic (ClaudeBot vs Claude-User).
Same origin, very different signal. When the training crawler jumps five-fold week-over-week but the user-side fetcher stays flat, that's vendor-initiated crawling — a re-crawl, an index refresh, expanded coverage. Not more people asking about you. Reading it as "we're getting popular in AI answers" would be wrong.
The other thing I've been watching: 404s to .well-known paths like agent-card.json and openid-configuration, and paths that look like MCP endpoints. Agents are probing for surfaces that don't exist yet. Whether we should serve them is a separate question — but at least knowing they're being probed is useful.
Might be wrong. Curious how this evolves as answer engines mature.