Reference

AI crawlers list: user agents by job

AI crawlers fall into three classes. Agent crawlers fetch pages live during a conversation, search crawlers build the index that picks candidates, and training crawlers collect data for future models. Blocking the wrong class can remove you from AI answers.

Run a free diagnosis

AI crawler user agents

These are the user agents our probe uses or tracks.

User agent	Operator	Class	Job
ChatGPT-User	OpenAI	Agent	Live fetch during a ChatGPT conversation
Claude-User	Anthropic	Agent	Live fetch during a Claude conversation
Perplexity-User	Perplexity	Agent	Live fetch for a Perplexity answer
OAI-SearchBot	OpenAI	Search	Index for ChatGPT search
Claude-SearchBot	Anthropic	Search	Index for Claude search
PerplexityBot	Perplexity	Search	Index for Perplexity
Bingbot	Microsoft	Search	Bing index, used upstream by ChatGPT search
Googlebot	Google	Search	Google index, also feeds Gemini and AI Overviews, renders JavaScript
GPTBot	OpenAI	Training	Training data collection
ClaudeBot	Anthropic	Training	Training data collection

robots.txt that keeps you in AI answers

Allow agent and search classes. Decide on training separately.

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

robots.txt is not the whole story

Many sites allow AI crawlers in robots.txt but block them at the CDN, bot manager or firewall. The agent then gets HTTP 403. Test with a real request, not just the file.

FAQ

Questions people ask

Should I block GPTBot?

Blocking GPTBot affects training data, not live ChatGPT answers. ChatGPT-User and OAI-SearchBot are separate user agents.

Which crawler does Gemini use?

Gemini relies on Googlebot, which renders JavaScript.

Keep reading