Robots.txt for AI crawlers: every bot and what to allow

To control AI crawlers with robots.txt, name each bot's user agent and allow or disallow it; the important choice is between search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot), which decide whether AI assistants can show and link your pages, and training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended), which only affect model training. Most businesses should allow the search bots and decide separately about training.

Every major AI company now runs more than one bot, and each bot answers to its own line in robots.txt. Blocking the wrong one can quietly remove your site from ChatGPT, Claude or Perplexity answers. This page lists the bots that matter as of October 2026, what each company says it does, and three complete policies you can paste.

Which AI crawlers exist, and what does each one do?

The table groups each bot by the job its owner documents for it. "Search" bots build the index an assistant uses to cite pages. "Training" tokens control whether content is used to train models. "User" agents fetch a page when a person asks the assistant to, and their owners say robots.txt may not apply to them.

Sources: each company's crawler documentation, linked at the end of this page. Checked October 19, 2026. “Sites that block it”: the share of 2,117 live websites whose robots.txt blocks the bot from the whole site, from Dottpace's October 2026 study.
Robots.txt name	Owner	Job	What blocking it does	Sites that block it
`OAI-SearchBot`	OpenAI	Search	Your pages aren't shown in ChatGPT search answers.	2.4%
`GPTBot`	OpenAI	Training	Signals your content shouldn't train OpenAI's foundation models.	7.1%
`ChatGPT-User`	OpenAI	User	May stop ChatGPT reading a page a user asks for; robots.txt "may not apply".	4.1%
`Claude-SearchBot`	Anthropic	Search	Stops your content being indexed for search in Claude.	2.0%
`ClaudeBot`	Anthropic	Training	Signals future content should be excluded from Anthropic's training data.	6.0%
`Claude-User`	Anthropic	User	Stops Claude retrieving your pages in response to a user's question.	1.8%
`PerplexityBot`	Perplexity	Search	Your site isn't surfaced and linked in Perplexity's search results.	4.3%
`Perplexity-User`	Perplexity	User	Generally ignores robots.txt, because a user requested the page.	1.9%
`Google-Extended`	Google	Training and grounding	Limits use in Gemini models. No effect on Google Search inclusion or ranking.	6.0%
`Applebot-Extended`	Apple	Training	Limits use for Apple's AI models. Pages can still appear in Apple's search features.	5.4%
`Applebot`	Apple	Search (and training)	Removes you from Spotlight, Siri and Safari search features.	1.4%
`meta-externalagent`	Meta	Training and other uses	Stops Meta's crawler that collects content for uses such as training AI models.	5.4%
`CCBot`	Common Crawl	Open web archive	Keeps new pages out of Common Crawl, a public dataset many AI models have been trained on.	7.8%

Training bots are blocked far more often than search bots. In our study of 2,117 live websites, 10.0% block at least one AI bot, GPTBot is blocked 2.9 times as often as OAI-SearchBot, and publishers block the most: 22.7% of them block at least one training bot, against 3.2% of SaaS sites.

Two names in the table never visit your site. Google-Extended and Applebot-Extended are control tokens: Google says Google-Extended "doesn't have a separate HTTP request user agent string", and Apple says Applebot-Extended "does not crawl webpages". They only tell the company how it may use what its normal crawler already fetched.

Should you block AI crawlers?

It depends on what you sell. A store, a service business or a SaaS product usually wants every chance to be named when a buyer asks an assistant for a recommendation, so blocking AI search bots works against it. A publisher whose articles are the product may reasonably refuse training, and some decide to refuse AI search too, accepting fewer citations.

Ask two separate questions:

Do I want AI assistants to show and link my pages? If yes, allow the search bots.
Do I want my content used to train AI models? Your answer here doesn't change the first one.

What are three ready-made robots.txt policies?

Each policy below is complete for the AI bots it names. Keep the rest of your file (rules for other crawlers and your Sitemap: line) as it is.

Policy 1: allow everything

If your file has no rules that block these bots, you're already here. Nothing to add. Effect: you can be cited in ChatGPT, Claude, Perplexity, Gemini and Google's AI features, and your content may be used for training.

Policy 2: allow AI search, block training

# AI search: allowed (ChatGPT, Claude and Perplexity can show and link your pages)
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# Training: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /

Effect: you stay eligible for citations in ChatGPT search, Claude and Perplexity, Google Search and its AI Overviews are unchanged (Google-Extended is not a Search control), and you signal "no training" to the companies that honor these tokens. Several User-agent lines can share one group of rules; that is part of the robots.txt standard.

Policy 3: block all AI use

User-agent: OAI-SearchBot
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /

Effect: your pages won't be shown in ChatGPT search, Claude search or Perplexity results. Google Search, including AI Overviews, still uses Googlebot, which this policy leaves alone; to stay out of Google's AI features you'd have to use snippet controls or leave Google Search entirely. User-initiated agents may still fetch pages a person asks for.

What does Google-Extended actually control?

Google-Extended is not a way out of AI Overviews. Google says it controls whether content its crawlers fetch "may be used for training future generations of Gemini models", and that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". AI Overviews and AI Mode are part of Google Search and use the normal Googlebot crawl.

To limit what appears from a page in Google's results, including its AI features, Google points to the nosnippet, data-nosnippet, max-snippet and noindex controls. Those cost you visibility in regular results too, so use them on purpose.

Why isn't robots.txt the whole story?

Robots.txt is a request, not a lock. Google's own guide says it "is not a mechanism for keeping a web page out of Google" and that while respectable crawlers obey it, "other crawlers might not". Three things can override what your file says:

Your CDN or firewall. Cloudflare and other providers can block AI bots by category before they ever read robots.txt. Your file may say "allowed" while the bot gets an error. See how to check Cloudflare's AI bot settings.
A managed robots.txt. Some hosts add AI rules to your file for you. Cloudflare's managed robots.txt, for example, prepends Disallow rules for training crawlers to your own file. Always read the live file at /robots.txt, not the one in your code.
User-initiated agents. ChatGPT-User, Claude-User and Perplexity-User act for a person, and their owners say robots.txt may not apply.

How do you write the rules correctly?

The file must live at the root: https://yourwebsite.com/robots.txt. A file in a subfolder is ignored.
A bot follows the most specific group that names it. If GPTBot has its own group, the User-agent: * rules no longer apply to GPTBot.
A group that only says Allow: / is fine; it states intent and blocks nothing.
Don't block your CSS or JavaScript files. Google needs them to render your pages.
Changes take time to be read: OpenAI and Perplexity each say about 24 hours.
Keep a comment above each group explaining the decision, with a date. The next person to edit the file will thank you.

How do you check what your file does today?

Type your address into the free AI bot checker: it reads your robots.txt for all 17 bots and quotes the line that decides each one. Or open your live /robots.txt in a browser and look for each name in the table above.
For any bot without its own group, read the User-agent: * group: those rules apply to it.
Check your CDN or host for AI bot settings that block outside the file.
Run a live request as the bot to see the real response. Dottpace's free audit does this for each AI search bot, and Dottpace Health keeps checking after every release.

Sources

OpenAI, Overview of OpenAI Crawlers.
Anthropic, How site owners can block Anthropic's crawlers.
Perplexity, Perplexity crawlers.
Google, Google's common crawlers, AI features and your website and Introduction to robots.txt.
Apple, About Applebot.
Meta, Meta web crawlers.
Common Crawl, FAQ.
Cloudflare, Managed robots.txt.

For the OpenAI bots in detail, read OAI-SearchBot vs GPTBot vs ChatGPT-User.

Robots.txt for AI crawlers: every major bot, what it does, and rules you can copy