OAI-SearchBot decides whether your pages can appear in ChatGPT search answers, GPTBot collects content that may be used to train OpenAI's models, and ChatGPT-User fetches a page only when a person asks ChatGPT to. If you want to be found in ChatGPT but keep your content out of training, allow OAI-SearchBot and block GPTBot.
OpenAI runs three separate crawlers, and each one has its own line in robots.txt. They are often lumped together as "the ChatGPT bot", which is how sites end up blocking the one that matters for visibility while trying to block the one that only matters for training. This guide explains what each bot does, quotes OpenAI's own documentation, and gives you the exact rules to copy.
What does each OpenAI bot do?
OpenAI documents all three on its crawlers and user agents page. In short:
| Bot | What it is for | Follows robots.txt? | If you block it |
|---|---|---|---|
OAI-SearchBot | Finds and indexes pages so they can be shown and linked in ChatGPT search answers. | Yes | Your pages won't be shown in ChatGPT search answers (they can still appear as plain navigational links). |
GPTBot | Collects content that may be used to train OpenAI's generative AI foundation models. | Yes | Your future content is signaled as not for model training. It does not remove you from ChatGPT search. |
ChatGPT-User | Visits a page when a ChatGPT user (or a Custom GPT or GPT Action) asks for it. Not an automatic crawler. | Not always: OpenAI says robots.txt rules "may not apply" to user-initiated visits. | It may still reach pages a user asks for. It does not decide whether you appear in search. |
The key sentence in OpenAI's documentation is about search: sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers. GPTBot is described separately: disallowing it "indicates a site's content should not be used in training generative AI foundation models." The two settings are independent, which is the whole point. You can say yes to one and no to the other.
OpenAI also lists a fourth agent, OAI-AdsBot, which only visits landing pages that were submitted as ads in ChatGPT to check them. If you don't advertise in ChatGPT, you won't see it.
What happens in ChatGPT if you block OAI-SearchBot?
When someone asks ChatGPT a question that needs current information, ChatGPT searches the web and cites the pages it used. Your pages can only be among those cited if OAI-SearchBot is allowed to read them. Block it, and ChatGPT won't show your pages in those answers, even if it knows your brand name from elsewhere.
This matters most for questions where buyers compare options: "best plumber in Austin", "Shopify apps for subscriptions", "project management tool for small teams". If a competitor is readable and you are not, the answer can only quote them.
Changes are not instant. OpenAI says it can take about 24 hours after you update robots.txt for its systems to adjust.
Does blocking GPTBot remove you from ChatGPT?
No. This is the most common misunderstanding. GPTBot is the training crawler. Blocking it tells OpenAI not to use your future content for training foundation models, and nothing in OpenAI's documentation links it to search visibility. Search inclusion is controlled by OAI-SearchBot alone.
The reverse is also true: allowing GPTBot does not make ChatGPT more likely to recommend you. If you have no reason to share your content for training, blocking GPTBot costs you nothing in ChatGPT search.
"I only want to stop training": what should robots.txt say?
Add these lines to the robots.txt file at the root of your site (for example https://yourwebsite.com/robots.txt). Keep any rules you already have for other bots.
# OpenAI: allow ChatGPT search, block model training
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /You don't need a rule for ChatGPT-User. It acts for a person who asked for your page, and blocking it would only stop ChatGPT from reading a page a user pointed it to.
If you want to be fully visible
If your robots.txt has no rule that names OpenAI's bots and no rule for all bots (User-agent: *) that disallows your pages, all three can already read your site. You don't need to add anything. Adding an explicit Allow for OAI-SearchBot is harmless and makes your intent clear to whoever edits the file next.
If you want to block all three
User-agent: OAI-SearchBot
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /Know the trade-off: this removes your pages from ChatGPT search answers. Because ChatGPT-User acts on a user's request, OpenAI notes that robots.txt may not apply to it in every case.
Why do so many sites get this wrong?
Three patterns cause most mistakes:
- A copied "block AI" list. Lists of AI user agents passed around in 2023 and 2024 often put OAI-SearchBot next to GPTBot under one
Disallow: /. The intent was to stop training; the effect is also to leave ChatGPT search. - A rule for all bots. A line such as
User-agent: *followed byDisallow: /blocks every bot that has no rule of its own, including OAI-SearchBot. This often comes from a staging site that went live with its "keep out" file. - A setting outside robots.txt. Your host or CDN can block bots before they ever read robots.txt. Cloudflare, for example, has settings that block AI crawlers by category. Read how to check whether Cloudflare blocks AI bots.
How common is it? In our October 2026 study of 2,117 live websites, 7.1% block GPTBot and 2.4% block OAI-SearchBot. Most GPTBot blockers get it right: 67.3% still let OAI-SearchBot in, while 32.7% shut out ChatGPT search too. The blanket rule is the bigger trap on less-tended sites: among 1,497 random top-1M domains, 37.3% of all AI-search blocking came only from User-agent: *, without naming any AI bot.
A quick way to see the problem: open your robots.txt and search for "OAI-SearchBot". If it sits under a Disallow: /, ChatGPT search can't use your pages.
How do you check your site in a minute?
- Type your address into the free AI bot checker. It shows OAI-SearchBot, GPTBot and ChatGPT-User side by side, with the line that decides each — or check by hand:
- Open
https://yourwebsite.com/robots.txtin your browser. - Find any group that names
OAI-SearchBot. If it hasDisallow: /, ChatGPT search is blocked. - If there's no group for OAI-SearchBot, look at the
User-agent: *group. Its rules apply to OAI-SearchBot too. - Check your CDN or firewall settings for AI bot blocking, since those don't show up in robots.txt.
Robots.txt only tells you what a bot is asked to do. To see what a bot actually gets, you need a live request as that bot. That is part of what Dottpace's free audit runs.
What about the other AI search bots?
Other AI companies use the same split between search and training, with their own names:
- Anthropic:
Claude-SearchBotindexes content for search in Claude,ClaudeBotis for training, andClaude-Userfetches pages for a user. - Perplexity:
PerplexityBotsurfaces and links sites in Perplexity's search results and, per Perplexity, is not used for training foundation models.Perplexity-Userfetches pages for a user. - Google:
Google-Extendedcontrols use of your content for Gemini training and grounding, and Google says it doesn't affect inclusion or ranking in Google Search.
The full table, with copy-paste policies for each, is in robots.txt for AI crawlers.
Sources
- OpenAI, Overview of OpenAI Crawlers (OAI-SearchBot, GPTBot, ChatGPT-User, OAI-AdsBot; the 24-hour note).
- Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity, Perplexity crawlers.
- Google, Google's common crawlers (Google-Extended).
If you're also wondering why ChatGPT doesn't mention your business even though bots are allowed, read how to get your business recommended by ChatGPT, or see how Dottpace tracks what AI assistants say about you every week.