How many websites block AI search and AI training bots?

One in ten live websites we checked (10.0% of 2,117) blocks at least one AI bot in its robots.txt file, and AI training bots are blocked far more often than AI search bots: GPTBot is blocked 2.9 times as often as OAI-SearchBot, the bot behind ChatGPT search.

Most site owners who block AI do it on purpose and by name, and many of them block AI training while leaving the door open to AI search. Blocks caused by a blanket rule are rare among live business sites but more common across random domains. And one network of 530 copy-paste sites could distort any study that doesn't look for it.

A second, independent sample of 1,497 random domains from the Tranco top-1M list gives almost the same top-line result: 9.7% block at least one AI bot, and 7.9% block GPTBot. Want the answer for your own site? The free AI bot checker reads your robots.txt for the same 17 bots.

Which bots did we check?

robots.txt is a plain text file at the root of a website (for example example.com/robots.txt). It tells crawlers which pages they may read. A crawler that respects it looks for a group that names it; if none does, it follows the User-agent: * group. AI companies run different bots for different jobs, and blocking one doesn't block the others.

For the headline comparisons we used three AI search and assistant bots (OAI-SearchBot, ChatGPT-User, PerplexityBot) and four AI training bots (GPTBot, ClaudeBot, Google-Extended, CCBot).

Which AI bots do websites block most?

Every AI training bot is blocked more often than any AI search bot. CCBot and GPTBot lead; Google's and Bing's own crawlers are almost never blocked.

Partial blocks, where a bot is kept off some pages but not the home page, were rare: 0.2% of sites or fewer for every bot.

Do sites that block AI training also block AI search?

Mostly not. 67.3% of the sites that block GPTBot still let OAI-SearchBot in. Across all four training bots, the largest group blocks training only.

Which kinds of websites block AI bots?

Publishers block AI far more than anyone else. News and media sites have the clearest reason to protect their content and the most leverage to license it. Startups, local businesses and online stores, which usually want to be found and recommended, rarely block AI at all.

Well-known sites block AI training about 3.0 times as often as lesser-known ones (18.5% against 6.2%). Type and source overlap in our sample — every publisher and store came from the Tranco list — so treat these as descriptive differences, not causes.

Does AI blocking differ by region?

Does the website platform make a difference?

The platforms differ by site mix as well as by default settings, so these are descriptive differences.

Do sites block AI on purpose or by a blanket rule?

Among live business sites, almost every block names the bot: 97.3% of GPTBot blocks come from a rule that names GPTBot. Blanket blocks, where User-agent: * with Disallow: / shuts out every bot that isn't named, are much more common across random domains — and they often shut out Google too: 30.7% of the random domains that block AI search also block Googlebot from the whole site.

How did one site network change the numbers?

We found 530 sites with the same robots.txt. It lets Google, Bing and Apple in and shuts out every AI bot. They look like one operator's network, so we excluded them. Had we counted them, the share of random domains blocking ChatGPT search would have jumped from 3.3% to 20.7%.

What do the numbers mean?

Training opt-outs are the main reason sites block AI. Site owners treat training and search differently, which fits how OpenAI, Anthropic, Google, Apple and Meta now separate training crawlers from search crawlers: owners can refuse one and accept the other.

Some AI search blocking is probably not about AI. A site that blocks ChatGPT search only through a catch-all rule may not know it. Among random domains, 37.3% of AI-search blocking comes only from User-agent: *, without naming any AI bot.

Duplicate sites can skew a study. Studies that sample top-list domains without checking for copy-paste networks can overstate AI search blocking several times over.

How can I check my own site?

1. Free AI bot checker. Type your address: it reads your robots.txt for the same 17 bots and shows, for each one, allowed, blocked or partly blocked, the line that decides it, and whether your home page answers GPTBot and OAI-SearchBot like a normal browser.
2. Free website audit. The same robots.txt check plus everything around it: what AI assistants say about you, indexing, phone speed and your business facts.
3. By hand. Open yoursite.com/robots.txt and look for User-agent: lines that name AI bots, each followed by Disallow: /, and for a User-agent: * group with Disallow: /, which blocks every bot without its own group, AI search included.

To opt out of AI training but stay visible in AI search, name the training bots and leave the search bots alone:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Also check your firewall or CDN settings, which can block bots that robots.txt allows — see how Cloudflare's AI bot blocking works. For every bot and the rules to copy, read robots.txt for AI crawlers and OAI-SearchBot vs GPTBot.

Questions about this study

What is the difference between AI training bots and AI search bots?

Training bots such as GPTBot and ClaudeBot collect pages to train AI models. Search bots such as OAI-SearchBot and PerplexityBot build the index an assistant uses to find and cite pages when it answers. In our sample, 7.1% of sites block GPTBot and 2.4% block OAI-SearchBot.

If I block GPTBot, will my site disappear from ChatGPT?

No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, which you control separately, and 67.3% of the sites in our sample that block GPTBot still allow OAI-SearchBot.

Does robots.txt actually stop AI bots?

It's a request, not a lock. The major AI companies say their training and search crawlers follow it. OpenAI, Perplexity and Meta say it may not apply to the fetchers that open a page because a user asked for it. To enforce a block you need firewall or CDN rules, which this study doesn't measure.

Why are these numbers lower than other studies of AI bot blocking?

Many studies look only at the most popular news and media sites, where blocking is common; in our sample, 22.7% of publishers block at least one training bot. Our main sample is mostly startups, stores and local businesses. We also excluded a network of 530 identical sites, which would have raised the share blocking ChatGPT search from 2.4% to 10.9% in our main sample.