Does Cloudflare block AI bots? How to check your site and what to change

If your site sits behind Cloudflare, its AI settings decide what ChatGPT, Claude and Perplexity can read before your own robots.txt is even consulted. The settings have changed several times since 2024, and many owners turned one on without knowing which bots it covers. This guide explains what each setting does, as documented by Cloudflare in October 2026, and how to check your own site.

What AI bot settings does Cloudflare have?

There are four places where Cloudflare can affect AI crawlers. Any one of them can block a bot that your own robots.txt allows.

1. AI bot policies (the current control)

Under Security Settings → Configure AI bot policies, Cloudflare sorts AI traffic into three categories and lets you pick an action for each:

• Search: crawlers that collect or index your content to answer questions about it later. This is the kind of crawler OAI-SearchBot, Claude-SearchBot and PerplexityBot are; AI Crawl Control shows how Cloudflare classifies each one.
• Agent: automated activity in real time on a person's behalf, such as a chat assistant fetching a page or a browser agent.
• Training: crawlers that take content to train or fine-tune models. Cloudflare puts mixed-purpose crawlers, used for both training and search, in this group.

For each category the choices are Block (on all pages), Block on pages with ads (Cloudflare detects which pages show ads) or Allow. Blocks apply to verified bots in the category and to unverified bots that behave the same way.

2. The legacy "Block AI bots" switch

The older one-click setting, also under Security Settings, blocks verified bots that Cloudflare classifies as crawling for AI training, plus similar unverified bots. Cloudflare says it is deprecated as of September 15, 2026. If it's still on for your zone, check what your current AI bot policies show.

3. Managed robots.txt (Bot Preference Sync)

If you set Training to disallow and turn on Enable Bot Preference Sync, Cloudflare writes AI rules into your robots.txt. If you already have a file, Cloudflare puts its rules before your content; if you don't, it creates one. The rules include a Disallow: / for training crawlers such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and meta-externalagent, plus a line such as Content-signal: search=yes, ai-train=no. This is why the robots.txt in your code can differ from the one bots receive.

4. AI Crawl Control

AI Crawl Control (formerly AI Audit) shows which AI services visit your site, how often, and which ones ignore your robots.txt, and lets you allow or block individual crawlers. Cloudflare says it works on all plans. It's the best place to see what is actually happening.

Does Cloudflare block AI crawlers by default?

For new domains, partly. Cloudflare's documentation says that from September 15, 2026, new domains get these defaults:

• Training bots: blocked on pages with ads.
• Agent bots: blocked on pages with ads.
• Search bots: allowed.

Two details matter. First, mixed-purpose crawlers that do both search and training are blocked by any setting that blocks training. Second, the defaults are described for new domains; Cloudflare gave existing customers an opt-out window before that date, and it doesn't spell out every case for older zones. Earlier, on July 1, 2025, Cloudflare announced it was changing its default to block AI crawlers unless they pay creators. If your domain was added around then, look at what was switched on.

So the honest answer for any one site is: it depends on the zone, and you need to look.

Should you block AI training bots on Cloudflare?

It's a business choice, and for most businesses it can be made separately from search. Blocking training crawlers doesn't remove you from ChatGPT, Claude or Perplexity answers, because those use separate search bots. What you must not do by accident is block the Search category: that takes your pages out of the answers buyers read.

A sensible setup for a store, service business or SaaS product:

• Search: Allow.
• Agent: Allow, unless agents are causing load or abuse. Agents are how assistants read a page a customer pointed them to, and how shopping agents check a product.
• Training: your choice. Blocking it is common and costs no search visibility.

Publishers whose income depends on page views may reasonably choose differently. Whatever you choose, write it down with the date.

How do you check your Cloudflare dashboard?

1. Log in to Cloudflare and open the domain.
2. Go to Security Settings and filter by bot traffic.
3. Open Configure AI bot policies and note the action for Search, Agent and Training.
4. Check whether the legacy Block AI bots switch is on.
5. See whether Bot Preference Sync (managed robots.txt) is enabled.
6. Open AI Crawl Control and look at which crawlers were allowed or blocked in the last days. A search bot with only blocked requests is your red flag.
7. Check custom WAF rules and rate limits for anything that matches bot user agents.

How do you check from outside, without the dashboard?

If you don't manage the Cloudflare account yourself, you can still see the signs:

1. Read the live robots.txt. Open https://yourwebsite.com/robots.txt. Rules you didn't write, or a Content-signal line, mean a managed robots.txt is on.
2. Request a page as the bot. From a terminal: curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot)" https://yourwebsite.com/. A 403 or a challenge page where a normal browser gets 200 points to a block by user agent. The free AI bot checker runs this request for you, as GPTBot and as OAI-SearchBot, next to a normal browser.

Know the limit of the second test. Cloudflare verifies well-known bots by where the request comes from, not only by the name it gives. A request from your laptop that only claims to be OAI-SearchBot can be treated differently from the real crawler: it can be blocked as a fake when the real one is allowed, or allowed when the real one would be blocked. A clean result from outside is a good sign, not proof. AI Crawl Control shows what the real bots received.

What did we find on our own site?

dottpace.com runs on Cloudflare. On October 19, 2026, we requested our home page as OAI-SearchBot, GPTBot, PerplexityBot, ClaudeBot, ChatGPT-User, Bingbot and Googlebot; every request returned 200, and our robots.txt allows all crawlers. We don't block AI training on our own site because our pages are about our product, and we want assistants to describe it correctly. That is a choice for our kind of site, not a recommendation for every site.

What should you do if you find a block?

• The Search category is blocked: set it to Allow, unless you meant to leave AI search.
• A training block catches a search bot: check whether the bot is mixed-purpose. If your goal is visibility, allow the specific search crawler in AI Crawl Control.
• Managed robots.txt surprises you: read the merged file, then either turn off Bot Preference Sync or make sure its rules match your intent.
• Then wait a day and recheck. OpenAI and Perplexity say it can take about 24 hours for their systems to pick up robots.txt changes.

For the full list of bot names and copy-paste rules, read robots.txt for AI crawlers. Dottpace can also connect to Cloudflare to record changes to these settings next to your traffic.

Sources

• Cloudflare, Block AI bots and AI bot policies (last updated July 1, 2026).
• Cloudflare, Managed robots.txt.
• Cloudflare, AI Crawl Control.
• Cloudflare blog, Content Independence Day (July 1, 2025).
• OpenAI, Overview of OpenAI Crawlers; Perplexity, Perplexity crawlers.