Can AI bots read my site?

What does this checker look at?

It reads your site's robots.txt the way Google and RFC 9309 specify and decides, for each of 17 bots, whether your home page is allowed, blocked, or partly blocked (the home page is open, but the bot's own rules close other paths). For every bot that isn't simply allowed, it quotes the lines that decide it. It's the same robots.txt check the free website audit runs, and the same 17 bots as our study of how many websites block AI bots.

It then requests your home page three times — as a normal browser, with GPTBot's name and with OAI-SearchBot's name — to see whether your server answers them the same way.

Which bots matter most?

• AI search bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Applebot, Meta AI's search, plus Googlebot and Bingbot) build the indexes assistants use to find and cite pages. Blocking one keeps your site out of that assistant's answers. Most businesses should allow all of them.
• Assistant fetchers (ChatGPT-User, Perplexity-User, Claude-User, meta-externalfetcher) open a page when someone asks an assistant to. OpenAI, Perplexity and Meta say robots.txt may not apply to theirs.
• Training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot) collect pages to train models. Blocking them is a licensing choice; it doesn't remove you from AI search.

The difference is explained in OAI-SearchBot vs GPTBot, and every bot with rules you can copy is in robots.txt for AI crawlers.

What can't it see?

robots.txt is a request, not a lock, and it isn't the only thing that can turn a bot away. A firewall or CDN — such as Cloudflare's AI bot blocking — can block a bot that robots.txt allows, and it can treat the real bot (checked by its network address) differently from our request that only uses the bot's name. So an “Allowed” here means your robots.txt lets the bot in, not that every request will get through. Your CDN's logs show what the real bots received.

To see what AI assistants actually say about your business, and what keeps your pages out of Google, run the full free audit.