I've seen the same problem on more than one site: the company wanted to show up in ChatGPT, and its robots.txt was telling ChatGPT to go away. It's a two-minute fix, and it can make a huge difference, because AI crawlers take robots.txt really seriously. If the file says no, the big ones listen.

Usually nobody meant to hide from ChatGPT. They meant to keep their content out of AI training, which is a fair call. The catch is that "AI crawler" isn't one bot, and it's easy to block the one that gets you into the answers along with the one that trains the models.

So I wanted to know how often that happens. I checked the robots.txt files of the top 1,000 websites plus a random sample of 1,500 smaller ones, and I built a free tool so you can check yours.

Just want the fix? Skip to how to block AI training and stay in AI search.

What 2,500 websites do

Counting only sites that publish real robots.txt rules and let Google in:

• 19%of the top 1,000 block OpenAI's training crawler
• 48%of those also block ChatGPT search
• 50%same thing on smaller sites

What stood out to me:

• Blocking AI is mostly about training. In the top 1,000, about twice as many sites block OpenAI's training crawler as block ChatGPT search.
• But about half the sites blocking training block search too, whether they meant to or not. The top 100 runs higher, thanks to the big platforms, but everywhere else it hovers around half, all the way down to sites ranked in the hundreds of thousands. It isn't a big-site problem or a small-site problem. It's a robots.txt problem.
• The biggest platforms block more, on purpose. In the top 100 it's Amazon, TikTok, Facebook, Instagram, X and Pinterest, which are protecting their data. If you're a business that wants customers to find you, copying them just means fewer people see you.
• Smaller sites block AI less overall. 13% of the smaller sites block OpenAI's training crawler, compared with 19% of the top 1,000. When they do, they make the same trade.

Two ways sites lose ChatGPT search

When I looked at how the 99 sites in the study that block ChatGPT search actually do it, every one of them fell into one of two patterns.

1. The "block every AI bot" list (about two-thirds). 65 sites name OAI-SearchBot directly. Every one of them also blocks GPTBot, and 58 block at least 10 of the 16 bots I checked. That looks a lot like a list copied from somewhere, the kind that piles training bots, search bots and assistant bots into one block:

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: PerplexityBot
...
Disallow: /

The fix is to split the list: keep the training bots in the block and move the search and assistant bots out (the setup further down does exactly that).

2. The allowlist (about one-third). The other 34 block everything under User-agent: * and then let in a short list of search engines by name. Every one of them names Googlebot:

User-agent: Googlebot
User-agent: Bingbot
Allow: /

User-agent: *
Disallow: /

That made sense when Google and Bing were the only search that mattered. AI search bots aren't on the list, so they're shut out without anyone deciding that. The fix is to add OAI-SearchBot, Claude-SearchBot and PerplexityBot to the allowlist.

Which bots get blocked most

Training crawlers get blocked more often than search crawlers, on big and small sites alike:

For comparison, Bing's crawler is blocked by under 1% of these sites, and Googlebot by none (I only counted sites that let Google in).

How sites end up blocking AI by accident

When it's an accident, it usually happens one of three ways:

• Your CDN did it for you. In July 2025, Cloudflare started blocking AI crawlers by default. Its managed robots.txt can quietly add Disallow: / lines for bots like GPTBot, ClaudeBot and Google-Extended. In my data it only showed up on about 3% of the top 1,000, so it's less common than people think, but if you're on Cloudflare and never looked, check.
• Someone pasted a "block AI" snippet during the wave of worry about AI training, and it never came back out.
• Old rules block more than they meant to. A line like Disallow: /_next (common on Next.js sites) also blocks the files a crawler needs to render your pages.

It happens to people who do this for a living, too. When I built the checker, the first site I ran was mine, and it turned out my own site wasn't serving a real robots.txt at all, just Cloudflare's boilerplate with no rules and no sitemap. Nothing was blocked, but it's fixed now.

Know which bot does what

This is the part that trips people up. "AI crawler" isn't one thing, and blocking the wrong one costs you more than you meant to give up. Most AI companies now run separate bots for training, for search, and for fetching a page when a person asks about it:

• OpenAI: GPTBot trains models, OAI-SearchBot is what puts you in ChatGPT search, and ChatGPT-User fetches pages when someone asks. Block GPTBot to opt out of training and you'll still show up in ChatGPT answers, as long as you leave OAI-SearchBot alone.
• Anthropic: ClaudeBot for training, Claude-SearchBot for search, Claude-User for user requests.
• Perplexity: PerplexityBot builds its search index and Perplexity-User fetches pages for users. Perplexity says neither one collects training data.
• Google: Google-Extended doesn't touch Google Search. It controls Gemini training and grounding, not your rankings, and blocking it won't take you out of AI Overviews. Those run on regular Googlebot.

The two-minute check

The fastest way is the free checker: type in your site and it shows every bot above as allowed or blocked. To do it by hand:

1. Open yoursite.com/robots.txt and look for Disallow: / under any AI user agent, or under User-agent: * with no AI bot named separately.
2. If you're on Cloudflare or another CDN, check its bot and AI crawler settings too. A CDN can block crawlers before they ever reach your site, and that won't show up in the file.
3. Check every host your site uses. If your pages pull data from an API or a separate subdomain, that host needs a robots.txt that actually loads. A file that returns a server error makes Google stop crawling for the first 12 hours, then fall back to an old copy.
4. Make sure you aren't blocking your CSS and JavaScript folders, or crawlers can't see the page the way people do.

How to block AI training and stay in AI search

This is the setup most businesses actually want, and you can do it in robots.txt because the companies use separate bots for each job. Block the training bot, leave the search and assistant bots alone:

One limit worth knowing: Google doesn't offer this split for AI Overviews. They're part of Search and use regular Googlebot, so there's no way to stay in Google Search but out of AI Overviews with robots.txt.

Here's the whole thing as a robots.txt you can paste in (it's the same setup the checker gives you):

# Keep AI training out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
User-agent: Bytespider
Disallow: /

# Let AI search and assistants in
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

One gotcha: a bot named in a group follows only that group. If your User-agent: * section hides paths like /admin/, copy those Disallow lines into the second group too.

Common questions

Can I block AI training but still show up in AI search? Yes, for OpenAI, Anthropic and Apple, because they use separate bots for training and search. Block the training bot and allow the search bot. The setup is above. The exception is Google's AI Overviews, which you can't separate from regular Google Search.

Will blocking GPTBot hurt my Google rankings? No. GPTBot is OpenAI's crawler. Google Search uses Googlebot, and nothing you tell GPTBot changes what Google does.

Does blocking Google-Extended keep me out of AI Overviews? No. Google says Google-Extended doesn't affect Search, and AI Overviews are part of Search. The controls that do limit them, like nosnippet, apply to Search as a whole, so they'd cut into your regular results too.

If I block training bots now, does that remove what they already collected? No. robots.txt only controls crawling from here on. It doesn't pull back anything that's already been collected.

Should I block ChatGPT-User, Claude-User or Perplexity-User? For most businesses, no. Those bots open your page when a person asks the assistant about it or pastes your link. OpenAI says ChatGPT-User isn't what decides whether you show up in search (that's OAI-SearchBot), so blocking it mostly means the assistant can't read your page for someone who's already asking about you.

What's the Content-Signal line in my robots.txt? If you're on Cloudflare, its managed robots.txt can add a line like Content-Signal: search=yes, ai-train=no, a newer way to say "find me, don't train on me." Fewer than 2% of the top 1,000 sites had one saying ai-train=no. It's a nice extra, but the user-agent rules above are what the big AI companies document, so don't rely on it alone.

Then measure it

OpenAI and Perplexity both say it takes about 24 hours for a robots.txt change to kick in. After that, track AI traffic like any other channel: in GA4, referrals from chatgpt.com, perplexity.ai and gemini.google.com show up as their own sources. The number I care about isn't visits, it's how many of those people go on to buy or apply.

One caveat: robots.txt is a request, not a lock. The big AI companies say they follow it, but not every crawler plays nice (Cloudflare accused Perplexity of getting around blocks in 2025, which Perplexity disputes). For being found, though, that doesn't matter. The goal is making sure you're not the one keeping them out.

How I ran the numbers: one request for /robots.txt on each site, parsed the way the robots.txt standard (RFC 9309) says crawlers should. Sites: the full Tranco top 1,000 (Oct 14, 2026) plus a random 500 from each of ranks 1,001–10,000, 10,001–100,000 and 100,001–1,000,000. "Blocked" means a normal page is off limits to that bot. I only counted sites that serve real robots.txt rules and let Google in (499 of the top 1,000 and 813 of the smaller sites), since a site that blocks everyone isn't trying to be found. Each row of the last column rests on a few dozen training blockers at most (the top 100 on just 13), so read it as "about half," not exact. The counts in parentheses are the raw numbers. Some sites serve crawlers a different file, so this is what a regular visitor sees. You can download the per-site results (CSV).

How I made this: I wrote the crawler and the free checker with Claude as my pair programmer, and both use the same robots.txt parser, so the study and the tool give the same answers. The analysis and opinions are mine.