Is your site blocking ChatGPT?
Most sites that block AI only want to keep their content out of training. About half block AI search too, often without meaning to. Check yours in a few seconds, see how much of your homepage AI crawlers can actually read, and get a fix if you need one.
What AI crawlers can read on your homepage
See the robots.txt we read
Most sites blocking AI just want out of training. About half block AI search too.
I checked the robots.txt of the top 1,000 websites, plus a random sample of 1,500 smaller ones (Tranco list, Oct 14, 2026). Counting only sites that publish real rules and let Google in:
- 19%of the top 1,000 block OpenAI's training crawler
- 48%of those also block ChatGPT search
- 50%same thing on smaller sites
| Sites ranked | Block AI training (GPTBot) | Block ChatGPT search | Of training blockers, also block ChatGPT search |
|---|---|---|---|
| Top 100 | 26% | 20% | 62% |
| 101–500 | 21% | 9% | 45% |
| 501–1,000 | 17% | 8% | 48% |
| 1,001–10,000 | 13% | 7% | 51% |
| 10,001–100,000 | 13% | 6% | 41% |
| 100,001–1M | 12% | 7% | 56% |
Bigger sites block AI more often, but the trade is the same at every size: about half the sites keeping OpenAI out of training also keep themselves out of ChatGPT answers. You don't have to choose. The fix the checker gives you blocks training and keeps search open.
What this checks (and what it doesn't)
This reads your robots.txt, the file that tells crawlers what they're allowed to read. The big AI companies say their crawlers follow it, so it's the first thing to get right.
It also reads your homepage the way most AI crawlers do: as the raw HTML, without running JavaScript. Sites built entirely in JavaScript can let AI crawlers in and still show them a blank page.
It can't see blocks that happen before a crawler reaches your site. Some CDNs and firewalls, Cloudflare included, can block AI crawlers on their own. If this check looks fine but you're still missing from AI answers, look at your CDN's bot and AI crawler settings next.
The 16 crawlers it checks
| Crawler | Company | What it does |
|---|---|---|
| GPTBot | OpenAI | Training OpenAI's models |
| OAI-SearchBot | OpenAI | ChatGPT search results |
| ChatGPT-User | OpenAI | Pages ChatGPT opens when a user asks |
| ClaudeBot | Anthropic | Training Anthropic's models |
| Claude-SearchBot | Anthropic | Claude's search results |
| Claude-User | Anthropic | Pages Claude opens when a user asks |
| PerplexityBot | Perplexity | Perplexity's search index |
| Perplexity-User | Perplexity | Pages Perplexity opens when a user asks |
| Googlebot | Google Search and AI Overviews | |
| Google-Extended | Gemini training and grounding (not Search) | |
| Bingbot | Microsoft | Bing search and Copilot |
| Applebot | Apple | Siri and Spotlight results |
| Applebot-Extended | Apple | Training Apple's models |
| meta-externalagent | Meta | Training Meta's models |
| CCBot | Common Crawl | Common Crawl, a dataset many models train on |
| Bytespider | ByteDance | Training ByteDance's models |
Common questions
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot only collects training data. ChatGPT search uses OAI-SearchBot, and pages ChatGPT opens when someone asks use ChatGPT-User. Block GPTBot alone and you stay out of training but still show up in answers. Many sites block all three by accident with a rule for every bot.
How do I block AI training but stay in AI search?
Give the training crawlers (GPTBot, ClaudeBot, Google-Extended and others) a Disallow: / group, and the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) an Allow: / group. The checker gives you the exact lines to paste.
Can ChatGPT read a website built with JavaScript?
OpenAI's crawlers, like Anthropic's and Perplexity's, read the HTML your server sends and don't run JavaScript. Text that only appears after JavaScript runs is invisible to them. Google and Bing do run JavaScript. Server-side rendering or pre-rendering fixes it for every crawler.
Can robots.txt keep my site out of Google's AI Overviews?
Not on its own. AI Overviews use regular Googlebot, so blocking it also removes you from Google Search. Google-Extended only covers Gemini training and grounding, not Search or AI Overviews.
How long do robots.txt changes take to work?
About a day. Google and OpenAI both cache robots.txt for up to 24 hours before they read it again.
Need a hand?
I'm Jay. I do SEO and AI search, and I care most about whether the traffic turns into customers. If you want to show up in AI answers and aren't, I'd like to hear about it.