How to Block AI Crawlers (and Which Ones Not To)

Every major AI company now runs several crawlers with different jobs. If you want to keep your content out of model training, you don’t have to vanish from ChatGPT or Perplexity answers. You only need to block the right user agents.

Three kinds of AI bots

Kind	What it does	Examples	Block it?
Training	Collects pages to train future models	GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent	Your choice
Search	Builds the index that AI answers cite	OAI-SearchBot, Claude-SearchBot, PerplexityBot	Usually no
User-triggered	Opens a page because a user asked about it	ChatGPT-User, Claude-User, Perplexity-User	Usually no

Two more are not crawlers but control tokens. Google-Extended tells Google not to use your content for Gemini training and grounding, while Googlebot keeps crawling for Search. Applebot-Extended does the same for Apple Intelligence.

Blocking training bots costs you nothing in traffic: training doesn’t send visitors. Blocking search bots removes you from that assistant’s cited sources.

A robots.txt that blocks training only

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Bytespider
User-agent: Meta-ExternalAgent
Disallow: /

User-agent: *
Allow: /

Several User-agent lines above one rule form a single group. A bot obeys only the most specific group that names it, so GPTBot here ignores the * group entirely.

You can also add a Content Signal line, which states your preference in words AI companies can parse:

Content-Signal: search=yes, ai-train=no

Build your own version with checkboxes in the AI robots.txt generator.

Common mistakes

• Blocking Googlebot to stop Gemini. It removes you from Google Search. Use Google-Extended.
• Forgetting groups don’t merge. If GPTBot has its own group, rules under * don’t apply to it.
• Blocking only ChatGPT-User. Then ChatGPT can still cite your page from its index but can’t open it when a user asks to read it.
• Expecting instant results. Crawlers cache robots.txt for up to a day, and content already collected isn’t deleted.

When robots.txt is not enough

robots.txt is a request, not a lock. OpenAI, Anthropic, Google, Apple and Perplexity say their declared crawlers respect it. Scrapers that copy a bot’s user agent don’t.

To enforce a block:

1. Use your CDN’s bot controls. Cloudflare’s AI Crawl Control lets you block AI crawlers by category with verified identity, not just by user agent.
2. Block by verified IP, not by name. OpenAI, Perplexity, Google, Microsoft, Apple and Common Crawl publish IP ranges. Anthropic doesn’t, so ClaudeBot can only be blocked by user agent.
3. Rate-limit unknown bots that ignore robots.txt.

Blocking by user agent alone stops honest bots, which already follow robots.txt, and does nothing against fakes. Verify suspicious IPs with the AI Bot IP Checker.

Check that it works

1. Run the AI Crawler Checker to see the verdict for each bot.
2. After a few days, drop your access log into the AI Bot Log Analyzer. Blocked training bots should stop appearing, or only fetch /robots.txt.
3. If a “blocked” bot keeps crawling pages, check its IP. It is often a fake.