Tools
Crawl accessAI SEO Site AnalysisOne report on how ready your site is for ChatGPT, Gemini and AI Overviews: 40 checks, a score and the fixes that matter.AI Visibility CheckerCan ChatGPT, Claude, Perplexity, Gemini and Copilot see your page? One check, every engine.AI Site AuditCrawl your whole site and find every page AI crawlers can’t read, cite or reach.Cloudflare AI Crawler AnalyticsConnect Cloudflare with a read-only token and see real GPTBot, ClaudeBot and PerplexityBot traffic.Sitemap Checker for AI CrawlersValidate your XML sitemap and find URLs that are broken, redirected, noindexed or blocked for AI bots.AI Crawler CheckerSee which of 21 AI bots your robots.txt allows or blocks, with the exact rule that decides.AI robots.txt GeneratorAllow AI search, block AI training. Build a robots.txt for 21 AI bots in a few clicks.AI Bot Log AnalyzerUpload server logs and see which AI bots crawl you, what they read and which ones are fake.AI Bot IP CheckerIs it really GPTBot? Verify any IP against official AI crawler IP lists and reverse DNS.
Content for AIAI Crawlability CheckerSee your page the way AI crawlers do: raw HTML, no JavaScript, converted to Markdown.AI Citation CheckerWill ChatGPT quote your page? Score every passage for how easy it is for AI to cite.Query Fan-out GeneratorAISee the hidden sub-queries AI Mode and ChatGPT run for a search, and which ones your page covers.Schema Markup CheckerFull structured data audit: what JSON-LD you have, what’s missing for this page type, what’s wrong, plus ready code.LLM Token CounterCount GPT tokens for any text or web page and see which context windows it fits.llms.txt GeneratorBuild a spec-compliant llms.txt from your sitemap in seconds, then edit it visually.llms.txt ValidatorCheck any llms.txt against the spec: structure, links, duplicates and common mistakes.
Agents & trustPrompt Injection ScannerFind hidden instructions for AI on any web page: invisible text, comments, Unicode tricks, bot-only content.AI Agent Readiness CheckerCan AI agents find, understand and use your site? Checks MCP, A2A, OpenAPI, llms.txt and page usability.AI Entity CheckerDoes AI know your brand? Check Wikidata, Wikipedia and your Organization schema for consistency.AI Referral Traffic TrackerTrack visits from ChatGPT, Perplexity, Gemini and Claude in GA4 with a ready regex and channel setup.
LearnBlogAI trackerPricing
EN
EnglishENУкраїнськаUADeutschDEFrançaisFR
AI crawlersBotReady team

Anthropic’s three crawlers explained

Blocking ClaudeBot doesn’t remove you from Claude’s search answers anymore, and blocking Claude-SearchBot can. Here is how Anthropic’s bots split up.

Anthropic’s crawler documentation lists three separate bots, each with its own user agent you can allow or block in robots.txt. Reports date the documentation update to late February 2026. The previous version described only ClaudeBot.

What happened

  • ClaudeBot collects public web content for training Anthropic’s models.
  • Claude-SearchBot crawls to index content for search, so Claude can find and cite pages in answers. Anthropic warns that blocking it may reduce your visibility and accuracy in Claude’s search results.
  • Claude-User fetches a page when a user asks Claude about it.

Older user agents, Claude-Web and anthropic-ai, are deprecated. Anthropic publishes the IP ranges its bots use, so you can verify that a request claiming to be Claude really comes from Anthropic.

What it means for you

  • Block training, keep search: disallow ClaudeBot and allow Claude-SearchBot and Claude-User.
  • Check old rules. A robots.txt that only blocks anthropic-ai does nothing today.
  • Verify before you trust. Scrapers often pretend to be ClaudeBot. Compare request IPs with Anthropic’s published ranges.

The AI Bot IP Checker tells you whether an IP really belongs to the bot it claims to be.

Sources

  1. Search Engine Journal
  2. Momentic, AI crawler list (October 2026)

Check your site

AI Crawler CheckerSee which of 21 AI bots your robots.txt allows or blocks, with the exact rule that decides.AI robots.txt GeneratorAllow AI search, block AI training. Build a robots.txt for 21 AI bots in a few clicks.AI Bot IP CheckerIs it really GPTBot? Verify any IP against official AI crawler IP lists and reverse DNS.