Tools
Crawl accessAI SEO Site AnalysisOne report on how ready your site is for ChatGPT, Gemini and AI Overviews: 40 checks, a score and the fixes that matter.AI Visibility CheckerCan ChatGPT, Claude, Perplexity, Gemini and Copilot see your page? One check, every engine.AI Site AuditCrawl your whole site and find every page AI crawlers can’t read, cite or reach.Cloudflare AI Crawler AnalyticsConnect Cloudflare with a read-only token and see real GPTBot, ClaudeBot and PerplexityBot traffic.Sitemap Checker for AI CrawlersValidate your XML sitemap and find URLs that are broken, redirected, noindexed or blocked for AI bots.AI Crawler CheckerSee which of 21 AI bots your robots.txt allows or blocks, with the exact rule that decides.AI robots.txt GeneratorAllow AI search, block AI training. Build a robots.txt for 21 AI bots in a few clicks.AI Bot Log AnalyzerUpload server logs and see which AI bots crawl you, what they read and which ones are fake.AI Bot IP CheckerIs it really GPTBot? Verify any IP against official AI crawler IP lists and reverse DNS.
Content for AIAI Crawlability CheckerSee your page the way AI crawlers do: raw HTML, no JavaScript, converted to Markdown.AI Citation CheckerWill ChatGPT quote your page? Score every passage for how easy it is for AI to cite.Query Fan-out GeneratorAISee the hidden sub-queries AI Mode and ChatGPT run for a search, and which ones your page covers.Schema Markup CheckerFull structured data audit: what JSON-LD you have, what’s missing for this page type, what’s wrong, plus ready code.LLM Token CounterCount GPT tokens for any text or web page and see which context windows it fits.llms.txt GeneratorBuild a spec-compliant llms.txt from your sitemap in seconds, then edit it visually.llms.txt ValidatorCheck any llms.txt against the spec: structure, links, duplicates and common mistakes.
Agents & trustPrompt Injection ScannerFind hidden instructions for AI on any web page: invisible text, comments, Unicode tricks, bot-only content.AI Agent Readiness CheckerCan AI agents find, understand and use your site? Checks MCP, A2A, OpenAPI, llms.txt and page usability.AI Entity CheckerDoes AI know your brand? Check Wikidata, Wikipedia and your Organization schema for consistency.AI Referral Traffic TrackerTrack visits from ChatGPT, Perplexity, Gemini and Claude in GA4 with a ready regex and channel setup.
LearnBlogAI trackerPricing
EN
EnglishENУкраїнськаUADeutschDEFrançaisFR
AI crawlersBotReady team

OpenAI: robots.txt may not apply to ChatGPT-User

If you blocked ChatGPT-User in robots.txt to keep ChatGPT away, that may no longer work. Here is how OpenAI’s three bots are documented now.

OpenAI’s crawler documentation now says that robots.txt rules may not apply to ChatGPT-User, the agent that fetches a page when a ChatGPT user asks about it. Earlier versions described it as controllable through robots.txt.

What happened

OpenAI documents three agents:

  • GPTBot collects content that may be used to train OpenAI’s models. robots.txt applies.
  • OAI-SearchBot crawls for ChatGPT search. OpenAI points site owners to this bot to manage whether pages appear in search answers.
  • ChatGPT-User visits pages on behalf of a user, for example when someone pastes a link or ChatGPT browses during an answer. These fetches are treated like a user’s request.

This is consistent with how Google treats user-triggered fetchers. One source also reports newer version numbers in GPTBot and OAI-SearchBot user agents. That is a single report.

What it means for you

  • To stay out of training, disallow GPTBot.
  • To stay in ChatGPT search, allow OAI-SearchBot.
  • To stop ChatGPT-User, robots.txt may not be enough. A firewall rule is the only hard block, and it also stops users who share your links in ChatGPT.

Check how often each OpenAI bot visits in your logs with the AI Bot Log Analyzer.

Sources

  1. Search Engine World
  2. Momentic, AI crawler list (October 2026)
  3. Honeyb, OAI-SearchBot

Check your site

AI Crawler CheckerSee which of 21 AI bots your robots.txt allows or blocks, with the exact rule that decides.AI robots.txt GeneratorAllow AI search, block AI training. Build a robots.txt for 21 AI bots in a few clicks.AI Bot Log AnalyzerUpload server logs and see which AI bots crawl you, what they read and which ones are fake.