AI crawler blocking keeps rising: Q3 2026 data
More sites are shutting AI crawlers out, both in robots.txt and at the firewall. The numbers differ by study, but the direction is the same.
New studies of the third quarter of 2026 point the same way: more websites are blocking AI crawlers.
What happened
- One analysis found 13.47% of AI bot requests got a 403 Forbidden response in Q3 2026, up from 5.98% a year earlier. A 403 usually means a firewall or bot-management rule, not robots.txt.
- A study of the Tranco top 10,000 sites found that 14.3% of valid robots.txt files block GPTBot from the whole site, and 13.4% block ClaudeBot.
- Cloudflare revised how it classifies training and search crawlers in October 2026, so year-over-year comparisons of crawler shares need care.
These figures come from independent vendors, not from OpenAI, Anthropic or Cloudflare.
What it means for you
The risk is blocking more than you intend. Training bots (GPTBot, ClaudeBot, Google-Extended) and search or user bots (OAI-SearchBot, Claude-SearchBot, ChatGPT-User, PerplexityBot) are separate. Blocking the first keeps your content out of training. Blocking the second removes you from AI answers.
Also check your firewall. A CDN rule can return 403 to a bot that your robots.txt allows.
The AI Visibility Checker tests robots.txt and firewall responses for each major AI bot.