How to Verify Real AI Bots and Catch the Fake Ones
A user agent is just text. Scrapers copy GPTBot and ClaudeBot to get past filters and to hide in your logs. Here is how to tell the real crawlers from the impostors.
What you’ll learn
- Why user agents can’t be trusted on their own
- Where AI companies publish their crawler IP ranges
- How reverse DNS verification works for Googlebot and Bingbot
- What to do with requests that fail verification
Every request carries a User-Agent header, and any program can set it to anything. If a scraper writes GPTBot there, your server can’t tell the difference from the header alone. That matters in three situations:
- you allow AI search bots through a firewall and don’t want scrapers riding on that rule;
- you analyse logs and want real AI crawler numbers, not inflated ones;
- you block AI training bots and want to know whether the blocked traffic was even real.
Two ways to verify a crawler
1. Published IP ranges
Most AI companies publish the IP addresses their bots use as JSON files. A request is genuine only if its IP falls inside the published ranges for that bot.
| Company | Bots | Where the ranges are |
|---|---|---|
| OpenAI | GPTBot, OAI-SearchBot, ChatGPT-User |
One JSON file per bot, linked from OpenAI’s crawler documentation |
| Anthropic | ClaudeBot, Claude-SearchBot, Claude-User |
A JSON file linked from Anthropic’s crawler page |
| Perplexity | PerplexityBot, Perplexity-User |
JSON files linked from Perplexity’s bot documentation |
Googlebot, special crawlers, user-triggered fetchers |
Separate JSON files for each group in Google Search Central | |
| Microsoft | bingbot |
A JSON file in Bing Webmaster documentation |
Ranges change, so load them fresh instead of copying them once into a config file.
2. Reverse DNS
Google and Microsoft also support reverse DNS verification:
- Look up the hostname for the request IP (a reverse DNS, or PTR, lookup).
- Check that the hostname ends in the right domain:
googlebot.comorgoogle.comfor Google,search.msn.comfor Bing. - Look up that hostname forward and confirm it resolves back to the same IP.
Both steps are needed. A scraper can control the reverse record of its own IP, but not the forward record of googlebot.com.
What a fake bot looks like
Typical signs, beyond a failed IP check:
- Hosting provider IPs: cheap cloud servers rather than the vendor’s network.
- Odd paths: login pages,
wp-admin,.envfiles or search URLs with random parameters. - Broken user agents: a missing version number or a wrong contact URL.
- No robots.txt request before crawling, while real crawlers fetch it regularly.
What to do with fakes
- Block them at the firewall, not in robots.txt. Fake bots ignore robots.txt anyway.
- Keep them out of your reports. Count verified crawler requests separately from user-agent matches, or your AI crawler numbers will be wrong.
- Don’t block by user agent alone. A rule that blocks every request saying
GPTBotalso blocks the real one, which may not be what you want.
Check it quickly
Paste an IP and a user agent into the AI Bot IP Checker to see whether the IP really belongs to the bot it claims to be. For a whole log file, the AI Bot Log Analyzer verifies every AI bot request and lists the fakes. The BotReady AI tracker does the same check automatically for every crawler visit and shows fake bots in a separate table.