How to See AI Crawlers in Your Server Logs
AI crawlers don’t run JavaScript, so analytics tools never see them. Your server does. Here is where to look and what the numbers tell you.
What you’ll learn
- Why GA4 and other analytics can’t show AI crawlers
- Where to get logs on common hosting setups
- Which fields and numbers matter for AI crawlers
- How to turn raw logs into a weekly report
GA4, Plausible and similar tools work by running a script in the visitor’s browser. AI crawlers such as GPTBot, ClaudeBot and PerplexityBot download your HTML and never execute that script. To analytics they are invisible.
The only complete record of AI crawler activity is on the server side: access logs, CDN logs or a small piece of server code that reports bot requests.
Where to get the logs
| Setup | Where the logs are |
|---|---|
| Nginx or Apache on your own server | access.log, usually in /var/log/nginx/ or /var/log/apache2/ |
| Shared hosting (cPanel and similar) | “Raw Access” or “Logs” section of the control panel |
| Cloudflare | Security Analytics, AI Crawl Control, or the GraphQL Analytics API |
| Vercel, Netlify and other platforms | Log drains or the platform’s request logs, often with short retention |
| WordPress without server access | A plugin or a small snippet that records bot requests |
Logs on many hosts are kept for only a few days. Export them regularly if you want trends.
What to look at
Each log line has the fields you need: time, IP, request path, status code and user agent. For AI crawlers, five numbers matter:
- Requests per bot per day. Which companies crawl you, and is it growing?
- Purpose split. Training bots (
GPTBot,ClaudeBot,CCBot) versus search and user bots (OAI-SearchBot,Claude-SearchBot,PerplexityBot,ChatGPT-User). Search bot visits are the ones that lead to citations. - Top crawled pages. If bots mostly hit tag pages, filters or old URLs, your important content may not be getting crawled.
- Status codes. A high share of 403, 404 or 5xx responses means bots are blocked or hitting broken pages.
- Verified versus claimed. Requests whose IP doesn’t match the vendor’s published ranges are fake. Count them separately.
A quick command-line check
On a server with Nginx, this counts requests per AI bot in today’s log:
grep -oE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Google-Extended|Bytespider|CCBot|Amazonbot" /var/log/nginx/access.log | sort | uniq -c | sort -rnIt counts user-agent matches only, so fake bots are included.
From logs to a weekly report
Raw logs are hard to read every week. Three ways to make them practical:
- Upload a log file to the AI Bot Log Analyzer. It finds AI bots, verifies their IPs and shows top pages and errors.
- Connect Cloudflare with a read-only token in Cloudflare AI Crawler Analytics to see crawler requests from your zone by bot and day.
- Install the BotReady AI tracker. A small server snippet reports every AI crawler request with its page and status, and a script tag counts visitors from ChatGPT, Perplexity and Gemini. Both appear in one dashboard.
Connect crawling to traffic
Crawling is only the first step. The useful question is which crawled pages later receive visitors from AI assistants. Pages that search bots read often but that never get AI visitors may need clearer answers or better structure. Pages that get AI visitors show what kind of content AI search trusts on your site.