WysLeap
Guide · AI crawlers

How can I see which AI crawlers visit my website?

Track them from your server or CDN, because most AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) don't run JavaScript and never show up in Google Analytics. WysLeap does this with a few lines in your middleware or CDN: it records every AI crawler visit by crawler and page, checks each IP against the ranges OpenAI, Perplexity, Google and Apple publish, and drops the fakes. You can also read your raw access logs, or use Cloudflare AI Crawl Control if your site is behind Cloudflare.

Updated

Tools that show AI crawler visits

  • WysLeap: server-side reporting from your middleware, CDN worker or server, with every hit checked against the crawler's published IP ranges, visits by crawler and page, and the AI-referred visitors those crawlers lead to. Included in the 90-day free trial.
  • Cloudflare AI Crawl Control: AI crawler activity and allow or block controls, for sites behind Cloudflare.
  • Your access logs: complete and free, but you parse, verify and chart them yourself.

Why your analytics don't show AI crawlers

Analytics scripts count a visit when JavaScript runs in a browser. AI crawlers mostly request the HTML and leave, so the script never runs. Google Analytics also filters known bots out on purpose. The visits happen; your analytics just can't see them.

Three places to find them

1. Server or CDN access logs

Every request is in your access logs with its user agent and IP address. Search for the crawler names (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and so on). Logs are complete and free, but you're left to parse, verify and chart them yourself, and many hosts keep them for only a couple of weeks.

2. Cloudflare AI Crawl Control

If your site is proxied through Cloudflare, its AI Crawl Control dashboard shows which AI crawlers visit and lets you allow or block them. It only covers traffic that passes through Cloudflare.

3. Server-side reporting to an analytics tool

A few lines in your middleware, CDN worker or server can send each AI crawler request to an analytics tool. WysLeap's server-side crawler guide has copy-paste examples for Cloudflare Workers, Next.js and Express. The SEO & GEO dashboard then shows visits by crawler and page, next to the AI-referred visitors those crawlers lead to.

Check the IP, not just the name

Anyone can send a request that says it's ChatGPT. On one of our own sites in October 2026, 76 of 79 requests calling themselves ChatGPT-User were scanners probing for secret files such as .env. We still see requests named OAI-SearchBot coming from cloud servers outside the ranges OpenAI publishes.

OpenAI, Perplexity, Google, Microsoft and Apple publish the IP ranges their crawlers use, so a request outside them is fake. WysLeap checks every server-reported hit against those lists and drops the ones that don't match. Anthropic, Amazon and Meta publish no lists; for those, a single IP claiming to be crawlers from several companies is a reliable sign of a scanner.

A gotcha: static files can skip your middleware

On hosts that serve static files straight from a CDN, requests for files such as robots.txt never reach your middleware or server code, so server-side reporting won't count them. If you want to know whether a crawler checked your robots.txt, look in the CDN's access logs.

What to do with the data

  • Make sure the crawlers that feed AI answers aren't blocked in robots.txt or by your firewall.
  • Compare which pages they read with the pages you want quoted. A pricing page no AI crawler has fetched in weeks can't be quoted accurately.
  • Watch the result: the brand mentions in AI answers and the visitors AI assistants send you.

Side by side

Ways to see AI crawler visits

FeatureWysLeapAccess logsCloudflare AI Crawl ControlGA4
Sees crawlers that don't run JavaScript
With server reporting
Checks crawler IPs against published ranges
Manual
Visits by crawler and page
Manual
Next to your visitor analytics
Works on any host
If your host keeps logs
Cloudflare sites only

FAQ

Frequently asked questions

Why doesn't Google Analytics show AI crawlers?
Google Analytics counts a visit when its JavaScript runs in a browser. Most AI crawlers fetch the page's HTML and never run JavaScript, and GA4 also excludes known bots on purpose. So AI crawlers don't appear in GA4 reports at all.
Which AI crawlers should I look for?
The ones that feed AI answers today: OAI-SearchBot and ChatGPT-User (OpenAI), Claude-SearchBot and Claude-User (Anthropic) and PerplexityBot and Perplexity-User. Training crawlers include GPTBot, ClaudeBot, Google-Extended, CCBot and Bytespider. WysLeap tracks 17 AI crawlers in three tiers.
How do I know a crawler is real?
Check the request's IP address against the ranges the operator publishes. OpenAI, Perplexity, Google, Microsoft and Apple publish them. Anthropic, Amazon and Meta don't, so those can only be judged by behavior, such as one IP claiming to be several different companies' crawlers.
Is AI crawler tracking included in the WysLeap trial?
Yes. AI crawler tracking is part of SEO & GEO, which is included in the 90-day free trial and in Pro ($30 a month for 1M events). Each stored crawler hit counts toward the monthly event allowance.

See what AI reads on your site.

AI crawler visits by crawler and page, checked against published IP ranges.