In an earlier post, we wrote about how automated traffic now outweighs human traffic (according to Imperva's 2025 Bad Bot Report), and how AI bots and agentic browsers are corrupting analytics metrics that were never built to handle them. Since then, the problem has evolved faster than most teams anticipated.
The first wave of AI traffic, the scrapers training large language models, is now a known quantity. A second wave has arrived: LLM-powered crawlers that operate on behalf of real users in real time. They go beyond scraping. They're question-answering engines, research agents, and AI browsing tools that visit your site, read your content, and report back, and your analytics sees none of it.
"Your bot filters know what Googlebot looks like. They have no idea what GPT-4o web search looks like."
That gap is growing. Sites that understand what LLM crawlers are, how they behave, and how to detect them will have a structural advantage in the AI-native web that's rapidly becoming the default.
What LLM Crawlers Actually Are
The term "LLM crawler" covers several distinct types of non-human visitors that have emerged as AI assistants have gone mainstream. They're related, but they differ from each other in important ways, and from the traditional bots your current filters are designed to catch.
1. Training Data Scrapers (The First Wave)
These crawl the web systematically to collect text for training or fine-tuning AI models. GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended, and CCBot (Common Crawl) are the most visible, because most declare themselves through user-agent strings and respect robots.txt when asked.
They're relatively easy to detect and block. Many sites have already updated their robots.txt to exclude them. The catch is that they represent only a fraction of AI traffic.
2. Real-Time Retrieval Crawlers (The Second Wave)
These visit your site on demand, in real time, to answer specific user questions. When someone asks Perplexity "What are WysLeap's pricing plans?" or asks ChatGPT to "summarize the latest articles from [your blog]," a crawler fetches your page, processes it, and returns an answer to the user, who never visits your site in their own browser.
Key characteristic: these crawlers are triggered by user intent, visit specific pages, and fetch only what they need. Their traffic pattern looks nothing like systematic scrapers, which is exactly why they evade traditional bot filters.
3. Agentic Browsers (The Third Wave)
These are AI systems that browse the web on behalf of users to complete tasks. Unlike retrieval crawlers that just fetch pages, agentic browsers can navigate, click, fill forms, and interact with dynamic content. Tools like Perplexity Comet, browser-use, Playwright-based AI agents, and increasingly, operating-system-level AI assistants fall into this category.
They're the hardest to detect because they often execute JavaScript, render pages fully, and exhibit behavior patterns that overlap with legitimate human browsing.
The Scale Is Already Significant
Industry analyses of publisher server logs suggest LLM-related crawl requests now make up a meaningful share of total traffic on content-heavy sites. For SaaS product sites with detailed feature documentation and pricing pages, the share may be even higher.
Almost none of this is captured in standard analytics dashboards. This is a structural gap in how the web is currently measured, not a niche edge case.
Why Your Existing Bot Filters Can't Catch Them
Traditional bot detection was designed for a different threat model. Understanding why it fails on LLM crawlers requires understanding exactly what it was built to catch.
User-Agent Blocklists Don't Scale Anymore
Traditional bot filters maintain lists of known bad user-agent strings. This works when there are dozens of known scrapers, each with a consistent signature. It doesn't work when there are thousands of AI frameworks, many of which rotate user agents, spoof browser signatures, or simply use legitimate browser user-agent strings.
GPTBot declares itself. So do ClaudeBot and Googlebot-Extended. But a custom agent built on Playwright or Puppeteer may present as Chrome 131 on macOS with no distinguishing characteristics at all. Blocklists catch the honest actors while missing the ones that matter.
Client-Side Analytics Scripts Don't Run
Most analytics platforms, including Google Analytics 4, are client-side. They rely on a JavaScript tag executing in the visitor's browser. When a retrieval crawler fetches a page without executing JavaScript (which many do, by design, for efficiency), the analytics tag never fires. The visit is completely unrecorded.
This is the invisibility problem. These crawlers aren't necessarily hard to detect. The detection mechanism simply doesn't run when they arrive.
Behavioral Heuristics Were Trained on Old Patterns
Traditional bot detection looks for patterns like: visits too many pages too fast, stays zero seconds, comes from suspicious IP ranges, doesn't accept cookies, has no mouse movement. LLM retrieval crawlers violate none of these. They visit specific pages, spend a normal amount of "time" fetching them, come from cloud provider IP ranges, and navigate sensible paths.
They look, to old detection models, like a careful human browsing from a cloud server. Which is, in a sense, exactly what they are.
The Robots.txt Problem
Robots.txt is the web's social contract: crawlers that respect it identify themselves and follow your access rules. The problem is compliance is voluntary. OpenAI, Anthropic, and a handful of large players respect robots.txt. The ecosystem of third-party AI tools, open-source agent frameworks, and enterprise AI platforms that are now building on top of these models largely don't.
You can opt out of GPTBot. You cannot opt out of a developer's custom AI agent built on GPT-4o that browses using a Chrome user-agent.
How to Actually Detect LLM Crawlers
Detection means combining several signals that are weak on their own but together create a reliable fingerprint. Here's the layered approach that works in practice.
Layer 1: Server-Side Request Analysis
Detection has to start here, before client-side JavaScript ever runs. Examine raw HTTP request headers for signals that correlate with automated fetching:
- Accept-Language header: Bots frequently send
Accept-Language: *or a minimal set. Humans have specific language preferences. - Missing or minimal Sec-Fetch headers: Modern browsers send a rich set of Sec-Fetch-* headers. Many crawlers omit these entirely.
- No Accept-Encoding diversity: Browsers accept multiple compression schemes. Minimal bots often send only
Accept-Encoding: gzip. - IP range analysis: Cloud provider IP ranges (AWS, GCP, Azure) are overrepresented in LLM crawler traffic versus human traffic.
Layer 2: JavaScript Execution Probe
The cleanest signal: does JavaScript run? Embed a lightweight, non-blocking beacon in your page that fires on load. Requests that arrive at your server without a corresponding beacon hit within a short window (2–5 seconds) are almost certainly non-JS-executing crawlers.
// Minimal JS beacon: add to your page layout
// (/api/js-probe is an endpoint on your own server)
fetch('/api/js-probe', {
method: 'POST',
body: JSON.stringify({ url: location.href, ts: Date.now() })
});Layer 3: Behavioral Fingerprinting
For LLM crawlers that do execute JavaScript, detection shifts to behavioral signals:
- No mouse movement or touch events: Humans move their cursor before clicking. Headless browsers and agents typically trigger clicks without prior mouse movement events.
- Abnormal timing patterns: Humans scroll, pause, read, scroll again. Crawlers that render pages typically scroll to a fixed depth immediately.
- Navigator API anomalies:
navigator.webdriveris true for browser automation frameworks unless explicitly patched. - Screen dimension consistency: Agentic browsers often run in default headless viewport sizes (1280x720, 1366x768) with unusual pixel ratios.
Layer 4: ML-Based Anomaly Detection
Each signal on its own can be weak, because LLM crawlers are getting better at mimicking human behavior. The strongest detection approach combines all signals into an ML model that:
- Learns the normal distribution of your site's human traffic (not a generic model)
- Identifies deviations from that distribution as anomaly scores
- Updates continuously as new crawler patterns emerge, so detection improves as the threat evolves
- Can cluster anomalous traffic into named patterns (e.g., "Perplexity fetch pattern," "headless Chrome agentic," "unknown retrieval crawler")
WysLeap's bot detection today covers the earlier layers of this stack: known bot and AI-crawler user agents, automation and headless-browser detection (such as navigator.webdriver), and simple browser signals. Bots are flagged at ingest and excluded from human metrics.
What to Do Once You Can See Them
Detection is only the first step. What you do with LLM crawler traffic depends on what type it is and what your goals are. The right response is not always "block."
For Training Scrapers: Robots.txt + Licensing
If your primary concern is your content being used to train AI models without compensation, robots.txt directives for known crawlers (GPTBot, ClaudeBot, CCBot) are the first line of defense. Beyond that, content licensing discussions with AI companies are becoming more viable, and knowing exactly which crawlers are accessing your content and how frequently puts you in a much stronger negotiating position.
For Real-Time Retrieval: Measure, Then Decide
Before blocking real-time retrieval crawlers, consider what they're actually doing. If Perplexity is fetching your pricing page ten times a day in response to user queries, those queries represent genuine commercial intent from humans who could become customers. Blocking the crawler may redirect that intent to a competitor's pricing page instead.
The more strategic move: optimize your content for AI retrieval, ensure your most important information is in the HTML (not behind JavaScript rendering), and monitor which pages get fetched most. Those pages are surfacing in AI answers, so they deserve your attention.
For Agentic Browsers: Intent-Based Classification
HUMAN Security found that 87% of AI agent page visits are product-related. Each of those visits stands for a human with genuine purchase intent who has delegated research to an AI assistant. Blanket blocking loses those leads.
A better approach: classify agentic traffic separately, exclude it from your human-audience metrics (so your analytics remain clean), but don't block it from accessing your content. Treat agentic visits as an engagement signal. If an AI agent is visiting your pricing page on someone's behalf, that someone is worth understanding.
The Metrics to Track
- LLM crawler share of total traffic: Trending up is expected, but sudden spikes may indicate targeted scraping
- Top pages by LLM fetch volume: Tells you which content is being surfaced in AI responses
- Crawler type breakdown: Training scrapers vs. real-time retrieval vs. agentic, treated differently
- Human-only traffic trend: Your actual audience, separated from machine traffic
- New bot pattern emergence: Novel traffic signatures you haven't seen before
Why Static Lists Will Always Lose, and What Wins Instead
The set of LLM crawlers changes every few months. New AI tools launch. Existing ones update their user agents. Enterprise customers build internal agents on top of public APIs. Open-source frameworks proliferate.
Any detection approach based on a static blocklist is already behind. By the time you add a new crawler signature to your filter, it's been joined by a dozen variants you haven't seen yet.
The Only Sustainable Approach: Behavioral ML That Adapts
A detection system that learns what your human traffic looks like, continuously updates that model, and identifies deviations without needing to know which crawler produced them will hold up far longer than any static approach.
That's where the industry is heading. WysLeap's detection today combines a maintained list of known bot and AI-crawler user agents with automation and headless-browser checks and simple browser signals. It catches self-declared crawlers and many automated browsers, and flags them at ingest.
The Three-Layer Architecture That Works
- Server-side collection: Capture all traffic before any JavaScript runs, so nothing is invisible by default
- Multi-signal classification: Combine header analysis, JS execution probes, and behavioral fingerprinting to classify each visitor
- Self-learning anomaly detection: Continuously identify new patterns, update the model, and improve accuracy without manual intervention
The result is a system that gets better at detecting LLM crawlers over time, while blocklists only get more stale.
The Taxonomy Cheat Sheet
| Crawler Type | Detectable by UA? | Runs JS? | Risk to Metrics | Recommended Action |
|---|---|---|---|---|
| Training scrapers (GPTBot, CCBot) | Often yes | No | Content theft | Block via robots.txt |
| Real-time retrieval (Perplexity, Bing AI) | Sometimes | Rarely | Invisible in analytics | Measure, then decide; optimize content for AI |
| Custom agents (Playwright, browser-use) | Rarely | Sometimes | Corrupts metrics | ML behavioral detection required |
| Agentic browsers (Comet, Gemini) | No | Yes | Inflates engagement | Classify separately; don't block |
See the bots that are hitting your site
WysLeap flags known bots, AI crawlers and automated browsers at ingest, so you get cleaner human-only metrics alongside bot visibility. Start free or read The Invisible Audience.
