Documentation

Onboarding guide

Get from zero to live analytics in minutes: installation, custom events, SaaS metrics, AI observability, and session replay.

Step 08 of 9: AI Crawlers

Onboarding guide

Server-side AI crawler detection

The WysLeap script only sees AI crawlers that run JavaScript. Most don't. Report them from your server, CDN or middleware and the SEO & GEO dashboard counts every visit.

How it works

Your server checks each request's user agent. When it matches a known AI crawler, it sends the page path to WysLeap in the background, after the response, so visitors never wait for it. WysLeap stores only AI crawler hits; anything else is dropped.

Endpoint: POST https://wysleap.com/api/telemetry/crawler
Header: x-api-key with your site's API key (the same key as in your tracking script)
Body: { siteId, hits: [{ userAgent, path, status?, timestamp? }] }, up to 50 hits per request

Once a site sends server-side hits, the AI crawler charts use them instead of the script's detections, so crawlers that also run JavaScript aren't counted twice. Each stored hit counts toward your monthly event allowance.

Cloudflare Worker

Add a Worker on your site's route and set WYSLEAP_SITE_ID and WYSLEAP_API_KEY as Worker variables.

javascript

const AI_BOTS = /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|anthropic-ai|PerplexityBot|Google-Extended|GoogleOther|Applebot-Extended|Amazonbot|FacebookBot|CCBot|Bytespider|cohere-ai/i;

export default {
  async fetch(request, env, ctx) {
    const response = await fetch(request);
    const userAgent = request.headers.get("user-agent") || "";
    if (AI_BOTS.test(userAgent)) {
      const url = new URL(request.url);
      ctx.waitUntil(
        fetch("https://wysleap.com/api/telemetry/crawler", {
          method: "POST",
          headers: { "Content-Type": "application/json", "x-api-key": env.WYSLEAP_API_KEY },
          body: JSON.stringify({
            siteId: env.WYSLEAP_SITE_ID,
            hits: [{ userAgent, path: url.pathname + url.search, status: response.status }],
          }),
        }).catch(() => {}),
      );
    }
    return response;
  },
};

Next.js (proxy.ts)

In Next.js 16 and later, middleware is called Proxy. Put this in proxy.ts at your project root (in older versions, use middleware.ts and name the function middleware). Set WYSLEAP_SITE_ID and WYSLEAP_API_KEY as environment variables.

javascript

import { NextResponse, type NextFetchEvent, type NextRequest } from "next/server";

const AI_BOTS = /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|anthropic-ai|PerplexityBot|Google-Extended|GoogleOther|Applebot-Extended|Amazonbot|FacebookBot|CCBot|Bytespider|cohere-ai/i;

export function proxy(request: NextRequest, event: NextFetchEvent) {
  const userAgent = request.headers.get("user-agent") || "";
  if (AI_BOTS.test(userAgent)) {
    event.waitUntil(
      fetch("https://wysleap.com/api/telemetry/crawler", {
        method: "POST",
        headers: { "Content-Type": "application/json", "x-api-key": process.env.WYSLEAP_API_KEY! },
        body: JSON.stringify({
          siteId: process.env.WYSLEAP_SITE_ID,
          hits: [{ userAgent, path: request.nextUrl.pathname + request.nextUrl.search }],
        }),
      }).catch(() => {}),
    );
  }
  return NextResponse.next();
}

// Skip static files and images
export const config = {
  matcher: ["/((?!_next/static|_next/image|favicon.ico).*)"],
};

Node.js (Express)

javascript

const AI_BOTS = /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|anthropic-ai|PerplexityBot|Google-Extended|GoogleOther|Applebot-Extended|Amazonbot|FacebookBot|CCBot|Bytespider|cohere-ai/i;

app.use((req, res, next) => {
  const userAgent = req.get("user-agent") || "";
  if (AI_BOTS.test(userAgent)) {
    res.on("finish", () => {
      fetch("https://wysleap.com/api/telemetry/crawler", {
        method: "POST",
        headers: { "Content-Type": "application/json", "x-api-key": process.env.WYSLEAP_API_KEY },
        body: JSON.stringify({
          siteId: process.env.WYSLEAP_SITE_ID,
          hits: [{ userAgent, path: req.originalUrl, status: res.statusCode }],
        }),
      }).catch(() => {});
    });
  }
  next();
});

On a busy site, collect hits for a few seconds and send them in one request of up to 50.

Check it works

Request one of your pages with a crawler user agent, for example curl -A "GPTBot/1.2" https://yoursite.com/, then open SEO & GEO → AI crawler activity in your dashboard. The hit appears within a minute. See the API documentation for the other endpoints.