Server-side AI crawler detection

How it worksCopy Anchor Link

Your server checks each request's user agent. When it matches a known AI crawler, it sends the page path to WysLeap in the background, after the response, so visitors never wait for it. WysLeap stores only AI crawler hits; anything else is dropped.

• Endpoint: POST https://wysleap.com/api/telemetry/crawler
• Header: x-api-key with your site's API key (the same key as in your tracking script)
• Body: { siteId, hits: [{ userAgent, path, ip?, status?, timestamp? }] }, up to 50 hits per request
• ip: the client IP, from your platform's trusted header (as in the examples below). Crawler names are easy to fake: WysLeap checks the IP against the ranges OpenAI, Perplexity, Google and Apple publish for their crawlers and drops hits that don't match. The IP is only checked, never stored. Without it, hits are kept but can't be verified.

Requests for secret files and dev-server paths (/.env, /.git, /@fs/…) are dropped: they come from scanners using crawler names.

Once a site sends server-side hits, the AI crawler charts use them instead of the script's detections, so crawlers that also run JavaScript aren't counted twice. Each stored hit counts toward your monthly event allowance.

ExamplesCopy Anchor Link

const AI_BOTS = /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|anthropic-ai|PerplexityBot|Perplexity-User|Google-Extended|GoogleOther|Applebot-Extended|Amazonbot|FacebookBot|CCBot|Bytespider|cohere-ai/i;

export default {
  async fetch(request, env, ctx) {
    const response = await fetch(request);
    const userAgent = request.headers.get("user-agent") || "";
    if (AI_BOTS.test(userAgent)) {
      const url = new URL(request.url);
      ctx.waitUntil(
        fetch("https://wysleap.com/api/telemetry/crawler", {
          method: "POST",
          headers: { "Content-Type": "application/json", "x-api-key": env.WYSLEAP_API_KEY },
          body: JSON.stringify({
            siteId: env.WYSLEAP_SITE_ID,
            hits: [{
              userAgent,
              path: url.pathname + url.search,
              status: response.status,
              ip: request.headers.get("cf-connecting-ip"),
            }],
          }),
        }).catch(() => {}),
      );
    }
    return response;
  },
};

Check it worksCopy Anchor Link

Request one of your pages with a crawler user agent, for example curl -A "GPTBot/1.2" https://yoursite.com/, then open SEO & GEO → AI crawler activity in your dashboard. The hit appears within a minute. See the API documentation for the other endpoints.