Prerequisites#

A server or edge function that sees each incoming request (URL, user agent, IP).
Your site token and website id from Bot traffic → Agent Analytics Setup in the app. The snippet there already has both filled in; the examples on this page use YOUR_SITE_TOKEN and YOUR_WEBSITE_ID. See the HTTP API reference for the full payload.

Steps#

1
Detect crawler requests
Check the user agent with a broad pattern like the ones below. You do not need an exact list: Rankealo classifies the user agent again and drops anything that is not a known crawler.
2
Send the event without blocking
POST the event after the response is sent, or in the background. Never make the crawler wait for Rankealo.
3
Keep the token server-side
Store the token in an environment variable. Never put it in browser code.

curl#

Try the endpoint by hand. A 200 with ingested: 1 means the token and website id work.

bash

curl -X POST "https://app.rankealo.ai/api/ai-visibility/crawler-hits/ingest" \
  -H "Authorization: Bearer YOUR_SITE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "websiteId": "YOUR_WEBSITE_ID",
    "domain": "www.your-site.com",
    "href": "https://www.your-site.com/pricing",
    "ai": {
      "userAgent": "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)",
      "ip": "203.0.113.8",
      "statusCode": 200
    }
  }'

Node / Express#

js

// Express middleware: register before your routes.
const CRAWLER_UA = /bot|crawler|spider|chatgpt|gptbot|claude|perplexity|bing|google|applebot|bytespider|ccbot/i;

app.use((req, res, next) => {
  const ua = req.get("user-agent") || "";
  if ((req.method === "GET" || req.method === "HEAD") && CRAWLER_UA.test(ua)) {
    res.on("finish", () => {
      // Fire and forget: never await this in the request path.
      fetch("https://app.rankealo.ai/api/ai-visibility/crawler-hits/ingest", {
        method: "POST",
        headers: {
          "content-type": "application/json",
          authorization: "Bearer " + process.env.RANKEALO_SITE_TOKEN,
        },
        body: JSON.stringify({
          websiteId: process.env.RANKEALO_WEBSITE_ID,
          domain: req.hostname,
          href: req.protocol + "://" + req.get("host") + req.path,
          ai: { userAgent: ua, ip: req.ip, statusCode: res.statusCode },
        }),
      }).catch(() => {});
    });
  }
  next();
});

PHP#

php

<?php
// Call at the end of your front controller, after the response is sent.
function rankealo_report_crawler(): void {
  $ua = $_SERVER['HTTP_USER_AGENT'] ?? '';
  if (!preg_match('/bot|crawler|spider|chatgpt|gptbot|claude|perplexity|bing|google/i', $ua)) return;
  if (!in_array($_SERVER['REQUEST_METHOD'] ?? '', ['GET', 'HEAD'], true)) return;

  if (function_exists('fastcgi_finish_request')) fastcgi_finish_request(); // respond first

  $host = strtolower($_SERVER['HTTP_HOST'] ?? '');
  $path = parse_url($_SERVER['REQUEST_URI'] ?? '/', PHP_URL_PATH) ?: '/';
  $ch = curl_init('https://app.rankealo.ai/api/ai-visibility/crawler-hits/ingest');
  curl_setopt_array($ch, [
    CURLOPT_POST => true,
    CURLOPT_HTTPHEADER => ['Content-Type: application/json', 'Authorization: Bearer YOUR_SITE_TOKEN'],
    CURLOPT_POSTFIELDS => json_encode([
      'websiteId' => 'YOUR_WEBSITE_ID',
      'domain' => $host,
      'href' => 'https://' . $host . $path,
      'ai' => [
        'userAgent' => $ua,
        'ip' => $_SERVER['REMOTE_ADDR'] ?? null,
        'statusCode' => http_response_code(),
      ],
    ]),
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_TIMEOUT_MS => 1500,
  ]);
  curl_exec($ch);
  curl_close($ch);
}

Batching

On busy sites, queue events and send them in batches of up to 500 with a hits array. The format is in the HTTP API reference.

Limitations#

Requests served by a cache or CDN in front of your server never reach your code, so they are not reported.
Hits are marked as claimed identities unless you pass ai.verifiedBot: truefrom a source you trust, such as your CDN's bot verification.
Up to 500 hits per request; larger batches are rejected with a 400.

Confirm data is flowing#

1
In Rankealo, open Bot traffic → Agent Analytics Setup.
2
Press Check now in step 3. Once a crawler has visited, it shows Receiving data with the time of the last hit.
3
Open the Bot traffic tab to see hits by crawler and by page.

The first hit depends on when a crawler next visits, which can take up to 24 hours. To test right away, request a page with a crawler user agent, for example curl -A "GPTBot/1.2" https://www.your-site.com/, then press Check now. A test like this is recorded as a claimed identity, like any hit without CDN verification.

Troubleshooting#

401 Unauthorized+

The token does not match the websiteId. Copy both again from Agent Analytics Setup.

200 but ingested is 0 and skipped is 1+

The host in href is not your Rankealo website, or the user agent is not a known crawler. Use a real crawler user agent such as GPTBot/1.2 to test.

503 AI crawler ingest is disabled+

Ingest is switched off on this Rankealo environment. Contact support.

Any backend (HTTP API)