Short-Circuiting Bot Traffic in Edge Middleware

This guide is part of Implementing Early Returns in Edge Middleware. It builds a scoring function that runs in a few microseconds on every request, decides whether the caller is worth serving, and exits the chain before any expensive work happens — without ever blocking a search engine.

The economics are what make this an edge problem. A scraper costs the operator a fraction of a cent per request and costs you a database query, a render, and an origin round trip. If the rejection happens at your origin, you have already paid. If it happens in the isolate at the PoP nearest the scraper, you paid for a few hundred microseconds of CPU and no bandwidth beyond the response. That gap is the entire justification for putting the check here rather than in your application.

The problem

You add a bot filter, and one of three things goes wrong.

The filter matches on user-agent strings, and within a week the scrapers all identify as Chrome 141 on macOS. Your block rate drops to zero and your traffic does not, because the string is a single line of configuration on their side.

Or the filter is more aggressive and it works — until an SEO report shows organic traffic collapsing. You blocked Googlebot, because its user-agent looked automated to a heuristic that could not tell it from the scrapers copying it.

Or you get the classification right but the response is wrong: you return 403 to a rate-abusing but otherwise legitimate integration, its retry logic treats that as permanent, and a customer’s nightly sync silently stops working three weeks before anyone notices.

Root cause: the only free signals are the ones the client cannot forge cheaply

Every bot-detection technique sits somewhere on a curve of cost against evasion difficulty. At the edge you have a hard CPU ceiling — single-digit milliseconds on Cloudflare Workers’ free tier, 50 ms on Vercel Edge Middleware — and no ability to run a browser challenge inline. That rules out fingerprinting scripts, behavioural analysis over a session, and anything requiring a model. What is left is the request itself: its headers, their order, the connecting network, and how often this shape of request has arrived recently.

The user-agent is the worst of those signals because it is a single string under complete client control with zero cost to change. It is not worthless as telemetry — it tells you what a cooperative client claims to be — but as a control it is decorative. Every serious scraping toolkit ships with a realistic user-agent by default.

What is expensive for an attacker to fake is consistency across many signals at once. A real browser sends Accept, Accept-Language, Accept-Encoding, Sec-Fetch-Site, Sec-Fetch-Mode, Sec-Fetch-Dest and Upgrade-Insecure-Requests on a navigation, in a stable order, with values that agree with the claimed user-agent. A script using a low-level HTTP client sends a user-agent and Accept: */* and nothing else. Making a scraper emit a fully coherent header set is possible; making it do so through a proxy pool at scale, while also matching the rate profile of a human, costs real engineering time. Scoring is how you charge them for it.

Step 1 — Extract the cheap signals

Everything in this step reads from the Request object and allocates nothing beyond a few short strings. It costs well under a hundred microseconds.

// lib/signals.ts
export interface Signals {
  /** Client sent no Accept header at all — no real browser does this. */
  missingAccept: boolean;
  /** Accept is the wildcard a low-level HTTP client sends by default. */
  wildcardAccept: boolean;
  /** How many of the headers a real browser navigation always sends are absent. */
  missingBrowserHeaders: number;
  /** Sec-Fetch-* present but internally inconsistent. */
  incoherentFetchMetadata: boolean;
  /** Header names arrived in an order no browser produces. */
  impossibleOrdering: boolean;
  /** Claimed user-agent, kept for logging only. */
  userAgent: string;
  /** Autonomous system number of the connecting network, when the platform exposes it. */
  asn: number | null;
}

const BROWSER_NAV_HEADERS = [
  'accept-language',
  'accept-encoding',
  'sec-fetch-site',
  'sec-fetch-mode',
  'sec-fetch-dest',
  'upgrade-insecure-requests',
] as const;

export function extractSignals(request: Request, asn: number | null): Signals {
  const h = request.headers;
  const accept = h.get('accept');

  const missingBrowserHeaders = BROWSER_NAV_HEADERS.reduce(
    (n, name) => (h.get(name) === null ? n + 1 : n),
    0,
  );

  const dest = h.get('sec-fetch-dest');
  const mode = h.get('sec-fetch-mode');
  // A top-level document navigation is always mode=navigate, dest=document.
  // Anything claiming one without the other is a partially spoofed header set.
  const incoherentFetchMetadata =
    (mode === 'navigate' && dest !== null && dest !== 'document') ||
    (dest === 'document' && mode !== null && mode !== 'navigate');

  return {
    missingAccept: accept === null,
    wildcardAccept: accept === '*/*',
    missingBrowserHeaders,
    incoherentFetchMetadata,
    impossibleOrdering: hasImpossibleOrdering(request),
    userAgent: h.get('user-agent') ?? '',
    asn,
  };
}

/**
 * Browsers emit a stable header order. Host always precedes User-Agent, and
 * Sec-Fetch-* always follows User-Agent. A client that assembles headers from
 * a dictionary usually emits them alphabetically or in insertion order, which
 * produces sequences no browser ever sends.
 */
function hasImpossibleOrdering(request: Request): boolean {
  const names = [...request.headers.keys()];
  const at = (n: string) => names.indexOf(n);
  const ua = at('user-agent');
  const secSite = at('sec-fetch-site');
  if (ua === -1 || secSite === -1) return false;
  return secSite < ua;
}

A caveat on ordering: the Headers iteration order you observe in an edge runtime is normalised by the platform on some providers and preserved on others. Verify empirically before you weight it heavily — the check above is written to return false rather than fire when the evidence is not available.

The ASN comes from platform-specific request metadata: request.cf.asn on Cloudflare Workers, the x-vercel-ip-as-number header on Vercel, and context.geo plus edge headers on Netlify. A small deny list of hosting ASNs known for scraping traffic is a strong signal, because residential proxy capacity is genuinely expensive and most scrapers run from cloud ranges.

Step 2 — Score, do not classify

Boolean classification is brittle: one false positive blocks a real customer. A score with two thresholds gives you a middle band where you can degrade rather than deny.

// lib/score.ts
import type { Signals } from './signals';

/** Higher is more suspicious. Zero is a clean request. */
export function scoreRequest(s: Signals, recentRate: number): number {
  let score = 0;

  // Nothing that renders HTML omits Accept entirely.
  if (s.missingAccept) score += 40;
  else if (s.wildcardAccept) score += 15;

  // Each absent browser header is weak on its own; four of them is not.
  score += Math.min(s.missingBrowserHeaders * 8, 40);

  if (s.incoherentFetchMetadata) score += 30;
  if (s.impossibleOrdering) score += 25;

  // Datacentre origin. Legitimate server-to-server integrations also land
  // here, which is exactly why this alone must never be decisive.
  if (s.asn !== null && HOSTING_ASNS.has(s.asn)) score += 20;

  // Rate is the strongest single input, and the only one that measures
  // behaviour rather than shape.
  if (recentRate > 600) score += 45;
  else if (recentRate > 200) score += 25;
  else if (recentRate > 60) score += 10;

  return score;
}

const HOSTING_ASNS = new Set<number>([
  // Illustrative. Maintain this from your own traffic analysis, not a blog post.
  14618, 16509, 15169, 8075, 14061, 16276, 24940, 63949,
]);

export type Verdict = 'allow' | 'throttle' | 'block';

export function verdictFor(score: number): Verdict {
  if (score >= 90) return 'block';
  if (score >= 55) return 'throttle';
  return 'allow';
}

The weights are a starting point, not a law. Run the scorer in shadow mode first — compute the score, log it, and take no action — for at least a week, then look at the distribution. Real traffic clusters near zero with a long thin tail; if your block threshold sits inside the fat part of the distribution you are about to have a bad day.

Step 3 — Allow-list real crawlers by verified reverse DNS

Search engines must never be scored. They will look automated to every heuristic above, because they are: no Accept-Language, no Sec-Fetch-*, datacentre ASN, high rate. Blocking them removes you from search results, and the damage takes weeks to reverse.

The wrong fix is trusting the user-agent, since Googlebot/2.1 is a string anyone can send. The right fix is the verification protocol the major engines publish: take the connecting IP, do a reverse DNS lookup to get a hostname, check the hostname ends in the engine’s official domain, then do a forward lookup on that hostname and confirm it resolves back to the original IP. Both directions are required — a reverse record alone can be set by whoever controls the IP’s in-addr.arpa delegation.

Edge runtimes have no DNS resolver API, so both lookups go over DNS-over-HTTPS with fetch, and the result is cached so the cost is paid once per IP rather than once per request.

// lib/crawler.ts
const DOH = 'https://cloudflare-dns.com/dns-query';

const VERIFIED_SUFFIXES: Record<string, readonly string[]> = {
  googlebot: ['.googlebot.com', '.google.com'],
  bingbot: ['.search.msn.com'],
  applebot: ['.applebot.apple.com'],
  duckduckbot: ['.duckduckgo.com'],
};

interface DohAnswer { name: string; type: number; data: string }

async function query(name: string, type: 'PTR' | 'A' | 'AAAA'): Promise<string[]> {
  const res = await fetch(`${DOH}?name=${encodeURIComponent(name)}&type=${type}`, {
    headers: { accept: 'application/dns-json' },
    // Short budget: a slow resolver must never hold the request open.
    signal: AbortSignal.timeout(400),
  });
  if (!res.ok) return [];
  const body = (await res.json()) as { Answer?: DohAnswer[] };
  return (body.Answer ?? []).map((a) => a.data.replace(/\.$/, ''));
}

function reverseName(ip: string): string {
  if (ip.includes(':')) return ''; // IPv6 PTR construction omitted for brevity.
  return ip.split('.').reverse().join('.') + '.in-addr.arpa';
}

/**
 * Forward-confirmed reverse DNS. Returns the crawler key when the IP genuinely
 * belongs to that engine, otherwise null. The user-agent only selects which
 * suffix list to check; it never grants trust on its own.
 */
export async function verifiedCrawler(ip: string, userAgent: string): Promise<string | null> {
  const ua = userAgent.toLowerCase();
  const key = Object.keys(VERIFIED_SUFFIXES).find((k) => ua.includes(k));
  if (!key) return null;

  const ptr = reverseName(ip);
  if (!ptr) return null;

  const hostnames = await query(ptr, 'PTR');
  const host = hostnames.find((h) =>
    VERIFIED_SUFFIXES[key].some((suffix) => h.endsWith(suffix)),
  );
  if (!host) return null;

  const forward = await query(host, ip.includes(':') ? 'AAAA' : 'A');
  return forward.includes(ip) ? key : null;
}

Cache the verdict. A Map at module scope covers the isolate; a KV namespace with a one-day TTL covers the PoP and survives isolate churn. Crawler IP ranges change slowly, so a day is conservative. Without caching you have added two network round trips to a fraction of your traffic, and a resolver outage becomes your outage — which is why the code above treats any failure as “not verified” and then falls through to scoring rather than blocking outright.

Step 4 — Return 429 with Retry-After, not 403

The verdict determines the response, and the two rejections are not interchangeable.

403 means never, and a well-built client will not retry. Use it only for the block band, where you are confident the traffic is abusive and you want it to stop permanently.

429 means not right now, and it is the correct answer for the throttle band. Paired with Retry-After, it tells a cooperative client exactly when to come back, which is the difference between a customer’s nightly sync backing off gracefully and it hard-failing. Search engines in particular treat 429 as a crawl-rate signal and reduce their request rate; they treat 403 as a permanent removal instruction.

// lib/reject.ts
import type { Verdict } from './score';

export function rejectionFor(verdict: Verdict, retryAfterSeconds = 60): Response {
  if (verdict === 'block') {
    return new Response('Forbidden', {
      status: 403,
      headers: { 'cache-control': 'no-store', 'content-type': 'text/plain; charset=utf-8' },
    });
  }

  return new Response('Too Many Requests', {
    status: 429,
    headers: {
      // Seconds, not a date. Both are legal; seconds avoid clock-skew disputes.
      'retry-after': String(retryAfterSeconds),
      'cache-control': 'no-store',
      'content-type': 'text/plain; charset=utf-8',
    },
  });
}

Both carry cache-control: no-store for the same reason a 401 does: the rejection is a property of the caller, not of the URL, and a shared tier that stores it will serve it to somebody innocent. The same hazard is covered in detail in returning 401 responses from edge middleware.

Step 5 — Assemble the middleware

// middleware.ts
import { extractSignals } from './lib/signals';
import { scoreRequest, verdictFor } from './lib/score';
import { verifiedCrawler } from './lib/crawler';
import { rejectionFor } from './lib/reject';

const crawlerCache = new Map<string, string | null>();

export default async function middleware(request: Request): Promise<Response> {
  const ip = request.headers.get('x-forwarded-for')?.split(',')[0].trim() ?? '';
  const ua = request.headers.get('user-agent') ?? '';

  // Verified crawlers bypass scoring entirely, before anything else runs.
  if (!crawlerCache.has(ip)) {
    crawlerCache.set(ip, await verifiedCrawler(ip, ua).catch(() => null));
  }
  if (crawlerCache.get(ip)) return fetch(request);

  const asnHeader = request.headers.get('x-vercel-ip-as-number');
  const signals = extractSignals(request, asnHeader ? Number(asnHeader) : null);
  const rate = await recentRateFor(fingerprint(ip, signals));
  const score = scoreRequest(signals, rate);
  const verdict = verdictFor(score);

  console.log(JSON.stringify({
    event: 'bot.score', score, verdict, rate, ua, path: new URL(request.url).pathname,
  }));

  if (verdict !== 'allow') return rejectionFor(verdict, verdict === 'throttle' ? 60 : 3600);
  return fetch(request);
}

/** Coarse identity: IP plus the stable shape of the header set. */
function fingerprint(ip: string, s: { userAgent: string; missingBrowserHeaders: number }): string {
  return `${ip}|${s.missingBrowserHeaders}|${s.userAgent.slice(0, 48)}`;
}

recentRateFor is a counter read; the implementations are covered in per-IP rate limiting with Cloudflare KV and, for stricter accounting, sliding window rate limiting with Durable Objects.

The matcher

export const config = {
  runtime: 'edge',
  matcher: [
    // Never score robots.txt or the sitemap. A crawler that cannot read
    // robots.txt cannot obey it, and a throttled sitemap looks like an outage.
    '/((?!robots.txt|sitemap.xml|_next/static|_next/image|favicon.ico|.well-known).*)',
  ],
};

Local versus production divergence

The ASN row causes most of the surprises. Locally every request scores twenty points lower than it will in production, so a threshold tuned against local traffic will be too aggressive the moment it deploys. Tune against shadow-mode production logs, never against local requests.

Validating with Vitest

The scorer is pure, which is the point of separating it from the middleware. It tests exhaustively without any network or runtime emulation.

// score.test.ts
import { describe, expect, it } from 'vitest';
import { extractSignals } from './lib/signals';
import { scoreRequest, verdictFor } from './lib/score';

const BROWSER_HEADERS = {
  accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
  'accept-language': 'en-GB,en;q=0.9',
  'accept-encoding': 'gzip, deflate, br',
  'sec-fetch-site': 'none',
  'sec-fetch-mode': 'navigate',
  'sec-fetch-dest': 'document',
  'upgrade-insecure-requests': '1',
  'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36',
};

const score = (headers: Record<string, string>, asn: number | null = null, rate = 5) =>
  scoreRequest(extractSignals(new Request('https://app.test/', { headers }), asn), rate);

describe('scoreRequest', () => {
  it('scores a real browser navigation at zero', () => {
    expect(score(BROWSER_HEADERS)).toBe(0);
  });

  it('scores a bare scripted request into the block band', () => {
    const s = score({ 'user-agent': 'python-requests/2.32' });
    expect(verdictFor(s)).toBe('block');
  });

  it('treats a wildcard Accept as weaker evidence than a missing one', () => {
    const missing = score({ 'user-agent': 'x' });
    const wildcard = score({ 'user-agent': 'x', accept: '*/*' });
    expect(wildcard).toBeLessThan(missing);
  });

  it('flags Sec-Fetch metadata that contradicts itself', () => {
    const coherent = score(BROWSER_HEADERS);
    const spoofed = score({ ...BROWSER_HEADERS, 'sec-fetch-dest': 'image' });
    expect(spoofed - coherent).toBe(30);
  });

  it('never blocks on a hosting ASN alone', () => {
    // A well-formed server-to-server call from a cloud range stays allowed.
    expect(verdictFor(score(BROWSER_HEADERS, 16509))).toBe('allow');
  });

  it('escalates a clean browser shape only when the rate is extreme', () => {
    expect(verdictFor(score(BROWSER_HEADERS, null, 50))).toBe('allow');
    expect(verdictFor(score(BROWSER_HEADERS, null, 250))).toBe('allow');
    expect(verdictFor(score({ 'user-agent': 'x', accept: '*/*' }, 16509, 700))).toBe('block');
  });

  it('is deterministic for identical inputs', () => {
    expect(score(BROWSER_HEADERS, 16509, 100)).toBe(score(BROWSER_HEADERS, 16509, 100));
  });
});

The fifth test is the guard rail that matters most. “Never blocks on a hosting ASN alone” is the assertion that stops someone raising the ASN weight to 70 in a late-night incident and taking out every server-side integration your customers have built.

Common pitfalls and resolutions

Organic search traffic collapses after deployment — a crawler was scored and blocked. Verify crawlers by forward-confirmed reverse DNS before scoring, and exclude robots.txt and the sitemap from the matcher.

Block rate drops to zero within a week — you weighted the user-agent. Move it to logging only and score header coherence, ordering and rate instead.

A customer’s nightly integration silently stopped — you returned 403 where 429 belonged. Reserve 403 for the block band and always pair 429 with Retry-After.

CPU limit exceeded on Workers — the DoH lookup is running for every request rather than only for claimed crawlers, and its result is not cached. Gate it on the user-agent hint and cache per IP.

Legitimate users in one country all get throttled — a carrier-grade NAT range shares one IP among thousands of subscribers, so the per-IP rate is meaningless there. Fingerprint on IP plus header shape, not IP alone.

Rejections served to innocent users — the 429 or 403 was cached by a shared tier. Set cache-control: no-store on every rejection.

Score distribution looks nothing like the local one — ASN metadata is absent locally, so every request scores lower. Tune thresholds from shadow-mode production logs.

Production deployment checklist

• The scorer ran in shadow mode for at least a week before enforcing
• The user-agent contributes zero points to the score
• Crawlers are verified by forward-confirmed reverse DNS, not by user-agent
• Crawler verification results are cached per IP with a bounded TTL
• A resolver failure falls back to scoring, never to blocking
• robots.txt and sitemap.xml are excluded from the matcher
• The throttle band returns 429 with a Retry-After in seconds
• 403 is reserved for the highest-confidence block band
• Every rejection carries cache-control: no-store
• No single signal can push a request across the block threshold alone
• Score, verdict and rate are logged for every scored request
• Vitest covers a clean browser, a bare script, and the ASN-alone guard rail

Frequently Asked Questions

Related

• Implementing early returns in edge middleware
• Rate limiting and abuse prevention at the edge
• Per-IP rate limiting with Cloudflare KV
• Returning 401 responses from edge middleware
• Structured logging for edge functions