Avoiding Cache Stampedes at the Edge

This guide is part of Edge Caching & CDN Integration. It solves one concrete problem: stopping a single cache entry’s expiry from turning into thousands of simultaneous origin fetches spread across every PoP in your network.

A stampede — also called a thundering herd or dog-pile — is what happens when a popular cache entry expires and every request that was being served from it arrives at the origin at once. On a single server this is annoying. At the edge it is a different order of problem, because the cache is not one cache: it is hundreds of independent caches that were all filled at the same moment by the same deploy or the same purge, and which therefore all expire at the same moment too. The failure is not caused by traffic growth. It is caused by correlation.

The problem

You deploy, traffic looks fine, and forty seconds later your origin’s p99 goes vertical and your error rate follows it. Or you purge a hot tag and watch the same thing happen instantly. Or your dashboard is healthy for exactly sixty seconds at a time and then spikes, on a metronome, forever.

Three symptoms, one cause. Each of them is a set of edge nodes whose cached copy of the same resource became invalid simultaneously, so the number of concurrent origin requests jumped from roughly one to roughly N, where N is however many PoPs and however many concurrent in-flight requests each of those PoPs happened to be holding. Origins that comfortably serve two requests per second for a resource are asked for six hundred, they queue, latency climbs, the queued edge requests time out, the edge retries, and the retry storm finishes the job.

The tempting fix is stale-while-revalidate, and it does help — but only for entries that already exist. It does nothing for an entry that is genuinely absent, and after a deploy, a purge, a PoP eviction under memory pressure, or a cold region, absence is exactly the state you are in.

Root cause: correlated expiry plus unbounded miss concurrency

Two independent constraints combine to produce a stampede, and you have to address both.

The first is correlated expiry. TTLs at the edge are almost always computed as “now plus a constant.” If a hundred PoPs fill their copy of /api/pricing within the same second — which is exactly what happens after a purge or a deploy, because the first request to each PoP arrives immediately — then a hundred PoPs will expire that copy within the same second, sixty seconds later. Nothing decorrelates them. The pattern is self-reinforcing: each stampede refills every cache simultaneously, which schedules the next stampede simultaneously.

The second is unbounded miss concurrency inside a single node. Even one PoP, taken alone, can produce a herd. When the entry is missing, every concurrently arriving request independently observes a miss and independently starts an origin fetch. A PoP serving 200 requests per second to a resource that takes 400 ms to regenerate will start roughly 80 origin fetches for the same bytes before the first one returns. There is no implicit deduplication; you must write it.

Understanding both matters because the mitigations are different. Jitter and probabilistic early expiration decorrelate across nodes and across time. A single-flight lock deduplicates within a node, and a distributed lock extends that across nodes. Serving stale on error is what stops the residual failure from becoming a user-visible outage. None of the four is sufficient alone.

Step 1 — Decorrelate expiry with TTL jitter

The cheapest fix, and the one to apply first, is to stop every node from choosing the same expiry. Add a bounded random offset when you compute the TTL, so a hundred nodes that fill within the same second spread their expiries over a window instead of a point.

// lib/ttl.ts

export interface TtlPolicy {
  /** Base freshness lifetime in seconds. */
  baseSeconds: number;
  /** Maximum fraction of baseSeconds to add as jitter, e.g. 0.15 = up to +15%. */
  jitterRatio: number;
  /** Extra window during which a stale copy may still be served. */
  staleWhileRevalidateSeconds: number;
  /** Window during which a stale copy may be served if the origin errors. */
  staleIfErrorSeconds: number;
}

export function jitteredTtl(policy: TtlPolicy, random: () => number = Math.random): number {
  const spread = policy.baseSeconds * policy.jitterRatio;
  // Jitter upward only. Jittering downward would shorten freshness below the
  // contract you advertised to clients and to any upper cache tier.
  return Math.round(policy.baseSeconds + random() * spread);
}

export function cacheControlFor(policy: TtlPolicy, random: () => number = Math.random): string {
  const maxAge = jitteredTtl(policy, random);
  return [
    'public',
    `s-maxage=${maxAge}`,
    `stale-while-revalidate=${policy.staleWhileRevalidateSeconds}`,
    `stale-if-error=${policy.staleIfErrorSeconds}`,
  ].join(', ');
}

Jitter upward, never downward. Downward jitter shortens the freshness lifetime you promised, which quietly increases total origin load — the opposite of the goal. A 10-20% upward spread is usually enough: on a 60-second base TTL that scatters expiries across a 6-12 second window, which is long enough to flatten the spike but short enough that no one notices the extra staleness.

Jitter alone does not fix a stampede; it converts one tall spike into a wider, lower plateau. That plateau is still N fetches for the same bytes. Step 2 removes the duplication.

Step 2 — Coalesce concurrent misses with a single-flight lock

Single-flight means: for a given cache key, at most one origin fetch is in flight at a time, and everyone else waits on its result. Inside one isolate this is a map from key to in-flight promise. It costs nothing and eliminates the within-node herd entirely.

// lib/single-flight.ts

type Pending<T> = Promise<T>;

const inFlight = new Map<string, Pending<unknown>>();

/**
 * Runs `work` for `key`, or joins the existing run if one is already active.
 * Module scope, so the map is shared by every request this isolate handles.
 */
export function singleFlight<T>(key: string, work: () => Promise<T>): Promise<T> {
  const existing = inFlight.get(key) as Pending<T> | undefined;
  if (existing) return existing;

  const run = (async () => {
    try {
      return await work();
    } finally {
      // Always clear, including on rejection, so a failed fetch does not
      // pin every subsequent request to the same rejected promise.
      inFlight.delete(key);
    }
  })();

  inFlight.set(key, run);
  return run;
}

export function inFlightCount(): number {
  return inFlight.size;
}

The finally block is the part people get wrong. If you delete the entry only on success, one origin failure poisons the key: every later request joins the already-rejected promise and fails instantly, forever, until the isolate recycles. Deleting in finally means the next request after a failure gets a fresh attempt.

Cross-node coalescing needs shared state. A Durable Object gives you a strongly consistent lock holder per key; KV does not, because its writes are eventually consistent and two PoPs can both believe they won. Use the DO when the origin fetch is genuinely expensive and correctness of the lock matters, and accept per-isolate single-flight otherwise — it already removes the large majority of duplicate fetches, since concurrency within a PoP is what multiplies N into the thousands.

// lib/distributed-lock.ts

export interface LockEnv {
  STAMPEDE_LOCK: DurableObjectNamespace;
}

/**
 * Attempts to acquire a short lease for `key`. Returns true if this caller
 * should perform the origin fetch, false if another node already is.
 */
export async function tryAcquire(env: LockEnv, key: string, leaseMs = 5_000): Promise<boolean> {
  const id = env.STAMPEDE_LOCK.idFromName(key);
  const stub = env.STAMPEDE_LOCK.get(id);
  const res = await stub.fetch('https://lock/acquire', {
    method: 'POST',
    body: JSON.stringify({ leaseMs }),
  });
  const { acquired } = (await res.json()) as { acquired: boolean };
  return acquired;
}

Keep the lease short — a few seconds, comfortably longer than a normal origin fetch but far shorter than a human notices. A lease that outlives the isolate holding it must expire on its own; never rely on a release call, because the isolate can be evicted mid-fetch and the release will never arrive. The losers of the lock race should not block on the winner across the network. They should serve stale if they have it, and only queue briefly if they do not.

Step 3 — Expire probabilistically before the TTL with XFetch

Jitter decorrelates the scheduled expiry, but every node still waits until its entry is dead before refreshing. The XFetch approach, from Vattani, Chierichetti and Lowenstein’s work on optimal probabilistic cache refresh, does something smarter: as an entry approaches expiry, each read has a small and growing chance of triggering an early refresh, weighted by how long the last regeneration actually took.

The rule is: refresh now if

now - delta * beta * ln(random()) >= expiry

where delta is the measured cost of the last recomputation in milliseconds, beta is a tuning factor (1.0 is the neutral default; higher refreshes earlier), and random() is uniform in (0, 1). Because ln(random()) is negative, the left-hand term pushes now forward by a random amount scaled by the recompute cost. Expensive resources therefore start trying to refresh earlier — which is exactly right, since they are the ones that hurt when they stampede.

// lib/xfetch.ts

export interface CachedEnvelope<T> {
  value: T;
  /** Absolute epoch ms at which the entry becomes stale. */
  expiresAt: number;
  /** How long the last origin regeneration took, in ms. */
  deltaMs: number;
}

/**
 * Returns true if this reader should proactively regenerate the entry,
 * even though it has not formally expired yet.
 */
export function shouldEarlyExpire<T>(
  entry: CachedEnvelope<T>,
  beta = 1,
  now = Date.now(),
  random: () => number = Math.random,
): boolean {
  if (entry.deltaMs <= 0) return now >= entry.expiresAt;
  // ln of a uniform (0,1) sample is negative, so this shifts `now` forward.
  const gap = entry.deltaMs * beta * Math.log(random());
  return now - gap >= entry.expiresAt;
}

Two properties make this worth the six lines. First, it is independent per reader, so it decorrelates nodes without any coordination at all. Second, it degrades gracefully: as now passes expiresAt the probability rises to certainty, so a resource nobody reads still refreshes on its first read after expiry rather than being pinned stale.

Set beta to 1 and leave it there unless you measure a reason to move. Raising it above 1 buys earlier refreshes at the cost of more total origin traffic; dropping it below 1 does the reverse and edges you back toward a synchronized expiry.

Step 4 — Serve stale on error, and treat a cold cache separately

stale-while-revalidate covers the case where you have a stale copy and the origin is healthy. stale-if-error covers the case where you have a stale copy and the origin is not healthy — and during a stampede, that is precisely the state the origin enters. Setting both is what keeps a stampede from escalating into an outage: as the origin starts failing, edge nodes fall back to the last good body rather than adding retries to the pile.

What neither directive covers is a genuinely empty cache. After a deploy, a hard purge, or a PoP eviction there is no stale copy to serve, so every one of those requests must reach the origin. This is the single most important thing to internalize about stampedes at the edge: stale-while-revalidate is a refresh strategy, not a fill strategy. Cold-cache protection comes from single-flight coalescing plus, where it matters, a lock — and from not hard-purging hot keys when a version bump would do, as covered in tag and surrogate-key invalidation.

Here is the whole policy assembled into one middleware path.

// middleware.ts
import { cacheControlFor, type TtlPolicy } from './lib/ttl';
import { singleFlight } from './lib/single-flight';
import { shouldEarlyExpire, type CachedEnvelope } from './lib/xfetch';

const POLICY: TtlPolicy = {
  baseSeconds: 60,
  jitterRatio: 0.15,
  staleWhileRevalidateSeconds: 120,
  staleIfErrorSeconds: 600,
};

interface Env {
  CACHE: KVNamespace;
}

async function regenerate(key: string, url: string, env: Env): Promise<CachedEnvelope<string>> {
  const startedAt = Date.now();
  const res = await fetch(url, { headers: { 'x-edge-refresh': '1' } });
  if (!res.ok) throw new Error(`origin ${res.status}`);
  const value = await res.text();
  const deltaMs = Date.now() - startedAt;

  const envelope: CachedEnvelope<string> = {
    value,
    expiresAt: Date.now() + POLICY.baseSeconds * 1000,
    deltaMs,
  };
  await env.CACHE.put(key, JSON.stringify(envelope), {
    // Keep the raw entry alive well past freshness so stale serving works.
    expirationTtl: POLICY.baseSeconds + POLICY.staleIfErrorSeconds,
  });
  return envelope;
}

export default async function middleware(
  request: Request,
  env: Env,
  ctx: { waitUntil(p: Promise<unknown>): void },
): Promise<Response> {
  const url = new URL(request.url);
  const key = `page:${url.pathname}`;

  const raw = await env.CACHE.get(key, 'text');
  const entry: CachedEnvelope<string> | null = raw ? JSON.parse(raw) : null;

  // Cold: nothing to serve. Coalesce and wait.
  if (!entry) {
    const filled = await singleFlight(key, () => regenerate(key, url.toString(), env));
    return new Response(filled.value, {
      headers: { 'cache-control': cacheControlFor(POLICY), 'x-cache': 'MISS' },
    });
  }

  // Warm but due for refresh: serve stale immediately, refresh behind the response.
  if (shouldEarlyExpire(entry)) {
    ctx.waitUntil(
      singleFlight(key, () => regenerate(key, url.toString(), env)).catch(() => {
        // Swallow: stale-if-error already covered the user-visible path.
      }),
    );
    return new Response(entry.value, {
      headers: { 'cache-control': cacheControlFor(POLICY), 'x-cache': 'STALE' },
    });
  }

  return new Response(entry.value, {
    headers: { 'cache-control': cacheControlFor(POLICY), 'x-cache': 'HIT' },
  });
}

Note that the refresh path never blocks the response, and that the refresh itself goes through singleFlight, so a hundred concurrent readers that all roll an early expiry produce one origin fetch, not a hundred. The x-cache header is there so you can distinguish the three paths in production without guessing; see reading cache status headers at the edge for how providers stack their own status headers alongside yours.

Matcher configuration

Restrict the pattern to routes that actually have an expensive origin behind them. Running coalescing logic on static assets adds latency and CPU for no benefit.

export const config = {
  runtime: 'edge',
  matcher: [
    '/api/pricing/:path*',
    '/api/catalog/:path*',
    '/((?!_next/static|_next/image|favicon.ico|robots.txt).*)',
  ],
};

Local versus production divergence

The third row is why stampedes are almost never reproduced locally. With a concurrency of one, single-flight is a no-op and every strategy looks identical. Reproduce the failure by driving concurrent load against a stub origin with an artificial 400 ms delay, and count origin hits — not response times.

Validating with Vitest

The whole point of factoring jitteredTtl, singleFlight and shouldEarlyExpire as pure-ish functions with injectable randomness is that they become deterministically testable. Inject random, count origin calls, and assert the count rather than the latency.

// stampede.test.ts
import { describe, expect, it, vi } from 'vitest';
import { jitteredTtl, cacheControlFor } from './lib/ttl';
import { singleFlight, inFlightCount } from './lib/single-flight';
import { shouldEarlyExpire, type CachedEnvelope } from './lib/xfetch';

const POLICY = {
  baseSeconds: 60,
  jitterRatio: 0.15,
  staleWhileRevalidateSeconds: 120,
  staleIfErrorSeconds: 600,
};

describe('ttl jitter', () => {
  it('never shortens the base lifetime', () => {
    for (const r of [0, 0.01, 0.5, 0.99]) {
      expect(jitteredTtl(POLICY, () => r)).toBeGreaterThanOrEqual(POLICY.baseSeconds);
    }
  });

  it('caps the spread at the configured ratio', () => {
    expect(jitteredTtl(POLICY, () => 1)).toBe(69); // 60 + 15%
  });

  it('emits stale-while-revalidate and stale-if-error together', () => {
    const header = cacheControlFor(POLICY, () => 0);
    expect(header).toContain('stale-while-revalidate=120');
    expect(header).toContain('stale-if-error=600');
  });
});

describe('single flight', () => {
  it('collapses concurrent misses into one origin call', async () => {
    const origin = vi.fn(async () => {
      await new Promise((r) => setTimeout(r, 20));
      return 'body';
    });

    const results = await Promise.all(
      Array.from({ length: 50 }, () => singleFlight('key:a', origin)),
    );

    expect(origin).toHaveBeenCalledTimes(1);
    expect(new Set(results)).toEqual(new Set(['body']));
  });

  it('does not poison the key after a failure', async () => {
    const origin = vi
      .fn()
      .mockRejectedValueOnce(new Error('origin 503'))
      .mockResolvedValueOnce('recovered');

    await expect(singleFlight('key:b', origin)).rejects.toThrow('origin 503');
    expect(inFlightCount()).toBe(0);
    await expect(singleFlight('key:b', origin)).resolves.toBe('recovered');
  });

  it('keeps distinct keys independent', async () => {
    const origin = vi.fn(async () => 'x');
    await Promise.all([singleFlight('k1', origin), singleFlight('k2', origin)]);
    expect(origin).toHaveBeenCalledTimes(2);
  });
});

describe('probabilistic early expiration', () => {
  const entry: CachedEnvelope<string> = {
    value: 'v',
    expiresAt: 10_000,
    deltaMs: 400,
  };

  it('almost never refreshes when the entry is young', () => {
    // random near 1 makes ln(random) near 0, so the gap is negligible.
    expect(shouldEarlyExpire(entry, 1, 5_000, () => 0.999)).toBe(false);
  });

  it('refreshes early when the recompute cost is high and the roll is low', () => {
    expect(shouldEarlyExpire(entry, 1, 9_500, () => 0.0001)).toBe(true);
  });

  it('always refreshes once the entry is past expiry', () => {
    expect(shouldEarlyExpire(entry, 1, 10_001, () => 0.999)).toBe(true);
  });

  it('scales the early window with beta', () => {
    const now = 9_600;
    expect(shouldEarlyExpire(entry, 0.1, now, () => 0.2)).toBe(false);
    expect(shouldEarlyExpire(entry, 4, now, () => 0.2)).toBe(true);
  });
});

The second single-flight test is the one that catches real production bugs. A version of singleFlight that deletes the map entry only on success passes every other test in this file and then locks out an entire key for the lifetime of the isolate the first time the origin returns a 503.

Common pitfalls and resolutions

Origin load spikes on a fixed period, forever — TTLs are computed as a constant and every fill is correlated. Add upward jitter and probabilistic early expiration; the period should dissolve into a flat line within one TTL cycle.

Jitter added, spike shrank but did not disappear — you decorrelated across nodes but not within them. Each node is still issuing tens of duplicate fetches. Add singleFlight.

Every request for one key fails instantly after a single origin error — the in-flight map entry is deleted only on resolve. Delete it in finally.

A purge immediately melts the origin — a hard purge of a hot tag is the worst case, because it removes the stale copy that stale-if-error would have used. Prefer soft purge or a version bump for hot keys.

Refresh work counted against the user’s request — the background refresh is being awaited instead of handed to ctx.waitUntil. The user should never wait for a refresh they did not need.

Distributed lock held forever after an eviction — the lock relies on an explicit release. Use a lease with a server-side expiry so an evicted isolate cannot deadlock the key.

beta tuned up to “refresh sooner,” origin load rose — raising beta trades origin traffic for freshness. If freshness is the goal, shorten the base TTL instead; that at least remains predictable.

Production deployment checklist

• Every computed TTL passes through the jitter helper, jittering upward only
• stale-while-revalidate and stale-if-error are both present on cacheable responses
• Concurrent misses for one key are coalesced by an isolate-scoped in-flight map
• The in-flight entry is cleared in a finally, not only on success
• Background refreshes run under ctx.waitUntil and never block a response
• Probabilistic early expiration uses the measured recompute cost, not a guess
• Hot keys are invalidated by version bump or soft purge, not a hard purge
• Any distributed lock uses a short, self-expiring lease
• An x-cache header distinguishes HIT, STALE and MISS in production logs
• A load test measures origin fetch count under concurrency, not just latency

Frequently Asked Questions

Related

• Stale-while-revalidate at the edge
• Tuning Cache-Control max-age for edge
• Multi-tier CDN cache architecture
• Tag and surrogate-key invalidation
• Debugging low cache hit ratios at the edge