Cache stampede: why it happens and how to stop it

What is a cache stampede?

A cache stampede happens when many requests discover the same missing or expired cache entry before one request has finished rebuilding it. Every request falls through to the origin, often a database or a slower upstream API, and repeats the same expensive work.

The problem is also called a thundering herd. In a cache, the defining feature is correlation: the requests do not miss independently over time. They miss together because one event, such as a hot key expiring, makes the cached value unavailable to all of them at once.

If a key takes 1.5 seconds to recompute and 50 requests arrive during that interval, a basic cache-aside implementation can start 50 recomputations. A cache hit rate measured over a full day can still look excellent. The short miss window is enough to exhaust the database pool and delay unrelated queries.

The failure sequence

The usual cache-aside path is safe under low concurrency:

1. Read the key from the cache.
2. On a miss, query the database.
3. Store the result with a TTL.
4. Return the result.

Concurrency changes step two. A request cannot read a value that another request has not stored yet. During the rebuild window, every new request sees the same miss and starts the same query.

That makes recomputation time as important as request rate. A five-millisecond query leaves a narrow overlap window. A 1.5-second report query gives hundreds of requests time to join the herd.

What we measured in the Torollo lab

The Torollo incident uses one hot catalog key, a query that takes about 1.5 seconds, 50 concurrent requests and the default PostgreSQL client pool of 10 connections. The numbers below are observations from that fixed local setup, not general performance claims.

The first result is worse than “50 requests cause 50 queries.” Ten connections process the work in roughly five waves. The fiftieth response waits for the earlier waves even though every query asks for the same catalog.

This is also why a larger pool is not an automatic fix. A larger pool may move the queue from the application into PostgreSQL while increasing the amount of simultaneous work the database must perform.

Why TTL alone does not prevent a stampede

TTL controls how long a value remains reusable. It does not coordinate the requests that arrive after expiry.

Adding random jitter to TTLs helps when many different keys were created together and would otherwise expire together. It does much less for one hot key. Every request to that key still observes the same expiry time.

Longer TTLs make cliffs less frequent but increase staleness. Shorter TTLs reduce the stale window but create more cliffs. The frequency changes; the coordination problem remains.

Approach 1: request coalescing

Request coalescing lets one request rebuild a key while concurrent requests share or wait for that work. Inside one process, this is often called single-flight. Across several application instances, the coordination point can be a distributed lock or lease.

The first request becomes the leader. The others become followers. A follower can wait for the new cache value, receive stale data, or stop waiting when its own deadline expires. That policy must be explicit.

Coalescing protects the database by reducing duplicate work. It does not make the recomputation itself faster. In the Torollo run, the database hit count fell from 50 to one, while 49 requests still waited about 1.5 seconds for the result.

A lock also needs a bounded lifetime and ownership-safe release. Redis documents the SET ... NX PX pattern and warns that a client must not delete a lock now owned by someone else. The Redis distributed lock guide describes those safety constraints.

Approach 2: serve stale while revalidating

Stale-while-revalidate separates two clocks:

• the freshness window says how long the value is current;
• the retention window says how long an older value may still be served.

After freshness ends, one request refreshes the key in the background while the other requests receive the retained value immediately. This removes both the origin burst and the follower latency, at the cost of returning known-stale data for a bounded period.

That trade is suitable only when the data has a clear staleness budget. A product catalog may tolerate a short delay before a new title appears. An authorization decision, inventory reservation or account balance may not.

The same concept exists in HTTP caching through the stale-while-revalidate response directive. MDN’s Cache-Control reference describes how a cache can reuse stale content while it revalidates in the background. Application caches need their own equivalent policy and hard age limit.

Approach 3: refresh before the cliff

Proactive refresh moves recomputation before the TTL boundary. A fixed schedule can work for a small, known set of keys, but it can refresh cold data that nobody will read.

Probabilistic early recomputation makes each cache hit increasingly likely to trigger a refresh as expiry approaches. XFetch includes the observed recomputation cost in that decision, so a slow value gets a wider safety margin than a fast one. The algorithm comes from the paper Optimal Probabilistic Cache Stampede Prevention.

Early refresh lowers the chance that traffic reaches a hard cliff. It does not replace coalescing. Two requests can decide to refresh at nearly the same time, so the refresh path still needs coordination.

The tuning parameter buys freshness with origin work. Refresh too cautiously and stale responses return. Refresh too aggressively and the system recomputes values before it needs to.

Approach 4: keep a local floor when the cache fails

A distributed lock stored in Redis disappears as a defense when Redis is unavailable. That is exactly when every request becomes a cache miss.

An in-process single-flight group gives each application process a local floor: one origin query per key per process rather than one query per request. It cannot coordinate a whole fleet when the shared cache is down. Ten application instances can still produce ten queries. The important point is that the fallback does not depend on the failed component.

The request path also needs cache timeouts and a decision about degraded service. A cache should remain an optimization. If an unavailable cache makes the application wait indefinitely, it has become a hard dependency.

How the approaches compare

These mechanisms are complementary. A common design keeps a stale copy, refreshes early under a coalescing lock and retains local single-flight for degraded cache operation.

What a cache stampede looks like in production

Average hit rate is a weak detector because it hides short correlated bursts. Look for time-aligned signals:

• origin query count by cache key or endpoint;
• concurrent cache misses for the same key;
• database pool wait time and active connections;
• recomputation duration;
• follower count and follower timeout count;
• stale responses by age;
• cache errors and the origin work they trigger;
• periodic latency spikes aligned with a TTL.

A graph shaped like a comb is a useful clue. Flat origin traffic followed by a narrow spike at a regular interval often points to synchronized expiry or refresh work.

A practical design review

Before shipping a hot-key cache, answer these questions:

1. What is the maximum acceptable age for a stale value?
2. Who recomputes the key when many requests miss together?
3. What do the followers do while recomputation runs?
4. What bounds the lock, the wait and the origin request?
5. What happens when the cache and its distributed lock are both unavailable?
6. Can the old recomputation overwrite a newer value?
7. Which metrics expose duplicated origin work rather than average hit rate?

The answers define the failure mode more clearly than the TTL value does.

Sources and further reading

• How to tame the thundering herd problem, Redis
• Distributed locks with Redis, Redis documentation
• Optimal Probabilistic Cache Stampede Prevention, Vattani, Chierichetti and Lowenstein
• Cache-Control: stale-while-revalidate, MDN