available now · advanced

The thundering herd

Stampede-proof caching with Redis, PostgreSQL and Deno

A hot cache key expires and every request reaches PostgreSQL in the same instant.

The catalog cache worked for months. Then one newsletter sent readers to the same page at once. PostgreSQL spiked on the TTL boundary while Redis stayed healthy.

the running environment

Three services. One failure you can observe.

Torollo creates each service as a real container on your Docker daemon. The network paths below are part of the system the validators inspect.

web

Deno store

container / online

cache

Redis hot key

container / online

db

PostgreSQL catalog

container / online

  • web to cacheTCP 6379 / open
  • web to dbTCP 5432 / open
  • cache to dbTCP 5432 / blocked

the failure sequence

The cache stays healthy. The database still queues.

  1. 01

    09:00:07

    The hot key expires

    The catalog value crosses its freshness boundary while requests are already arriving.

  2. 02

    + 1 ms

    Every request sees the same miss

    No request can see a value that another request has not finished rebuilding yet.

  3. 03

    + 1.5 s

    The connection pool becomes a queue

    Identical database work waits in waves behind the pool limit.

  4. 04

    next TTL

    The same tooth returns

    The load graph becomes a comb, with one spike at each expiry boundary.

what you will investigate

The brief gives you evidence, not the repair.

  • 01Diagnosing correlated load on an expired hot key
  • 02Coalescing concurrent misses behind one lookup
  • 03Separating freshness from cache retention
  • 04Refreshing a hot key before its expiry cliff
  • 05Protecting the database when the cache is unavailable

graded against the containers

A pass comes from the repaired system.

The scenario records database hits, waiting requests, stale responses and the slowest response for each drill. Validators read those observations from the running system.

included 01

Failure state

included 02

Live observations

included 03

Commented debrief

read before you run it

Cache stampede: causes, fixes and trade-offs

Read the mechanisms before you reproduce them on a running system.

Read the technical field note

production incidents

Run The thundering herd locally.

$29

The purchase includes both available incidents, the October incident when it ships, hints and commented debriefs. There is no subscription or license server.

Get the Production Incidents pack for $29

One-time purchase · keep forever · runs locally · 30-day money-back guarantee