available now · advanced
Charged twice
Idempotent payment workers on RabbitMQ, Redis and Deno
A worker completes a charge, dies before acknowledging the message and leaves another worker free to run it again.
Monday starts with a support ticket: one order, two charges. The queue kept its promise and recovered the job. The payment code made the recovery unsafe.
the running environment
Three services. One failure you can observe.
Torollo creates each service as a real container on your Docker daemon. The network paths below are part of the system the validators inspect.
broker
RabbitMQ payments queue
container / online
worker
Two Deno cashiers
container / online
ledger
Redis order state
container / online
- worker to brokerAMQP 5672 / open
- worker to ledgerTCP 6379 / open
the failure sequence
The system recovers. The customer still gets hurt.
01
delivery 1
A worker receives the payment job
RabbitMQ keeps the delivery unacknowledged while the worker performs the side effect.
02
side effect
The charge succeeds
The external action has happened, but the broker has not received an acknowledgement.
03
crash
The worker connection closes
The broker makes the unacknowledged message available for redelivery.
04
delivery 2
Another worker runs the job
Without an idempotent boundary, the same business operation can create a second charge.
what you will investigate
The brief gives you evidence, not the repair.
- 01Reading at-least-once delivery as an application contract
- 02Choosing a business idempotency key
- 03Closing check-then-act races between workers
- 04Handling leases that expire while work is still running
- 05Recovering an unacknowledged message after a crash
graded against the containers
A pass comes from the repaired system.
The scenario observes queue state, per-order ledger state, competing consumers and crash recovery. Validators grade the live containers after each repair.
included 01
Failure state
included 02
Live observations
included 03
Commented debrief
read before you run it
At-least-once delivery: why jobs run twice
Understand redelivery, idempotency and the acknowledgement boundary first.
Read the technical field noteproduction incidents
Run Charged twice locally.
$29
The purchase includes both available incidents, the October incident when it ships, hints and commented debriefs. There is no subscription or license server.
One-time purchase · keep forever · runs locally · 30-day money-back guarantee