Idempotency
The Retries lesson resends a failed call so a brief failure does not reach the user. Retrying a read is harmless, but retrying a write can repeat the write.
A payment service now joins the service chain from the earlier lessons. To charge $20 for an order, the order service calls the payment service with POST /payments and a 300 ms timeout. If the first attempt times out and the retry succeeds, the card was charged once or twice, and the order service cannot tell which.
Why a Timeout Cannot Be Trusted
A timeout tells the caller only that no response arrived in time. It does not say where the failure happened. That failure can occur in three places.
In the first case, the network drops the request, so the payment service never sees it. In the second case, the payment service crashes before committing anything. In the third case, the payment service charges the card and commits the payment, but the network loses the response on the way back. The customer pays in case three, though not in cases one and two.
The caller sees the same timeout in all three cases. Without a retry, cases one and two lose a sale the customer wanted. With a retry, cases one and two succeed, while case three charges the customer a second time.
Skipping retries is not fully under the service's control either. Users tap Pay again, and HTTP client libraries and proxies retry failed requests. A message queue also redelivers a message when the consumer does not acknowledge it. The system therefore needs a retry that cannot repeat the effect.
What Idempotent Means
An operation is idempotent if running it twice leaves the system in the same state as running it once.
Consider an account with a $90 balance. One request says "add $10 to the balance," and another says "set the balance to $100." Running each request twice shows how a retry behaves.
The addition ends at $110, which is $10 too much. The set operation ends at $100 both times because the second run writes a value that is already there.
Many operations are naturally idempotent. A read changes nothing, a PUT that replaces a whole record writes the same record again, and a DELETE of a deleted row has nothing left to delete. An upsert (insert the row, or overwrite it if a row with that key exists) is a common way to turn a write into a "set".
A card charge cannot become a "set". Every run moves money. The fix is to give the request an identity that the server can recognize when the request arrives again.
Idempotency Keys
An idempotency key is a unique value, typically a random UUID, that the client generates for one logical operation such as a single purchase. The client sends the key with the request, usually in a header, and sends the same key on every retry. Here, the client is whichever caller invokes the payment service: the app for a user's tap, or the order service for its own retries.
The server keeps a table of keys it has handled along with the response returned for each key. On each request, the server looks up the key first.
For a new key, the server does the work and saves the result under that key. For a known key, the server skips the work and returns the saved result.
Suppose the first response is lost and the retry's response arrives. Attempt 1 charges $20 and saves 9f2c → pay_1. Attempt 2 finds 9f2c and returns pay_1 without charging again. The client made two attempts, and the card was charged once. Without the key, those two attempts charge $40.
The widget below sends a payment through a network that can lose the response. Press "Send a new payment" and press "Step" until the response is lost. Turn off "Network loses the response", press "Retry", press "Step" until the attempt ends, and count the charges. Then press "Reset", turn on "Send an idempotency key", and do the same again.
Without the key, the retry charges a second time. With the key, the server finds the key stored on the first attempt and returns the stored result. The card is charged once.
The client must create the key once before the first attempt. A common bug builds the whole request inside the retry loop and generates a new key on every attempt. The server then sees three different purchases and charges three times without raising an error.
# Wrong: a new key on every attempt
for attempt in range(3):
key = uuid4()
send("POST /payments", key, amount)
# Right: one key per purchase, reused on every retry
key = uuid4()
for attempt in range(3):
send("POST /payments", key, amount)Stripe's API works this way. In its v1 API, once a request starts to run, Stripe saves the status code and body of the first request for a key. It then returns the same result for later requests with that key. It also compares the parameters of a repeated request against the original and returns an error if they differ. That check catches a client reusing a key for a different purchase (Stripe docs).
The code below runs three attempts of the same payment against a server with key checks, a server without them, and a client that makes a new key per attempt. Change the attempts and run it again.
Idempotency Keys: Three Attempts, Lost Responses
How Long to Keep a Key
Because the key table grows with every request, the server must remove old keys. However, a key must remain in the table longer than any client could still retry with it. If a retry arrives after its key is deleted, the server treats it as a new request and charges again.
Stripe, for example, may remove keys once they are at least 24 hours old, and reusing a key after that starts a new request (Stripe docs). The retention period therefore sets a limit on the retry policy. Retries, queue redeliveries, and manual replays must all finish inside that window.
Two Requests With the Same Key at Once
The basic key check has a gap when two requests with the same key arrive together. A user can double-tap Pay. The order service can also retry while the first attempt is still running. The order service's 300 ms timeout fires, but the payment service keeps working on the first attempt (the Timeout lesson describes this wasted work). The retry then arrives while that first attempt runs.
With a "check, then insert" sequence, request A looks up 9f2c and finds nothing. Before A saves anything, request B looks up 9f2c, also finds nothing, and both requests charge the card. Because the check and the write are two separate steps, the other request can run between them. This bug is a race condition: the result depends on the order in which two requests interleave.
Reversing the order fixes this race. Before doing any work, the server inserts the key into a column with a unique constraint, which makes the database reject a second row with the same value. Request A's insert succeeds with status in_progress, and request B's insert fails because the key already exists. The database uses the constraint to decide which request wins.
Request A charges the card and marks the key done with the result pay_1. Meanwhile, request B returns 409 Conflict, telling the client that the request is still in progress and to retry later. When the client retries B, the key is done and the server returns pay_1.
The widget below runs both requests through the payment service, one database action at a time. Here, a charge is a payment row in the payment service's own database. Press "Next action" until the run ends and count the charges. Then select "Insert key first" and run it again.
With "check, then insert", both lookups run before either request saves the key, so the card is charged twice. A unique constraint on the key column rejects B's later insert, but only after B has charged. With "Insert key first", B's insert fails on its first action, so B does no work.
The third option, "Insert key first, one transaction", fixes a problem from the next section.
A Crash in the Middle
A crash between database writes exposes a second gap. Suppose the server saves the payment in one write and marks the key done in a second write. If the process crashes between the two writes, the payment is saved while the key remains in_progress.
The server then cannot tell whether that first attempt is still running or dead. If retries wait, the client keeps getting 409. If a time limit lets a retry take over a key that stayed in_progress too long, the retry runs the work again, and the customer pays twice.
The fix puts the key and the work in the same database transaction (a group of writes that the database commits together or not at all). Inside one transaction, the server inserts the key, inserts the payment, marks the key done, and commits.
A crash before the commit discards all three writes, including the key, so the retry starts fresh and charges once. A crash after the commit keeps all three writes, so the retry finds the key. In the widget above, select "Crash after the charge, then retry" and compare "Insert key first" with "Insert key first, one transaction". The first option ends with two charges. The second ends with one.
The transaction also changes the double-tap case. Request A's key row is not committed yet, so in many relational databases B's insert waits until A commits or rolls back. After A commits, B's insert fails, and B reads the committed key row and returns pay_1. B never sees in_progress, so it gets no 409. Select "Double-tap: two requests at once" with "Insert key first, one transaction" to step through it. The 409 path remains for work that cannot sit inside one local transaction, such as the call in the next section.
The rule is to record the key in the same transaction as the effect that it protects. For more on transaction scope, see Database Transactions.
When the Work Happens in Another System
That transaction rule requires the effect to live in the service's own database. In most payment systems, an external payment provider charges the card through its own API. Because the payment service's transaction cannot span the provider's system, the local commit and the provider's charge cannot commit together.
The payment service therefore splits the work. In one local transaction, it records the key and a payment row with status pending. A payment worker then calls the provider, and the payment row's status tracks progress, as in_progress did before. Suppose the call times out. The worker is now in the caller's position from the first section: the charge may or may not have happened. Two practices handle this ambiguity.
First, the worker does not mark the payment failed. A timeout means that the worker did not hear back. The worker puts the payment into an explicit unknown state such as in_doubt instead of guessing.
Second, the worker passes the same idempotency key down to the provider. A provider that supports keys returns the original result on retry, such as the charge or a saved error, instead of charging again. When the worker gets the charge, it marks the payment captured, which means the card was charged. For anything still unresolved, a reconciliation job compares the in_doubt records against the provider's settlement report (the provider's record of what was charged) and finalizes each payment.
Reconciliation. A batch job matches internal records against an external source of truth. Suppose payment
pay_1isin_doubtin the payments database, and the provider's daily report lists chargech_1with the referencepay_1, which the payment service attached to the charge. The job markspay_1ascapturedand writes the ledger entries.
The Payment System solution builds this flow step by step, from a version that double charges to the idempotency gate, the in_doubt state, reconciliation, and a double-entry ledger.
Queues and Jobs: Exactly-Once Effects
In many systems, no client waits for a response. A queue holds the work, and a worker processes it. Many queues provide at-least-once delivery, where the worker processes a message and then sends an acknowledgment (ack). If the worker crashes before sending the ack, the queue delivers the message again. Under normal operation no message is lost, and some messages are processed twice. Delivery Guarantees compares at-most-once, at-least-once, and exactly-once delivery.
Job schedulers face the same problem with leases. A lease is a time-limited claim on a job. If the worker does not renew the lease in time, it expires and another worker can take the job.
Suppose worker A claims job job-7 with a 30-second lease and then freezes for 45 seconds in a long garbage-collection pause. At 30 seconds the lease expires, and worker B claims and runs the job. At 45 seconds, worker A resumes and runs the job as well. A longer lease does not fix this issue because a pause can last longer than the lease. Martin Kleppmann describes this failure mode for distributed locks (How to do distributed locking).
Because a gap remains between doing the work and recording that it was done, exactly-once delivery is not practical across machines. Instead, a system can achieve an exactly-once effect by combining at-least-once delivery with idempotent work. The job carries a key, and the worker records that key in the same transaction as the effect, using the same insert-first pattern as before. Worker B records the key and runs the job. Worker A finds the key and does nothing.
Kleppmann's own fix is a fencing token: a number that increases with each lease grant, which the storage checks so that it rejects writes from an older lease holder. The job key plays a similar role when the effect lives in the same database as the key. The Distributed Job Scheduler solution covers leases, crash recovery, and this effectively-once execution in detail.
The Same Pattern at Every Hop
Every scenario above has a sender that might send twice and a receiver that must not repeat the effect.
| Sender → receiver | Who makes the key | Where it is checked |
|---|---|---|
| Client → payment service | The client, once per purchase | Payment service, in the same transaction as the payment |
| Payment worker → provider | The payment service, passed down | The provider's idempotency layer |
| Queue → worker | The producer, in the message or job | The worker, in the same transaction as the effect |
At each hop, the sender keeps the same key across retries, and the receiver records the key together with the work.
Safe Retries Still Add Load
An idempotency key makes a retry safe, though it does not limit how many retries arrive or when they arrive. A dependency that is down still receives every retry, and each retry still holds a thread in the caller. The Retries lesson covers retry counts, backoff, and jitter, and the next lesson, Circuit Breaker, stops calls to a dependency that keeps failing.
Key idea. Because a timeout does not tell the caller whether the work happened, the client generates one idempotency key per operation and reuses it on every retry. The server inserts the key first under a unique constraint and records it in the same transaction as the effect. For an external provider, the payment service passes the key down, marks timed-out calls
in_doubt, and reconciles them. For queues, at-least-once delivery combined with a key check produces an exactly-once effect.
Further Reading
- Idempotent requests — Stripe API docs: saved responses per key, parameter comparison, and removal of keys after at least 24 hours.
- Implementing Stripe-like Idempotency Keys in Postgres — Brandur Leach: a key table with a unique constraint,
409 Conflictfor a key in progress, and transactions between calls to external systems. - How to do distributed locking — Martin Kleppmann: why a process pause can outlast a lease.