Idempotency

An operation is idempotent when performing it several times has the same intended effect as performing it once. In a distributed system that property is what makes a retry safe: a client that cannot tell whether its request succeeded can send it again without charging a card twice, creating a second order or sending a second shipment. In HTTP APIs, idempotency is partly built into method semantics (PUT and DELETE) and partly added with an idempotency key on operations such as POST /payments. It is a business guarantee that the server has to enforce, not something the network provides.

The problem exists because failures often happen after the work succeeded. The client sends a payment request, the service authorises the card and commits, and the response is lost on the way back. The client sees a timeout. If it blindly repeats the request, the customer pays twice; if it refuses to retry, a valid payment looks failed. Idempotency lets the client retry while the server guarantees that the business side effect happens once.

Three ways a request can time out

A client that times out is in one of three situations and cannot distinguish them. The request may never have arrived, in which case retrying is harmless. It may have arrived and failed before any committed side effect, in which case retrying is also harmless. Or it may have succeeded with the response lost, which is the dangerous case: the retry now repeats a charge, an order or an email. Because the third case looks exactly like the first two, the only safe design is to assume the client will retry and make retrying safe. How and when to retry (backoff, jitter, budgets) is covered in Retry & Backoff; this lesson is about making the retried operation itself safe to repeat.

The mental model: delivery versus execution

The useful distinction is between transport delivery and business execution. A client library, a load balancer, a message broker or an impatient user can deliver the same logical request several times. Whether those deliveries amount to one purchase or three is a decision the application must make explicitly. The goal is not to prevent duplicate requests, which the network will produce regardless, but to prevent duplicate business outcomes. Concretely, a customer who clicks “Pay ₹4,999” once must be charged once, whether the request was delivered once or four times.

Take a payment endpoint and follow one request carefully:

1. The client sends POST /payments with key K.
2. The service authorises the payment with the provider and records it.
3. The database transaction commits; the payment now exists durably.
4. The response is lost between the server and the client (a dropped connection, a gateway that gave up first).
5. The client times out and sends POST /payments again with the same key K.

At step 5 the server must recognise a second delivery of an operation it has already executed, and answer with the original recorded result or a documented equivalent such as the payment's current status. Executing again would charge twice; refusing to retry would leave a valid payment looking failed. Retrying with a new key defeats the mechanism, because the server would reasonably treat it as a new purchase. The sequence diagram below draws the exact message order.

The client generates one key per logical operation and reuses it for every retry of that operation. Two separate purchases need two keys even when their payloads are byte-identical. A random UUID is the usual choice for user-initiated actions. For system-initiated work, such as a worker issuing a refund, a deterministic key derived from the business identity (order, payment and a version of the intent) lets a restarted process rebuild the same key instead of inventing a new one.

POST /v1/payments HTTP/1.1
Idempotency-Key: 9c7c96d4-812e-47c1-b712-2899a5947f64
Content-Type: application/json

{ "order_id": "ord_8f32", "amount_minor": 499900, "currency": "INR" }

HTTP/1.1 201 Created
{ "id": "pay_1042", "status": "AUTHORIZED" }

POST /v1/payments HTTP/1.1
Idempotency-Key: 9c7c96d4-812e-47c1-b712-2899a5947f64
Content-Type: application/json

{ "order_id": "ord_8f32", "amount_minor": 499900, "currency": "INR" }

HTTP/1.1 201 Created
Idempotent-Replayed: true
{ "id": "pay_1042", "status": "AUTHORIZED" }

A key by itself is not an identity, because two tenants can generate the same string and one client can reuse a key for a different request by mistake. The server therefore stores records under a scope of tenant or account, operation, and key, and keeps a request fingerprint beside them. Including the method and path in the scope means the same key on POST /payments and POST /refunds cannot collide, and including the account means one user's retry can never return another user's payment. If the same scoped key arrives with a different amount or order, the server should reject it rather than replay a result for unrelated input. A common choice is 422 Unprocessable Content, which the IETF draft for an Idempotency-Key header (still a draft, not a standard) also uses for a mismatch. Whatever you choose, document it.

scope:                tenant_42 / POST /v1/payments
key:                  9c7c96d4-812e-47c1-b712-2899a5947f64
request_fingerprint:  sha256(canonical(order_id, amount_minor, currency))
state:                COMPLETED
payment_id:           pay_1042
provider_operation:   ch_77
stored_status:        201
stored_response:      {"id":"pay_1042","status":"AUTHORIZED"}
expires_at:           2026-10-30T10:00:00Z

Storing the response, not just the fact that the key was seen, is what makes a replay useful. A server that remembers only “key X was processed” can answer a retry with nothing more than an empty 200, leaving the client unable to learn the payment ID.

Computing the fingerprint is where subtle bugs live. Hashing raw request bytes treats the same fields in a different order as different requests, so a client library that serialises maps non-deterministically would see spurious mismatches on retry. Normalising too aggressively is the opposite failure: if canonicalisation drops a field that changes the business meaning (currency, destination account, a nested line item), two different operations hash identically and one is silently replayed as the other. Hash the operation-defining fields after a stable canonical form (sorted keys, one number representation, insignificant whitespace removed) and exclude only transport noise such as trace headers, timestamps added by a proxy, or the key itself.

The flow has four steps. The server tries to claim the scoped key atomically. If the claim succeeds, this request owns execution: it runs the operation once. It then stores the final status and body against the key. Any later request carrying the same key reads that record and replays the stored answer instead of executing again. The decision diagram below shows what each outcome of the claim leads to, including the cases where the duplicate arrives while the owner is still running or with a different payload. The rest of the article is about making each step hold under concurrency and crashes, which is where implementations usually fail.

Implemented literally as “check key, then execute”, with a SELECT followed by an INSERT, the flow has a gap between the two statements. Two requests carrying K (a double-click, a client retry racing a slow original, a load balancer retry) can both read “not found” inside that gap and both proceed to charge. No care in the application code closes the gap, because the decision and the write are two separate steps. The claim must be a single atomic operation that the data store arbitrates: a unique constraint, or an equivalent conditional write such as a compare-and-set on an absent key in a linearizable store (Redis SET key value NX works for a single node but needs care around expiry and failover). The diagram below shows the outcome the constraint produces.

CREATE TABLE idempotency_records (
  tenant_id             TEXT        NOT NULL,
  operation             TEXT        NOT NULL,
  idempotency_key       TEXT        NOT NULL,
  request_fingerprint   TEXT        NOT NULL,
  state                 TEXT        NOT NULL,  -- CLAIMED | IN_PROGRESS | UNCERTAIN | COMPLETED | FAILED
  lease_owner           TEXT,
  lease_expires_at      TIMESTAMPTZ,
  provider_operation_id TEXT,
  response_status       INTEGER,
  response_body         JSONB,
  expires_at            TIMESTAMPTZ NOT NULL,
  PRIMARY KEY (tenant_id, operation, idempotency_key)
);

-- Claim: exactly one concurrent caller gets a row back.
INSERT INTO idempotency_records
  (tenant_id, operation, idempotency_key, request_fingerprint, state, expires_at)
VALUES ($1, $2, $3, $4, 'CLAIMED', now() + interval '24 hours')
ON CONFLICT (tenant_id, operation, idempotency_key) DO NOTHING
RETURNING state;

The winner owns execution. The loser must not run the business logic, and it has three honest options depending on what the stored record says. If the record is COMPLETED and the fingerprint matches, replay it. If the fingerprint differs, reject the request as a key reuse. If the record is still in flight, the contract decides: return 409 Conflict with a retry hint, return 202 Accepted with a stable operation identifier the client can poll, or wait for a bounded interval (a few seconds, never indefinitely) and replay if the owner finishes. The guarantee that matters is that duplicates create no second side effect and the client can still discover the outcome.

One PostgreSQL detail worth knowing: when the claim is inserted inside a longer transaction, a concurrent INSERT ... ON CONFLICT DO NOTHING on the same key blocks until that transaction commits or rolls back. That accidental wait is correct but ties up a connection, which is why many designs commit the claim in its own short transaction and carry state explicitly instead.

When the whole operation lives in one transactional database, the business rows and the deduplication result can commit together: either both exist or neither does, and the crash windows mostly disappear. The hard case is an external call. Suppose the service inserts the record, calls the payment provider, and crashes before saving the provider's answer. On retry the record says IN_PROGRESS, yet the provider may already have charged the card. No local transaction can span your database and the provider's ledger, so the design has to make the uncertainty explicit instead of hoping it away.

Three mechanisms work together. First, write the intent durably before the call and pass the provider a stable idempotency identifier derived from your key, when the provider supports one, so a repeated call is recognised on their side. Second, store the provider's operation ID as soon as it is known. Third, when a record is stuck in an uncertain state, reconcile by asking the provider what happened, using that identifier, and then either complete the record, or retry the call with the same identifier, or record a final failure. The state diagram below shows the states.

Be honest about the limit. If the provider offers neither an idempotency identifier nor a way to look a charge up, exactly-once charging cannot be guaranteed from the API layer alone. You can narrow the window and detect duplicates afterwards, but you are then managing a residual risk with refunds, alerts and daily reconciliation against the provider's settlement report, not eliminating it.

Production failure: the key was stored after the charge

A payment service calls the provider, and only then inserts its idempotency record. A deploy kills the process after the charge but before the insert. The client retries; the service finds no record, believes this is a first attempt, and charges again. Nothing is wrong with the retry logic. The deduplication state was never coordinated with the side effect it was meant to guard, so the window between “charged” and “recorded” stayed open. Moving one line is not a fix. Claim first, persist intent, give the provider a stable identifier, store its operation ID, and give a recovery job the authority to reconcile any record that sits in IN_PROGRESS or UNCERTAIN past its lease. This incident and the double-click race above are two ends of the same gap.

No rule says every duplicate must get the same bytes forever. The API should state its behaviour. The common design replays the first completed response, including its status code, for the same key and fingerprint within the retention window; another design returns a stable operation identifier and lets the client query current status. Be deliberate about failures too. A definitive business outcome (a declined card, a validation error found after the claim) can be stored and replayed, but a transient failure that happened before any side effect, such as a database outage, should release the claim so the retry can actually run; replaying a stored 503 would turn a blip into a permanent failure for that key.

Retention cannot be unlimited, so a service promises protection for a documented period (24 hours or seven days are common, chosen from client retry behaviour and business risk). After expiry, reusing the key may be treated as a new operation, which makes the TTL an external contract, not merely a cleanup setting. It has to exceed the client's longest retry horizon, including queued mobile retries and manual operator replays. A client with a two-hour backoff window against a one-hour TTL will, at the 90-minute retry, find “key not found” and execute a duplicate (illustrative numbers). Expired records still need a background job to evict them, so retention is also a storage cost.

Keep two clocks apart. A worker lease says who may process or recover an operation right now; it is short (seconds to minutes) and is renewed or taken over by compare-and-set on lease_expires_at. The deduplication retention period says how long the same logical operation stays recognisable; it is long. A short lease and a long retention window coexist, as the clocks diagram below shows. The mistake to avoid is deleting an in-progress record because its lease expired: that erases the only evidence that a charge may be outstanding, so the next retry runs from scratch. An expired lease should trigger recovery, never deletion. Where a lease can be taken over, give the external call a fencing identifier so a slow former owner cannot act after its replacement.

For high-value operations, do not rely on key retention alone. Enforce domain uniqueness where the business allows it: one captured payment per authorised transaction, one refund per (payment, refund request), one shipment per order line. A unique constraint on that natural identity keeps working long after the idempotency record has been purged, and it also catches the case where a client mistakenly sent a fresh key for the same purchase. The key handles retries; the domain constraint handles invariants. Both belong in the contract you publish in API design.

By RFC 9110, PUT and DELETE are idempotent methods, and GET, HEAD and OPTIONS are both safe and idempotent. Idempotent means the intended server state is the same after one or many identical requests; the responses may differ. A second DELETE /orders/42 can return 404 and still be idempotent. PUT /users/7 with { "status": "ACTIVE" } leaves the same state however often it is repeated. POST is not idempotent by definition, which is why POST /payments without protection charges again on every delivery, and why the key adds the property at the application level. PATCH depends on the patch: setting status to CANCELLED is idempotent, adding 1 to a counter is not. The method label is a contract the implementation must honour; a PUT handler that appends to a list or fires a notification is not idempotent just because of its verb. Protocol-level retry behaviour is discussed in REST.

Four patterns

The idempotency key header is the general-purpose tool for payments, orders and calls to external APIs: the client generates the key per business operation and the server stores key to result, as described above. A natural unique key uses a business-level constraint instead, such as UNIQUE(email) on user registration, so the second insert fails cleanly and the handler can map the conflict to the existing user. An upsert, INSERT ... ON CONFLICT (user_id) DO UPDATE SET ..., suits configuration and state synchronisation, where either path converges on the same row. State replacement sends the desired final state rather than a delta: PUT { "quantity": 5 }, not PATCH { "delta": +1 }. Prefer the last three where they fit, because they need no extra store; use the key when the operation creates something new and has external side effects that no constraint can describe.

An Idempotency-Key header does not give exactly-once anything. A broker can redeliver a message (at-least-once delivery) and a consumer can re-run a handler for it (at-least-once processing); a key changes neither. What it protects is the third layer: whether the actual side effect, the charge, the order or the email, happens more than once. An idempotent consumer makes repeated deliveries produce one intended effect. A consumer of order events can record the event ID in the same transaction that reserves inventory, with a unique constraint on the event ID, so a redelivery violates the constraint and rolls back instead of reserving twice.

If the same handler also sends an email, that effect sits outside the transaction: a crash after the send but before the commit repeats it. It needs its own deduplication (a provider message ID, or an outbox row marked sent) or reconciliation. That is why the practical outcome is called effectively-once and why the guarantee is scoped to your system's boundary: once the call crosses into a third-party provider, that system needs its own strategy; your key stops you asking it twice, but it cannot make the other side's delivery exactly-once. End-to-end exactly-once behaviour requires defining the effect, its boundary, the storage and every external dependency. Queues and job systems are at-least-once by design, which is why consumer-side idempotency matters; see Queues & Workers and Job Processing.

A retry policy needs two inputs: whether the operation is safe to repeat, and what kind of failure just happened. A timeout on an operation with guaranteed deduplication is a reasonable retry. A 400 or 422 validation failure is not: the same bytes will fail the same way, so the retry only burns capacity. 429 and transient 503 responses are retryable with backoff, and should respect Retry-After when the server sends it. A 409 because the original is in flight is retryable after a short delay, while a 409 or 422 for a key reused with a different payload is not, because it signals a client bug. Delay, jitter and attempt limits are covered in Retry & Backoff, and sustained failures are better stopped by a circuit breaker than retried.

The dangerous interaction is nesting. If the client, the gateway and the service each make up to three attempts, one user action can produce 27 calls at the bottom layer while the dependency is already struggling (illustrative: attempts multiply across layers, they do not add, as the diagram below shows). Idempotency keeps those 27 calls from producing 27 charges, but they still produce 27 units of load. Give the whole request an end-to-end retry budget: retry at one layer (usually the outermost one that holds the key), pass a deadline downward, and let inner layers fail fast.

Two failures are covered above because they need structural fixes: the check-then-insert race and the key stored after the charge. The remainder are mostly contract and client bugs. They are cheap to prevent and expensive to find after a customer has been charged twice.

• No key at all

  The gateway times out and the customer clicks Pay again; every retry of POST /payments is a new payment. Result: a double charge, a support ticket, a refund and lost trust.

  Fix: Require the key on payment and order-creation endpoints and return a clear error when it is missing.

• A new key on every retry

  An SDK generates a fresh UUID per HTTP attempt, so every retry looks like a new operation and the protection is silently void.

  Fix: Generate the key once per business operation, before the first attempt, and reuse it for all retries.

• The key expires before the last retry

  Retention is shorter than the longest client retry horizon, so a late retry finds nothing and executes again.

  Fix: Set retention above the longest retry window, including manual operator replays, and back it with a domain constraint.

• Only the key is stored, not the response

  The server remembers that X was processed but not what it returned, so retries get an empty 200 and the client cannot learn the payment ID.

  Fix: Store status and body (or a stable operation reference) and replay them.

• Check-then-insert race

  A double-click sends two simultaneous requests; both see "not found" and both execute.

  Fix: Claim with a unique constraint or an atomic conditional write; the loser never executes.

• Keys not scoped

  A global key namespace lets one tenant's key collide with another's, and a retry can return somebody else's payment.

  Fix: Scope by tenant or account and operation, and compare a request fingerprint before replaying.

An idempotent API lets a client repeat a request, such as after a timeout, without repeating its effect, and public APIs usually achieve this with a client-supplied idempotency key.

Stripe accepts an Idempotency-Key header on POST requests. Reusing a key returns the stored status and body of the first request, including failures, requires the parameters to match, and keys may be pruned after at least 24 hours. PayPal uses a client-supplied PayPal-Request-Id on supported REST operations to recognise a retried request; support and retention vary by API. AWS EC2 RunInstances accepts a ClientToken so a retried launch does not start duplicate instances. Square makes idempotency_key a required field in the body of CreatePayment. The shared shape is a client-generated opaque token, a server-side record bound to the request, and a documented retention period. Check each vendor's current documentation for exact names and windows.

The rule of thumb: if repeating a mutation costs money, ships goods or sends something to a person, protect it. If repeating it is harmless or already converges on the same state, use the cheaper mechanism from the patterns section rather than a key store.

The costs are real. Every protected request adds a write to the key store, and the store becomes part of the availability path: if it is down, the endpoint either fails closed or runs unprotected, and you must choose deliberately. Retention is storage that must be evicted. Protection only works if clients cooperate, so make the key required and validate it on every request rather than hoping SDKs send it. And the guarantee is scoped to your boundary, as explained under exactly-once above. Without any of this you keep a simpler implementation and no extra storage, at the price of being unable to retry a timed-out mutation safely at any layer.

This sketch combines the pieces for POST /v1/payments backed by PostgreSQL and a provider that accepts its own idempotency identifier and supports lookup. It is pseudocode for the control flow, not a drop-in implementation.

handle(tenant, key, body):
  fp = sha256(canonical(body.order_id, body.amount_minor, body.currency))

  claimed = INSERT ... ON CONFLICT DO NOTHING RETURNING   -- atomic claim
  if not claimed:
    rec = SELECT ... WHERE (tenant, 'POST /payments', key)
    if rec.fingerprint != fp:        return 422            -- key reused for other input
    if rec.state == COMPLETED:       return rec.status, rec.body   -- replay
    if rec.state == FAILED:          return rec.status, rec.body   -- definitive outcome
    return 409 with Retry-After                            -- owner still running

  -- owner path, in its own short transactions
  UPDATE ... SET state='IN_PROGRESS', lease_owner=$me,
                 lease_expires_at = now() + interval '2 minutes',
                 provider_operation_id = 'pay-' || key      -- intent persisted first
  result = provider.charge(idempotency_id = 'pay-' || key, ...)
  BEGIN;
    INSERT INTO payments (...) VALUES (...);                -- UNIQUE(order_id) as domain guard
    UPDATE ... SET state='COMPLETED', response_status=201, response_body=$json;
  COMMIT;
  return 201, body

recovery job (every minute):
  for rec in state IN ('IN_PROGRESS','UNCERTAIN') and lease_expires_at < now():
    charge = provider.lookup(rec.provider_operation_id)
    if charge exists:  finish rec as COMPLETED
    elif declined:     finish rec as FAILED
    else:              retry provider.charge with the same identifier

Walk the failure cases against it. A double-click produces two inserts; one returns a row and executes, the other gets 409 and, on retry, the replayed 201. A crash after the provider call leaves the record IN_PROGRESS with an expired lease; the recovery job finds the charge by provider_operation_id and completes it, so the client's next retry replays a normal result. A client that retries after the retention window has a fresh key from the server's view, but the UNIQUE(order_id) constraint on payments still refuses a second capture. The one thing this design cannot do is repair a provider that has no identifier or lookup; that residual risk belongs to reconciliation.

An idempotency key is the retry-safety half of an API's contract, so API Design is the natural companion, and REST covers the method semantics the key builds on. For the timing and amplification side of retries, read Retry & Backoff, and for what to do when a dependency keeps failing, the Circuit Breaker lesson. Idempotency is a business guarantee implemented across retries, concurrency, persistence and recovery; a header alone does not prevent duplicate side effects.