Idempotency Keys in Practice: Designing APIs That Survive Retries

The problem retries create

Consider the canonical failure: a client calls POST /payments to charge a customer. The server debits the card, writes a payment record, and starts sending the response — and the connection drops. From the client's point of view, this is indistinguishable from a request that never reached the server at all. The only reasonable thing to do is retry. But retrying a non-idempotent write means the server might process the charge a second time.

This is not an edge case. It is the default behavior of TCP under packet loss, of load balancers under connection draining, of client libraries with default retry-on-timeout behavior, and of any queue-based worker that redelivers a message after a lease expires. In a system with enough request volume, ambiguous responses happen constantly, and "just don't lose the response" is not a fix — you cannot control the network between you and every caller.

The industry's answer, made mainstream by Stripe's API design and now close to a de facto standard, is the idempotency key: a client-generated unique value attached to a mutating request that lets the server recognize "I have seen this exact request before" and return the original result instead of repeating the side effect.

What an idempotency key actually guarantees

An idempotency key does not make an operation idempotent in the mathematical sense (f(f(x)) = f(x)). It makes an operation safe to retry: the same key, with the same parameters, produces the same result no matter how many times you send it. That distinction matters because it defines the contract precisely:

• Same key + same parameters → return the cached result of the original attempt, without re-executing the side effect.
• Same key + different parameters → this is a client bug (reusing a key for a different logical request) and should be rejected with a 422 or 409, not silently processed with the new parameters.
• New key → new logical operation, always processed.

The key is scoped to one specific request, not to a resource or a time window. This is the detail that trips people up when they first design one: an idempotency key is not a "don't let this user submit the order twice" business rule — that's a separate concern (often solved with a client-side order draft ID or a distinct uniqueness constraint on the resource itself). An idempotency key protects against retries of the same request, which is a transport-layer concern layered under whatever business rules exist above it.

Reference implementation

The shape that has converged across Stripe, GitHub, and most serious payment and messaging APIs looks like this:

1. Client generates the key. A v4 UUID or another string with enough entropy to avoid collisions, generated once per logical operation and reused across all retry attempts of that operation.

POST /v1/payments
Idempotency-Key: 7fd2c1e4-9c3d-4b8a-8f21-1e9a6d3c5b70
Content-Type: application/json

{ "amount": 4999, "currency": "usd", "customer": "cus_9f2a" }

2. Server atomically claims the key before doing any work. This is the step most naive implementations get wrong — checking "does this key exist?" and then inserting it are two separate operations unless you make them one. Race two concurrent requests with the same key and a check-then-insert pattern will let both through.

INSERT INTO idempotency_keys (key, request_fingerprint, status, created_at)
VALUES ($1, $2, 'in_progress', now())
ON CONFLICT (key) DO NOTHING
RETURNING key;

If the insert returns no row, another request with this key is already in flight or has completed — look up its state instead of proceeding. In Redis, the equivalent is SET key value NX EX <ttl>.

3. Compare the request fingerprint. Store a hash of the meaningful request parameters (method, path, body) alongside the key. If a retry arrives with the same key but a different fingerprint, that's a client error — the caller is misusing the key, not retrying the original request. Returning the cached result in that case would silently execute the wrong operation for a different logical request.

4. Execute the operation and persist the result against the key, including failure results. Stripe's implementation is explicit about this: a request that failed with a 500 is cached too, so a retry of a genuinely broken request returns the same 500 rather than trying again and potentially succeeding in an inconsistent way. (Client errors like 400s are typically not cached, since the client is expected to fix the request and resend with a new key.)

5. Return the cached response on any subsequent identical request, regardless of how long ago the original attempt happened, until the key expires.

The state machine you're actually building

The naive mental model — "check if key exists, if not do the work" — misses the case that matters most in production: what happens when a retry arrives while the original request is still being processed? This happens routinely with client-side timeout-and-retry logic, where the client gives up on a slow request before the server has finished.

A correct implementation needs at least three states per key:

Without the in_progress state, two near-simultaneous retries can both pass a "key not found" check before either finishes writing, and you're back to a double-charge. This is why the claim step in the reference implementation above has to be atomic against the actual persistence layer, not just checked in application code.

Storage and TTL

Idempotency records need a bounded lifetime — they cannot live forever, both for storage cost and because "was this key used" needs a defined answer. The pattern that's converged in practice:

• TTL longer than your longest plausible retry window. Stripe checks idempotency for at least 24 hours; systems with long-lived retry queues or offline clients (mobile apps that queue writes and replay them after reconnecting) often use 48–72 hours.
• Scope keys by source and environment. A key from a test-mode client should never collide with a key from production, and keys from different API consumers (if you multiplex several tenants through one key namespace) should be scoped to avoid cross-tenant collisions.
• Prune asynchronously, not synchronously. A cron job or TTL-based expiry (Redis EXPIRE, Postgres partition pruning) rather than checking expiry on every read keeps the hot path simple.
• Don't index on the key alone if you support multi-tenant access — index on (tenant_id, key) so lookups stay fast and collisions across tenants are structurally impossible rather than a fingerprint check away from a bug.

Idempotency for webhooks: the other half of the problem

Idempotency keys solve the client-initiated retry problem. Webhooks create the mirror-image problem: the provider retries delivery to you, and your endpoint has to be idempotent as a consumer, without ever having generated a key of your own.

Every serious webhook provider — Stripe, GitHub, Shopify — delivers at-least-once, never exactly-once, because there is no way to guarantee delivery without occasionally double-sending. A dropped payment_intent.succeeded event is worse than a duplicated one, so providers bias toward re-sending on any ambiguity: a timeout, a 5xx, a TCP reset before your 200 OK is acknowledged.

The fix mirrors the idempotency key pattern but uses the provider's event ID as the key instead of a client-generated one:

def handle_webhook(event):
    # event['id'] is Stripe's evt_... , GitHub's X-GitHub-Delivery, Shopify's X-Shopify-Webhook-Id
    claimed = db.execute(
        "INSERT INTO processed_webhook_events (event_id, source, received_at) "
        "VALUES (%s, %s, now()) ON CONFLICT (event_id) DO NOTHING RETURNING event_id",
        (event['id'], 'stripe')
    )
    if not claimed:
        return  # already processed — ack and exit, do not re-run side effects
    apply_side_effects(event)

Two details determine whether this actually works under load:

1. The claim and the side effect must be in the same transaction, or the side effect must itself be idempotent. If you claim the event, then crash before running apply_side_effects, that event is now silently dropped forever unless you also handle "claimed but never completed" the same way the payment example above handles in_progress.
2. Order is not guaranteed. Providers do not promise webhooks arrive in the order they were generated. If your handler assumes subscription.updated always follows subscription.created, a fast retry or network reordering will break that assumption. Store the event's own timestamp and compare against the current record state before applying an update, rather than trusting arrival order.

Retry backoff on the provider side typically escalates — something like 1 minute, then 5, then 30, then hours — which means your endpoint needs to keep accepting retries correctly for potentially a day or more after the first delivery attempt, not just handle a quick double-send.

Failure modes to design against

The fingerprint check gets skipped. The most common corner-cutting in idempotency implementations is caching the response by key alone, without validating that the request body matches. This silently turns a client bug — reusing a key across two different logical requests — into a data integrity bug, where request B gets request A's cached result.

The claim isn't atomic. Read-then-write idempotency checks look correct in a single-threaded test and fail immediately under concurrent load. If your data store doesn't support a native atomic claim (INSERT ... ON CONFLICT, SET NX), you need an explicit lock, not a check.

Idempotency keys are treated as a substitute for database constraints. They protect against retries of the same request; they do not protect against a different bug producing two different requests that both want to create "the same" resource. A unique constraint on, say, (customer_id, external_order_ref) is still necessary — idempotency keys and business-level uniqueness constraints solve different problems and you generally need both.

TTL is too short for the client's actual retry behavior. Mobile clients that queue writes offline and mobile networks with long-tail latency can produce retries hours after the original attempt. A 5-minute TTL that looked fine in a synchronous web app will silently double-execute requests from a client that was offline for twenty minutes.

Failed requests aren't cached, so a permanently broken request gets retried forever. If a request fails due to a bug (malformed downstream call, a schema mismatch), and you don't cache that failure, every retry re-triggers the same expensive, doomed call. Caching failures — with a shorter TTL than successes, if you want to allow eventual recovery — avoids turning a bug into a self-inflicted denial-of-service.

A short decision checklist

• Does every mutating (non-GET) endpoint that a client might reasonably retry accept an idempotency key or header?
• Is the key claim atomic against your actual data store, not just checked in application code before a separate write?
• Do you compare a fingerprint of the request body/params, not just the key, before returning a cached result?
• Is there an explicit in_progress state so concurrent retries don't race past each other?
• Is the TTL longer than your slowest realistic client retry window, and scoped per tenant/environment?
• For inbound webhooks: are you deduplicating on the provider's event ID with the same atomic-claim pattern, and are you tolerant of out-of-order delivery?
• Do idempotency keys coexist with, not replace, resource-level uniqueness constraints for genuine business-rule enforcement?

Sources

• Stripe: Designing robust and predictable APIs with idempotency
• Stripe API Reference: Idempotent requests
• Brandur Leach: Implementing Stripe-like Idempotency Keys in Postgres
• Webhook Idempotency and Deduplication — Hooklistener
• Webhook Reliability 2026: Idempotency & Retry Reference — DigitalApplied

Idempotency is one half of a stable API contract; API versioning is the other — together they're what let clients retry safely and upgrade without breakage.

Syslabs' engineering team builds and hardens integration layers like this — idempotency handling, webhook reconciliation, retry design — as part of production API work for clients across fintech, travel, and SaaS platforms.