Webhook Delivery at Scale: Building a Retry and Dead-Letter System That Does Not Lose Events

TL;DR: Webhooks are at-least-once, not exactly-once — your endpoint will see duplicates, and it will occasionally see nothing at all unless you build for both failure modes explicitly. The fix is a three-part system: bounded exponential backoff with full jitter on the sender side, idempotent processing keyed on the event ID on the receiver side, and a dead-letter queue with manual or automated redrive once retries are exhausted. None of these three pieces is optional — skip any one and you either lose events silently or double-charge a customer during a retry storm.

Most teams discover their webhook system is broken the same way: a downstream consumer goes down for twenty minutes, retries pile up, the consumer comes back, and now every queued event fires in a burst that either overwhelms the consumer again or — worse — processes some events twice because nobody built idempotency in from the start. This article is a practical architecture for avoiding that outcome, on both the sending and receiving side of a webhook relationship.

Why "at-least-once" is the only honest guarantee

Distributed systems theory settled this a long time ago: you cannot build exactly-once delivery across an unreliable network. A sender can't know whether a request failed before or after the receiver processed it — the acknowledgment itself might be what got lost. Every major webhook provider, Stripe included, operates on at-least-once delivery and pushes the deduplication problem onto the receiver. Stripe's own documentation is explicit about this: if your endpoint is slow, crashes mid-request, or the network blips, Stripe retries with exponential backoff for up to 72 hours, and the same event.id can legitimately arrive more than once.

The practical implication: exactly-once processing is achievable, but only as an application-level property, built by treating every incoming webhook as a duplicate until proven otherwise.

Part 1: retry logic on the sending side

If you're the one delivering webhooks to customer endpoints — think an integration platform, a billing system, a marketplace notifying sellers — your retry logic needs four properties.

Bounded attempts with exponential backoff. A simple formula works: delay = base_delay * 2^attempt, capped at some maximum interval (commonly 15–60 minutes) and an overall deadline (24–72 hours is typical across providers). Retrying forever isn't kindness to the receiver; it's a resource leak on your side and a signal-loss problem on theirs, since a webhook delivered three days late is often worse than no webhook at all.

Full jitter, not just backoff. Without jitter, every event that failed against a struggling endpoint retries at the same intervals, so when the endpoint recovers it gets hit by every queued retry simultaneously — a self-inflicted thundering herd. Full jitter (sleep = random(0, base_delay * 2^attempt)) spreads retries out and is the AWS-recommended approach for this exact reason.

def backoff_delay(attempt, base=1, cap=900):
    max_delay = min(cap, base * (2 ** attempt))
    return random.uniform(0, max_delay)

Fast, decisive timeouts. A webhook call should time out in single-digit seconds (5 seconds is a common default). If the receiver takes 30 seconds to return a 200, that's not reliability, it's a queue backing up behind a slow consumer. Push all real work to an async worker on the receiver's side and acknowledge fast — more on this below.

Observable attempt history. Every attempt — timestamp, HTTP status, response body snippet, latency — should be logged and queryable, both for your own debugging and because customers integrating with your webhooks will ask "why didn't event X arrive" and the honest answer needs to be a delivery log, not a shrug.

Part 2: idempotent processing on the receiving side

This is the half of the system most teams get wrong, because it requires discipline on the consumer side of an integration you don't control the timing of.

Every webhook payload should carry a stable, unique event identifier — Stripe calls it event.id, GitHub uses a delivery UUID header, most systems have some equivalent. Your endpoint's first move, before touching any business logic, is to check whether you've already processed that ID.

CREATE TABLE processed_webhook_events (
  event_id      TEXT PRIMARY KEY,
  received_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
  source        TEXT NOT NULL
);

def handle_webhook(event):
    try:
        insert_processed_event(event.id, source=event.source)  # UNIQUE constraint does the work
    except UniqueViolation:
        return respond_200()  # already handled, ack and stop — this is not an error
    process_event(event)
    return respond_200()

The UNIQUE constraint on event_id is the actual deduplication mechanism — not an application-level if exists() check, which has a race condition when two retries arrive concurrently. Let the database enforce uniqueness and catch the constraint violation.

Do the insert and the state change atomically where possible. If marking an event "processed" and mutating application state (crediting an account, updating an order) happen in separate transactions, a crash between them reintroduces the exact duplicate-processing bug idempotency was supposed to prevent. Wrap both in one transaction, or make the state mutation itself idempotent (an upsert keyed on a business identifier, not just a counter increment).

Verify the signature before you do anything else — including parsing the body. Most providers sign webhooks with HMAC-SHA256 over the raw request bytes. Validate against the raw payload, not a re-serialized version of it (JSON key ordering and whitespace differences will silently break signature checks if you parse-then-verify instead of verify-then-parse). Include a timestamp in the signed payload and reject anything outside a tolerance window (5 minutes is common) — this closes the replay-attack window where someone captures a legitimately signed webhook and resends it later.

Acknowledge fast, process async. The pattern that scales:

Provider → Webhook endpoint
             │
             ├─ verify signature (before parsing)
             ├─ dedupe on event.id (DB unique constraint)
             ├─ persist raw event + enqueue job
             └─ return 2xx immediately
                         │
                         ▼
                  Async worker pool
                  (the actual business logic:
                   emails, billing, fulfillment)

Returning a 200 only after all business logic completes is the single most common cause of provider-side timeout-triggered retries, which in turn is the most common cause of duplicate processing. Decouple "I received this" from "I finished acting on this."

Part 3: the dead-letter queue

Retries are not infinite, and they shouldn't be. Once an event exhausts its retry budget — a typical policy is 5–8 attempts over several hours, though some systems extend to 72 hours — it needs a terminal state that isn't "silently dropped."

A dead-letter queue (DLQ) is that terminal state. The design goals:

• Preserve the full original request — headers, raw body, all retry attempt metadata — so a human (or an automated redrive job) can reprocess it without any information loss.
• Isolate failures per-event. One malformed or permanently-failing payload should never block the queue behind it. This is the core reason DLQs exist as a separate structure rather than "just retry forever in the main queue."
• Alert, don't just log. A DLQ that nobody looks at is a silent data-loss mechanism with extra steps. Wire an alert on DLQ depth or on any single message crossing a few days old.
• Support redrive. Once the root cause is fixed (endpoint was down, a bug in the consumer, a schema mismatch), you want to replay DLQ contents back through the same idempotent handler — which is exactly why Part 2's idempotency work pays for itself here. A redrive is just another delivery attempt to an idempotent consumer; nothing special has to happen for it to be safe.

A decision table for retry budgets

Treating a 422 the same as a 503 is a common and expensive mistake — retrying a malformed payload for 72 hours accomplishes nothing except delaying the alert that would have caught the bug sooner.

Failure modes worth designing against explicitly

Silent endpoint changes. A customer rotates their webhook secret or changes their endpoint URL without telling you. Signature verification failures should be distinguishable in your logs from delivery failures, and should alert differently — a spike in 401s from a previously-healthy endpoint is a different problem than a spike in timeouts.

Ordering assumptions. At-least-once delivery combined with retries means events can arrive out of order (a retried older event can land after a newer one that succeeded on the first try). If your consumer logic assumes ordering, include a sequence number or timestamp in the payload and have the consumer reject or reorder stale events rather than assuming arrival order is send order.

Retry storms after extended outages. If a consumer is down for hours, don't let every queued retry fire the instant it comes back. Combine jittered backoff with a rate limit on delivery per-endpoint so a recovering consumer gets a ramp, not a wall.

Monitoring and alerting checklist

A webhook pipeline that isn't instrumented will fail silently long before anyone notices, because the failure mode is "events stop arriving," not "the service crashes." A minimal monitoring setup should track:

• Delivery success rate per endpoint, not just globally — a single misconfigured customer endpoint shouldn't be invisible inside an otherwise-healthy aggregate number.
• DLQ depth and age of oldest item. Depth alone hides a DLQ that has one message sitting unreviewed for two weeks; age catches that.
• Retry attempt distribution. If most successful deliveries are succeeding on attempt 4 or 5 instead of attempt 1, that's an early signal of a degrading downstream dependency worth investigating before it fails outright.
• Duplicate-detection hit rate. A rising rate of UniqueViolation catches on the receiving side isn't necessarily bad — it means idempotency is doing its job — but a sudden spike usually correlates with a sender-side retry storm and is worth correlating against sender-side error rates.
• Signature verification failure rate, tracked separately from delivery failures, since a spike here points to a credential rotation or an active attack rather than a network problem.
• End-to-end latency from event generation to consumer acknowledgment, especially for use cases (payment confirmations, inventory updates) where staleness has a direct business cost.

None of this requires exotic tooling — a dashboard built on the delivery-attempt log described in Part 1, plus a handful of threshold alerts, covers the vast majority of what teams need to catch problems before customers report them.

Testing the system before it's under load

Webhook reliability code is exactly the kind of thing that looks correct in review and then breaks the first time it meets a real retry storm, because the failure paths (duplicate delivery, out-of-order arrival, DLQ redrive) are rarely exercised in normal development. Worth building into a test suite explicitly:

• A test that fires the same event twice in quick succession and asserts the second call is a no-op with respect to side effects (no duplicate email, no double charge).
• A test that simulates a slow consumer (artificial delay past the timeout threshold) and confirms the sender retries rather than treating it as a permanent failure.
• A test that pushes a deliberately malformed payload through the pipeline and confirms it lands in the DLQ on the first attempt rather than burning through the full retry budget.
• A load test that fires a burst of retries simultaneously (simulating a consumer recovering after an outage) and confirms jitter actually spreads the load rather than converging on the same intervals by coincidence of test timing.

Teams that skip this kind of test coverage tend to discover their idempotency logic was subtly wrong — often a race condition between the duplicate check and the state mutation — during an actual incident rather than in a code review, which is the more expensive way to find it.