Webhook Exactly-Once Delivery: Why It's Impossible and What to Do Instead

At some point, a customer will ask you: "Can your webhook system guarantee that each event is delivered exactly once?"

The correct answer is no. Not because your infrastructure is incomplete, but because exactly-once delivery is a distributed systems impossibility under normal network conditions. Understanding why — and knowing what guarantees you can offer — is what separates webhook systems that hold up in production from those that silently lose or double-process events.

Why Exactly-Once Is Impossible

The impossibility comes from a fundamental problem in distributed systems: you cannot atomically perform an HTTP delivery and confirm the recipient processed it.

Consider the sequence of events during a delivery:

1. ›Your worker sends the HTTP POST to the destination.
2. ›The destination receives the payload and processes it (charges a card, sends an email, marks an order fulfilled).
3. ›The destination sends a 200 OK response.
4. ›Your worker receives the 200 OK and marks the event as delivered.

Now ask: what happens if the network drops between steps 3 and 4? Your worker never sees the 200 OK. From its perspective, the delivery failed. So it retries. The destination receives the same event again — and processes it again.

The destination has no way to distinguish a retry from a new event unless it has been built to do so. Your delivery layer has no way to know the destination already processed the event. This is the two generals problem applied to HTTP: you cannot confirm both sides have agreed on the outcome of a communication without the possibility of the confirmation message itself being lost.

The only way to guarantee exactly-once delivery would be for you and your customer's server to share a transactional commit protocol — and that would require your delivery worker and their application to participate in a two-phase commit across a network boundary. No production system does this for webhooks.

What You Can Guarantee Instead

The realistic guarantees you can offer are:

At-most-once is the weakest. It's the default behavior if your system has no retry logic — and it means your customers lose events every time there's a transient network hiccup, a deployment restart, or a cold container startup. Do not offer at-most-once for anything important.

At-least-once is what well-engineered webhook systems provide. It's the right trade-off: durability over uniqueness. Retries are safe as long as consumers are built to handle them — and "built to handle them" means idempotent consumers.

Effectively-once is the operational outcome your customers actually want. It requires two things working together: your delivery layer providing at-least-once guarantees with a stable event ID per event, and consumers using that ID to deduplicate on their end.

Designing for At-Least-Once: The Delivery Layer

Your delivery layer needs three properties to provide solid at-least-once guarantees:

1. Stable event IDs. Every event must have a unique, immutable identifier that survives retries. If you retry an event, the id in the payload must be the same as the original attempt. Never generate a new ID per delivery attempt.

2. Idempotent HTTP semantics. Your delivery worker must retry on any non-2xx response, including timeouts, without creating a new event. The retry is a re-delivery of the same event, not a new one.

3. Durable event storage before delivery. Events must be persisted to your database before any delivery attempt begins. If your worker crashes mid-delivery, the event must still be in the queue waiting to be picked up again.

Here's how a durable delivery worker handles this correctly in Go:

func (w *Worker) processEvent(ctx context.Context, event Event) error {
    for attempt := 1; attempt <= maxAttempts; attempt++ {
        outcome, err := w.forwarder.Deliver(ctx, event, attempt)
        if err == nil && outcome.Success {
            return w.store.MarkDelivered(ctx, event.ID)
        }

        // Record the failed attempt — same event.ID, different attempt number
        if recordErr := w.store.RecordAttempt(ctx, DeliveryAttempt{
            EventID:       event.ID,   // stable across retries
            AttemptNumber: attempt,
            Outcome:       outcome,
        }); recordErr != nil {
            return recordErr
        }

        if attempt < maxAttempts {
            delay := backoff(attempt)
            if err := w.store.ScheduleRetry(ctx, event.ID, time.Now().Add(delay)); err != nil {
                return err
            }
            return nil // release the job; it will be picked up again after delay
        }
    }

    return w.store.MarkDeadLetter(ctx, event.ID)
}

The key detail: event.ID is the same on every attempt. The attempt number increments, but the event identity does not change. This is what allows consumers to deduplicate correctly.

Designing for Effectively-Once: The Consumer Side

The consumer's job is to make processing idempotent. The pattern is straightforward: record event IDs you have already processed, and skip re-processing if you see one again.

CREATE TABLE processed_webhook_events (
    event_id    TEXT        PRIMARY KEY,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

In your webhook handler:

func (h *Handler) HandleWebhook(w http.ResponseWriter, r *http.Request) {
    var payload WebhookPayload
    if err := json.NewDecoder(r.Body).Decode(&payload); err != nil {
        http.Error(w, "bad request", http.StatusBadRequest)
        return
    }

    // Attempt to record the event ID atomically
    _, err := h.db.ExecContext(r.Context(), `
        INSERT INTO processed_webhook_events (event_id)
        VALUES ($1)
        ON CONFLICT (event_id) DO NOTHING
    `, payload.ID)
    if err != nil {
        http.Error(w, "internal error", http.StatusInternalServerError)
        return
    }

    // Check if we actually inserted (rows affected = 0 means duplicate)
    var count int
    h.db.QueryRowContext(r.Context(), `
        SELECT COUNT(*) FROM processed_webhook_events
        WHERE event_id = $1
          AND processed_at >= NOW() - INTERVAL '1 second'
    `, payload.ID).Scan(&count)

    if count == 0 {
        // Already processed — return 200 so the sender stops retrying
        w.WriteHeader(http.StatusOK)
        return
    }

    // Process the event exactly once
    if err := h.processEvent(r.Context(), payload); err != nil {
        // Don't return 200 here — let the sender retry
        http.Error(w, "processing failed", http.StatusInternalServerError)
        return
    }

    w.WriteHeader(http.StatusOK)
}

A cleaner approach uses INSERT ... ON CONFLICT DO NOTHING and checks sql.Result.RowsAffected():

result, err := h.db.ExecContext(ctx, `
    INSERT INTO processed_webhook_events (event_id) VALUES ($1)
    ON CONFLICT (event_id) DO NOTHING
`, payload.ID)
if err != nil {
    return err
}

rows, _ := result.RowsAffected()
if rows == 0 {
    // Duplicate — already processed
    return nil
}

return h.processEvent(ctx, payload)

This is safe as long as processEvent and the insert happen in the same database transaction — or processEvent is itself idempotent (e.g., it uses an INSERT ... ON CONFLICT DO UPDATE internally). If your business logic is not transactional with the deduplication insert, you can still get duplicates in the event of a crash between processing and committing.

The Deduplication Window

You don't need to keep event IDs forever. Most webhook providers guarantee uniqueness within a bounded window — typically 24 to 72 hours. A standard practice is to keep event IDs for slightly longer than the maximum retry window of the sender.

If the sender retries for up to 72 hours (as Stripe does), keep event IDs for 96 hours:

-- Run this as a scheduled job every few hours
DELETE FROM processed_webhook_events
WHERE processed_at < NOW() - INTERVAL '96 hours';

Add an index on processed_at to make the cleanup fast:

CREATE INDEX processed_webhook_events_processed_at
    ON processed_webhook_events (processed_at);

At 10,000 webhook events per day, your deduplication table will hold roughly 400,000 rows at steady state — a manageable size with minimal storage cost.

What This Means for Your API Docs

When you document your webhook delivery semantics, be precise. Don't write "we deliver webhooks reliably" — write what you actually guarantee:

Events are delivered with at-least-once semantics. Each event has a stable id field that is identical across all retry attempts. Consumers should use this id to deduplicate events and ensure idempotent processing.

This sentence does three things: it sets accurate expectations, tells consumers exactly what to key their deduplication on, and implicitly admits that retries happen. Customers who read this and implement deduplication will never have a duplicate-processing bug. Customers who skip it will eventually notice — but that's on them.

GetHook includes the stable event ID in every delivered payload and records each attempt separately, so your customers can always inspect the full delivery history for a given event ID and verify whether they acknowledged it correctly.

Summary

Exactly-once delivery is not achievable across a network boundary without a shared transactional protocol. The practical engineering answer is:

• ›Your delivery layer provides at-least-once delivery with stable event IDs and exponential backoff retries.
• ›Your consumers provide idempotent processing using those event IDs as deduplication keys.
• ›Together, they achieve effectively-once behavior — no duplicate side effects, no lost events.

The two-layer approach is more robust than any guarantee a single side could offer alone, because it survives failures at both ends without coordination.

Ready to build webhook infrastructure with durable at-least-once delivery and full event history? Get started with GetHook →