At-Least-Once Delivery: Writing Webhook Consumers That Handle Duplicates Safely

Every webhook provider — Stripe, GitHub, Shopify, and your own outbound system — delivers events with an at-least-once guarantee. This means the same event ID may arrive at your endpoint two, three, or even a dozen times. Network retries, provider-side delivery bugs, and your own 5xx responses during a restart all cause re-delivery.

This is not a bug. It is an intentional property of reliable distributed systems. But it means the contract between you and any webhook provider is: "You will eventually receive this event at least once. You may receive it more than once. Handle it."

The failure mode is silent. A duplicate invoice.paid event that double-charges a customer, sends two fulfillment requests to a 3PL, or provisions two accounts is not caught by any monitoring unless you specifically build for it.

This post walks through exactly how to build webhook consumers that tolerate duplicates correctly.

Why Re-Delivery Happens

Understanding the source of duplicates helps you design defenses. The three main causes:

1. Your 5xx response during a restart. Your deploy restarts the process mid-request. The provider sees a connection reset or 502 and retries the event — even though your handler already ran the business logic.

2. Provider-side retry with no confirmation. The provider sends the event, your endpoint responds 200, but the provider's delivery confirmation step fails internally. It retries conservatively.

3. Your network stack responds before your handler completes. A reverse proxy (nginx, envoy) times out and returns 504 before your handler finishes. The provider retries. Your handler eventually succeeds. Now both the original and retry have been processed.

The correct mental model: a 200 response confirms receipt, not processing. Providers retry until they get a 200. Your processing may have already happened.

The Idempotency Key Is the Event ID

Every well-designed webhook provider includes a stable event ID in the payload:

{
  "id": "evt_01HX9P3KQ2ZVNR7Y8W4M",
  "type": "invoice.paid",
  "created_at": "2026-03-29T10:44:00Z",
  "data": {
    "invoice_id": "inv_abc123",
    "amount_paid": 14900,
    "currency": "usd"
  }
}

This id is your idempotency key. The provider guarantees that all deliveries of the same logical event share the same id. Your job is to record which event IDs you have already processed and skip re-processing if you see one again.

Implementation: The Idempotency Table

The simplest robust implementation uses a dedicated table:

CREATE TABLE processed_webhook_events (
    event_id        TEXT        NOT NULL,
    source          TEXT        NOT NULL,   -- e.g. 'stripe', 'github'
    processed_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
    outcome         TEXT        NOT NULL,   -- 'success', 'skipped', 'failed'
    PRIMARY KEY (source, event_id)
);

CREATE INDEX ON processed_webhook_events (processed_at);

The PRIMARY KEY (source, event_id) ensures uniqueness. The processed_at index supports retention cleanup (covered below).

Your handler logic follows this pattern:

func (h *WebhookHandler) Handle(w http.ResponseWriter, r *http.Request) {
    var event WebhookEvent
    if err := json.NewDecoder(r.Body).Decode(&event); err != nil {
        http.Error(w, "bad request", http.StatusBadRequest)
        return
    }

    // Attempt to record the event atomically.
    // If the INSERT fails with a unique constraint violation,
    // this event has already been processed — return 200 immediately.
    inserted, err := h.store.MarkProcessed(r.Context(), event.Source, event.ID)
    if err != nil {
        // Transient DB error — return 500 so the provider retries later.
        http.Error(w, "internal error", http.StatusInternalServerError)
        return
    }
    if !inserted {
        // Already processed. Return 200 to stop further retries.
        w.WriteHeader(http.StatusOK)
        return
    }

    // Now run your business logic.
    if err := h.processEvent(r.Context(), event); err != nil {
        // Mark as failed so you can inspect it later,
        // but still return 500 to allow provider retry.
        h.store.MarkFailed(r.Context(), event.Source, event.ID)
        http.Error(w, "processing error", http.StatusInternalServerError)
        return
    }

    w.WriteHeader(http.StatusOK)
}

The MarkProcessed function uses INSERT ... ON CONFLICT DO NOTHING and returns whether the row was actually inserted:

func (s *Store) MarkProcessed(ctx context.Context, source, eventID string) (bool, error) {
    result, err := s.db.ExecContext(ctx,
        `INSERT INTO processed_webhook_events (event_id, source, outcome)
         VALUES ($1, $2, 'success')
         ON CONFLICT (source, event_id) DO NOTHING`,
        eventID, source,
    )
    if err != nil {
        return false, err
    }
    rows, _ := result.RowsAffected()
    return rows == 1, nil
}

This is atomic. Two concurrent requests carrying the same event ID will race on the INSERT — exactly one will succeed, and the other will see zero rows affected and short-circuit. No application-level locking required.

The Race Condition You Can't Avoid (and How to Handle It)

There is one gap in the pattern above: the window between "INSERT succeeded" and "business logic completes." If your process crashes or is killed in that window, the event is recorded as processed but the business logic never ran.

The crash window is unavoidable with this design. There are two ways to handle it:

Option A: Mark failed on crash recovery. On startup, query for events with outcome = 'success' and processed_at > now() - interval '5 minutes' that don't have a corresponding record in your business table. Re-process them.

Option B: Use a transaction. Insert into the idempotency table and perform the business logic within the same database transaction. This is only viable when your business logic is a database write — not an HTTP call or external side effect.

tx, err := db.BeginTx(ctx, nil)
// INSERT into processed_webhook_events
// INSERT or UPDATE into your business table
// tx.Commit()

Option B is cleaner but only works for pure database operations. If your handler calls an external API (send an email, charge a card), you cannot include it in a transaction.

For external side effects, Option A — or accepting the rare silent drop — is the pragmatic choice. The key is to know which events landed in that window (check your idempotency table against your business table) and decide consciously.

Designing Business Logic to Be Idempotent

Even with an idempotency table, designing your business logic to tolerate duplicates is worth the effort — because the table can be unavailable (DB failover), and because it simplifies debugging.

The core principle: make every write use upsert semantics, not insert.

-- ❌ Will fail or double-charge on duplicate
INSERT INTO invoices (id, status, paid_at) VALUES ($1, 'paid', now());

-- ✅ Idempotent
INSERT INTO invoices (id, status, paid_at)
VALUES ($1, 'paid', now())
ON CONFLICT (id) DO UPDATE
  SET status = 'paid', paid_at = COALESCE(invoices.paid_at, EXCLUDED.paid_at);

The COALESCE on paid_at ensures a replay doesn't overwrite the original payment timestamp with the replay time — only set it if it was previously null.

For external API calls, check before acting:

// Check if the customer was already provisioned before provisioning again
existing, _ := crm.GetCustomer(ctx, event.CustomerID)
if existing != nil {
    return nil // already exists, skip
}
return crm.CreateCustomer(ctx, event.CustomerID, event.Email)

This pattern — "check then act" — has a TOCTOU race condition, but for webhook processing at normal throughput, it is almost always sufficient. The idempotency table handles the true concurrent case.

Retention: Keep Your Idempotency Table Small

The idempotency table grows unbounded unless you prune it. Most providers retry for 24–72 hours. A seven-day retention window covers even the most aggressive retry schedules:

-- Run daily via cron or pg_cron
DELETE FROM processed_webhook_events
WHERE processed_at < now() - interval '7 days';

Add a partial index to make this fast:

CREATE INDEX ON processed_webhook_events (processed_at)
WHERE processed_at < now() - interval '6 days';

At 100K events/day, a seven-day retention window is ~700K rows — small enough to fit in memory for most Postgres configurations.

What GetHook Provides

If you use GetHook as your inbound gateway, each event receives a stable id assigned at ingestion time. Even if the upstream provider sends the same event twice with inconsistent IDs (some providers do this), GetHook deduplicates at the ingest layer based on a content hash before the event reaches your consumer.

This shrinks your idempotency surface: instead of handling provider-level duplicates, you only handle GetHook delivery retries — which carry a consistent, stable ID every time.

Your idempotency table still belongs in your application. GetHook handles provider-side duplication; your table handles delivery-side duplication.

Checklist

Building webhook consumers with these patterns takes an afternoon. Not building them takes an incident. Start with the idempotency table — it handles 95% of real-world duplicate scenarios — and layer in upsert business logic from there.

Ready to simplify your inbound webhook infrastructure? Get started with GetHook and let the gateway handle provider-level deduplication before events reach your consumers.