Ordering Guarantees in Webhook Delivery: When They Matter and How to Enforce Them

When you're building a payment integration, you eventually hit an event like this: your system processes charge.succeeded for a charge that your records show as still pending, because charge.pending arrived two seconds later. The customer's order looks paid. Your internal state says otherwise. Reconciliation runs at midnight and flags the discrepancy.

Out-of-order webhook delivery isn't a bug in any one system — it's the default behavior of distributed infrastructure. HTTP requests race. Retries interleave with new deliveries. Workers process jobs in parallel. The question isn't whether events will arrive out of order, but whether your system handles it correctly when they do.

This post covers how to reason about ordering guarantees, which scenarios actually require strict ordering, and the concrete techniques for enforcing it when you need it.

Why Webhooks Don't Guarantee Order by Default

Every step in the delivery pipeline is a potential reordering point:

1. ›

   Provider-side queuing. The provider may buffer events internally before delivery. If their queue has multiple workers, worker A might pick up event E2 before worker B finishes delivering E1.

2. ›

   Network variance. Two HTTP requests sent 50ms apart can arrive in any order. A TCP connection reset on the first request adds a retry delay; the second request lands first.

3. ›

   Your delivery worker. If your worker processes a batch of 10 jobs in parallel, events for the same destination are delivered concurrently without any sequencing.

4. ›

   Retries. An event that fails and is retried with exponential backoff will arrive after events that were sent after it but delivered successfully on the first attempt.

5. ›

   Fan-out latency. In a fanout scenario, E1 may be delivered to destination A immediately but delayed to destination B due to a transient error — while E2 has already been delivered to both.

This is not a design flaw you can patch around the edges. It's the natural result of building reliable systems from unreliable components. The correct response is to design consumers that tolerate it.

When Out-of-Order Delivery Is Fine

For the majority of event types, out-of-order delivery causes no harm, because the events describe independent facts:

For these, building consumers that are idempotent (safe to process more than once) and commutative (produce the same result regardless of order) is the right approach. Accept any order, track which event IDs you've processed, and move on.

When Out-of-Order Delivery Causes Real Bugs

The dangerous scenarios are state machines where event order determines the final state:

Payment lifecycle transitions. A charge goes created → pending → succeeded → refunded. If refunded arrives before succeeded, your system might mark an order as refunded that it never recorded as paid — or ignore the refund because it can't apply to a "pending" charge.

Inventory reservations. inventory.reserved followed by inventory.released leaves stock available. If released arrives first and your handler ignores it (no reservation to release), then reserved arrives and creates a reservation that's never released.

Account provisioning. account.suspended followed by account.reactivated should leave the account active. Reversed, you permanently suspend an account that had been reactivated.

Audit log integrity. If you're building an immutable audit log from webhook events, out-of-order delivery creates a log that contradicts the actual sequence of events.

The common thread: these are all state machines where applying transition T2 without T1 having been applied first produces incorrect state.

Technique 1: Sequence Numbers in Event Payloads

The cheapest defense is to use the event metadata the provider gives you. Most mature webhook providers include either a sequence number or a monotonically increasing timestamp in their events.

Stripe includes created (Unix timestamp) on every event object. GitHub includes X-GitHub-Delivery (UUID) but no sequence number. Shopify includes X-Shopify-Webhook-Id and the updated_at field on the resource.

When a provider gives you sequence numbers or timestamps, use them:

type EventRecord struct {
    ID           string
    ResourceID   string
    EventType    string
    SequenceNum  int64   // provider's sequence number or event timestamp
    Payload      []byte
    ProcessedAt  *time.Time
}

func (h *PaymentHandler) Apply(ctx context.Context, event EventRecord) error {
    current, err := h.store.GetCurrent(ctx, event.ResourceID)
    if err != nil {
        return err
    }

    // Reject events older than the last one we processed
    if current != nil && event.SequenceNum <= current.LastSequenceNum {
        log.Printf("skipping out-of-order event %s: seq %d <= current %d",
            event.ID, event.SequenceNum, current.LastSequenceNum)
        return nil // not an error, just a no-op
    }

    return h.store.ApplyTransition(ctx, event)
}

This works well when events are independent across resources (each ResourceID has its own sequence) and when the provider's sequence numbers are reliable. The main risk: if the provider reuses sequence numbers after a retry, you'll incorrectly discard a valid re-delivery. Read the provider's documentation carefully.

Technique 2: Conditional Updates with Version Fields

A more robust approach that doesn't rely on provider-issued sequence numbers is tracking a version on your own state:

CREATE TABLE subscriptions (
    id           UUID PRIMARY KEY,
    account_id   UUID NOT NULL,
    status       TEXT NOT NULL,
    version      INT NOT NULL DEFAULT 1,
    updated_at   TIMESTAMPTZ NOT NULL DEFAULT now()
);

Every transition requires the caller to specify the version they're updating from:

func (s *SubscriptionStore) Transition(
    ctx context.Context,
    subscriptionID string,
    fromVersion int,
    newStatus string,
) (updated bool, err error) {
    result, err := s.db.ExecContext(ctx, `
        UPDATE subscriptions
        SET status = $1, version = version + 1, updated_at = now()
        WHERE id = $2 AND version = $3
    `, newStatus, subscriptionID, fromVersion)
    if err != nil {
        return false, err
    }

    rowsAffected, err := result.RowsAffected()
    return rowsAffected == 1, err
}

If version doesn't match, the update is a no-op and you know an out-of-order event arrived. You can then decide to dead-letter it, retry after a delay, or discard it depending on your business rules.

The key insight: optimistic concurrency control naturally prevents out-of-order transitions. An event that arrives late will find the version has already advanced past what it expected.

Technique 3: Per-Destination Delivery Serialization

For scenarios where you genuinely need strict ordering at the delivery layer — not just at the consumer — you need to serialize delivery per destination. This means no two events for the same destination are in-flight simultaneously.

The trade-off is significant: serialized delivery eliminates the parallelism that makes webhook delivery fast. For a destination that takes 200ms per event, you can deliver at most 5 events/second. For most destinations this is acceptable. For high-volume streams it's not.

Implementation with a Postgres queue:

-- Add a per-source sequence number at ingest time
ALTER TABLE events ADD COLUMN source_sequence BIGSEQ GENERATED ALWAYS AS IDENTITY;

-- Claim only the lowest-sequence undelivered event per destination
SELECT da.*
FROM delivery_attempts da
JOIN events e ON e.id = da.event_id
WHERE da.destination_id = $1
  AND da.status = 'queued'
ORDER BY e.source_sequence ASC
LIMIT 1
FOR UPDATE SKIP LOCKED;

By only claiming the next-in-sequence event per destination, you ensure that E2 is never delivered before E1. The downside: if E1 is stuck retrying, E2 (and all subsequent events) wait behind it. This is the correct behavior for true strict ordering — but it means one bad delivery blocks the queue for that destination.

The Practical Tradeoff Matrix

Choosing the right approach depends on your use case:

The last row is worth calling out. You don't have to choose between "all ordering" and "no ordering." If only payment state events require strict ordering, route them to a separate destination configured with serialized delivery. Everything else — notifications, analytics, non-critical updates — goes to a parallel destination with no ordering constraints.

What GetHook Does

GetHook's delivery model processes per-destination jobs in parallel by default, which is the right behavior for the majority of workloads. For teams that need per-destination serialization, the route configuration supports ordered delivery mode — which claims one event at a time per destination, ordered by ingestion sequence.

The event timeline in the GetHook dashboard shows each event's delivery attempts with timestamps, making it straightforward to identify when out-of-order delivery has occurred and reconstruct the actual sequence.

If you're building a payment integration or financial state machine on top of webhooks, the combination of optimistic version control at the consumer and sequence-aware event inspection in the delivery layer covers most real-world ordering requirements without introducing the full cost of serialized delivery.

Summary

Out-of-order webhook delivery is the default, not the exception. The right response depends on what you're building:

• ›For most event types, idempotency is sufficient. Deduplicate by event ID, ignore duplicates, don't assume order.
• ›For state machines, use optimistic version control — require a version match on every state transition. Out-of-order events fail the version check and are either dead-lettered or discarded.
• ›For strict ordering requirements at the delivery layer, serialize per destination — but accept that this constrains your throughput.
• ›For mixed workloads, route ordered and unordered events to separate destinations with different delivery policies.

The mistake most teams make is not thinking about ordering until a bug surfaces in production. Adding a version column to your state tables costs almost nothing upfront and prevents an entire class of subtle data corruption bugs.

Start building reliable webhook infrastructure with GetHook →