Zero-Downtime Secret Rotation for Webhook Signing Keys

Every webhook signing key has a useful life. A key that never rotates is one compromise away from an attacker forging events that look legitimate to every consumer you have. Whether your rotation is driven by a security audit, a suspected breach, a compliance requirement, or just policy ("rotate every 90 days"), the question is always the same: how do you swap the secret without breaking anything?

The naive approach — update the secret, redeploy, tell your consumers to update theirs — creates a window where either you're still signing with the old key (after consumers update) or consumers are validating with the old key (after you update). Either way, you get verification failures. In a production system that means dropped events, manual re-ingestion, and support tickets.

The right approach is a dual-secret overlap. This post explains how it works, how to implement it on both the sending and receiving side, and what operational runbook to follow when you do a rotation.

Why Signing Key Rotation Is Hard

When you sign a webhook with HMAC-SHA256, you produce a signature that ties the payload to a specific secret. The consumer verifies the signature with the same secret. If the secrets differ, verification fails.

The coordination problem: your producer and all consumers must agree on the active secret at the same point in time. In practice, you cannot do this atomically. Consumers are third-party systems you don't deploy. Even your own internal services have rolling deployments where some pods run the old code and some run the new.

Any approach that requires simultaneous cutover is operationally fragile. You need a transition period where both the old and new secrets are valid.

The Dual-Secret Overlap Pattern

The solution is to treat a rotation as three phases, not one:

During the overlap window, the producer signs with both secrets (sending two signatures in the header), and consumers accept a signature valid under either secret. Once you confirm all consumers have updated to the new secret, you retire the old one.

This eliminates the coordination requirement. Consumers can update at any time during the overlap window. The window can be hours, days, or weeks — whatever your operational tempo requires.

Implementing Dual Signatures on the Sending Side

The Webhook-Signature header (or X-Webhook-Signature, depending on your format) needs to carry both signatures during the overlap period.

Using a Stripe-compatible format (t=<timestamp>,v1=<sig>), the dual-signature header looks like:

Webhook-Signature: t=1742947200,v1=aef3c8b2...,v1=9d1f7a45...

Two v1= entries, one per active secret. Here's how to produce that in Go:

type SigningSecret struct {
    ID     string
    Secret []byte
}

func SignPayload(payload []byte, timestamp int64, secrets []SigningSecret) string {
    message := fmt.Sprintf("%d.%s", timestamp, payload)

    var parts []string
    parts = append(parts, fmt.Sprintf("t=%d", timestamp))

    for _, s := range secrets {
        mac := hmac.New(sha256.New, s.Secret)
        mac.Write([]byte(message))
        sig := hex.EncodeToString(mac.Sum(nil))
        parts = append(parts, "v1="+sig)
    }

    return strings.Join(parts, ",")
}

When secrets has one entry, you get a normal single signature. When it has two entries (old and new), you get a dual-signature header. The consumer's verification function must accept any valid signature in the header.

Implementing Dual-Secret Verification on the Receiving Side

Consumer verification needs to accept a signature valid under any of the provided secrets. Most webhook verification libraries do this already if you pass multiple secrets — but check before assuming.

func VerifySignature(payload []byte, header string, secrets [][]byte, tolerance time.Duration) error {
    parts := strings.Split(header, ",")

    var timestamp int64
    var signatures []string

    for _, part := range parts {
        if strings.HasPrefix(part, "t=") {
            t, err := strconv.ParseInt(strings.TrimPrefix(part, "t="), 10, 64)
            if err != nil {
                return errors.New("invalid timestamp in signature header")
            }
            timestamp = t
        }
        if strings.HasPrefix(part, "v1=") {
            signatures = append(signatures, strings.TrimPrefix(part, "v1="))
        }
    }

    if timestamp == 0 {
        return errors.New("missing timestamp in signature header")
    }

    // Reject replays outside the tolerance window
    age := time.Since(time.Unix(timestamp, 0))
    if age > tolerance || age < -tolerance {
        return fmt.Errorf("signature timestamp out of tolerance window: age=%s", age)
    }

    message := fmt.Sprintf("%d.%s", timestamp, payload)

    for _, secret := range secrets {
        mac := hmac.New(sha256.New, secret)
        mac.Write([]byte(message))
        expected := hex.EncodeToString(mac.Sum(nil))

        for _, sig := range signatures {
            if hmac.Equal([]byte(sig), []byte(expected)) {
                return nil // Any valid signature under any accepted secret passes
            }
        }
    }

    return errors.New("no valid signature found")
}

The key behavior: iterate all secrets, iterate all signatures in the header, accept if any combination is valid. Use hmac.Equal for constant-time comparison to avoid timing attacks.

Storing Multiple Active Secrets

On the producer side, you need a data model that supports multiple active secrets per destination:

CREATE TABLE signing_secrets (
    id           UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    destination_id UUID NOT NULL REFERENCES destinations(id),
    secret_enc   BYTEA NOT NULL,       -- AES-256-GCM encrypted
    label        TEXT NOT NULL,        -- e.g. "primary", "retiring"
    status       TEXT NOT NULL DEFAULT 'active',
    -- active | retiring | revoked
    created_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
    revoked_at   TIMESTAMPTZ
);

CREATE INDEX ON signing_secrets (destination_id)
    WHERE status = 'active';

At signing time, load all active secrets for the destination and sign with all of them. During normal operation there is one active secret. During a rotation there are two. After the old secret is retired, there is one again.

Never store secrets in plaintext. Encrypt with AES-256-GCM before writing to the database, decrypt at signing time. A compromised database dump should not expose signing material.

The Rotation Runbook

A rotation has five steps. Document and automate as much of this as possible before your first production rotation.

Step 1: Generate the new secret. Generate a cryptographically random secret (32 bytes minimum). Insert it into signing_secrets with status active. Do not revoke the old secret yet.

# Generate a new 32-byte secret, base64-encoded
NEW_SECRET=$(openssl rand -base64 32)

# Insert via your management API or directly
curl -X POST https://api.yoursaas.com/v1/destinations/$DEST_ID/secrets \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"label\": \"primary-2026-03\"}"

Step 2: Verify dual-signature delivery is working. Send a test event. Confirm the Webhook-Signature header contains two v1= entries. Confirm the consumer's existing verification code accepts the event (it should, since the old signature is still present).

Step 3: Distribute the new secret to consumers. This is the human/operational step. Notify consumers (internal teams or external customers) that a new secret is available. Give them a deadline to update. Most webhook platforms expose the new secret through a dashboard or API at this point.

Step 4: Wait for the overlap window to expire. Monitor your delivery logs. Once you stop seeing consumers validate with the old secret — or once your deadline has passed — you can retire it.

Step 5: Revoke the old secret. Update the old secret's status to revoked. Set revoked_at. From this point, only the new secret is used to sign, and only the new secret should be accepted by consumers.

UPDATE signing_secrets
SET status = 'revoked', revoked_at = now()
WHERE destination_id = $1
  AND label = 'primary-2026-02';

What Can Go Wrong

You revoke the old secret too early. Consumers who haven't updated yet will start seeing verification failures. Their processing stops, events accumulate in your retry queue, and you have a support incident. The overlap window exists to prevent this — be generous with it.

You never revoke the old secret. An indefinite overlap window defeats the purpose of rotation. If you're running with two active secrets permanently, you've doubled your attack surface. Set a calendar reminder when you start the overlap; actually revoke at the end.

You rotate without telling consumers. Even with a dual-signature overlap, consumers need to know the new secret exists. If they're hardcoding the old secret in their configuration, they will eventually fail when the overlap window closes. Proactive communication is part of rotation.

Encrypted secret decryption fails after key rotation. If your encryption key also rotates, you need to re-encrypt stored signing secrets with the new encryption key before the old one is retired. These are two separate rotation concerns — keep them separate operationally.

Automating Rotation at Scale

For platforms with many customers and per-customer signing secrets, manual rotation doesn't scale. The same pattern applies automatically:

1. ›On a rotation schedule (or triggered by an API call), generate a new secret and insert it alongside the old one
2. ›Mark the old secret as retiring with a timestamp
3. ›A background job monitors retiring secrets; once the configured overlap duration has passed, mark them revoked
4. ›Emit an event notification to the customer when their secret is rotated, with instructions to update

This is how GetHook handles signing secret rotation for managed destinations — the overlap window is configurable per destination, defaulting to 72 hours. The dual-signature delivery happens automatically during the window without any consumer coordination required.

Summary

Zero-downtime secret rotation requires:

1. ›A data model that supports multiple active secrets per destination simultaneously
2. ›A sender that signs with all active secrets and includes all signatures in the header
3. ›A receiver that accepts a payload if any signature in the header is valid under any accepted secret
4. ›A disciplined three-phase rotation: add new → overlap window → retire old
5. ›Encrypted storage for all secret material — a database dump should not expose signing keys

The dual-secret overlap pattern eliminates the coordination window that makes naive rotation fragile. The complexity is contained to the signing and verification logic; the rest of your system doesn't need to change.

If you want signing secret rotation built into your webhook infrastructure without managing it yourself, GetHook handles rotation, overlap windows, and encrypted secret storage out of the box.

Get started with GetHook →