Mirroring Production Webhook Traffic to Staging

Synthetic test fixtures are a necessary starting point, but they diverge from reality faster than most teams realize. The Stripe invoice.payment_failed event in your test suite has five fields. The one that arrives in production has eleven, including two your code has never seen. Your GitHub Actions webhook fixture uses ref: "refs/heads/main". One Monday morning a contributor pushes directly to a release branch and your handler silently drops the event because the prefix check doesn't match.

Traffic mirroring — cloning live production webhook events and replaying them against a staging environment — closes this gap. It lets you validate new handler code against the full entropy of real provider behavior, not just the subset you remembered to document.

This post covers the mechanics: how to split traffic at the ingest layer, how to protect your staging environment from side effects, what to do with the responses, and how to operationalize the whole thing without creating a compliance headache.

The Core Architecture

The idea is straightforward: when a webhook arrives at your production ingest endpoint, forward a copy to your staging endpoint in parallel, asynchronously. The production response path is untouched. Staging processes the copy independently.

Provider ──→ Production Ingest
                │
                ├──→ Production handler (real path)
                │
                └──→ [mirror] Staging Ingest (async, fire-and-forget)

The key properties of a correct mirror:

1. ›Asynchronous. The mirror write must not block the production response. A slow or offline staging environment should never cause production latency.
2. ›Fire-and-forget. Mirror delivery failures must not surface as errors on the production path. Log them; don't alert on them like you would production failures.
3. ›Sanitized. Some payloads contain production customer data. Before mirroring, you may need to strip or mask fields. More on this below.
4. ›Non-authoritative. Staging handlers must not take actions that affect production state — no real emails, no payment captures, no external API calls.

Implementing the Mirror at the Ingest Layer

If you control your ingest layer, adding a mirror is a straightforward HTTP fork. Here's a minimal Go implementation:

type MirrorConfig struct {
    Enabled    bool
    TargetURL  string
    SampleRate float64 // 0.0–1.0; 1.0 = mirror all traffic
    Timeout    time.Duration
}

func mirrorAsync(cfg MirrorConfig, body []byte, headers http.Header) {
    if !cfg.Enabled {
        return
    }
    if cfg.SampleRate < 1.0 && rand.Float64() > cfg.SampleRate {
        return
    }

    go func() {
        ctx, cancel := context.WithTimeout(context.Background(), cfg.Timeout)
        defer cancel()

        req, err := http.NewRequestWithContext(ctx, http.MethodPost, cfg.TargetURL, bytes.NewReader(body))
        if err != nil {
            log.Printf("mirror: failed to create request: %v", err)
            return
        }

        // Copy relevant headers; strip provider signatures (staging can't verify them)
        for key, vals := range headers {
            if isSafeToMirror(key) {
                req.Header[key] = vals
            }
        }
        // Mark as a mirror so staging knows not to re-emit downstream side effects
        req.Header.Set("X-Mirror-Source", "production")

        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            log.Printf("mirror: delivery failed to %s: %v", cfg.TargetURL, err)
            return
        }
        defer resp.Body.Close()
        log.Printf("mirror: delivered to staging, status=%d", resp.StatusCode)
    }()
}

func isSafeToMirror(header string) bool {
    blocked := map[string]bool{
        "Stripe-Signature":        true,
        "X-Hub-Signature-256":     true,
        "X-Shopify-Hmac-Sha256":   true,
        "Authorization":           true,
    }
    return !blocked[http.CanonicalHeaderKey(header)]
}

Call mirrorAsync from your ingest handler immediately after you've persisted the event to the production queue — after validation, before returning 200. The goroutine runs in the background; your handler continues normally.

The SampleRate field is important for high-volume sources. At 50,000 events/hour, mirroring 100% creates a 50,000 event/hour load on staging. Start at 1–5% to validate the pipeline, then ramp up as confidence grows.

Handling Signatures on the Mirror

Provider HMAC signatures are computed against your production webhook secret and the raw body bytes. Your staging environment has a different signing secret (it should — never share secrets between environments). This means every mirrored event will fail signature verification in staging.

You have two options:

Option A: Disable signature verification in staging for mirrored requests.

Check the X-Mirror-Source: production header and skip HMAC verification for those requests. This is pragmatic and safe as long as staging is not publicly reachable — only your mirror infrastructure can set that header.

func (h *IngestHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    isMirror := r.Header.Get("X-Mirror-Source") == "production"

    body, _ := io.ReadAll(http.MaxBytesReader(w, r.Body, 2<<20))

    if !isMirror {
        if err := h.verifier.Verify(body, r.Header.Get("Webhook-Signature")); err != nil {
            httpx.Unauthorized(w, "invalid signature")
            return
        }
    }

    // ... rest of handler
}

Option B: Re-sign the mirrored event with the staging secret before forwarding.

At the mirror layer, strip the original signature headers and compute a new HMAC using the staging signing secret. Staging verifies normally. This is cleaner architecturally — staging behaves identically to production — but requires the mirror component to have access to the staging signing secret.

func resignForStaging(body []byte, stagingSecret string) string {
    ts := strconv.FormatInt(time.Now().Unix(), 10)
    message := ts + "." + string(body)
    mac := hmac.New(sha256.New, []byte(stagingSecret))
    mac.Write([]byte(message))
    return "t=" + ts + ",v1=" + hex.EncodeToString(mac.Sum(nil))
}

Option B is preferable if you want staging to behave identically to production and if your CI infrastructure already has access to staging configuration. Option A is fine for a quick start.

Protecting Against Side Effects

This is the most important operational concern. Your staging webhook handlers must not:

• ›Send emails or push notifications
• ›Charge payment methods
• ›Create records in external CRMs
• ›Call third-party APIs that have real effects

The standard approach is to gate all external calls behind an environment check:

func (s *EmailService) Send(ctx context.Context, to, subject, body string) error {
    if s.cfg.Env == "staging" {
        log.Printf("[staging] would send email to=%s subject=%s", to, subject)
        return nil
    }
    return s.mailer.Send(ctx, to, subject, body)
}

But relying on every developer to remember this pattern across every integration is fragile. A more robust approach is to stub all external service clients at the application boundary in staging:

GetHook's destination config supports a dry_run mode flag per destination — events are delivered to the handler logic but the HTTP POST to the destination URL is skipped and logged instead. This is useful for mirrored events where you want to exercise the routing and processing logic without actually forwarding to downstream endpoints.

What to Do With Mirror Responses

On the production path, you care deeply about delivery response codes. On the mirror path, responses are purely diagnostic.

Collect them for comparison:

CREATE TABLE mirror_delivery_log (
    id           UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    event_id     UUID NOT NULL,            -- production event ID
    source_name  TEXT NOT NULL,
    payload_hash TEXT NOT NULL,            -- SHA-256 of body, for dedup
    prod_status  INT,                      -- production handler response
    mirror_status INT,                     -- staging handler response
    mirror_latency_ms INT,
    mirror_error TEXT,
    created_at   TIMESTAMPTZ NOT NULL DEFAULT now()
);

A weekly query against this table tells you a lot:

-- Events where staging returned a different status than production
SELECT
    source_name,
    prod_status,
    mirror_status,
    COUNT(*) AS count
FROM mirror_delivery_log
WHERE created_at > now() - interval '7 days'
  AND prod_status != mirror_status
GROUP BY 1, 2, 3
ORDER BY count DESC;

Discrepancies where production returns 200 and staging returns 400 or 500 are the signal you're looking for. They indicate code in the staging branch that doesn't handle a real event shape the same way the production code does.

This comparison becomes especially valuable before a deploy: run your staging candidate against mirrored traffic for 24–48 hours, check the discrepancy rate, and deploy with high confidence only after it drops to zero.

Data Privacy and Compliance

Mirroring production webhook payloads to staging raises a real concern: production data (PII, financial details, health records) ending up in a staging environment with weaker controls.

Your approach depends on your compliance posture:

Low risk (no regulated data in payloads): Mirror freely. Label events as env=staging and ensure staging logs don't feed into production log aggregation.

Medium risk (some PII, no regulated industries): Apply a field masking step before forwarding. A simple approach:

func maskPayload(body []byte, fieldsToMask []string) ([]byte, error) {
    var m map[string]interface{}
    if err := json.Unmarshal(body, &m); err != nil {
        return nil, err
    }
    maskFields(m, fieldsToMask)
    return json.Marshal(m)
}

func maskFields(m map[string]interface{}, fields []string) {
    masked := make(map[string]bool, len(fields))
    for _, f := range fields {
        masked[f] = true
    }
    for k, v := range m {
        if masked[k] {
            m[k] = "***"
        } else if nested, ok := v.(map[string]interface{}); ok {
            maskFields(nested, fields)
        }
    }
}

High risk (HIPAA, PCI, financial): Either don't mirror raw payloads at all — generate synthetic payloads from the real event schema instead — or require staging to meet the same data handling standards as production, including encryption at rest and audit logging.

A pragmatic middle ground is to mirror only the event envelope (type, ID, timestamp, provider metadata) and replace the data field with a schema-valid synthetic payload generated from the event type. You lose some fidelity but gain full compliance safety.

Operationalizing the Mirror

A few lessons from running traffic mirrors in production:

Run the mirror out-of-band. Don't put it in the same process as your production ingest worker. A memory leak or crash in the mirror logic must not take down production. A separate, lightweight mirror service that reads from a side-channel queue is safer.

Rate-limit the mirror. At 10% of production volume, staging should be comfortable. If a traffic spike hits production, the mirror should drop events rather than overwhelm staging. A token bucket at the mirror layer is appropriate.

Alert on mirror pipeline health, not mirror delivery failures. You don't want to be paged at 2am because staging returned 500s. You do want to know if the mirror pipeline itself has stopped draining — that means you're flying blind.

Make it opt-in per source. Not all webhook sources need mirroring. High-volume, stable sources (GitHub, Stripe) are the valuable ones. One-off integrations with custom payloads add noise. Add a mirror_enabled flag to your source config and start with the sources that have caused production incidents.

Traffic mirroring won't replace unit tests or integration tests against known fixtures. It's a complementary layer that catches the class of bugs that only real-world provider behavior exposes: undocumented fields, unexpected event shapes, edge cases in event sequencing, and payload sizes you never anticipated.

If you want to route production traffic copies to a staging endpoint without modifying your ingest infrastructure, GetHook supports per-source destination routing — you can add a staging destination to your production source and let the delivery layer handle the fan-out, with independent retry policies per destination.