Reading a provider's webhook documentation is not the same as actually integrating it. The docs tell you the happy path: an event fires, a payload arrives, you return 200. What they don't tell you is how each provider handles signature verification, what they consider a successful response, how they retry, and which undocumented behaviors will surprise you at 3am.
This post covers four commonly used webhook providers — Twilio, SendGrid, PagerDuty, and HubSpot — and the specific quirks that matter when you're building a reliable integration.
Twilio
Signature verification is URL-sensitive
Twilio signs webhooks using HMAC-SHA1 over a concatenation of the full request URL and the sorted POST parameters. This is different from most providers that sign the raw request body.
The signed string is:
https://your-domain.com/hooks/twilio?AccountSid=AC...
AccountSid=AC...Body=Hello+worldFrom=+15551234567MessageSid=SM...NumMedia=0NumSegments=1To=+19875550123Note the structure: the URL comes first, then the POST parameters in alphabetical order, concatenated as key + value with no separator between pairs.
If your webhook sits behind a load balancer or reverse proxy that terminates TLS, Twilio will sign against https://... but your app may see http://.... This mismatch causes every signature check to fail silently. The fix:
func twilioSignedURL(r *http.Request) string {
scheme := r.Header.Get("X-Forwarded-Proto")
if scheme == "" {
scheme = "https" // default to https in production
}
host := r.Header.Get("X-Forwarded-Host")
if host == "" {
host = r.Host
}
return fmt.Sprintf("%s://%s%s", scheme, host, r.URL.RequestURI())
}Always reconstruct the URL from X-Forwarded-Proto and X-Forwarded-Host when Twilio webhooks go through a proxy.
Form-encoded, not JSON
Twilio sends webhook bodies as application/x-www-form-urlencoded, not JSON. This trips up generic webhook routers that assume Content-Type: application/json. Parse with r.ParseForm() in Go or the equivalent in your stack, not json.Unmarshal.
Retry behavior
Twilio retries on any non-2xx response, with 3 attempts over roughly 15 minutes. There is no exponential backoff — intervals are roughly fixed. If your handler is slow, Twilio will time out (default 15 seconds per attempt). Design for fast acknowledgment: persist the event and return 200 immediately, then process asynchronously.
SendGrid
Event webhook batches, not individual events
SendGrid delivers events in batches — a single POST body contains a JSON array of multiple events, not a single event object. If your handler expects {"event": "delivered", "email": "..."}, it will fail. The actual payload is:
[
{
"email": "user@example.com",
"timestamp": 1711112400,
"event": "delivered",
"sg_message_id": "abc123.filterdrecv-...",
"sg_event_id": "abc456"
},
{
"email": "user@example.com",
"timestamp": 1711112405,
"event": "open",
"sg_message_id": "abc123.filterdrecv-...",
"sg_event_id": "def789"
}
]Batch sizes are variable — expect anywhere from 1 to several hundred events per request. Always iterate over the array, not index into [0].
Signature verification uses Ed25519, not HMAC
SendGrid's Event Webhook now supports Ed25519 signature verification (the older HMAC-based verification is deprecated). The public key is available in the SendGrid dashboard under Settings > Mail Settings > Event Webhook.
Verification uses the X-Twilio-Email-Event-Webhook-Signature and X-Twilio-Email-Event-Webhook-Timestamp headers (yes, they kept the X-Twilio-Email- prefix after the Twilio acquisition).
import "crypto/ed25519"
func verifySendGridSignature(body []byte, signature, timestamp, publicKeyBase64 string) error {
pubKeyBytes, err := base64.StdEncoding.DecodeString(publicKeyBase64)
if err != nil {
return fmt.Errorf("invalid public key: %w", err)
}
pubKey := ed25519.PublicKey(pubKeyBytes)
// Signed payload is timestamp + body (no separator)
signed := append([]byte(timestamp), body...)
sigBytes, err := base64.StdEncoding.DecodeString(signature)
if err != nil {
return fmt.Errorf("invalid signature encoding: %w", err)
}
if !ed25519.Verify(pubKey, signed, sigBytes) {
return fmt.Errorf("signature verification failed")
}
return nil
}Rotate the signing key in the SendGrid dashboard, not via API. There's no programmatic key rotation endpoint — a limitation worth knowing before you design your secret management process.
Deduplication via sg_event_id
SendGrid can deliver the same event more than once. The sg_event_id field is stable across retries and is your deduplication key. Store processed sg_event_id values and discard duplicates before processing.
PagerDuty
Webhooks as "Extensions" or "V3 Subscriptions"
PagerDuty has two webhook systems: legacy Extensions (V1/V2) and the newer V3 Subscriptions. They have different payload structures, different signature schemes, and different event type naming conventions. If you're integrating an older PagerDuty account, you may be dealing with V2; new accounts default to V3. Check which your account is configured for before assuming either payload format.
V3 subscription payloads look like:
{
"event": {
"id": "01BXXX",
"event_type": "incident.triggered",
"resource_type": "incident",
"occurred_at": "2026-03-28T10:00:00.000Z",
"agent": { "type": "user_reference", "id": "PXXXXXX" },
"client": { "name": "PagerDuty" },
"data": {
"id": "QXXXXXX",
"type": "incident",
"title": "Database connection pool exhausted",
"status": "triggered",
"urgency": "high",
"service": { "id": "PXXXXXX", "type": "service_reference", "summary": "Payments API" }
}
}
}HMAC-SHA256 with a secret per subscription
V3 webhooks are signed with HMAC-SHA256. The secret is set per subscription (not account-wide) and is available in the webhook subscription configuration. The signature header is X-PagerDuty-Signature and the format is v1=<hex>.
PagerDuty signs the raw request body. Keep the raw bytes before any JSON parsing.
The "acknowledge within 5 seconds" rule
PagerDuty expects a 2xx response within 5 seconds. If your handler takes longer, PagerDuty marks the delivery as failed and retries. Their retry schedule: 5 minutes, 30 minutes, 2 hours, 12 hours, 24 hours. Total delivery window is about 38 hours.
Unlike Twilio, PagerDuty uses exponential backoff, which means a stuck destination won't be spammed — but a slow one will have a long tail of retries showing up hours later.
Incident state machine events
| Event Type | Trigger |
|---|---|
incident.triggered | New incident created |
incident.acknowledged | On-call engineer acknowledged |
incident.resolved | Incident resolved |
incident.escalated | Escalated to next policy level |
incident.reassigned | Assigned to different responder |
incident.annotated | Note added to incident |
incident.priority_updated | Priority changed |
incident.responder.added | Responder added |
If you're building an integration that tracks incident lifecycle (e.g., syncing to a ticketing system), you need all eight event types. Subscribing to incident.triggered and incident.resolved only gets you two states of a multi-step process.
HubSpot
App-level vs. portal-level webhooks
HubSpot has two distinct webhook systems with completely different purposes:
App webhooks — configured in the HubSpot developer portal for a specific app. These fire when CRM objects (contacts, deals, companies) are created, updated, or deleted. Intended for ISVs building HubSpot integrations.
Workflow webhooks — configured inside HubSpot's workflow builder by HubSpot users. These fire as an action step in a workflow, using the HTTP request action. The payload is user-configurable, not standardized.
If a customer says "our HubSpot webhook isn't working," clarify which type before debugging.
App webhook batching and the 1000-event limit
HubSpot delivers app webhooks in batches, with up to 1,000 events per request. The payload is a JSON array similar to SendGrid's batching pattern. HubSpot fires batches every few seconds during high activity, so a busy portal can deliver thousands of events in a short window.
[
{
"eventId": 1234567890,
"subscriptionId": 12345,
"portalId": 987654,
"appId": 11111,
"occurredAt": 1711112400000,
"subscriptionType": "contact.creation",
"attemptNumber": 0,
"objectId": 56789,
"changeSource": "CRM_UI",
"changeFlag": "NEW"
}
]Note: occurredAt is milliseconds since epoch, not seconds. Divide by 1000 before converting to a timestamp.
Signature verification changed in V3
HubSpot V1 and V2 webhooks used HMAC-SHA256 over client_secret + request_body. V3 changed this to HMAC-SHA256 over client_secret + http_method + request_uri + request_body + timestamp. The X-HubSpot-Signature-Version header tells you which version is in use.
func verifyHubSpotV3(clientSecret, method, uri string, body []byte, timestamp, signature string) error {
// Reject if timestamp is older than 5 minutes
ts, err := strconv.ParseInt(timestamp, 10, 64)
if err != nil {
return fmt.Errorf("invalid timestamp")
}
if time.Since(time.UnixMilli(ts)).Abs() > 5*time.Minute {
return fmt.Errorf("timestamp out of tolerance window")
}
source := clientSecret + method + uri + string(body) + timestamp
mac := hmac.New(sha256.New, []byte(clientSecret))
mac.Write([]byte(source))
expected := base64.StdEncoding.EncodeToString(mac.Sum(nil))
if !hmac.Equal([]byte(expected), []byte(signature)) {
return fmt.Errorf("signature mismatch")
}
return nil
}The timestamp tolerance check matters: HubSpot sends the X-HubSpot-Request-Timestamp header in milliseconds, so use time.UnixMilli, not time.Unix.
Deduplication key: eventId
HubSpot can deliver the same event more than once on retry. The eventId field is stable across retries and is your deduplication key. Unlike SendGrid's sg_event_id, HubSpot's eventId is a plain integer, not a string.
Cross-Provider Comparison
| Provider | Body Format | Signature Scheme | Retries | Batch Delivery | Dedup Key |
|---|---|---|---|---|---|
| Twilio | Form-encoded | HMAC-SHA1 (URL + params) | 3 attempts, ~15 min | No | MessageSid / CallSid |
| SendGrid | JSON array | Ed25519 | Not specified | Yes (up to ~1000) | sg_event_id |
| PagerDuty | JSON object | HMAC-SHA256 | 5 retries, up to 38h | No | event.id |
| HubSpot | JSON array | HMAC-SHA256 (V3) | Not specified | Yes (up to 1000) | eventId |
Patterns That Apply Everywhere
Across all four providers, the same engineering patterns make your integration more reliable:
Return 200 fast. All four providers timeout at or under 15 seconds. Acknowledge immediately, process asynchronously. A gateway like GetHook handles this by design — the ingest layer returns immediately while the delivery pipeline processes events durably in the background.
Verify signatures before processing. Each provider has a different scheme, but all four provide one. Skipping verification means any HTTP client can trigger your webhook handler. Store the raw body bytes before any parsing so you can pass them to the verification function.
Idempotency is not optional. SendGrid, HubSpot, and PagerDuty all explicitly document that they may deliver the same event more than once. Twilio can too. Store your deduplication keys in a database with a unique constraint and check before processing.
Log the raw payload. When a provider's behavior changes or you encounter an unexpected payload structure, you want the original bytes — not a partially-parsed object. Log the raw body (subject to your data retention policy) before any transformation.
If you're building an integration layer that receives events from multiple providers and fans them out to internal services, a webhook gateway centralizes the signature verification, deduplication, and retry handling so your application code doesn't have to implement it per-provider.