Integrations look finished when the first webhook returns 200.

Then the provider retries, your handler double-charges a side effect, or your endpoint is down for ten minutes and the event vanishes into a void. Webhooks are asynchronous distributed systems dressed as form posts. They need contracts for identity, retries, and poison messages.

Pneumatic capsule with indigo stripe

Delivery is not arrival. Arrival is durable processing.

Why the happy path lies

A single successful POST in a demo proves almost nothing. Production includes:

  • At-least-once delivery (duplicates)
  • Out-of-order events
  • Partial outages on your side
  • Payload version drift
  • Replay after bugfix

If your handler is not idempotent, retries become corruption.

Webhook path diagram

Receiver rules

  1. Verify signatures (HMAC or provider standard) before any side effect
  2. Idempotency keys — store processed event IDs; ignore duplicates safely
  3. Ack fast — enqueue work; do not hold HTTP open for slow CRM calls
  4. Status codes with meaning — 2xx only when accepted for processing; 4xx for payloads you will never accept; 5xx when the sender should retry
  5. Dead-letter poison messages with alerts and a human path

Sender rules (when you emit)

  • Exponential backoff with jitter
  • Bounded retries and a visible failure state for ops
  • Delivery logs with correlation IDs
  • Replay tooling for missed windows

Ordering and versions

Document whether order is guaranteed (usually it is not). Version event types (invoice.paid.v2). Consumers that assume total order will invent subtle bugs.

Observability

Track accept rate, processing lag, dead-letter depth, and signature failures. A silent webhook is worse than a loud failure.

Closing

If your runbook for “webhook missed” is empty, you do not have an integration — you have a hope with a URL.

Need webhook hardening between CMS, billing, or CRM? Start a project inquiry.

Idempotency patterns

  • Store event_id primary key; insert-ignore on conflict
  • Side effects behind “exactly once” outbox when talking to payment systems
  • Natural keys (invoice id + event type) when providers lack event ids

Security

  • Rotate secrets; support dual secrets during rotation
  • IP allowlists only as defense-in-depth (they break under cloud egress changes)
  • Timestamp tolerance to limit replay windows

Testing

Contract tests with signed sample payloads. Chaos: drop 50% of deliveries in staging and prove the system converges.

Runbook

“Webhook backlog growing” → check receiver 5xx, queue depth, provider status, signature mismatch rate. Page the owner of the consumer system, not “ whoever last touched it.“

Multi-tenant receivers

If one endpoint receives many event types, route by type early. Isolate failure domains so a bad catalog.updated handler does not block invoice.paid.