Webhook reliability: a practical guide for senders and receivers
Webhook reliability is the share of events that reach their endpoint and get processed once, measured across retries rather than per request. A single webhook is one HTTP POST that fails whenever the receiver is deploying, slow, or unreachable, so reliability comes from the system around it: retries with backoff, a dead letter queue, signatures, and idempotent receivers.
That work is split between the two ends. The sender decides whether a failed delivery is retried, parked for replay, or dropped. The receiver decides whether the retry that follows is harmless or charges a customer twice.
Measure it per endpoint, after retries
The number that matters is eventual delivery: the share of events that got a 2xx at some point inside the retry window. Track first-attempt success next to it, because the gap between the two is how much work your retries are doing. A 90% first-attempt rate with 99.99% eventual delivery means retries are covering for a receiver that times out one request in ten, and that receiver is seeing a steady stream of duplicates.
Measure per endpoint, never as one global figure. A few large customers usually hold most of the endpoints, so a healthy average can hide the one that fails every request. Our webhook monitoring guide lists the four metrics to track and what a change in each one means.
Why webhook deliveries fail
Most failed deliveries are not exotic. The receiver is mid-deploy and refuses connections for thirty seconds. A load balancer returns a 502 while an instance rotates. A TLS handshake fails once. None of these mean the event should be lost, and all of them are fixed by trying again a moment later.
The next most common cause is self-inflicted: the receiver does real work before answering, and providers do not wait long. GitHub records a delivery as failed after 10 seconds and Shopify after 5. A handler that calls two external APIs before returning 200 will trip a webhook timeout under load, and the sender records a failure even though the work completed.
Then there are failures no retry can fix. A receiver returning 400 because of a schema mismatch will return 400 on every attempt. A signature check that fails because the webhook secret was rotated on one side only keeps failing until both sides agree. A decommissioned endpoint will never come back. A sender that cannot tell these apart from transient errors spends its whole schedule on them, and what happens when a webhook fails shows how each case looks in a delivery log.
What a reliable sender does
Seven mechanisms carry most of the weight. The reliability section of Webhooks University maps each failure above to the mechanism that handles it.
Record the event before sending it
Write the outgoing message to durable storage in the same database transaction as the change that caused it, then let a separate worker deliver it. This transactional outbox closes the gap where an application commits, crashes, and never sends the event, a failure no delivery log will ever show. It also keeps delivery out of your API’s request path, so a slow subscriber never slows down your product.
Retry with exponential backoff and jitter
Delays should grow rather than stay fixed, so a struggling receiver gets room to recover, and random jitter keeps deliveries that failed together from retrying in lockstep. Exponential backoff with a cap is the standard shape, and webhook retry strategies covers the formulas and which status codes deserve a retry.
Publish the exact schedule, including the rule that only a 2xx counts as success and redirects do not, because receivers cannot plan around a schedule they cannot see. Svix, for example, retries immediately, then after 5 seconds, 5 minutes, 30 minutes, 2 hours, 5 hours, 10 hours, and 10 hours: eight attempts across a little over 27 hours.
Time out in seconds
A delivery worker waiting on a hung endpoint is doing nothing else. Cut each attempt off after a fixed limit, count the timeout as a failure, and let the retry schedule take over. Most providers allow between 5 and 30 seconds; webhook timeout best practices covers how receivers stay under it.
Keep exhausted events and make them replayable
When the schedule runs out, the event should land in a dead letter queue with its payload, destination, and attempt history rather than disappear. Replay lets a customer recover from a day-long outage on their own: resend one message, or everything that failed since a given time, once their endpoint is fixed. A replay must reuse the original message ID so the receiver’s deduplication still recognizes it. Dead letter queues for webhooks covers what to store and how to replay safely.
Disable endpoints that stay broken
An endpoint that has failed every attempt for days is not coming back on the next one. Disable it automatically, tell its owner, and stop spending workers on it. Cap concurrency per endpoint as well, so one slow customer cannot occupy the pool that serves everyone else. Circuit breakers are the general form of this, and webhook scalability covers the capacity math.
Sign every attempt
A receiver cannot safely act on a request it cannot verify, so signing is part of reliability as much as security. Generate the signature fresh on each attempt: it covers a timestamp, and receivers reject anything outside a timestamp tolerance of about five minutes, so a retry sent hours later with the original signature would be refused. The Standard Webhooks specification defines the headers for this, so receivers can verify with an off-the-shelf library.
Give receivers an ID and an order hint
Retries mean duplicates, so every message needs an ID that stays the same across attempts and replays. That ID is the receiver’s idempotency key. Do not promise ordering unless you enforce it, because a retried event can land after events created later. Put a sequence number or the resource’s updated-at time in the payload so receivers can discard stale updates. Webhook delivery guarantees explains what at-least-once delivery does and does not promise.
What a reliable receiver does
At-least-once delivery pushes work onto the receiver, and three habits handle nearly all of it.
Answer immediately and process later. Verify the signature, write the event to a queue or a table, return 200, and do the real work in a background queue. The handler drops from seconds to milliseconds, and timeouts stop being a failure mode.
Deduplicate on the event ID. The sender cannot tell a lost response from a lost request, so the same event will eventually arrive twice. Store processed IDs behind a unique constraint and skip repeats, which is the whole subject of idempotency and deduplication.
Treat each event as a notification, not the full truth. When order or completeness matters, fetch the resource’s current state from the provider’s API instead of applying the payload blindly, and reconcile periodically for events you cannot afford to miss. Polling is the safety net for whatever a long outage dropped. Best practices for receiving webhooks has the rest of the checklist.
How to evaluate a provider’s webhook reliability
A provider that promises “reliable delivery” without numbers is promising nothing you can check. Ask for these:
- The delivery guarantee, and whether every message carries an ID that stays stable across retries
- The exact retry schedule and total retry window
- The request timeout, and which status codes count as success
- How long failed events stay available for replay, per message and in bulk
- Whether endpoints that keep failing are disabled, and how the owner finds out
- A delivery log you, and ideally your customers, can search per endpoint
A provider that publishes all of these has thought about failure. One that says “we retry a few times” has not. Stripe, for instance, documents retries for up to three days in live mode. Our guide to evaluating webhook infrastructure covers what a good answer to each question sounds like.
When webhooks are the wrong reliability model
Webhooks fit server-to-server notification across a trust boundary. They fit poorly when you need strict ordering, replay from an arbitrary offset, or millions of internal events per second. That is a log or a broker’s job, and webhooks vs message queues walks through where the line sits.
Getting reliable delivery without building it
Each mechanism above is simple to describe. Running all seven together is the expensive part: persisting scheduled retries so a crash does not lose them, isolating slow endpoints, storing attempt history, and answering tickets that ask whether an event was ever sent. That is the webhook infrastructure most teams end up rebuilding.
Svix provides it as a service. Messages follow the eight-attempt schedule above with a 30-second timeout per attempt. An endpoint that has failed every attempt for five days is disabled, with an operational webhook to tell you, and failed messages can be resent one at a time or recovered in bulk from a chosen date, through the API or the App Portal your customers use. The exact rules are in the Svix retry docs.
Frequently asked questions
Can webhooks be lost?
Yes, if the sender does not retry. With retries over a long window and a dead letter queue for exhausted messages, loss requires the receiver to be unreachable for the entire retry period, commonly a day or more. Without retries, one failed request means the event is gone.
Are webhooks guaranteed to be delivered in order?
No. Most providers make no ordering guarantee, because a retried event can arrive after events created later. If order matters, include a sequence number or timestamp in the payload and reorder or discard stale updates on the receiving side.
How many times should a failed webhook be retried?
A common default is around eight attempts spread across roughly a day with exponential backoff and jitter, after which the message goes to a dead letter queue. The exact count matters less than publishing the schedule so receivers know how long they have to recover.
Why did my webhook fail even though my server processed it?
Almost always a timeout. The handler did its work but responded after the provider gave up, so the sender recorded a failure and retried. Return 2xx first and process asynchronously, and deduplicate on the event ID so the retry is harmless.