Webhook retry strategies
Webhooks fail. Networks are unreliable, servers go down, deployments cause brief outages. A webhook system without retries would lose events constantly. But retries done wrong can overwhelm recovering servers, waste resources on permanently broken endpoints, and create confusing duplicate deliveries.
This article covers the strategies for retrying failed webhooks effectively: exponential backoff to avoid hammering struggling servers, jitter to prevent thundering herds, and knowing when to stop retrying entirely.
Why naive retries cause problems
The simplest retry approach is to immediately try again when a delivery fails. This is almost always wrong. If a server returned an error because it is overloaded, immediately retrying adds more load. If a network blip caused a timeout, the request might still be in flight, and retrying creates duplicates. If a deployment is rolling out, the server might be down for 30 seconds, and rapid retries just pile up failed attempts.
Immediate retries also create feedback loops. Imagine a server that starts failing under load. Immediate retries double the incoming traffic. More requests fail, causing more retries, causing more failures. The server never recovers because the retry storm keeps it overwhelmed.
Exponential backoff spreads out retries
Exponential backoff solves these problems by waiting progressively longer between retry attempts. The first retry might happen after 30 seconds, the second after 2 minutes, the third after 8 minutes, and so on. Each retry waits a fixed multiple of the previous delay. The schedule below multiplies by 4, and a multiplier of 2 is also common.
This gives transient failures time to resolve. A brief network hiccup clears up in seconds. A deployment completes in a minute or two. A rate limit window resets. By the time your retry arrives, the problem has likely fixed itself.
A typical exponential backoff schedule looks like this:
Attempt 1: immediate (initial delivery)
Attempt 2: 30 seconds after failure
Attempt 3: 2 minutes after attempt 2
Attempt 4: 8 minutes after attempt 3
Attempt 5: 32 minutes after attempt 4
Attempt 6: 2 hours after attempt 5
Attempt 7: 8 hours after attempt 6
The formula is usually base_delay * (multiplier ^ attempt_number), sometimes with a cap to prevent absurdly long waits. You might cap the maximum delay at 8 hours even if the exponential formula would suggest 24 hours.
Adding jitter prevents thundering herds
Pure exponential backoff has a subtle problem. If a server goes down and 10,000 webhooks fail at the same moment, they all schedule their first retry for exactly 30 seconds later. When that retry happens, 10,000 requests hit the server simultaneously. Even if the server has recovered, this coordinated spike might knock it down again.
Jitter solves this by adding randomness to the retry delay. Instead of waiting exactly 30 seconds, you wait somewhere between 15 and 45 seconds. The 10,000 retries spread out over a 30-second window instead of arriving simultaneously.
There are two common approaches to jitter. Full jitter picks a random delay between zero and the calculated backoff time. Decorrelated jitter uses the previous delay to calculate a range for the next one, creating more variation across attempts.
A simple implementation multiplies the backoff by a random factor, a milder variant than full jitter that keeps delays close to the planned schedule:
import random
def calculate_delay(attempt, base_delay=30, multiplier=4, max_delay=28800):
exponential = base_delay * (multiplier ** attempt)
with_jitter = exponential * (0.5 + random.random()) # 0.5x to 1.5x
return min(with_jitter, max_delay)
The combination of exponential backoff and jitter gives you the best of both worlds: spreading out individual endpoint retries over time while also spreading out retries across different endpoints.
Knowing when to stop
Retrying forever wastes resources and can mask configuration problems. If an endpoint has been returning 404 for three days, something is fundamentally wrong, and continued retries accomplish nothing.
Most webhook systems set a maximum retry window, typically 24 to 72 hours. After exhausting all attempts within this window, the delivery is marked as permanently failed. Some systems send a notification to the webhook owner so they can investigate and potentially replay the failed events manually.
The maximum attempts and time window should balance reliability against practicality. Too few retries and you lose events to brief outages. Too many and you waste resources on dead endpoints while the failed event queue grows unbounded.
A reasonable default might be 6 to 8 attempts over 24 to 48 hours. This handles most transient failures while giving up on endpoints that are clearly broken.
Distinguishing retryable from non-retryable failures
Not all failures deserve retries. A 500 Internal Server Error suggests a temporary server problem and is worth retrying. A 404 Not Found usually means the endpoint does not exist. A 401 Unauthorized indicates a credential problem that retrying will not fix on its own.
Smart retry logic categorizes responses:
Retry these: 408 Request Timeout, 429 Too Many Requests, 500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable, 504 Gateway Timeout, and network errors like connection refused or DNS failure.
Do not retry these: 400 Bad Request, 401 Unauthorized, 403 Forbidden, 404 Not Found, 405 Method Not Allowed. These indicate problems with the request itself or the endpoint configuration, not transient failures.
There is a tradeoff here. A 404 or 401 during a botched deploy is often temporary, so many senders, Svix included, retry every non-2xx response and rely on disabling persistently failing endpoints to stop wasting attempts. A 410 Gone is the clearest signal to stop immediately.
For 429 Too Many Requests, check if the response includes a Retry-After header. If it does, respect that delay instead of your normal backoff schedule.
How to handle webhook retries, failures, and dead letter queues
Handle webhook failures in three layers. Retry transient failures (timeouts, 429, 5xx) with exponential backoff and jitter for roughly a day. When retries are exhausted, move the message to a dead letter queue instead of dropping it. Then alert the endpoint owner and let them replay failed messages once the endpoint is fixed.
The DLQ record should keep the original payload, the event ID, the target endpoint, the attempt count, and the last status code and response body. Replays must resend the identical message with the same ID, so the receiver’s deduplication still works. Track health per endpoint as well: if an endpoint fails every message for days, disable it with a circuit breaker rather than filling the DLQ with thousands of entries. The dead letter queues article and the DLQ redrive guide cover replay in more depth.
What receivers should know
If you are receiving webhooks, understand that retries are coming. Build your endpoint to be idempotent using the event ID, so processing the same webhook twice produces the same result. Respond quickly with a 2xx (200 or 202) before doing heavy processing, so you do not trigger unnecessary retries due to timeouts.
If you need the sender to back off because you are overwhelmed, return 429 with a Retry-After header. This is more graceful than returning 500 errors or timing out, and good webhook senders will respect it.
Should you build webhook infrastructure yourself or use a service?
Build it yourself if you send a few event types to internal consumers and already run a durable queue. Use a service once you deliver to customer endpoints, because retries, per-endpoint isolation, signing, secret rotation, a DLQ with replay, SSRF protection, and a customer-facing delivery log together become an ongoing product.
The retry loop itself is a small job. The cost is everything around it: persisting scheduled retries so a crash does not lose them, keeping one slow endpoint from delaying everyone else, and answering support tickets that ask whether an event was sent. Svix handles this layer: failed messages are retried immediately, then after 5 seconds, 5 minutes, 30 minutes, 2 hours, 5 hours, 10 hours, and 10 hours, and endpoints that keep failing are disabled automatically, with failed messages recoverable from the dashboard or API. The webhook infrastructure glossary entry and the guide to evaluating webhook infrastructure walk through the build-versus-buy criteria.
Choosing your retry strategy
For most webhook systems, a sensible default is exponential backoff starting at 30 seconds, growing by a factor of about 4 up to a maximum delay of 8 hours, with jitter applied, over 7 to 8 attempts spanning roughly a day. Distinguish retryable from non-retryable status codes to avoid wasting attempts on permanent failures, and send exhausted messages to a dead letter queue so nothing is silently lost.
This strategy handles the common cases well: brief outages, rate limiting, network hiccups, and deployment windows. It gives struggling servers room to recover while eventually giving up on endpoints that are genuinely broken.
If you are building a webhook sender for customers, consider letting them customize the retry policy. Some customers have endpoints that recover quickly and want aggressive retries. Others have systems that need more breathing room. Allowing configuration of the retry count, backoff multiplier, or maximum retry window accommodates different operational needs without requiring you to pick a one-size-fits-all default. If you would rather not maintain that machinery, Svix ships retries, endpoint disabling, and message recovery out of the box.