Webhook scalability: how to scale webhook delivery
Webhook scalability is a system's ability to keep delivering events on time as event volume, subscriber count, and payload size grow. Sending one HTTP request is easy. What breaks under load is everything around it: one event becomes many deliveries, some receivers answer slowly or not at all, and failed attempts compete with new traffic for the same workers.
Where the load actually comes from
Start with the multiplication rather than the event rate. If your application produces 500 events per second and the average event type has three endpoints subscribed to it, you are sending 1,500 requests per second. If receivers respond in 300 milliseconds on average, you need roughly 450 requests in flight at all times just to keep pace, and that is the healthy case. This expansion of one event into many independent deliveries is webhook fan-out, and it is the number that decides how much delivery capacity you need.
Two things make the arithmetic worse than it looks. Subscriber counts grow with your customer base, so delivery volume grows faster than event volume. And large customers concentrate: a handful of accounts will hold most of the endpoints, which means most of your traffic depends on the response times of a few third-party servers you have no control over.
The bottleneck is other people's servers
Delivery throughput is a function of receiver latency, not your own CPU. A worker that posts to an endpoint taking 20 seconds to time out is doing nothing for 20 seconds, and a few hundred deliveries to that endpoint can consume a pool sized for thousands of healthy requests per second. This is why a single customer's bad deploy can look like an outage in your own system.
Three controls keep it contained. Set an aggressive delivery timeout, because a receiver that has not acknowledged in a few seconds is not going to. Cap concurrency per endpoint so no single destination can hold more than its share of workers. And pause endpoints that have been failing long enough to be considered dead with a circuit breaker, which turns a slow failure into a fast one and frees the capacity it was consuming.
Retry storms compound the problem
Retries are the part that turns a small incident into a large one. When a popular receiver goes down for an hour, every delivery in that hour fails and enters a retry schedule. The receiver comes back, and your system now has the backlog plus the normal traffic plus every retry that happens to be due at the same moment. Fixed retry intervals make that synchronization exact, so use exponential backoff with jitter to spread attempts out.
Bound the total work retries can create. Give each delivery a fixed number of attempts and move it to a dead-letter queue rather than retrying indefinitely, and run retries at a lower priority than first attempts so a backlog never delays events that are happening now. A rate limit per endpoint protects receivers from getting the entire backlog at once when they recover.
Keep the send path out of the request path
Never fan out inside the API request that produced the event. Write the event once, return, and let a separate process expand it into per-endpoint deliveries in a background queue. Otherwise your own API latency becomes a function of your slowest subscriber, and a traffic spike in webhooks becomes a traffic spike in the product.
Horizontal scaling then comes from the queue rather than from bigger machines. Partition delivery work by endpoint or by tenant so that consumers can be added without reordering anything within a partition, and so one noisy tenant occupies one partition instead of the whole pool. Fairness across tenants is a related problem with its own answers, covered in building a multi-tenant webhook service.
Receiving webhooks at scale is a different problem
If you are on the receiving end, the spike is not yours to control: providers send when their events happen, often in bursts after their own incidents. Acknowledge with a 2xx as soon as you have durably stored the request, then process it asynchronously. Handle events idempotently, because at-least-once delivery means you will see the same event twice, and a duplicate that charges a customer twice is worse than a delivery you dropped.
What grows besides throughput
Delivery logs grow faster than traffic does, since every attempt is a record and every record may store a payload. Decide retention early, because customers expect to inspect a failure from last week and storing full payloads forever is the line item that surprises teams. Watch queue depth, delivery latency at the 99th percentile, and success rate per endpoint rather than a single global success number, which hides the one large customer that is failing. Our webhook monitoring guide covers what to record and alert on.
Written out, scaling webhook delivery is a queue, per-endpoint isolation, jittered retries with a dead-letter path, partitioned workers, and a delivery log with a retention policy. That is the webhook infrastructure most teams end up building twice. Svix provides it as a service, so an API call fans out to every subscribed endpoint with retries, signatures, and logs already handled. If you would rather build, start with building a webhook sender, and read the downsides of webhooks before you commit.
Frequently asked questions
How many webhooks per second can one sender handle?
It depends on receiver latency, not on your hardware. With 300ms average responses, one worker sustains about three deliveries per second, so throughput is roughly your concurrency limit divided by average response time. Size the pool from that number and measure it with real endpoints rather than a local test server.
Should retries share a queue with new deliveries?
No. A backlog of retries from one failing receiver will delay events that just happened. Give retries a separate lower-priority queue or a scheduled queue keyed on when each attempt is due, so first attempts always go out on time.
Is adding more workers enough to scale webhook delivery?
Only until the slow endpoints absorb them. Without per-endpoint concurrency caps, timeouts, and circuit breaking, a larger pool just means more workers blocked on the same unresponsive receiver. Add the isolation controls first, then scale the pool.