How webhooks work under the hood
When you click "save" on a webhook configuration and events start flowing to your endpoint, a lot happens behind the scenes. Understanding this process helps you build more reliable integrations, debug delivery failures, and design better webhook systems of your own.
This article traces the complete lifecycle of a webhook from the moment an event occurs to when your server processes it, covering the HTTP mechanics, network considerations, and failure modes along the way.
The event triggers
Everything starts with an event in the source system. A customer completes a purchase, a user updates their profile, a file finishes uploading. The application detects this change and decides it needs to notify external systems.
At this point, the webhook sender constructs a payload representing the event. This typically includes an event type identifier, a timestamp, a unique event ID, and the relevant data. The sender serializes this payload as JSON and prepares to dispatch it to all registered endpoints that have subscribed to this event type.
Most webhook systems do not send the webhook synchronously as part of the original transaction. Instead, they write the event to a queue and return immediately. A separate worker process picks up queued events and handles the actual HTTP delivery. This decoupling is important because webhook delivery can be slow or fail entirely, and you do not want that blocking your main application.
Building the HTTP request
The webhook sender constructs an HTTP POST request to your registered endpoint URL. The request includes several important components beyond just the payload.
The Content-Type header is typically application/json, telling your server how to parse the body. The sender adds custom headers for metadata and security. Providers following the Standard Webhooks spec include a unique message ID (webhook-id), a timestamp (webhook-timestamp), and crucially, a signature (webhook-signature) that lets you verify the request is authentic.
The signature is computed by taking the timestamp, the raw request body, and a shared secret, then running them through an HMAC-SHA256 function. This proves the request came from someone who knows the secret and that the body has not been tampered with in transit. The timestamp prevents replay attacks where an attacker captures a valid webhook and resends it later.
The network journey
Once constructed, the request travels across the network to your server. This involves DNS resolution to find your server's IP address, establishing a TCP connection, and usually negotiating TLS encryption. Each step can fail or introduce latency.
DNS failures are rare but happen, especially if you recently changed your endpoint URL. TCP connection failures occur when your server is down, overloaded, or unreachable due to network issues. TLS handshake failures happen with certificate problems, expired certs, or misconfigured servers.
The webhook sender typically sets a timeout for the entire operation, often between 5 and 30 seconds. If your server does not respond within this window, the sender treats it as a failure and schedules a retry.
Your server receives the request
Your web server receives the incoming POST request. The first thing your webhook handler should do is read the raw request body and verify the signature before doing anything else. This is your authentication step, confirming the request actually came from the expected sender.
If the signature is invalid or missing, return a 401 Unauthorized response immediately. Do not process the payload. Someone might be probing your endpoint or attempting to inject fake events.
After signature verification, parse the JSON body and check the event ID against your store of processed events. Webhook senders use at-least-once delivery semantics, meaning you might receive the same event multiple times due to retries. Idempotency checking using the event ID prevents double-processing.
Processing and responding
Here is where most developers make a mistake: they try to do all their processing before responding. The webhook sender is waiting for your response, and if you take too long, it will time out and retry. Now you have duplicate delivery plus wasted work.
The correct pattern is to acknowledge receipt immediately, then process asynchronously. Your handler should validate the request, store the event in your own queue or database, return a 200 OK response, and then have a separate worker process handle the actual business logic.
Your response status code tells the sender what happened. A 2xx status means you received the webhook successfully. A 4xx status indicates a client error, like invalid signature (401) or bad request format (400). Most senders will not retry 4xx errors since they indicate a problem on your end that retrying will not fix. A 5xx status indicates a server error, and the sender will typically retry.
The response body usually does not matter, but returning a simple JSON acknowledgment like {"received": true} is good practice for debugging.
When delivery fails
If your endpoint returns an error or times out, the webhook sender schedules a retry. Most systems use exponential backoff, waiting longer between each attempt. A typical schedule might retry after 5 minutes, then 30 minutes, then 2 hours, then 8 hours, spreading out over 24 to 72 hours.
After exhausting all retry attempts, the sender marks the delivery as failed. Some systems notify you via email or dashboard alert. Others provide a way to manually replay failed webhooks once you have fixed the underlying issue.
During an outage, failed webhooks can pile up. When your endpoint comes back online, you might receive a burst of retried webhooks all at once. Your endpoint needs to handle this load spike gracefully, which is another reason to acknowledge quickly and process asynchronously.
The complete picture
From event to processing, a webhook typically flows through these stages: event occurs in source system, event is queued for delivery, worker constructs HTTP request with signature, request travels over network to your endpoint, your server verifies signature and checks for duplicates, you acknowledge receipt with 200 OK, your worker processes the event asynchronously. If any step fails, the retry mechanism kicks in and the process repeats.
Understanding this lifecycle helps you debug issues. Signature failures point to secret mismatches or body parsing problems. Timeouts suggest your handler is doing too much work synchronously. Duplicate events indicate idempotency problems in your processing logic. Knowing where in the chain things break is half the battle in building reliable webhook integrations.