Webhook Monitoring Guide
Monitoring webhooks can be a real challenge. You have to understand what data to collect, how to collect it, and what metrics to track.
If you gather too much data, you can lose the signal in the noise. Collect bad data and your insights are unreliable.
To make sure your monitoring efforts yield actionable metrics, you should first solidify the reasons why we want to monitor our webhook system. What can go wrong and how can we identify the problem?
Status Monitoring
The goal of status monitoring is to provide instant feedback on how a system is functioning at any given moment. If a user is experiencing an issue, they need to know if their endpoint is failing or if the webhook service is down.
Because webhook systems have several components, you’ll want to provide status information for the entire system as well as for specific components to make it easier to diagnose any issues.
Availability Monitoring
Availability monitoring is similar to status monitoring with a couple key differences. First, the motivation behind status monitoring is to provide the current status of the system while the motivation for availability monitoring is to understand the reliability of the system over time.
This helps you monitor infrastructure usage to prevent problems from arising and you can focus on other important but less time-consuming tasks.
This is why you should record timestamps of any data related to the cause of any availability issues.
Performance Monitoring
Not only does your usage steadily increase over time, some users can have dramatic spikes in usage that your system will need to respond to.
As volume increases, our system needs to process more webhooks and collect more data. This can put strain on our infrastructure, increasing the chances of something going wrong.
In order to keep track of system performance and ensure that you avoid a substantial drop in performance, you need to define some key metrics which will serve as a benchmark for the webhook infrastructure’s expected throughput. This way you’ll be able to identify any slow downs early and resolve any bottlenecks/problems before your system slows to a crawl.
How to measure webhook reliability
Webhook reliability is the share of events that reach the receiver with a 2xx response inside the retry window. Measure it per endpoint with four numbers: first-attempt success rate, eventual success rate after retries, delivery latency at the 95th percentile, and the count of deliveries that exhausted retries and landed in the dead letter queue.
| Metric | What a change means |
|---|---|
| First-attempt success rate | Falling while eventual success holds: the receiver is flapping or timing out, and retries are masking it. |
| Eventual success rate | Falling: retries are not enough, so check the retry schedule and whether the endpoint is disabled. |
| p95 delivery latency | Rising: queue backlog or a slow receiver holding worker capacity. |
| Dead-lettered deliveries | Any nonzero count is an incident for that endpoint and needs a replay path. |
Other Monitoring
While these are key baseline components of a webhook monitoring system, you will probably find that there are other key items to track based on your specific implementation and use case. Just make sure you’re clearly defining the problem and how you’re going to identify it.