What is webhook deduplication?
Webhook deduplication is detecting that an incoming delivery repeats one already processed, and dropping it before it causes a second side effect. It is the concrete mechanism behind idempotent webhook handling: because delivery is at-least-once, the same event will occasionally arrive twice, and deduplication is how the second arrival becomes harmless.
Deduplicate on the message ID
The reliable key is the unique ID the provider attaches to each message. Providers following the Standard Webhooks specification send it as the webhook-id header, stable across every retry of the same message, and the spec recommends receivers key their deduplication on it.
The handler flow: verify the signature, then attempt to record the ID in a store of processed messages. If the record already exists, return 2xx immediately without processing, because acknowledging a duplicate is exactly what the sender needs to stop retrying it. If it doesn't, process the event.
Content-based keys, like hashing the payload, are a fallback when a provider sends no ID, but they misfire in both directions: two legitimately distinct events with identical payloads collapse into one, and the same event re-sent with a trivially different field, like a fresh timestamp, slips through. Use the provider's ID whenever one exists.
Making the check atomic
The subtle bug in deduplication is the gap between checking and recording. A handler that checks the store, processes, and then records the ID can crash after processing but before recording; the retry then finds no record and processes again. Two patterns close the gap.
Unique-constraint insert. Insert the message ID into a table with a unique constraint as part of the same database transaction as the side effect. A duplicate delivery's insert fails, the transaction rolls back, and nothing runs twice. This is the strongest pattern when the side effect lives in the same database.
Record-then-process. Insert the ID first, then process. A crash between the two leaves a recorded ID with no completed work, so the retry is dropped and the event is lost from the handler's perspective. This pattern therefore needs a reconciliation path, and it trades a rare stuck event for a guarantee of no duplicates. Which failure is worse depends on the event: for a payment, processing twice is worse; for a notification, dropping is.
Scoping the store
The deduplication store does not need to remember forever. Duplicates come from retry schedules measured in hours or days, and anything older than the timestamp tolerance fails verification before deduplication runs. A TTL comfortably longer than the provider's retry schedule, a few days to a week, bounds the store's size while catching every realistic duplicate. Redis with per-key expiry and a database table with a periodic delete are both common homes for it.
One scope caveat: deduplication drops repeats of the same message. A provider that emits two distinct messages for what your business logic considers one occurrence, or a replay initiated from the provider's dashboard with a fresh ID, passes deduplication by design. Handlers that must guard against those cases key on a business identifier from the payload, such as an order ID plus state, in addition to the message ID.
The full receiving-side treatment, including code, is in the idempotency and deduplication lesson.
Frequently asked questions
What should I use as the deduplication key for webhooks?
The provider's unique message ID, sent as the webhook-id header under the Standard Webhooks specification and stable across retries. Payload hashes are a last resort: they conflate distinct events with identical bodies and miss re-sends with trivially changed fields.
How long should deduplication records be kept?
Slightly longer than the provider's retry schedule, typically a few days to a week. Older duplicates cannot occur through retries, and anything beyond the timestamp tolerance is rejected during signature verification anyway.
Should I return an error for a duplicate webhook?
No, return 2xx. The sender retries on errors, so failing a duplicate invites more copies of it. Acknowledging says "this message is handled," which is true whether it was handled just now or earlier.
Does deduplication protect against replay attacks?
It closes the window that timestamp tolerance leaves open. A replayed capture inside the tolerance window carries the original message ID, so a receiver that deduplicates drops it. The signature, the timestamp check, and deduplication together cover forgery, stale replays, and fresh replays.
Ready to send webhooks?
Svix handles signing, retries, rate limiting, and delivery observability for the webhooks you send to your users, so your team can stay focused on your product.
Start sending webhooks with Svix or read the build vs. buy analysis