Design idempotent webhook handlers that survive retries

At-least-once delivery is a practical promise. A provider would rather deliver the same event twice than lose it when a network connection drops after your server completed the work but before the response reached the provider. From the sender's perspective, a timeout is ambiguous. The receiver may have failed before processing, or it may have processed successfully and lost the acknowledgement. Retrying is the only safe default. Your handler must therefore treat duplicates as expected traffic, not as exceptional behavior.

Idempotency means that processing the same logical event more than once produces the same durable outcome as processing it once. It does not necessarily mean every line of code runs once. A duplicate request may still be parsed and authenticated. The important boundary is the side effect: one invoice, one shipment, one account change, or one notification. Designing that boundary deliberately is more reliable than trying to deduplicate after an incident.

Prefer a provider-generated event ID when the provider guarantees that the ID represents one logical event. If no event ID exists, derive a stable key from fields that define the business action, but be careful: hashing an entire payload can treat harmless metadata changes as new events, while using a customer ID alone can incorrectly collapse separate actions. The key should be stable across retries and specific enough not to merge unrelated events. Store the provider name with the key when multiple providers can generate the same-looking IDs.

Do not use a timestamp generated by your receiver as the idempotency key. It changes on every attempt. Do not use the request's network connection or process ID. Those identify a delivery attempt, not the event. Include the event type and version when the same provider ID could be reused across domains. Write down the key derivation rule because future maintainers need to know whether changing it would reopen duplicate side effects for historical events.

An in-memory cache can reduce duplicate work on one instance, but it cannot provide a correctness guarantee when the service scales horizontally or restarts. The durable idempotency record should have a unique constraint on the chosen key and provider. The handler attempts to insert the event before performing the side effect. If the insert conflicts, the handler can return the previously recorded outcome or acknowledge the duplicate without doing the side effect again.

The race is important. Two identical deliveries can arrive at nearly the same time on different instances. A read-then-insert sequence can let both requests observe no record and continue. Use an atomic insert with a unique constraint, an upsert that records ownership, or a transaction with an explicit state transition. The database is the component that can arbitrate the race. Application-level checks are still useful for friendly responses, but they should sit behind the constraint.

A single boolean such as processed is rarely enough. A useful event record distinguishes received, verified, accepted, processing, completed, and failed states. Store the attempt count, last error category, and next retry time. A duplicate arriving while processing should not start a second worker. It can return an acknowledgement and let the original owner finish. If the owner crashes, a lease or timeout allows another worker to take over without losing the event.

State transitions should be monotonic where possible. An event that is completed should not move back to processing because a late duplicate arrived. If a business action can be corrected, model that correction as a new event instead of mutating the history of the original delivery. This makes audits easier and helps operators understand whether a duplicate was ignored, retried, or intentionally compensated.

Your database can protect its own writes, but a handler often calls another service. Pass the same idempotency key to payment, email, shipping, or provisioning APIs when they support one. If the downstream service does not support idempotency, create an outbox record that uniquely represents the intended side effect and let a worker own delivery. Record the downstream request ID and response so a retry can determine whether the call succeeded before the connection failed.

A common mistake is to mark the event completed before calling the external service, because the database transaction is easier that way. That prevents duplicate work but can lose the side effect if the external call fails. The opposite mistake is to call the external service first and mark completed afterward, which can duplicate the side effect when the database write fails. An outbox or a downstream idempotency key resolves this tension by making both operations observable and retryable.

Idempotency tests should simulate the awkward moments, not only a clean duplicate after completion. Deliver the same event concurrently. Kill the worker after the downstream call but before the completion record. Return a timeout after the database insert. Deliver an event while a previous attempt holds its lease. Send a duplicate with a changed non-business field. Each scenario should have an expected durable outcome and an expected acknowledgement status.

Use a fake downstream service that can fail at precise points. A test that only mocks a successful API response cannot expose the ambiguity between a completed call and a lost response. Include a reconciliation job that can find events stuck in processing and compare them with downstream records. The best idempotency design is one that remains understandable when an operator has to recover an event at 2 a.m.

Expose idempotency behavior in metrics and runbooks. Track duplicate rate, conflict rate, processing latency, events stuck in processing, and downstream retry counts. A rising duplicate rate can indicate provider latency, a slow database, or an acknowledgement path that is failing. It is not automatically a sender problem. Alert on the business consequence, such as uncompleted orders, rather than on every duplicate request.

Document how to replay an event safely, how to inspect the idempotency record, and how to compensate an external side effect. A replay tool should require an event ID, show the current state, and explain what will happen if the event is already complete. PingFlow's inspector is useful during the observation phase, but correctness ultimately belongs in the durable handler and its data model.

Give each logical event a durable idempotency key and claim it before performing a side effect. Store the key, payload hash, processing state, result, and timestamps in the same transactional boundary as the business change when possible. A duplicate should return the recorded result or a clear in-progress response, not execute the action again.

Test a first delivery, an immediate duplicate, a retry after a worker timeout, a duplicate with a changed payload, and two concurrent deliveries. Assert that the side effect occurs once, the duplicate is observable, and a failed attempt can be retried without losing the original event. Treat a payload-hash mismatch as an integrity incident rather than silently overwriting the record.

Expire idempotency records only after the provider's maximum retry window plus a safety margin. Track duplicate rate, in-progress age, permanently failed events, and manual replays. If a handler is released with a bug, pause the affected action, preserve incoming events, and replay from the durable record after the fix. Keep a compensating operation for side effects that cannot be rolled back automatically.

Review the provider's webhook retry contract, Stripe's idempotent request guidance where applicable, and the transactional outbox pattern. The article's implementation should state which database constraint is the final duplicate barrier.