Debug webhook failures with evidence, not guesses

Webhook debugging becomes expensive when a team starts by reading application code instead of preserving the evidence. The first useful artifact is the exact request that arrived, or the exact delivery attempt that never arrived. Record the destination URL, HTTP method, timestamp with timezone, response status, response body, relevant headers, and the raw request body. Keep the body unmodified because a formatter can hide duplicate fields, invalid encoding, or an unexpected newline. If a provider offers an event identifier, record it alongside the request so you can distinguish a retry from a separate event.

A temporary capture endpoint is often the fastest way to answer basic questions. It tells you whether DNS resolved, whether TLS negotiation completed, whether the sender reached the expected path, and what the sender actually transmitted. This is different from inspecting a typed application object after a framework has already parsed and normalized it. The raw request is the boundary between systems. Preserve it before you decide which system is responsible. That one habit prevents hours of arguments based on assumptions about what the provider must have sent.

Most webhook incidents fall into a small set of classes: the request never left the sender, the request could not reach your edge, the edge rejected it, your handler rejected or crashed on it, or your handler accepted it but failed later while processing the event. The response status is a strong first signal. A timeout points toward networking, cold starts, or a handler that did not finish. A 404 usually means the URL or routing contract is wrong. A 401 or 403 suggests authentication or signature verification. A 400 means the receiver understood the request but rejected its shape. A 500 means the receiver accepted responsibility and then failed internally.

Do not treat every retry as a new bug. Providers retry after timeouts and many non-2xx responses, so a single logical event can produce a burst of nearly identical requests. Group attempts by provider event ID, payload hash, and a small time window. Then ask whether the first attempt had a different status from the later attempts. If the first attempt timed out but the second succeeded, you may have a latency or idempotency problem rather than a parsing problem. Classification turns a noisy timeline into a narrow investigation.

Replaying a webhook is useful only when the receiver can tolerate a duplicate. A handler that sends an email, creates a shipment, or grants access before recording an event ID can perform the side effect twice during a normal provider retry. Before replaying, identify the event's idempotency key and the durable record that should own it. A database uniqueness constraint is more reliable than an in-memory set because it survives restarts and multiple instances. The handler should check that record as part of the same transaction or use a state transition that makes a duplicate harmless.

Use a sanitized replay in development when the payload contains personal data or production secrets. Preserve the fields that affect validation and business logic, but replace recipient addresses, tokens, and large binary values. If you must replay against production, use a provider-supported replay feature or a one-time diagnostic route with strict access controls. Never paste a production signature into a public request tool. A replay is a controlled experiment, not a shortcut around the trust boundary.

Signature verification depends on the exact bytes the sender signed. Parsing JSON and serializing it again can change whitespace, key order, escaping, or number representation. Verify the signature against the untouched request body before handing the body to a JSON parser. Also verify the timestamp or replay window if the provider includes one. A correct signature with an old timestamp should still be rejected. Log the verification result and a request identifier, but never log the signature header or the secret used to verify it.

When verification fails, compare the sender's documented signing scheme with the receiver's implementation. Common mismatches include hashing a decoded body instead of raw bytes, using a different character encoding, concatenating timestamp and body in the wrong order, or comparing hexadecimal output with base64 output. Test known-good and known-bad fixtures in a unit test. A fixture should include the exact body, timestamp, header, and expected result so a framework upgrade cannot quietly move verification after parsing.

A webhook endpoint has two jobs: acknowledge receipt quickly and process the event reliably. Doing both synchronously makes provider timeouts more likely, especially when processing calls another API. A safer shape is to verify the request, persist a small event envelope with an idempotency key, return a success response, and process the envelope through a worker or an internal queue. If a queue is not available yet, keep the synchronous work bounded and make every downstream operation retryable. The acknowledgement should not claim success until the event is durably recorded.

This design also improves observability. You can measure receipt latency separately from processing latency, see whether an event is waiting, running, or failed, and retry processing without asking the sender to resend the request. Keep the original body available only as long as your retention policy allows, and store a redacted summary for routine operations. The goal is not to make every webhook asynchronous on day one. The goal is to make the boundary honest about what it has completed.

Useful webhook logs are structured around a correlation ID. Include the provider event ID, your internal event ID, route name, verification result, response status, processing state, attempt number, and elapsed milliseconds. Avoid logging the full payload by default. A hash or a small set of non-sensitive fields is usually enough to correlate attempts. If a payload must be inspected, put it behind an explicit diagnostic mode with a short retention period and access review.

Build a small timeline for each incident: received, verified, persisted, acknowledged, processed, and completed or failed. The first missing transition is often the fault boundary. If the request is received but never verified, look at signature configuration. If it is verified but not persisted, inspect the database. If it is persisted and acknowledged but not processed, inspect the worker. This timeline is more durable than a collection of arbitrary log lines because it reflects the lifecycle the system promises.

Before calling a webhook integration finished, test a valid request, an invalid signature, an old timestamp, a malformed body, a duplicate event, a provider retry, a slow downstream dependency, and an unknown event type. Confirm that unknown events are acknowledged safely or rejected intentionally, rather than crashing a shared handler. Confirm that the provider sees the response status you expect and that your alerting distinguishes a sender outage from a receiver outage.

Finally, document the contract next to the code: endpoint ownership, authentication method, retry behavior, idempotency key, maximum body size, timeout budget, event retention, and the operator who can replay an event. A clear contract reduces future debugging time because the next engineer can compare observations with an explicit promise. When you need a neutral place to capture incoming payloads, PingFlow's Webhook Inspector can help you observe the raw request before you change your application.

Create a short-lived capture endpoint that stores the raw body, headers, delivery ID, and receive timestamp before parsing. Keep the capture separate from business processing so a malformed payload is still evidence. Restrict access, redact credentials before export, and give every capture a correlation ID that follows the request through logs and the replay tool.

Run one known-good delivery, one invalid-signature delivery, one malformed body, and one dependency timeout through the same path. Compare edge status, handler status, latency, retry count, and the provider event ID. A successful response with a missing side effect points downstream; a timeout with no application log points to routing, startup, or an edge limit. Preserve the first failed attempt before replaying.

Keep captures and replay records on a bounded retention schedule, with access logging and deletion ownership. Replays must use the same idempotency key as the original event and require an operator reason. During an incident, roll back the handler or disable a failing downstream action while continuing to acknowledge safely classified events. Never solve a retry storm by accepting unauthenticated payloads.

Use HTTP Semantics (RFC 9110), the provider's delivery and retry contract, and the OWASP Logging Cheat Sheet as review references. Document the exact provider event schema and the retention policy beside the capture implementation.