Debugging Webhook Failures

Delivery or processing?

The very first question to settle in any webhook failure is where it actually broke. A delivery failure means the request never reached your endpoint at all, or that you returned a non-2xx response — it is fundamentally the provider's view of the world. A processing failure means the request did arrive and you acknowledged it correctly, but your own logic then went wrong somewhere afterwards. These are two very different problems wearing similar clothes.

The two need entirely different evidence to diagnose. Delivery failures show up in the provider's own delivery logs or dashboard, where you can see what it tried to send and how you responded. Processing failures, by contrast, show up in your application logs. Establishing which side of the line a failure falls on immediately narrows the search and stops you wasting time looking in completely the wrong place.

Trace by event id

Every delivery should carry a unique event id, and logging that id at each step of processing is, without much competition, the single most useful debugging habit you can adopt. Armed with the id, you can follow one specific event all the way through its journey: from the moment it arrives, through verification, into your queue, and out the far side into whatever it ultimately changed.

log.info('received', { eventId: event.id });
// ... verify, enqueue, process ...
log.info('processed', { eventId: event.id });

When something fails, you simply search your logs for that one id and read the whole story it tells. Without an id threading through, you are reduced to guessing among thousands of near-identical deliveries, which is slow and frustrating in equal measure.

The usual suspects

Reassuringly, most webhook failures fall into a fairly short and familiar list. Signature mismatches caused by verifying a re-serialised body instead of the raw bytes are extremely common and catch nearly everyone once. Timeouts arise from doing too much heavy work inline on the request path. Unhandled event types can throw an error if you forgot to ignore the ones you do not recognise. And misconfigured or stale subscription URLs quietly stop deliveries arriving altogether, with no error to alert you.

Run through this checklist of usual suspects before you start a deep investigation, because it resolves a surprisingly large share of cases in minutes. The signature and timeout issues in particular are treated in detail in verifying webhook signatures and the related guidance on responding quickly.

Build for diagnosability

The best debugging actually starts long before any incident occurs, by building the system to be diagnosable in the first place. Log every delivery with its id, its status, and its timing; expose metrics for received, processed, and failed counts; and set up alerts that fire when the failure rate climbs above normal. A dead-letter queue then hands you the actual problem payloads to inspect directly, rather than leaving you with only a vague error message and a hunch.

When something does go wrong, cross-reference your own logs against the provider's delivery history to pinpoint exactly where a delivery diverged from the happy path. The provider's delivery logs and any retry detail they expose are described in the API documentation, and together with your own observability they give you the complete picture.