Dead-Letter Queues

When retries are not enough

Retries are excellent at handling transient failures — a brief database outage, a momentary network timeout, a dependency that was restarting. But some events fail every single time, no matter how often you try them: a malformed payload, a genuine bug in your handler, or a reference to a record that simply will never exist. Retrying these forever wastes resources and, worse, can block the queue behind them while they take their turn over and over.

A dead-letter queue, usually shortened to DLQ, is where such stubborn events go once they have exhausted their retry budget. Instead of being silently dropped or endlessly retried, the problem event is set aside in a separate queue for human attention, while healthy events continue to flow through the main pipeline completely unobstructed.

Deciding what to dead-letter

The usual rule is to move an event into the DLQ once it has failed processing a set number of times. That threshold is what distinguishes a stubborn, genuine failure from a passing blip — a couple of retries comfortably cover transient issues, while persistent failure beyond the limit is a clear signal that something needs a human to look at it rather than more automated attempts.

if (attempts >= MAX_RETRIES) {
  deadLetter.push(event, lastError);
} else {
  retryLater(event);
}

Choosing the threshold is a balancing act worth getting roughly right. Set it too low and you give up on events that would have recovered on the next try; set it too high and you waste effort and queue time on events that were never going to succeed.

Keep the evidence

A dead-letter queue is only as useful as the context it captures, so store enough to actually diagnose each failure. At a minimum, keep the full original event payload, the number of attempts that were made, and the error from the last attempt. Without that surrounding context, a dead-lettered event is just a mystery you cannot act on — you know something failed, but not what, why, or how to fix it.

Pair the DLQ with alerting so that a growing queue gets noticed quickly rather than discovered weeks later. A silent DLQ quietly filling up is a missed problem compounding in the dark. Treat each new arrival as a signal worth investigating promptly, much like the failures examined in debugging webhook failures, where the same logs and ids help you trace the root cause.

Recovering dead-lettered events

The DLQ is a holding area, deliberately, and not a graveyard where events go to be forgotten. Once you have identified and fixed the root cause — patched the bug, corrected the bad data, restored the missing record — you can reprocess the events the queue is holding by feeding them back through the normal pipeline. Idempotent processing is what makes this safe, even for events that were partially handled before they failed the first time.

Some events will turn out to be genuinely unrecoverable, and those can be discarded after review with a clear conscience. The important thing is that the decision to drop an event is deliberate and informed, never an accident or a silent timeout. The provider's retry behaviour, which feeds directly into how you set your own thresholds, is documented in the API documentation.