Designing for Failure

Failure is the default state

Any call that leaves your process can fail: DNS lookups, TLS handshakes, slow responses, dropped connections, and partial reads all happen in production. A resilient client treats these as expected events, not exceptions to be logged and forgotten. The question is never whether a request will fail, but how often and what happens next.

Begin by listing the things outside your control: the network, the remote service, third-party identity providers, and your own host being restarted mid-request. For each, decide on a deliberate response rather than letting an unhandled error bubble up to your user. A request that fails predictably is far cheaper to operate than one that fails in a new way every time.

The four building blocks

Four mechanisms cover most cases. Timeouts bound how long you wait. Retries with backoff recover from transient errors. Idempotency makes a safe-to-repeat request truly safe. Circuit breakers stop hammering a service that is already down.

They work as a set, not in isolation. Retrying without a timeout can pile requests onto a struggling server; retrying without idempotency can double-charge or duplicate a record. Treat the four as one design, and apply them at the boundary where your code talks to ours.

See timeouts and deadlines for the bounding half of the pattern.

Isolate the blast radius

When one dependency degrades, contain the damage. Give each external call its own timeout budget and, where possible, its own connection pool, so a slow identity provider does not starve calls to unrelated endpoints. This is sometimes called the bulkhead pattern, after the watertight compartments in a ship.

Avoid sharing a single global lock or thread pool across every outbound request. If one upstream stalls, an unbounded queue of waiting callers will exhaust memory and threads, turning a partial outage into a total one. Bounded queues that reject early are friendlier than queues that grow without limit.

Fail loud, recover quiet

A resilient client is observable. Emit a metric for every retry, timeout, and circuit-breaker trip, and tag it with the endpoint so you can see where trouble starts. Recovery itself should be quiet: once the dependency is healthy again, traffic should resume without manual intervention.

Test the failure paths the same way you test the happy path. Inject latency, return errors, and kill connections in a staging environment, then confirm your client behaves as designed. Code that has never failed in testing will fail for the first time in production, where it is most expensive.