Circuit Breakers

The problem it solves

When a dependency fails, naive clients keep calling it — and often retrying — which wastes resources, piles up slow requests, and can drag your own system down with it. Worse, the relentless traffic can stop the struggling dependency from ever recovering.

A circuit breaker borrows from electrical safety: when failures cross a threshold, it trips and stops calls flowing for a while. This protects your system from wasting effort on a dependency that is clearly down, and protects that dependency from being hammered while it tries to recover. It converts a slow, cascading failure into a fast, contained one.

Three states

A breaker moves between three states:

• Closed — normal operation; requests flow and failures are counted.
• Open — too many failures have occurred, so calls are rejected immediately without even trying the dependency.
• Half-open — after a cool-down, a few trial requests are allowed through to test recovery.

From half-open, success closes the breaker and normal flow resumes; failure re-opens it for another cool-down. This cycle lets the system probe gently for recovery rather than either giving up forever or flooding the moment it might be back. The half-open state is what makes the breaker self-healing.

Tuning thresholds

A breaker needs thresholds: how many failures trip it, and how long it stays open before testing again. Set these to match the dependency's behaviour.

Too sensitive, and brief blips trip the breaker needlessly, cutting off a service that was fine. Too lax, and it never trips when it should, leaving you exposed to the very cascade it exists to prevent. A failure rate over a window is usually better than a raw count, because it accounts for traffic volume. The open duration should give the dependency real time to recover but not leave you blind to its return for too long. Expect to tune these against real behaviour rather than guessing once.

Pairing with fallbacks

An open breaker fails fast — but failing fast is only useful if you have something to do instead. Pair the breaker with a fallback so an open circuit produces a graceful response rather than a bare error.

if breaker.is_open():
    return cached_or_default()
return client.call(request)

Depending on the operation, the fallback might be cached data, a queued retry for later, or a clear temporarily unavailable message. The breaker decides when to stop trying; the fallback decides what to do instead. Together they keep your system responsive even while a dependency is down. This pairing is a core part of a resilient client.