Circuit Breaker Builder

Demo

Click to play with sound.

---
name: Circuit Breaker Builder
description: Wraps flaky upstream dependencies in circuit breakers, aggressive timeouts, and per-dependency bulkheads so a slow or failing service degrades gracefully instead of cascading into a full outage. Use when a slow or unavailable upstream is stalling your threads, outbound calls hang with no timeout, one dependency's outage is taking down unrelated traffic, or you are integrating a network call that can realistically be slow or down. Do NOT use when the goal is staying under a provider's request quota or handling 429s — use rate-limit-handler instead; do NOT use to size a database connection pool — use connection-pool-tuner instead.
---
# Circuit Breaker Builder

Protect a service from slow or unavailable upstreams: fail fast, fall back, and isolate the blast radius so one sick dependency can't drain your whole capacity.

## Workflow

1. **Set an explicit timeout on every outbound call.** Configure connect and read timeouts separately; never rely on library defaults (often minutes or infinite). The total budget must be shorter than your own caller's timeout so you fail before they give up. A call with no timeout is a guaranteed resource leak under load.

2. **Wrap the call in a three-state breaker.** Closed: calls pass through. Open: calls fail instantly without touching the dependency. Half-open: a few probe calls test recovery. Implement these states explicitly per dependency, not a single global breaker.

3. **Trip on a rolling error rate, not a raw count.** Open the breaker when the error rate crosses a threshold over a sliding window with a minimum request volume (e.g. >50% over the last N calls, ignore until at least M calls). A raw failure count flaps at low traffic and trips on noise.

4. **Define a fallback per call site.** While open, return immediately: stale cache for reads, queue-for-later or accept-and-reconcile for writes, a sane default or degraded response where one exists, and a clean fast error only when there is no safe degradation. The point is to stop spending threads, connections, and latency on a call you expect to fail.

5. **Probe and recover from half-open.** After a cooldown, allow a limited number of probe calls. If they succeed, close the breaker. If any fail, re-open and reset (and back off) the cooldown. Never flood the recovering dependency with full traffic the instant the cooldown expires.

6. **Give each dependency its own bulkhead.** Isolate every upstream behind its own concurrency limit — a dedicated connection pool, thread pool, or concurrency semaphore. A saturated dependency then exhausts only its own bulkhead and leaves capacity for unrelated traffic. Never share one pool across multiple upstreams.

7. **Emit metrics and alert.** Record state transitions, trip counts, fallback rates, and timeout rates per dependency. Alert when a breaker opens. Tune thresholds and cooldowns against this real data, not against guesses.

## Quality bar