Wraps flaky upstream dependencies in circuit breakers, aggressive timeouts, and per-dependency bulkheads so a slow or failing service degrades gracefully instead of cascading into a full outage. Use when a slow or unavailable upstream is stalling your threads, outbound calls hang with no timeout, one dependency's outage is taking down unrelated traffic, or you are integrating a network call that can realistically be slow or down. Do NOT use when the goal is staying under a provider's request quota or handling 429s - use rate-limit-handler instead; do NOT use to size a database connection pool - use connection-pool-tuner instead.
Click to play with sound.
---
name: Circuit Breaker Builder
description: Wraps flaky upstream dependencies in circuit breakers, aggressive timeouts, and per-dependency bulkheads so a slow or failing service degrades gracefully instead of cascading into a full outage. Use when a slow or unavailable upstream is stalling your threads, outbound calls hang with no timeout, one dependency's outage is taking down unrelated traffic, or you are integrating a network call that can realistically be slow or down. Do NOT use when the goal is staying under a provider's request quota or handling 429s - use rate-limit-handler instead; do NOT use to size a database connection pool - use connection-pool-tuner instead.
---
# Circuit Breaker Builder
Protect a service from slow or unavailable upstreams: fail fast, fall back, and isolate the blast radius so one sick dependency can't drain your whole capacity.
## Workflow
1. **Set an explicit timeout on every outbound call.** Configure connect and read timeouts separately; never rely on library defaults (often minutes or infinite). A sane starting point: connect timeout of 1-2 seconds, read timeout anchored to the dependency's observed p99 latency plus modest headroom - not a round number pulled from the air. The total budget must be shorter than your own caller's timeout so you fail before they give up. A call with no timeout is a guaranteed resource leak under load.
2. **Wrap the call in a three-state breaker.** Closed: calls pass through. Open: calls fail instantly without touching the dependency. Half-open: a few probe calls test recovery. Implement these states explicitly per dependency, not a single global breaker.
3. **Trip on a rolling error rate, not a raw count.** Open the breaker when the error rate crosses a threshold over a sliding window with a minimum request volume. Widely-used defaults from Hystrix and resilience4j are a good anchor: trip at ≥50% errors over the window, but only once at least 10-20 requests have been observed in it. A raw failure count flaps at low traffic and trips on noise.
4. **Define a fallback per call site.** While open, return immediately: stale cache for reads, queue-for-later or accept-and-reconcile for writes, a sane default or degraded response where one exists, and a clean fast error only when there is no safe degradation. The point is to stop spending threads, connections, and latency on a call you expect to fail.
5. **Probe and recover from half-open.** After a cooldown - 30-60 seconds is a reasonable starting range - allow a limited number of probe calls (single digits, e.g. 3-5). If they succeed, close the breaker. If any fail, re-open and reset (and back off) the cooldown. Never flood the recovering dependency with full traffic the instant the cooldown expires.
6. **Give each dependency its own bulkhead.** Isolate every upstream behind its own concurrency limit - a dedicated connection pool, thread pool, or concurrency semaphore. A saturated dependency then exhausts only its own bulkhead and leaves capacity for unrelated traffic. Never share one pool across multiple upstreams.
7. **Emit metrics and alert.** Record state transitions, trip counts, fallback rates, and timeout rates per dependency. Alert when a breaker opens. Tune thresholds and cooldowns against this real data, not against guesses.
## Deliverable