Breakers stop you from calling the dead; bulkheads stop the dead from taking you with them. State machine mechanics, the config that actually matters, fallback design, and thread-pool isolation for a JVM estate.
~2.5h study~1.5h exercise2 interactive simsFast track ~70 min
Fast track — Resilience4j in prod already?
Compresses to ~70 min. Motivation collapses; state-machine subtleties, config reasoning, fallback design, and bulkheads stay mandatory — most estates have breakers installed and misconfigured.
§01 takeaway · 5 min
§02 state machine + sim · 15 min · MANDATORY
§03 config reasoning, in full · 15 min · MANDATORY
§04 fallback design, in full · 15 min · MANDATORY
§05 bulkheads + pool sim · 15 min · MANDATORY
STEP 01
Why breakers exist
SKIM
Fast-track takeawayWithout a breaker, calling a dead dependency costs a full timeout per call — threads and connections held for seconds each, at full request volume. The math: 200 RPS × 2 s timeouts = 400 concurrent held threads → your pool (usually 200) exhausts → you are now down, for a dependency you could have lived without. The breaker converts that to a sub-millisecond local failure. It's a bet: after enough failures, the next call will probably fail too — so spend it on the fallback instead, and probe recovery cheaply.
Two framings worth keeping: the breaker as fail-fast converter (expensive distributed failure → cheap local failure), and as recovery protection — a browning-out service can't heal under full load + retry amplification (Module 12); breakers shed the load that prevents recovery.
Send a mix of calls. At ≥50% failures over the last 10 calls, the breaker opens; advance time to reach half-open; probes decide reopening vs closing.
The subtleties that separate installed-breakers from working-breakers:
Slow calls are failures. A dependency answering 200 OK in 4 s is operationally down. Count slowCallRateThreshold (with slowCallDurationThreshold) alongside errors, or the breaker sleeps through brownouts — the most common misconfiguration.
Half-open must cap probes. Uncapped, the accumulated herd floods through the moment the breaker half-opens, re-killing the recovering service — the breaker itself becomes a synchronized-wave generator (Module 12). Fixed small permittedNumberOfCallsInHalfOpenState, everyone else still fails fast.
Minimum call volume: without minimumNumberOfCalls, one failure in a quiet period = 100% rate = spurious open. Statistics need a denominator.
Scope: breaker per dependency per pod. Each pod learns independently (fine — its connections, its view); but alert on fleet open-rate: 1 pod open = pod problem, 80% of pods open = dependency incident, and the dashboards must distinguish.
STEP 03
Config is a set of claims about the dependency
MANDATORY
Every number encodes a belief — write the belief next to the number:
Setting
The claim it encodes
OTT example: DRM vendor
slidingWindow(100, TIME_BASED 10s)
how much evidence before acting; time-based windows react to rate changes, count-based to sparse traffic
10 s window: license calls are high-volume; react fast
failureRateThreshold 50%
"below this, partial failure is tolerable and retries handle it"
30% — license failures block playback; act early
slowCallDuration 800ms + rate 50%
"slower than this is as bad as failing"
p99.5 of vendor SLA + margin
waitDurationInOpenState 10s
expected recovery timescale — too short probes futilely, too long ignores recovery
vendor incidents last minutes: 20–30 s, or exponential
permittedCallsInHalfOpen 5
probe cost you'll spend on the question "is it back?"
5 real user requests as probes — with fallback behind them
recordExceptions / ignoreExceptions
what counts as the dependency's fault — business errors (404, INVALID_ARGUMENT) must NOT trip breakers
entitlement-denied is a valid answer, not a failure
Staff expectationReviewing breaker configs by asking for the claims, not the numbers — "what's this dependency's measured p99.5? what's its typical incident duration? which of its errors are answers vs failures?" A config that can't answer those is copied from a blog, and copied configs fail in both directions: tripping on business errors and sleeping through brownouts.
STEP 04
Fallback design — the actual hard part
MANDATORY
The breaker is plumbing; the product decision is what happens when it's open. The fallback hierarchy, best first:
Serve stale — Module 01's cache with extended/logical TTL: recommendations rail from 10 minutes ago is indistinguishable from fresh. The strongest fallback because it's invisible.
Serve degraded — generic rails instead of personalized; SD license path if the premium DRM route is down; default artwork. Product-visible but functional.
Fail open vs fail closed — the decision that needs a paper trail. Entitlement service down: fail closed (deny playback) protects revenue and breaks every legitimate viewer; fail open (allow with logging, bounded duration, non-premium content only) protects experience and leaks rights. Neither is an engineering call alone — get the business to sign the policy before the incident, encode it, and alert loudly while it's active.
Fail fast with honesty — a clear error beats a hang; it feeds the client's own retry/backoff correctly.
Never: fallback that calls another unprotected dependency. Fallback paths need the same timeout/breaker discipline — the backup path taking down the service is a classic.
STEP 05
Bulkheads: isolation before intelligence
MANDATORY
Breakers react to failure statistics; bulkheads make failure unable to spread, statistics or not. The JVM scenario: one slow dependency, one shared thread/connection pool:
Shared pool (24 threads) · DRM vendor goes slow
Blue = serving normal traffic (catalog, resume, profile). Red = stuck waiting on the slow DRM call. Watch what a shared pool does.
Semaphore bulkhead (Resilience4j Bulkhead): cap concurrent calls per dependency on the caller's threads — cheap, no context switch; the default for most call sites, and the natural fit for virtual-thread estates (limit concurrency, not carrier threads).
Thread-pool bulkhead: dedicated executor + queue per dependency — full isolation including timeout enforcement from outside the call; costs threads and switches. Reserve for the few dependencies whose client libraries can't be trusted to time out.
Bulkhead everything shared: DB connection pools per workload class (interactive vs batch), RabbitMQ channels, HTTP client pools per dependency. "One pool for all outbound HTTP" is the pre-incident state of most estates.
Order of operations per call: Bulkhead (may I add load?) → CircuitBreaker (is it worth calling?) → TimeLimiter (how long may it take?) → Retry (outermost, budgeted, Module 12). Getting this order wrong — e.g. retries inside the breaker — makes each retry a separate breaker event and burns bulkhead permits on doomed attempts.
STEP 06
Exercise
MANDATORY
Fast trackStep 1 (~30 min): reproducing pool exhaustion and fixing it with a bulkhead is the module's core scar.
1
Exhaust a pool. Spring Boot service, two endpoints: /fast (local) and /drm (calls a stub that sleeps 5 s on demand). Fixed Tomcat pool (24). Load both at 50 RPS, flip the stub slow: watch /fast die too. Add a semaphore bulkhead (max 8) on the DRM call: /fast stays at 100% while /drm degrades alone. Dashboards for both runs.
2
Tune a breaker with claims. Add a Resilience4j breaker on the DRM call. First: defaults, and demonstrate both failure modes (business-404s tripping it; 2 s slow calls not tripping it). Then write the §03 claims table for the stub and configure from it; verify open→half-open→closed transitions in /actuator metrics under scripted fault patterns.
3
Fallback ladder. Implement serve-stale (Redis, logical TTL) → degraded (default response) → fail-fast for the same endpoint, choosing by breaker state and cache availability. Chaos-test the full ladder.
4
Paper. For your real playback path: every dependency, its bulkhead limit (from Module-02 math: pool size vs timeout vs RPS), breaker claims, and the signed fail-open/closed policy for entitlement. This document is the incident-response pre-work.
Self-check
Breaker never opens during a brownout where p99 went from 60 ms to 3 s but errors stayed ~0. Why, and the fix?
It's counting only failures, and slow-successes aren't failures. Configure slow-call rate + duration thresholds so latency degradation trips it — a 3 s "success" is operationally a failure holding your threads (which is the §05 pool-exhaustion path).
Why must half-open cap concurrent probes?
All blocked callers are waiting; at half-open, uncapped they all rush the recovering dependency simultaneously — a breaker-synchronized stampede that re-kills it and re-opens the breaker in a flap loop. Few probes carry the information "is it back?"; the rest keep failing fast until the answer is yes.
Entitlement service is down for 90 s during a match. Argue both fallback policies in two sentences each.
Fail closed: no unlicensed playback ever — but every paying viewer is blocked during your peak, converting a backend blip into a customer-facing outage and a refund storm. Fail open (bounded): viewers keep watching, incident is invisible — you accept bounded rights leakage, mitigated by allowing only already-cached-entitled users or non-premium content, with loud alerting and automatic expiry. The point: it's a pre-signed business policy, not an on-call improvisation.
Why do retries belong OUTSIDE the breaker (and bulkhead)?
Inside, each retry is invisible to the breaker as an independent decision and consumes bulkhead permits on attempts the breaker would have rejected. Outside (Retry → Breaker → Bulkhead → call), the breaker sees every attempt, open-state rejections short-circuit retries instantly (fail-fast, budget preserved), and permits go only to calls that will actually be made.