System Design Masterclass · Module 31 · Extension of Week 4 · Capstone precursor

Chaos Engineering

Modules 12–19 installed defenses; this module is how you find out whether they're real. Chaos engineering as controlled experimentation — steady-state hypotheses, minimal blast radius, automatic abort — plus the fault catalog mapped to the course, and the organizational path from first game day to continuous verification. Do this before conducting Module 26's drills: the capstone assumes the method.

~2.5h study~2h exercise1 interactive simFast track ~65 min

Fast track — run a game day before?

Fast track mode

Compresses to ~65 min. Motivation collapses; the experimental method (with the sim), the fault-to-module map, and the maturity path stay mandatory.

§01 takeaway · 5 min
§02 experiment sim, both runs · 20 min · MANDATORY
§03 fault catalog · 15 min · MANDATORY
§04 organizational path, in full · 15 min · MANDATORY

STEP 01

Why deliberately break production-shaped systems

SKIM

Fast-track takeawayYour resilience mechanisms (breakers, retries, failovers, fallback ladders) are code paths that almost never execute — which means they rot silently: the fallback that references a decommissioned cache, the failover runbook with last year's hostnames, the breaker config nobody re-tuned after the dependency's SLA changed. Testing can't reach them because the interesting failures are emergent — they live in the interactions between services, timeouts, and retries (M12's amplification, M13's pool exhaustion), not in any unit. Chaos engineering is the discipline of executing those paths on purpose, under controlled conditions, while you're watching — trading small, bounded, scheduled pain for the elimination of large, unbounded, 3 AM pain. The one-line justification for leadership: we will find out whether the failover works; the only choice is whether we find out on Tuesday afternoon or during the final.

STEP 02

The method: it's an experiment, not vandalism

MANDATORY

Fast trackThe five steps below are the entire discipline; everything else is tooling. Run the sim both ways.

Experiment: "playback survives entitlement-cache latency"

Steady state: define normal numerically — playback-start success ≥99.5%, p99 <800 ms (your M24 SLIs, reused verbatim)

Hypothesis: "injecting +500 ms on entitlement-cache calls for 5% of traffic will NOT move the steady state" — falsifiable, written down, with the expected defense named (M13 slow-call breaker → serve-stale fallback)

Minimize blast radius: 5% of traffic, one AZ, off-peak, feature-flagged injection (M29) — and an automatic abort: kill the experiment if the SLI dips below the guardrail

Inject & observe: run the fault; watch the SLI, the defense's own metrics (breaker state, fallback rate), and nothing-you-expected-to-move

Conclude: hypothesis held → widen next time (25%, peak-adjacent); falsified → you found a real vulnerability for the price of a blip — fix, then re-run the same experiment as the regression test

The abort condition is what makes this engineering: an experiment you can't stop instantly is an incident you scheduled.

Steady state must be user-facing (M24's measure-at-the-edge rule): "CPU stayed fine" proves nothing; "playback-start success didn't move" proves the thing that matters.
One variable at a time, smallest first: the discipline mirrors canary analysis (M29) — earn each widening with evidence. Compound scenarios (latency + node loss) come only after singles pass.
A falsified hypothesis is the win condition, not the failure — you paid 5%-for-5-minutes to learn what would otherwise have cost a match night. Write it up like an incident (blameless, with actions), because that's exactly what it is, minus the pager.

STEP 03

The fault catalog, mapped to the course

MANDATORY

Fault to inject	Tooling shape	The defense on trial
Dependency latency (+N ms, the gray failure)	mesh/Envoy fault injection, Toxiproxy	slow-call breakers, bulkheads, deadline propagation (M13, M07) — the highest-yield first experiment
Dependency errors (5xx on X%)	mesh fault injection, chaos middleware	retry classification & budgets, fallback ladder (M12, M13)
Pod/instance kill	chaos-mesh/Litmus pod-kill, or kubectl	lifecycle correctness, registry staleness handling, PDBs (M29, M09, M08)
Network partition	netem/iptables, chaos-mesh netem	CP/AP behavior as designed (M03), quorum refusal (M30), split-brain guards (M11)
Dropped/duplicated/delayed messages	broker chaos, consumer-side injection	at-least-once handling, idempotency, DLQ flow, oldest-age alerts (M15, M16, M24)
Resource exhaustion (CPU/mem/disk/conn-pool)	stress-ng, cgroup squeezes, pool shrink	load shedding & admission control, bulkheads, JVM-under-throttle behavior (M14, M13, M29)
Clock skew	libfaketime, container time offset	token exp/nbf tolerance, cert validity, lease/TTL logic (M22, M21, M19)
Zone/region evacuation	traffic drain + scheduled game day	the M11 failover: RTO/RPO for real, cold-cache warming, fail-back (the drill that finds the hardcoded region name)
Expired cert / rotated key	short-lived cert in staging-prod mirror	rotation automation, JWKS overlap, the alert that should have fired at day 30 (M21, M22)

Sequencing rule: start where your incident history points (for you: dependency latency and cache/invalidation faults — the 428 family), in the environment closest to production you can defend, and graduate each experiment toward production as its blast-radius controls prove out. Staging-only chaos validates staging; the goal, reached incrementally, is production — because that's where the real config, data volumes, and traffic mixes live.

STEP 04

The organizational path

MANDATORY

Stage 1 — Game days (start here, this quarter): scheduled, announced, one scenario, all hands watching — half the value is the humans: does the on-call find the dashboard? does the runbook's command still exist? who actually has prod access? Output: a fix list and a repeat date. M26's capstone drills are game-day scripts; run one for real.
Stage 2 — Automated experiments in the pipeline: the passed game-day scenarios become CI-adjacent chaos suites (latency injection against every new build's canary — M29's gates extended with fault conditions), so resilience regressions are caught like functional ones.
Stage 3 — Continuous, randomized, in production: the monkey-style background process within strict budgets — only after stages 1–2 built the abort machinery, the SLO guardrails (M24), and the cultural trust. Skipping to stage 3 is how chaos programs get banned by their first incident review.
The error budget is the permission slip (M24): chaos spends budget deliberately — healthy budget funds experiments; exhausted budget pauses them. This single linkage answers "why are we breaking things on purpose?" with arithmetic the whole org already signed.
Blast-radius controls are non-negotiable everywhere: kill switch reachable in seconds, experiments tagged in traces (M24 — so "is this chaos or an incident?" is a query), never during declared freezes/events, and never on the paths currently burning budget.

Staff expectationPositioning chaos as verification of claims already made: every design doc from this course asserted defenses ("the breaker opens on slow calls," "failover holds RTO < 10 min," "consumers absorb duplicates") — chaos experiments are those assertions turned into repeatable tests, which makes the program's backlog trivially derivable: grep your architecture docs for claims, sort by blast radius, verify top-down. That framing also wins the political argument, because you're not proposing to break production; you're proposing to check whether the documents everyone approved are still true.

STEP 05

Exercise

MANDATORY

Fast trackStep 1 (~45 min): one complete experiment, method and all, on your Week-3 lab stack — then take the method into M26.

One full experiment, by the book. Use your Module-13 lab (service + slow-able DRM stub): write the steady-state definition, the hypothesis naming the defense, the abort condition; inject +500 ms via Toxiproxy at 10% of calls; observe SLI + breaker metrics; conclude in writing. Then falsify on purpose: disable the slow-call threshold and re-run — watch the experiment catch the misconfiguration your dashboards wouldn't.

Message chaos. Against your Module-15/16 lab: inject duplicate deliveries and a poison message under load; verify idempotency absorbs the former and the DLQ ladder + oldest-age alert catch the latter within the experiment's window. These two are the async claims most estates have never actually tested.

Design the first real game day. Pick the scenario your incident history nominates (the entitlement-cache latency one writes itself), and produce the one-page plan: hypothesis, blast radius, abort, roles (who injects, who observes, who can veto), success criteria, and the fix-list template. Get it on a calendar — the plan without a date is a document, not a program.

Then: Module 26. Conduct the capstone's five drills as tabletop game days using this module's method — hypothesis and abort condition per drill. The capstone is the final exam; this module was how to sit it.

Self-check

What separates a chaos experiment from an outage you caused?

Four things, all decided in advance: a numeric steady-state definition, a falsifiable hypothesis naming the defense under test, a minimized blast radius, and an automatic abort tied to user-facing guardrails. Remove any one and you've scheduled an incident. The method — not the tooling — is the discipline.

Why inject latency before injecting errors or kills?

Gray failure is both the most common real-world mode and the one defenses most often miss: error-counting breakers sleep through it (M13's top misconfiguration), retries amplify it (M12), and pools exhaust under it (the M13 sim). Kills test crash paths that platforms mostly handle (M29 reconciliation); latency tests the configs humans wrote. Highest probability × weakest coverage = first experiment.

Your first production experiment falsified its hypothesis and caused a 4-minute SLI dip. Leadership asks whether the program should continue. The answer?

The experiment worked exactly as designed: it found a real vulnerability (which would have fired during a peak event, unbounded, unwatched) at the cost of a bounded, aborted, off-peak blip — spent from an error budget allocated for exactly this. The fix ships this week and the re-run becomes a permanent regression test. The alternative wasn't "no dip"; it was "the same dip, ×50, during the final." Continue — and the 4-minute number proves the abort machinery works.

Why must chaos experiments be tagged in traces, and what goes wrong without it?

During an experiment, injected faults are indistinguishable from real ones in every signal (M24) — on-call sees breaker opens and error spikes and starts incident response against your own injection (wasted pages, eroded trust, and worst case a "fix" deployed against a synthetic fault). A chaos-experiment attribute on affected traces/metrics makes "is this us?" a one-query answer and lets dashboards exclude or highlight experiment traffic deliberately.