System Design Masterclass · Module 24 / Week 4

Observability at Scale

The three signals and what each is actually for, cardinality as the economics of metrics, head vs tail sampling for traces, SLOs and error budgets as the interface between engineering and everyone else — and observing the event-driven paths where most tracing setups go blind.

~2.5h study~1.5h exercise2 interactive simsFast track ~70 min

Fast track — Datadog power user already?

Compresses to ~70 min. Signal definitions collapse; cardinality economics, tail sampling, SLO design, and async observability stay mandatory.

  1. §01 takeaway · 5 min
  2. §02 cardinality calculator, in full · 15 min · MANDATORY
  3. §03 sampling sim, in full · 15 min · MANDATORY
  4. §04 SLOs, in full · 20 min · MANDATORY
  5. §05 async paths, in full · 15 min · MANDATORY
STEP 01

Three signals, three jobs

SKIM
Fast-track takeawayMetrics — cheap aggregates over time: detection and trends ("is it broken? since when? how much?"); pre-aggregation is why they're cheap and why they can't answer "which request". Traces — per-request causality across services: localization ("where in the chain?"); sampled, because complete tracing at your volume is a second production system. Logs — arbitrary detail at a point: explanation ("why exactly?"); structured (JSON), leveled honestly, and joined to traces via trace_id on every line — an un-correlated log estate is grep archaeology. The workflow they compose: metric alert → trace to the failing hop → logs of that exact request. Histograms over averages everywhere (M02: tails are the product), RED per service (rate/errors/duration), USE per resource (utilization/saturation/errors) — and instrument via OpenTelemetry so the vendor is a backend choice, not an architecture.
STEP 02

Cardinality is the bill

MANDATORY

A metric's cost = the product of its label values (each combination is a stored series). Multiply your way into bankruptcy:

Series count for http_request_duration

SERIES (× ~10 histogram buckets)

VERDICT

STEP 03

Sampling: head vs tail

MANDATORY

100 traces, 3 with errors (rare tail events — exactly what tracing exists to catch). Sample at 10%:

Which traces survive?
Red = error traces. Outlined = kept by the sampler.
STEP 04

SLOs and error budgets

MANDATORY
Staff expectationRunning the SLO review as a leadership instrument: monthly, per journey — budget spent on what (incidents? deploys? dependency X?), trend, and the one investment that would buy the most budget back. That meeting, fed by burn-rate data, is how reliability work gets prioritized against features without heroics or table-pounding — and it's a manager-architect's home turf.
STEP 05

Observing the async estate

MANDATORY

Request-path tracing is table stakes; your platform's hard incidents live on the event paths (M15–M18), where default setups go blind:

STEP 06

Exercise

MANDATORY

Fast trackStep 3 (~30 min): defining one real burn-rate SLO end-to-end changes how your team pages forever.

1
Wire the correlated stack. Two Spring services + RabbitMQ between them, OTel auto-instrumentation, trace context through the message headers, logs carrying trace_id. Verify one trace spans HTTP → queue → consumer, and that clicking from a latency histogram exemplar lands on a real trace.
2
Sample deliberately. OTel collector with tail sampling: keep all error traces + latency > p99 + 5% baseline. Generate 1k requests with 1% rare errors; confirm every error trace survives while volume drops ~90%. Compare against head-only 10%.
3
One real SLO. For playback-start (or your license path): define the SLI query at the edge, set the SLO with product, implement the two burn-rate alerts, and back-test against last quarter's incidents — would it have paged when it mattered and stayed quiet otherwise? Tune until yes.
4
Paper. Audit your estate's async observability against §05: which topics/queues carry trace context, which have age (not just depth) alerts, whether outbox age exists, and whether "purchase → playable" is measured anywhere. The blanks are the reason your last event-path RCA took days.

Self-check

Someone adds user_id as a metric label "temporarily". Walk the failure and the correct alternative.
Series count multiplies by active users (millions) × existing labels × histogram buckets — memory/cost explosion at the metrics backend, slow queries, possibly an ingestion outage; and it's unusable anyway (nobody graphs 5M series). Correct: user_id goes on trace/log attributes (per-event systems), with exemplars linking the metric spike to sampled traces where the user is visible.
Why must head sampling be deterministic on trace ID, and what breaks otherwise?
Every service in the chain makes its own keep/drop decision; if they flip independent coins, a trace is kept in some hops and dropped in others — fragments everywhere, complete traces almost nowhere (0.1⁵ for five hops at 10%). Hashing the trace ID makes all services agree, so 10% sampling yields 10% *complete* traces.
Your queue-depth alert stayed green while entitlement grants stalled for 40 minutes. What happened and what alert was missing?
A poison message (or a wedged consumer on one partition) blocked processing while producers paused or depth stayed within threshold — depth measures backlog size, not progress. Oldest-message age (or consumer-lag *age*) measures progress directly and would have climbed linearly from minute one. Depth for capacity, age for liveness; you need both.
Product wants "five nines". Use the error-budget frame to have the real conversation.
99.999% = 26 seconds of budget per month — less than one deploy hiccup, incompatible with weekly releases, human on-call response times, or your dependencies' own SLAs (a 99.95% CDN caps you below it regardless). Translate targets into what they buy and cost: 99.9% (43 min) supports current velocity; each added nine multiplies infra redundancy and freezes release cadence. The budget makes the trade explicit — pick the number whose costs you'll actually pay.