The three signals and what each is actually for, cardinality as the economics of metrics, head vs tail sampling for traces, SLOs and error budgets as the interface between engineering and everyone else — and observing the event-driven paths where most tracing setups go blind.
~2.5h study~1.5h exercise2 interactive simsFast track ~70 min
Fast track — Datadog power user already?
Compresses to ~70 min. Signal definitions collapse; cardinality economics, tail sampling, SLO design, and async observability stay mandatory.
§01 takeaway · 5 min
§02 cardinality calculator, in full · 15 min · MANDATORY
§03 sampling sim, in full · 15 min · MANDATORY
§04 SLOs, in full · 20 min · MANDATORY
§05 async paths, in full · 15 min · MANDATORY
STEP 01
Three signals, three jobs
SKIM
Fast-track takeawayMetrics — cheap aggregates over time: detection and trends ("is it broken? since when? how much?"); pre-aggregation is why they're cheap and why they can't answer "which request". Traces — per-request causality across services: localization ("where in the chain?"); sampled, because complete tracing at your volume is a second production system. Logs — arbitrary detail at a point: explanation ("why exactly?"); structured (JSON), leveled honestly, and joined to traces via trace_id on every line — an un-correlated log estate is grep archaeology. The workflow they compose: metric alert → trace to the failing hop → logs of that exact request. Histograms over averages everywhere (M02: tails are the product), RED per service (rate/errors/duration), USE per resource (utilization/saturation/errors) — and instrument via OpenTelemetry so the vendor is a backend choice, not an architecture.
STEP 02
Cardinality is the bill
MANDATORY
A metric's cost = the product of its label values (each combination is a stored series). Multiply your way into bankruptcy:
Series count for http_request_duration
SERIES (× ~10 histogram buckets)
—
VERDICT
—
The rule: metrics carry bounded, low-cardinality dimensions you'll actually group by; anything identifying (user, session, request, content ID) belongs in traces/logs, which are built for per-event data. "Just add a label" is how one deploy 10×es the observability bill or knocks over the metrics backend.
Guardrails as platform policy: allowed-label lists in shared instrumentation libraries, relabel/drop rules at collection, per-team series budgets with dashboards — cardinality is a resource like CPU and gets governed like one.
Exemplars bridge the gap: modern histograms attach sampled trace IDs to buckets — click the p99 spike, land on an actual slow trace. Cheap metrics for detection, linked evidence for diagnosis.
STEP 03
Sampling: head vs tail
MANDATORY
100 traces, 3 with errors (rare tail events — exactly what tracing exists to catch). Sample at 10%:
Which traces survive?
Red = error traces. Outlined = kept by the sampler.
Head sampling decides at trace start (deterministic on trace ID — every service keeps or drops the same traces, or your traces arrive as fragments): cheap, predictable volume, and statistically blind — a 1-in-10k error class appears in your kept set once per 100k requests. Fine for aggregate latency truth; useless for rare-failure forensics.
Tail sampling decides after the trace completes, at a collector that buffers spans: keep 100% of errors and slow outliers + a small baseline of normal traffic. You trade collector infrastructure (buffer memory, all-spans-through-it bandwidth, a stateful tier to operate) for exactly the traces on-call needs. The standard estate answer: head-sample a baseline in SDKs + tail-sample errors/outliers at the OTel collector.
Context propagation is the foundation under all of it: W3C traceparent through every HTTP/gRPC hop and every broker message (§05) — one service dropping headers cuts every trace that passes through it, and the symptom (orphan spans) is subtle until the incident where you needed the join.
STEP 04
SLOs and error budgets
MANDATORY
SLI = a ratio of good events measured where users are (edge/RUM, not pod-local): "manifest requests answered <500 ms, successfully" / all. SLO = the target (99.9% over 30 days). Error budget = 1−SLO made spendable: 0.1% of a month ≈ 43 minutes of full outage, or proportionally more partial degradation.
The budget is a decision protocol, not a dashboard: budget healthy → ship fast, run risky migrations, spend it deliberately (chaos drills, M11 failovers). Budget exhausted → feature freeze in favor of reliability work — pre-agreed with product, which is the entire point: the SLO converts "is this reliable enough?" from a feelings argument into arithmetic both sides signed.
Alert on burn rate, not point thresholds: multi-window burn-rate alerts (e.g. page when burning 14× budget over 1h AND 5m — fast-burn; ticket at 2× over 24h — slow-burn) page for what threatens the monthly promise and stay silent for noise. This single change is most teams' largest pager-fatigue fix.
Choose SLIs per user journey, few and meaningful: for OTT — playback start success/latency, rebuffer ratio, license success (M23), API availability. Twenty SLOs are zero SLOs; each one needs an owner who can actually halt features when it burns.
Staff expectationRunning the SLO review as a leadership instrument: monthly, per journey — budget spent on what (incidents? deploys? dependency X?), trend, and the one investment that would buy the most budget back. That meeting, fed by burn-rate data, is how reliability work gets prioritized against features without heroics or table-pounding — and it's a manager-architect's home turf.
STEP 05
Observing the async estate
MANDATORY
Request-path tracing is table stakes; your platform's hard incidents live on the event paths (M15–M18), where default setups go blind:
Propagate trace context through messages: traceparent in RabbitMQ headers / Kafka record headers, consumers continuing (or linking) the trace — "purchase → outbox → relay → entitlement grant → cache invalidation" becomes one queryable causality chain instead of five islands. This is precisely the chain your 428 RCA had to reconstruct by hand.
The async golden signals: consumer lag (Kafka) / queue depth (Rabbit) as saturation (M14's signal), oldest-message age (depth can look fine while one poison message blocks a partition — age catches it), DLQ depth > 0 as a paging condition (M15), and outbox oldest-unpublished age (M18) — the invisible-event-outage detector.
End-to-end freshness SLOs for async journeys: "purchase → playable" as a measured duration (emit event timestamps, compute the distribution) with its own SLO — because the user experiences the pipeline's total, not any queue's local health. This is the async analog of measuring at the edge.
Correlation IDs ≠ trace IDs: business correlation (sagaId, orderId — M17) belongs in logs/events alongside trace context; traces answer "what called what", correlation answers "which business flow" — incidents need the join of both.
STEP 06
Exercise
MANDATORY
Fast trackStep 3 (~30 min): defining one real burn-rate SLO end-to-end changes how your team pages forever.
1
Wire the correlated stack. Two Spring services + RabbitMQ between them, OTel auto-instrumentation, trace context through the message headers, logs carrying trace_id. Verify one trace spans HTTP → queue → consumer, and that clicking from a latency histogram exemplar lands on a real trace.
2
Sample deliberately. OTel collector with tail sampling: keep all error traces + latency > p99 + 5% baseline. Generate 1k requests with 1% rare errors; confirm every error trace survives while volume drops ~90%. Compare against head-only 10%.
3
One real SLO. For playback-start (or your license path): define the SLI query at the edge, set the SLO with product, implement the two burn-rate alerts, and back-test against last quarter's incidents — would it have paged when it mattered and stayed quiet otherwise? Tune until yes.
4
Paper. Audit your estate's async observability against §05: which topics/queues carry trace context, which have age (not just depth) alerts, whether outbox age exists, and whether "purchase → playable" is measured anywhere. The blanks are the reason your last event-path RCA took days.
Self-check
Someone adds user_id as a metric label "temporarily". Walk the failure and the correct alternative.
Series count multiplies by active users (millions) × existing labels × histogram buckets — memory/cost explosion at the metrics backend, slow queries, possibly an ingestion outage; and it's unusable anyway (nobody graphs 5M series). Correct: user_id goes on trace/log attributes (per-event systems), with exemplars linking the metric spike to sampled traces where the user is visible.
Why must head sampling be deterministic on trace ID, and what breaks otherwise?
Every service in the chain makes its own keep/drop decision; if they flip independent coins, a trace is kept in some hops and dropped in others — fragments everywhere, complete traces almost nowhere (0.1⁵ for five hops at 10%). Hashing the trace ID makes all services agree, so 10% sampling yields 10% *complete* traces.
Your queue-depth alert stayed green while entitlement grants stalled for 40 minutes. What happened and what alert was missing?
A poison message (or a wedged consumer on one partition) blocked processing while producers paused or depth stayed within threshold — depth measures backlog size, not progress. Oldest-message age (or consumer-lag *age*) measures progress directly and would have climbed linearly from minute one. Depth for capacity, age for liveness; you need both.
Product wants "five nines". Use the error-budget frame to have the real conversation.
99.999% = 26 seconds of budget per month — less than one deploy hiccup, incompatible with weekly releases, human on-call response times, or your dependencies' own SLAs (a 99.95% CDN caps you below it regardless). Translate targets into what they buy and cost: 99.9% (43 min) supports current velocity; each added nine multiplies infra redundancy and freezes release cadence. The budget makes the trade explicit — pick the number whose costs you'll actually pay.