System Design Masterclass · Module 02 / Week 1

Latency Numbers & Back-of-Envelope Estimation

The numbers every architect must know cold, and the 5-step estimation method that turns "will this scale?" from opinion into arithmetic. Worked against OTT-scale traffic throughout.

~2.5h study~1h practice3 interactive toolsFast track ~60 min

Fast track — comfortable with the numbers?

Fast track mode

Compresses to ~60 min. The tables collapse to ratios; the estimation method, latency budgets, and tail-latency math stay mandatory — they're where senior engineers actually differ from mid-level ones.

Skim §01–§02 takeaways; quiz yourself on the ratios · 10 min
§03 estimation method + calculator, in full · 20 min · MANDATORY
§04 budgets & tail amplification, in full · 20 min · MANDATORY
Exercise part B only: budget your real playback-start path · 30 min

STEP 01

The latency ladder

SKIM

Fast-track takeawayCarry the ratios: memory is ~100× a CPU cache ref · a same-DC network hop is ~5× an SSD read · a DB query is ~10× a Redis call · a cross-region hop is ~1,000× a same-DC hop and cannot be optimized away — only avoided (caching, replication, edge). Also: a mutex is ~25 ns but a context switch is ~5 µs — 200× — which is why thread-per-request dies before NIO/virtual threads do.

These are the constants of your universe. Estimation, capacity planning, and design reviews all reduce to multiplying these together. Grouped by tier:

Operation	Latency	Intuition anchor
CPU cycle (3 GHz)	~0.3 ns	the unit everything else is measured against
L1 / L2 cache ref	1 / 4 ns	on-die
Mutex lock/unlock (uncontended)	~25 ns	cheap — contention is what's expensive
Main memory ref	~100 ns	100× L1 — why CPU-cache-friendly data layouts matter
Compress 1 KB (LZ4-class)	~2 µs	compression is nearly free vs any network hop
Context switch / syscall	~1–5 µs	200× a mutex — the cost of thread-per-request
Read 1 MB from RAM	~3 µs	sequential RAM ≈ 250 GB/s effective
SSD random read (4 KB)	~100–150 µs	NVMe; ~100k IOPS per device
Same-AZ network RTT	~500 µs	includes kernel + serialization, not just wire
Read 1 MB from SSD	~1 ms	sequential; ~1–3 GB/s
Read 1 MB over 10 Gbps network	~1 ms	network ≈ SSD for sequential throughput
HDD seek	~10 ms	why HDDs survive only for cold blobs
Same-region, cross-AZ RTT	~1–2 ms	the cost of AZ-redundant sync replication
Cross-region RTT (e.g. Bahrain→Frankfurt)	~80–120 ms	speed of light — cannot be engineered away
TLS handshake (cold)	~1–2 RTT	why connection pooling & TLS resumption matter

Staff expectationYou don't recite this table — you compute with it. "The recommendation service adds a cross-AZ hop plus one Redis read plus deserialization of a 200 KB payload — that's ~1.5 ms + 0.3 ms + ~0.6 ms ≈ 2.5 ms before it does any work." That sentence, produced live in a design review, is the skill.

STEP 02

Throughput rules of thumb

SKIM

Fast-track takeawayPer-node ceilings to carry: Postgres/MySQL ~5k mixed TPS (reads to ~50k if indexed & cached) · Redis ~100k simple ops/s per core · Kafka/RabbitMQ ~50k–1M msgs/s depending on ack mode & size · one Spring Boot pod ~1–10k RPS depending on work per request · NVMe ~100k IOPS · 10 Gbps NIC ≈ 1.25 GB/s. And the day math: 1M requests/day ≈ 12 RPS average — daily volumes always sound scarier than they are; peak factor (3–10× for video prime time) is what sizes systems.

Latency tells you how long one operation takes; these tell you how many a node survives. All are order-of-magnitude, single-node, and assume sane tuning:

System	Ballpark ceiling	What moves it
Postgres / MySQL	~5k mixed TPS · ~50k cached reads/s	working set in RAM, index quality, connection count
Redis (single core)	~100k simple ops/s	pipelining ↑, big values / Lua / O(N) cmds ↓
Kafka partition	~10–50 MB/s	batch size, acks, compression
RabbitMQ node	~50k msgs/s transient · ~10–20k persistent	persistence, publisher confirms, queue count
Spring Boot pod	~1k–10k RPS	work per request; virtual threads/WebFlux for IO-bound
NVMe SSD	~100k IOPS · ~2 GB/s seq	queue depth, block size
10 Gbps NIC	1.25 GB/s	the hard cap on any single-node data plane

Time conversions you'll use constantly: 1 day ≈ 86,400 s ≈ 10⁵ s for mental math · 1M/day ≈ 12/s · 1B/month ≈ 400/s · 30M seconds ≈ 1 year.

STEP 03

The estimation method

MANDATORY

Fast trackRead in full and drive the calculator. The method is five steps of disciplined multiplication — the discipline (peak factors, read/write split, stating assumptions out loud) is what separates a defensible estimate from a guess.

Every estimate follows the same shape. State assumptions out loud — being auditable matters more than being precise; the goal is the right power of ten.

1 · Scope the unit. What one "use" costs: one playback start = 1 auth check + 1 entitlement + 1 manifest + 1 DRM license + ~4 catalog reads.
2 · Traffic. DAU × actions/user/day ÷ 86,400 = average RPS. Multiply by peak factor — for video that's 5–10× (everyone watches at 21:00, and a match kickoff is a step function, not a curve).
3 · Read/write split. Route reads to cache math (× miss ratio → DB), writes to DB/queue math.
4 · Storage & bandwidth. objects × size × replication factor × growth; RPS × payload = egress.
5 · Divide by per-node ceilings (§02) → node counts. Then sanity-check against reality: if the answer says 2,000 DB nodes, your design is wrong, not your arithmetic.

Envelope calculator · OTT playback-start traffic

DAU 5M Playback starts / user / day 6 Peak factor 6× Cache hit ratio (catalog+manifest reads) 95% Backend reads per start 7

Avg starts/s

—

DAU × starts ÷ 86,400

Peak starts/s

—

× peak factor — size for this

Peak backend reads/s

—

× reads per start

Reaches DB (post-cache)

—

The lesson the sliders teachDrag hit ratio from 95% → 70% and watch the DB row. Cache hit ratio is a capacity multiplier, not a latency nicety — the §01 module's stampede content and this arithmetic are the same fact viewed from two sides.

STEP 04

Latency budgets & tail amplification

MANDATORY

An SLO like "playback start p99 < 800 ms" is meaningless until it's decomposed into a budget that each team can be held to:

p99 ≤ 800 ms playback start CDN + TLS + edge 120 ms API gateway + auth 60 ms entitlement check 80 ms ← includes 1 DB round trip manifest assembly 150 ms ← fan-out to 4 services, parallel DRM license 200 ms ← external vendor: the budget hog client parse + first byte 150 ms reserve / jitter 40 ms

Two staff-level rules: budget the p99, not the mean (means hide everything), and parallel fan-out costs the max of its branches, sequential costs the sum — so the architecture diagram is the latency budget.

Tail latency amplification

The counterintuitive killer in microservice estates: even rare slowness becomes common under fan-out. If each dependency is slow 1% of the time, a request touching N of them is slow with probability 1 − 0.99ᴺ:

Tail amplifier · P(request hits ≥1 slow server)

Services touched per request 1 Per-service P(slow) 1%

Requests experiencing tail latency

Drag services to 100 — a fan-out of 100 with 1% slow servers makes 63% of user requests slow. Your p99 problem is now a p37 problem.

Mitigations (each gets depth in Week 3): hedged requests (send a second copy after the p95 mark, take the first answer), tight per-hop timeouts with budgets passed downstream, cutting fan-out via caching/denormalization, and isolating the slow 1% (GC tuning, connection pool sizing) because tails are usually caused, not random.

Staff expectationGiven any incident graph, you can decompose "p99 went from 300 ms to 2 s" into which hop's budget broke using distributed traces, and you know that adding one more synchronous dependency to a 40-service call path is a tail-latency decision, not just a coupling decision. That argument — made with the 1−0.99ᴺ math — is how you win the "just call the service synchronously" debate.

STEP 05

Exercise

MANDATORY

Fast trackDo part B only (~30 min) — it uses your production numbers and produces an artifact you can actually use at work.

Cold estimation drill (30 min, no tools). On paper: size the stream-harness for a World Cup final — 2M concurrent viewers joining over 10 minutes, each start = 1 license + 1 manifest + 5 catalog reads. Compute peak starts/s, license QPS, catalog QPS at 90% and 99% hit ratio, and how many Redis cores and DB replicas §02's ceilings imply. Then write down which single number you'd challenge first.

Budget your real path (30–45 min). Pull one real playback-start trace from Datadog APM. Write the hop-by-hop p50/p99 table, assign each hop a budget that sums to your SLO, and flag every hop currently over budget. Count N = services touched and compute your own 1−(1−p)ᴺ tail amplification using your real per-service slow rate.

Stretch. Verify one §02 ceiling empirically: redis-benchmark -t get,set -P 16 and pgbench on a dev box; compare measured vs ballpark and explain the gap.

Self-check

Your API does 50M requests/day. Peak RPS to size for?

50M ÷ 86,400 ≈ 580 avg RPS. Video peak factor 5–10× → size for ~3–6k RPS. Anyone who sizes for 580 has designed an outage for 21:00.

Sync replication across AZs adds how much write latency, and cross-region?

Cross-AZ ~1–2 ms per round trip — usually acceptable. Cross-region ~80–120 ms — usually unacceptable on a user-facing write path, which is why cross-region is normally async and why geo-distributed strong consistency is expensive (see CAP/PACELC, Module 03).

Why can compressing payloads be a latency *win* despite CPU cost?

Compressing 1 KB costs ~2 µs of CPU; sending fewer bytes saves network time that dwarfs it — 1 MB over 10 Gbps is ~1 ms, so 5× compression saves ~800 µs for a few hundred µs of CPU. Below same-rack latencies it stops paying; across regions it always pays.

A request fans out to 30 services, each slow 2% of the time. What fraction of requests are slow?

1 − 0.98³⁰ ≈ 45%. Nearly half — from dependencies that are individually "fine 98% of the time." This is the strongest quantitative argument for reducing synchronous fan-out.