System Design Masterclass · Module 10 / Week 2

CDN & Edge Architecture

For an OTT platform the CDN is not an optimization — it is the data plane. Cache hierarchies and shielding, the Cache-Control contract done properly, purge vs surrogate keys, and live-event segment delivery where every mistake multiplies by a million viewers.

~3h study~1.5h exercise1 interactive simFast track ~80 min

Fast track — CDN operator already?

Compresses to ~80 min. Anatomy collapses; the header contract, origin shielding with request collapsing, and live-event delivery stay mandatory — this is the layer where your platform's peak events live or die.

  1. §01 takeaway · 5 min
  2. §02 header contract, in full · 20 min · MANDATORY
  3. §03 shield + collapsing + sim · 20 min · MANDATORY
  4. §04 invalidation takeaway · 10 min
  5. §05 live-event delivery, in full · 20 min · MANDATORY
STEP 01

Anatomy

SKIM
Fast-track takeawayLayers: client → edge POP (hundreds, closest to eyeballs, terminates TLS/HTTP-3) → regional/mid-tier caches → origin shield (one designated POP that all others fetch through) → your origin. Each layer multiplies hit ratio and divides origin load; the shield is the difference between "N-POPs × misses" and "1 × miss" hitting origin. Routing to the nearest POP is DNS- and/or anycast-based (Module 11). Two mental models: for VOD the CDN is a giant read-through cache (Module 01 rules apply verbatim); for live it's a fan-out tree.

Hit-ratio math across tiers behaves like Module 01's multi-tier cache: edge hit 90% + shield hit 80% of the remainder → origin sees 2% of requests. At 5M concurrent viewers requesting a segment every 4 s, that's the difference between 25k RPS and 1.25M RPS at origin — the CDN is your capacity plan.

STEP 02

The header contract

MANDATORY

Fast trackFull read. Cache-Control is the API between your services and a distributed system you don't operate — imprecision here is how private data ends up shared and how origins melt.

STEP 03

Origin shield & request collapsing

MANDATORY

Two mechanisms that decide whether a cold hot object is a non-event or an origin stampede:

Cold hot object · 100k concurrent requests, 4 POPs shown
POP 1
POP 2
POP 3
POP 4
Origin shield
Your origin
Each POP receives 25k concurrent requests for a segment nobody has cached yet. Compare what reaches your origin.
STEP 04

Invalidation at the edge

SKIM
Fast-track takeawaySame hierarchy as Module 01, CDN-flavored: TTL as the safety net · path purge — slow-ish (seconds–minutes to propagate globally), operational last resort · surrogate keys / cache tags — tag objects at response time (Surrogate-Key: content-42 series-7), purge by tag in near-real-time: the event-driven-invalidation pattern, with your RabbitMQ ContentUpdated events driving tag purges · versioned URLs/v42/art.jpg: immutable objects + max-age=31536000, immutable, "invalidate" by referencing a new URL — zero purge latency, the strongest pattern wherever a manifest/HTML references the asset. Rule: version what you can, tag-purge what you can't, path-purge in anger.
STEP 05

OTT delivery specifics

MANDATORY
Staff expectationOwning the origin offload ratio as a first-class SLO (e.g. ≥98% for segments, per-object-class dashboards), and running event-readiness reviews before marquee lives: prewarm list, per-CDN header-contract verification, origin capacity at assumed worst-case offload, and the multi-CDN failover drill actually executed — not just documented. The CDN bill and the origin fleet size are both downstream of decisions in this module.
STEP 06

Exercise

MANDATORY

Fast trackStep 1 (~30 min): building the shield + collapsing demo locally makes §03's multiplication table permanent.

1
Build a mini-CDN. Two nginx "POPs" + one nginx "shield" (proxy_cache) in front of a Spring Boot origin with a request counter and 300 ms artificial latency. Fire 200 concurrent requests at both POPs for a cold key: record origin hits with collapsing off (proxy_cache_lock off) vs on, with and without the shield tier. Reproduce the sim's table with real processes.
2
Header lab. Serve one endpoint with max-age=10, s-maxage=60, stale-while-revalidate=30, stale-if-error=600. Kill the origin and demonstrate stale-if-error serving; watch X-Cache/age headers across states. Then break it: add Vary: User-Agent and watch your hit ratio fragment per client.
3
Audit production. Pull real response headers for: one segment URL, one live manifest, one catalog API response, one personalized rail. Grade each against §02/§05; file the diffs as tickets. Check your actual origin-offload ratio per object class in CDN analytics — most estates find a surprise in the manifest class.
4
Paper — event readiness. Write the prewarm + failover runbook for a hypothetical final: objects to prewarm, expected origin RPS at 95%/98%/99% offload (Module 02 math), the multi-CDN switch trigger, and who executes it.

Self-check

Why is s-maxage + purge better than a short max-age for catalog responses?
Short TTLs make every cache re-fetch constantly — origin load scales with TTL inverse — while updates still wait out the TTL. Long s-maxage + surrogate-key purge gives near-perfect offload AND near-instant invalidation on actual change: you pay origin load only when content changes, not per interval.
A "just-published" segment gets 100k concurrent requests. Walk the request-count math through each §03 configuration.
No shield/no collapse: every POP's every request misses → up to 100k origin hits (bounded by POP count × concurrency). Shield only: each POP collapses nothing but all misses funnel to the shield → origin sees ~1 per shield-node miss, shield absorbs POP-count fetches — origin ~1–few, shield takes the herd. Shield + collapsing at every layer: each POP → 1 fetch, shield coalesces those → exactly 1 origin fetch.
Why do signed segment URLs threaten your hit ratio, and what's the fix?
If the signature/token becomes part of the cache key, every user's URL is a distinct object — hit ratio collapses to per-user. Fix: token validated at the edge but excluded from the cache key (signed cookies, or CDN config that strips the token before keying), so authorization is per-request while the cached object stays shared.
Live manifest TTL: why "half the segment duration" rather than zero?
TTL 0 forwards every player's manifest poll to origin — your highest-frequency request class at full fan-in. Half a segment duration means players still discover new segments within one segment period (no QoE impact) while the CDN absorbs ~all polls; SWR covers the refresh so the herd never sees a miss.