System Design Masterclass · Module 11 / Week 2

DNS, Anycast & Global Traffic

The resolution path and its many caches, TTL as your failover clock, anycast vs GeoDNS vs latency-based routing, and designing multi-region failover you can actually execute at 21:00 on match night.

~2.5h study~1h exercise1 interactive simFast track ~65 min

Fast track — know your resolvers?

Compresses to ~65 min. Resolution mechanics collapse; TTL-failover math, steering trade-offs, and multi-region design stay mandatory.

  1. §01 takeaway · 5 min
  2. §02 TTL sim + the disobedience problem · 20 min · MANDATORY
  3. §03 steering comparison, in full · 15 min · MANDATORY
  4. §04 multi-region design, in full · 20 min · MANDATORY
STEP 01

Resolution: a chain of caches

SKIM
Fast-track takeawaystub resolver (OS) → recursive resolver (ISP / 8.8.8.8) → root → TLD → authoritative — with caches at every hop plus two more the exam forgets: the browser/app cache and the JVM (networkaddress.cache.ttl — historically cached forever under a security manager; verify your base images, because a JVM that never re-resolves makes every DNS failover a rolling restart). Negative caching (NXDOMAIN) bites during launches: query a name before its record exists and resolvers cache the absence for the SOA-negative TTL.

Design consequence: "change DNS" is never an action, it's a proposal that a fleet of caches you don't control will adopt over a TTL-shaped window (§02). Everything in this module is about designing within that constraint.

STEP 02

TTL is your failover clock

MANDATORY

Fast trackRun the sim, then read the disobedience list. Your real failover time is the TTL you set plus the lies resolvers tell about it.

Region failover · % of users reaching the healthy region
t + 30 s
t + TTL/2
t + TTL
t + 2×TTL

Why the curve never reaches 100% on schedule — resolver disobedience:

The working compromiseKeep user-facing records at moderate TTL (60–300 s) pointing at stable traffic-steering endpoints (LB VIPs, CDN hostnames), and do fast failover below DNS — health-checked routing policies at the DNS provider, anycast withdrawal, or LB-level cross-region spill — so DNS changes are rare and planned, not your emergency lever. If DNS *is* the emergency lever, your RTO includes the disobedient tail, and the runbook must say so.
STEP 03

Steering: anycast vs GeoDNS vs latency-based

MANDATORY
MechanismHowWinsFails when
Anycastsame IP announced from many sites; BGP routes each user to the "closest"instant failover (route withdrawal), DDoS absorption, zero DNS dependencyBGP "closest" ≠ lowest latency; long-lived TCP can break on route shifts mid-connection (fine for DNS/UDP and CDN edges, risky for long sessions)
GeoDNSauthoritative server answers by resolver locationsimple, policy-friendly (rights/regulatory routing — relevant to content licensing)sees the resolver's location, not the user's — public resolvers without ECS mislocate; TTL-bound agility
Latency-based DNSanswers by measured latency from resolver networks to your regionstracks real performance, not geographysame resolver-location blindness; measurement lag vs sudden shifts
Client-side steeringapp SDK measures endpoints/CDNs, picks per-sessiontrue per-user QoE decisions, multi-CDN switching mid-sessionyou own the steering logic, its telemetry, and its failure modes

They compose rather than compete — the standard OTT stack: anycast for DNS itself and CDN edges (fast, connectionless-friendly), Geo/latency DNS for regional API steering with rights-compliance overrides, client-side steering for multi-CDN segment delivery where QoE data is the deciding signal (Module 10).

STEP 04

Multi-region failover you can execute

MANDATORY
Staff expectationProducing the two-number answer — RTO and RPO per tier, derived: RTO = detection + decision + DNS/steering propagation (§02, incl. disobedient tail) + DB promotion + cache warm; RPO = replication lag at failure. And pushing the org to buy those numbers explicitly, because "we're multi-region" without them is a slogan. For an OTT: playback (stateless+CDN) can be active/active with RTO≈minutes; billing accepts RTO tens-of-minutes with RPO≈seconds — different tiers, different truths.
STEP 05

Exercise

MANDATORY

Fast trackSteps 1–2 (~30 min): tracing your own resolution chain and catching the JVM cache are immediately actionable in your estate.

1
Walk the chain. dig +trace your platform's API and CDN hostnames; identify every delegation and TTL. Then dig the same name via 8.8.8.8, 1.1.1.1, and your ISP resolver — compare answers (Geo-steering visible?) and observed TTL countdowns (clamping visible?).
2
Catch the JVM. Small Java app resolving a hostname in a loop while you flip a /etc/hosts or local dnsmasq record. Measure adoption time with default settings vs networkaddress.cache.ttl=30; then add an HTTP client with pooled connections and show traffic still doesn't move until connection max-lifetime kicks in. Write the two fixes into your platform defaults.
3
Simulate failover. Two "regions" (docker networks) behind dnsmasq with health-checked records (or Route53 in a sandbox). Kill region A under load; measure the §02 curve empirically at TTL 30 s vs 600 s. Compare with a below-DNS failover (HAProxy backend switch) doing the same job in ~1 s.
4
Paper. For your platform: RTO/RPO table per tier (playback, API, entitlement, billing), the §04 derivation for each, and the fail-back reconciliation step your current runbook is missing.

Self-check

You set TTL=60 and failed over 10 minutes ago; 8% of traffic still hits the dead region. Why, and what should the runbook have said?
The disobedient tail: TTL-clamping resolvers, stacked OS/app/JVM caches, and connection pools reusing established connections. Runbook: expect an exponential tail (minutes to hours for the last few %), keep the old region returning redirects/errors-fast rather than black-holing, and enforce connection max-lifetime so pooled traffic drains.
Why is anycast ideal for DNS and CDN edges but risky for a long-lived gRPC session?
BGP path changes can land mid-connection packets at a different site that has no TCP state → reset. DNS (UDP, single exchange) and CDN edges (short HTTP fetches, QUIC connection IDs help further) barely notice; an hours-long stream does. Long sessions want stable unicast endpoints behind steering, or QUIC's migration tolerance.
GeoDNS sends a user in Riyadh to Frankfurt. Most likely cause?
The user's resolver is what GeoDNS sees — a public resolver egress or corporate VPN resolver located in Europe, without EDNS Client Subnet passing the user's real subnet. Fixes: honor ECS at the authoritative, prefer latency/health-based policies, and let client-side steering correct per-session.
Why does active/passive for the data core force an RPO conversation?
Async replication means the standby trails the primary by the replication lag; promotion abandons the unshipped tail — those transactions are gone. RPO = that lag. Making it ~0 requires sync cross-region replication, which Module 02 priced at ~100 ms per write — a cost the business must explicitly accept or decline; silence means they've accepted data loss without knowing it.