The resolution path and its many caches, TTL as your failover clock, anycast vs GeoDNS vs latency-based routing, and designing multi-region failover you can actually execute at 21:00 on match night.
~2.5h study~1h exercise1 interactive simFast track ~65 min
Fast track — know your resolvers?
Compresses to ~65 min. Resolution mechanics collapse; TTL-failover math, steering trade-offs, and multi-region design stay mandatory.
§01 takeaway · 5 min
§02 TTL sim + the disobedience problem · 20 min · MANDATORY
§03 steering comparison, in full · 15 min · MANDATORY
§04 multi-region design, in full · 20 min · MANDATORY
STEP 01
Resolution: a chain of caches
SKIM
Fast-track takeawaystub resolver (OS) → recursive resolver (ISP / 8.8.8.8) → root → TLD → authoritative — with caches at every hop plus two more the exam forgets: the browser/app cache and the JVM (networkaddress.cache.ttl — historically cached forever under a security manager; verify your base images, because a JVM that never re-resolves makes every DNS failover a rolling restart). Negative caching (NXDOMAIN) bites during launches: query a name before its record exists and resolvers cache the absence for the SOA-negative TTL.
Design consequence: "change DNS" is never an action, it's a proposal that a fleet of caches you don't control will adopt over a TTL-shaped window (§02). Everything in this module is about designing within that constraint.
STEP 02
TTL is your failover clock
MANDATORY
Fast trackRun the sim, then read the disobedience list. Your real failover time is the TTL you set plus the lies resolvers tell about it.
Region failover · % of users reaching the healthy region
t + 30 s
—
t + TTL/2
—
t + TTL
—
t + 2×TTL
—
Why the curve never reaches 100% on schedule — resolver disobedience:
Some ISP resolvers enforce TTL floors (clamping your 30 s to minutes); a long tail caches far beyond TTL.
Client-side caches stack on top: OS, browser, app, JVM (networkaddress.cache.ttl), and connection pools that keep using established connections regardless of DNS — pools must also enforce max connection lifetime or DNS failover doesn't move existing traffic at all.
Low TTL costs: every expiry = a resolution on the request path (add ~one RTT to cold requests) and multiplied authoritative-DNS load — you're trading steady-state latency and DNS bill for failover agility.
The working compromiseKeep user-facing records at moderate TTL (60–300 s) pointing at stable traffic-steering endpoints (LB VIPs, CDN hostnames), and do fast failover below DNS — health-checked routing policies at the DNS provider, anycast withdrawal, or LB-level cross-region spill — so DNS changes are rare and planned, not your emergency lever. If DNS *is* the emergency lever, your RTO includes the disobedient tail, and the runbook must say so.
STEP 03
Steering: anycast vs GeoDNS vs latency-based
MANDATORY
Mechanism
How
Wins
Fails when
Anycast
same IP announced from many sites; BGP routes each user to the "closest"
instant failover (route withdrawal), DDoS absorption, zero DNS dependency
BGP "closest" ≠ lowest latency; long-lived TCP can break on route shifts mid-connection (fine for DNS/UDP and CDN edges, risky for long sessions)
GeoDNS
authoritative server answers by resolver location
simple, policy-friendly (rights/regulatory routing — relevant to content licensing)
sees the resolver's location, not the user's — public resolvers without ECS mislocate; TTL-bound agility
Latency-based DNS
answers by measured latency from resolver networks to your regions
tracks real performance, not geography
same resolver-location blindness; measurement lag vs sudden shifts
you own the steering logic, its telemetry, and its failure modes
They compose rather than compete — the standard OTT stack: anycast for DNS itself and CDN edges (fast, connectionless-friendly), Geo/latency DNS for regional API steering with rights-compliance overrides, client-side steering for multi-CDN segment delivery where QoE data is the deciding signal (Module 10).
STEP 04
Multi-region failover you can execute
MANDATORY
Choose the pattern per tier, honestly. Active/active for stateless tiers (both regions serve; failover = weight shift — continuously proven by normal traffic). Active/passive for the PC/EC data core (Module 03): async-replicated standby, promoted on failover, accepting the replication-lag data-loss window (RPO > 0) — write it down as a number, not a vibe.
The hard part is state, not traffic. Traffic moves in minutes (§02/§03); the questions that decide RTO are: who promotes the database (automated with what quorum, or a human), what happens to writes accepted-but-unreplicated (RPO), and how sessions/caches rehydrate in the surviving region (cold-cache stampede on arrival — Module 01 warming applies at region scale).
Fail back is harder than fail over. Data written in region B during the incident must reconcile with A before A takes traffic again. Runbooks that end at "traffic moved" are half-runbooks.
Drill it or it doesn't exist. Quarterly game-days that actually shift production traffic; the first three drills find the hardcoded region names, the IAM gaps, and the JVM DNS caches. An undrilled failover plan is documentation of intent.
Split-brain guard: region isolation ≠ region death. Promotion must require external quorum (a third witness region/service), or a partitioned-but-alive primary keeps taking writes while the standby is promoted — two primaries, Module 03's divergence, now with your billing data.
Staff expectationProducing the two-number answer — RTO and RPO per tier, derived: RTO = detection + decision + DNS/steering propagation (§02, incl. disobedient tail) + DB promotion + cache warm; RPO = replication lag at failure. And pushing the org to buy those numbers explicitly, because "we're multi-region" without them is a slogan. For an OTT: playback (stateless+CDN) can be active/active with RTO≈minutes; billing accepts RTO tens-of-minutes with RPO≈seconds — different tiers, different truths.
STEP 05
Exercise
MANDATORY
Fast trackSteps 1–2 (~30 min): tracing your own resolution chain and catching the JVM cache are immediately actionable in your estate.
1
Walk the chain.dig +trace your platform's API and CDN hostnames; identify every delegation and TTL. Then dig the same name via 8.8.8.8, 1.1.1.1, and your ISP resolver — compare answers (Geo-steering visible?) and observed TTL countdowns (clamping visible?).
2
Catch the JVM. Small Java app resolving a hostname in a loop while you flip a /etc/hosts or local dnsmasq record. Measure adoption time with default settings vs networkaddress.cache.ttl=30; then add an HTTP client with pooled connections and show traffic still doesn't move until connection max-lifetime kicks in. Write the two fixes into your platform defaults.
3
Simulate failover. Two "regions" (docker networks) behind dnsmasq with health-checked records (or Route53 in a sandbox). Kill region A under load; measure the §02 curve empirically at TTL 30 s vs 600 s. Compare with a below-DNS failover (HAProxy backend switch) doing the same job in ~1 s.
4
Paper. For your platform: RTO/RPO table per tier (playback, API, entitlement, billing), the §04 derivation for each, and the fail-back reconciliation step your current runbook is missing.
Self-check
You set TTL=60 and failed over 10 minutes ago; 8% of traffic still hits the dead region. Why, and what should the runbook have said?
The disobedient tail: TTL-clamping resolvers, stacked OS/app/JVM caches, and connection pools reusing established connections. Runbook: expect an exponential tail (minutes to hours for the last few %), keep the old region returning redirects/errors-fast rather than black-holing, and enforce connection max-lifetime so pooled traffic drains.
Why is anycast ideal for DNS and CDN edges but risky for a long-lived gRPC session?
BGP path changes can land mid-connection packets at a different site that has no TCP state → reset. DNS (UDP, single exchange) and CDN edges (short HTTP fetches, QUIC connection IDs help further) barely notice; an hours-long stream does. Long sessions want stable unicast endpoints behind steering, or QUIC's migration tolerance.
GeoDNS sends a user in Riyadh to Frankfurt. Most likely cause?
The user's resolver is what GeoDNS sees — a public resolver egress or corporate VPN resolver located in Europe, without EDNS Client Subnet passing the user's real subnet. Fixes: honor ECS at the authoritative, prefer latency/health-based policies, and let client-side steering correct per-session.
Why does active/passive for the data core force an RPO conversation?
Async replication means the standby trails the primary by the replication lag; promotion abandons the unshipped tail — those transactions are gone. RPO = that lag. Making it ~0 requires sync cross-region replication, which Module 02 priced at ~100 ms per write — a cost the business must explicitly accept or decline; silence means they've accepted data loss without knowing it.