Design the Playback Path

Detection: entitlement-service latency/error burn-rate + cache-hit ratio collapse (M24). License requests fall through to the DB — which the defense-in-depth sized for a bounded multiple, with single-flight collapsing per key (M01) and admission control shedding background traffic first (M14). Breakers on the DB edge convert saturation into fast failures feeding the fallback ladder: serve-stale entitlements from L1/logical-TTL where present, then the pre-signed bounded fail-open for already-playing sessions (M13). In-flight viewers keep watching (licenses already held; renewals hit the grace window — M23). Human decision: extend fail-open scope/duration vs start refusing new joins — the policy doc names who decides. Recovery: warm the rebuilt cluster before readmitting it (M01 warming; M08 slow start), or the recovery causes the second stampede.

The gray failure: error-rate breakers sleep through it — but slow-call-rate thresholds trip (M13's most important config line), and bulkheads cap the threads the vendor edge may hold, so the license service's own pool survives (the M13 pool sim, live). Deadline propagation means requests arriving with <150 ms budget skip the vendor call into the fallback immediately (M07). Retry budget prevents amplification onto the struggling vendor (M12). Blast radius: new premium-tier license issuance degrades; everything else — playback in progress, non-premium, manifests — never notices, because the bulkhead made the failure unable to spread. Human decision: engage the vendor's secondary region/endpoint per runbook, and the comms line to the vendor with your trace evidence (M24).

Detection: per-app-version request-rate anomaly (the authenticated version header earning its keep) + license-service admission metrics. Defense already standing: per-device+app-version rate limits at the gateway (M14) cap each broken client; 429+Retry-After feeds any client-side backoff that exists; priority shedding protects playback-critical traffic from the flood; the license service's own admission control guards goodput. Blast radius: the broken version's users degrade (they were degraded anyway — their app is broken); everyone else unaffected. Human decisions: force-update/kill-switch that app version via config, and the retro item — the client SDK's retry policy now goes through the same review as server retry configs (M12's one-law).

Detection: edge SLIs (user-measured) collapse for the region while pod metrics look green — the measure-at-the-edge argument (M24) cashing out. Traffic moves below DNS where possible (anycast withdrawal / steering policy), DNS records already at moderate TTL with the disobedient tail expected in the RTO math (M11). Stateless playback tier is active/active — surviving region pre-scaled per the event playbook to absorb (M25's N−1 at region scale). The hard parts: session/entitlement caches in region B are cold for the arriving millions — warming + collapsing + shedding is the difference between failover and cascade (M01/M14); the data core promotes per its written RPO with split-brain guarded by external quorum (M11/M03). Human decision: the fail-back timing — after the match, never during, with the reconciliation step the runbook must contain.

Reconstruct with the course's vocabulary: grants written but invalidation events lost or delayed = a dual-write on the invalidation path (M18 — the outbox wasn't under it) or replica-lag read-your-writes anomaly on fresh purchases (M04); players retried a non-transient error code without backoff, amplifying (M12); the error taxonomy collapsed "pending" into "denied" so the player couldn't wait politely (M23/M17 — saga still in its retriable tail). The fixes write themselves as module numbers — which is the point of the course: incidents become compositions of known parts, and RCAs become lookups instead of investigations.