Closing the loop that Module 02 opened: from demand model to pod count to invoice, utilization targets as risk decisions, autoscaling for spiky OTT traffic, and cost as an architectural fitness function reviewed like latency.
~2.5h study~1.5h exercise1 interactive modelFast track ~70 min
Fast track — sized fleets before?
Compresses to ~70 min. Demand modeling collapses (M02 built it); the capacity chain calculator, autoscaling-for-spikes, and cost levers stay mandatory.
§01 takeaway · 10 min
§02 calculator, tuned to your numbers · 20 min · MANDATORY
§03 autoscaling, in full · 15 min · MANDATORY
§04 cost levers, in full · 15 min · MANDATORY
STEP 01
The demand model
SKIM
Fast-track takeawayM02's envelope, made a living artifact: DAU → concurrency curves per journey → per-service RPS via the call-fan-out map (one play-start = N backend calls — count them, including retries at the M12 budget) → growth and event overlays. OTT's defining feature: the planned spike — a marquee match is 5–20× baseline concurrency with a vertical leading edge at kickoff; your capacity plan is really two plans (baseline economics + event playbook), and the ratio between them decides your whole autoscaling and headroom philosophy. Keep the model in a spreadsheet/notebook with named assumptions — the artifact's value is that every number has a defender.
STEP 02
The capacity chain
MANDATORY
Fast trackPut your real license-service numbers into this and keep the result. The chain: peak RPS → per-pod ceiling (measured, at SLO latency — not at 100% CPU) → utilization target → pods → money.
Fleet sizer
BASELINE PODS
—
EVENT PEAK PODS
—
N−1 AZ CHECK
—
BASELINE $/MO
—
Per-pod ceiling comes from load tests at SLO latency — the RPS where p99 crosses budget, not where CPU hits 100% (the knee is earlier; M14's goodput logic). Re-measure per release class; payload mixes drift.
Utilization target is a risk statement: 60% means "we absorb a 1.6× surprise or an AZ loss without paging". Higher utilization = cheaper and more fragile; the number should differ by tier (Tier-0 playback path lower, batch higher) and be chosen, not inherited from a default HPA config.
N−1 everything: capacity must hold with one AZ gone (so each AZ carries ≤ (N−1)/N of peak) and during deploys (surge capacity while old+new overlap). Both checks are one line of arithmetic each and both are commonly skipped until the incident.
Dependencies inherit the plan: your pods scale in seconds; the DB connection budget (pods × pool size — M02's check), Redis throughput, license-vendor rate contracts, and NAT/LB limits do not. Every scale-out plan needs the "what saturates next" column — the answer is usually a connection count or a vendor contract, not CPU.
STEP 03
Autoscaling when traffic is vertical
MANDATORY
Reactive autoscaling arrives late by construction: metric scrape + decision + pod schedule + image pull + JVM warmup (M08's slow start) = minutes — while a kickoff edge is seconds. HPA handles the diurnal curve; it cannot handle the match. Tuning notes for the part it does handle: scale on the leading indicator (RPS or queue depth per pod, not CPU — M14's argument again), fast up / slow down (stabilization windows), and scale-down caps so a metrics blip doesn't halve the fleet.
Planned spikes get scheduled pre-scaling: the event playbook (M10's readiness review) sets floors ahead of kickoff — calendar-driven KEDA/cron-HPA raising minReplicas for the playback path, license service (M23's herd), gateway, and the dependency checklist (§02) pre-verified. Autoscaling then handles the error term, not the event.
Warm capacity beats fast capacity: for JVM estates, over-provisioned warm pods (paused/low-weight, M08 slow-start pools) or reduced-startup images (CDS/AOT, smaller heaps) matter more than scheduler speed. Measure your true time-to-serving (schedule→ready→warmed) — it's the constant that decides how early pre-scaling must start.
Scale-down is the dangerous direction: post-match, millions leave in minutes; aggressive scale-down + the long-tail audience = latency spikes for the remainder, and connection-draining bugs (M08 §04) surface here. Slow, staged descent with SLO guard metrics.
STEP 04
Cost as an architectural property
MANDATORY
Know your unit economics: cost per streaming hour, per play-start, per subscriber-month — architecture decisions become comparable ("this cache tier costs $X/mo and saves $Y of DB + $Z of latency SLO risk"). For OTT the invoice usually ranks: CDN/egress ≫ compute ≫ storage ≫ observability — which means M10's offload ratio is your biggest cost lever, ahead of any pod tuning: a 1% offload improvement at scale outweighs months of rightsizing.
The compute levers, in leverage order: delete idle/zombie workloads (audit finds 10–20% in most estates) → rightsizing requests/limits to measured usage (the gap between requested and used is pure waste × fleet) → commitment discounts (RIs/Savings Plans sized to the baseline floor — the demand model tells you the floor; commit to that, let spot/on-demand carry spikes) → spot for stateless/batch tiers with M08-grade draining → efficiency work (JVM tuning, caching, protocol choices) last, because it's engineering-time-expensive.
Cross-AZ traffic is the silent line item: chatty east-west at fan-out (M02's ×N) with per-GB cross-AZ pricing adds up to real money — topology-aware routing (prefer same-AZ endpoints) and honest accounting of replication traffic (M04) belong in the review.
Observability bills are architecture too: M24's cardinality and sampling decisions are cost decisions; log verbosity × retention is often a top-five line. Budget per team, reviewed with the same cadence as error budgets.
Staff expectationRunning cost review like performance review: a monthly per-service dashboard of unit cost trend, the top three movers explained, and cost added as a section in design docs ("this design costs ~$X/mo at target load, dominated by Y") — estimated with M02-style envelope math, not finance-team precision. Engineers who bring the cost number to their own reviews get trusted with bigger systems; that's the cultural loop this discipline builds.
STEP 05
Exercise
MANDATORY
Fast trackStep 1 with real numbers (~30 min) — the measured per-pod ceiling is the datum everything else in this module stands on.
1
Measure a real ceiling. Load-test one of your services (staging) with stepped RPS; find the knee where p99 crosses SLO. Record ceiling-RPS, the resource that saturated (CPU? pool? downstream?), and re-run at two pod sizes to find the cheaper shape. Feed the §02 calculator with the results.
2
Time your true time-to-serving. Instrument pod-schedule → readiness → first-request-at-SLO-latency for one JVM service (warmup visible). That number × the kickoff curve tells you how many minutes before the match pre-scaling must fire; write it into the event playbook.
3
Paper — the model. Build the demand-model spreadsheet for your playback path: journeys → fan-out → per-service peak RPS, baseline vs marquee-event columns, the "what saturates next" row per service, and the resulting baseline commitment vs burst split. This is the artifact §01 described — it should fit on two sheets and survive an architecture review.
Self-check
Why is "pods at 60% CPU" not a capacity plan, per this module?
Three gaps: the ceiling that matters is RPS-at-SLO-latency (the knee precedes CPU saturation, and IO-bound services saturate pools with idle CPU); 60% is a risk decision that should vary by tier and be justified (what surprise does it absorb?); and pod capacity says nothing about what saturates next — connection budgets, Redis, vendor contracts — which is where scale-outs actually die.
Your HPA is perfectly tuned, yet the derby kickoff still browns out the license service. Why, structurally?
Reactive scaling's control loop (scrape+decide+schedule+pull+JVM warm) is minutes; the kickoff edge is seconds — by the time new pods serve, the herd already hit the old fleet, and M12 retries amplified it. Planned spikes need scheduled pre-scaling to the event playbook's floor, with autoscaling handling only the residual. HPA tuning cannot fix a control-loop-vs-edge-speed mismatch.
Where should commitment discounts stop and burst capacity begin?
Commit to the demand model's reliable floor — the baseline diurnal minimum-to-typical band you'll pay for regardless — and leave the event surges and growth uncertainty on on-demand/spot. Overcommitting turns the discount into a fixed cost of capacity you idle; undercommitting pays list price for your own predictable base. The model's baseline column IS the commitment sizing input — which is why the model has to exist first.
Rank for an OTT platform: 1% CDN offload improvement vs 20% compute rightsizing. Defend with structure, not numbers.
Egress/CDN dominates the OTT invoice (video bytes dwarf API compute), so a 1% offload gain moves the largest line item — typically worth more absolute money than 20% of the much-smaller compute line, while also cutting origin fleet needs (compound effect). Structure of the argument: always lever the biggest line first; the invoice ranking, not effort or visibility, sets the order. Verify with your actual bill — and it's why M10 is a cost module in disguise.