System Design Masterclass · Module 29 · Extension of Weeks 2 & 4

Container Orchestration & Deployment

The compute substrate under everything since Module 08, and the pipelines that move code onto it. Kubernetes as a reconciliation machine, pod lifecycle done right, sidecars vs daemonsets, and progressive delivery — canaries, blue-green, and feature flags as the decoupling of deploy from release.

~3h study~1.5h exercise1 interactive simFast track ~75 min

Fast track — kubectl in muscle memory?

Fast track mode

Compresses to ~75 min. The k8s mental model collapses; lifecycle correctness (it completes M08), placement trade-offs, and progressive delivery stay mandatory.

§01 takeaway · 10 min
§02 lifecycle, in full · 15 min · MANDATORY
§03 placement, in full · 15 min · MANDATORY
§04 canary sim + flags, in full · 25 min · MANDATORY

STEP 01

Kubernetes is a reconciliation loop

SKIM

Fast-track takeawayThe one model that organizes everything: you declare desired state (Deployment: "8 replicas of image v42"); controllers run reconciliation loops comparing desired vs actual and acting on the diff, forever. Everything is this loop: Deployments manage ReplicaSets manage Pods; Services select pods into endpoints (M09's third-party registration); HPA reconciles replica count against metrics (M25). Consequences worth owning: self-healing is free (a killed pod is just a diff), but so is fighting the controller (hand-edited resources revert — change the declaration, never the instance); the API server + etcd (a Raft store — M30) is the cluster's PC/EC heart (M03) — control-plane outage means no changes, while data-plane traffic keeps flowing, a distinction that matters at 21:00. Networking model in one line: every pod gets a routable IP, flat network, Services are virtual IPs load-balanced by kube-proxy at L4 (which is exactly why M07/M08's gRPC balancing needed help).

STEP 02

Pod lifecycle, done correctly

MANDATORY

Fast trackThis section is M08 §04's contract, now with the platform's exact mechanics — the difference between deploys that are invisible and deploys that page.

Startup: scheduled → image pull → init containers → app start → startup probe (protects slow JVM boots from liveness killing them) → readiness probe gates Service endpoints → (LB slow start ramps — M08). Readiness = "send traffic"; liveness = "restart me" — wiring liveness to anything dependency-shaped creates restart loops during brownouts (M08's rule, enforced by probe config).
Shutdown is a race you must sequence: SIGTERM and endpoint-removal happen concurrently — the pod can receive traffic after SIGTERM. The correct config: preStop: sleep N (N > endpoint-propagation time) so the pod keeps serving while the fleet forgets it, then Spring server.shutdown=graceful drains in-flight, then exit — all inside terminationGracePeriodSeconds (default 30 s; size it ≥ preStop + p99.9 request + drain, or k8s SIGKILLs mid-request).
Requests/limits are your M13 bulkheads at the node level: requests drive scheduling and are what you actually reserve; CPU limits throttle (beware: aggressive CPU limits + JVM = GC pauses that look like M19's stale-holder scenarios); memory limits OOM-kill. JVM must be told the truth: container-aware heap sizing (MaxRAMPercentage), or the heap fights the cgroup.
PodDisruptionBudgets + anti-affinity: PDBs cap voluntary disruption ("≥6 of 8 ready during node drains") so cluster maintenance can't violate your N−1 math (M25); anti-affinity/topology-spread puts replicas across nodes/AZs so the math holds when a node dies.

STEP 03

Sidecars, daemonsets, and where cross-cutting code lives

MANDATORY

	Sidecar (per pod)	DaemonSet (per node)	Library (in process)
Isolation	per-workload config & blast radius; scales with the app	shared per node — one instance serves all pods on it	none — it's your process
Cost	overhead × every pod (the mesh bill, M09)	overhead × nodes — order of magnitude cheaper	~free at runtime; paid in upgrade coordination
Upgrade	with the pod (restart-coupled) — native sidecar/ambient trends aim to fix this	independently, node-rolling	every service must rebuild & redeploy (the N-versions problem that motivated meshes)
Fits	per-request work needing workload identity: mesh proxies (M21 mTLS), per-app config	node-scoped agents: log/metric collection, node security, CNI	hot-path logic where a hop is too expensive; single-stack estates with a platform team (M09's "starter as mesh")

The decision echoes M09's mesh rule: uniformity requirements and stack heterogeneity push code out of process (sidecar/daemonset); latency sensitivity and single-stack discipline pull it in (library). Observability collection is daemonset-shaped almost always (node-level, no per-request identity needed); mTLS is sidecar/ambient-shaped (per-workload identity is the point); resilience logic (M12–M14) is genuinely contested — your Spring Boot starter vs mesh policy — and the honest answer is your platform team's capacity, not the technology.

STEP 04

Progressive delivery

MANDATORY

The strategies, by blast-radius shape: rolling (default; gradual but version-mixing — M07's compatibility rules are what make it safe at all) · blue-green (full parallel fleet, instant switch and instant rollback; costs 2× capacity for the window, and "instant" switches are M11-style traffic moves with connection-drain caveats; DB migrations must be compatible with both colors — expand-migrate-contract again) · canary (small weighted slice + automated metric gates — the one that scales to 200 services because humans don't watch dashboards at 3 AM):

Automated canary · gates: error rate <1%, p99 < +20% vs baseline

traffic split

0% canary

gate @ 5% · analyze 5 min

gate @ 25% · analyze 10 min

gate @ 50% · analyze 10 min

promote to 100%

Argo-Rollouts/Flagger-style: the controller shifts weight, compares canary metrics against the stable baseline, and promotes or rolls back — no human in the loop unless a gate fails.

Gate on the SLIs that page you (M24): error rate, latency vs concurrent baseline (not last week), and business canaries (playback-start success) — a release can be HTTP-healthy and product-broken. Compare canary pods against stable pods at the same moment so diurnal traffic doesn't fake a regression.
Feature flags decouple deploy from release: code ships dark; exposure ramps by user/device/region independently of pods — which converts release risk into a config change with instant rollback, enables the kill-switch your M28 under-attack mode and M13 fail-open policies need, and (for your world) lets a risky change ship Tuesday but activate after Saturday's final. Discipline: flags have owners and expiry dates (flag debt is real code debt), flag state is observable (which flags served this request → trace attributes, M24), and permanent operational switches are declared as such, not left as zombie flags.
CI is the trust chain: build once, promote the same artifact through environments (never rebuild per env); contract tests (M07) and migration compatibility checks as pipeline gates; image signing/provenance so what runs is what was built (the M20 signing rule, applied to your own supply chain). GitOps (declared state in git, controllers reconciling — §01's loop applied to deployment itself) makes "what's running and who changed it" a git log instead of an investigation.

Staff expectationThe maturity question isn't "do we canary?" but "what percentage of rollbacks are automated, and what's our deploy-to-detect time?" A platform where a bad release is caught at 5% traffic by a gate and reverted in 4 minutes without paging anyone has converted deployment risk into arithmetic — and that, plus feature flags for the business-risk dimension, is what lets 200 services deploy daily in front of five million users without the release calendar becoming a war room.

STEP 05

Exercise

MANDATORY

Fast trackStep 2 (~35 min): the automated canary catching your own injected regression is the module's proof.

Lifecycle to zero errors. kind/minikube: your echo service with startup/readiness/liveness probes, preStop sleep, graceful shutdown, PDB, and topology spread. Rolling restart under k6 load → zero non-200s (reproducing M08's exercise on the real platform). Then break it deliberately: remove preStop and measure the error burst; set an aggressive CPU limit and watch JVM p99 (GC throttling) — two configs, two incident classes.

Automated canary. Argo Rollouts (or Flagger) with a Prometheus analysis template gating on error rate + p99-vs-baseline. Ship a healthy image → watch auto-promotion; ship one with an injected 200 ms regression → watch the gate fail and auto-rollback at 5%. Save both rollout histories.

Flag discipline. Add a feature flag (any provider or a config-map + library) to a code path; expose by device-class targeting; wire flag state into trace attributes; then write the flag's expiry ticket at creation time — practicing the discipline, not just the mechanism.

Paper. Audit your real pipeline against §04: same-artifact promotion? contract-test gates? canary with automated analysis or "watch the dashboard"? rollback time measured? flag inventory with owners? The gaps, ranked by blast radius, are the platform team's next quarter.

Self-check

Why can a pod receive traffic after SIGTERM, and what's the correct defense?

Endpoint removal and SIGTERM are parallel, independently-propagating events (§01's reconciliation: kube-proxy rules and LB endpoint lists update asynchronously across nodes). Defense: preStop sleep longer than propagation (pod serves while being forgotten), then graceful drain, with terminationGracePeriodSeconds sized to cover the sum — sequencing by sleeping, because you can't order the events themselves.

Your JVM service has CPU limit = request = 1 core and mysterious multi-second p99 spikes. Mechanism?

CFS throttling: GC and JIT are multi-threaded bursts; with a hard 1-core limit the JVM gets throttled mid-collection, stretching pauses into the seconds — which then look like M19 stale-holder or M13 slow-call scenarios. Fixes: raise/remove CPU limits for latency-sensitive JVMs (keep requests honest for scheduling), size GC threads to the actual quota, and watch container throttling metrics, not just CPU usage.

Blue-green with a DB column rename: why does "instant rollback" become a lie, and what restores it?

Both colors share the database; a rename means green's schema breaks blue — after the switch, rolling back to blue fails against the migrated DB. Expand-migrate-contract (M07) restores it: add the new column (both colors work), dual-write/backfill, switch traffic, and only contract (drop old) after blue is retired. Schema compatibility across N and N±1 is the invariant that makes *any* rollback claim true — rolling, blue-green, or canary alike.

A canary gate compares this release's p99 to last Tuesday's. Why is that broken, and what's correct?

Traffic mix and load differ by day/hour — the comparison conflates release effects with diurnal effects (a kickoff during analysis "fails" a healthy release; a quiet window passes a bad one). Correct: A/B at the same instant — canary pods vs stable pods serving the same live traffic split, same time window; the only variable left is the release. It's the controlled-experiment principle, and it's why canary analysis needs baseline pods, not baseline history.