L4 vs L7, the algorithms and when each one loses, health checking without flapping, and the lifecycle mechanics — draining, slow start — that make deploys invisible instead of incident-shaped.
~2.5h study~1.5h exercise1 interactive simFast track ~70 min
Fast track — run LBs already?
Compresses to ~70 min. Layer definitions and topology tour collapse; algorithm failure modes, health-check design, and lifecycle mechanics stay mandatory.
§01 takeaway · 5 min
§02 algorithms + race sim, in full · 25 min · MANDATORY
§03 health checking, in full · 15 min · MANDATORY
§04 draining & slow start, in full · 15 min · MANDATORY
Exercise step 2 · 20 min
STEP 01
L4 vs L7
SKIM
Fast-track takeawayL4 balances connections (TCP/UDP tuples): microsecond overhead, millions of flows, sees no requests — so long-lived HTTP/2/gRPC connections make it lumpy (Module 07's pinned-pod problem). L7 terminates the protocol and balances requests: routing on path/headers, retries, per-call balancing — for CPU, latency, and a TLS-termination position in the trust model. Standard stack: L4 (NLB-class) at the very edge for scale → L7 (Envoy/ALB-class) behind it for brains. Choose per hop by asking "do I need to see requests here?"
L4 (transport)
L7 (application)
Balances
connections/flows
individual requests
Sees
IPs, ports
paths, headers, cookies, gRPC methods
Cost
~µs, near line-rate
parse + terminate TLS: real CPU, +ms
Enables
raw scale, non-HTTP protocols
routing, retries, canaries, per-call gRPC LB, WAF
Weakness
blind to request skew on pooled connections
is itself a service to capacity-plan and secure
STEP 02
Algorithms — and when each loses
MANDATORY
Fast trackRun the race with the slow node enabled. The point of this section is not the algorithm list — it's seeing why round robin degrades exactly when you need balancing most.
The scenario that separates algorithms: request durations are unequal (real traffic), and node 3 is degraded — 4× slower (GC storm, noisy neighbor). 60 requests arrive; bars show in-flight queue depth:
Algorithm race · node 3 degraded 4×
MAX QUEUE DEPTH —
REQUESTS ON SLOW NODE —
Round robin keeps feeding the slow node its fair share — its queue grows without bound. Least-connections adapts. P2C gets ~the same result with O(1) state.
Round robin / weighted RR — assumes equal cost and equal nodes; both assumptions fail in production. Fine as a default only with health checks + outlier ejection covering for it.
Least connections / least outstanding requests — adapts to real load; the workhorse. Danger: a fast-failing node has few outstanding requests and attracts more traffic — the black-hole effect. Pair with outlier detection (§03) always.
Power of two choices (P2C) — pick 2 random nodes, send to the less loaded. Near-optimal balance, trivial state, no herd behavior when many LB instances share stale load info. Envoy's modern default; the right answer in most reviews.
Latency-aware (peak-EWMA) — weights by observed response time; strongest against gray degradation, and the strongest black-hole risk when fast-fail = low latency. Requires error-aware scoring.
Consistent-hash / ring balancing — same math as Module 05, applied to routing: session-to-pod stickiness for WebSockets, cache-affinity routing so each pod's L1 cache owns a keyspace slice. You trade balance for locality — monitor the hot-shard risk you just imported.
IP-hash stickiness — legacy stickiness; breaks on NAT (one corporate IP = thousands of users on one pod) and on topology change. Prefer cookie- or token-based stickiness at L7 if you must be sticky at all.
STEP 03
Health checking without self-harm
MANDATORY
Liveness ≠ readiness. Liveness = "process should not be restarted"; readiness = "send me traffic". A pod warming caches or draining is alive but not ready. Wiring one probe to both jobs causes restart loops during startup — a classic k8s estate bug.
Shallow vs deep checks. A deep check that pings the database makes every pod unready when the DB blips — the LB then removes all capacity, converting a partial dependency brownout into a total outage. Rule: readiness reflects the pod's own health; dependency health is handled by circuit breakers per-request (Module 13). Deep checks belong in monitoring, not routing.
Passive checks / outlier ejection. Active probes sample; real traffic tells the truth. Eject a node on consecutive 5xx/timeout, with bounded ejection (never eject >X% of the pool — the same all-capacity-gone protection) and exponential re-admission.
Flap damping. Thresholds (3 fails to eject, 2 passes to return) and hysteresis, or a node on the edge oscillates and connection churn amplifies its degradation.
Staff expectationThe panic-mode question: "what does our LB do when every backend fails checks?" Good answers exist (serve-stale-DNS, fail-open to all backends, static fallback pool) but only if chosen beforehand. Envoy's panic threshold — below X% healthy, ignore health and spray all — encodes the judgment that during a mass event, health signal is probably wrong and load spreading beats load concentration on the few "healthy" survivors.
STEP 04
Lifecycle: draining & slow start
MANDATORY
Most "deploy caused 502s" incidents are lifecycle bugs, not code bugs. The correct shutdown sequence, in order:
Skip step 2 and the LB routes to a pod that's already refusing — the deploy-time 502 spike. Skip step 4 and clients holding keep-alive connections get resets mid-request.
Slow start on the way in: a fresh pod has cold JIT, cold connection pools, cold L1 caches (Module 01). Full traffic share instantly → its p99 is terrible → latency-aware LBs and users both notice. Ramp weight over 30–120 s (Envoy slow_start); for JVM estates this is not optional polish, it's the difference between deploys being invisible and deploys being a sawtooth on the latency dashboard.
Connection draining for long-lived protocols: WebSockets/gRPC streams can be hours old — draining must send GOAWAY and give clients a resume path (Module 07's reconnect protocol). Bounded MAX_CONNECTION_AGE keeps the drain window finite.
STEP 05
Topologies
SKIM
Fast-track takeawayFour placements: edge (L4→L7 pair, TLS termination, WAF) · internal per-service (k8s Service = L4 kube-proxy — hence gRPC needs help: headless Service + client-side LB, or a mesh) · client-side (Spring Cloud LoadBalancer: no extra hop, per-call decisions, at the cost of LB logic in every client) · mesh sidecar/ambient (Envoy everywhere: per-call L7 for all protocols, uniform retries/outlier detection/mTLS — paid in resource overhead and operational surface, Module 09). The gRPC thread from Module 07 resolves here: the answer to "L4 balances connections" is client-side LB or mesh.
Selection heuristic: start with edge L4→L7 + platform-provided internal Services; add client-side LB precisely where per-call balancing over long-lived connections matters (gRPC-heavy paths); adopt a mesh when the uniformity of retries, mTLS, and telemetry across 200 services outweighs running the mesh — a platform-team capacity question more than a technical one.
STEP 06
Exercise: build the failure modes
MANDATORY
Fast trackStep 2 (~30 min): the black-hole demo. Watching least-connections flood a failing node rewires your instincts about "adaptive" algorithms.
1
Race the algorithms. 3 Spring Boot echo pods behind Envoy (docker-compose); one pod given a 200 ms sleep. Load-test under ROUND_ROBIN, LEAST_REQUEST; compare per-pod RPS and global p99. Reproduce the sim's result with real wires.
2
Create a black hole. Make the degraded pod return instant 500s instead of slow 200s. Watch LEAST_REQUEST send it more traffic. Fix with outlier detection (consecutive_5xx: 3, max_ejection_percent: 50) and verify ejection + re-admission in Envoy stats.
3
Make deploys invisible. Rolling restart under load (k6/wrk running). Count non-200s with naive shutdown; then implement the §04 sequence (readiness flip + preStop sleep + Spring server.shutdown=graceful) and get to zero errors. Add Envoy slow start and show the new-pod p99 sawtooth flatten.
4
Paper. For your real playback path, draw every LB hop (CDN→edge→gateway→services), label each L4/L7 and its algorithm, and mark where long-lived gRPC/WebSocket connections make L4 lumpy. That diagram is a genuine architecture-review artifact.
Self-check
Why does least-connections make a fast-failing node worse, and what's the guard?
Instant errors → near-zero outstanding requests → the algorithm reads it as the least-loaded node and concentrates traffic on it: a black hole eating a growing share of requests. Guard: passive outlier ejection on error rate — load signals and health signals must be separate inputs.
Why is a DB-pinging readiness probe an outage amplifier?
One dependency brownout makes all pods unready simultaneously; the LB removes 100% of capacity, so even requests that don't need the DB (cache hits, static paths) now fail. Readiness = own health; dependency failures degrade per-request via breakers/fallbacks, keeping partial service partial.
Deploys cause a 30 s p99 spike on new pods. Three coordinated fixes?
(1) LB slow start ramping new-pod weight over 60 s. (2) In-pod warmup before readiness: touch JIT-hot paths, prime connection pools, preload hot cache keys (Module 01 warming). (3) Deploy strategy that adds before removing (maxSurge) so warm capacity covers the ramp.
When is consistent-hash load balancing worth its imbalance risk?
When per-pod locality is worth more than even spread: L1 cache affinity (each pod's Caffeine owns a key range → much higher hit ratio), session-to-pod stickiness for stateful streams. You've re-imported Module 05's hot-shard problem into your LB — so per-pod load metrics and a salting/spill plan come with it.