The single-node truths under the distributed course: the atomic primitive (CAS) beneath every lock-free structure and half this course's patterns, the thread-model decision your Java 21 platform is facing right now (pools vs Loom vs event loops), the two laws — Little's and the Universal Scalability Law — that were silently underneath Modules 13 and 25, and the data-integrity discipline (checksums, scrubbing, end-to-end) that catches the corruption replication happily copies.
~3h study~1.5h exercise1 interactive simFast track ~75 min
Fast track — sized a thread pool with math before?
Compresses to ~75 min. The hierarchy collapses to its rule; CAS, the Loom decision, the laws calculator, and integrity stay mandatory.
§01 takeaway · 5 min
§02 CAS/ABA, in full · 15 min · MANDATORY
§03 thread models, in full · 20 min · MANDATORY
§04 calculator, both laws · 20 min · MANDATORY
§05 integrity, in full · 15 min · MANDATORY
STEP 01
The coordination hierarchy
SKIM
Fast-track takeawayShared mutable state is where concurrency bugs live, and every mechanism for taming it has a price. Fastest to slowest: immutable (nothing to coordinate) → thread-local/partitioned (nothing shared) → lock-free/CAS (~25 ns, progress guaranteed) → lock-based (25 ns uncontended, unbounded when contended) → distributed coordination (network RTT + M19's whole module). The rule: take the highest level that's correct — and notice it's the same ladder this course climbed at system scale: immutable events (M15/M18), partition-per-owner (M05), optimistic version guards (M16), locks only when demotion fails (M19's demote-the-lock tree). One design instinct, two scales — that's the "mechanical sympathy" the module title means.
STEP 02
CAS: the primitive under everything
MANDATORY
Compare-and-swap — atomically: if (*addr == expected) { *addr = new; return true } else return false — one CPU instruction, and the entire foundation of lock-free programming: read, compute, CAS, retry on failure. Java surfaces it as AtomicLong/AtomicReference, VarHandle.compareAndSet, and inside ConcurrentHashMap, lock striping, and the JVM's own biased-lock machinery.
You've been using CAS at every scale all course: Redis SET NX (M19's lock acquire) is CAS on a key; SQL UPDATE ... WHERE version = ? (M16's guard) is CAS on a row; Raft's vote-once-per-term (M30) is CAS on a ballot. The pattern — optimistic attempt, atomic verify, retry — is scale-invariant; only the latency of the attempt changes (25 ns → 0.5 ms → quorum RTT).
ABA — the classic trap: value went A→B→A between your read and your CAS; the CAS succeeds but the world it validated is gone (a freed-and-reallocated node in a lock-free stack; at system scale: a lock released and re-acquired — the M19 stale-holder in miniature). Fix is identical at every scale: pair the value with a monotonic version/stamp (AtomicStampedReference; the row's version column; the fencing token). M32 named this trick "logical clocks"; here it is at nanosecond scope.
Contention economics: under heavy contention CAS loops burn CPU retrying (and cache-line ping-pong between cores dominates — false sharing puts unrelated hot fields on one line and makes strangers contend). The high-throughput fix is reducing sharing, not smarter sharing: LongAdder beats AtomicLong for hot counters by striping across cells and summing on read — which is M05's partitioning strategy, applied to eight bytes.
STEP 03
Thread models — and the Loom decision
MANDATORY
Model
Mechanics
Fits
Thread-per-connection
one OS thread per client; ~1 MB stack each → 10k conns ≈ 10 GB of stacks
<~1k concurrent; simple internal tools
Pool + queue (classic Tomcat/Spring MVC)
fixed OS threads, work queued; a blocked thread is a lost slot
bounded concurrency; the model your M13 bulkheads assume
Event loop (Netty/WebFlux/Node)
few threads, non-blocking IO, callbacks/reactive; nothing may block the loop
C10K+: gateways, proxies, push/SSE fan-out (M07)
Virtual threads (Loom, Java 21)
thread-per-request syntax, event-loop economics: JVM parks virtual threads on block, ~KB footprint, millions cheap
IO-bound services — i.e., most of your 200
The staff-level framing of Loom: it removes the concurrency ceiling reason for reactive code — you keep debuggable, blocking-style Spring MVC and stop losing a pool slot per in-flight downstream call (the very coupling that made M12 slow-call amplification so vicious). It does not remove event loops' place at the extreme edge (per-connection memory at 500k SSE connections still matters), doesn't speed up CPU-bound work, and imports one new failure mode: pinning — a virtual thread blocking inside synchronized or native calls pins its carrier OS thread; audit hot paths for synchronized-around-IO (older JDBC pools and legacy libs are the usual suspects) before migrating.
What survives the migration unchanged: bulkheads and admission control (M13/M14) — "unlimited cheap threads" means the thread stops being the natural limiter, so without explicit semaphores/limits your service will now happily accept work far past what downstreams and its own CPU can bear (Little's Law, next section, tells you exactly how far). Loom changes the cost of concurrency, not the need to bound it.
Sizing rule for classic pools while you still have them: IO-bound ≈ cores × (1 + wait/service time); CPU-bound ≈ cores. Both are corollaries of the next section's first law.
STEP 04
Little's Law & the Universal Scalability Law
MANDATORY
Little's Law: L = λ × W — in-flight work = arrival rate × time-in-system. No assumptions, no distribution fine print; it's conservation. It was silently under M13 (pool size must cover λ×W or the queue grows), M12 (a timeout stretch multiplies W, thus L — why slow is worse than down), M02 (throughput ceilings), and M25 (pods = λ×W ÷ per-pod concurrency). USL adds what happens when you scale out: C(N) = N / (1 + α(N−1) + βN(N−1)) — α is contention (the serialized fraction: locks, single writers, one hot partition), β is coherence (crosstalk: cache invalidation, consensus rounds, gossip — everyone syncing with everyone). α caps your speedup (Amdahl); β makes throughput decrease past a peak N — retrograde scaling, where adding nodes makes it slower.
The laws, live
λ arrival (rps)
1000
W latency (ms)
200
α contention
5%
β coherence
0.5%
148121620242832
Reading the bars: amber = peak useful cluster size; red = retrograde territory where node N+1 subtracts throughput. Set β to 0 and watch the curve flatten toward Amdahl's ceiling (1/α) but never fall; any β > 0 guarantees a peak. Then recognize your systems: a hot partition (M05) is α; consensus on the write path (M30's warning) is β; cache-invalidation chatter and full-mesh gossip are β — and "we added nodes and it got slower" stops being a mystery and becomes a measurement of β.
Little's Law as the red-flag detector (use in every review): proposed λ × expected W = required concurrency — compare against pool sizes, connection limits, per-pod capacity. 2,000 rps × 1.5 s tail = 3,000 in-flight: if the fleet holds 800 slots, the design collapses at the whiteboard instead of at kickoff. Same math sizes M14's shed point and explains M12's amplification arithmetic in one line.
STEP 05
Data integrity: corruption is a workload
MANDATORY
Silent corruption happens at scale — bit rot, controller bugs, cosmic-ray flips, buggy NICs. At hundreds of services × petabytes of segments, "rare" multiplies into "weekly." And note what your redundancy does about it: nothing — replication (M04) faithfully copies corrupt bytes, and backups retain them. Detection must be explicit: checksum at rest (per block/object — ZFS, S3 ETags, Kafka per-record CRCs), in flight (per message), and verify on read — a mismatch triggers read-from-replica + repair, turning corruption from wrong-answers into a handled fault class.
Scrubbing: a background pass re-reading cold data, verifying checksums, and repairing from good replicas — because unread data rots invisibly until the day all copies have independently rotted. ZFS scrubs, S3 does it internally, HDFS block scanners — and your own long-retention stores (the content archive, the event lake) need an answer to "who re-reads this, ever?" (It's also M04's anti-entropy with a checksum instead of a vector — same epidemic repair idea.)
The end-to-end argument (Saltzer/Reed/Clark, the most cited principle in systems): reliability features in lower layers are optimizations only — TCP checksums, disk ECC, and TLS integrity each cover their hop, but nothing below the application can catch corruption that happens between layers (in your process memory, during transcode, in a buggy serializer). Only an application-level check spanning source to destination actually verifies the thing you care about: checksum the segment at packaging, verify after the CDN hop; checksum before compression, verify after decompression. This is also why M16 put idempotency at the application layer and M24 measured at the edge — the end-to-end argument is the same reasoning applied to delivery and to observability.
CRC vs MAC (the M20 boundary): checksums/CRCs detect accidents and are trivially forgeable; against adversaries you need MACs/signatures (M20). Same field in the packet, different threat models — don't let a design review conflate them.
STEP 06
Exercise
MANDATORY
Fast trackStep 2 (~30 min): the Loom pinning audit is the one with immediate production value for your estate.
1
Feel the hierarchy. JMH benchmark a hot counter four ways: synchronized, AtomicLong, LongAdder, thread-local-with-sum — at 1, 8, and 32 threads. Plot ops/sec vs threads; you'll watch §01's ladder and a live USL curve (the synchronized version goes retrograde) in one chart.
2
Loom pilot + pinning audit. Take one IO-bound service: enable virtual threads (spring.threads.virtual.enabled=true on Boot 3.2+), load-test before/after at high concurrency, and run with -Djdk.tracePinnedThreads=full to catch synchronized-around-IO. Write the two-paragraph migration recommendation for your architecture group: which service classes, what to audit, what stays reactive.
3
Apply the laws to real numbers. Pull one hot service's peak λ and p99 W from M24 dashboards; compute L and compare to its actual pool/semaphore limits — headroom or hidden collapse? Then fit USL: throughput at 2, 4, 8 pods from a load test → estimate α and β → predicted useful ceiling. One page, real numbers.
4
Integrity walk. Trace one content segment source→CDN→player and one event producer→Kafka→consumer→store: where do checksums exist, where are they *verified*, and what's the first end-to-end check in each path? Any long-retention store without a scrub answer gets a ticket.
Self-check
Connect ABA, the M19 stale-holder, and the fix they share.
Both are validate-then-act races where the validated state was recycled in between: ABA — the memory word returned to value A via B, so CAS's equality check passes against a different world; stale-holder — the lock was released and re-acquired by another process, so "I hold the lock" is true-looking but stale. Shared fix: make state non-recyclable by pairing it with a monotonic stamp (AtomicStampedReference / version column / fencing token) so equality of value can't masquerade as continuity of state. It's one bug and one fix, at nanosecond and network scale.
Why doesn't Loom make bulkheads obsolete — argue from Little's Law.
Pre-Loom, the thread pool was an accidental admission controller: L couldn't exceed pool size, so excess arrivals queued visibly. Loom makes threads ~free, so L = λ×W is now unbounded by default — during a downstream brownout W stretches and in-flight work balloons (memory, connections, downstream pressure) with no natural ceiling. Bulkheads/semaphores (M13) must become *explicit*: you're no longer limited by thread cost, so you must limit by policy. Loom removed the accident, not the need.
A service scales linearly to 8 pods, plateaus at 12, and gets *slower* at 20. Diagnose with USL and name two likely culprits per parameter.
Plateau = α (contention: a serialized fraction — single hot partition M05, one shared DB connection pool, a global lock/single-writer table); decline past a peak = β > 0 (coherence: work that grows with N² — cache-invalidation chatter between pods, full-mesh health gossip, consensus or leader coordination on the request path M30). Fit the two runs to C(N) to quantify, then attack α by partitioning the serialized thing and β by removing pairwise crosstalk (hierarchies, batching, taking consensus off the hot path). Adding pods past the peak is paying to be slower.
TCP has checksums and you use TLS everywhere — make the case that a transcoded video segment still needs an application-level hash.
End-to-end argument: TCP's checksum covers the wire hop (and is weak — 16-bit), TLS covers its session — but corruption in the transcoder's memory, a buggy serializer, a bad disk write at the origin, or a caching layer's truncation happens *between* protected hops, and every lower layer will then faithfully protect the corrupt bytes onward. Only a hash computed at packaging and verified at (or near) the player spans the actual journey. Lower layers optimize by catching most errors early; they cannot provide the guarantee.