What CAP actually proves (narrower than the folklore), why PACELC is the version you should design with, and how to classify every dataset on your platform into a consistency stance you can defend.
~2.5h study~1.5h exercise2 interactive simsFast track ~75 min
Fast track — know your CAP folklore?
Compresses to ~75 min. Definitions collapse; the partition simulator, PACELC, and the dataset-classification method stay mandatory — most engineers "know CAP" and still misapply it, because the folklore version is wrong.
§01 takeaway — the corrected statement of the theorem · 5 min
§02 partition sim, both modes · 15 min · MANDATORY
§03 PACELC + system mapping table · 20 min · MANDATORY
§04 classify your own datasets · 15 min · MANDATORY
Exercise part B (config-level proof) · 20 min
STEP 01
What CAP actually says
SKIM
Fast-track takeawayCAP is not "pick 2 of 3." Partitions are not optional — networks fail, so P is a fact, not a choice. The theorem's real content: while a partition is happening, a system must choose to answer either consistently (C = linearizability — refuse/redirect on the cut-off side) or available (A — every reachable node answers, allowing divergence). During normal operation the theorem says nothing — that's the gap PACELC (§03) fills. Also: CAP's C is linearizability, not ACID's C.
The folklore triangle — "Consistency, Availability, Partition tolerance: pick two" — is how CAP is taught and precisely how it's misused. The precise claims:
C — Consistency means linearizability: every read sees the most recent completed write, as if there were one copy of the data. It is not ACID consistency (constraint validity) — an unfortunate naming collision.
A — Availability means every request to any non-failed node gets a non-error response, without a guarantee it's the latest data.
P — Partition tolerance means the system keeps operating when the network drops or delays messages between nodes. In a distributed system this isn't a menu option — switches die, AZs split, GC pauses look like partitions. P is the weather, not a choice.
So the real theorem is narrow: when a partition occurs, you must pick C or A for the duration of that partition. "CA system" is a meaningful phrase only for a single node — the moment data lives on two machines connected by a network, you're choosing between CP and AP behavior during partitions.
STEP 02
Feel it: the partition simulator
MANDATORY
Fast trackRun both modes. The sim makes the abstract choice concrete: CP = someone gets an error; AP = someone gets stale data and you inherit a merge problem. There is no third button.
Two replicas of a viewer's concurrent-streams counter, one in each AZ. Cut the network, then try writes in each mode:
Partition simulator · streams:user42
AZ-1 · majority side
streams:user42 = 1
AZ-2 · minority side
streams:user42 = 1
Network healthy — writes replicate to both AZs. Cut the network, then try writing from both sides in each mode.
What you just saw, named:
CP: the minority side refuses writes (or the whole key routes to the majority). Users behind the partition see errors — reduced availability, preserved truth. Concretely how quorum systems, ZooKeeper, etcd, and single-leader DBs behave.
AP: both sides accept writes and diverge. Everyone gets an answer; on heal you hold two truths — user42 at "2 streams" in both AZs when the real total is 3, i.e. a concurrency-limit bypass, a real OTT revenue leak. Divergence must be resolved: last-write-wins (silently loses data), merge functions, or CRDTs (Module 04).
STEP 03
PACELC: the version you design with
MANDATORY
CAP is silent about the 99.99% of time when there is no partition. PACELC completes it: if Partition, choose Availability or Consistency; Else, choose Latency or Consistency. The "else" clause is the one you live with daily — synchronous replication buys consistency by paying replica round trips on every write (§Module 02: +1–2 ms cross-AZ, +100 ms cross-region).
Consistency everywhere; pay latency and error responses. Money, entitlements, inventory.
PC / EL
Fast normally, but refuses rather than diverges during partitions. Rare but real (PNUTS).
PA / EC
Consistent normally, availability-first during partitions — unusual; heal-time conflicts return.
Tap a system you actually run:
Pick a system to see its default PACELC stance — and which knobs move it.
Staff expectationKnowing that the stance is per-configuration, not per-product. "Is MongoDB CP?" is a junior question; "MongoDB with w:majority + readConcern:majority is PC/EC, but with w:1 and secondary reads you've silently bought PA/EL and its anomalies" is the staff answer. Same product, opposite guarantees, one YAML line apart.
STEP 04
Classifying your platform's datasets
MANDATORY
The method: for each dataset ask two questions — what breaks if a read is stale? and what breaks if a write is refused? Whichever answer is worse decides the stance.
OTT dataset
Stance
Reasoning — the defensible sentence
Payments / billing state
PC/EC
A refused write is retryable; a stale read double-charges. Consistency wins absolutely.
Entitlement grant (post-purchase)
PC/EC
Stale read = paying user blocked (the 428 class). Pay the latency on this one path.
Concurrent-stream limit
PC preferred
AP divergence = limit bypass = revenue leak. But total lockout during partition is worse for UX → bounded AP with conservative merge (take max, enforce on heal) is a defensible middle.
Catalog / metadata
PA/EL
Stale title art for 60 s harms nobody; an unavailable catalog is an outage.
Watch history / resume points
PA/EL
Last-write-wins per device is fine; users tolerate a 10 s-old resume point.
Recommendations
PA/EL
Inherently approximate; staleness is invisible.
Live-match scores overlay
PA/EL
Availability during the exact moment of peak partition-risk is the whole product.
Notice the shape: most of the platform is PA/EL; a thin money-and-rights core is PC/EC. Architecturally that means the consistency-critical core should be small, isolated, and never on the hot read path — which is why entitlement results are cached (PA) everywhere except the seconds after purchase (PC read-through to source).
STEP 05
Misconceptions that fail design reviews
SKIM
Fast-track takeawayFive in one breath: (1) "CA distributed system" isn't a thing. (2) CAP's C ≠ ACID's C. (3) CAP binds only during partitions — daily trade-offs are PACELC's else-clause. (4) Quorum overlap (W+R>N) alone doesn't give linearizability (Module 04 shows the counterexample). (5) The choice is per-dataset and per-operation, not per-company — the same request may take a PC path for entitlement and a PA path for catalog.
"We use an AP database, so we're AP." Stances are per-configuration and per-operation. Cassandra at QUORUM reads+writes behaves very differently from ONE.
"Partitions are rare, ignore them." GC pauses, overloaded NICs, and slow disks are indistinguishable from partitions to a timeout. Partition behavior is really timeout behavior, and timeouts happen daily.
"Quorums give us strong consistency." Overlap prevents lost acknowledged writes; it doesn't order concurrent operations. Linearizability needs more (consensus or read-repair-with-sync) — Module 04.
"Choose once for the platform." The §04 table is the refutation: one platform, seven stances.
STEP 06
Exercise: prove a stance with config
MANDATORY
Fast trackPart B only (~20–30 min): flipping one system between stances and observing the anomaly teaches more than any diagram.
A
Classify (30 min). Extend the §04 table with 5 datasets from your actual platform (DRM sessions, EPG, profiles, download rights, ad decisions). For each, write the one-sentence defense. This table is a real architecture artifact — keep it in your Obsidian vault.
B
Prove it (45 min). Docker: 3-node MongoDB replica set (or Redis master+replica). Write with w:1, read from a secondary, and capture a stale read. Switch to w:"majority" + readConcern:"majority" and show it's gone — then measure the write-latency difference between the two configs. That delta is PACELC's else-clause, in milliseconds, on your screen.
C
Partition it (stretch).docker network disconnect the minority node mid-writes. Observe CP behavior (writes error/step-down) vs what Redis async replication does instead (accepts, then discards the minority's writes on heal — data loss, quantified in the replica log).
Self-check
Why is "CA" not a real option for a distributed system?
Because P isn't chosen — networks partition whether you like it or not. A system that hasn't decided its partition behavior has decided by accident: it will do something when the switch dies, usually the worst of both (unavailable AND inconsistent).
Your team says "make the concurrency limiter strongly consistent." What's the full trade you're accepting?
Every stream start now pays a quorum/majority write (§M02: +1–2 ms cross-AZ, more cross-region), and during any partition, users on the minority side cannot start streams at all — an availability incident during exactly the network events that accompany big live events. The alternative: bounded AP with a conservative merge and post-heal enforcement. Present both; the business chooses the leak vs the lockout.
Where does CAP's guarantee stop and PACELC's begin?
CAP constrains behavior only while a partition is in progress. PACELC adds the everyday clause: absent partitions you still trade consistency against latency, because synchronous coordination costs round trips on every operation. Most systems' daily character is set by the EL/EC choice, not the P clause.
Same product, opposite stances — give a concrete example.
MongoDB: w:majority + readConcern:majority → PC/EC; w:1 + secondary reads → effectively PA/EL with read-your-writes violations. Or Redis: async replication (PA-ish, loses minority writes) vs WAIT numreplicas pushing toward EC at latency cost. The knob, not the logo, defines the system.