System Design Masterclass · Module 04 / Week 1

Replication & Consistency Models

Topologies, the anomalies replication lag actually causes users, the consistency spectrum from linearizable to eventual, and why quorum arithmetic doesn't buy what most engineers think it does.

~3h study~2h exercise2 interactive simsFast track ~90 min

Fast track — replicated before?

Compresses to ~90 min. Topology definitions collapse; lag anomalies, the spectrum, and quorum subtleties stay mandatory — they're the parts that produce user-visible bugs that pass all unit tests.

  1. §01 takeaway · 5 min
  2. §02 anomalies + lag sim, in full · 25 min · MANDATORY
  3. §03 spectrum, in full · 20 min · MANDATORY
  4. §04 quorums incl. the counterexample · 20 min · MANDATORY
  5. §05–§06 takeaways; Exercise step 3 only · 25 min
STEP 01

Topologies in one pass

SKIM
Fast-track takeawayThree families: single-leader (all writes → one node; simple, ordered; followers lag; failover is the hard part — split brain, lost un-replicated writes) · multi-leader (writes accepted in several regions; great for geo-latency and offline; buys permanent conflict-resolution duty) · leaderless/Dynamo-style (client/coordinator writes to N, quorum reads; no failover event, but consistency is statistical — §04). Sync replication = durability at latency cost; async = fast with a loss window; semi-sync (one sync follower) is the common production compromise. Your Postgres+Redis estate is single-leader async almost everywhere — every §02 anomaly applies to you.
TopologyWrite pathStrengthYou pay
Single-leaderall writes → leader → replicatesimple, totally ordered writesread lag on followers; failover complexity (split brain, lost writes)
Multi-leaderwrites in every region, async cross-synclocal write latency, region survivalconcurrent-write conflicts, forever
Leaderlesswrite to N replicas, read from Rno failover moment; tunableconsistency is probabilistic without care (§04)

Sync vs async is orthogonal: sync = follower acks before commit (durability, +RTT per write, follower outage blocks writes) · async = fast, but leader death loses the un-shipped tail · semi-sync = exactly one sync follower as the durable copy — the pragmatic default.

STEP 02

Replication-lag anomalies

MANDATORY

Fast trackThese three anomalies are the source of the "user saved their profile and it vanished" bug class. Read fully; run the sim.

Async lag is normally milliseconds and occasionally minutes (bulk load, follower restart, network blip). "Eventually consistent" is a fine property until a user sits inside the window:

Lag simulator · profile update

Leader

displayName = "Sib"

Follower lag: 800 ms

displayName = "Sib"
Write a new value, then immediately read. The load balancer alternates leader/follower — with lag > 0 and the fix off, you'll catch the vanishing write.
STEP 03

The consistency spectrum

MANDATORY

Models are contracts about what reads may observe, ordered strongest → weakest. Stronger = fewer surprises = more coordination = more latency and less availability. Tap each for its cost:

Linearizable
One-copy illusion: every read sees the latest completed write, real-time order respected.
Sequential
All clients see the same order of ops, but not necessarily in real time.
Causal
If A could have influenced B, everyone sees A before B; concurrent ops may differ in order.
Eventual
Absent writes, replicas converge... eventually. No promise about any individual read.
Staff expectationNaming the weakest sufficient model per operation — not per system. "Playback resume: eventual with per-device LWW. Entitlement read post-purchase: linearizable via leader read. Continue-watching rail: causal so episodes never reorder within a session." Choosing stronger than needed silently spends your latency and availability budgets from Modules 02–03.
STEP 04

Quorums — and what W+R>N doesn't buy

MANDATORY

Leaderless systems write to W of N replicas and read from R. If W + R > N, read and write sets must overlap, so a read touches at least one replica with the newest acknowledged write. Play with it:

Quorum explorer · N=5

Now the part that fails design reviews. Overlap ≠ linearizability:

STEP 05

Conflict resolution

SKIM
Fast-track takeawayAccepting concurrent writes (multi-leader, leaderless, AP-during-partition) means owning merges. Three tools: LWW — simple and silently destroys data; clock skew decides the winner, so never use it where losing a write is losing money. Version/vector clocks — detect concurrency instead of hiding it; app decides the merge (Dynamo's shopping-cart union). CRDTs — data types whose merge is automatic and correct (G-Counter for view counts, OR-Set for my-list, LWW-Register where LWW is genuinely fine). Rule: pick the data structure that makes the conflict impossible before writing merge code.
STEP 06

Code: routing reads by consistency need

SKIM
Fast-track takeawayThe Spring pattern: a routing DataSource sends @Transactional(readOnly=true) to replicas and everything else to the leader — then the one crucial exception: paths needing read-your-writes (post-purchase entitlement) force the leader despite being reads. Skim the snippet to see where that override hooks in.
public class ReplicaRoutingDataSource extends AbstractRoutingDataSource {
  @Override protected Object determineCurrentLookupKey() {
    if (ConsistencyContext.leaderRequired())        // read-your-writes paths set this
      return "leader";
    return TransactionSynchronizationManager
        .isCurrentTransactionReadOnly() ? "replica" : "leader";
  }
}

// entitlement check right after purchase — a READ that must hit the leader
public Entitlement verifyPostPurchase(String userId, String contentId) {
  try (var scope = ConsistencyContext.requireLeader()) {  // ThreadLocal / ScopedValue
    return entitlementRepo.find(userId, contentId);       // linearizable-enough: leader read
  }
}

Wrap the target in LazyConnectionDataSourceProxy so the routing decision happens after @Transactional attributes are known. Equivalent knobs elsewhere: Mongo readPreference=primary per-operation; Redis WAIT; JDBC readOnly hints on Aurora endpoints.

STEP 07

Exercise: catch the anomaly on your own screen

MANDATORY

Fast trackStep 3 alone (~40 min) delivers the module's core scar tissue: a reproduced read-your-writes violation and its fix.

1
Stand up replication. Docker Compose: Postgres 16 primary + streaming replica (or Bitnami images). Confirm with pg_stat_replication.
2
Induce lag. recovery_min_apply_delay = '2s' on the replica — deterministic 2-second lag beats hoping for a race.
3
Reproduce & fix. Spring Boot app with the routing DataSource: POST /profile then immediate GET /profile → capture the stale read in a failing integration test. Add ConsistencyContext.requireLeader() on the read-after-write path → test green. Keep both test runs as artifacts.
4
Quorum math (paper). N=5: give (W,R) for write-latency-optimized, read-latency-optimized, and balanced; state each config's data-loss window if two replicas die.
5
Stretch — CRDT. Implement a PN-Counter in Java for concurrent streams (map of node→increments/decrements, merge = element-wise max). Unit-test that merge is commutative, associative, idempotent — the three properties that make convergence inevitable.

Self-check

User saves a setting and the next page shows the old value. Diagnose and give two fixes with their costs.
Read-your-writes violation: write hit the leader, read hit a lagging replica. Fix A: route that user's own-data reads to the leader — simple, adds leader load. Fix B: client tracks last-write LSN/timestamp; replicas serve only if caught up, else redirect — scales better, needs plumbing through every read path.
W=3, R=3, N=5 — is a read guaranteed the latest write?
The latest acknowledged write reaches at least one of the R replicas — but version-resolution must pick it, in-flight partial writes create windows where concurrent readers disagree, and sloppy quorums void the overlap. Guaranteed freshness under concurrency needs consensus or leader reads.
Why is LWW dangerous for a concurrent-streams counter but fine for resume points?
LWW drops one of two concurrent writes. For resume points the dropped write is a few seconds of playback position per device — invisible. For a counter, dropping a write means a stream isn't counted → concurrency-limit bypass → revenue. Counters need merges that preserve both updates (PN-Counter / sum-of-deltas).
Which consistency model does "continue watching must never reorder within a session" require, and what's the cheapest implementation?
Monotonic reads (a session-level fragment of causal). Cheapest: pin each session's reads to one replica via consistent hashing — no version tracking needed, survives everything except that replica's failure (then re-pin, accepting one possible regression).