Topologies, the anomalies replication lag actually causes users, the consistency spectrum from linearizable to eventual, and why quorum arithmetic doesn't buy what most engineers think it does.
~3h study~2h exercise2 interactive simsFast track ~90 min
Fast track — replicated before?
Compresses to ~90 min. Topology definitions collapse; lag anomalies, the spectrum, and quorum subtleties stay mandatory — they're the parts that produce user-visible bugs that pass all unit tests.
§01 takeaway · 5 min
§02 anomalies + lag sim, in full · 25 min · MANDATORY
§03 spectrum, in full · 20 min · MANDATORY
§04 quorums incl. the counterexample · 20 min · MANDATORY
§05–§06 takeaways; Exercise step 3 only · 25 min
STEP 01
Topologies in one pass
SKIM
Fast-track takeawayThree families: single-leader (all writes → one node; simple, ordered; followers lag; failover is the hard part — split brain, lost un-replicated writes) · multi-leader (writes accepted in several regions; great for geo-latency and offline; buys permanent conflict-resolution duty) · leaderless/Dynamo-style (client/coordinator writes to N, quorum reads; no failover event, but consistency is statistical — §04). Sync replication = durability at latency cost; async = fast with a loss window; semi-sync (one sync follower) is the common production compromise. Your Postgres+Redis estate is single-leader async almost everywhere — every §02 anomaly applies to you.
Topology
Write path
Strength
You pay
Single-leader
all writes → leader → replicate
simple, totally ordered writes
read lag on followers; failover complexity (split brain, lost writes)
Multi-leader
writes in every region, async cross-sync
local write latency, region survival
concurrent-write conflicts, forever
Leaderless
write to N replicas, read from R
no failover moment; tunable
consistency is probabilistic without care (§04)
Sync vs async is orthogonal: sync = follower acks before commit (durability, +RTT per write, follower outage blocks writes) · async = fast, but leader death loses the un-shipped tail · semi-sync = exactly one sync follower as the durable copy — the pragmatic default.
STEP 02
Replication-lag anomalies
MANDATORY
Fast trackThese three anomalies are the source of the "user saved their profile and it vanished" bug class. Read fully; run the sim.
Async lag is normally milliseconds and occasionally minutes (bulk load, follower restart, network blip). "Eventually consistent" is a fine property until a user sits inside the window:
Read-your-writes violation — user updates their profile (→ leader), next page reads a follower that hasn't caught up: their edit "vanished". Fixes: read own data from the leader; or session-sticky reads; or client sends last-seen LSN/timestamp and replicas serve only if caught up.
Non-monotonic reads — successive reads hit differently-lagged replicas: a comment appears, then disappears. Time runs backwards. Fix: pin a session to one replica (consistent-hash users → replicas).
Consistent-prefix violation — causally ordered writes (question → answer) observed out of order via differently-lagged partitions. Fix: keep causal chains in one partition, or causal-consistency machinery.
Lag simulator · profile update
Leader
displayName = "Sib"
Follower lag: 800 ms
displayName = "Sib"
Write a new value, then immediately read. The load balancer alternates leader/follower — with lag > 0 and the fix off, you'll catch the vanishing write.
STEP 03
The consistency spectrum
MANDATORY
Models are contracts about what reads may observe, ordered strongest → weakest. Stronger = fewer surprises = more coordination = more latency and less availability. Tap each for its cost:
Linearizable
One-copy illusion: every read sees the latest completed write, real-time order respected. Cost: consensus or leader-reads on every op; cross-AZ RTT per write; unavailability during partitions. Reserve for: locks, leader election, uniqueness (one active stream token), payments.
Sequential
All clients see the same order of ops, but not necessarily in real time. Cost: total-order broadcast; cheaper than linearizable (no real-time bound). Rarely selected explicitly; useful for state machines.
Causal
If A could have influenced B, everyone sees A before B; concurrent ops may differ in order. Cost: version/dependency tracking (vector clocks or session tokens). The sweet spot for social-ish features: replies never precede their parent. Read-your-writes and monotonic reads are its session-level fragments.
Eventual
Absent writes, replicas converge... eventually. No promise about any individual read. Cost: nearly free — and worth exactly what it promises: nothing per-read. Fine for catalog, counters, telemetry. The design task is bounding "eventually" and alerting when lag exceeds the bound.
Staff expectationNaming the weakest sufficient model per operation — not per system. "Playback resume: eventual with per-device LWW. Entitlement read post-purchase: linearizable via leader read. Continue-watching rail: causal so episodes never reorder within a session." Choosing stronger than needed silently spends your latency and availability budgets from Modules 02–03.
STEP 04
Quorums — and what W+R>N doesn't buy
MANDATORY
Leaderless systems write to W of N replicas and read from R. If W + R > N, read and write sets must overlap, so a read touches at least one replica with the newest acknowledged write. Play with it:
Quorum explorer · N=5
Now the part that fails design reviews. Overlap ≠ linearizability:
Partial-write races: a write lands on replica 1 but not yet 2–3; a concurrent read at R=2 from {1,2} vs another from {2,3} — two readers, two answers, both "valid quorum reads". Ordering concurrent ops needs consensus, not arithmetic.
Sloppy quorums (Dynamo-style, for availability): during faults, writes accept on substitute nodes outside the home set, with hinted handoff shipping them home later. Availability goes up; the overlap guarantee quietly disappears until handoff completes.
Read repair & anti-entropy are the convergence machinery — stale replicas get fixed on read or by background sync; they set how long "eventual" lasts.
STEP 05
Conflict resolution
SKIM
Fast-track takeawayAccepting concurrent writes (multi-leader, leaderless, AP-during-partition) means owning merges. Three tools: LWW — simple and silently destroys data; clock skew decides the winner, so never use it where losing a write is losing money. Version/vector clocks — detect concurrency instead of hiding it; app decides the merge (Dynamo's shopping-cart union). CRDTs — data types whose merge is automatic and correct (G-Counter for view counts, OR-Set for my-list, LWW-Register where LWW is genuinely fine). Rule: pick the data structure that makes the conflict impossible before writing merge code.
Last-write-wins: pick by timestamp. Cheap, deterministic, and lossy — with clock skew, "last" is a lie. Acceptable for resume points; unacceptable for anything counted or owned.
Vector clocks / versions: track causality; if versions are concurrent, surface both to a merge function. Correct, but pushes complexity into the application.
CRDTs: counters, sets, registers with mathematically convergent merges. A distributed concurrent-streams counter as a PN-Counter merges cleanly after the Module 03 partition — the principled version of "take the max and reconcile."
STEP 06
Code: routing reads by consistency need
SKIM
Fast-track takeawayThe Spring pattern: a routing DataSource sends @Transactional(readOnly=true) to replicas and everything else to the leader — then the one crucial exception: paths needing read-your-writes (post-purchase entitlement) force the leader despite being reads. Skim the snippet to see where that override hooks in.
public classReplicaRoutingDataSourceextendsAbstractRoutingDataSource {
@Override protectedObjectdetermineCurrentLookupKey() {
if (ConsistencyContext.leaderRequired()) // read-your-writes paths set thisreturn"leader";
returnTransactionSynchronizationManager
.isCurrentTransactionReadOnly() ? "replica" : "leader";
}
}
// entitlement check right after purchase — a READ that must hit the leaderpublicEntitlementverifyPostPurchase(String userId, String contentId) {
try (var scope = ConsistencyContext.requireLeader()) { // ThreadLocal / ScopedValuereturn entitlementRepo.find(userId, contentId); // linearizable-enough: leader read
}
}
Wrap the target in LazyConnectionDataSourceProxy so the routing decision happens after @Transactional attributes are known. Equivalent knobs elsewhere: Mongo readPreference=primary per-operation; Redis WAIT; JDBC readOnly hints on Aurora endpoints.
STEP 07
Exercise: catch the anomaly on your own screen
MANDATORY
Fast trackStep 3 alone (~40 min) delivers the module's core scar tissue: a reproduced read-your-writes violation and its fix.
1
Stand up replication. Docker Compose: Postgres 16 primary + streaming replica (or Bitnami images). Confirm with pg_stat_replication.
2
Induce lag.recovery_min_apply_delay = '2s' on the replica — deterministic 2-second lag beats hoping for a race.
3
Reproduce & fix. Spring Boot app with the routing DataSource: POST /profile then immediate GET /profile → capture the stale read in a failing integration test. Add ConsistencyContext.requireLeader() on the read-after-write path → test green. Keep both test runs as artifacts.
4
Quorum math (paper). N=5: give (W,R) for write-latency-optimized, read-latency-optimized, and balanced; state each config's data-loss window if two replicas die.
5
Stretch — CRDT. Implement a PN-Counter in Java for concurrent streams (map of node→increments/decrements, merge = element-wise max). Unit-test that merge is commutative, associative, idempotent — the three properties that make convergence inevitable.
Self-check
User saves a setting and the next page shows the old value. Diagnose and give two fixes with their costs.
Read-your-writes violation: write hit the leader, read hit a lagging replica. Fix A: route that user's own-data reads to the leader — simple, adds leader load. Fix B: client tracks last-write LSN/timestamp; replicas serve only if caught up, else redirect — scales better, needs plumbing through every read path.
W=3, R=3, N=5 — is a read guaranteed the latest write?
The latest acknowledged write reaches at least one of the R replicas — but version-resolution must pick it, in-flight partial writes create windows where concurrent readers disagree, and sloppy quorums void the overlap. Guaranteed freshness under concurrency needs consensus or leader reads.
Why is LWW dangerous for a concurrent-streams counter but fine for resume points?
LWW drops one of two concurrent writes. For resume points the dropped write is a few seconds of playback position per device — invisible. For a counter, dropping a write means a stream isn't counted → concurrency-limit bypass → revenue. Counters need merges that preserve both updates (PN-Counter / sum-of-deltas).
Which consistency model does "continue watching must never reorder within a session" require, and what's the cheapest implementation?
Monotonic reads (a session-level fragment of causal). Cheapest: pin each session's reads to one replica via consistent hashing — no version tracking needed, survives everything except that replica's failure (then re-pin, accepting one possible regression).