A lock with a timeout can expire while you think you hold it — which means a distributed lock alone can never protect correctness. Redis locks and their honest limits, fencing tokens, leases, leader election, and when to reach for a real coordination service.
~2.5h study~1.5h exercise1 interactive simFast track ~70 min
Fast track — burned by a lock before?
Compresses to ~70 min. Motivation and Redis mechanics collapse; the stale-holder sim with fencing tokens and the efficiency-vs-correctness framing stay mandatory — they're the module.
§01–§02 takeaways · 10 min
§03 stale holder + fencing, in full · 25 min · MANDATORY
§04 leases & election takeaway · 10 min
§05 decision guide, in full · 10 min · MANDATORY
Exercise step 1 · 15 min
STEP 01
Efficiency locks vs correctness locks
SKIM
Fast-track takeawayThe distinction that decides everything downstream: an efficiency lock prevents wasted duplicate work (two pods refreshing the same cache — M01's single-flight; two schedulers running the same nightly job). If it fails, you pay double compute. A correctness lock guards an invariant (don't double-charge, don't exceed 2 concurrent streams). If it fails, you corrupt data. Efficiency locks: Redis SET NX is perfect. Correctness: no lock alone suffices (§03) — you need fencing at the resource, or better, no lock at all (DB constraints, M16's atomic ops). Most "we need a distributed lock" conversations dissolve when you ask which kind it is.
STEP 02
Redis locks done properly (and their ceiling)
SKIM
Fast-track takeawayThe correct single-Redis lock: SET lock:{key} {token} NX PX 30000 — atomic acquire with TTL (no expiry = a crash wedges the key forever), unique holder token, and release via Lua compare-and-delete (never bare DEL — you'd release someone else's lock after your TTL slipped). Extension via watchdog for long work (compare-and-extend, same Lua discipline). Ceilings: async replication means failover can forget the lock (two holders — M04's lost-write, lock edition); Redlock (quorum across N independent Redis nodes) hardens against single-node loss but remains debated for correctness under clock skew/pauses — the field's practical consensus: fine for efficiency locks, don't bet invariants on it. For invariants, §03. Redisson packages all of this for the JVM (watchdog included) — use it rather than hand-rolling.
STEP 03
The stale holder — and fencing tokens
MANDATORY
Fast trackThis is the canonical failure (Kleppmann's example) every senior engineer must be able to draw from memory. Run both modes.
Client A acquires the lock (TTL 10 s), then stalls — a long GC pause, VM migration, slow IO. The TTL expires; B legitimately acquires; A wakes with no idea time passed and writes. Two writers, lock "working as designed":
GC pause outlives the TTL
Client A
Client B
Storage
A acquires, pauses through its own TTL, and wakes up still believing it holds the lock. No heartbeat, watchdog, or TTL tuning fixes this — the pause is undetectable from inside.
Fencing token = the lock service hands out a monotonically increasing number with each grant (A gets 33, B gets 34). Every write carries its token; the storage/resource rejects tokens older than the highest seen. A's late write (33 < 34) bounces. The correctness moved from the lock to the resource — which is the honest place, because only the resource sees every write.
The resource must be able to check — a DB (token column + WHERE token >= ? guard or version CAS), an object store with conditional puts. A dumb resource (fire-and-forget HTTP to a vendor) can't be fenced — then the answer is making the operation idempotent/CAS-shaped (M16) or accepting the efficiency-lock framing.
Recognize the family: fencing tokens ≈ M16's version guards ≈ M17's leased semantic locks ≈ optimistic-locking CAS. One idea — attach a monotonic epoch to authority, verify at the point of effect — appearing at four layers. This is also exactly how Kafka fences zombie producers (epochs) and how k8s leader election works (resourceVersion CAS on a lease object).
STEP 04
Leases, leader election, coordination services
SKIM
Fast-track takeawayA lease is a lock that tells the truth: time-bounded authority, renewed by heartbeat, with the holder obligated to check remaining validity before acting and to stop when it can't renew — the basis of k8s Lease-object leader election (CAS on renewal, the API server's Raft doing the real work). Leader election = a lease on "the right to run the singleton" (schedulers, your outbox relay, partition assigners): the elected leader still carries its epoch/token into every effect (§03), because two leaders during a transition is a when, not an if. ZooKeeper/etcd are CP coordination services (M03): linearizable CAS, watches, sessions — the correct substrate when you genuinely need coordination primitives; in a k8s estate, the Lease API gives you etcd's guarantees without operating etcd yourself. For JVM apps: Spring Integration's LockRegistry/leader abstractions, or ShedLock for the common "run this @Scheduled job on exactly one pod" case — which, note, is an efficiency problem and should be treated as one.
STEP 05
The decision guide
MANDATORY
"We need a distributed lock"
│
├─ Is there an invariant that breaks if two proceed?
│ NO → efficiency lock: Redis SET NX PX (+ Redisson watchdog).
│ Accept rare double-work; measure it; done.
│ YES ↓
├─ Can the invariant live in ONE system's atomic op instead?
│ DB unique constraint / conditional update / M16 upsert /
│ Redis Lua counter (your concurrent-streams limit!)
│ → USE THAT. A constraint is a lock you can't hold wrong.
│ Can't ↓
├─ Can the resource verify fencing tokens?
│ YES → lock (Redis/etcd/k8s Lease) + monotonic token
│ + resource-side rejection. §03.
│ NO → redesign the resource interaction to be
│ idempotent / CAS-shaped (M16), or route all
│ writes through one owner (leader per key —
│ partition ownership, M05 thinking).
└─ Singleton process (scheduler, relay)?
→ leader election via k8s Lease / ShedLock,
epoch carried into effects, split-brain assumed.
Staff expectationThe instinct to demote locks: your concurrent-streams limit is not a lock problem — it's an atomic counter with a ceiling (Redis Lua: check-and-incr, M14's bucket shape) or a DB constraint on active sessions. Playback heartbeats make it self-healing (a crashed session's slot frees by TTL — a lease, naturally). The review question "which atomic operation replaces this lock?" kills 80% of proposed distributed locks, and every kill removes a coordination dependency from the serving path.
STEP 06
Exercise
MANDATORY
Fast trackStep 1 (~30 min): manufacturing the stale-holder write yourself, then bouncing it with a token, is the durable version of §03.
1
Manufacture the stale holder. Two JVM clients + Redis lock (TTL 5 s) + a Postgres "protected" table. Client A: acquire, Thread.sleep(8000) (the "GC pause"), write. Client B: acquire at t=6 s, write. Observe A's late write landing. Then add fencing: lock grants INCR lock:token; table gets a fence_token column; writes go UPDATE ... WHERE fence_token < :mine — rerun and watch A's write reject with 0 rows updated.
2
Break bare DEL. Implement release as plain DEL, engineer the TTL-slip so A deletes B's lock; then fix with the compare-and-delete Lua. Two runs, one log line of difference, whole incidents of consequence.
3
Demote a lock. Implement the concurrent-streams limit twice: (a) lock-per-user around read-count-write; (b) one Redis Lua check-and-incr with per-session TTL slots. Load-test both under contention; compare p99, correctness under crash (kill a client mid-hold), and code size. Write the two-sentence conclusion.
4
Paper. Inventory every distributed lock in your estate (grep Redisson/ShedLock/SETNX usage): classify efficiency vs correctness; for each correctness lock, name the fencing mechanism or the atomic-op replacement. Unfenced correctness locks go on the risk register.
Self-check
Why can't heartbeats/watchdogs fix the stale-holder problem?
The watchdog runs in the same paused process — during the GC pause it isn't extending anything, and after waking, both the app thread and watchdog believe no time passed until they next check. The failure is that the holder's belief about time diverges from the world's; only verification at the point of effect (fencing at the resource) closes that gap, because the resource's view of order is the one that matters.
Your team proposes Redlock for payment deduplication. Response?
Wrong tool twice over: (1) it's a correctness invariant, and Redlock's guarantees under pauses/clock skew are exactly the debated territory — the safe posture is efficiency-only; (2) payment dedup has a natural atomic home: the M16 idempotency insert (unique key, ON CONFLICT) in the payments DB — one system, one atomic op, no coordination service on the money path. Propose that; keep Redlock off the design.
Leader election gave you exactly one outbox relay — yet events got published twice during a deploy. Explain calmly.
Two leaders overlapped during lease transition (old leader's last batch in flight while the new one started) — split-brain-for-seconds is inherent to lease-based election. And it was fine: the relay is at-least-once by design (M18), consumers are idempotent (M16). The lesson: election bounds duplication for efficiency; the correctness was always downstream. If it *hadn't* been fine, the fix is epoch-fenced publishing, not "better" election.
State the one-sentence version of this module.
A distributed lock is a performance optimization wearing a correctness costume — real safety comes from an atomic operation or a fencing check at the resource that every write must pass, and the best lock is the one you replaced with a constraint.