ACID stops at the service boundary. Why 2PC isn't the answer, sagas as sequences of local transactions with compensations, choreography vs orchestration for real team topologies, and the isolation anomalies sagas re-introduce.
~3h study~2h exercise1 interactive simFast track ~80 min
Fast track — shipped a saga before?
Compresses to ~80 min. 2PC history collapses; saga mechanics, the topology decision, and the isolation anomalies stay mandatory — the anomalies are what production sagas actually get wrong.
§01 takeaway · 5 min
§02 mechanics + sim, both runs · 20 min · MANDATORY
§03 topology decision, in full · 15 min · MANDATORY
§04 anomalies, in full · 20 min · MANDATORY
§05 design rules · 10 min
STEP 01
Why not 2PC
SKIM
Fast-track takeawayTwo-phase commit gives cross-system atomicity by having a coordinator collect prepare-votes then commit — and pays with blocking: between prepare and commit, every participant holds locks waiting; a crashed coordinator leaves them locked indefinitely (in-doubt transactions needing manual resolution). Add PACELC: 2PC is maximal EC — multiple round trips of coordination per transaction, availability hostage to the slowest/deadest participant. Across microservices (heterogeneous stores, Redis and Rabbit and vendors that don't speak XA, teams that deploy independently) it's operationally dead. The alternative isn't "no transactions" — it's atomicity replaced by guaranteed eventual completion: the saga.
STEP 02
Saga mechanics
MANDATORY
A saga = a sequence of local transactions, each atomic in its own service, each publishing its completion. On failure at step N, run compensations for steps N-1…1 in reverse — semantic undo, not rollback. Watch a subscription purchase:
Purchase saga · orchestrated
T1
Order service · create order (PENDING)
—
T2
Payment service · authorize & capture
—
T3
Entitlement service · grant subscription
—
T4
Notification · send confirmation
—
Each step is a local ACID transaction. Between steps, the system is in a visible intermediate state — hold that thought for §04.
Compensations are business operations: refund (not "unsee the charge"), revoke entitlement, cancel order — each with its own failure modes, each needing the same retry+idempotency discipline as forward steps (M12/M16: sagaId:stepName is the idempotency key).
Classify steps:compensatable steps may precede the pivot (the go/no-go step — payment capture); after the pivot come only retriable steps that must eventually succeed (entitlement grant retries forever; it cannot "fail" the saga post-payment). Ordering steps as compensatable → pivot → retriable minimizes compensation surface — put the hardest-to-undo step last before the pivot, cheap-to-retry steps after.
Compensation can itself fail — refund API down. Retriable-forever + DLQ + human escalation queue (M15's discipline); a saga engine's job is largely bookkeeping these stuck states visibly.
Every step transition rides the outbox (M18): "commit local transaction + publish event" is a dual write; the saga's reliability is only as good as that seam.
STEP 03
Choreography vs orchestration
MANDATORY
Choreography (events, no conductor)
Orchestration (explicit coordinator)
Flow lives in
each service's subscriptions — emergent
one state machine, readable end-to-end
Coupling
loose; adding consumers is free
orchestrator knows every participant
Debuggability
"where is order 4711?" = archaeology across N services' logs
centralized: timeouts, retries, compensation sequencing in one place
Risk profile
invisible complexity — the flow exists nowhere
the orchestrator becoming a god-service with business logic pulled in
Staff expectationThe working rule: 2–3 steps, single team, no pivot ambiguity → choreography; anything with money, ≥4 steps, or cross-team ownership → orchestration — because at 3 AM the question is always "what state is this purchase in and what happens next?", and only an orchestrator's state table answers it cheaply. Guard the orchestrator's scope: it sequences and books state; it must not accrete the participants' business rules (the fat-gateway disease, M09, saga edition). Tooling spectrum: hand-rolled state table → Spring Statemachine → Temporal/Camunda-class engines; adopt an engine when saga count, not saga complexity, justifies the platform.
STEP 04
The isolation you gave up
MANDATORY
Fast trackThis is the section production sagas fail on. ACID's I is gone: intermediate states are visible and interleavable. The anomalies and their countermeasures:
Lost update: saga A (purchase) and saga B (plan change) interleave writes to the same subscription — B overwrites A's step. Countermeasure: semantic locking — the record carries a state flag (PENDING_PURCHASE) that other sagas respect (wait/abort), set in T1, cleared at saga end. It's an application-level lock with all of M19's lessons attached (lease it, or a dead saga wedges the record).
Dirty read: another flow reads entitlement between grant and a later compensation — user starts playback on a subscription about to be revoked. Countermeasures: order steps so the pivot precedes visible effects (grant only after capture); or mark intermediate state explicitly (entitlement.status=PROVISIONAL) and let readers decide (playback may accept provisional; invoicing may not).
Non-repeatable reads across steps: T3 re-reads data T1 validated — price changed mid-saga. Countermeasure: pass values through the saga context, don't re-read; the saga executes against its snapshot.
Commutative-update design shrinks the exposure: steps that add/remove (reserve one concurrent-stream slot) interleave more safely than steps that set totals — M16's absolute-vs-relative logic, inverted deliberately where interleaving is the norm.
STEP 05
Design rules, compressed
SKIM
Fast-track takeawayThe checklist: (1) every step + compensation idempotent with sagaId:step keys · (2) step order = compensatable → pivot → retriable · (3) every transition through the outbox · (4) saga state persisted and queryable (the 3 AM table) · (5) timeouts per step with explicit timeout-actions (a saga that can wait forever is a leak) · (6) semantic locks leased, never eternal · (7) end-to-end saga metrics: started/completed/compensated/stuck, age of oldest in-flight · (8) before building any saga, try to shrink it — often re-drawing service boundaries so the invariant lives in ONE service (single ACID transaction) beats coordinating two. The best saga is the one you redesigned away.
STEP 06
Exercise
MANDATORY
Fast trackSteps 1–2 (~60 min): building the orchestrator and then breaking it mid-saga is the module.
1
Build the purchase saga. Three Spring services (order, payment-stub, entitlement) + an orchestrator with a saga_state table, steps wired through RabbitMQ, every command carrying sagaId:step idempotency keys (M16's pattern). Happy path green.
2
Break it everywhere. (a) Payment fails → verify compensation order and final CANCELLED state. (b) Kill the orchestrator between T2 and T3; restart; verify resume from state table. (c) Redeliver T3's command twice; verify one grant. (d) Make the refund fail 3× then succeed; watch the retry ladder + stuck-saga alert fire.
3
Reproduce an anomaly. Run a concurrent "cancel subscription" flow against an in-flight purchase saga; capture the lost-update. Fix with a semantic lock (status flag + lease) and show the second flow now waits/aborts cleanly.
4
Paper. Take one real multi-service flow from your platform. First, argue whether a boundary re-draw could make it a single local transaction. If not: step table with compensations, pivot marked, isolation anomalies listed with countermeasures, topology chosen with the §03 rule.
Self-check
What exactly does a saga guarantee that 2PC guaranteed differently?
2PC: atomic visibility — all-or-nothing at one instant, bought with blocking and coordinated availability. Saga: guaranteed eventual completion in one of two terminal states (all steps done, or all executed steps compensated) — no atomicity, no isolation, full availability of participants in between. You trade "never see partial" for "never stuck holding locks"; §04 is the bill for that trade.
Why must post-pivot steps be retriable-forever rather than failable?
The pivot committed the business outcome (money captured). A post-pivot "failure" would require compensating the pivot — refunding a purchase the user legitimately made — because a notification email bounced. Post-pivot steps therefore retry until success (with DLQ + human escalation as the asymptote); anything that can legitimately fail belongs before the pivot.
Order 4711 is "stuck" — walk the diagnosis under orchestration vs choreography.
Orchestration: one query on saga_state → current step, attempts, last error, age; act on that step. Choreography: no single truth — grep order service (emitted?), broker (delivered? DLQ?), payment service (consumed? crashed?), entitlement (waiting on an event that never came?); the flow must be reconstructed from N partial views. That asymmetry, at incident frequency × saga count, is the strongest orchestration argument.
Why does a semantic lock need a lease, and what breaks without one?
The lock is cleared by the saga's completion — and sagas die (orchestrator crash before recovery, poison state). Without an expiry, the record stays PENDING_PURCHASE forever: every future saga on that subscription waits/aborts eternally — a wedged customer. Lease + expiry-with-reconciliation (check the actual saga state, then release or resume) is M19's lock discipline applied at the business layer.