System Design Masterclass · Module 18 / Week 3

Outbox, CDC & Event Sourcing

The dual-write problem is the original sin of event-driven estates. The transactional outbox that fixes it, change-data-capture with Debezium, and event sourcing — including the honest account of when it isn't worth its cost.

~2.5h study~2h exercise1 interactive simFast track ~75 min

Fast track — outbox in prod already?

Compresses to ~75 min. The dual-write demo stays (it's 5 minutes and foundational); outbox design details and the event-sourcing honesty check stay mandatory; CDC mechanics collapse.

  1. §01 sim, both runs · 10 min · MANDATORY
  2. §02 outbox design, in full · 20 min · MANDATORY
  3. §03 CDC takeaway · 10 min
  4. §04 event sourcing, in full · 20 min · MANDATORY
  5. Exercise step 1 · 15 min
STEP 01

The dual-write problem

MANDATORY

"Save to the DB and publish to RabbitMQ" is two systems with no shared transaction. Whatever order you pick, the crash between them corrupts one side — Module 01's invalidation race was one instance; here's the general disease:

Grant entitlement + publish EntitlementGranted

Database

Broker → consumers

The service will crash between the DB commit and the publish. Compare outcomes.

Both orderings lose: DB-then-publish + crash = state changed, world never told (downstream caches, notifications, analytics silently diverge — undetectably). Publish-then-DB + crash = world told about state that doesn't exist (consumers act on a grant that was never made). Wrapping both in @Transactional is a lie — the rabbitTemplate call isn't in the DB transaction.

STEP 02

The transactional outbox

MANDATORY

Make the event a row: write the business change and the event into the same database, same ACID transaction. A separate relay reads the outbox table and publishes. The dual write becomes single-write + asynchronous delivery of something durably recorded:

@Transactional
public void grant(String userId, String planId) {
  entitlementRepo.save(new Entitlement(userId, planId, ACTIVE));
  outboxRepo.save(OutboxEvent.of(               // SAME transaction — atomic with the state change
      "EntitlementGranted", userId,             // aggregateId → partition key downstream (M15 §04)
      json(userId, planId), UUID.randomUUID())); // eventId → consumer idempotency key (M16)
}   // commit: either both rows exist or neither — the race is gone by construction

Design decisions that matter:

STEP 03

CDC: the database as event source

SKIM
Fast-track takeawayCDC (Debezium-class) tails the DB's replication log (Postgres logical decoding / MySQL binlog) and emits every committed change to Kafka — the same stream replication uses (M04), productized. Two uses: (1) the outbox relay done properly — Debezium's outbox router reads only outbox rows, giving purpose-built events; (2) raw table CDC for integrating systems you can't change — powerful, but it publishes your schema as your contract: every column rename becomes a breaking event change (the anti-corruption layer between raw CDC and consumers is not optional). Ops notes: replication slots must be monitored (an abandoned slot pins WAL and fills your disk — a classic self-inflicted DB outage), snapshots for bootstrapping, and exactly-once only within Kafka (M15 §02 scope rules).
STEP 04

Event sourcing — and when it isn't worth it

MANDATORY

Event sourcing inverts storage: the event log is the truth; current state is a fold over events (state = replay(events)), cached as snapshots. Distinct from "using events" (outbox publishes events about state) — here there is no state table to diverge from.

Staff expectationBeing able to place the three patterns on one line: outbox = reliable events about state (adopt by default, everywhere you publish) · CDC = the transport that industrializes it + legacy integration (adopt per pipeline) · event sourcing = a storage philosophy (adopt per aggregate, rarely, with the cost list above read aloud). Estates get in trouble by treating these as a maturity ladder to climb rather than three tools with three grips.
STEP 05

Exercise

MANDATORY

Fast trackStep 1 (~35 min): the kill-test on your own outbox is the proof that matters.

1
Build + kill-test the outbox. Entitlement service with outbox table + polling relay (SKIP LOCKED batches → RabbitMQ). Kill-test A: crash between business write and... there is no between — demonstrate by killing during the transaction (nothing committed) and after (both committed, relay delivers). Kill-test B: crash the relay after publish, before mark → verify redelivery and that your M16 consumer absorbs it. Add the oldest-unpublished-age metric + alert.
2
Graduate to CDC. Debezium (docker) with the outbox event router on the same table; retire the poller. Measure end-to-end latency both ways. Break it educationally: stop the connector for an hour, watch WAL/slot growth, write the runbook line that prevents the disk-full incident.
3
Event-source one aggregate. Subscription as an event stream (Created, Renewed, PlanChanged, Cancelled) in a Postgres events table: fold to state, snapshot every N, one projection ("active subscriptions") that rebuilds from scratch. Then the real lesson: add a field to an event and write your first upcaster.
4
Paper. Sweep your estate for dual writes (grep for publish/send calls inside or adjacent to @Transactional methods) — list the top 10 by blast radius; that's the outbox migration backlog. Then apply §04's selection rule to your aggregates and defend which two, if any, deserve event sourcing.

Self-check

Why doesn't putting the RabbitMQ publish inside @Transactional fix the dual write?
The annotation scopes the DB transaction only; the broker publish is an external call that either happens before commit (event about state that may roll back) or effectively after (crash window between commit and publish). No Spring annotation can enroll RabbitMQ in a Postgres transaction — atomicity requires both writes in one system, which is exactly the outbox's move.
Outbox depth is 0 but consumers report missing events. Where do you look?
Depth 0 + missing events means rows were published-and-cleaned or never written. Check: (1) the producing code path actually writes outbox rows in the same transaction (a new endpoint bypassing it is the usual culprit); (2) relay marked rows published but the broker confirm was lost (publisher confirms enabled?); (3) events published to the wrong routing key/topic. The outbox's guarantee is only as wide as the code discipline that writes to it — hence the lint/architecture test that forbids direct publishes.
Raw-table CDC vs outbox events for integrating your billing DB with analytics — argue it.
Raw CDC ships fast (no producer changes) but couples analytics to billing's physical schema: every migration is a breaking change to a consumer the billing team forgot exists. Outbox (or CDC-on-outbox) publishes intentional, versioned events — a contract. Rule: raw CDC for systems you cannot change (vendor DBs, legacy), always behind an anti-corruption transform; owned services earn the outbox.
Your PM asks for "entitlements as of last March" for a rights audit and your store is CRUD. What does this reveal about §04's rule?
CRUD stores destroy history by design — the update that changed the row erased the audit answer. This aggregate (entitlements) sits squarely in the history-is-a-business-asset category, which is the §04 signal it deserved event sourcing (or at minimum an append-only grants/revocations ledger — the lightweight middle: temporal table / audit journal without full CQRS). The rule cuts both ways: it also says your profile service, which will never get this question, was right to stay CRUD.