Queues are buffers that convert latency problems into throughput problems — and coupling problems into consistency problems. Delivery guarantees as they actually are, RabbitMQ vs Kafka semantics, ordering, backpressure, and the DLQ discipline that keeps event estates debuggable.
~3h study~2h exercise1 interactive simFast track ~85 min
Fast track — running both brokers already?
Compresses to ~85 min. Motivation collapses; delivery-guarantee mechanics, the broker-semantics comparison, ordering, and failure handling stay mandatory — this module is your estate's native territory and its sharpest edges.
§01 takeaway · 5 min
§02 delivery sim, all three modes · 20 min · MANDATORY
§03 semantics table, in full · 20 min · MANDATORY
§04 ordering, in full · 15 min · MANDATORY
§05 DLQ discipline, in full · 15 min · MANDATORY
STEP 01
What events buy — and cost
SKIM
Fast-track takeawayAsync messaging buys: temporal decoupling (consumer down ≠ producer blocked), load leveling (the queue absorbs the 21:00 spike, consumers drain at their pace — Module 06's buffer trade), fan-out without producer knowledge, and failure isolation. It costs: eventual consistency everywhere downstream (Modules 03–04 apply to every consumer), loss of request/response simplicity (correlation, sagas — M17), an ops-critical broker, and debuggability that must be designed (correlation IDs, tracing through the broker). The test for event vs RPC: does the producer need the answer to proceed? Yes → call. No → event.
The architectural upgrade events enable: inverting dependencies. "Purchase service calls entitlement, notification, analytics, recommendations" (knows 4 teams, breaks with each) becomes "purchase emits PurchaseCompleted; interested parties subscribe" — new consumers cost the producer nothing. That inversion is why event estates scale organizationally, and why their failure modes (this module) are worth mastering.
STEP 02
Delivery guarantees as they actually are
MANDATORY
Fast trackRun all three modes. The punchline to internalize: at-least-once is the only honest guarantee for work that matters, and it makes duplicates a certainty to design for, not an edge case.
10 events · consumer crashes mid-batch, then recovers
At-most-once (ack/commit before processing, or fire-and-forget): crash between ack and work = message gone. Acceptable only where loss is cheaper than duplication machinery: raw telemetry, ephemeral presence.
At-least-once (process, then ack): crash after work but before ack = redelivery = duplicate processing. The default for everything that matters — which makes consumer idempotency a hard requirement, not a virtue (Module 16 is the companion to this one).
"Exactly-once" is exactly-once processing effect, never exactly-once delivery: Kafka transactions give it within a consume-transform-produce loop into Kafka; the moment a side effect leaves the transactional world (a DB write, an HTTP call, an email), you're back to at-least-once + idempotency. Treat every "exactly-once" claim as a scoped statement and ask for the scope.
STEP 03
RabbitMQ vs Kafka: different machines, not competitors
Staff expectationThe selection question is "queue or log?" — does anyone need to re-read the past? Commands (do this once, then it's done) → queue. Facts (this happened; multiple presents and futures care) → log. Estates that put commands on Kafka fight rebalancing and per-message error handling; estates that put event streams on RabbitMQ rebuild retention and replay badly. Running both, each for its shape — as you do — is the correct answer, and the review-worthy skill is policing which shape each new use case is.
STEP 04
Ordering: scarce, local, and paid for
MANDATORY
The only ordering anyone gives you is per-partition (Kafka) / per-queue-with-one-consumer (Rabbit). Global order across a topic doesn't exist at scale. Design for per-entity order: partition key = the entity whose history must be sequential (userId for watch events, contentId for content updates) — the same key decision as Module 05, with the same hot-key consequences (one celebrity user ≠ one hot partition, if you chose well).
Order-destroyers to audit for: producer retries without idempotence (enable.idempotence=true fixes reorder-on-retry within a partition); max.in.flight>5 without idempotence; consumer-side parallelism that fans one partition across threads (key-affinity executors restore it); Rabbit redelivery re-queuing behind newer messages; and changing partition count (Module 05: key→partition remaps, histories straddle).
Design escape: make consumers order-tolerant instead of demanding order — version numbers in events with last-write-wins per entity, or full-state ("current resume point is X") rather than delta ("advance by 10 s") events. State-carrying events tolerate reorder and duplication; delta events tolerate neither. Prefer state-carrying at boundaries.
Backpressure: Rabbit — prefetch/QoS caps in-flight per consumer (unbounded prefetch = consumer OOM during backlogs); Kafka — max.poll.records × processing time must stay under max.poll.interval.ms or the group rebalances mid-work (the classic "consumer keeps getting kicked" incident); lag is your queue-depth SLO (M14's saturation signal, broker edition).
STEP 05
Failure handling: the DLQ discipline
MANDATORY
Classify before retrying: transient (dependency timeout) → in-place redelivery with backoff (Rabbit: delayed-retry queues via TTL+DLX; Kafka: retry topics topic.retry.5s/1m/10m ladder). Permanent (deserialization failure, business-invalid) → straight to DLQ/parking-lot; retrying a poison message forever is how one bad event halts a partition.
Bound in-place retries hard: a message redelivered in a tight loop blocks everything behind it (per-partition HOL — Module 06's concept, broker edition) and can dominate consumer CPU. Max redeliveries, then escalate to the retry ladder or DLQ.
DLQ is a workflow, not a bin: alert on depth > 0 per source; every entry carries original payload + headers + exception + attempt count; a redrive tool (with idempotent consumers making redrive safe — M16); and a weekly-zero policy. A DLQ nobody drains is silent data loss with extra steps.
Schema discipline = messaging's API versioning: Module 07's expand-migrate-contract applies verbatim to event payloads; a schema registry (or the JSON ignore-unknowns discipline) with CI compatibility checks, because producers and consumers never deploy together. The registry's compatibility mode is a real decision, not a default: backward = new schema reads old data → consumers upgrade first (the usual choice: readers must survive whatever's already in the topic — and note retention/replay makes "old data" mean everything retained, not just in-flight); forward = old schema reads new data → producers may upgrade first; full = both, restricting you to add-optional-with-default / remove-optional changes — the discipline worth paying for on long-retention and compacted topics, where a reader may meet every schema version ever written. The mode also dictates rollout order across teams, which is why it belongs in the topic's contract, not tribal knowledge — and renames/type-changes are never compatible in any mode: that's a new field (expand) or a new topic, exactly like M07's URL-version rule.
Observability through the broker: trace context propagated in headers (W3C traceparent) so a purchase→entitlement→notification chain is one trace; per-consumer lag and DLQ depth as first-class SLIs; and event catalogs (which team owns which topic/exchange, with what contract) — at 200 services the catalog is the difference between an event estate and event spaghetti.
STEP 06
Exercise
MANDATORY
Fast trackStep 1 (~40 min): producing the duplicate with your own crash teaches more than the whole §02 text.
1
Manufacture a duplicate. Rabbit consumer (manual acks) writing "entitlement granted" rows; System.exit(1) after the DB write, before the ack. Restart → redelivery → duplicate row. Repeat on Kafka with commit-after-process. You have now seen why Module 16 exists. Keep both duplicate rows as trophies.
2
Lose a message honestly. Kafka producer with acks=1; kill the partition leader right after a send; observe the acked-but-lost write with acks=all + min.insync.replicas=2 as the fix. Then break ordering: enable.idempotence=false, max.in.flight=5, inject a retry, watch per-key reorder; fix with idempotence on.
3
Build the retry ladder + DLQ. Rabbit: work queue → 5 s and 1 m delayed-retry queues (TTL+DLX) → parking-lot, with attempt-count headers and an alert on parking-lot depth. Poison a message and watch it walk the ladder without blocking the queue.
4
Paper. Audit five real topics/exchanges in your estate against this module: guarantee mode, ordering key + hot-key risk, retry/DLQ path, schema evolution policy, owner. The gaps become the platform backlog.
Self-check
Where exactly does Kafka's exactly-once stop applying?
At the boundary of the Kafka transaction: consume→transform→produce into Kafka topics with transactional producer + read_committed consumers is exactly-once in effect. The first non-Kafka side effect — a JPA write, an HTTP call — sits outside the transaction and can happen 0 or 2 times on crash/retry. Cross that boundary and you're back to at-least-once + idempotent effects (M16) or outbox-style patterns (M18).
Your consumer group rebalances every few minutes under backlog. Mechanism and fixes?
Processing a full max.poll.records batch exceeds max.poll.interval.ms → broker assumes the consumer died → rebalance → redelivery → more backlog: a self-sustaining loop. Fixes: smaller poll batches, longer poll interval, move heavy work off the polling thread (with manual offset management), and treat lag as a scaling signal before it becomes this.
Content updates must apply in order per title, but the consumer is 8-way parallel. Reconcile.
Partition by contentId (broker guarantees per-partition order), then preserve it inside the consumer: shard work across the 8 threads by hash(contentId) so each title's events stay on one thread — parallelism across titles, sequence within a title. Or sidestep: version-stamped full-state events + LWW make order irrelevant.
Why is a delayed-retry queue better than sleep-and-nack in the consumer?
Sleeping holds the consumer thread and its prefetch slot hostage — under a burst of failures the whole consumer stalls (and Rabbit immediate-redelivery nack just hot-loops the same message at the queue head). TTL+DLX delayed queues park the message in the broker, freeing the consumer for other work, with the backoff schedule explicit in topology and attempt-count in headers.