Timeouts, Retries & Backoff

• Setting values: read timeouts come from the dependency's measured p99.5 + margin — not round numbers. A 2 s timeout on a 50 ms-p99 dependency doesn't "add safety"; it means that when things go wrong you hold threads/connections 40× longer than useful, converting a dependency brownout into your own pool exhaustion.
• Timeout < caller's patience: serving a beautiful response at t=9 s to a client that gave up at t=3 s is pure waste — and with retries above you, it's waste × attempts. Deadlines (Module 07) are the systematic fix; per-hop timeouts are the floor when deadlines aren't plumbed.
• Every await is a timeout site: HTTP, DB (javax.persistence.query.timeout, socket + statement), Redis, RabbitMQ publishes and consumer acks, lock acquisition, CompletableFuture.get. The audit finds the ones with no value set; those are where incident threads pile up.

Fast trackThe multiplication below is the single most important number in this module. Run it.

Retries at every layer multiply. If each of 4 layers does 3 attempts, one user tap can become 3⁴ = 81 requests at the bottom — arriving precisely when the bottom layer is least able to serve them:

The rules that keep this survivable:

• Retry at one layer — ideally the top (it knows user intent) — pass failures through elsewhere. Meshes/gateways doing silent retries under app retries is how 81 happens without anyone writing 81.
• Retry only retryable errors: connect failures, 503, gRPC UNAVAILABLE — yes. 4xx/INVALID_ARGUMENT — never. Timeouts — only if idempotent (the request may have succeeded: Module 16 owns this).
• Retries need remaining budget: a retry whose deadline already expired is amplification with zero possible benefit.

Exponential backoff (base × 2ⁿ, capped) spaces attempts out — but without jitter, every client that failed together retries together: synchronized waves hammering the recovering service at t=1s, 2s, 4s. Compare:

Full jitter — sleep = random(0, min(cap, base × 2ⁿ)) — spreads the entire wave flat and is the recommended default (per the canonical AWS analysis: best server-load profile, negligible completion-time cost). Equal jitter (half fixed + half random) is the compromise when you must guarantee minimum spacing. No jitter is a bug, not a style.

• Retry budgets beat retry counts. Per-attempt counts look at one request; budgets look at the system: retries may be at most X% of recent request volume (e.g. 20%, token-bucket implemented). Healthy system: occasional retries sail through. Broken dependency: budget exhausts, retries stop fleet-wide, the storm never forms. This is the fleet-level guard that per-call config cannot provide — and what Envoy/Finagle-class systems implement natively.
• Hedged requests attack tail latency, not errors (Module 02's amplification): after the p95 mark with no response, send a second attempt to a different replica; take the first answer, cancel the loser. Bounds: hedge only idempotent reads, cap hedges at ~5% of volume, and cancel aggressively — hedging without cancellation is deliberate load doubling.
• Retries interact with everything downstream of this week: they're the amplifier that circuit breakers (M13) interrupt, the duplicate-source idempotency (M16) exists for, and the reason rate limiters (M14) see 3× traffic during brownouts. Design them as one system.

RetryConfig config = RetryConfig.custom()
    .maxAttempts(3)
    .intervalFunction(IntervalFunction
        .ofExponentialRandomBackoff(100, 2.0, 0.6))  // base 100ms, ×2, randomized — jittered
    .retryOnException(e ->
        e instanceof ConnectException ||
        (e instanceof WebClientResponseException w
            && w.getStatusCode().value() == 503))     // transient ONLY — never 4xx
    .failAfterMaxAttempts(true)
    .build();

// order matters: TimeLimiter (total budget) OUTSIDE Retry (attempts) —
// so attempts can never outlive the caller's deadline
Supplier<CompletionStage<Entitlement>> call =
    TimeLimiter.decorateCompletionStage(timeLimiter, scheduler,
        Retry.decorateCompletionStage(retry, scheduler,
            () -> client.check(userId, contentId)));

Fast trackStep 2 (~25 min): watching your own retry storm triple the load on a browning-out dependency is permanent education.

Self-check