REST, gRPC, GraphQL & Real-time

Fast trackThe three subsections below — deadlines, load balancing, streaming semantics — are where gRPC estates succeed or melt. Full read.

Deadlines, not timeouts

gRPC's deadline is absolute and propagated: the client sets "this whole operation is worthless after T", and every downstream hop receives the remaining budget in metadata. A 300 ms playback-auth deadline arriving at the entitlement service with 120 ms left means it can skip the optional enrichment call — deadline-aware degradation. Contrast per-hop timeouts, which compose blindly: five sequential 2 s timeouts can burn 10 s on a request the user abandoned at 1 s. Rule: every gRPC call sets a deadline; no infinite defaults survive code review. Servers should check Context.current().isCancelled() before expensive work — the client may be long gone.

The load-balancing caveat

gRPC = long-lived HTTP/2 connections carrying many calls. An L4 (connection-level) balancer balances connections, not calls — 3 pods, 3 connections, and one chatty client pins one pod at 100% while the LB dashboard shows "balanced". Fixes: an L7/HTTP-2-aware proxy (Envoy — per-call balancing), client-side LB with service discovery, or connection MAX_CONNECTION_AGE to force periodic re-balance. This single fact explains most "gRPC load is lumpy" incidents (full treatment: Module 08).

Streaming semantics

• Server streaming — one request, response stream: live score feeds, watch-progress sync down.
• Client streaming — request stream, one response: batched telemetry/QoE beacon upload.
• Bidirectional — independent streams both ways: session control channels. With flow control inherited from HTTP/2 — backpressure you don't have to build (and with WebSockets, you do).

Errors are first-class: status codes (DEADLINE_EXCEEDED, UNAVAILABLE, RESOURCE_EXHAUSTED) with typed detail payloads — and retry policy should key on them: retry UNAVAILABLE, never retry INVALID_ARGUMENT, retry DEADLINE_EXCEEDED only at the top with budget left.

Answer per edge, not per company. Tick what's true for the edge you're designing:

The defaults it encodes — also the defensible-in-review shape for your estate: gRPC inside (owned both ends, contracts + codegen + deadlines), REST at the public edge (reach + caching), GraphQL only at a BFF layer where screen-aggregation pain is real (never service↔service), SSE before WebSockets unless the client genuinely pushes, and events via the message bus rather than any of these when the interaction is notification-shaped (Module 15).

With hundreds of services, you can never deploy both sides of an edge atomically — so every change must be compatible with the version currently running on the other side. The rules:

• Protobuf: field numbers are the contract — never reuse or renumber; only add fields (new fields are unknown-but-preserved to old readers); mark removed fields reserved; never change a field's type; everything effectively optional with defaults, so absence must be semantically safe.
• JSON/REST: consumers must ignore unknown fields (Jackson: FAIL_ON_UNKNOWN_PROPERTIES=false — deliberately, estate-wide); additive changes only; removals/renames go through deprecate → dual-write → migrate → remove.
• Breaking changes get a new major surface (/v2, new proto package) with both versions served during migration — and a sunset date enforced by gateway metrics on per-consumer version usage, or v1 lives forever.
• Contract tests in CI: compatibility checked mechanically (buf breaking-change detection for proto; consumer-driven contracts for REST). Review-time vigilance does not scale to 200 services; pipelines do.

A · Contract that evolves safely

syntax = "proto3";
package ott.entitlement.v1;

message CheckRequest {
  string user_id    = 1;
  string content_id = 2;
  reserved 3;                         // was device_type — removed, number retired forever
  string device_id  = 4;              // added later: old servers ignore, old clients omit —
}                                       // so absence MUST be handled as "unknown device"

service EntitlementService {
  rpc Check(CheckRequest) returns (CheckResponse);
  rpc WatchGrants(WatchRequest) returns (stream GrantEvent);  // server streaming
}

B · Deadline + status-aware retry

CheckResponse check(String userId, String contentId) {
  try {
    return stub
        .withDeadlineAfter(250, TimeUnit.MILLISECONDS)   // ALWAYS. propagates downstream
        .check(CheckRequest.newBuilder()
            .setUserId(userId).setContentId(contentId).build());
  } catch (StatusRuntimeException e) {
    return switch (e.getStatus().getCode()) {
      case UNAVAILABLE      -> retryOnceWithBudget();     // transient: retry is correct
      case DEADLINE_EXCEEDED -> fallbackDeny("latency");  // budget gone: do NOT pile on
      default               -> throw e;                  // INVALID_ARGUMENT etc: a bug, surface it
    };
  }
}

Fast trackStep 2 only (~30 min): breaking compatibility on purpose and catching it with tooling is the durable lesson.

Self-check