Core Concepts
gRPC
gRPC makes service-to-service calls compact, fast, and strongly typed. The schema is the contract — the compiler enforces it.
What gRPC Is and the Problem It Solves
Typed remote calls over HTTP/2, driven by a schema both sides compile against
gRPC is a framework for calling methods on remote services through a contract. You describe services and message types in a .proto file, generate client and server code in each language, and call a method on a generated client stub. Messages are encoded with Protocol Buffers (Protobuf) and each call travels as an HTTP/2 stream, so the caller writes inventory.ReserveStock(req) instead of building a URL, serializing JSON and parsing a response by hand.
The problem it addresses is agreement between teams. Imagine an order service asking an inventory service to reserve stock. Both sides must agree on the operation name, the request and response fields, and the error behaviour. With hand-written HTTP clients that agreement lives in documentation and in each caller's assumptions, and a mismatch such as a renamed field or a number that became a string is discovered at runtime, in production. A Protobuf definition makes the agreement machine-checkable: the compiler generates both sides from the same file, so a breaking change shows up as a build failure or a failed compatibility check instead of a surprised caller.
syntax = "proto3";
package inventory.v1;
service InventoryService {
rpc ReserveStock(ReserveStockRequest) returns (ReserveStockResponse);
}
message ReserveStockRequest {
string order_id = 1;
string sku = 2;
int32 quantity = 3;
}
message ReserveStockResponse {
string reservation_id = 1;
bool accepted = 2;
}The numbers 1, 2 and 3 are field tags. They are part of the wire contract, not a display order.
The mental model to keep through the rest of this page is that a gRPC call looks local and is not. The stub hides serialization and transport, but the call still has latency, can fail part-way, can time out with the work done or undone, and crosses a compatibility boundary between independently deployed services. Everything below, from field numbers to deadlines to retries, is about the places where that illusion leaks.
Visual Story
REST overhead → Protobuf compression → Unary RPC → Streaming — watch the full flow
gRPC — Communication Patterns Deep-Dive
⤢ Expand for synchronized narration timeline
How a gRPC Call Travels From Stub to Handler
Five layers between a method call and your business logic
A unary call has five stages. Application code calls a method on the generated stub. The stub attaches call options such as the deadline and serializes the request message with Protobuf. The gRPC library frames that message and sends it on a new HTTP/2 stream. On the server, the HTTP/2 layer hands the stream to a dispatcher that picks the handler from the request path, decodes the bytes into a typed message and invokes your method. The response retraces the same layers in the other direction.
What the bytes actually look like
Take ReserveStockRequest with order_id = "o-17", sku = "A1" and quantity = 3. Protobuf writes each field as a tag followed by its value. The tag is (field_number << 3) | wire_type: field 1 with wire type 2 (length-delimited) is 0x0A, field 2 is 0x12, and field 3 as a varint (wire type 0) is 0x18. Field names never appear on the wire, which is why the numbers matter so much in the next section. The whole message is 12 bytes. gRPC then adds a 5-byte prefix to each message: one byte saying whether the message is compressed and four bytes of length. That 17-byte unit is what goes into an HTTP/2 DATA frame.
Message bytes (hex): 0a 04 6f 2d 31 37 12 02 41 31 18 03
| | o - 1 7 | | A 1 | |
| len=4 | len=2 | value=3
tag: field 1 tag: field 2 tag: field 3 (varint)
gRPC message frame: 00 | 00 00 00 0c | <12 message bytes>
compressed flag | length = 12
HTTP/2 request: HEADERS :method POST
:path /inventory.v1.InventoryService/ReserveStock
content-type application/grpc
te trailers
grpc-timeout 400m
DATA <17-byte frame>, END_STREAM
HTTP/2 response: HEADERS :status 200, content-type application/grpc
DATA <response frame>
HEADERS grpc-status 0 (trailers, END_STREAM)Simplified: real traffic also carries user-agent, authority, encoding negotiation and metadata headers.
Two details of this exchange surprise people. The outcome of the call is carried in the trailers as grpc-status, so a failed RPC still travels in an HTTP response with status 200; HTTP status codes are not how gRPC reports errors, and generic HTTP tooling that only watches :status cannot see failures. And the deadline travels as a relative duration in the grpc-timeout header (for example 400m for 400 ms), not as an absolute timestamp, so servers do not need synchronized clocks to honour it.
Protobuf: Compact Encoding, Field Numbers and the Limits of Compatibility
The schema is a wire contract, and decoding successfully is not the same as agreeing on meaning
Protobuf encodes a message as a sequence of numeric tags and values. Because field names are replaced by small numbers and integers use variable-length encoding, the result is usually smaller than the equivalent JSON, and generated code serializes it without runtime reflection or text parsing. A message with user_id = 42, name = "Mangesh" and role = "admin" takes 18 bytes (2 for the id, 9 for the name, 7 for the role), against 46 bytes for the same data as compact JSON, about 60% less. That is one small, name-heavy message; the real gap depends on field types, repeated fields and whether the JSON is compressed, and for large string-dominated payloads it can nearly vanish. Generated code also gives typed accessors, which removes a category of schema-mismatch bugs from application code. The price is readability: you cannot inspect a message with curl or browser devtools, so keep grpcurl, server reflection or logging interceptors in the development workflow.
The numbers in a message definition are the identity of a field on the wire. Names and declaration order are irrelevant to the encoding; renaming a field is wire-compatible (though it changes generated code and any JSON mapping), whereas changing its number is equivalent to deleting the field and adding a new one. The practical consequence is that a field number must never be reused for a different meaning. Suppose field 3 was string email and was later removed, and a new string phone took number 3. An old client still writing the retired field and a new server reading the same number as something else decode without any error and produce wrong data on every call.
message ReserveStockRequest {
reserved 4; // was: string warehouse_code = 4;
reserved "warehouse_code"; // stops the name being reused, too
string order_id = 1;
string sku = 2;
int32 quantity = 3;
string reason = 5; // new field gets a new number
}The compiler rejects any later attempt to declare field 4 or the name warehouse_code.
Reserving numbers and names turns an easy-to-miss review mistake into a compile error, which is why it belongs in the standard workflow for removing a field. Numbers 1 to 15 encode their tag in a single byte, so they are worth keeping for frequently populated fields.
What Protobuf compatibility does and does not give you
Protobuf makes certain evolutions mechanically safe. Adding a field is safe in both directions: old readers skip the unknown tag, and new readers see the default value when an old writer omits it. Removing a field is safe if its number is reserved and readers tolerate the default. This is the reason a fleet of services can be deployed in any order during a rollout, as long as every change stays within these rules. A schema linter or breaking-change check in CI (for example buf breaking) enforces them automatically.
What the format cannot protect is meaning. If quantity changes from individual units to cartons, every old caller still decodes cleanly into the same integer and silently reserves the wrong amount. Narrowing a type, tightening validation, or changing what a status code means can break callers without a single decoding error. Two proto3 behaviours add to the risk: a scalar field that is absent and a field set to its zero value are indistinguishable unless the field is declared optional, so "not provided" and "zero" must be modelled deliberately; and an enum value unknown to an older reader must be handled by that reader. A strong contract also means a field change needs coordinated rollout discipline across teams, which is the trade you accept in exchange for catching drift at build time.
Unknown fields deserve a similar caution. Current Protobuf runtimes generally keep unknown fields when parsing and re-serializing a message, so a proxy or a read-modify-write service can pass new fields through. That is a runtime behaviour, not a property of the wire format: older runtimes dropped them, and code that copies fields one by one into a new message discards them. A service that must preserve data it does not understand should be tested for exactly that. Semantic changes need the same discipline as in any API; the guidance in API Design on versioning and deprecation applies here unchanged.
HTTP/2 Multiplexing, Flow Control and Where They Stop Helping
Streams are independent logically, but they still share one TCP connection
Every RPC runs on its own HTTP/2 stream, and many streams share one connection. With HTTP/1.1 a connection carries one request at a time, so N concurrent calls need N connections, each with its own TCP and TLS setup; with HTTP/2, concurrent calls become streams with odd numbers (1, 3, 5, ...) on a single warm connection. HTTP/2 also provides framing, HPACK header compression, which sends repeated headers such as :path and content-type as short table references after the first request, and per-stream and per-connection flow control. This is real value, but a common summary of "no head-of-line blocking" needs a qualification: HTTP/2 removes head-of-line blocking at the HTTP layer and leaves it in place at the transport layer.
TCP-level head-of-line blocking
TCP delivers one ordered byte stream. HTTP/2 interleaves frames from many streams into that stream, so if one packet is lost, the receiver cannot hand any later bytes to the application until the missing packet is retransmitted, even if those bytes belong to streams that lost nothing. On a clean data-centre network this barely registers; on lossy mobile or cross-region links, a single connection carrying many latency-sensitive RPCs can stall them all at once. Protocols built on QUIC address this at the transport layer, but gRPC is defined over HTTP/2, so the practical mitigation is operational: spread traffic over several connections when loss is a problem, and measure rather than assume.
- Stream A
- Stream B
- Stream C
Flow control, concurrency limits and idle connections
HTTP/2 flow control works with credits. A receiver advertises a window per stream and per connection, and the sender may not exceed it until a WINDOW_UPDATE grants more. The protocol default is small (65,535 bytes), and many gRPC implementations adjust windows dynamically, but the effect is constant: a consumer that reads slowly stops granting credit, and the producer blocks. This is the backpressure mechanism for streaming RPCs, and it only works if application code actually reads; a handler that buffers unbounded messages in memory hides the problem until the process runs out of heap.
A server also advertises SETTINGS_MAX_CONCURRENT_STREAMS. When a client has more RPCs in flight than the peer allows, the extra calls typically queue in the client until a stream frees up, which shows up as latency before the request is even sent. Slow RPCs and long-lived streams can therefore starve short ones that share the channel; if that happens, use separate channels or raise the limit rather than adding retries.
Connections also die quietly. Load balancers and NAT devices drop idle connections after a timeout, and a client may not notice until it tries to send. Keepalive pings (HTTP/2 PING frames) detect dead peers and keep intermediaries from reaping the connection, but servers enforce a minimum ping interval and may close a connection whose client pings too often, so keepalive settings on both sides and on every proxy in the path have to agree. Two operational limits are also worth knowing: many implementations cap received messages at a default of a few megabytes (large payloads fail with RESOURCE_EXHAUSTED), and HTTP/2 header lists have size limits that bound how much metadata a call can carry.
Why gRPC Is Often Faster, and When It Is Not
Several compounding factors, and the size of the gap depends on the workload
"HTTP/2 makes it fast" is only part of the story. The advantage comes from several factors compounding. Messages are smaller and cheaper to encode and decode, because generated Protobuf code avoids reflection and text parsing. Many calls share one long-lived connection, so there is no TCP or TLS handshake per request. Headers are compressed after the first call on a connection. And streaming lets one call carry many messages instead of paying a request/response cycle for each. For one illustrative service-to-service workload, that can look like roughly 35 ms per call over REST/JSON against roughly 12 ms over gRPC; this is a made-up shape to show direction, not a ratio you should expect. When the time is dominated by a database query or a downstream dependency, the transport is a small fraction of the total and switching protocols will not move latency much. Measure your own payloads before using performance as the reason to adopt gRPC.
The same properties produce sharp failure modes when misused, and the common ones are covered where they bite: connection reuse and load balancing and retry amplification.
gRPC Streaming: The Four RPC Communication Patterns
Unary plus three streaming shapes: pick by the shape of the data flow, not by default
gRPC streaming comes in three shapes beside the unary call, and each runs over a single HTTP/2 stream.
Unary is one request and one response, the right default for calls like GetProduct or ReserveStock. Server streaming is one request followed by many responses, useful for live updates such as a price feed, log tails, or delivering a large result set in pieces without waiting for all of it. Client streaming is many requests followed by one response, a fit for batched ingestion, sensor data or chunked uploads where the server returns one summary. Bidirectional streaming lets both sides send sequences independently over the same RPC, useful for interactive coordination such as chat or a collaborative session.
service TelemetryService {
rpc GetSnapshot(SnapshotRequest) returns (SnapshotResponse); // unary
rpc WatchMetrics(WatchRequest) returns (stream MetricUpdate); // server streaming
rpc UploadMetrics(stream MetricUpdate) returns (UploadSummary); // client streaming
rpc Exchange(stream ClientEvent) returns (stream ServerEvent); // bidirectional
}Message ordering in streaming RPCs
Within one direction of one stream, messages are delivered in the order they were sent, and each arrives exactly once or the stream fails. Across the two directions of a bidirectional stream there is no shared ordering: the client's messages and the server's messages are two independent sequences. If the server sends S1 while the client is sending C2, neither side can know which "happened first". A protocol that needs causality, such as "this reply answers message 7", has to carry it in the messages with sequence numbers or acknowledgement fields. Across different streams, even on one connection, there is no ordering guarantee at all.
Streaming is a capability, not a reason to choose gRPC for every endpoint. A stream owns state: you have to decide how cancellation is handled, how backpressure is applied, how a client resumes after a reconnect (a stream does not survive a broken connection, so the application needs a resume token or offset), and how the stream is drained during deploys. Reach for a stream when the data really is a sequence, and use unary calls otherwise. Fifty unary calls with a small payload are usually better replaced by one coarse-grained call than by a stream, as the chatty-service failure mode below shows.
Deadline Propagation and the Unknown-Outcome Problem
A deadline is a shrinking budget, and an expired one proves very little about what the server did
A deadline is not a per-hop timeout. It is a point in time by which the whole operation must finish, and each hop receives what is left of it. Suppose a client request reaches the API gateway with a 900 ms deadline. The gateway spends 200 ms authenticating and routing, so it calls the order service with 700 ms remaining. The order service spends 300 ms validating and loading data, so it calls inventory with 400 ms remaining. If inventory needs 120 ms, it answers at 620 ms into the request with 280 ms of slack. All numbers here are illustrative.
The key property in the figure is that all three deadlines end at the same instant. If the order service ignored the budget and started a fixed two-second call to inventory, it would keep working long after the gateway gave up and answered the client, burning capacity on a result nobody will read. Propagation is not automatic in every runtime: in several languages you must pass the incoming context (or its remaining time) to every outgoing call, and a fresh background context silently removes the limit. In the worst case, service A calls B, B calls C, and C hangs with no deadline anywhere; the whole chain waits until operating-system or load-balancer timeouts fire minutes later. Good practice is to set a realistic deadline at the edge, forward the remaining budget everywhere, and let handlers check for cancellation before starting expensive steps and stop when the caller is gone. A call with 500 ms left should be allowed to fail fast on a dependency that normally needs 2 s, rather than start it. When a retry policy is configured, the deadline covers all attempts together; it does not restart on each one.
DEADLINE_EXCEEDED does not mean "nothing happened"
When a deadline expires, the caller stops waiting and the RPC fails with DEADLINE_EXCEEDED. The server may not have started, may still be running, or may have committed the change a moment before the response was due. Cancellation behaves the same way: it tells the server, and through it any downstream calls, to stop if it can, but it does not roll back a committed transaction. For a read, this ambiguity is harmless. For ReserveStock, it means the caller does not know whether stock is held. The durable answer is to make the operation safe to repeat or to look up (see the next section), and to make servers protect multi-step writes so that cancellation in the middle cannot leave half-finished state. The Idempotency lesson works through the crash windows in detail.
gRPC Retries: Semantic Safety, Backoff, Jitter and Budgets
Retrying is a design decision about the operation, not a transport setting
Retrying GetProduct is usually harmless. Retrying ReserveStock may reserve the stock twice unless the server recognizes the repeat. Whether a retry is safe depends on the operation's semantics, not on the status code alone. Some implementations transparently retry calls that provably never reached application code, for example when a stream is refused or the connection is already draining. Anything beyond that, such as a retry after UNAVAILABLE or DEADLINE_EXCEEDED, is an application decision. For mutations, carry a stable operation identifier in the request, and have the server store the result of the first accepted attempt and return a compatible result for duplicates. Protobuf and HTTP/2 give you none of this.
Note that order_id alone, as in the opening contract, is not a sufficient key if one order may reserve several SKUs, or if the same order can legitimately reserve again after a cancellation. Use a key that identifies the operation, such as (order_id, sku) or an explicit request ID, and define what a duplicate with different parameters returns.
Configuring retries
gRPC supports retries through the channel's service configuration. The policy limits attempts, spaces them with exponential backoff, randomizes each delay (jitter) so that clients do not retry in lockstep, and lists the status codes worth retrying. A separate throttling block acts as a token-bucket retry budget: failures drain tokens, successes refill them, and retries stop while the bucket is low, so a dependency that is failing for everyone does not receive a wave of extra work.
{
"methodConfig": [{
"name": [{ "service": "inventory.v1.InventoryService", "method": "GetStock" }],
"retryPolicy": {
"maxAttempts": 3,
"initialBackoff": "0.05s",
"maxBackoff": "0.5s",
"backoffMultiplier": 2,
"retryableStatusCodes": ["UNAVAILABLE"]
}
}],
"retryThrottling": { "maxTokens": 10, "tokenRatio": 0.1 }
}ReserveStock is deliberately absent: enable retries for it only after the server deduplicates by operation key.
Retry UNAVAILABLE because it signals a transient transport or server condition. Do not retry INVALID_ARGUMENT, NOT_FOUND or FAILED_PRECONDITION, since repeating the same request returns the same answer. Treat RESOURCE_EXHAUSTED with care: if it means a quota or a load-shedding signal, retrying immediately works against the server. The backoff, jitter and budget concepts are explained on the Retry & Backoff page, and a failing dependency should eventually be cut off entirely by the Circuit Breaker.
Mapping Failures to gRPC Status Codes, Handling Metadata and Observing Calls
Callers branch on the code, so the code has to carry the right meaning
gRPC has its own status set (OK, CANCELLED, INVALID_ARGUMENT, DEADLINE_EXCEEDED, NOT_FOUND, ALREADY_EXISTS, RESOURCE_EXHAUSTED, FAILED_PRECONDITION, UNAVAILABLE and others), conceptually parallel to HTTP status codes but not the same values. Clients decide whether to retry, repair the request or give up from the code, so a service that returns INTERNAL for every rejection forces them to guess. The set is small enough to apply consistently.
| Situation | Status | Caller should |
|---|---|---|
| quantity is negative or sku is malformed | INVALID_ARGUMENT | fix the request; never retry |
| sku does not exist | NOT_FOUND | fix the identifier |
| reservation for this key already exists with different parameters | ALREADY_EXISTS | treat as a conflict |
| order is already shipped, so stock cannot be reserved | FAILED_PRECONDITION | change system state first, then retry |
| per-tenant quota or rate limit reached | RESOURCE_EXHAUSTED | back off; retry only with delay |
| server overloaded, draining or unreachable | UNAVAILABLE | retry with backoff and jitter |
| caller is not authenticated / lacks permission | UNAUTHENTICATED / PERMISSION_DENIED | fix credentials; never retry |
| deadline elapsed | DEADLINE_EXCEEDED | outcome unknown; retry only if safe |
Business outcomes that are an expected result, such as "not enough stock", are often better as data in the response (accepted = false with a reason) than as an error, because the call itself worked. Whichever you choose, document it in the contract so every caller treats it identically.
Metadata and observability
Metadata is the gRPC equivalent of headers and carries credentials, trace context and routing hints. Treat it as untrusted input: validate it, enforce size limits, never log tokens, and do not let clients set metadata that internal systems treat as authoritative (for example a user identity header injected by a trusted proxy). For monitoring, record latency by method and status code, request and response sizes, active streams, deadline failures and downstream dependency errors. Streaming methods need their own measures, namely stream duration, messages per stream and throughput, since a ten-minute watch is not a slow unary call. Propagating trace context in metadata lets one request be followed across every hop in the deadline figure above.
Production Scenario: A Retry Storm When Inventory Slows From 40 ms to 800 ms
Each caller behaves sensibly alone; together they finish off the dependency
Inventory normally answers in about 40 ms and starts responding in about 800 ms after a database failover (illustrative figures). Callers with a 500 ms deadline begin to see DEADLINE_EXCEEDED and UNAVAILABLE. Three upstream services each retry twice, so every failed call can become three attempts. Inventory, already slow, now receives up to three times the traffic. Its queues grow, latency rises further, more calls fail, and more retries are launched. Meanwhile, abandoned attempts from callers that already gave up still occupy server threads. If the retrying services are themselves called by retrying services, the factors multiply; two layers of three attempts mean up to nine.
- Load multiplier, no budget
- 3.00x
- Load multiplier, budgeted
- 1.10x
- Over capacity, no budget
- yes
- Over capacity, budgeted
- no
The default settings of the explorer reproduce the case above: with every attempt failing and three attempts, load triples and exceeds the dependency's capacity. Move the layers slider to see how nesting compounds it, and compare the budgeted bar, which grows by at most 10% per layer however bad the failure rate becomes.
Raising the deadline is not the fix, because it lets slow calls hold resources longer. What helps is a set of defences that work together: limit attempts and retry only UNAVAILABLE for calls known to be safe; add exponential backoff with jitter; retry at one layer of the stack rather than at each; propagate the remaining deadline so work on abandoned requests stops; cap concurrency and queue depth on the server and shed load early with a clear status; make mutating RPCs idempotent so a duplicate is harmless; and open a circuit breaker so callers stop sending while the dependency recovers. The monitoring signals to watch are queue depth, retry rate as a share of traffic, and the ratio of deadline failures to total calls. Detailed mechanics are on the Retry & Backoff and Circuit Breaker pages, and admission control is covered under Rate Limiting.
Channels, Load Balancing and Chatty Calls
Three ways to lose the performance gRPC promised
A new connection per request. An engineer creates a client for every incoming request. Each one pays TCP setup, a TLS handshake and HTTP/2 negotiation, adding tens of milliseconds of latency (illustratively 50 to 200 ms) and exhausting ephemeral ports under load. A channel is meant to be long-lived: create one at startup, share it across calls, and let it manage reconnection and name resolution.
Layer-4 load balancing. gRPC clients keep a few long-lived connections, and a layer-4 balancer distributes connections, not individual RPCs. Once a client has connected, every call it makes lands on the same backend, so one server can carry most of the traffic while its neighbours sit idle, and a newly scaled-out replica receives nothing from existing clients. Use a layer-7 proxy that balances per RPC (Envoy, or nginx with grpc_pass), client-side load balancing with a service registry or DNS-based discovery, or a service mesh. Setting a maximum connection age on servers, which sends GOAWAY and makes clients reconnect, is a common supplement.
Chatty microservices. A service makes forty tiny calls to another to assemble one response, because each field or entity got its own RPC. Per-call overhead, serialization and the added chance of a slow tail dominate, and latency ends up worse than a coarser REST endpoint would have given. Design RPCs around what callers need, batch related data into one call, and treat call count per user request as a metric. A cheap stub does not make a network hop cheap.
Browser Access, Proxies and Long-Lived Streams During Deploys
Where gRPC meets infrastructure it was not designed around
Browser JavaScript cannot control HTTP/2 framing or read trailers the way a backend client can, so it cannot speak native gRPC. Browser-facing systems use one of three approaches: gRPC-Web, a protocol variant translated by a compatible proxy such as Envoy; Connect, which defines protocols that browsers can call directly and that also interoperate with gRPC servers; or an HTTP/JSON gateway that maps REST-style requests onto the RPC methods. Each trades some streaming ability for reach, and bidirectional streaming in particular is not generally available to browsers. A gRPC-first design should therefore decide early whether the public edge is gRPC or REST, and use gRPC behind it.
Every proxy and load balancer between client and server must support HTTP/2 end to end, trailers and the streaming timeouts you need. Layer-7 proxies balance per RPC but add their own idle and maximum stream-duration timeouts. Those timeouts matter most for long-lived streams. During a deploy, a draining server sends an HTTP/2 GOAWAY: in-flight streams may finish within the grace period, new calls go to another connection, and any stream still open when the process exits fails with UNAVAILABLE. A watch stream therefore has to expect disconnection, reconnect with backoff and jitter, and resume from a known position instead of assuming it will live forever. Idle timeouts in the path cut quiet streams in the same way, so use keepalives or application heartbeats on streams that can be silent.
gRPC vs REST: When to Use gRPC, When Not To, and How It Compares
Pick the simplest tool that solves the problem; many systems end up using more than one
In the gRPC vs REST decision, gRPC suits internal, typed, streaming-friendly service calls, while REST suits public, browser-facing, cache-friendly APIs.
gRPC fits internal service-to-service traffic, especially with many services in several languages, where a shared typed contract and generated clients pay for themselves. It fits low-latency, high-throughput paths where encoding cost and connection reuse matter, genuine streaming workloads, and service-mesh environments built around Envoy or Istio. It is a weaker fit for public browser APIs and third-party integrations, where reach and curl-friendliness matter; for simple CRUD applications that gain little from code generation; for teams new to Protobuf who would take on tooling and compatibility discipline without a matching need; and for APIs that people will debug by hand. The usual shape of a larger system is both: REST (or a gateway) at the public edge, gRPC between internal services.
| Dimension | REST | gRPC | GraphQL |
|---|---|---|---|
| Payload readability | JSON, readable with curl and devtools | binary; needs grpcurl or a decoder | JSON, readable |
| Streaming | not built in (SSE, WebSockets, chunked responses) | first-class, four call shapes | subscriptions, usually over WebSockets |
| Browser clients | native | via gRPC-Web, Connect or a gateway | native |
| Contract | OpenAPI, optional and often drifts | required .proto, code generated | required schema, queried by clients |
| Caching | HTTP caching and CDNs work | no HTTP caching of POST-based calls | hard; usually application-level |
| Learning curve | low | medium: tooling, codegen, deadlines | higher: resolvers, query cost control |
The reasoning behind these dimensions, such as client diversity, streaming needs, tooling and team familiarity, is in API Design, and the conventional HTTP side is in REST.
The trade-offs you are accepting
Performance comes at the cost of observability: binary payloads need decoding tools. Strong contracts catch drift at compile time and also mean that field changes need coordination across services. Streaming is powerful and brings flow control, mid-stream errors, reconnection logic and load-balancer complications. And none of it removes the remote-call problems from the first section. Choose gRPC where typed contracts, service-to-service calls and streaming justify code generation and infrastructure support; choose conventional HTTP APIs where broad interoperability, caching and browser simplicity matter more.
End to End: Order Service Reserving Stock
One contract, a propagated deadline, and a server that deduplicates
Putting the pieces together for the order and inventory services. The contract gains an explicit operation key under a new field number, while the retired number stays reserved. With REST the caller would write a fetch, check res.ok and read data.stock_level hoping the field still exists; with the generated client, a renamed or retyped field fails the build instead.
message ReserveStockRequest {
reserved 4; // retired in the earlier example
string order_id = 1;
string sku = 2;
int32 quantity = 3;
string request_id = 5; // operation key, stable across retries
}The order service calls inventory with the incoming context, so the deadline from the gateway applies, and it may tighten but never extend it. It branches on the status code rather than on error text.
func (s *OrderService) reserve(ctx context.Context, o Order) error {
// ctx already carries the gateway's deadline; WithTimeout can only shorten it.
ctx, cancel := context.WithTimeout(ctx, 400*time.Millisecond)
defer cancel()
resp, err := s.inventory.ReserveStock(ctx, &invpb.ReserveStockRequest{
RequestId: o.ReservationKey, // generated once per order attempt, reused on retry
OrderId: o.ID,
Sku: o.Sku,
Quantity: o.Quantity,
})
switch status.Code(err) {
case codes.OK:
if !resp.Accepted { return ErrOutOfStock }
return nil
case codes.InvalidArgument, codes.NotFound, codes.FailedPrecondition:
return err // repeating the same request cannot succeed
default:
// Unavailable, DeadlineExceeded: outcome unknown, safe to retry only because of RequestId.
return retryable(err)
}
}On the server, the reservation record and the stock decrement are written in one transaction, with a unique key on request_id, and the transaction claims the key first. A duplicate attempt cannot decrement twice: if the first attempt committed, the insert finds the key and returns no row, and the server returns the stored result without touching stock; if two attempts race, the second insert waits on the first, then finds the key and returns the stored result. If the stock update matches no row, the transaction rolls back, so the claim disappears with it and the server responds accepted = false.
CREATE TABLE reservations (
request_id text PRIMARY KEY,
order_id text NOT NULL,
sku text NOT NULL,
quantity int NOT NULL,
reservation_id text NOT NULL,
created_at timestamptz NOT NULL DEFAULT now()
);
BEGIN;
INSERT INTO reservations (request_id, order_id, sku, quantity, reservation_id)
VALUES ($1, $2, $3, $4, $5)
ON CONFLICT (request_id) DO NOTHING
RETURNING reservation_id; -- no row: duplicate, ROLLBACK, return stored row
UPDATE stock SET available = available - $4
WHERE sku = $3 AND available >= $4; -- 0 rows: ROLLBACK, respond accepted = false
COMMIT;On a duplicate key, read the stored row: same sku and quantity means return the original result; different parameters means ALREADY_EXISTS.
Now trace the failure that started this page. The caller sends the request, inventory commits the reservation, and the response is lost. The caller sees DEADLINE_EXCEEDED, retries with the same request_id, and gets the original reservation back instead of a second hold. The retry is limited to a few attempts with jittered backoff inside the one deadline, behind a retry budget and a circuit breaker, and the metrics (latency by method and status, deadline failures, retry rate) tell you when any of it starts to degrade. This deduplication is an application guarantee that you build; neither Protobuf nor HTTP/2 supplies it, and a key by itself does not make execution exactly-once, only repeat-safe.
Key Takeaways
The few rules that keep a gRPC service safe under failure
- Set a realistic deadline at the edge and forward the remaining budget on every outgoing call; a deadline is a point in time shared by all hops, and with retries it covers all attempts together.
- DEADLINE_EXCEEDED means the outcome is unknown, not that nothing happened. Make writes safe to repeat with a request key and return the stored result when the same key arrives with the same parameters.
- Retry only operations that are semantically safe, with backoff, jitter and a budget. Nested layers multiply load (two layers of three attempts mean up to nine), which is how a 40 ms dependency becomes an 800 ms outage.
- Treat field numbers as the wire contract: never reuse or renumber them, and reserve retired numbers and names, because decoding successfully is not the same as agreeing on meaning.
- Map failures to status codes callers can branch on (INVALID_ARGUMENT never retried, UNAVAILABLE retried with backoff, FAILED_PRECONDITION after changing state).
- Reuse channels, check that every proxy supports HTTP/2 end to end, and plan for GOAWAY and long-lived streams during deploys.
Where to Go Next
The surrounding lessons that make gRPC safe to run
gRPC makes typed remote calls convenient; it cannot make a remote operation local. For the contract-level choices around it, read API Design and compare with REST. For the failure side, Idempotency explains the crash windows behind unknown outcomes, and the Core Patterns lessons on Retry & Backoff, the Circuit Breaker and Rate Limiting cover the protections a gRPC service needs under load.