Animated Architecture Patterns

Retry with Backoff & Jitter

Retry transient failures with growing, randomised delays so clients don't stampede.

Also known as Retry pattern, Exponential backoff, Jittered backoff

429 wait for Retry-After: 1 s ok on attempt 2 503 wait 100 ms ok on attempt 2 400 not retried blip retry storm: 6 calls in flight, nonstop 100 ms 200 ms 400 ms 800 ms idle between waves every wave overloads it again ok on attempt 3 of 5 ok on attempt 3 of 5 ok on attempt 3 of 5 ok on attempt 3 of 5 ok on attempt 3 of 5 ok on attempt 4 of 5 retries spread out: under capacity attempts per client waiting to retry succeeded failed client 1 client 2 client 3 client 4 client 5 client 6 Service HEALTHY BLIP OVERLOADED calls in flight over capacity: every call in flight times out capacity 4 calls at once 0 0.5 s 1 s 1.5 s 2 s RETRY POLICY What to retry timeout, 503 transient: retry after a wait 429 wait as long as Retry-After says 400, 401 never: a retry can't fix them Retry immediately wait = 0 Every client resends the moment its call fails, again and again. Retries add to the overload that caused the failures. Exponential backoff wait = min(800, 100×2n) ms n = retries so far Waits double: 100, 200, 400, 800 ms, then stay at the cap. Same schedule on every client. Backoff + full jitter backoff = min(800, 100×2n) wait = random(0, backoff) Each client draws its own wait, so retries no longer line up. Max 5 attempts, then fail. RESULT AFTER 2 s attempts succeeded – – 8 5 of 6 the 400 is not retried 120 0 of 6 nothing gets through 30 0 of 6 fewer attempts, no success 19 6 of 6 recovered within 0.75 s

Step 1 · Retry what can succeed. During a brief blip a call fails with 503; the client waits 100 ms and the second attempt succeeds. Retry only errors that can go away: timeouts, 503, and 429 once its Retry-After delay has passed. A 400 or 401 is returned at once, because sending the same request again can't succeed.

Step 2 · Immediate retries. Six calls arrive together, more than the service can handle at once, and all of them time out. Every client retries the moment its call fails, so six calls stay in flight nonstop: 120 attempts in two seconds, none succeed, and the retries keep the service overloaded instead of letting it recover.

Step 3 · Exponential backoff. Each client waits longer after every failure: 100, 200, 400, then 800 ms, which is the cap (min(cap, base × 2ⁿ)). Attempts drop from 120 to 30 and the service is idle between rounds, but the clients failed together and follow the same schedule, so each round arrives as one wave and overloads it again.

Step 4 · Add jitter. With full jitter each wait is drawn at random between zero and the backoff (random(0, min(cap, base × 2ⁿ))). The first retries still collide, then they spread out, the load stays under capacity, and all six succeed within 0.75 s using 19 attempts. A limit of five attempts stops a client from retrying forever when a failure isn't transient.

The problem

Remote calls fail for reasons that fix themselves: a connection drops, a node restarts, a service is briefly overloaded or throttles a busy client. Retrying turns most of these blips into successes nobody notices. But every retry is one more request, sent just when the service is least able to take it. If it is failing because it is overloaded, clients that retry at once keep the load above capacity, so the failures continue and the attempts multiply: a retry storm. Amazon's Builders' Library warns that retries "can even delay recovery by keeping the load high long after the original issue is resolved." Waiting a fixed or doubling time helps less than it seems: clients that failed together wait the same time and come back together, in waves.

How it works

A retry policy answers three questions.

  • Is this failure worth retrying? Retry what may succeed next time: connection failures and timeouts, 503 Service Unavailable and 429 Too Many Requests, and usually 502, 504 and 408 too (in gRPC, UNAVAILABLE). Don't resend a 400, 401, 403, 404 or 422 unchanged: the same request fails the same way, and a 401 needs fresh credentials rather than a wait. (Eventual consistency can blur the line: a 404 just after a create may turn into a success once the new state has propagated.) Google's SRE book puts it plainly: don't retry permanent errors or malformed requests, "because neither will ever succeed."
  • How long should it wait? Use capped exponential backoff with jitter. The backoff grows with every retry, min(cap, base × 2ⁿ), and with full jitter the client sleeps a random time between zero and that backoff. Backoff makes each client retry less often; jitter breaks up the waves, so retries reach the service at a fairly steady rate instead of all at once. In the simulations in Marc Brooker's 2015 article, exponential backoff without jitter was "the clear loser". Full jitter and decorrelated jitter (each wait drawn between the base and three times the previous wait, then capped) did best, and with 100 contending clients jitter cut the number of calls by more than half. If the response carries Retry-After, a number of seconds or an HTTP date (RFC 9110), wait at least that long.
  • When should it stop? After a fixed number of attempts (three in total by default in the AWS SDKs' standard retry mode, four in Polly), when the next attempt could not finish before the caller's deadline, or when a retry budget runs out. Then return the error.

The numbers in the animation come from a small model: six clients send 100 ms calls to a service that can work on four at once, and while more than four are in flight every one of them times out. Step 4 is one seeded random run with the median number of attempts (19) out of 20,000 simulated runs.

When to use it

  • Any call across a network: service-to-service APIs, databases, message brokers and cloud APIs. Most cloud SDKs already retry, and so does a service mesh such as Istio (twice for an HTTP request, by default), so find out what yours does before adding another layer.
  • Background and batch work, which can afford more attempts and longer waits than a request a user is waiting for.
  • Not for permanent errors, and not to wait out a dependency that is down for minutes: stop calling it and fail fast with a Circuit Breaker.
  • Not for operations with side effects, unless they are idempotent (see below).

Trade-offs

  • Retries are extra load. Each one asks a struggling service for more work, and the Builders' Library calls them "selfish" for that reason. Attempt limits and budgets keep the extra load bounded during a real outage.
  • Latency or success. Every wait adds to the caller's latency. Azure's guidance is to fail after fewer, shorter retries in an interactive app and show a "try again later" message, and to save longer exponential schedules for batch jobs.
  • Duplicates. A timeout doesn't tell you whether the server did the work. Retrying a payment that went through charges the customer twice unless the operation is idempotent.
  • Amplification. Retries at several layers of a call chain multiply (see below).
  • Backoff alone isn't enough. Without jitter, clients that failed together stay in step, and once their waits reach the cap they all retry at the capped rate together. Limit the number of attempts as well.

Implementation notes

  • Make retried operations idempotent. RFC 9110 defines GET, HEAD, OPTIONS, TRACE, PUT and DELETE as idempotent and says a client "SHOULD NOT automatically retry a request with a non-idempotent method" unless it knows the request is idempotent or was never applied. For POST and other unsafe operations, send an idempotency key: a unique ID per logical operation that stays the same across retries. The server stores the first result for that key and returns it for every retry. Stripe's Idempotency-Key header works this way, saving the status code and body of the first request. Message consumers need the same protection against redelivery (Idempotent Consumer).
  • Retry at one layer only. Retries multiply down a call chain. In the Google SRE book's example, the JavaScript, frontend and backend layers each make 4 attempts, so one user action can send 64 attempts (4³) to an overloaded database. The Builders' Library describes a five-deep stack that retries at every layer and multiplies the load on the database 243 times (3⁵). Hidden layers count too: an application that makes three attempts behind an Istio sidecar that retries twice can already send nine. Retry at a single point, where the whole operation is understood, and let the layers below fail fast and report why.
  • Budget retries, not just attempts. An attempt limit caps one request. A budget caps a whole client or process, so a widespread outage doesn't turn every request into several. The SRE book describes a per-request limit of three attempts plus a per-client budget that allows retries only while they stay below 10% of requests, which cuts the worst-case growth in traffic from almost 3× to about 1.1×. It also suggests a server-wide limit such as 60 retries per minute. Token buckets are a common way to build a budget. The AWS SDKs' standard retry mode spends tokens on each retry, returns them when requests succeed and stops retrying when the bucket is empty. gRPC's retryThrottling takes one token per failure, adds tokenRatio per success and pauses retries while fewer than half the tokens are left. Envoy's retry_budget caps concurrent retries at a percentage of active and pending requests (20% by default).
  • Set timeouts and a deadline. Give every attempt a timeout and the whole operation a deadline that includes the waits. Don't start an attempt that can't finish before the deadline, and pass the remaining time to downstream calls (deadline propagation in the SRE book) so no layer keeps working on a request its caller has abandoned. For the per-attempt timeout, the Builders' Library starts from the dependency's latency: pick an acceptable rate of false timeouts, such as 0.1%, and use the matching percentile (p99.9).
  • Honour Retry-After, and send it. With a 503, Retry-After says how long the service expects to be unavailable, and a 429 (RFC 6585) may carry it too. On the server side, return a clear overload status instead of letting calls hang, so clients back off; the SRE book recommends exactly that. Vary the delay you hand out, or every throttled client comes back in the same second.
  • Pair it with a circuit breaker. Retries ride out blips; a Circuit Breaker handles failures that outlast a few attempts. Treat "breaker open" as non-retryable, so retries don't keep knocking on a dependency the breaker has already stopped calling. Azure's Retry pattern makes the same point: after repeated failures, stop sending requests and fail immediately. A bulkhead caps how many threads and connections calls to one dependency, retries included, can hold (Bulkhead).
  • Libraries. Polly (.NET), Resilience4j (Java, IntervalFunction.ofExponentialRandomBackoff), tenacity (Python, wait_random_exponential) and the AWS SDKs all support exponential backoff with randomised waits, but check the defaults: Polly's retry strategy waits a constant 2 seconds with jitter off unless you set BackoffType = DelayBackoffType.Exponential and UseJitter = true.
  • Observe it. Count first attempts, retries and give-ups separately and graph the retry rate: the SRE book notes that it can reveal bad retry behaviour during an incident. Azure suggests logging failed attempts that were later retried as informational, and only the final failure as an error.