Retries are a budget, not a reflex
A retry is useful when the next attempt has a reasonable chance of succeeding. It is harmful when it repeats a permanent error, multiplies load during an outage, or repeats a non-idempotent operation. Before adding a retry loop, classify the operation and the failure. A connection reset before the server received a request is different from a 400 validation error. A 429 rate limit is different from a 503 overload response. The client needs a policy that reflects those differences.
Set an explicit retry budget: a maximum attempt count, a maximum elapsed time, and a maximum delay. The elapsed-time limit matters more than the attempt count when a service is slow. A request that waits through five short retries may be worse than one that fails clearly after a bounded deadline. Pass the remaining budget through nested calls so a parent request cannot accidentally spend several minutes inside a child retry loop.
Use exponential backoff with jitter
Exponential backoff increases the delay after each failed attempt, giving a recovering service time to reduce pressure. Without jitter, many clients that observed the same outage can retry on the same schedule and create synchronized waves. Add randomness to the delay so clients spread their attempts. A practical policy starts with a small delay, doubles it up to a cap, and applies full or equal jitter. The exact numbers should match the service's documented limits and the user's latency expectations.
Honor a server-provided Retry-After value when it is present and valid, but cap it by the client's deadline. A provider may return a date instead of a number of seconds, so parse both formats and handle clock skew. If the value is absurdly large or malformed, fall back to your safe policy. Log the chosen delay category, not every random value, so operators can understand the policy without drowning in noise.
Retry only failures that can recover
Transient network errors, connection resets, DNS timeouts, and selected 5xx responses are common retry candidates. A 429 is usually retryable with the server's guidance. A 401 may be transient during a token refresh, but repeating it with the same credential will not help. A 403, 404, and most 400 responses usually require a code or configuration change. Treat a malformed response as an integration failure that deserves an alert, not as permission to retry forever.
The method and operation matter. GET requests are often safe to retry when the server contract is read-only, but a POST may create a side effect. If the API supports an idempotency key, send one and reuse it for the logical operation. If it does not, prefer a durable outbox or a query that can confirm the result before retrying. Never assume that a timeout means the server did nothing. It only means the client did not observe the result.
Make cancellation and deadlines first-class
A retry policy should cooperate with cancellation. If the caller navigates away, shuts down, or cancels a job, do not keep retrying in the background. Use an abort signal or context deadline and pass it to the HTTP client. Reserve a small portion of the overall budget for parsing and cleanup. A request that uses its entire deadline inside the network call leaves no time to record the failure or release a lease.
Different operations need different budgets. A user-facing autocomplete request might allow one quick retry, while a background synchronization job can tolerate several minutes. Put the policy near the client method and name the intent, such as userInteractive or backgroundSync. Avoid one global retry helper that silently retries every endpoint. The policy is part of the API contract and should be visible in code review.
Prevent retry storms at the fleet level
Per-request backoff is not enough when thousands of workers share a dependency. Add a circuit breaker or a fleet-wide rate limiter that can stop new work after a sustained failure. A breaker should have clear open, half-open, and closed states, with a small probe volume during recovery. Do not make the breaker so aggressive that one noisy tenant blocks every customer. Partition by dependency, region, or credential when the failure domain supports it.
Queue-based work should also have a maximum delivery age. A message that is several hours old may no longer be useful, especially if it represents a temporary user action. Move exhausted messages to a dead-letter path with context and a replay procedure. Alert on the age and volume of that path. Infinite retries turn a temporary outage into permanent operational debt.
Test the policy with a controllable server
A good retry test can instruct a fake server to return a sequence such as timeout, 503, 429 with Retry-After, and success. Assert the number of attempts, delay bounds, idempotency key reuse, cancellation behavior, and final error classification. Use a fake clock where possible so tests do not sleep for real time. Add a property that total elapsed time never exceeds the caller's deadline even when the server keeps failing.
Test the negative path too. A 400 should not retry. A malformed Retry-After should not produce an unbounded delay. A response with an unknown status should follow a documented default. A duplicate POST should be detectable through the idempotency key. These tests are cheap compared with investigating a production retry storm that causes a provider to block your account.
Tell operators what the client decided
Emit structured metrics for attempts, retryable failures, permanent failures, exhausted budgets, circuit state, and time spent waiting. Include dependency name, operation, status class, and outcome. Do not include authorization headers or request bodies. A trace should show one logical operation with child attempts rather than making each retry look like a separate user action.
Write the retry policy in the runbook with examples. Explain which statuses are retryable, how Retry-After is handled, what the maximum age is, and how to replay a dead-lettered item. When a webhook or API integration fails, a controlled payload inspector can help reproduce the request, but the client still needs a bounded, idempotent policy so recovery does not become another incident.
Implementation example
Retry only failures that are plausibly temporary and safe to repeat. Combine exponential backoff with a cap and jitter, honor Retry-After when it is valid, and use an idempotency key for operations that create state. Give each request a total deadline so retries cannot consume the entire user or worker budget.
const delayMs = Math.min(capMs, baseMs * 2 ** attempt)
* (0.5 + Math.random());
await sleep(Math.min(delayMs, deadlineMs - elapsedMs));Verify and troubleshoot
Test timeouts, connection resets, 408, 429, 500, 502, 503, and 4xx validation errors separately. Assert that retries stop at the deadline, use jitter, and preserve the original request identity. Inspect attempt count, backoff duration, provider response headers, and total user latency. A retry that turns a provider outage into synchronized traffic is a client-side incident.
Operations and recovery
Set a retry budget per route and expose exhausted operations to a queue or operator workflow. Alert on retry amplification, not just final error rate. When a dependency is unhealthy, reduce optional traffic, honor its quota, and prefer a truthful degraded response over an unbounded retry loop. Document whether a manual replay is safe and how to find the original correlation ID.
References and further reading
Use RFC 9110 for HTTP semantics, the provider's rate-limit contract, and the Google SRE guidance on handling overload. State which methods and status codes are retryable for this specific client.