Timeouts, Retries and Circuit Breakers: Designing for Failure

A payment provider slows down. Each request to it now takes 30 seconds instead of 300 milliseconds. Your API keeps accepting checkout requests, each one waiting on the provider, until every connection and worker is busy waiting. Now the pages that never touch payments are timing out too.

This is how one slow dependency becomes a full outage. It isn't a bug in any single line of code; it's the absence of three habits: timeouts, careful retries and circuit breakers.

#Rule 1: Every Network Call Gets a Timeout

The default timeout for many HTTP clients is "wait forever" or several minutes. That's never what you want.

const res = await fetch(`${PAYMENTS_URL}/charges`, {
  method: "POST",
  body: JSON.stringify(charge),
  headers: { "content-type": "application/json", "idempotency-key": charge.id },
  signal: AbortSignal.timeout(2_000), // give up after 2 seconds
});

How long? Look at the dependency's real latency: a timeout a little above its 99th percentile (p99) is a good starting point. If p99 is 800 ms, 1.5–2 s is reasonable; 30 s is not.

#Rule 2: Retry Only What's Safe, With Backoff and Jitter

Retries hide brief, transient failures: a dropped connection, a 503 during a deploy. Done carelessly, they turn a struggling dependency into a dead one.

Retry only when both are true:

• The error is transient: timeouts, connection resets, 502, 503, 504, or 429 (respecting Retry-After). Never retry a 400 or 401; it will fail the same way.
• The operation is safe to repeat: reads, or writes that carry an idempotency key, like the charge above.

async function withRetry<T>(fn: () => Promise<T>, attempts = 3): Promise<T> {
  for (let attempt = 1; ; attempt++) {
    try {
      return await fn();
    } catch (err) {
      if (attempt >= attempts || !isTransient(err)) throw err;
      // Exponential backoff with full jitter: 0–200ms, then 0–400ms, then 0–800ms...
      const cap = 200 * 2 ** (attempt - 1);
      await new Promise((r) => setTimeout(r, Math.random() * cap));
    }
  }
}

Here isTransient is a small helper that returns true for timeouts, connection resets and the status codes above.

Jitter matters. Without it, every client that failed at the same moment retries at the same moment, hitting the recovering service in synchronized waves.

#Rule 3: Stop Calling Something That's Down

Timeouts and retries still send traffic to a dependency that's clearly broken. A circuit breaker notices repeated failures and short-circuits: it fails fast for a while instead of waiting, then carefully checks whether the dependency has recovered.

• Closed: normal. Calls go through; failures are counted.
• Open: after too many failures (say, 50% of the last 20 calls), calls fail immediately without touching the network.
• Half-open: after a cool-down (say, 30 seconds), a few trial calls are allowed. If they succeed, close; if not, open again.

A minimal version:

class CircuitBreaker {
  private failures = 0;
  private openedAt = 0;
  private state: "closed" | "open" | "half-open" = "closed";

  constructor(private readonly threshold = 5, private readonly coolDownMs = 30_000) {}

  async call<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === "open") {
      if (Date.now() - this.openedAt < this.coolDownMs) throw new Error("circuit open");
      this.state = "half-open";
    }
    try {
      const result = await fn();
      this.failures = 0;
      this.state = "closed";
      return result;
    } catch (err) {
      if (this.state === "half-open" || ++this.failures >= this.threshold) {
        this.state = "open";
        this.openedAt = Date.now();
      }
      throw err;
    }
  }
}

In production, use a maintained library (for example opossum in Node.js, or Polly in .NET) that adds rolling windows, metrics and limits on how many trial calls are allowed in half-open.

#Rule 4: Decide What Happens When It Fails

Failing fast is only useful if the caller has something better to do. Decide this per feature, with the product owner:

• Show cached or stale data ("prices as of 2 minutes ago").
• Degrade the feature (hide recommendations, keep the product page).
• Queue the work and confirm later ("we'll email your receipt").
• Fail clearly with a useful message, rather than a spinner that never ends.

#Rule 5: Isolate Resources

Give each dependency its own limit on concurrent calls (a bulkhead), so a slow payment provider can use up its own share of connections but not the ones search and login need.

#Checklist

• Every outbound call has a timeout based on the dependency's real latency.
• Retries are limited, use backoff with jitter, happen in one layer, and only for transient errors on safe operations.
• Critical dependencies sit behind a circuit breaker with a clear fallback.
• Each dependency has its own concurrency limit.
• Dashboards show timeouts, retries and breaker state per dependency.

#Summary

Failures in distributed systems aren't exceptional; they're scheduled, you just don't know when. Timeouts bound how long you wait, careful retries absorb blips, circuit breakers stop you hammering something that's down, and fallbacks keep users moving. Together they turn "the payment provider is slow" from an outage into a degraded checkout.