Step 2 of 15 · 1 min

How a slow dependency takes everything down

It's worth walking through the mechanism, because it's the same every time.

Each Chirp API server could handle about 200 requests at once (a pool of worker threads). Normally, a home screen request took 80 ms, and each server handled about 500 a second. By Little's Law (lesson 1.2), that's 500 × 0.08 = 40 workers busy, with 160 to spare.

Then each home screen request started including a 20-second wait for recommendations. Requests that used to hold a worker for 80 ms now held one for 20 seconds, 250 times longer. Little's Law says the number of busy workers grows by the same factor: 500 × 20 = 10,000 workers needed, on a server that had 200. Every worker on every API server filled up with requests waiting for recommendations, and new requests had no worker to run on. Including requests that had nothing to do with recommendations: posting, liking, loading a profile.

A slow dependency is far more dangerous than a dead one. A dead service fails fast, and your code moves on. A slow one ties up your resources while you wait.

Step 3 of 15 · 1 min

Timeouts

The first fix is the most basic: never wait forever. Every call to another service, database or cache needs a timeout.

Choosing the number

Too long, and you're back to tying up workers. Too short, and you fail requests that would have succeeded. A reasonable starting point is a little above the dependency's normal slowest responses: if the recommendation service's p99.9 latency (lesson 1.2) is 150 ms, a timeout of 250 to 300 ms lets nearly all normal requests finish and cuts off anything stuck.

Two kinds of timeout matter separately:

• Connect timeout: how long to wait to establish a connection. This should be short, tens or hundreds of milliseconds inside a data center, because a server that can't accept a connection quickly usually isn't going to.
• Request timeout: how long to wait for the response once connected.

Deadlines, not just timeouts

Inside a single user request, Chirp's API server called several services one after another, and some of those called others. Each call had its own timeout, but nothing tracked the total. A request could spend 2 seconds in one service, then start a fresh 2-second timeout in the next, and so on, long after the user's app had given up waiting.

The fix is a deadline: the request as a whole gets a time budget when it arrives ("this must finish within 800 ms"), and every call made on its behalf gets whatever is left of that budget, not a fresh one. Deadlines are passed along with each internal call, so a service three hops deep knows it has only 120 ms left and doesn't start a 2-second query. gRPC does this natively; with HTTP, it's a header your services agree on.

When the deadline has passed, stop working on the request. A result nobody is waiting for is wasted work, and under load, wasted work is what tips a system over.

Step 4 of 15 · 1 min

Retries, and how they multiply

Retrying failed calls is a good idea, used carelessly. Lesson 1.5 covered the basics for phones: only retry what's safe to repeat, and back off with randomness (jitter) so retries don't arrive in synchronized waves. Inside a system of many services, there's a further danger.

During the incident, here's what happened to one home screen request:

• The API server called the recommendation service. It timed out. The API server retried, 3 times.
• For each of those calls, the recommendation service ran its query, which timed out at the database. It retried, 3 times.

So one user request became up to 3 × 3 = 9 attempts at the slowest query in the system. With retries at three layers, it would have been 27. This is retry amplification: retrying at every layer multiplies load on the deepest layer, which is usually the one that's already struggling.

The rules that keep retries safe:

• Retry at one layer, usually the one closest to the user or the one that knows whether the operation is safe to repeat. Lower layers fail fast and report the error upward.
• Use a retry budget. Limit retries to a fraction of normal traffic, for example "retries may be at most 10% of requests to this service in the last minute." When things are healthy, the occasional failure gets retried. When everything is failing, the budget runs out quickly and retries stop, instead of tripling the load on a service that's already down. (Lesson 2.2 mentioned this for load balancers.)
• Back off with jitter, always.
• Don't retry what can't succeed: 400s, permission errors, or anything the dependency has told you to stop sending (429, or a 503 with Retry-After).

Step 5 of 15 · 1 min

Circuit breakers

Even with timeouts and limited retries, Chirp's API servers kept calling the broken recommendation service thousands of times a second, each call waiting 300 ms to time out. The service was obviously broken. Why keep asking?

A circuit breaker is a small piece of code wrapped around calls to a dependency that tracks how they're going and stops making them when they're clearly failing. The name comes from the electrical device that cuts the power when there's a fault. It has three states:

• Closed (normal): calls go through. The breaker counts failures and timeouts.
• Open: if the failure rate passes a threshold (say, more than 50% of calls in the last 10 seconds), the breaker opens, and calls fail immediately without reaching the dependency. No waiting, no workers tied up, no extra load on the struggling service.
• Half-open: after a cool-down (say, 30 seconds), the breaker lets a few trial calls through. If they succeed, it closes again. If they fail, it opens for another cool-down.

Circuit breakers help in two directions at once: they protect the caller (no workers stuck waiting) and they protect the dependency (it gets a chance to recover without being hammered). Michael Nygard's book Release It! popularized the pattern; most service frameworks and service meshes have one built in.

Step 6 of 15 · 1 min

Fallbacks: what to do instead

When a call fails fast (because of a timeout or an open breaker), the caller needs something to do instead. That's a fallback, and choosing it is a product decision as much as an engineering one:

• Leave it out. The home screen without "People you might know" is a perfectly good home screen. This was the right answer for recommendations, and it should have been there all along.
• Use a cached or stale answer. Show yesterday's suggestions, or the last value seen (lesson 2.4's stale-while-revalidate idea).
• Use a simpler default. A fixed list of popular accounts instead of personalized suggestions.
• Fail the request, clearly. For some things, like a payment or a password change, there's no sensible fallback, and saying "please try again" is correct.

The pattern that matters is graceful degradation: decide in advance which parts of each screen are essential and which are optional, and make sure an optional part failing never takes an essential part down with it. Chirp went through every screen and marked each element as one or the other.

Step 7 of 15 · 1 min

Bulkheads

Ships are divided into watertight compartments, called bulkheads, so a hole in one compartment doesn't sink the whole ship. The same idea applies to a server's resources.

During the incident, all calls from an API server to all dependencies shared one pool of workers. When the recommendation calls filled it, nothing else could run. With bulkheads, each dependency gets its own limited share: at most 20 concurrent calls to the recommendation service, at most 50 to the post store, and so on. If recommendations hang, they fill their 20 slots, further recommendation calls fail fast, and the other 180 workers carry on serving everything else.

Bulkheads can be separate thread pools, separate connection pools, or simply a counter of in-flight calls per dependency that rejects new calls past a limit.

Step 8 of 15 · 2 min

Load shedding

Everything so far protects a service from its dependencies. The last pattern protects a service from too many requests of its own.

When a service is overloaded, every request it accepts makes all the others slower (lesson 2.1's queueing table). Past a certain point, it's accepting work it can't finish before the caller gives up, so it spends all its effort on requests that will time out anyway. Throughput of useful work collapses even though the service is "busy."

Load shedding means rejecting excess requests quickly and early, with a 503, so the ones you do accept finish in time. A few practical forms:

• Limit in-flight requests per server. Past the limit, reject immediately.
• Drop requests that have already waited too long. If a request has been queued for longer than its deadline, the caller has already given up, so don't start it.
• Shed the least important work first. Chirp gave background and analytics traffic lower priority than user-facing requests, so under overload, analytics was rejected first.

Rejecting work feels wrong, but it's what lets a service keep serving most users instead of failing all of them slowly.

Systems that stay broken

There's a nastier version of overload, which researchers have called a metastable failure: a system gets pushed into overload by some trigger (a spike, a slow dependency, a cache emptying), and then stays overloaded after the trigger goes away, because the overload itself keeps generating extra load. Retries are the classic cause: timeouts cause retries, retries add load, load causes more timeouts. Lesson 2.4's cold cache is another: an empty cache sends everything to the database, which is too slow to refill the cache.

Chirp's Wednesday incident had this shape. Rolling back the recommendation service didn't fix it straight away, because the API servers' retries kept the recommendation service overloaded even after the slow query was gone. It took turning off retries entirely for a few minutes to break the loop. Retry budgets, circuit breakers and load shedding are what prevent this, because they all cut the feedback loop that keeps a system stuck.

Step 9 of 15 · 1 min

From the real world: DynamoDB's metastable morning

This lesson described metastable failures: overload that keeps itself going after the trigger is gone. The clearest public example is from AWS, in the early hours of Sunday, 20 September 2015.

Each DynamoDB storage server periodically asks a metadata service for its "membership": the list of table partitions it's responsible for. If it doesn't get an answer within a time limit, it stops serving requests and retries. Over the preceding months, customers had rapidly adopted a new feature, global secondary indexes, which made those membership lists much bigger. Answers took longer and longer to produce, creeping close to the time limit, without anyone noticing.

At 2:19 in the morning (Pacific time), a brief network disruption made many storage servers request their membership at once. The metadata service slowed, answers went past the time limit, and those servers stopped serving and retried. The retries kept the metadata service overloaded. Then healthy servers, checking in on their normal schedule, also got slow answers, timed out, and stopped serving too. Customer error rates settled around 55%, and AWS services built on DynamoDB (SQS, Auto Scaling, CloudWatch) suffered as well.

The network disruption was over in minutes. The overload wasn't, because the retries were now the load. At 5:06, engineers paused metadata requests entirely to stop the retry flood, added capacity while it was quiet, and re-enabled requests gradually. Errors fell to normal levels from about 7:10.

Things to recognise:

• The trigger was small and the cause was gradual: a time limit that silently got closer to the normal response time. Alert on response time as a fraction of its timeout.
• The fix was to stop the retries, not to add capacity under load. Adding capacity only worked once the retry storm was paused.
• Behaviour on failure ("stop serving and retry") turned one slow dependency into a fleet-wide outage. Where you can, keep serving with the last known good state (static stability, lesson 5.5).

Step 10 of 15 · 1 min

Going deeper: limits that find their own number

Load shedding needs a limit: "accept at most N requests at a time." The problem is choosing N. Set it from a load test, and it's wrong the day the code, the hardware or the traffic mix changes. Too high, and requests queue inside the service, so everything admitted is slow. Too low, and you reject traffic you could have served.

Adaptive concurrency limits let the service find N itself, by borrowing an idea from TCP congestion control. The signal is latency:

• Track the no-load latency: the fastest recent response time, roughly what a request takes with no queueing.
• If current latency is close to that, nothing is queueing, so it's safe to raise the limit a little.
• If current latency rises well above it, requests are waiting in line, so lower the limit.

The limit settles near the point where the service is fully busy but nothing is queueing, which is exactly where you want it. Requests above the limit are rejected immediately, which is far better than accepting them and timing out later. Netflix's open-source concurrency-limits library implements several variants (named after TCP algorithms: Vegas and gradient), and Envoy has an adaptive concurrency filter based on the same idea.

Two practical notes: reject with a clear "overloaded, retry later" status (HTTP 503 with Retry-After) so clients back off instead of retrying immediately, and give important traffic priority, so that when the limit shrinks, it's background work that gets shed first.

Step 11 of 15 · 1 min

What Chirp changed

• Every outgoing call has a timeout, set from the dependency's measured latency. Library defaults of "wait forever" are banned by a code check.
• Deadlines are set at the edge (800 ms for most API requests) and passed to every internal call.
• Retries at one layer only, with backoff, jitter and a 10% retry budget per dependency.
• Circuit breakers on every dependency, with alerts when one opens.
• Bulkheads: per-dependency limits on concurrent calls.
• Fallbacks for every optional element of every screen, decided with the product team.
• Load shedding on every service: in-flight limits, dropping requests past their deadline, and lower priority for background traffic.

The recommendation service had another bad deploy two months later. Its circuit breaker opened within ten seconds, the "People you might know" row disappeared from home screens, and nothing else was affected. Most users never noticed, which was the point.

Step 12 of 15 · 1 min

In the simulation

Chirp's epilogue, "incident day," starts with exactly this: a non-essential service slows down. Without timeouts, it takes down the home screen. With timeouts but retries at every layer, the system gets stuck in overload. With breakers, bulkheads and fallbacks, a row of suggestions quietly disappears.

Open the simulation →

Step 13 of 15 · 1 min

If you're here for an interview

Reliability follow-ups ("what happens when this service is slow?") are where many designs fall apart. Strong answers name the patterns and how they fit together:

• Timeouts on every call, and a deadline for the whole request.
• Retries only where safe, at one layer, with backoff, jitter and a budget.
• Circuit breakers to stop calling a failing dependency, plus fallbacks for what to show instead.
• Bulkheads so one dependency can't use up all the resources.
• Load shedding to stay useful under overload.

Saying "a slow dependency is more dangerous than a dead one," and explaining why, shows you understand the mechanism rather than the vocabulary.

Step 14 of 15 · 2 min

Try these yourself

Step 15 of 15 · 1 min

Further reading

• Marc Brooker, "Timeouts, retries, and backoff with jitter," Amazon Builders' Library.
• Michael T. Nygard, Release It!, on circuit breakers, bulkheads and stability patterns.
• Nathan Bronson et al., "Metastable Failures in Distributed Systems," HotOS 2021.
• Site Reliability Engineering (Google), chapters 21 and 22, on handling overload and cascading failures.
• Amazon Web Services, "Summary of the Amazon DynamoDB Service Disruption and Related Impacts in the US-East Region," September 2015.
• Netflix Technology Blog, "Performance Under Load" (2018), and the Netflix/concurrency-limits repository; Envoy documentation, "Adaptive Concurrency."