Step 2 of 17 · 1 min

What a load balancer does

A load balancer receives requests on one address and passes each one to one of several servers behind it. That's the core job, but in practice it does several things at once:

• Gives clients one stable address. Phones talk to api.chirp.app. Which servers sit behind it can change every minute without anyone noticing.
• Spreads load across servers, using a rule you choose (the next section).
• Keeps track of which servers are healthy, and stops sending traffic to ones that aren't.
• Hides server changes. Deploys, autoscaling and failures all happen behind it.
• Often terminates TLS (lesson 1.1), so the expensive handshake happens once at the load balancer rather than on every app server, and certificates live in one place.

Chirp used a load balancer managed by its cloud provider. You can also run one yourself with software like nginx, HAProxy or Envoy. The ideas are the same.

Step 3 of 17 · 1 min

Layer 4 and layer 7

You'll often hear load balancers described as "L4" or "L7". The numbers come from the old OSI model of networking. In practice they mean:

A layer 4 load balancer works with connections. It sees IP addresses and ports, and when a new TCP connection arrives, it picks a server and forwards that connection's packets there. It never looks inside the traffic, so it can't see URLs, headers or cookies. It's simple and very fast.

A layer 7 load balancer works with HTTP requests. It completes the TCP and TLS handshakes itself, reads each request, picks a server for that request, and forwards it, usually over a connection it keeps open to that server. Because it understands HTTP, it can route /v1/feed to one group of servers and /v1/uploads to another, add headers, retry a failed request on a different server, and report per-endpoint error rates. It costs more CPU per request than L4.

The difference that bites people most often is about long-lived connections. Lesson 1.1 showed why apps keep connections open and send many requests over them, especially with HTTP/2. An L4 load balancer chooses a server once per connection. If a phone opens one HTTP/2 connection and sends 500 requests over it, all 500 go to the same server. With a few thousand phones that mostly evens out, but if your clients are a handful of other services with a few long-lived connections each, one server can end up with most of the traffic while the others sit idle. An L7 load balancer chooses a server per request, so this doesn't happen.

Chirp's load balancer was L7, which is the usual choice for HTTP APIs.

Step 4 of 17 · 2 min

Choosing a server

Every load balancer needs a rule for which server gets the next request. The common ones:

Round robin. Server 1, 2, 3, 4, 1, 2, 3, 4… Simple and fair when servers are identical and requests take about the same time. It has no idea what's actually happening on the servers.

Weighted round robin. Same, but a server with twice the capacity gets twice the turns. Useful when the fleet has a mix of machine sizes.

Least connections (or least outstanding requests). Send the request to whichever server has the fewest requests in progress right now. This adapts to reality: a server stuck on slow requests naturally receives fewer new ones. It's usually a better default than round robin, and it's exactly the rule that went wrong on Thursday, because it can't tell a fast success from a fast failure.

Least response time. Prefer the server that's been answering fastest. Same blind spot.

Random. Pick any server at random. Surprisingly decent, but with many requests, some servers will get unlucky streaks.

Power of two choices. Pick two servers at random, and send the request to whichever of the two has fewer requests in progress. It sounds like a small tweak to random, but it makes a dramatic difference. A well-known result in computer science shows that with purely random placement, the busiest server ends up noticeably busier than average, and this extra load keeps growing as you add servers. Checking just two random servers and picking the less busy one shrinks that gap enormously. Checking three helps only a little more.

Power of two choices has another, more practical advantage. Big load balancers are really many machines, each with its own partial view of how busy every server is. If they all use "least connections," they can all spot the same idle server at the same moment and pile onto it together. Random pairs make that kind of stampede much less likely. That's why many modern load balancers and service meshes use it.

Hashing. Choose the server from a hash of something in the request, like the user ID. The same user always goes to the same server, which helps if each server caches per-user data. The catch is that when servers are added or removed, a naive hash reshuffles almost everyone. Lesson 5.2 shows how consistent hashing fixes that.

Here's a fleet of four servers, where server 4 is slower than the rest. Switch between the algorithms and watch the queues.

Round robin keeps sending server 4 its full share, so its queue builds up and the slowest requests get very slow. Least connections and two choices both notice the backlog and send it less.

Step 5 of 17 · 1 min

The fast-failure trap

Back to Thursday. Any rule that rewards servers for answering quickly has the same weakness: a server that fails instantly looks like the best server you have. It never has a backlog, and its response times are excellent. They're just all errors.

This isn't rare. Lots of failures are fast: a missing configuration value, a dependency that refuses connections immediately, a bug that throws an exception on the first line. A server in that state can soak up a large share of traffic in a fleet, which is what made a 25% problem into a 61% one.

Two defences:

Passive health checking, also called outlier detection. The load balancer watches the real responses going through it. If one server's error rate is much worse than the others, for example five 5xx responses in a row, or an error rate over 50% in the last 30 seconds, it stops sending that server traffic for a while, then tries it again. This reacts within seconds, using traffic that's happening anyway. Envoy, HAProxy and most service meshes support it, and many cloud load balancers have some version.

Health checks that test what matters. Which is the next section.

Step 6 of 17 · 2 min

Health checks

Besides watching real traffic, a load balancer can actively check each server: every few seconds, it requests a URL such as /health. If a server fails a few checks in a row, it's taken out of rotation. If it passes a few in a row, it comes back.

The timing settings add up. Chirp's check ran every 10 seconds and needed 3 failures before removing a server, so a server that died completely kept receiving traffic for up to 30 seconds. Faster checks find problems sooner, but are more likely to remove a server that just had a brief hiccup.

The harder question is what the health check should actually check.

Too shallow

Chirp's original /health endpoint just returned 200 OK. It proved the process was running and could answer HTTP. It said nothing about whether the server could do real work, and server 3 passed it for fourteen minutes while failing every real request.

Too deep

After Thursday, someone rewrote /health to be thorough. It loaded the configuration, queried the database, and checked Redis. If any of them failed, it returned 503 Service Unavailable.

That caught a broken server 3 immediately. Three weeks later, it caused Chirp's worst outage of the year.

A long-running analytics query slowed the database down for about four minutes. Real requests got slower, but most still worked. The health checks, though, had a short timeout. One after another, every server's /health timed out waiting for the database, and the load balancer removed every server from rotation. A slow database became a complete outage, because every server failed its check for the same shared reason at the same moment.

The general principle: a health check should answer "is something wrong with this server?", not "is something wrong anywhere?" If a dependency that every server shares is down, removing servers doesn't help, since there's no healthy server to send traffic to instead. All it does is turn a partial failure into a total one.

Some managed load balancers protect against this by failing open: if every server is unhealthy, they send traffic to all of them anyway, on the reasoning that it's better than sending it nowhere. It's a useful safety net, but it's better not to need it.

About right

Many systems separate two different questions, a split popularized by Kubernetes:

• Liveness: is this process stuck beyond repair? If yes, restart it. This should be very simple, because restarting a healthy server by mistake is costly.
• Readiness: should this server receive traffic right now? It checks things that belong to this server: it finished starting up, its configuration loaded, its own connection pool works. It flips to "not ready" while the server is shutting down. It doesn't fail just because a shared database is slow.

Combine a readiness check with passive outlier detection, and you catch both kinds of failure. Broken servers fail their readiness check or get ejected for their error rate. Shared problems leave every server in place and serving, which is what you want, because a slow app usually beats no app.

Step 7 of 17 · 1 min

Deploys without dropped requests

Lesson 2.1 promised that scaling out makes deploys painless. The load balancer is what makes that true, if a few things happen in the right order.

When a server is about to be replaced:

1. Mark it not ready. Its readiness check starts failing, so the load balancer stops sending it new requests.
2. Let in-flight requests finish. The load balancer keeps the server's existing connections open for a while. This is called connection draining or a deregistration delay, and something like 30 seconds is common. It should be longer than your slowest normal request.
3. Shut down gracefully. The app finishes its last requests and exits (lesson 2.1).
4. Start the new version, and wait until it's ready before moving to the next server.

Skip step 1, and the load balancer keeps sending requests to a server that's in the middle of shutting down. Those users see errors. Chirp's early deploys produced a small burst of 502s every time for exactly this reason.

There's one more subtlety: a freshly started server is often slower than one that's been running for a while. Its caches are empty, and some runtimes are still warming up. Some load balancers support a slow start setting, which ramps traffic to a new server up gradually over a minute or so instead of giving it a full share at once.

Step 8 of 17 · 1 min

Timeouts, keep-alive and retries

The mysterious 502s

A few weeks after moving to the load balancer, Chirp noticed a strange pattern: a small number of 502 Bad Gateway errors, a few per minute, spread across all servers, with nothing wrong in the app logs.

The cause was a mismatch between two timeouts. The load balancer kept idle connections to each server open for 60 seconds, so it could reuse them. The app servers closed idle connections after 5 seconds. Every so often, the load balancer picked a connection that it thought was still open, sent a request down it, and at that exact moment the server closed the connection. The request failed with a 502.

The fix is a simple rule: the server's keep-alive timeout must be longer than the load balancer's idle timeout, so it's always the load balancer that closes idle connections. Chirp set the app servers to 75 seconds and the 502s stopped.

Retries at the load balancer

An L7 load balancer can retry a failed request on a different server. That's helpful when one server has a problem. It's also dangerous, for the reasons in lesson 1.5:

• Only retry requests that are safe to repeat. Retrying a GET is fine. Retrying a POST /v1/posts can create duplicate posts, unless it carries an idempotency key that the server checks.
• Limit how much retrying can happen. If every server is overloaded, retrying every failed request adds even more load to an already overloaded system. Many load balancers support a retry budget, for example "retries may add at most 20% extra traffic."

Step 9 of 17 · 1 min

Knowing who the user is

Once requests pass through a load balancer, the app server sees every connection coming from the load balancer's IP address, not the user's. That breaks anything that uses the client's IP: rate limiting, abuse detection, rough location.

L7 load balancers solve this by adding a header, usually X-Forwarded-For, containing the original client IP. L4 load balancers can use the PROXY protocol to pass it along. Either way, only trust that header when it comes from your own load balancer. Anyone can send a request with a fake X-Forwarded-For header, so if your servers can be reached directly, an attacker can claim any IP they like.

Step 10 of 17 · 1 min

Who balances the load balancer?

If every request goes through the load balancer, it's a single point of failure too. Lesson 2.1 all over again.

Managed load balancers handle this for you: what looks like one load balancer is really a group of machines across several availability zones, and the provider replaces failed ones. If you run your own, you typically run at least two, and use one of these to send traffic to whichever are healthy:

• DNS with several addresses, one per load balancer. Simple, but clients and resolvers cache DNS answers (lesson 1.1), so it reacts slowly when one dies.
• A floating IP address that automatically moves to a standby machine when the active one stops responding.
• Anycast, where many machines in different places announce the same IP address and the internet routes each user to the nearest one. This is how large CDNs and DNS providers work, and it's coming back in lesson 2.3.

Step 11 of 17 · 1 min

From the real world: the health check that took the site down

This lesson warned that a health check can remove every healthy server at once. Stack Overflow had a very pure example of it on 20 July 2016.

Someone posted a question containing about 20,000 consecutive whitespace characters on one line. That's strange but harmless text. To render posts, Stack Overflow ran a regular expression that trimmed whitespace from the ends of lines, essentially \s+$.

For most text, that regex is instant. For this one, it isn't. The engine starts at the first space and matches all 20,000 spaces, then finds the line doesn't end there (there's another character after them), so it fails. Then it starts again at the second space and does it all again, and so on. That's 20,000 + 19,999 + 19,998 + … checks, about 200 million, to render one post.

The post appeared on the home page, so every web server rendered it, and each render pinned a CPU. And the home page was what the load balancer's health check requested. Health checks started timing out on every server, the load balancer took them all out of rotation, and Stack Overflow was down. The servers hadn't crashed. They were busy rendering the one page that was used to judge whether they were alive.

The team found it within about 10 minutes, replaced the regex with a simple loop that trims from the end of the string, and had the site back about 34 minutes after it started.

Lessons that apply directly to Chirp:

• A health check should test whether the server can serve, not whether your most complicated page works. A dedicated /ready endpoint that checks local things (as this lesson recommended) wouldn't have failed.
• The busiest page is a shared dependency. Anything on the home page is rendered by every server, so one bad piece of content becomes a fleet-wide problem.
• Super-linear work on user input (regex, parsing, sorting) needs limits. It's the same lesson as Cloudflare's outage in lesson 1.2.

Step 12 of 17 · 1 min

Going deeper: one connection, a million requests

Chirp's services started talking to each other over gRPC, which runs on HTTP/2. Soon after, the team noticed that adding servers to the feed backend didn't help: one backend sat at 90% CPU while the new ones were nearly idle.

The cause is the difference between balancing connections and balancing requests. A layer-4 load balancer (lesson's "Layer 4 and layer 7") picks a backend when a connection opens and then forwards that connection's bytes there for as long as it lives. With HTTP/1.1, that's fine: clients open many short connections, so connections and requests are roughly the same thing.

HTTP/2 changed that. It sends many requests over one connection at the same time, and a client typically keeps that connection open for hours. A busy service with a handful of long-lived connections can put most of its traffic on one backend, and a backend added later receives no connections at all until something reconnects.

There are three common fixes:

• Use a layer-7 load balancer that understands HTTP/2. It terminates the client's connection, keeps its own connections to every backend, and balances each request separately. Envoy, NGINX, HAProxy and the cloud providers' application load balancers can all do this.
• Balance on the client side. gRPC clients can resolve all backend addresses, open a connection to each, and spread requests themselves (gRPC's round_robin policy, or a service mesh sidecar that does the same).
• Make connections expire. Servers can tell clients to reconnect every few minutes (gRPC's MAX_CONNECTION_AGE), which slowly re-spreads connections, including onto new backends. It's crude but effective, and it also helps deploys drain.

The general lesson: a load balancer can only balance the units it can see. If one unit carries a huge, variable amount of work, balancing the units won't balance the work.

Step 13 of 17 · 1 min

What Chirp changed

• Turned on outlier detection: a server is ejected after 5 consecutive 5xx responses, then retried after 30 seconds.
• Replaced /health with a readiness check covering only server-local things (startup finished, configuration loaded, local pool working).
• Switched the balancing rule to power of two choices.
• Added deregistration delay and graceful shutdown to every deploy, plus a one-minute slow start for new servers.
• Set the app's keep-alive timeout to 75 seconds, above the load balancer's 60.
• Allowed the load balancer to retry only GET requests and requests with idempotency keys, within a retry budget.
• Started alerting on error rate per server, not just for the whole fleet. A single bad server shows up immediately.

The next bad deploy, a month later, failed fast on one server just like before. Outlier detection removed it from rotation within four seconds. The fleet's error rate peaked at 0.3%.

Step 14 of 17 · 1 min

In the simulation

The Stage 1 dashboard shows your load balancer's choices: the balancing rule, the health check type, and whether outlier detection is on. Stage 1 includes a bad deploy and a slow-database event. See whether your settings turn either one into an outage.

Open the simulation →

Step 15 of 17 · 1 min

If you're here for an interview

"Put a load balancer in front" is a box in every diagram. The follow-ups are where you show you've run one:

• "How does it know a server is unhealthy?" Talk about active readiness checks and passive outlier detection, and why the check shouldn't fail because of a shared dependency.
• "Which algorithm?" Least outstanding requests or power of two choices for most HTTP services, and why plain round robin struggles with uneven requests.
• "Isn't the load balancer a single point of failure?" Several instances across zones, with DNS, a floating IP or anycast in front.
• "How do you deploy without errors?" Readiness flips, draining, graceful shutdown, then the next server.

Mentioning the fast-failure trap, unprompted, is a strong signal. Few candidates know it.

Step 16 of 17 · 2 min

Try these yourself

Step 17 of 17 · 1 min

Further reading

• Michael Mitzenmacher, "The Power of Two Choices in Randomized Load Balancing," IEEE Transactions on Parallel and Distributed Systems, 2001.
• Site Reliability Engineering (Google), chapters 19 and 20, on load balancing at the front end and inside the data center. Free online.
• Envoy proxy documentation on outlier detection and health checking. Clear, with good diagrams.
• Kubernetes documentation on liveness, readiness and startup probes.
• Stack Exchange, "Outage Postmortem – July 20, 2016" (stackstatus.tumblr.com).
• gRPC blog, "gRPC Load Balancing" (2017), on why connection-level balancing fails for gRPC and the client-side alternatives.
• Envoy documentation, "HTTP/2 and gRPC" load-balancing notes.