Step 2 of 16 · 1 min

Two ways to get more capacity

When one machine can't keep up, there are two ways to get more:

• Scale up (also called vertical scaling): replace the machine with a bigger one. More CPU cores, more memory, faster disks.
• Scale out (also called horizontal scaling): add more machines of the same size, and spread the work across them.

People sometimes talk about scaling up as the naive option and scaling out as the grown-up one. That's not right. Each has real strengths, and good engineers reach for scaling up more often than you might think. The skill is knowing where each one stops working.

Step 3 of 16 · 2 min

Scaling up is underrated

Scaling up has one enormous advantage: nothing about your code changes. Your app still runs on one machine. There's no network between parts of it, nothing to keep in sync, no new ways for things to fail. Every problem in the rest of this lesson simply doesn't exist.

Machines also go much bigger than most people realize. The largest general-purpose machines you can rent from the big cloud providers have hundreds of CPU cores and several terabytes of memory. Plenty of well-known services have been run from a surprisingly small number of large servers for a long time.

It's also often the best answer for databases, which are much harder to spread across machines than application servers are. You'll see in Part 5 how much work it takes to split a database. Many teams scale their database up for years before they need to do that.

So when Chirp's server was overloaded, resizing it was the right first move. It took an hour and bought time.

Where it stops

Scaling up has limits, and Chirp hit three of them in the following three weeks.

A single machine is a single point of failure. Every problem with that machine is an outage for the whole app: a hardware fault, a kernel update that needs a restart, a full disk, a bad configuration change. Chirp found this out on a Saturday morning, when the cloud provider emailed to say the physical host under their server was being retired. The server was stopped and moved, and Chirp was down for 20 minutes. Nothing was wrong with Chirp's code. The machine just had to go somewhere else.

Resizing usually means a restart. Going from 32 cores to 64 would mean another outage. With a growing user base, there's no good time for that.

There's a ceiling. Even the largest machine has a limit, and you're paying for the peak all day long. Chirp's evening peak was about three times its afternoon traffic, so for most of the day two thirds of that big machine sat idle.

There's a quieter fourth limit too: not all software uses extra cores well. Chirp's backend was written in Python, which runs one thread of Python code at a time per process. To use 32 cores, they already had to run many separate processes on the machine. That's a small taste of scaling out, inside a single box.

Step 4 of 16 · 1 min

Scaling out: same app, more machines

Scaling out means running several copies of the application on several machines, with a load balancer in front that spreads incoming requests across them. (Lesson 2.2 is all about load balancers. For now, treat it as something that receives every request and passes it to one of the servers.)

The benefits are the mirror image of scaling up's limits:

• Survive failures. If one of four servers dies, the other three carry on. Users might not notice.
• Change things without downtime. You can restart, update or replace servers one at a time while the others keep serving. This is how rolling deploys work.
• Grow in small steps. Need 20% more capacity? Add one more server, not a machine twice the size.
• Match the daily cycle. Run six servers in the evening and two at 4 am. This is autoscaling, which we'll come back to.

But there's a catch, and it's the real subject of this lesson. The moment there are two copies of your app, everything that assumed there was only one breaks. Chirp found five such things in its first week on two servers.

Step 5 of 16 · 5 min

What broke when there were two

1. People kept getting logged out

When you log in to Chirp, the server creates a session: a record saying "this browser belongs to Priya," with a random ID that goes into a cookie. The first version of Chirp kept sessions in the server's memory. With one server, that was fine.

With two, Priya logged in on server A, and A stored her session. Her next request went to server B, which had never heard of her session ID, so it treated her as logged out. Depending on which server each request happened to reach, she'd be logged in, then out, then in again.

There are three ways to fix this:

Keep sessions in a shared store. Both servers read and write sessions in a store they share, usually Redis. Any server can handle any request, because the session is no longer inside one of them. This adds a network round trip per request (half a millisecond, from lesson 1.2), and the session store becomes something that must stay up.

Put the session in the token itself. Instead of a random ID that points to stored data, the server gives the browser a signed token containing the user's ID and an expiry time, signed with a secret key only the servers know. Any server can check the signature and trust the contents, without looking anything up. JSON Web Tokens (JWTs) are a common format for this. The trade-off is that you can't easily take a token back. If Priya's phone is stolen and she clicks "log out everywhere," a token that's valid for 30 days stays valid for 30 days, unless you add a check against a list of revoked tokens (which brings back the shared store). A common compromise is a short-lived token (say 15 minutes) plus a longer-lived refresh token that is checked against a database whenever a new short-lived token is issued.

Sticky sessions. Tell the load balancer to always send a given user to the same server. This hides the problem rather than fixing it. Load spreads unevenly, because some servers end up with heavier users. When a server dies or restarts during a deploy, everyone on it is logged out anyway. And it quietly lets the rest of the app keep depending on per-server memory. Chirp tried it for one day and moved sessions to Redis.

2. Half the photos were missing

Photos were uploaded to the app server and saved to its local disk. With two servers, a photo uploaded through A existed only on A's disk. Every request for it that landed on B got a 404. About half of all photo views failed.

The fix is to store files somewhere every server can reach, and that isn't a local disk. Object storage, the subject of lesson 2.3, was built for exactly this.

3. Everyone got the daily digest twice

Chirp sends a "here's what you missed" email every morning at 8. A scheduled task on the server (a cron job) did it. Once there were two servers, there were two cron jobs, and every user got two identical emails. A few hundred people unsubscribed that morning.

Scheduled work needs to run exactly once, no matter how many app servers there are. Some ways to get that:

• Run scheduled jobs in one place. A small separate worker process, not part of the web servers, is responsible for scheduled tasks. Simple, but if that one worker is down at 8 am, nothing runs.

• Take a lock first. Each server tries to take a lock before running the job, and only the one that gets it proceeds. In Redis, that's one command:

  SET lock:daily-digest:2026-06-14 server-a NX PX 600000

  NX means "only set it if it doesn't exist," and PX 600000 makes the lock expire after ten minutes, so a crashed server doesn't hold it forever.

• Make the job itself safe to run twice. Before sending a digest, record "digest for user 4821 on June 14 was sent" in a table with a unique constraint. A second run finds the row already there and skips that user.

Chirp did the last two together, and that's the important lesson. Locks with expiry times have an awkward failure mode: if a job runs longer than its lock lasts (a slow database, a long garbage-collection pause), the lock expires while the first server is still working, and a second server starts the same job. The lock reduces duplicates; the unique constraint guarantees there aren't any. Lesson 6.5 comes back to why locks across machines are harder than they look.

4. Limits and caches that were only per server

Chirp limited each account to 100 posts an hour, to slow down spammers. The counter lived in server memory. With two servers, each kept its own count, so the real limit became 200. With six servers in the evening, 600. A spammer found this within a week.

Small in-memory caches had the same issue in a milder form. Each server had its own copy of popular profiles, cached for 60 seconds. After someone changed their display name, server A might show the new one while server B still showed the old one, so the name seemed to flicker between the two on refresh.

Counters that have to be correct across all servers need to live in a shared store. Lesson 6.3 covers rate limiting properly, and lesson 2.4 is about caching. Small per-server caches are still fine for data that can be a little stale, as long as you accept that different servers will briefly disagree.

5. Work that disappeared when a server stopped

After a new post was saved, the first version of Chirp started a background thread to send push notifications, then immediately returned "post created" to the app. It made posting feel fast.

During rolling deploys, servers were stopped and replaced. Any notification threads that were still running on a server when it stopped were simply gone. Some followers never got notified. Nobody noticed for weeks, because nothing logged an error: the process that would have logged it was the one that had been stopped.

Two fixes:

• Shut down gracefully. When a server is told to stop (on Linux, by a SIGTERM signal), it should stop accepting new requests, finish the ones in progress, then exit. Lesson 2.2 covers how the load balancer helps with this.
• Don't keep important work in memory. Anything that must happen after the response is sent goes into a durable queue, where any worker can pick it up and retry it if needed. Part 4 is all about this.

Step 6 of 16 · 1 min

What "stateless" actually means

All five problems have the same root cause: the app server was holding state, information that outlives a single request: sessions, files, scheduled jobs, counters, unfinished work.

The usual advice is "make your servers stateless," which sounds as if state disappears. It doesn't. Chirp still has sessions, photos and counters. They've just moved out of the app servers and into systems built to hold them:

A useful working definition: a server is stateless if any request can go to any server, and any server can be destroyed at any moment without losing anything that matters. Once that's true, servers become interchangeable. You stop caring about any individual machine, because none of them hold anything you'd miss. People sometimes put this as treating servers like cattle, not pets.

This is also why "just add more servers" works so smoothly for application servers and so poorly for databases. The application tier became easy to copy by pushing all its state into the database and a few other stores, which now have to cope with the combined load from every app server.

Step 7 of 16 · 2 min

How many servers do you need?

With interchangeable servers, sizing becomes arithmetic. You need three numbers:

• Peak requests per second. Chirp's evening peak at 100,000 daily users was about 180 requests per second.
• CPU time per request. Chirp measured this from its logs: about 40 milliseconds of CPU per request on average (Python, an ORM, building JSON).
• Cores per server. Chirp used servers with 4 cores.

At peak, the work in progress is (Little's Law, from lesson 1.2):

180 requests/s × 0.040 s of CPU each = 7.2 cores busy

So you'd think two 4-core servers (8 cores) would do it. That would be a mistake, for two reasons.

Why you never run servers at 100%

Requests don't arrive evenly. Even if a server is busy only 80% of the time on average, there are moments when several requests arrive together and have to wait in line. As a server gets busier, those waits grow, and they grow much faster than the load does.

A classic simple model of a queue (it's called M/M/1, and real servers are more complicated, but the shape holds) says that the average time a request spends in the system is:

response time = service time ÷ (1 − utilization)

Going from 50% to 90% busy doesn't make requests 80% slower. It makes them five times slower. That's why Chirp's pages went from 60 ms to two seconds at 95% CPU: the server wasn't doing more work per request, it was making requests wait.

So teams aim to keep servers somewhere around 50–70% busy at peak, leaving room for bursts.

Planning for a server to fail

The second reason: if Chirp needs every server it has, losing one means overload. So you add at least one spare. This is called N+1: enough servers to handle peak with N, plus one more.

Putting it together for Chirp:

Work at peak:             7.2 cores
Usable per server:        4 cores × 60% target = 2.4 cores
Servers needed:           7.2 ÷ 2.4 = 3
Plus one spare (N+1):     4 servers

Four 4-core servers cost about the same as the one 16-core server Chirp might otherwise have used, and they can survive a failure, a deploy, or a bad configuration on one machine.

Spread them out

Four servers in the same rack share the same power and network switch. Cloud providers group their data centers into availability zones: separate buildings with separate power and networking, close enough for low latency between them (typically a millisecond or two). Putting Chirp's servers in at least two zones means one building losing power doesn't take all of them down.

Step 8 of 16 · 1 min

Autoscaling

Once servers are interchangeable, you can add and remove them automatically. Chirp set up a rule: keep average CPU across the fleet near 55%, with at least 2 servers and at most 10.

It works well with a few caveats:

• New servers aren't instant. Starting a machine, installing the app and warming it up took Chirp about three minutes. If traffic doubles in thirty seconds (say, at the end of a cricket match), autoscaling reacts too late. For predictable peaks, schedule extra capacity ahead of time: Chirp starts scaling up at 7:30 pm every evening.
• Scaling in needs care. Removing a server must use the graceful shutdown from earlier, or in-flight requests fail.
• Don't flap. Without a cool-down period, the fleet can bounce between 4 and 5 servers every few minutes. Most autoscalers have a setting for how long to wait after a change.
• Autoscaling can't fix a shared bottleneck. If the database is the slow part, adding app servers only adds more connections to the database (lesson 1.4) and makes things worse.

Step 9 of 16 · 1 min

The part that doesn't scale out (yet)

After all this, Chirp's application tier could grow almost without limit. The database couldn't. Every app server sends queries to the same single Postgres server, so doubling app servers doubles the load on one machine.

At 100,000 users, that database is fine. Scaling it up (a bigger machine) is the right move for now, and the next lessons cover what comes next: caching to avoid queries altogether (lesson 2.4) and replicas to spread reads across several databases (lesson 2.5). Spreading writes across machines is much harder, and that's what Part 5 is for.

Step 10 of 16 · 2 min

From the real world: when autoscaling made it worse

Autoscaling looks like the answer to everything in this lesson: watch a metric, add servers when it's high, remove them when it's low. On 4 January 2021, Slack's autoscaling turned a network problem into a much longer outage, and the reasons are worth knowing before you trust it.

It was the first working day after the holidays. Clients that hadn't been opened for two weeks had cold caches, so they pulled more data than usual, right at the morning peak. Slack's networks inside AWS were connected through AWS Transit Gateways, which are supposed to scale up automatically. They didn't scale fast enough, and started dropping packets.

Then Slack's own systems reacted, each sensibly by its own rules:

1. Web servers spent their time waiting on the network, so their CPU use went down. The autoscaler, which used CPU as one signal, removed servers.
2. Then worker threads filled up, all waiting, and the other signal (thread use) shot up. The autoscaler swung the other way and tried to add about 1,200 servers in fourteen minutes.
3. The service that sets up each new server had limits nobody had hit before. It ran out of Linux open-file handles, and hit an AWS quota. Most new servers never finished setup, and the group hit its configured maximum size.
4. Slack's dashboards ran in a different network, behind the same gateways, so the people fixing it lost their main view of the system.

The provisioning service recovered around 08:15 and enough healthy servers were serving by about 09:15. The network itself was back to normal at 10:40, after AWS raised the gateways' capacity by hand.

What to take for Chirp's design:

• A scaling signal can lie when the bottleneck is somewhere else. Low CPU meant "waiting," not "idle." Scale on something closer to what users feel (requests in flight, latency), and know what each signal means when a dependency is slow.
• Scaling out depends on a scale-out machine: provisioning, images, quotas, service discovery. That machine only runs at full speed during emergencies, so load-test it.
• Rate-limit your own autoscaler, and give it a floor, so a confused signal can't remove half your fleet or request a thousand servers at once.
• Keep your monitoring off the path it monitors.

Step 11 of 16 · 1 min

Going deeper: why the tenth server adds less than the second

The "how many servers" section treated capacity as linear: one server does 1,200 requests a second, so ten do 12,000. For truly stateless servers that's close. But every system has parts the servers share, and those parts bend the line. Neil Gunther's Universal Scalability Law gives the shape with two numbers:

throughput(N) = N ÷ (1 + σ(N − 1) + κN(N − 1))

• σ (contention) is the fraction of work that has to wait its turn for something shared: a lock, a single hot database row, a single queue. It's Amdahl's law: if 5% of each request is serialized, you can never get more than 20× one server, no matter how many you add.
• κ (coherence) is the cost of servers keeping each other up to date: invalidating each other's caches, agreeing on cluster membership, gossip. Every server has to hear from every other, so this cost grows with the number of pairs, roughly N². Past a certain size, adding a server makes the whole system slower.

With σ = 5% and κ = 0.2%, throughput peaks around 22 servers and then falls. Those numbers aren't exotic. A cluster where every node sends a heartbeat to every other node, or a cache layer where each write must invalidate on every server, has exactly this shape.

Two practical uses:

• Measure it. Run a load test at 1, 2, 4 and 8 servers, fit σ and κ to the results, and you'll know where your ceiling is before production finds it. It's a few hours' work.
• Design the coordination out. Make shared things partitionable (Part 5) so contention is per partition, not global. Replace all-to-all communication with something hierarchical or sharded. AWS's Kinesis outage in lesson 4.1 is a real case of κ: every front-end server kept a thread for every other front-end server.

Step 12 of 16 · 1 min

What Chirp changed

• Moved from one 32-core server to four 4-core servers across two availability zones, behind a load balancer.
• Moved sessions to Redis, and photos to object storage.
• Moved the daily digest to a separate worker, with a Redis lock and a unique (user_id, digest_date) constraint.
• Moved rate-limit counters to Redis.
• Added graceful shutdown, and a rule: nothing important runs in a background thread.
• Set up autoscaling on CPU, with a scheduled scale-up before the evening peak.

Their first rolling deploy afterwards, on a Tuesday evening at peak traffic, was the first time they'd shipped code without either waiting until midnight or causing an outage.

Step 13 of 16 · 1 min

In the simulation

Stage 1's decision D3 asks how Chirp should handle growing load. "Bigger server" is a real option, and it works for a while. Watch what happens to it in Stage 3, when a server fails at the worst possible moment. And if you choose to scale out, the simulation checks whether you also moved the state, because it will send the same user to different servers.

Open the simulation →

Step 14 of 16 · 1 min

If you're here for an interview

Almost every design interview includes "and then we put the app servers behind a load balancer." The candidates who stand out say one or two more sentences:

• Say the servers are stateless, and where the state went. "Sessions in Redis, files in object storage, so any server can handle any request."
• Size the fleet out loud. Peak requests × time per request ÷ usable capacity per server, plus a spare. Mention that you don't plan to run at 100%.
• Know when scaling up is fine. Saying "for the database, I'd scale up first, it's much simpler than sharding" shows judgment, not laziness.

A common follow-up is "how would you make sure a scheduled job runs only once?" A lock alone is a weak answer. A lock plus an idempotent job is a strong one.

Step 15 of 16 · 2 min

Try these yourself

Step 16 of 16 · 1 min

Further reading

• The Twelve-Factor App (12factor.net), especially the sections on processes, disposability and backing services. Short and still relevant.
• Martin Kleppmann, Designing Data-Intensive Applications, Chapter 1, on scalability and describing load.
• Neil Gunther's work on the Universal Scalability Law, for why adding machines gives less than linear gains once they have to coordinate.
• Your cloud provider's documentation on availability zones and autoscaling policies.
• Slack Engineering, "Slack's Outage on January 4th 2021," 2021.
• Neil J. Gunther, Guerrilla Capacity Planning (Springer, 2007), for the Universal Scalability Law, with worked examples of fitting σ and κ.
• Baron Schwartz, Practical Scalability Analysis with the Universal Scalability Law (free ebook, VividCortex), a shorter, hands-on introduction.