Step 2 of 15 · 1 min

The numbers

Here's roughly how long common operations take on today's hardware. Don't memorize the exact values. They change every few years, and they vary between machines. What matters is the order of magnitude, and how many times bigger each row is than the one above it.

(ns is a nanosecond, a billionth of a second. µs is a microsecond, a millionth. ms is a millisecond, a thousandth.)

Those numbers are so small that the gaps between them stop meaning anything. A trick that helps: pretend that reading from the L1 cache takes one second and scale everything else up by the same amount.

So when the "Who to follow" loop made 150 trips to a database 23ms away, it was the equivalent of making 150 trips that each took most of a year, when the data could have been fetched in one.

Step 3 of 15 · 1 min

Pattern 1: memory is fast, everything else is far away

Reading something from RAM is about a thousand times faster than reading it from an SSD, and thousands of times faster than asking another machine for it. That single fact is the reason caching exists, and a large part of this course is about caches in one form or another.

It's also why databases try very hard to keep frequently used data in memory. When Chirp's database is small, the whole thing fits in RAM and every query is fast. As it grows past the size of memory, some queries start hitting the disk, and you'll see response times jump even though the code hasn't changed. Lesson 1.4 looks at what's happening inside the database when that happens, and lesson 2.4 is about building caches on purpose.

Step 4 of 15 · 1 min

Pattern 2: in a row beats all over the place

Look at two rows from the table again: a random read from an SSD takes about 100µs, while reading a full megabyte sequentially takes about 500µs. A megabyte is 256 blocks of 4 KB. Reading those 256 blocks scattered across the disk would take 256 × 100µs, about 25ms. Reading them in a row takes 0.5ms. That's about 50 times faster for the same amount of data.

On a spinning hard drive the gap is even bigger, because each random read means physically moving an arm across the disk.

Storage systems are built around this. Databases write changes to a log first, appending to the end of a file, because appending is sequential (lesson 1.4). Some databases are designed so that all writes are sequential, which is why they handle heavy write loads so well (lesson 3.2). When you see a system design that seems strangely obsessed with appending to the end of things, this is why.

Step 5 of 15 · 1 min

Pattern 3: the trip costs more than the work

This is the one that caught Chirp. A simple database query by primary key does maybe 50 microseconds of actual work. The round trip to reach the database is 500 microseconds, ten times more, even inside the same data center. Across regions, it's hundreds of times more.

So the cost of talking to another machine is mostly the number of trips, not the amount of data in each one. The fix is almost always to make fewer, bigger trips:

suggestions = db.query("SELECT id FROM users WHERE ... LIMIT 50")
ids = [s.id for s in suggestions]

profiles  = db.query("SELECT * FROM users WHERE id = ANY(%s)", ids)
followers = db.query("""
    SELECT followee_id, count(*) FROM follows
    WHERE followee_id = ANY(%s)
    GROUP BY followee_id
""", ids)

Three trips instead of 150. Here's the difference, racing side by side. Try moving the database further away and see which version cares.

This pattern of one query to get a list and then one more query per item is common enough to have a name: the N+1 query problem. ORMs (libraries that let you treat database rows as objects) make it very easy to write without noticing, because user.profile.follower_count looks like reading a field, not making a network call.

The same idea applies everywhere you talk to another machine: fetching many cache keys at once instead of one by one, sending writes in batches, and designing APIs so the app can get everything it needs in one call (lesson 1.5). Round trips are a theme you'll see in every lesson of this course.

Step 6 of 15 · 2 min

Latency and throughput are different things

Two words get mixed up constantly, and the difference matters as soon as more than one person is using your app.

• Latency is how long one thing takes. "A feed request takes 20ms."
• Throughput is how many things get done per second. "The server handles 2,000 feed requests per second."

They're related, but you can't get one from the other without knowing how many things happen at the same time. A highway's speed limit (latency) doesn't tell you how many cars it carries per hour (throughput); that also depends on how many lanes it has.

There's a simple and very useful formula connecting them, called Little's Law:

things in progress = throughput × latency

If Chirp's server handles 1,000 requests per second and each takes 20ms, then on average 1,000 × 0.020 = 20 requests are in progress at any moment.

That sounds academic until it breaks something. Here's the second half of Chirp's Wednesday.

Each request held a database connection while it ran. The app had a connection pool, a fixed set of 20 open connections that requests take turns using, because opening a new database connection for every request is slow (it's the handshake problem from lesson 1.1 again). With the database half a millisecond away, a typical request held a connection for about 5ms. At 200 requests per second, that's 200 × 0.005 = 1 connection busy on average. Twenty was plenty.

When the app was accidentally pointed at the faraway staging database, each request held a connection for well over 100ms, because every query inside it took 23ms. Now 200 × 0.1 = 20 connections busy on average, which is the entire pool. Requests started waiting for a connection before they could even begin, and that waiting added more latency, which kept connections busy longer, which made the wait longer still.

So the slowness didn't stay in the "Who to follow" page. It spread to every page that needed the database, which was every page.

The general lesson: when something gets slower, everything that's holding a resource while it waits needs more of that resource. You'll use Little's Law to size connection pools, worker counts, and queue capacity throughout the course.

Step 7 of 15 · 1 min

Averages lie

When the team first looked at their dashboard on Wednesday, the average response time across the whole app was about 180ms. That didn't sound too bad. But the average blended thousands of fast requests to other pages with the "Who to follow" requests taking several seconds. The people opening that page were having a terrible time, and the average hid it.

That's why engineers look at percentiles instead:

• p50 (the median): half of requests are faster than this.
• p99: 99% of requests are faster than this. 1 in 100 is slower.
• p99.9: 1 in 1,000 is slower.

On that Wednesday, Chirp's p50 was 25ms, and its p99 was 3.6 seconds. Those two numbers tell the real story; the average didn't.

It's tempting to think the slowest 1% doesn't matter much. Two reasons it does:

• Your most active users hit it the most. Someone who opens the app 50 times a day hits a 1-in-100 slow request about every other day.
• One page often makes several requests. If a screen needs five requests and each has a 1% chance of being slow, the chance that at least one of them is slow is 1 − 0.99⁵, about 5%. The screen is slow far more often than any single request is.

That second effect gets much worse as systems grow, and it plays a big part in later lessons. For now, the habit to build is: always ask for p50 and p99, never just the average.

Step 8 of 15 · 1 min

Measuring instead of guessing

These numbers are for building intuition, not for capacity planning. Before making real decisions, measure your own system:

• Inside the app: log how long each database query and outgoing call takes. Most frameworks and database libraries have a setting for this.
• In the database: Postgres's EXPLAIN ANALYZE shows exactly how long a query took and where the time went (lesson 1.4).
• On the network: ping shows the round trip time to another machine. Run it from your app server to your database server and compare it to the numbers in this lesson.
• For the whole request: the curl timing command from lesson 1.1.

If you'd run ping from Chirp's server to the database host on Tuesday, you'd have seen 23ms where you expected 0.5ms, and found the wrong config in two minutes.

Step 9 of 15 · 2 min

From the real world: when the work is suddenly not small

This lesson's third pattern says the trip usually costs more than the work. That's true right up until the work explodes. On 2 July 2019, it did, across Cloudflare's whole network.

Cloudflare runs a web application firewall (WAF): rules, mostly regular expressions, that inspect every request for signs of attacks. At 13:42 UTC an engineer deployed a new rule. WAF rules were pushed to every server worldwide within seconds, because new attacks need fast responses. Part of the new rule's pattern was, in effect:

.*(?:.*=.*)

Almost immediately, CPU use on every core that handled web traffic went to 100%. Cloudflare's traffic dropped by about 82%, and visitors to a huge share of the web saw 502 errors. It lasted 27 minutes, until the team used a global switch to turn off the managed WAF rules.

Why would one line do that? Most regex libraries use backtracking: when part of a pattern fails, they go back and try a different way of matching the earlier parts. .* can match any number of characters, so for a string with no = in it, the engine tries every way of splitting the string between the two .*s before giving up. That's about n²/2 attempts for a string of length n.

For a 20-character field, that's 200 steps: nothing. For a 10,000-character field, it's around 50 million steps for one field of one request. Multiply by every request and every server, and your latency numbers stop being about network trips at all. Cloudflare's postmortem also found that a protection meant to limit how much CPU one regex could use had been removed by accident during a refactor weeks earlier.

What Cloudflare changed, and what to take for your own systems:

• Use a regex engine that guarantees linear time (RE2, Rust's regex, Go's regexp) for anything that runs on untrusted input. They give up a few features, like backreferences, to make that guarantee.
• Put a budget on work that depends on input size: maximum lengths, timeouts, step limits.
• Even "configuration" deserves a staged rollout. A rule that reached 1% of servers first would have broken 1% of servers.

The same shape caused Stack Overflow's outage three years earlier (lesson 2.2). When something's cost grows with the square of its input, one unusual input is enough.

Step 10 of 15 · 1 min

Going deeper: why your load test said 1 ms

You now know to look at p99, not the average. There's a trap that can make your p99 wrong, and it's built into many load-testing tools. Gil Tene, who named it, calls it coordinated omission.

Many load tools work like this: send a request, wait for the reply, record how long it took, send the next one. Imagine testing Chirp's feed this way at 100 requests a second for 100 seconds: 10,000 requests. Each normally takes 1 ms. Once during the test, the server freezes for 10 seconds (a long garbage-collection pause, say).

The tool sent a request just before the freeze. It waited 10 seconds for the reply and recorded one 10-second sample. During those 10 seconds it sent nothing else, because it was waiting. So its results show 9,999 fast requests and one slow one. Its p99 is 1 ms.

Real users don't take turns like that. At 100 a second, about 1,000 of them arrived during the freeze, and each waited somewhere between 0 and 10 seconds. That's 10% of all requests. The true p95 is about 5 seconds, and the true p99 about 9 seconds.

The tool didn't lie on purpose. It "coordinated" with the server: whenever the server was slow, the tool politely stopped asking, so the slow period was omitted from the data.

How to avoid it:

• Generate load on a fixed schedule ("open loop"), not one request after another. Tools like wrk2, k6's arrival-rate executors, and Gatling's open models do this.
• Measure from when each request was supposed to be sent, not from when it actually went out.
• Be suspicious of monitoring that has the same flaw: a health checker that waits for one reply before sending the next will also miss most of a stall.

It changes decisions, not just numbers. Chirp's team once tuned garbage collection for "better p99" using a closed-loop test. The improvement was real for the rare pauses the tool saw, but users mostly felt pauses the tool never counted.

Step 11 of 15 · 1 min

What Chirp changed

• Fixed the config, and added a check at startup that logs a warning if the database round trip is above a few milliseconds.
• Rewrote the loop into three batched queries.
• Turned on query logging in development with a warning for any request that makes more than 20 database queries. It catches N+1 loops before they ship.
• Started tracking p50 and p99 for each page instead of one average for the whole app.

None of these took more than an hour. Each one would have saved most of a day.

Step 12 of 15 · 1 min

In the simulation

Stage 0's latency panel splits each request into its parts: network, application, and database. When you make decisions later in the simulation, like adding a cache in Stage 1, watch which part of that breakdown moves. If a choice doesn't move the biggest part, it won't help much, however clever it is.

Open the simulation →

Step 13 of 15 · 1 min

If you're here for an interview

You won't be asked to recite this table, but you'll use it in every design question. When you say "let's add a cache here," you're relying on RAM being ~1,000× faster than disk and ~5,000× faster than a network trip. When you say "let's batch these writes," you're relying on round trips costing more than work. Saying that reasoning out loud, briefly, is much stronger than just naming the technique.

Two things that come across well:

• Quoting latency in orders of magnitude ("a cross-region round trip is tens of milliseconds, so we can't make several of these per request").
• Talking in percentiles ("the p99 matters here because the home screen makes five calls").

Step 14 of 15 · 2 min

Try these yourself

Step 15 of 15 · 1 min

Further reading

• Jeff Dean's original "Latency numbers every programmer should know," which is where most versions of this table come from.
• Colin Scott, "Latency numbers every programmer should know" (interactive). It shows how the numbers have changed over the years.
• Your cloud provider's published latencies between zones and regions, which are the numbers that matter for your own system.
• John Little's 1961 paper proving Little's Law, if you like seeing where things come from. Any operations-research textbook explains it more gently.
• John Graham-Cumming, "Details of the Cloudflare outage on July 2, 2019," Cloudflare blog. One of the clearest postmortems published, including a step-by-step explanation of the backtracking.
• Russ Cox, "Regular Expression Matching Can Be Simple And Fast," 2007. Why linear-time regex engines exist.
• Gil Tene, "How NOT to Measure Latency" (talk, several versions online), where coordinated omission is explained with examples; and the wrk2 README.