Step 1 of 17 · 2 min

Meet Chirp

Chirp is the app this whole course is about. It's made up, but it's built to feel familiar: a social app where people post short updates and photos, follow each other, and scroll a feed of posts from the people they follow. Think of a small mix of Twitter and Instagram, started by two friends.

Why follow one made-up app instead of jumping between topics? Because real systems don't arrive fully designed. They start small and break in specific ways as they grow, and each fix only makes sense once you've seen what broke. So the course grows Chirp step by step, and each part of the course is about the problem Chirp hits next:

At the end of each part, the simulation lets you run Chirp yourself at that size: you make the design decisions, real traffic hits them, and you see what holds up.

Everything about Chirp is fictional: the founders, the users, the bug reports. The technology and the numbers are real. When a lesson says "Chirp's server took 20 milliseconds", that's a realistic figure for that kind of request, not a made-up one.

Now, Chirp's very first problem.

Chirp's first real bug report didn't come from a crash. It came from Priya, a friend of one of the founders, who was testing the app from Mumbai.

"It works," she wrote, "but it takes a second or so every time I open it. Is that normal?"

The founders checked the server logs. The feed endpoint was answering in about 20 milliseconds. The database query took 4. Nothing was slow. One of them replied, a bit defensively, that it was probably her Wi-Fi.

It wasn't her Wi-Fi. Priya's phone was waiting on about 1.2 seconds of round trips between Mumbai and a data center in Virginia, and Chirp's own code was responsible for less than 2% of that time. The rest went to things that happen before the server ever sees a request, and to how the response travels back.

This lesson follows one request from the moment Priya taps the Chirp icon to the moment her feed appears. Almost every lesson after this one assumes you know this path, because every system you'll design is built on top of it.

Step 2 of 17 · 1 min

The thing that dominates everything: round trips

Before the details, one idea that explains most of what follows.

Data in a fiber-optic cable travels at roughly 200,000 kilometers per second, about two thirds the speed of light in a vacuum. Mumbai to northern Virginia is around 13,000 km in a straight line, and real cables don't run in straight lines. So a message from Priya's phone to Chirp's server and back (a round trip) takes around 200 milliseconds in practice. No amount of engineering makes that faster, because you can't speed up light.

That number matters because opening an app isn't one round trip. It's several, one after another, each waiting on the last. Each step in this lesson is really a question: how many round trips does this cost, and can we avoid paying them?

Keep a running total as we go.

Step 3 of 17 · 2 min

Step 1: Finding the server (DNS)

Priya's phone knows it needs to talk to api.chirp.app. It doesn't know where that is. Computers on the internet find each other by IP address, something like 203.0.113.42, so the first job is turning the name into an address. That's what the Domain Name System, DNS, does.

The phone doesn't figure this out alone. It asks a resolver, usually run by the mobile carrier or the Wi-Fi provider, sometimes a public one like Google's 8.8.8.8 or Cloudflare's 1.1.1.1. If the resolver has looked up api.chirp.app recently, it answers right away from memory.

If it hasn't, it works through a chain:

1. It asks a root server: "Who handles .app?" There are 13 named root server identities, but each one is really hundreds of machines spread around the world, so a nearby copy answers.
2. The root points it to the .app servers: "Who handles chirp.app?"
3. Those point it to Chirp's own DNS servers, usually run by whoever Chirp registered the domain with.
4. Chirp's DNS servers answer: api.chirp.app is at 203.0.113.42.

The resolver usually already knows the root and .app steps from earlier lookups, so a cold lookup often costs only one trip to Chirp's DNS servers. If those happen to sit in the US, that one trip is about 200ms. A warm lookup, answered from the resolver's memory, is a few milliseconds.

TTL, and the day Chirp's users vanished

Every DNS answer comes with a TTL (time to live): how many seconds anyone may remember it. Chirp's answer above said 300, so resolvers keep it for five minutes and then ask again.

Chirp learned what TTL really means a few weeks later, when they moved to a bigger server with a new IP address. They updated their DNS record and switched the old server off. For the rest of the day, a chunk of users couldn't load anything.

The record's TTL had been set to 86,400 seconds, a full day, which was the default their domain provider had filled in. Resolvers that had cached the old address kept handing it out until their copy expired, and that address now pointed at nothing.

The habit they took away is one most teams learn the same way: a day or two before changing an address, lower the TTL (to 60 seconds, say), wait for the old long TTL to run out, make the change, and keep the old server running a little longer than you think you need to. Some resolvers hold on to answers longer than they're told to, so you can never be completely sure every user has moved.

This also explains a trade-off you'll see throughout the course. A long TTL means fewer lookups and faster connections. A short TTL means you can move things quickly. You can't have both.

Running total (cold): about 200ms. And nothing has been sent to Chirp yet.

Step 4 of 17 · 1 min

Step 2: Opening a connection (TCP)

Now the phone knows where the server is. Before it can send a request, it needs a connection, and almost all web traffic has historically used TCP for that.

TCP's job is to make an unreliable network look reliable. Underneath, the internet just moves small packets, and packets get lost, duplicated, or arrive out of order. TCP numbers every byte, resends anything that goes missing, and puts everything back in order, so the app on each end sees a clean stream of data.

To set that up, the two sides first agree on starting numbers. This is the three-way handshake:

1. The phone sends SYN: "I'd like to connect. I'll start counting from here."
2. The server sends SYN-ACK: "Fine. I got yours, and I'll count from here."
3. The phone sends ACK: "Got yours." It can send its first data right alongside this.

That's one full round trip before any real data moves.

Running total: about 400ms.

Step 5 of 17 · 1 min

Step 3: Making it private (TLS)

Chirp uses https, so before the request goes out, the connection needs to be encrypted. That's TLS, the protocol behind the padlock in a browser.

TLS has two jobs. It encrypts traffic so nobody between Priya and Chirp (her café's Wi-Fi, her carrier, anyone else) can read or change it. And it proves the server really is Chirp, not someone pretending to be Chirp. The proof is a certificate: a document signed by an organization the phone already trusts, saying "this public key belongs to api.chirp.app."

In the handshake, the two sides agree on encryption settings, the server presents its certificate, and both sides work out a shared secret key without ever sending it across the network.

How many round trips that costs depends on the version:

• TLS 1.2, still widespread: two round trips.
• TLS 1.3, the current version: one round trip. The phone sends its guesses about the encryption settings in its very first message, so the server can usually answer everything in one go.

Chirp's server supported TLS 1.3. That saved them 200ms per new connection in Mumbai without changing a line of code, which is the kind of free win worth checking for.

TLS 1.3 also has a 0-RTT mode: a phone that has talked to Chirp before can send its request in the very first message, before the handshake finishes. The catch is that an attacker who records that first message can send it again, and the server can't tell the difference. That's harmless for "load my feed," and dangerous for "post this" or "send money." So 0-RTT is usually allowed only for requests that are safe to repeat. (Requests that are safe to repeat come back in lesson 1.5, and again in a big way in Part 4.)

Running total: about 600ms. Still no request sent.

Step 6 of 17 · 1 min

Step 4: Asking for the feed (HTTP)

Finally, the phone sends the actual request over the encrypted connection:

GET /v1/feed HTTP/1.1
Host: api.chirp.app
Authorization: Bearer eyJhbGciOi...
Accept: application/json

HTTP is the language of the request and response: a method (GET to read, POST to create), a path, some headers, and sometimes a body. The server sends back a status code (200 OK, 404 Not Found, 500 when something broke), its own headers, and the feed as JSON.

Sending the request and getting the first bytes of the response takes one round trip, plus however long the server spends working. For Chirp, that was the 20 milliseconds in their logs.

Running total: about 820ms. Chirp's code, the part the founders had been looking at, accounts for 20 of them.

Step 7 of 17 · 1 min

Step 5: Why the response took three more trips

The feed response was about 60 KB of JSON. You might expect that to arrive in one go. It doesn't, because of how TCP protects the network.

A brand-new TCP connection doesn't know how much the path between two machines can handle. If it sent everything at full speed, it could overload some router in the middle and cause packet loss for everyone. So it starts carefully and speeds up as data gets through. This is called slow start.

On most modern servers, a new connection may send about 10 packets, roughly 14 KB, before it has to wait for the other side to confirm they arrived. Each round trip that goes well, the allowance roughly doubles:

Round trip 1:  ~14 KB   (total so far: 14 KB)
Round trip 2:  ~28 KB   (total so far: 42 KB)
Round trip 3:  ~56 KB   (total so far: 98 KB)  ← the 60 KB feed finishes here

So Chirp's 60 KB response needed about two extra round trips on a new connection. From Mumbai, that's another 400ms.

Running total: about 1.2 seconds. That's what Priya was seeing.

Here's the whole thing in one place:

And if Priya opens the app again a minute later, on the same connection?

Same server, same code, same data, five times faster. The only difference is whether the expensive setup has already been paid for.

Here's the same journey as a timing diagram, with time running downward. Each diagonal line is one message crossing the ocean. Switch between a new and a reused connection, the two TLS versions, and compression, and watch which round trips disappear.

Step 8 of 17 · 1 min

Measuring it yourself

The founders stopped arguing once they measured it. curl can print exactly when each step finished:

curl -s -o /dev/null https://api.chirp.app/v1/feed \
  -w "dns:      %{time_namelookup}s
connect:  %{time_connect}s
tls:      %{time_appconnect}s
1st byte: %{time_starttransfer}s
total:    %{time_total}s
"

Each number is the time since the start, not the time for that step, so subtract each from the next to get the step itself. Run it against any site you use, ideally one hosted far away, and you'll see the same pattern: most of the time goes to setup and distance, not the server.

Browser developer tools show the same breakdown. Open the Network tab, click a request, and look at its Timing section.

Step 9 of 17 · 2 min

From the real world: the day Facebook's name disappeared

Step 1 of Priya's request, the DNS lookup, feels like a formality. On 4 October 2021, it was the reason Facebook, Instagram and WhatsApp were unreachable for about six hours.

It started in Facebook's backbone, the private network that links its data centers. During routine maintenance, an engineer ran a command meant to measure how much backbone capacity was available. Instead it took every backbone link down. Facebook had an audit tool that checks commands like this before they run, and that tool had a bug, so it let the command through.

That alone would have been a huge outage. What made it total was a safety rule in Facebook's DNS servers. Those servers are spread over many sites, and each site announces its addresses to the rest of the internet using BGP, the protocol routers use to tell each other "you can reach these addresses through me." Each DNS site was built to stop announcing itself if it couldn't reach the data centers, on the reasonable theory that a site cut off from the data centers shouldn't be answering questions. That's a good rule when one site is unhealthy. When the backbone went down, every site was cut off at once, and every site withdrew. Facebook's name servers vanished from the internet's routing tables.

From the outside, there was nothing to talk to. Resolvers couldn't find Facebook's name servers at all, so every lookup for facebook.com failed. Phones and browsers kept retrying, and public resolvers saw a flood of extra queries for Facebook's names.

Fixing it was slow for a reason that turns up in many big outages: the tools engineers would have used to fix it ran on the network that was down. Some of the work needed people physically inside data centers, past security systems built to be hard to get through. Once the backbone was back, Facebook brought traffic back in stages, because switching billions of users back on at once can cause its own surges, in power draw and in cold caches.

Three things to take from it:

• Step 1 is a dependency of everything. If nobody can find your servers, it doesn't matter how healthy they are.
• A safety rule that's correct for one machine can be catastrophic when every machine hits it together. Ask "what if all of them decide this at the same moment?"
• Keep a way in that doesn't depend on the thing that's broken: out-of-band access to your network, and runbooks that don't live on the systems they describe.

Step 10 of 17 · 2 min

Going deeper: anycast, or how one address lives in many cities

Earlier, you read that there are 13 root server identities, but each is "really hundreds of machines spread around the world, so a nearby copy answers." Cloudflare's 1.1.1.1 and Google's 8.8.8.8 work the same way. How can one IP address be in hundreds of places at once?

The answer is anycast. Normally, one address is announced by one place. With anycast, many sites announce the same address. Every router on the internet hears several routes to it, and each keeps the one it considers best, which is usually the closest in network terms. So Priya's packets to 1.1.1.1 land at a site in Mumbai, while someone in Frankfurt reaching the same address lands in Frankfurt. Neither phone knows there's more than one site.

Anycast gives you two things almost for free:

• Low latency, because users reach a nearby copy without any DNS tricks.
• Failover without anyone changing anything. If a site stops announcing the route (because it failed a health check, or an operator pulled it), routers forget it within seconds and send traffic to the next best site. It also spreads attack traffic across many sites, which is why DNS providers and CDNs lean on it.

It has two sharp edges, and both showed up in the Facebook outage or around it:

• Withdrawal is all-or-nothing per site, and it's automatic. That's exactly the mechanism that made Facebook's DNS disappear: correct per site, fatal when every site withdrew together.
• Routes can change in the middle of a connection. DNS questions are usually a single packet over UDP, so it doesn't matter if the next question goes to a different site. A long TCP connection is different: if the route shifts, packets suddenly arrive at a site that has never heard of that connection, and it gets reset. CDNs that use anycast for web traffic work hard to keep routes stable, and to make connections cheap to re-establish.

For Chirp, this is why "put the DNS with a big provider" and, later, "put a CDN in front" (lesson 2.3) get you global reach without Chirp running servers on every continent: the provider's anycast network does that part.

Step 11 of 17 · 1 min

What Chirp changed

None of these needed new servers or a bigger budget. They were the right first moves for a two-person team.

Reuse connections. Since HTTP/1.1, connections stay open by default after a response (keep-alive), but the client has to actually reuse them. Chirp's app had been creating a new HTTP client for every request, which threw away the warm connection each time. Sharing one client across the app fixed it, and repeat requests dropped to roughly one round trip.

Make the response smaller. Turning on gzip compression shrank the 60 KB feed JSON to around 12 KB. Repetitive text like JSON compresses very well. 12 KB fits inside the first 14 KB a new connection can send, so the two extra slow-start round trips disappeared entirely.

Ask once instead of five times. The first version of the app made separate calls for the feed, the user's profile, unread notifications, and a couple of settings, one after another. Folding the home screen into one request removed several round trips on every cold start. (Lesson 1.5 goes into designing APIs this way.)

Keep DNS answers cached. They raised the TTL back to a sensible value once the move was done. Five minutes to an hour is common.

Together, those took Priya's cold start from about 1.2 seconds to around 600ms, and her warm opens to about 220ms. The remaining cost is distance. The only way to remove it is to put something closer to Priya, which is what CDNs do (lesson 2.3) and what going multi-region means (lesson 5.5). Neither makes sense for a team with forty users, and recognizing that is part of the lesson too.

Step 12 of 17 · 1 min

Newer versions of HTTP

You'll hear about HTTP/2 and HTTP/3 in almost any system design conversation. Here's what they actually change.

HTTP/1.1 handles one request at a time per connection. If an app needs ten things, it either waits for each in turn or opens several connections, and each new connection pays its own TCP and TLS setup.

HTTP/2 lets many requests share one connection at the same time, interleaved in small pieces. One handshake, many requests. It also compresses headers, which helps because headers like long Authorization tokens repeat on every request.

But HTTP/2 still runs over TCP, and TCP delivers bytes strictly in order. If one packet is lost, everything behind it waits for the resend, even data belonging to completely unrelated requests. This is head-of-line blocking, and on a lossy mobile network it can make HTTP/2 slower than you'd expect.

HTTP/3 fixes that by replacing TCP with QUIC, a protocol that runs on top of UDP. Three changes stand out:

• A lost packet only holds up the request it belongs to. The others keep going.
• The connection setup and the encryption setup happen together, so a new connection costs one round trip instead of two (TCP plus TLS 1.3).
• Connections survive network changes. When Priya walks out of her house and her phone switches from Wi-Fi to mobile data, her IP address changes. A TCP connection breaks and has to start over. A QUIC connection is identified by an ID rather than by addresses, so it carries on.

Most large sites and CDNs support HTTP/3 today, and browsers fall back to older versions automatically when they can't use it.

Step 13 of 17 · 1 min

What happens inside Chirp's server

This lesson stops at the edge of the server, but it helps to know roughly what's on the other side. At Chirp's current size:

1. The request arrives at a reverse proxy (often nginx), which handles TLS and passes plain HTTP to the app.
2. The application checks who Priya is, works out what her feed should contain, and asks the database.
3. The database finds the rows and sends them back.
4. The application turns them into JSON and sends the response back through the proxy.

Each of those boxes gets its own lesson. The database is next in line (lesson 1.4), and the proxy grows into a load balancer in lesson 2.2.

Step 14 of 17 · 1 min

In the simulation

Stage 0's dashboard shows a latency breakdown for a user in a different country from your server, and you'll see the same split this lesson walked through: setup and distance first, your code a distant last. Keep an eye on it. In Stage 4, when Chirp's users spread across continents, that slice called "distance" becomes the biggest problem you have.

Open the simulation →

Step 15 of 17 · 1 min

If you're here for an interview

"What happens when you type a URL into a browser?" is one of the oldest interview questions there is. A weak answer lists the steps: DNS, TCP, HTTP. A strong answer counts the round trips and explains which ones can be avoided: cached DNS, reused connections, TLS 1.3, HTTP/2 or 3, compression to beat slow start, and moving content closer to users.

If you're asked a system design question and latency comes up, this is the thinking interviewers want to see: you don't just make the server faster, you work out where the time actually goes.

Step 16 of 17 · 2 min

Try these yourself

Step 17 of 17 · 1 min

Further reading

• Ilya Grigorik, High Performance Browser Networking. Free online, and the best single book on this lesson's material. The chapters on TCP, TLS, and HTTP/2 are especially good.
• RFC 8446: TLS 1.3. The introduction is readable even if the rest isn't.
• RFC 6928: the proposal to raise TCP's initial window to 10 packets.
• RFC 9000 (QUIC) and RFC 9114 (HTTP/3).
• Cloudflare Learning Center, "What is DNS?" A clear, illustrated walkthrough of resolvers and caching.
• Santosh Janardhan (Meta), "More details about the October 4 outage," Engineering at Meta, 2021. The official explanation: the backbone command, the audit tool bug and the DNS withdrawal.
• Cloudflare, "Understanding How Facebook Disappeared from the Internet," 2021. What the outage looked like from outside: BGP withdrawals and the surge of DNS queries.
• Cloudflare Learning Center, "What is anycast?" A short, clear introduction.