Step 2 of 14 · 1 min

Regions, and why distance dominates

Cloud providers run regions in cities around the world. Each region contains several availability zones (lesson 2.1), a few kilometres apart and connected by very fast links. Between regions, the distance is thousands of kilometres, and lesson 1.2's numbers apply: round trips of tens to hundreds of milliseconds, which no hardware can shorten.

Inside one region, the trade-offs in Parts 2 to 5 are mostly about machines and software. Across regions, they're mostly about the speed of light. Every design decision in this lesson comes down to one question: when a user in Mumbai does something, how many times does the request have to cross an ocean?

Step 3 of 14 · 2 min

Four ways to run in several regions

There are four main patterns, roughly in order of how much they change your system.

1. One active region, one standby

Run everything in Virginia as before, and keep a standby copy in another region: replicas of the databases, and servers that can start up if needed. If Virginia fails, switch to the standby.

This protects against losing a region. It does nothing for latency, because all traffic still goes to one place. It's the cheapest way to survive a regional disaster, and a reasonable first step.

2. Read locally, write to one region

Put read replicas of every database in every region, along with app servers. Reads are served nearby. Writes still go to the leaders in one home region.

For a read-heavy app like Chirp, this helps a lot, since most requests are reads. But every write from Mumbai still crosses the ocean, so posting and liking stay slow. And lesson 2.5's lag problems get much worse: replicas on another continent lag by hundreds of milliseconds at best, so read-your-writes needs careful handling (reading your own recent writes from the home region, lesson 5.3).

3. Give each user a home region

Put each user's data in the region nearest to them. Indian users' shards (lesson 5.1) have their leaders in Mumbai; European users' in Frankfurt; everyone else's in Virginia. Each user's reads and writes are local, most of the time.

This is what Chirp chose, and the rest of this lesson is mostly about the parts that make it tricky: everything that involves more than one user.

4. Every region writes everything

Let every region accept writes for any data, and replicate changes between regions in all directions (multi-leader replication). Every write is local. But two regions can now change the same thing at the same time, so you're in lesson 5.3's world of conflicts: last-write-wins losing data, or merging. Some data fits this well (lesson 5.3's CRDT examples), but for most applications it creates more problems than it solves. Chirp used it for nothing.

Step 4 of 14 · 3 min

Home regions

A user's home region is where their data lives and where their writes go. Chirp assigned it at sign-up, from the user's location, and stored it in a small global directory: user ID to home region.

Getting each request to the right place

When a request arrives, two things need to happen:

1. Reach a nearby region quickly. DNS (lesson 1.1) can answer api.chirp.app differently depending on where the user is (geo or latency-based DNS), or the regions can share one anycast address (lesson 2.2) so the network delivers each user to the nearest one. Either way, Priya in Mumbai connects to Mumbai, and the TLS handshake happens close by.
2. Get to the user's home region for anything that needs it. If Priya is travelling in Germany, she connects to Frankfurt, but her data lives in Mumbai. Frankfurt's servers check her home region (cached from the directory) and forward her writes, and reads of her own private data, to Mumbai over the cloud provider's network. That's one long hop for those requests, but only while she's travelling.

Chirp put the home region inside each user's login token, so most requests didn't need a directory lookup at all.

What crosses regions

Most of what Chirp users do involves other users, who may live elsewhere. Here's how each kind of interaction was handled:

Reading someone else's public content. Priya (Mumbai) opens the profile of a footballer whose home region is Frankfurt. Chirp replicates public content to every region: posts, public profiles and follower counts, as read-only copies kept up to date by change events (lessons 3.4 and 4.1). So Priya reads the footballer's posts from Mumbai. They might be a second or two behind Frankfurt, which is fine for public posts (eventual consistency, lesson 5.3).

Liking or replying to someone in another region. Where does Priya's like on the footballer's post live? Lesson 5.1 keyed likes by the liker's user ID, so it lives in Priya's home region, Mumbai, and the write is local and fast. An event carries it to Frankfurt, where the footballer's like count and notifications are updated (lesson 4.5's batched counts). Lesson 5.1's choice of key, made for hot-shard reasons, turned out to be exactly right for regions too.

Following someone in another region. Lesson 5.1 stored follows twice: by follower (the source of truth) and by followee (derived). The source-of-truth row lives in the follower's home region; the derived row is written in the followee's region by an event a moment later.

Feeds. Each region builds feeds for its own users (lesson 4.4), from the replicated public posts. A follower in Mumbai gets a Frankfurt user's post in their feed a couple of seconds after it's written, once it has replicated.

Things that must be globally unique. Usernames (lesson 5.1) need one answer worldwide. Chirp kept the usernames table in a single region, with read replicas everywhere. Claiming a username at sign-up (once per user) makes one cross-region round trip. Rare, and acceptable. Systems that need many global, strongly consistent decisions use databases built for it (lesson 5.4's Spanner-style systems), paying a cross-region round trip on every such write.

The pattern underneath: each piece of data has one home where it's written; everything else is a copy, kept up to date by events. Cross-region interactions become a local write plus an asynchronous event, never a synchronous call across the ocean in the middle of a user's request.

Moving a user

People move. When a user's location changes for good (they've used Chirp mostly from a new country for a month), their data can be moved to a new home region, using the same copy-catch-up-switch process as moving a shard (lesson 5.2), but for one user's data. Chirp did this in a nightly batch, a few thousand users at a time.

Step 5 of 14 · 1 min

Where data lives, legally

The European partner's questions were about data residency: rules about where people's personal data may be stored or sent. Europe's GDPR, for example, puts conditions on transferring personal data outside the EU, and other countries (including India) have their own rules, which change over time.

This lesson isn't legal advice, and the details always need lawyers. But home regions make the engineering much simpler: a European user's private data (email, phone, private messages, settings) lives in Frankfurt and is written there. What's replicated elsewhere is their public content, which they've chosen to publish. Chirp's design documents classified each kind of data as "stays in home region" or "may be replicated globally," and the legal team reviewed the list.

Step 6 of 14 · 2 min

Losing a region

Surviving the loss of a whole region was one of the three reasons for this project, so the team tested it before declaring victory. On a quiet Sunday morning, they cut Virginia off completely, to see whether Chirp would keep running in Mumbai and Frankfurt.

It didn't. Within a minute, nobody anywhere could log in.

The reason: the authentication service, which checks passwords and issues login tokens, had only ever been deployed in Virginia. Every region's app servers called it for each login and token refresh. Nobody had moved it, because it was "just a small internal service." Cutting off Virginia didn't just take out Virginia's users; it took out the one service everyone depended on.

This is the most common failure in multi-region designs: a hidden dependency on one region. Authentication, configuration services, feature-flag services, the ID-generation lease table from lesson 4.3, a monitoring system, the place where secrets are stored. Any of these, if it only lives in one region, quietly makes every region depend on that one.

After that Sunday, Chirp listed every service and data store, and for each one wrote down where it ran and what would happen if each region disappeared. Authentication was deployed in all three regions, with user credentials stored in each user's home region and tokens verifiable anywhere.

What failover costs

When a region really fails, two numbers from lesson 2.5 decide how bad it is:

• How much recent data is lost (RPO). Replication between regions is almost always asynchronous, because waiting for another continent on every write would add 100 ms or more (PACELC, lesson 5.4). So if Mumbai fails, writes from its last second or two may not have reached any other region. Chirp's design kept replicas of each home region's data in a second zone in the same region synchronously (cheap, lesson 5.4), and in another region asynchronously. Losing one zone loses nothing. Losing a whole region can lose a couple of seconds of writes.
• How long until users are served again (RTO). Detecting the failure, promoting the other region's replicas for the affected users' data, and moving traffic. If traffic moves by changing DNS, remember lesson 1.1: resolvers and phones cache answers for the record's TTL, so a 5-minute TTL means up to 5 minutes of some users still trying the dead region. Anycast moves traffic faster.

And the surviving regions need room to absorb the load. If Mumbai fails, its users (70% of Chirp) are now served from Frankfurt and Virginia. That's lesson 2.1's "N+1" at the scale of regions: each region needs spare capacity, or the failover moves the outage instead of ending it. Chirp sized Frankfurt and Virginia to absorb Mumbai at reduced quality (some features turned off, caches cold) for a few hours.

Practise it

A failover plan that's never been run doesn't work. Chirp scheduled a regional failover exercise every quarter, and each one found something: a cache that took 20 minutes to warm up (lesson 2.4), a queue consumer that only ran in one region, a DNS record someone had changed by hand.

Step 7 of 14 · 1 min

What it costs

Running in three regions isn't free, and it's worth being honest about it:

• Infrastructure: roughly the cost of one region times the number of regions, plus spare capacity for failover.
• Data transfer between regions, which cloud providers charge for. Replicating every public post and event to three regions adds up.
• Complexity: every feature now has to answer "where does this data live, and what happens across regions?" in design review.

For Chirp, with users concentrated in a few parts of the world, the speed improvement and the resilience were worth it. For a smaller app, or one whose users are mostly in one place, a single region with a CDN and a standby is usually the right answer for a long time.

Step 8 of 14 · 1 min

From the real world: when the control plane went dark

This lesson warned about hidden dependencies on one region. AWS's outage in us-east-1 on 7 December 2021 showed the same thing one layer down: a dependency on a control plane.

AWS runs an internal network for its own services (monitoring, internal DNS, parts of the control systems), connected to its main network by networking devices. That morning, an automated job scaled up a service on the main network. It triggered unexpected behaviour in a large number of clients on the internal network, and a surge of connection activity overwhelmed the devices between the two networks. Latency and errors spiked; clients retried, which added more load.

AWS's own monitoring ran on the internal network, so engineers had poor visibility into what was happening, which slowed diagnosis. Over the next several hours, anything that needed to cross that boundary struggled: launching new EC2 instances, some API and console operations, and AWS's ability to update its status page. Recovery came in the afternoon, after AWS moved internal DNS traffic away from the congested paths and stopped the scaling activity.

The most useful observation in AWS's summary is about what didn't break. Workloads that were already running mostly kept working. Running instances, existing connections and stored data (the data plane) didn't need the control plane minute to minute. What broke were systems that needed to change something during the event: scale up, launch replacements, update configuration.

Step 9 of 14 · 1 min

Going deeper: static stability

AWS's name for designing around that observation is static stability: a system should keep working, with no changes, when a dependency fails, especially the control plane that would normally help it recover.

The classic example is capacity across availability zones. Say Chirp's API runs in three zones, and each zone needs about a third of the capacity at peak.

• Reactive design: run each zone at about 67% busy. If a zone fails, the other two must launch more servers quickly. That needs the control plane (launching instances, attaching them to load balancers), and during a big event, the control plane is exactly what's overloaded, because everyone is trying to do the same thing.
• Statically stable design: run enough capacity in each zone that any two zones can carry everything, about 50% busy normally. When a zone fails, nothing needs to happen. The remaining zones are busier, but nobody has to launch anything.

The second design costs a third more servers all the time. That's the trade: you pay for resilience in advance, instead of depending on recovery machinery at the worst moment. Other forms of the same idea:

• Cache the control plane's answers in the data plane. Load balancers keep routing with their last known configuration if the configuration service is unreachable. Services keep using cached credentials and certificates that are valid for hours, not minutes.
• Prefer "keep doing what you were doing" over "ask before every action." A service that checks a feature-flag server on every request fails when the flag server does; one that refreshes flags in the background and keeps the last values doesn't.
• Keep your eyes outside the blast radius. Monitoring and incident tools that share a region or network with production go dark with it (Roblox, lesson 6.4, and AWS here).

Step 10 of 14 · 1 min

What Chirp changed

• Three regions: Mumbai, Frankfurt and Virginia, each with app servers, caches, queues and databases.
• Home regions per user, set at sign-up, carried in the login token, and stored in a small global directory.
• Writes go to the user's home region; public content is replicated to every region through change events; cross-region interactions are a local write plus an event.
• Usernames stay in one global table, with replicas everywhere.
• Synchronous replication within a region, asynchronous between regions.
• Latency-based routing to the nearest region, then forwarding to the home region where needed.
• No single-region dependencies: authentication, configuration and ID leases run in every region.
• Quarterly regional failover exercises.

Priya's feed now opens in about 180 ms. And when, a few months later, a real outage hit the Mumbai region for half an hour, Chirp stayed up. Slower for Indian users, with some features switched off, but up.

Step 11 of 14 · 1 min

In the simulation

Stage 4 is where Chirp goes global. You choose between a standby region, read replicas everywhere, home regions and multi-leader replication. The simulation shows read and write latency for users on each continent, then takes a region down, with one dependency you might have forgotten, to see what survives.

Open the simulation →

Step 12 of 14 · 1 min

If you're here for an interview

Multi-region questions test whether you can reason about latency and failure at the same time. Strong answers:

• Start from latency numbers: cross-region round trips are tens to hundreds of milliseconds, so a request shouldn't cross an ocean synchronously if it can avoid it.
• Name the patterns (standby, read-local and write-global, home regions, multi-leader) and choose one with a reason.
• Explain cross-region interactions: one home per piece of data, copies via events.
• Discuss failover honestly: asynchronous replication loses recent writes; DNS TTLs slow traffic shifts; surviving regions need spare capacity.
• Mention hidden single-region dependencies. Few candidates do, and it's the failure that actually happens.

Step 13 of 14 · 2 min

Try these yourself

Step 14 of 14 · 1 min

Further reading

• Martin Kleppmann, Designing Data-Intensive Applications, Chapter 5, on multi-leader replication and its use across data centers.
• Your cloud provider's documentation on regions, availability zones, and latency-based or geolocation DNS routing.
• James C. Corbett et al., "Spanner: Google's Globally-Distributed Database," OSDI 2012, for what globally consistent writes cost and how they're made practical.
• The European Data Protection Board's guidance on international data transfers, for the legal background (with a lawyer).
• Amazon Web Services, "Summary of the AWS Service Event in the Northern Virginia (US-EAST-1) Region," December 2021.
• Becky Weiss and Mike Furr, "Static stability using Availability Zones," The Amazon Builders' Library.