Before we start

You’ve almost certainly felt this outage, even if you don’t remember it. On 20 October 2025, apps and websites that had nothing to do with each other all broke at the same time. They had one thing in common: somewhere underneath, they depended on Amazon Web Services in us-east-1, its oldest and biggest region in northern Virginia.

The cause sounds tiny: one DNS record had no addresses in it. DNS is the internet’s phone book: it turns a name like dynamodb.us-east-1.amazonaws.com into the IP addresses of real servers. An empty entry means nobody can call.

The system as it was

DynamoDB, AWS’s big key-value database, is reached through hundreds of thousands of DNS records across many load balancers. Managing them by hand is impossible, so it’s automated with two parts:

• The DNS Planner watches the health and capacity of DynamoDB’s load balancers and regularly writes a plan: which load balancers each address should point to, and with what weights.
• The DNS Enactors apply plans to Route 53, AWS’s DNS service. There are three of them, in three different Availability Zones, so that losing one doesn’t stop updates.

After applying a plan, an Enactor also cleans up: it deletes plans much older than the one it just applied, so they don’t pile up.

Every part of that sounds sensible. Having three Enactors is meant to make things safer. The problem was how they could interfere with each other.

What happened

1. Oct 19, 11:48 PM PDTDynamoDB’s regional DNS record becomes empty. Clients can’t connect.
2. Oct 20, 12:38 AMEngineers identify DynamoDB’s DNS as the source.
3. 1:15 AMTemporary workarounds let some internal services connect.
4. 2:25 AMDNS is fully restored by hand. DynamoDB recovers by 2:40 AM as cached records expire.
5. 2:25 AMEC2’s internal fleet manager tries to catch up and gets stuck: “congestive collapse”.
6. 4:14 AMEngineers throttle work and restart its hosts.
7. 5:28 AMEC2 server leases are restored. A new backlog builds in network configuration.
8. 5:30 AMNetwork Load Balancers start failing health checks for new instances.
9. 9:36 AMAutomatic health-check failover is switched off to stop it making things worse.
10. 1:50 PMEC2 fully recovered.
11. 2:20 PMThe last affected services recover.

The race

Here’s the sequence, step by step. Each step is reasonable on its own; together they empty the record.

1. One Enactor picked up a plan, then became unusually slow, retrying updates on other endpoints.
2. Meanwhile the Planner kept producing newer plans, and a second Enactor applied one of the newest and finished quickly.
3. The slow Enactor finally applied its much older plan, overwriting the newer one. It had checked that its plan was fresh, but that check was made long before, and was stale by now.
4. The fast Enactor then ran its routine clean-up, deleting plans much older than the one it had applied. The old plan, now live, was one of them.
5. With the live plan deleted, the record was left with no IP addresses at all. And because the system was now in a state its automation didn’t expect, no Enactor could repair it. People had to.

This is a textbook race condition: a bug that only appears when things happen in an unlucky order. It had existed quietly, waiting for one Enactor to be slow at exactly the wrong moment. (Lesson 4.2 shows a different race with the same shape: two steps, each fine alone, broken by timing.)

Why fixing DNS didn’t end it

DynamoDB was back by 2:40 AM. But many AWS systems use DynamoDB themselves, and one of them is EC2’s fleet manager, which tracks the physical servers that virtual machines run on. It keeps a lease on every server, renewed regularly, and those renewals had been failing for hours. Leases timed out one by one.

When DynamoDB came back, the fleet manager tried to renew every lease at once. The work took so long that it timed out and was retried, adding even more work. AWS called this “congestive collapse”: the system was busy, but making no progress. New virtual machines couldn’t launch.

Engineers had to throttle incoming work and restart the fleet manager’s hosts. Then a backlog of network configuration built up, and new machines came up without working networking. Load balancers saw those machines failing health checks and kept pulling them in and out of service. Each recovery step uncovered the next jam.

This is how a three-hour root cause becomes a fifteen-hour outage: systems that were healthy, but depended on the broken one, don’t simply bounce back. The retry storm from lesson 6.1 shows the same kind of jam:

What they changed

• The DNS automation is off worldwide until the race condition is fixed and protections are added against applying incorrect plans.
• Load balancers get a limit on how much capacity health checks can remove at once during a failover.
• EC2 gets a test suite for the fleet manager’s recovery, and rate limits on incoming work based on queue size.

Lessons for your own designs

1. Check right before you act, not long before. The slow Enactor’s freshness check was correct when it ran, and wrong when it mattered. Make “is this still the newest?” part of the same step as the write, like the conditional updates in the ticket-booking question.
2. Clean-up jobs need care too. Deleting “old” things is dangerous when “old” might still be in use.
3. Know what’s underneath your dependencies. Many companies were multi-zone, but all in one region, and AWS’s own services depended on DynamoDB. Lesson 5.5 covers spreading across regions.
4. Plan the recovery, not just the fix. When a dependency comes back, everything waiting on it rushes back at once. Throttle the catch-up, or it becomes the next outage.
5. Limit what automatic failover can do. Health checks that remove capacity can make a bad situation worse. A cap on how much they can remove at once is a simple, powerful safety net.

How this shows up in interviews

When your design has several workers updating the same thing, interviewers may ask “what if two of them do it at once?” This outage gives you a concrete, memorable answer: show how you’d make updates conditional on a version number, so an older update can never overwrite a newer one. And when you say “we’ll deploy in multiple availability zones”, it’s worth adding that zones in one region still share regional services, which is why some systems go multi-region for their most critical paths.