Before we start

Imagine three friends trying to decide where to eat, over a group call with a bad connection. Asha can hear Ravi, Ravi can hear Meera, but Asha and Meera can’t hear each other. Asha keeps saying “let’s pick a new place, I’ll decide”, and Ravi keeps saying “Meera already decided”. Nobody’s lying, and nobody’s wrong about what they can hear. But they never agree.

That’s very close to what happened inside Cloudflare’s control plane. Fully broken things are easy to spot. Half-broken things, where some messages get through and others don’t, are much harder, because every part of the system has a different view of reality.

The system as it was

This incident wasn’t about Cloudflare’s main network, the part that serves websites. It hit the control plane: the API and dashboard customers use to change their settings.

Three layers matter:

• etcd, a small, strongly consistent store running on three nodes. It uses Raft, which needs a leader that most nodes can reach. (Lesson 6.5 explains Raft.)
• A database cluster manager that uses etcd to agree on which Postgres database is the primary. If it can’t confirm a healthy primary, it promotes a replica.
• Postgres databases, including the one that checks authentication for every API call and dashboard login.

What happened

1. 14:43A network switch partly fails. It still runs some protocols, but stops forwarding some traffic.
2. 14:43The etcd nodes can’t agree on a leader. Elections keep tying, so etcd stops accepting writes.
3. 14:45The database manager can’t confirm a healthy primary and promotes a replica. No data is lost.
4. 14:49The switch recovers by itself, about six minutes after it failed.
5. After 14:49A bug means every replica must be rebuilt after the promotion. The auth database loses a busy read replica.
6. During the afternoonAPI success drops as low as 75%; the dashboard is up to 80 times slower. Engineers shed load and reroute reads.
7. 21:20The first replica finishes rebuilding. Performance returns to normal immediately.

When three nodes can’t agree

Here’s what the half-broken switch did to etcd. Node 3 was the leader. Node 1 could only reach node 3 through the broken switch, so it couldn’t hear the leader. But it could reach node 2 through a healthy switch, and node 2 could reach node 3 fine.

So node 1 thought “the leader is gone”, and kept starting elections, voting for itself. Node 2 could still see node 3, so it kept voting for node 3. Every election tied. No leader that everyone could reach was chosen, and etcd couldn’t accept any writes.

The consensus lesson plays this out:

Cloudflare’s post called this a Byzantine failure, the term for nodes holding conflicting views of reality. In a postscript, they noted it’s better described as an omission fault: messages were silently dropped, not falsified. Either way, the point stands. Raft handles “a node is down” well. It handles “a node is up, but can only see some of the others” much worse.

Modern Raft implementations have defences against exactly this: pre-vote (a node checks it could actually win before disrupting everyone with an election) and check quorum (a leader steps down if it can’t reach a majority). Lesson 6.5 shows them:

Six minutes becomes six hours

The switch was only broken for about six minutes. Here’s how the damage outlasted it:

The database promotion itself went well: a synchronous replica took over with no data lost. The trouble was a bug in the cluster manager. After a promotion, every replica had to be rebuilt from scratch.

For most databases that hardly mattered. For the authentication database, it removed a heavily used read replica, so the new primary suddenly carried all the read traffic too. And every API call and dashboard login checks authentication, so both slowed to a crawl. Rebuilding a replica means copying a whole database, which takes hours.

There was one more complication. The obvious move was to send authentication traffic to the backup data centre. But dashboard sessions were stored in a way that couldn’t yet move between data centres without disruption, so doing that made the dashboard worse.

How they got through it

While waiting for the rebuild, engineers kept the platform alive by hand:

• Shedding load: they slowed down less urgent work, like pushing SSL certificates and sending emails.
• Rerouting reads: they sent API reads to replicas in the secondary data centre, which “substantially improved” API availability.
• Moving dashboard auth back to the primary data centre, once moving it had proved harmful.

At 21:20 the first replica finished rebuilding, and performance returned to normal immediately.

What they changed

• They fixed the bug that forced every replica to be rebuilt after a promotion.
• They changed when databases are promoted automatically. For at least one database, they decided promoting quickly can do more harm than good.
• They kept tuning these systems to handle failures in the things they depend on.
• Work on making dashboard sessions movable between data centres, already under way, continued.

Lessons for your own designs

1. Partial failures are the hard ones. Plan for “slow” and “sometimes” as well as “down”. A component that half works can confuse everything around it.
2. Automatic failover is a tool, not a free win. It can trigger on a blip and leave you worse off than doing nothing. Know what happens after a failover, not just during it.
3. Recovery time matters as much as failure time. Here, the switch was broken for six minutes, but rebuilding replicas took hours. If something takes hours to rebuild, you need spare capacity so losing it doesn’t hurt.
4. Know your hidden dependencies. The API depended on auth, which depended on a database, which depended on a cluster manager, which depended on etcd, which depended on a switch.

How this shows up in interviews

When you put a consensus system (etcd, ZooKeeper, a Raft-based database) into a design, interviewers may ask “what happens during a network partition?” Most candidates describe a clean split: some nodes on one side, some on the other. This story lets you go a level deeper: real partitions are often partial, where A can reach B and B can reach C, but A can’t reach C. Mentioning pre-vote and check quorum as defences, and that automatic failover can make things worse, shows you’ve thought about how these systems behave in practice. Lesson 5.4 covers partitions.