Before we start

Imagine asking a librarian: “Can you give me the books marked for disposal?” but you mumble the last part. The librarian hears “give me the books” and hands you every book in the library. You, believing they’re all marked for disposal, start throwing them out.

That’s almost exactly what happened inside Cloudflare. Nothing was hacked and nothing broke in the usual sense. Two pieces of software disagreed about what an empty value meant, and one of them was allowed to delete things.

The system as it was

Most websites on Cloudflare use Cloudflare’s own IP addresses. Some large customers instead bring their own ranges of addresses, a feature called BYOIP (Bring Your Own IP). Cloudflare then announces those ranges to the rest of the internet using BGP, the system networks use to tell each other “you can reach these addresses through me”. A range of addresses is called a prefix.

If Cloudflare stops announcing a customer’s prefix, the internet simply forgets how to reach it. The sites behind it become unreachable, even though every server is perfectly healthy.

When a customer wanted to remove a prefix, they used to ask, and an engineer did it by hand. Cloudflare decided to automate this with a new background task: regularly ask the API for prefixes waiting to be deleted, and delete them.

What happened

1. Feb 5, 21:53The code for the new clean-up task is merged.
2. Feb 20, 17:46The task is deployed.
3. 17:56It runs, receives every BYOIP prefix instead of a handful, and starts withdrawing them.
4. 18:13Cloudflare notices failures on its own 1.1.1.1 website.
5. 18:18An internal incident is declared.
6. 18:46An engineer stops the task and disables it. The withdrawals end.
7. 19:19Customers are told they can re-announce their prefixes from the dashboard.
8. 20:20Undoing changes restores about 800 prefixes.
9. 21:08A global configuration rollout begins for the prefixes whose settings were deleted.
10. 23:03The last prefixes are restored.

One empty value

The task asked the API for its list like this: /v1/prefixes?pending_delete. Notice there’s no value after pending_delete. The person writing the task meant “where pending_delete is true”.

On the server, the code only applied the filter when the value was not empty, something like if v != "". The value was empty, so the filter was skipped entirely, and the server returned every BYOIP prefix. Cloudflare’s write-up puts it plainly: the API server “interprets this as a request for all BYOIP prefixes”.

The task then did what it was built to do. It treated every prefix it received as waiting for deletion, and started removing them, along with the settings that connected them to Cloudflare’s services. Try both versions of the request:

The change was applied gradually, so not every customer was hit. By the time it was stopped, about 1,100 of the 6,500 prefixes Cloudflare announced to one network peer had been withdrawn, roughly a quarter of all BYOIP prefixes.

What customers saw

For affected customers, it looked like their sites had vanished. Connections simply failed, because the internet no longer knew where to send them. Other Cloudflare products built on those addresses broke too: proxied apps, protected networks, and customers’ outgoing traffic.

Even Cloudflare’s own 1.1.1.1 website was affected and showed errors. Its DNS service itself kept working.

Why undoing it took five hours

The task ran for about 50 minutes. Repairing it took over four hours more, because customers ended up in three different states:

• Prefix withdrawn, settings intact. These could simply be announced again, and customers could even do it themselves from the dashboard. About 800 were restored by reverting changes.
• Some settings deleted. Partly recoverable from the dashboard, but not completely.
• All settings deleted. About 300 prefixes had their service settings removed from Cloudflare’s edge by a software bug. These had to be restored manually and rolled out to every edge machine worldwide, which took the longest.

There’s a pattern here worth remembering: deleting is fast, but recreating the right state everywhere is slow. Mistakes that delete data and configuration cost far more to recover from than mistakes that just turn something off.

Why tests didn’t catch it

Cloudflare’s write-up is honest about this: the test environment’s sample data wasn’t realistic enough, and the tests didn’t cover a background task changing customer data without explicit input from a person. The task worked in testing because, in testing, there was nothing dangerous to delete.

What they changed

• Stricter API inputs. The flag was “interpreted as a string”, so they’re improving the API’s schema so inputs like this are validated, and an empty value can’t silently mean “everything”.
• Separating what’s configured from what’s live. Customer settings will be deployed as snapshots through gradual, health-checked rollouts, so they can “near-instantly revert individual customers (or all customers)”.
• A circuit breaker for big changes. Monitoring will detect when changes are “happening too fast or too broadly” and stop the process automatically.
• Fail small. A wider programme, which Cloudflare calls “Code Orange: Fail Small”, aims to roll out every configuration change in controlled steps.

Lessons for your own designs

1. An empty value should never mean “everything”. Filters that silently disappear when a parameter is missing or blank are dangerous, especially on endpoints that feed deletions. Reject unclear input instead of guessing. Lesson 1.5 covers API design.
2. Automation needs a size limit. A task that normally deletes a few things a day should refuse, or at least stop and ask, when it’s about to delete a thousand. That’s the circuit-breaker idea from lesson 6.1, applied to changes instead of requests.
3. Roll out changes gradually, and check health between steps. Then a bad change hurts a few customers, not a quarter of them.
4. Plan how you’ll undo it. Before automating a deletion, ask how long it would take to restore everything it could touch.

How this shows up in interviews

When you design a delete or clean-up job, say how you’d stop it from deleting too much: a maximum per run, a dry-run mode, soft deletes that can be undone, and alerts when deletions spike. Interviewers rarely expect this, so it stands out. And if you describe an API filter, mention that missing or empty filters should be rejected, not treated as “match everything”. This outage is a perfect one-sentence example.