Before we start

If you were travelling, banking or visiting a hospital on 19 July 2024, you may remember this day. Screens around the world showed the Windows “blue screen of death”, restarted, and showed it again. Airport departure boards, bank branches, supermarket tills and TV channels all went down together.

It wasn’t a cyberattack. It was a routine update to CrowdStrike Falcon, security software installed on millions of company computers. The bug inside it is almost painfully simple. The scale is what made it the largest IT outage in history.

The system as it was

CrowdStrike’s Falcon sensor protects computers by watching what happens on them: programs starting, files changing, programs talking to each other. To see all of that, part of it runs inside the Windows kernel, the core of the operating system. Many security products do this, because it gives them the deepest view.

The trade-off: a bug in normal software crashes that program. A bug in the kernel crashes the whole computer.

The sensor gets two kinds of updates:

• Sensor releases: new versions of the software itself. Customers can choose how quickly to adopt them.
• Rapid Response Content: small, frequent updates with new detection rules, sent to every sensor so they can spot new attacks quickly. These went everywhere at once.

The rules fit into templates called Template Types, built into each sensor release. A Template Type says what data a rule can look at. Each rule (a Template Instance) says what to look for in that data.

What happened

1. Feb 2024Sensor version 7.11 adds a new Template Type for spotting attacks that abuse how programs talk to each other. It’s defined with 21 fields; the code feeding it supplies 20.
2. Mar–Apr 2024Several rules of this type are deployed. They all use a wildcard for field 21, so the missing field is never read.
3. 19 Jul, 04:09 UTCA routine content update (Channel File 291) adds two new rules. One compares a real value in field 21.
4. 04:09 onwardWindows machines that receive it crash, restart, and crash again.
5. 05:27 UTCCrowdStrike reverts the update. Machines that hadn’t received it yet are safe.
6. Days that followCrashed machines can’t download the fix. IT teams restore them one by one, often by hand.
7. Late JulyCrowdStrike reports about 99% of Windows sensors back online.

The 21st field

This is the whole bug. A new Template Type was defined with 21 input fields, but the part of the sensor that collected the data only provided 20.

Why didn’t anyone notice for five months? Because every rule written for it so far had put a wildcard in field 21, meaning “match anything”. To match anything, the sensor never has to look at the field, so the missing 21st value was never read. The tests used wildcards too.

Then, on 19 July, a new rule put a real value in field 21. The sensor went to compare it, reached for the 21st item in a list of 20, and read memory past the end of the list. In the kernel, that’s fatal. Flip the toggle to see why months of testing passed:

CrowdStrike’s analysis lists the combination that caused it: the mismatch between the 21 inputs the validator expected and the 20 the sensor supplied, a missing check on reading past the end of the list, and no test using a real value in field 21.

Why it hit millions of machines at once

Two things turned a bug into a global event.

It went everywhere at once. Rapid Response Content was designed to reach every sensor quickly, because new attacks spread fast. There were no stages: no first 1%, then 10%, then everyone. So within minutes, every connected Windows machine running the sensor received it. Microsoft estimated about 8.5 million devices were affected, under 1% of all Windows machines, but concentrated in exactly the businesses that run airports, hospitals and banks.

Crashed machines couldn’t fix themselves. CrowdStrike pulled the update after 78 minutes. But a computer stuck in a crash loop can’t download a fix. Each one had to be started in a special recovery mode and have the bad file deleted, often by a person standing at the machine. For companies with encrypted disks, that also meant finding a recovery key for every single computer. That’s why the outage lasted days, not hours.

What they changed

CrowdStrike’s report lists several fixes:

• Check the number of fields when building the sensor, so a Template Type and its data source can’t disagree again.
• Check boundaries at runtime, so a rule asking for a field that doesn’t exist is rejected instead of crashing.
• Fix the validator, and test new rules inside the real interpreter, not just against the definition.
• Staged rollouts for content: new rules first go to internal machines, then a small canary group, then wider rings, with waiting time between each to watch for crashes.
• Customer control: customers can now choose where and when content updates are deployed.

Lessons for your own designs

1. Configuration is code. “It’s only a content update” or “it’s only a config change” is how many of the biggest outages begin. If it can change behaviour, it needs the same staged rollout as code.
2. Roll out in stages, and watch between them. 1% first, then more, with automatic stops if crash rates rise. This one practice would have turned a global outage into a small one. It’s also the lesson of the Cloudflare 2026 post.
3. Never trust the shape of input. Code should check that a field exists before reading it, especially where a mistake is catastrophic.
4. Test the unusual path, not just the common one. Every test passed because every test used a wildcard.
5. Think about recovery when the thing you’d use to fix it is broken. If an update can stop machines from booting, it can stop them receiving the fix.

How this shows up in interviews

Whenever you deploy something in a design, whether code, configuration or machine-learning models, you can say how it rolls out: a canary first, then gradual stages, with health checks and automatic rollback between them. Interviewers increasingly ask “how would you deploy this safely?”, and CrowdStrike is the example everyone recognises. It’s also a strong answer to “what’s the blast radius of this change?”: the bigger the potential blast radius, the slower and more careful the rollout should be.