https://systemdesignroad.in/blogs/ Real outages and engineering stories retold in plain words: what broke, why, and what the engineers learned. en https://systemdesignroad.in/blogs/cloudflare-byoip-2026/https://systemdesignroad.in/blogs/cloudflare-byoip-2026/Tue, 10 Nov 2026 00:00:00 GMTA new clean-up task asked for "prefixes pending delete" but sent the flag with no value. The API read that as "all prefixes", and the task began removing customers' addresses from the internet. https://systemdesignroad.in/blogs/clerk-query-plan-flip/https://systemdesignroad.in/blogs/clerk-query-plan-flip/Tue, 10 Nov 2026 00:00:00 GMTA routine automatic statistics refresh sampled a column that was almost always empty, concluded it was always empty, and changed how a hot query ran. Over 95% of sign-in traffic was rejected. https://systemdesignroad.in/blogs/aws-dynamodb-dns-2025/https://systemdesignroad.in/blogs/aws-dynamodb-dns-2025/Tue, 10 Nov 2026 00:00:00 GMTTwo copies of DynamoDB's DNS automation raced each other. One applied an old plan, the other deleted it, and DynamoDB's address in us-east-1 pointed nowhere. Apps around the world went down, and EC2 took until the afternoon to recover. https://systemdesignroad.in/blogs/crowdstrike-2024/https://systemdesignroad.in/blogs/crowdstrike-2024/Tue, 10 Nov 2026 00:00:00 GMTA routine security content update told millions of Windows PCs to read a 21st field from a list that only had 20. The code ran inside the operating system's kernel, so every one of them crashed, again and again. https://systemdesignroad.in/blogs/roblox-73-hours/https://systemdesignroad.in/blogs/roblox-73-hours/Tue, 10 Nov 2026 00:00:00 GMTMillions of players locked out for three days. The cause sat deep inside Consul, the system everything else used to find each other, and the monitoring that should have pointed to it depended on the same system. https://systemdesignroad.in/blogs/slack-january-4/https://systemdesignroad.in/blogs/slack-january-4/Tue, 10 Nov 2026 00:00:00 GMTEveryone came back from the holidays on the same morning. A network hub couldn't keep up, servers looked idle while they waited, and the autoscaler removed capacity at exactly the wrong moment. https://systemdesignroad.in/blogs/cloudflare-half-broken-switch/https://systemdesignroad.in/blogs/cloudflare-half-broken-switch/Tue, 10 Nov 2026 00:00:00 GMTA network switch failed for six minutes, but only partly. The machines it connected disagreed about who could reach whom, a database failover kicked in, and a hidden bug turned six minutes into six and a half hours. https://systemdesignroad.in/blogs/gitlab-database-deletion/https://systemdesignroad.in/blogs/gitlab-database-deletion/Tue, 10 Nov 2026 00:00:00 GMTA tired engineer, a late night, and a command run on the wrong server. Then the real shock: of five ways to restore the data, almost none worked. GitLab livestreamed the recovery.