Before we start

Picture the first day back at work after a long holiday. Everyone arrives at the same time, everyone’s inbox is full, and the lift is packed. Slack’s January 2021 outage was the internet version of that morning.

Over the holidays, few people use Slack, so caches on phones and laptops go stale. On the first Monday back, everyone opens it at once, and every client has to download a lot of fresh data. Slack’s own servers handled that fine. One piece of network in between did not.

The system as it was

Slack runs on AWS. Over the years it had split its infrastructure into many separate accounts and private networks (VPCs), which is good for security and isolation. Those networks were connected through AWS Transit Gateways: managed network hubs that route traffic between VPCs. Slack’s write-up notes they are “intended to scale transparently”, meaning AWS grows them automatically as traffic grows.

Two more pieces matter for this story:

• The web tier is a large group of servers that grows and shrinks automatically. It scales based on two signals: CPU use, and how many Apache worker threads are busy.
• provision-service sets up and tests every new server. It talks to other internal systems and to AWS, over the same network.

What happened

1. Early morningTraffic climbs much faster than usual as people return. One Transit Gateway starts dropping packets.
2. 6:57About 99% of messages are sending successfully. Normal is above 99.999%.
3. 7:00The usual top-of-the-hour rush, plus the struggling network, overwhelms the web tier. Slack becomes unavailable.
4. 7:00The autoscaler removes servers. Engineers’ terminal sessions end as their servers are shut down.
5. 7:01–7:15About 1,200 new servers are requested. provision-service can’t keep up and starts failing.
6. 8:15provision-service is fixed. Healthy servers begin joining.
7. 9:15Enough servers are working. Slack is “degraded, not down”.
8. 10:40AWS has raised the Transit Gateway’s capacity everywhere. The network is back to normal.

The lesson on scaling shows the outage from the outside:

The feedback loop

This is the heart of the story. Watch what the autoscaler saw, step by step:

Let’s slow down on the key moment. A web server that sends a request over a slow network just waits for the answer. Waiting uses almost no CPU. So as the network got worse, CPU use on the web servers went down, and the autoscaler read that as “we have too many servers”. It removed some, right as the morning peak arrived.

A few minutes later, the other signal took over: every worker thread was stuck waiting, so “busy threads” shot up, and the autoscaler asked for a huge number of new servers at once. That’s when the second problem appeared.

The service that builds servers broke too

Every new server has to be set up by provision-service. Normally that’s a steady trickle. Now it had about 1,200 to handle in a quarter of an hour, while talking to AWS over the same unhealthy network.

It hit two limits. The most significant was the Linux limit on how many files (including network connections) one process can have open. The other was an AWS quota. New servers came up half-configured and broken, and the autoscaling groups filled up to their maximum size with servers that couldn’t serve traffic.

Once engineers fixed provision-service, at about 8:15, healthy servers started arriving. Meanwhile, Slack’s load balancers went into “panic mode”, spreading traffic across whatever servers they had, while retries and circuit breakers kept some requests flowing. Lesson 6.1 covers circuit breakers.

Flying without dashboards

One more thing made this harder. Slack’s dashboard and alerting services ran in a different VPC from the databases behind them, so they too depended on the Transit Gateways. When the network struggled, the dashboards failed. Engineers could still query their metrics systems directly, but that’s much slower than glancing at a dashboard when every minute counts.

Sound familiar? It’s the same trap as the Roblox outage: monitoring depending on the thing that’s broken.

What they changed

• AWS reviewed how Transit Gateways scale when traffic jumps suddenly, and Slack asked for capacity to be raised ahead of the next post-holiday return.
• Dashboards were moved into the same VPC as their databases, so they no longer depend on the Transit Gateways.
• provision-service gets regular load tests, since this event asked more of it than any scale-up before.
• Autoscaling health checks and settings were re-examined, so a scaling burst can’t overload provisioning.

Lessons for your own designs

1. Choose scaling signals carefully. CPU measures work being done, not work waiting to be done. A server stuck waiting on the network or a database looks idle. Signals like queue length, busy workers or response time often tell the truth better.
2. Scale-up has dependencies too. Adding servers relies on provisioning tools, image downloads, config services and cloud quotas. If those can’t handle a sudden burst, scaling up can make things worse.
3. Managed doesn’t mean infinite. “Scales automatically” usually means “scales at some speed”. A sudden jump can outrun it.
4. Keep monitoring off the broken path. Dashboards that need the same network as production will fail exactly when you need them.
5. Know your predictable spikes. The first workday after a holiday, a big match, a product launch: some surges can be seen coming, and capacity can be raised ahead of time.

How this shows up in interviews

When you add autoscaling to a design, interviewers sometimes ask “what does it scale on?” Answering “CPU” is fine for a start, but this story lets you go further: say you’d also watch busy workers or request latency, because servers waiting on a dependency look idle. And if you’re asked about handling a predictable spike, mention scaling up before it arrives, rather than reacting during it. Lesson 2.1 covers scaling up and out.