Before we start

Think of a school where every message goes through one office: room changes, keys, announcements. Most days you never notice the office. Then one day it slows down, and suddenly nothing in the school works, including the noticeboard that would have told everyone what was wrong.

That’s roughly what happened to Roblox. They ran their own data centres, with more than 18,000 servers and 170,000 containers. Almost all of it relied on one system, Consul, to find each other, and when Consul struggled, the whole platform went with it.

The system as it was

Roblox ran on three tools from a company called HashiCorp, often called the “HashiStack”:

• Nomad decides which server each container runs on, and checks they’re healthy.
• Consul is a directory: services ask it “where is the payments service?” It also stores small settings and handles locks.
• Vault hands out secrets, like passwords and keys.

Consul itself was a small cluster: five servers that vote on every change, plus five read-only copies. The five voting servers use Raft, an agreement method where one leader handles every write and the others follow. Lesson 6.5 explains how Raft works.

That’s a lot of the company resting on five machines. Most of the time that’s fine. But it means that if those five machines are slow, everything is slow.

What happened, hour by hour

1. Oct 27, 14:00A new Consul feature called “streaming” is switched on for a busy traffic-routing service.
2. Oct 28, 13:37One Consul server is using a lot of CPU, and Vault is struggling. Players aren’t affected yet.
3. Oct 28, 16:35The number of players online drops to half of normal.
4. Oct 28, 19:00The team has moved Consul onto new, bigger machines. It’s still unhealthy.
5. Oct 29, 02:00They restore Consul from a snapshot. It degrades again within hours.
6. Oct 29, 16:00A second full restart. By early the next morning Consul is unhealthy again.
7. Oct 30, 15:51Streaming is turned off everywhere. Write delays drop sharply.
8. Oct 31, 05:00Consul and the caches are healthy, 61 hours in.
9. Oct 31, 10:00Players start to be let back in, a slice at a time.
10. Oct 31, 16:45Everyone is back, 73 hours after it began.

Here’s how the failure spread, from the lesson on consensus:

Problem one: a busy lock inside a new feature

Consul has two ways for services to say “tell me when something changes”. The older way is long polling: ask, and the server holds the question open until there’s news. The newer way, streaming, was meant to be more efficient.

The day before the outage, Roblox turned streaming on for one very busy service, and that service had grown by half. Streaming happened to funnel a lot of work through fewer shared internal locks (Go channels, in Consul’s code). Under a heavy mix of reads and writes, everything queued up on one of them, and writes got stuck behind it.

Worse, the team had just moved Consul to bigger machines, 128 cores instead of 64. That sounds like it should help. But more cores fighting over the same lock can make the queue worse, not better, especially on machines whose memory is split between two processor sockets. More hardware was making it slower.

Problem two: a database file that never shrinks

Consul keeps its Raft log in a small database called BoltDB. Every so often it saves a snapshot and deletes old log entries. But BoltDB never gives disk space back. It just remembers which pages are free, in a list called the freelist.

Over time, Roblox’s file grew to 4.2 GB while holding only 489 MB of real data. The freelist had to be rewritten on every single write. On an unhealthy leader, simple writes started taking two to three seconds.

This one wasn’t solved during the outage at all. HashiCorp found it days later. The fix was simple once known: compact the database file.

Why it took three days to diagnose

Here’s the part that turned a bad day into three bad days. Roblox’s own write-up says critical monitoring systems “relied on affected systems, such as Consul”, and that this “severely hampered the triage process”.

So the engineers were trying to diagnose Consul with dashboards that needed Consul. They tested theories one at a time: faulty hardware, too much traffic, too many CPU cores. Each test meant resetting the cluster and waiting to see. Flip the toggle to see what independent monitoring changes:

This is a classic trap, and it shows up in many big outages: the tools you use to fix a system shouldn’t depend on that system. Lesson 6.4 covers what good monitoring looks like.

Coming back from zero

Once Consul was healthy, they couldn’t just switch everything back on. When a big system starts completely cold, everything it normally takes for granted is missing.

• Empty caches. Roblox’s caches normally answer around a billion requests a second. Starting empty, every request would go to the databases behind them and overwhelm them. So the caches were redeployed and refilled carefully first.
• Old information. Some scheduling data stored in Consul was out of date, and the deployment tools had been built for small updates, not for rebuilding everything at once.
• Letting players in slowly. Using DNS, they admitted randomly chosen players about 10% at a time, checking database load and cache health after each step, until everyone was back at 16:45 on 31 October.

Starting a system from nothing is a skill, and it’s rarely practised. Lesson 2.4 explains why a cold cache can take a database down.

What they changed

• Monitoring that stands alone. They removed the circular dependency, so monitoring no longer relies on the systems it watches, and alerts now fire earlier.
• Not one cluster for everything. Critical services got their own separate Consul clusters, so one service’s load can’t slow everyone else down.
• A second site. They built a data centre in a different location and started on multiple zones, so one site having problems isn’t the end of the world.
• Fixing the root causes. Moving Consul to bbolt, an improved version of BoltDB, and only bringing streaming back after testing at Roblox’s scale.
• Practising cold starts. Redesigning cache deployment so it can bootstrap from nothing.

They reported no significant production incident during their busy December holiday traffic two months later.

Lessons for your own designs

1. Find the thing everything depends on. Draw your system and ask: if this one box stopped, what else would stop? Consul was that box. Often it’s a config service, a DNS server, a secrets store or an auth service.
2. Keep your eyes outside the patient. Monitoring, logging and the tools engineers use during an outage must keep working when the main system doesn’t.
3. More hardware isn’t always more speed. When the bottleneck is a shared lock, more cores can make it worse.
4. Practise starting from zero. Know how you’d refill caches and let traffic back in gradually. Turning everything on at once usually knocks it straight over again.

How this shows up in interviews

Interviewers love asking “what’s the single point of failure here?” This story is a perfect answer, because the weak point wasn’t a database or a load balancer, but a coordination service everyone assumed would always be there. If your design uses something like ZooKeeper, etcd or Consul, say what happens when it’s slow, not just when it’s down. And if you’re asked about monitoring, mention that it must not depend on the system it’s monitoring. That one sentence shows real operational experience.