Before we start

Imagine a shop that checks its stock by looking at a few shelves at random. One product is so rare that almost every shelf is empty of it. Today the shop samples ten shelves, finds nothing on any of them, and writes down “we never have this”. The next decision it makes, based on that note, is a bad one.

That’s what happened to Clerk’s database. Postgres didn’t crash, and nothing was misconfigured. It made a reasonable guess from a sample, and the guess was wrong in exactly the wrong place.

The system as it was

Clerk runs on Postgres. When you send Postgres a query, it doesn’t just run it. A component called the query planner first decides how to run it: which index to use, which table to read first, how to join them. It picks the plan it thinks will be fastest.

To decide, the planner uses statistics about each table: how many rows there are, how common each value is, and what fraction of a column is empty (NULL). Postgres collects these by sampling rows, not reading every row, and refreshes them automatically from time to time. That refresh is called ANALYZE.

Most of the time this works beautifully, and you never think about it. (Lesson 1.4 explains how databases find rows.)

What happened

1. 16:11 UTCAn automatic ANALYZE refreshes the statistics, the plan for a hot query changes, and the outage begins.
2. 16:15The incident is reported and escalated.
3. 16:32The team finds a customer with a traffic spike and suspects them.
4. 16:35They block that customer. It doesn’t help.
5. 16:50They realise the spike was the customer’s apps retrying because of the failures, not the cause.
6. 16:55The automatic backup path for sessions hasn’t kicked in, because the database is up, just slow.
7. 17:10A new session fallback is switched on and helps many users, but sign-in is still down.
8. 17:25The automatic ANALYZE is identified as the cause.
9. 17:27ANALYZE is run again by hand. The query goes back to its old plan, and recovery begins.
10. 18:06Performance is stable.

One wrong guess

One column in the affected table was empty almost all the time: 99.9996% NULL. That means a tiny number of rows did have a value.

When ANALYZE sampled the table, the sample was too small to include any of those rare rows. It saw only NULLs, so the statistics said the column was 100% NULL.

Believing that, the planner chose a different plan for a query that runs very frequently. That plan would have been great if the column really was always empty. But the query actually returned over 17,000 rows. Run constantly, the new plan consumed nearly all of the database. Try both sample sizes:

This kind of sudden change is called a query plan flip: same query, same data, but a different plan, and wildly different speed.

How a slow database became rejected requests

When the database slows down, requests don’t fail straight away. They wait. Each waiting request holds a worker, and soon every worker is waiting. To protect themselves, Clerk’s services started turning traffic away: over 95% of requests got a 429 Too Many Requests reply without being handled at all. The few that got through succeeded, but slowly.

That’s actually the system working as designed: rejecting quickly is better than everyone waiting forever. (Lesson 6.3 covers rate limiting and load shedding.) The real problem was upstream, in the database.

Why it took over an hour to find

Clerk’s timeline shows a very human detour. The first thing the team saw was one customer sending a big burst of traffic, so they blocked that customer. It didn’t help. The burst was caused by the outage: that customer’s apps were retrying failed requests.

This is one of the most common traps in incidents: a symptom looks like a cause. Lots of retries look like an attack. A busy database looks like too much traffic.

Their backup path for sessions didn’t kick in either. It was designed to take over when Postgres was down. Postgres was up, just painfully slow, so the failover never triggered. That’s a pattern from many outages: systems are often built for “dead”, not for “slow”.

Once they spotted the automatic ANALYZE in the timeline, the fix took two minutes: run ANALYZE again. With a fresh sample, the statistics were accurate enough, and the planner went back to the old plan.

What they changed

• Alerts for plan flips. In Clerk’s words, “we need to detect it immediately rather than relying on downstream symptoms”.
• Stable plans. They raised the statistics target on that table, so Postgres samples more rows, and rewrote the query so its plan can’t flip. They’re checking other queries for the same risk.
• A better fallback. The session fallback now covers any failure at the origin, not just a dead database.
• Clearer communication during incidents, with a dedicated communications lead and regular status updates.

Lessons for your own designs

1. Your database can change behaviour without any deploy. Statistics refreshes, data growth and version upgrades can all change query plans. For your most important queries, know which plan they use, and alert if it changes.
2. Skewed data is dangerous for sampling. A column that’s almost always empty, or almost always one value, is exactly where a sample can be misleading. Bigger statistics samples help.
3. Design failovers for “slow”, not only “down”. A health check that only asks “are you alive?” won’t notice a database that’s alive but useless.
4. Retries are a symptom before they’re a cause. When one client suddenly sends much more traffic, check whether it’s retrying your errors.

How this shows up in interviews

When you say “we’ll add an index”, strong candidates add a second sentence: how they’d know the database is actually using it, and what happens if the plan changes later. You can mention monitoring slow queries and query plans as part of your observability story. And this outage is a great example of why “the database is up” isn’t the same as “the database is healthy”, which matters whenever you design health checks or failover. Lesson 6.4 covers monitoring.