Before we start

Most of us have deleted a file we didn’t mean to, and felt that drop in our stomach. Then relief: “it’s fine, it’s backed up”. Now imagine going to the backup and finding it empty.

That’s the GitLab story. The deletion itself is a very human mistake, made at the end of a long, stressful evening. The real lesson is about everything that was supposed to make that mistake harmless, and didn’t.

GitLab is also known for how it handled it: it shared a public document of the recovery as it happened, livestreamed the work, and published a detailed postmortem.

The system as it was

GitLab.com ran its main Postgres database on one primary server, db1, with one standby replica, db2, kept up to date by streaming replication. (Lesson 2.5 explains replication.) Only the primary served traffic; the replica was there to take over if the primary failed.

If the replica fell too far behind, the fix was to empty its data folder and copy everything fresh from the primary using a tool called pg_basebackup.

On paper, the data was protected five different ways:

1. A daily pg_dump of the database, uploaded to cloud storage.
2. Cloud disk snapshots.
3. The replica itself.
4. Regular disk-level (LVM) snapshots.
5. Webhooks and other data copied to a staging environment.

What happened

1. ~17:20An engineer takes a disk snapshot of production by hand, to refresh the staging environment. This will matter later.
2. ~19:00Database load jumps, apparently from spam. Users can’t comment. It takes hours to bring under control.
3. ~23:00The replica falls behind. The primary has already discarded the data it needs, so it must be rebuilt from scratch.
4. 23:00–23:30pg_basebackup seems to hang, with no output. Engineers change settings and try again, late at night.
5. ~23:30An engineer deletes the data folder on what they believe is the replica. It’s the primary. They stop it a second or two later.
6. After 23:30About 300 GB is gone. The team starts checking backups, one by one.
7. Feb 1, ~17:00After about 18 hours of copying, the database is restored to its 17:20 state.
8. Feb 1, ~18:00Webhooks restored, checks done. GitLab.com is back.

The replication lesson shows how the replica fell behind:

The mistake

A replica that has fallen behind needs to catch up from the primary’s log of changes. But GitLab’s primary had already thrown away the log entries the replica needed, so the only option was a full rebuild.

The rebuild tool appeared to hang, printing nothing. (GitLab later found that waiting silently was normal behaviour; it just wasn’t documented.) It was late, and the engineer had been fighting the spam problem for hours. They decided to clear the replica’s data folder and start again.

They ran the command in a terminal connected to db1, the primary, instead of db2. Within a second or two they realised and stopped it. About 300 GB was already gone.

It’s worth saying clearly: this is a mistake any of us could make. Two terminals, two hostnames that differ by one character, a long night. GitLab’s response, rightly, was to fix the system, not blame the person.

Five backups, one that worked

Then the team went looking for a way back. Step through what they found:

The most painful one is the first. The daily database dumps had been failing for a while. The backup job ran an older version of pg_dump (9.2) against the newer database (9.6), so it produced nothing. The system did send error emails, but they were being rejected because of an email setting, so nobody ever saw them. The storage bucket was simply empty.

And the replica, the thing that feels most like a backup, couldn’t help either. A replica faithfully copies every change, including deletions, and in this case it was the server being rebuilt, so its data had been cleared as part of the fix.

What saved them was luck: the disk snapshot an engineer took by hand at 17:20 that afternoon, to give the staging environment fresher data. Restoring from it meant losing about six hours of work instead of a full day.

A slow road back

The snapshot was sitting on the staging environment, which used slow, throttled cloud disks. Copying the data back to production took around 18 hours. GitLab’s post says plainly that “the bottleneck was in the drives”.

When the database came back, it was as it had been at about 17:20. GitLab estimated that changes from roughly 17:20 to 23:30 were lost, including at least 5,000 projects, 5,000 comments and about 700 new user accounts. The Git repositories themselves were safe; they lived elsewhere.

One clever detail: before going live, they bumped every ID sequence in the database up by 100,000. The lost hours had already handed out IDs, and some of those IDs might still be referenced in emails, links or other systems. Jumping ahead meant the restored database would never hand out the same ID twice. Lesson 4.3 talks about why IDs must never be reused.

What they changed

GitLab listed many follow-ups. The most important:

• Hourly disk snapshots of the production database, and cloud snapshots for database servers too.
• Point-in-time recovery: continuously archiving the database’s change log, so it can be restored to any moment, not just to when a snapshot happened.
• Testing restores automatically, so a backup that doesn’t restore is noticed immediately.
• Monitoring the backups themselves, and alerting when they fail.
• Clearer terminal prompts showing which host you’re on, so production looks different from everything else.
• Someone owning data durability, so it’s somebody’s job, not everybody’s assumption.
• More replicas and better documentation for rebuilding them.

Lessons for your own designs

1. A backup you haven’t restored is not a backup. The only proof a backup works is restoring it, regularly and automatically.
2. Replication isn’t a backup. Replicas copy mistakes, deletions and corruption as faithfully as good data. You also need copies from the past.
3. Watch your backups like production. A silent failure is the worst kind. Alert when a backup is missing, too small or too old.
4. Know your recovery time, not just your recovery point. GitLab had a usable snapshot, but restoring it took 18 hours. Test how long a full restore takes.
5. Make dangerous actions hard to do by accident. Clear prompts, confirmations on production, and tools that check which machine they’re on.

How this shows up in interviews

Almost every design has a database, and interviewers sometimes ask “how would you recover if someone deleted the data?” Saying “we have replicas” is the trap this story warns about. A strong answer separates the two jobs: replicas for staying up when a machine fails, and backups (snapshots plus point-in-time recovery) for going back in time after a mistake. Then add that backups must be restored regularly to prove they work, and that you’d know how long a full restore takes. Those two sentences come straight from GitLab’s hardest day.