Step 2 of 16 · 1 min

What we're talking about

Throughout this lesson, picture data with several copies: a leader and replicas (lesson 2.5), a cache in front of a database (lesson 2.4), or several databases that all accept writes (coming up). A consistency model is a promise about what reads can return, given the writes that have happened.

Note that this is a different meaning of "consistency" from the "C" in ACID (lesson 1.4), which is about a database's own rules, like unique keys, holding. Same word, different idea. It's one of the more confusing bits of vocabulary in the field.

Step 3 of 16 · 1 min

The strongest: linearizability

The strongest common promise is linearizability, sometimes just called strong consistency. It means: the system behaves as if there were only one copy of the data, and every operation happens at a single instant somewhere between when it started and when it finished.

In practice: once a write has completed, every read that starts afterwards, by anyone, anywhere, sees it (or something newer). There's never a moment where one person sees the new value and another, reading later, sees the old one.

That's what the giveaway needed. "Is this code still unclaimed? If so, claim it" must have one answer for everyone. If two servers each check a replica that hasn't heard about the other's claim, both succeed. Usernames (lesson 5.1), money, stock levels, and locks (lesson 6.5) all need this.

The cost: to guarantee it, the copies have to coordinate on every write, and often on reads too. That means waiting on round trips, sometimes between data centers or regions, and if the copies can't talk to each other, some requests must fail rather than risk a wrong answer. Lesson 5.4 is about that trade.

Chirp fixed the giveaway by doing the claim as a single conditional update on the leader of the shard that held the code:

UPDATE giveaway_codes SET claimed_by = :user, claimed_at = now()
WHERE code = :code AND claimed_by IS NULL;

If this changes one row, the user won. If it changes none, someone got there first. Because there's one leader for that row and the row lock serializes the updates (lesson 1.4), exactly one person can win. Reading from a replica first and then writing was the bug.

Step 4 of 16 · 1 min

The weakest: eventual consistency

At the other end is eventual consistency: if writes stop, all copies will eventually agree. That's all. It says nothing about what you'll see in the meantime, or how long "eventually" is.

It sounds alarmingly weak, but most of Chirp's data is fine with it. A like count that's three likes behind for a second, a follower count that lags, a search index that's a second out of date (lesson 3.4), a post appearing in followers' feeds a few seconds after it's written (lesson 4.4). Nobody is harmed, and in exchange, those systems can be fast, cheap, and keep working when parts of them fail.

The trouble with eventual consistency isn't usually the delay. It's that, without anything else promised, a user can see strange sequences of states. That's where the in-between guarantees come in.

Step 5 of 16 · 1 min

The ones in between

Between linearizability and "eventually," there are several smaller promises. Each fixes one specific kind of strangeness, and each is much cheaper than full linearizability. You've met some of them already.

Read-your-writes. After you write something, you will see it. Others might not yet. That's lesson 2.5's "my post disappeared" fix.

Monotonic reads. Once you've seen something, you won't later see an older state. That was Priya's 12-then-11 comment count: her refreshes were going to different replicas, one more up to date than the other. Lesson 2.5's fix: stick each user to one replica.

Consistent prefix (or causal consistency, its fuller form). If one thing happened because of another, everyone sees them in that order. A reply never appears before the question it answers. A "Meera liked your post" notification never appears before the post. That was the second bug.

These are often called session guarantees, because they're promises to one user's session, not to the whole world. They're what makes a system that's only eventually consistent feel consistent to the person using it.

Here's how the main models line up:

Step 6 of 16 · 1 min

Causality, and the reply before the question

Why did Meera's reply arrive first? Chirp's comments lived on the post author's shard, but comment notifications and the activity view were built by an event consumer reading from two Kafka partitions (lesson 4.1). The question and the reply were written by different users, so their events were keyed differently and went into different partitions. One partition's consumer was a few seconds behind the other.

The fix for this particular case was to key comment events by post ID, so all comments on one post go through one partition, in order. Generally, if events on one thing must be seen in order, they need to go through a single ordered path: one partition, one leader, one log.

When that's not possible (writes to unrelated things that still have a cause-and-effect relationship), systems track causality explicitly: each write carries a record of what its writer had seen when writing, and a copy holds back a write until everything it depends on has arrived. That's the idea behind version vectors and causal consistency in some databases. It costs more bookkeeping, but much less than linearizability, because it doesn't need copies to wait for each other on every write.

Step 7 of 16 · 1 min

Quorums: tuning consistency with numbers

Leaderless databases (lesson 2.5 mentioned them; Cassandra and Dynamo-style stores) don't have one leader. Instead, each piece of data is stored on N replicas, and you choose two numbers:

• W: how many replicas must confirm a write before it counts as done.
• R: how many replicas a read asks, taking the newest answer it gets.

If R + W > N, every read set overlaps every write set by at least one replica, so a read should always reach at least one replica that has the latest successful write. With N = 3, the common choice is W = 2 and R = 2: writes survive one replica being down, reads survive one replica being down, and they overlap.

Quorums are useful, but "R + W > N means strongly consistent" is a myth, and it's worth knowing why:

• Concurrent writes. Two writes to the same key at the same moment can land on different overlapping replicas in different orders. The quorum doesn't decide which one wins. Something else has to (next section).
• A failed write isn't undone. If a write reaches one replica but not W, the client is told it failed, but that one replica still has it, and later reads may return it.
• Sloppy quorums. Some systems, when the usual replicas are unreachable, accept writes on other nodes temporarily and pass them on later. That keeps writes working, but reads from the usual replicas won't see them for a while.

So quorums give you a dial between latency, availability and how fresh reads are. For true linearizability, you need more: a single leader per key, or a consensus protocol (lesson 6.5).

Step 8 of 16 · 2 min

When two copies disagree

With a single leader, writes to one row happen in one order, so there's never a question of which one wins. As soon as more than one copy accepts writes (leaderless databases, or several regions each taking writes, lesson 5.5), two people can change the same thing at the same time on different copies. Those are conflicts, and something has to resolve them.

Last write wins, and what it loses

The simplest rule is last write wins (LWW): each write carries a timestamp, and when two versions meet, the one with the later timestamp is kept. Cassandra does this by default.

Two problems:

• The "loser" is silently thrown away. If Priya edits her bio on her phone and her laptop at nearly the same moment, one edit disappears with no error. For a bio, maybe acceptable. For a shopping cart, two people adding items to a shared list, or a collaborative document, it's data loss.
• Clocks lie. Different machines' clocks drift (lesson 4.3). If one server's clock is two seconds fast, its writes win for two seconds even when they happened earlier. An edit made after another can lose to it.

Better options

• Avoid conflicts by design. Route all writes for a given thing to one place: one leader per key, or one home region per user (lesson 5.5). Most of Chirp works this way, which is why conflicts are rare.
• Make changes that can't conflict. New posts and new likes are new rows with new IDs, so two of them never compete for the same row. Designing data as additions rather than overwrites removes most conflicts.
• Merge, rather than pick. For some data, there's a correct way to combine two versions. A set of tags edited on two devices can be merged by taking both additions. A counter incremented on two copies can be merged by adding both increments (each copy keeps its own count, and the total is the sum, a trick related to lesson 4.5's sharded counters). Data types designed so that any two copies can always be merged automatically are called CRDTs (conflict-free replicated data types). They power collaborative editors and some distributed databases.
• Keep both, and ask. Store both versions (detected with version vectors, which can tell "one happened after the other" from "these happened at the same time") and let the application or the user choose. Some databases return all conflicting versions to the application on the next read.

Step 9 of 16 · 1 min

Choosing, feature by feature

The most useful habit from this lesson is to stop asking "is our system consistent?" and ask, for each feature, what's the weakest guarantee it can tolerate? Weaker guarantees are faster, cheaper and stay up when things break. Use the strong ones only where they're needed.

Here's Chirp's list:

Most of the system sits at the bottom of the table, and that's what lets it scale.

Step 10 of 16 · 1 min

From the real world: the item that came back

Amazon's 2007 Dynamo paper, which inspired Cassandra and much of the "eventually consistent" world, is unusually frank about what its consistency choice costs, using the shopping cart as its example.

Amazon decided the cart must always accept writes. A customer who clicks "add to cart" during a network problem and gets an error may simply leave, so availability beat consistency. Dynamo therefore lets any replica accept a write even when replicas can't talk to each other, and tracks the resulting versions with vector clocks, so it knows when two versions are concurrent rather than one being newer.

When concurrent versions meet, Dynamo doesn't pick one. It hands both to the cart service, which merges them by taking the union of the items. That guarantees an addition is never lost, whichever replica it went to. The paper is explicit about the side effect: an item the customer deleted can reappear, if the deletion happened on one version and the item still existed in the other.

That's the same trade-off as this lesson's cart exercise, made by a real business with real numbers behind it: re-showing a removed item is an annoyance; losing an added item is lost revenue. The key move is deciding per feature which mistake is cheaper, then choosing the model and the merge rule to make only that mistake.

Step 11 of 16 · 1 min

Going deeper: write skew, the anomaly that isn't about replicas

Most of this lesson is about copies disagreeing. There's another consistency problem that happens inside a single database, with no replicas at all, and it surprises people because both transactions look correct.

A classic example: a hospital rule says at least one doctor must be on call. Alice and Bob are both on call, and both feel unwell. Each clicks "go off call" at the same moment. Each request runs a transaction:

BEGIN;
SELECT count(*) FROM shifts WHERE on_call AND shift_id = 42;  -- sees 2
-- 2 > 1, so it's safe for me to leave
UPDATE shifts SET on_call = false WHERE doctor = 'me' AND shift_id = 42;
COMMIT;

Under snapshot isolation (PostgreSQL's REPEATABLE READ, and the default in many databases), each transaction sees a snapshot where both doctors are on call, and each updates a different row. There's no conflict on any single row, so both commit. Nobody is on call. This is write skew: two transactions read overlapping data, make decisions based on it, and write to different places, and together they break a rule neither broke alone.

It shows up in ordinary apps: two people claiming the last seat, a username check-then-insert without a unique constraint, a budget that two expenses each "fit" into.

The fixes, from strongest to most targeted:

• Serializable isolation. PostgreSQL's SERIALIZABLE level (serializable snapshot isolation) detects that each transaction read something the other changed, and aborts one; the app retries it. It costs some throughput and requires retry handling.
• Lock what you read: SELECT … FOR UPDATE on the rows the decision depends on, so the second transaction waits for the first.
• Turn the rule into a constraint on one row: a unique index, or a counter row both transactions must update, so the database's normal row-level conflict detection catches it.

Note the vocabulary: this is serializability (transactions behave as if run one at a time), which is a different guarantee from linearizability (one copy, reads see the latest write). A system can offer either without the other.

Step 12 of 16 · 1 min

What Chirp changed

• The giveaway claim became a single conditional UPDATE on the leader. Every "check, then act" on shared state was reviewed for the same bug.
• Users stick to one replica per shard for their session, so counts don't flicker.
• Comment events are keyed by post ID, so a post's comments are always processed in order.
• Wrote down the consistency each feature needs (the table above) in the design guide, and made it a question in design reviews: "what's the weakest guarantee this can live with?"

Step 13 of 16 · 1 min

In the simulation

Stage 4 adds replicas in more places and more consumers between them. The simulation tracks complaints by type: flickering counts (monotonic reads), "my post vanished" (read-your-writes), replies before questions (causal), and double-claimed items (linearizability). Each one points to a specific guarantee you haven't provided where it was needed.

Open the simulation →

Step 14 of 16 · 1 min

If you're here for an interview

Consistency questions come up as soon as there's a replica in your design. Strong answers:

• Use the right words for the right things: linearizable, causal, read-your-writes, monotonic, eventual. And don't confuse them with ACID's "C."
• Pick per feature: "Likes can be eventually consistent; usernames need to be linearizable, so they go through one leader with a unique constraint."
• Know quorums and their limits: R + W > N, and why it doesn't guarantee linearizability on its own.
• Have a conflict strategy if more than one copy accepts writes, and know why last-write-wins can lose data.

Step 15 of 16 · 2 min

Try these yourself

Step 16 of 16 · 1 min

Further reading

• Martin Kleppmann, Designing Data-Intensive Applications, Chapter 5 (replication lag, quorums, conflicts) and Chapter 9 (linearizability, ordering and causality).
• Jepsen, "Consistency Models" (jepsen.io/consistency): a clear map of how the models relate, with definitions.
• Werner Vogels, "Eventually Consistent," Communications of the ACM, 2009.
• Marc Shapiro et al., "Conflict-free Replicated Data Types," 2011.
• Giuseppe DeCandia et al., "Dynamo: Amazon's Highly Available Key-value Store," SOSP 2007, sections 4.4 (data versioning) and 6 (lessons).
• Martin Kleppmann, Designing Data-Intensive Applications, chapter 7, "Write Skew and Phantoms"; PostgreSQL documentation, "Transaction Isolation" (Serializable).