Step 2 of 16 · 1 min

Getting messages to phones instantly

The prototype used polling: ask the server every few seconds. It's simple, but it has two costs that pull in opposite directions. Poll often, and you send huge numbers of empty requests. Poll rarely, and messages arrive late. At three seconds, Chirp had both problems at once.

There are three better options, each holding a connection open so the server can send data the moment it has some:

• Long polling. The app makes a request, and the server holds it open until there's a message (or until a timeout, say 30 seconds), then responds. The app immediately makes a new request. Messages arrive almost instantly and empty responses are rare. It works over plain HTTP, which makes it robust, but each message still costs a full request.
• Server-Sent Events (SSE). One long-lived HTTP response that the server keeps writing events into. Simple and efficient, but one-way: the server can push, but the client still sends with normal requests.
• WebSockets. A connection that starts as HTTP, then switches into a two-way channel. Either side can send small messages at any time, with very little overhead. This is what most chat apps use.

Chirp Messages used WebSockets on its apps and website, with long polling as a fallback for the few networks that interfere with WebSockets.

Step 3 of 16 · 3 min

Millions of open connections

Here's where chat breaks an assumption from lesson 2.1. Chirp's API servers were stateless: any request could go to any server, and servers could be replaced freely. A WebSocket is the opposite. It's a connection, held open for minutes or hours, between one phone and one specific server. That server is now holding state: "Priya's phone is connected to me."

The napkin math (lesson 1.3) for Chirp:

20M daily users who use messages
peak share online at once:  ~25%     → 5M open connections
per connection memory:      ~20–50 KB (buffers, TLS state, app state)
per server:                 ~100K connections comfortably
→ about 50 connection servers, plus spares

So Chirp split its chat system into two kinds of servers:

• Gateway servers hold the WebSocket connections. They do as little as possible: authenticate the connection, receive messages and pass them on, push messages down to phones. Because they hold connections, they're stateful.
• Chat services behind them do the real work (validating messages, storing them, deciding who receives them), and are stateless, like the rest of Chirp.

Keeping the stateful part small and simple is the main design idea. Gateways rarely need new code, so they rarely need restarting, and that matters a great deal, as you'll see.

Who is connected where?

When Priya sends Meera a message, it arrives at whichever gateway Priya is connected to. To deliver it, the system needs to know which gateway Meera is connected to, if any.

That's a connection registry: a fast lookup from user (or device) to gateway, kept in Redis (lesson 2.4). When Meera's phone connects to gateway 17, gateway 17 records meera:phone → gw-17, with a short expiry it renews while the connection is alive. When she disconnects, the entry is removed, or expires on its own if the gateway crashes.

Delivery then looks like this:

1. Priya's phone sends the message to its gateway (gw-4).
2. gw-4 passes it to the chat service, which stores it and assigns its position in the conversation (next section).
3. The chat service looks up Meera's devices in the registry: phone on gw-17, laptop on gw-9.
4. It sends the message to gw-17 and gw-9 (directly, or through a pub/sub channel per gateway), and they push it down to Meera's devices.
5. If Meera has no connected devices, the message waits in storage, and a push notification (lesson 4.1's push service) goes to her phone.

Deploying gateways without a stampede

The first time Chirp deployed a new version of the gateways, it restarted them all within two minutes. Five million phones were disconnected, and all of them immediately tried to reconnect. Each reconnection meant a TLS handshake (lesson 1.1, expensive for the server), an authentication check, a registry write, and a request for missed messages. The authentication service received in seconds what it normally saw in an hour, slowed down, and reconnections started failing and retrying. It was lesson 6.1's retry storm, triggered by a deploy.

The fixes:

• Restart gateways slowly, a few at a time, and have each one tell its clients to reconnect gradually before it shuts down (drain over a minute, rather than dropping everyone at once).
• Reconnect with backoff and jitter (lesson 1.5) on the client, so a crowd of disconnected phones spreads out over a minute.
• Make reconnecting cheap: let a phone resume its session with a short-lived token rather than a full login, and fetch only what it missed.
• Change gateways rarely. Keep logic out of them, so most deploys touch only the stateless services behind.

Keeping connections alive

A phone's connection can die silently: the phone goes into a tunnel, a mobile network drops the route, a NAT box forgets it. Neither side is told. So both sides send small heartbeats (WebSocket pings) every 30 seconds or so. If the gateway hears nothing for a minute, it treats the connection as dead and cleans up the registry. If the phone hears nothing, it reconnects. Mobile networks often drop idle connections after a few minutes, which is another reason to keep heartbeats going.

Step 4 of 16 · 2 min

Ordering

Priya's two messages arrived in the wrong order because the prototype sorted by the timestamp on the server that received each one. Her two messages took different paths (a retry, two different API servers whose clocks differed by a few milliseconds), and the later one got an earlier timestamp. Lesson 4.3 warned about exactly this: clocks on different machines don't agree precisely.

What a conversation needs is a single order that everyone sees, which is lesson 5.3's consistency question in miniature. Chirp's approach:

• Each conversation has one place that decides its order. Messages are stored partitioned by conversation ID (lesson 5.1), so all messages for one conversation go to the same shard. Within it, each new message gets the next sequence number for that conversation: 1, 2, 3, … assigned in a single-row update (lesson 1.4) or as an atomic counter.
• Clients display messages in sequence-number order, not by any timestamp. Timestamps are only for showing "10:42 pm."

Two people sending at the same moment will get two different sequence numbers, and both their phones will show the same order. That order might not match who pressed "send" first by a few milliseconds, but nobody can tell, and everyone sees the same conversation, which is what matters.

Showing your own message straight away

Waiting for the server before showing your own message would feel slow. So the app shows it immediately, marked as "sending," with a temporary position at the bottom. When the server replies with the real sequence number, the app moves it into place (almost always exactly where it already was).

No duplicates

The duplicates came from retries. Priya's phone sent a message, the response was lost in a tunnel, and the phone sent it again. Lesson 1.5's fix applies directly: the phone generates a client message ID (a UUID, lesson 4.3) when the message is written, and sends it with every attempt. The chat service stores it with a unique constraint per conversation. A retry with the same ID returns the original message's sequence number instead of storing a second copy.

Step 5 of 16 · 1 min

Storing messages

Message storage has a very clear access pattern (lesson 3.1):

• Write: append a message to a conversation. Enormous volume.
• Read: the most recent N messages in a conversation, then older pages as the user scrolls up (cursor pagination, lesson 1.5).
• Almost nothing else.

That's the textbook shape for wide-column storage (lessons 3.1 and 3.2): partition by conversation ID, cluster by sequence number in descending order. Writes are appends, reads are one range scan in one partition. Several large chat services have described storing messages exactly this way; Discord, for example, has written publicly about storing billions of messages in Cassandra and later migrating to ScyllaDB.

The napkin math:

20M users × 40 messages/day      = 800M messages/day
                                 ≈ 9,000/s average, ~30,000/s at peak
× ~300 bytes stored per message  ≈ 240 GB/day, ~90 TB/year (before replication)

That's well past one Postgres server's comfortable write rate (Part 5's starting point) and grows forever, which is why Chirp chose a wide-column store for message bodies, while keeping conversations, memberships and settings in its existing sharded Postgres.

Two things need care:

• Huge conversations make huge partitions. A group chat with years of history can hold millions of messages in one partition, which wide-column stores handle badly (lesson 3.1). The usual fix is to include a time bucket in the partition key, such as (conversation_id, month), so each partition stays a manageable size.
• Very active conversations are hot partitions (lesson 4.5). A 256-person group during a cricket match can get hundreds of messages a second. That's one partition, but at that rate still well within what one node handles. For really huge channels (thousands of members), see the next section.

Step 6 of 16 · 1 min

Receipts: sent, delivered, read

Each message moves through states that the sender sees:

• Sent (one tick): the server has stored it and assigned it a sequence number.
• Delivered (two ticks): at least one of the recipient's devices has received it.
• Read: the recipient has viewed it.

The naive way, a separate "delivered" and "read" record per message per recipient, means several extra writes per message, multiplied by group size. Instead, Chirp stores, per person per conversation, just two numbers: the highest sequence number they've received, and the highest they've read. Because messages are ordered, "read up to 482" means messages 1 to 482 are all read. One small update per person, however many messages they catch up on at once.

For a two-person chat, the sender's ticks come from the other person's numbers. For groups, "read by 14 of 20" comes from comparing each member's read position to the message's sequence number. That's a computation over 20 small rows, cached, not 20 writes per message.

Step 7 of 16 · 1 min

Many devices, and catching up

Priya uses Chirp on her phone, her tablet and the website. Each one needs every message, including the ones she sent from another device. And each needs to catch up after being offline.

Sequence numbers make this simple. Each device remembers, per conversation, the highest sequence number it has. When it reconnects, it asks: "for each of my conversations, give me everything after the sequence number I have." The server replies with exactly the gap: no duplicates, nothing missing. That's lesson 1.5's cursor, applied to every conversation at once.

To avoid asking about thousands of conversations on every reconnect, the server keeps a per-user "conversations changed since…" list, so the device only asks about the few that have new messages.

The laptop-to-phone bug was this: messages Priya sent were delivered to the recipient's devices, but not pushed to her own other devices. The fix was to treat the sender's other devices as recipients too.

Step 8 of 16 · 1 min

Presence and typing

"Online" dots and "Meera is typing…" look small and are surprisingly expensive, because they change constantly, and each change is interesting to many people.

If every user's online status were pushed to everyone who has them in a conversation list, a single person going online could trigger hundreds of updates, and with millions of people coming and going, presence traffic would dwarf actual messages.

So presence is designed as cheap and lossy, very differently from messages:

• Only send it to people who are looking. A device subscribes to the presence of the few people currently visible on its screen, and unsubscribes when they scroll away.
• It lives only in memory, with expiry: "Meera last active 2 minutes ago" is computed from a timestamp in Redis that her gateway renews. Nothing is stored durably. If a gateway crashes, the information just expires.
• It's allowed to be wrong for a few seconds. Updates are batched and delayed slightly, so someone briefly losing signal doesn't flicker offline and online.
• Typing indicators are sent through the gateways only, never stored, and expire after a few seconds if not refreshed.

The lesson generalizes: different data in the same feature can need completely different guarantees (lesson 5.3). Messages must never be lost or reordered; presence can be dropped freely.

Step 9 of 16 · 1 min

Group chats

A message to a group of 256 must reach up to 256 people's devices. That's lesson 4.4's fan-out problem, and the same trade-off applies:

• For normal groups (up to a few hundred), the message is stored once, in the conversation, and delivery fans out: the chat service looks up each member's connected devices and sends to their gateways. Members who are offline get a push notification, or simply catch up when they next connect.
• For very large broadcast channels (tens of thousands of members), pushing every message to every member is lesson 4.4's celebrity problem. Those channels switch to a pull model: members' apps fetch new messages when the channel is opened, and only a light "something new" signal is pushed.

Because storage is per conversation, not per recipient, a group message is still one write. That's a big difference from feeds, where each follower had their own copy.

Step 10 of 16 · 1 min

A word on end-to-end encryption

Many chat apps (WhatsApp and Signal, using the Signal Protocol) encrypt messages so that only the participants' devices can read them, not even the company's servers. That changes the design substantially: the server stores and routes encrypted blobs it can't read, each of a user's devices has its own keys, group messages are encrypted for each member's devices, and features like server-side search or spam detection on message content stop being possible. Chirp chose not to do this in the first version. It's worth knowing that it's a major architectural decision, not a setting, and that interviewers may ask about it.

Step 11 of 16 · 1 min

From the real world: two million connections on one box

"Millions of open connections" sounds like it needs thousands of servers. In January 2012, WhatsApp wrote a short post titled "1 million is so 2011": it had passed 2 million simultaneous connections on a single server, running Erlang on FreeBSD.

The post is mostly about tuning, and it shows where the limits of a chat gateway really are:

• Memory per connection is the budget. With around 100 GB of usable memory, two million connections leaves a few tens of kilobytes each, for socket buffers, session state and the process handling it. A design with a thread per connection (each with its own stack) can't get near that.
• The runtime matters. Erlang processes are tiny (a few hundred bytes to start) and cheap to switch between, so "one process per connection" works at that scale. Go's goroutines and async tasks in Rust or Java's virtual threads aim at the same thing.
• The operating system has limits of its own: open file descriptors, socket memory, ephemeral ports, connection-tracking tables. Each one is a number someone has to raise and then test.

For Chirp's gateways, the useful outputs of this are a napkin number (connections per gateway at your memory budget, at 60–70% to leave headroom, as lesson 1.3 says) and a list of OS and runtime settings to check before launch. The other lesson is the one from lesson 2.1: scaling up is underrated. A small team ran a global messaging service on remarkably few, very well-tuned machines.

Step 12 of 16 · 1 min

Going deeper: fan-out by node, not by member

Group chats multiply work: one message to a group of 1,000 means 1,000 deliveries. The obvious way to deliver is a loop: for each member, look up their gateway, send the message there. On a cluster with a few dozen gateway nodes, that means a thousand network sends from one process, mostly to the same few nodes, for every message.

The better shape is to group recipients by the node that holds their connection, send one message to each node with its list of local recipients, and let each node fan out to its own connections locally, in parallel. Network sends drop from the number of members to the number of nodes. The sending process stops being a bottleneck, and each node's local fan-out is just memory operations.

Discord described running into this as its servers (communities) grew to hundreds of thousands of members. In Elixir, a guild's process sending to every connected member's session process across the cluster became too slow for busy guilds. Discord wrote and open-sourced a small library, Manifold, that does exactly this grouping: it splits the recipient list by node, sends one message per node, and fans out there.

The same idea appears all over distributed systems: batch by destination before crossing the network. Kafka producers batch by partition leader; databases group writes by shard; CDNs collapse many requests for one object into one fetch to the origin. Whenever a loop sends many small messages across a network, ask whether they can be grouped by where they're going.

Step 13 of 16 · 1 min

What Chirp built

• WebSockets, with long-polling fallback. Gateways hold connections and do as little as possible; chat services behind them are stateless.
• A connection registry in Redis mapping devices to gateways, with expiring entries and heartbeats.
• Gradual gateway restarts and jittered client reconnects, with cheap session resumption.
• Per-conversation sequence numbers for ordering, client message IDs for de-duplication.
• Messages in a wide-column store, partitioned by conversation and month.
• Receipts as per-person "received up to" and "read up to" positions.
• Device sync by asking for everything after each device's last sequence number, including the user's own messages from other devices.
• Presence and typing in memory, only for people on screen, allowed to be briefly stale.
• Groups stored once and delivered by fan-out; very large channels use pull.

Messages launched to everyone in March. Median delivery time, from Priya's phone to Meera's, was 120 ms within India.

Step 14 of 16 · 1 min

If you're here for an interview

"Design WhatsApp" or "design a chat system" is a classic. A strong answer covers:

• Connections: WebSockets, gateways as the only stateful tier, a registry of who's connected where, and heartbeats.
• Routing: online recipients through their gateway; offline recipients through storage plus push notifications.
• Ordering and de-duplication: per-conversation sequence numbers (not timestamps), and client message IDs for retries.
• Storage: partitioned by conversation, sorted by sequence, with a napkin estimate.
• Receipts and sync: per-person read positions, and catching up by sequence number per device.
• Presence: cheap, lossy, and only for people who are looking.
• Groups: store once, fan out delivery, and switch to pull for huge channels.

Step 15 of 16 · 2 min

Try these yourself

Step 16 of 16 · 1 min

Further reading

• RFC 6455, The WebSocket Protocol.
• Discord Engineering, "How Discord Stores Billions of Messages" (2017) and "How Discord Stores Trillions of Messages" (2023).
• Martin Kleppmann, Designing Data-Intensive Applications, Chapter 5 (ordering and consistency) and Chapter 11 (messaging systems).
• The Signal Protocol documentation (signal.org/docs), for how end-to-end encrypted messaging works.
• Rick Reed, "1 million is so 2011," WhatsApp blog, January 2012, and his talk "That's 'Billion' with a 'B': Scaling to the Next Level at WhatsApp" (Erlang Factory 2014).
• Stanislav Vishnevskiy, "How Discord Scaled Elixir to 5,000,000 Concurrent Users," Discord blog, 2017; the discord/manifold repository.