Step 2 of 15 · 1 min

Do the work later

Ask, for each step: does the user need this to be finished before we tell them "done"?

• Saving the like: yes. If it isn't saved, we shouldn't say it was.
• The like count: ideally in the same transaction (lesson 3.3).
• The notification, the push, analytics, search: no. The author doesn't need their phone to buzz within the same 50 ms. A few seconds later is fine.

So the request does the essential part, records that the rest needs doing, and returns. Something else does the rest in the background.

The "something" is usually a message queue: a system that holds messages (also called jobs or tasks) until a worker takes them and processes them. The code that adds messages is the producer; the workers are consumers.

Moving work into a queue gets you more than speed:

• Isolation. When Apple's push service is slow, push jobs back up in the queue. Likes stay fast.
• Absorbing bursts. At the end of a cricket match, likes spike tenfold for a minute. The queue holds the extra while workers work through it at their own pace, instead of the whole app slowing down.
• Retries. A failed push can be tried again in a minute, without the user ever knowing.
• Independent scaling. Need more push capacity? Add push workers. The web servers don't change.

Step 3 of 15 · 1 min

A queue that loses jobs

Chirp's first queue was a Redis list. The web server pushed a job onto the list:

LPUSH queue:push '{"user": 4821, "text": "Priya liked your post"}'

and each worker sat in a loop, taking jobs off the other end:

BRPOP queue:push 0      # wait for a job, remove it, return it

It worked well for three weeks. Then the team noticed, comparing counts, that a small but steady number of push notifications were never sent.

The problem was in the word "remove." BRPOP takes the job off the list at the moment it hands it to the worker. If the worker then crashes, gets restarted by a deploy, or is killed by autoscaling (lesson 2.1) before it finishes, the job is gone. It isn't in the list, and it was never completed.

Step 4 of 15 · 1 min

Acknowledgements

Every serious queue fixes this the same way: a job isn't deleted when it's handed to a worker. It's deleted when the worker says it's finished. That message from the worker is an acknowledgement, or ack.

Until the ack arrives, the job is "in progress." If no ack arrives in time, because the worker crashed or hung, the queue gives the job to another worker.

Different systems implement this differently:

• Amazon SQS hides a message from other consumers for a visibility timeout (say, 30 seconds) after it's received. If the consumer deletes it within that time, it's done. If not, it becomes visible again and someone else gets it.
• RabbitMQ keeps the message assigned to a consumer until that consumer acks it. If the consumer's connection drops, unacked messages go back on the queue.
• Redis Streams track a list of pending messages per consumer, until they're acknowledged with XACK. Other consumers can claim messages that have been pending too long.

With plain Redis lists, the old fix was to atomically move the job to a separate "processing" list (BLMOVE), and remove it from there when done, with a cleanup process to rescue jobs stuck in "processing." It works, but you're building a small queue system yourself.

The catch: jobs run twice

Acknowledgements solve lost jobs, and create a new situation: a job can be processed more than once.

Imagine a push worker sends the notification, and then crashes before sending its ack. The queue doesn't know the push was sent. All it knows is that no ack arrived. So it gives the job to another worker, which sends the push again. The author's phone buzzes twice.

It's not a bug in the queue. It's unavoidable. The worker can't send the push and the ack at the very same instant, so there's always a moment where one has happened and the other hasn't. This is the same reason lesson 1.5 needed idempotency keys for phones on trains.

Step 5 of 15 · 2 min

At most, at least, exactly once

Queues describe their guarantees with three phrases:

• At most once. Each message is delivered once or not at all. Ack before processing. If the worker crashes, the message is lost. Fine for things you don't mind losing, like a metric sample.
• At least once. Each message is delivered one or more times. Ack after processing. Nothing is lost, but duplicates happen. This is what almost every real system uses.
• Exactly once. Each message has its effect exactly one time. Everyone wants this.

You'll see systems advertise "exactly once." Read the details carefully. Kafka, for example, can guarantee exactly-once processing within Kafka: reading from one topic and writing to another, as one transaction. But as soon as the work affects something outside (sending a push, charging a card, calling another company's API), no queue can make it happen exactly once, for the reason above.

What you can do is make processing a message twice have the same effect as processing it once. That's an idempotent consumer, and it's the real-world meaning of "exactly once":

• Natural idempotency. Some operations already are: "set this user's email preference to off," "add post 9001 to feed X" with a sorted set (lesson 4.4). Doing them twice changes nothing.
• Record what you've done. Give each message a unique ID. Before processing, check whether that ID has been processed; after processing, record it, ideally in the same database transaction as the work itself. Lesson 4.2 shows exactly how.
• Pass the key along. When calling an outside service, send an idempotency key (lesson 1.5) derived from the message ID, so their side can recognize the retry.

Push notifications are a hard case, because Apple's and Google's push services don't deduplicate for you. Chirp recorded each sent push's message ID in Redis for a day, and skipped any it had already sent. A tiny window for duplicates remained (a crash between sending and recording). For a push notification, that was acceptable. For a payment, it wouldn't be, and Part 7 covers that.

Step 6 of 15 · 1 min

When jobs fail

Retries, with patience

When a job fails because of something temporary (a timeout, a 503 from Apple), try again. But not immediately, and not forever. Lesson 1.5's exponential backoff with jitter applies here too: retry after a few seconds, then longer, with some randomness so a thousand failed jobs don't all retry at the same instant and hammer a struggling service.

Most queue systems can delay a retry for you. With SQS, for instance, you change the message's visibility timeout so it reappears later.

Poison messages

Some jobs will never succeed: a push for a user whose account was deleted, a malformed message from a bug, a notification about a post that no longer exists. Retried forever, they waste worker time. With some queues, they can also block everything behind them.

Chirp met one of these in its second week. A job with a bad payload made the push worker crash on start-up. The queue redelivered it, the next worker crashed, and so on. Within a minute, every push worker was stuck in a crash loop on the same message, and no pushes were sent for twenty minutes.

The fix is a dead-letter queue: after a job has failed a certain number of times (say, five), it's moved to a separate queue, out of the way. Someone can look at it later, fix the cause, and send it back. SQS calls this a redrive policy; most queue systems have the same idea.

Two habits go with it: alert when the dead-letter queue isn't empty, because a growing one means a bug, and catch errors per job in workers, so one bad message fails by itself rather than crashing the whole worker process.

Step 7 of 15 · 1 min

Watching a queue

The number to watch isn't how many messages are in the queue. Lesson 4.4 makes this point too: watch the age of the oldest message. A million tiny jobs might clear in seconds, while a hundred stuck ones mean something is broken.

Alongside it:

• Arrival rate and processing rate. If more arrives than workers can process, the queue grows until something gives. That's when you add workers.
• Failure and retry rate, and the dead-letter queue size.
• Processing time per job, to spot a slow dependency (like Apple on that Saturday) before the queue backs up.

Queues also let you choose what to do when you can't keep up. You can add workers. You can drop the least important work: Chirp's analytics events have a short expiry, and old ones are simply discarded under heavy load. Or you can push back on producers, so the system slows down instead of piling up an ever-growing backlog. That last idea is called backpressure, and lesson 6.1 returns to it.

Step 8 of 15 · 3 min

Queues and logs

There are two quite different kinds of system that people call "message queues," and the difference matters.

A queue

RabbitMQ, SQS, and job systems like Sidekiq or Celery work as described so far: a message is handed to one worker, and deleted once it's acknowledged. Think of it as a to-do list. Several workers share the list, each taking the next item, and done items disappear.

This is ideal for jobs: send this push, resize this photo, generate this report.

A log

Kafka, Amazon Kinesis, Redpanda and Redis Streams work differently. A log is an append-only list of messages that isn't deleted when read. Messages stay for a configured time (days or weeks), or forever. Each message gets a sequential position number, called an offset.

Readers don't take messages. Each consumer group just remembers the offset it has reached, and reads forward from there. So:

• Many different systems can read the same messages independently. The push service, the search indexer and the analytics pipeline each have their own consumer group, each at its own position, each reading every message. A queue would need a separate copy of each message for each of them.
• You can replay. Deploy a new search indexer with a bug fix, and point it at an offset from three days ago. It reprocesses everything since then. With a queue, those messages would be gone.
• Order is kept, within a partition (next section).

This is why the change stream in lesson 3.4 usually goes into a log, not a queue. Lots of systems want to hear about every change to a post, now and in the future, and replaying from the start is how you rebuild a search index.

Partitions

A single log on a single machine can only go so fast. So a log topic is split into partitions, each its own ordered log, spread across machines. When a message is written, a key chooses its partition: messages with the same key always go to the same partition.

That gives you the most important property of logs: messages with the same key are kept in order. If Chirp uses the post ID as the key, every change to post 9001 lands in the same partition, in the order it happened, and the indexer sees the edits in the right order (the out-of-order problem from lesson 3.4).

Within a consumer group, each partition is read by exactly one consumer at a time. So partitions are also the unit of parallelism: a topic with 12 partitions can have up to 12 consumers in a group working in parallel. Choose partition counts with growth in mind, because changing them later reshuffles which keys go where.

One warning, which lesson 4.5 explores: if one key is far busier than the others (one viral post, one celebrity), its partition gets far more traffic than the rest, and its single consumer falls behind while the others sit idle.

Which to use

Many companies run both: a log as the record of what happened ("post created," "like added"), and queues for jobs to do ("send this push"), often with a small service reading the log and creating jobs.

Chirp ended up with Kafka for events (post, like and follow changes, with search, analytics and notifications all consuming them), and SQS for jobs (push notifications, photo resizing, emails).

Step 9 of 15 · 2 min

From the real world: the threads that brought down Kinesis

Amazon Kinesis is AWS's managed log, much like the Kafka-style logs in this lesson. On 25 November 2020 it failed in its biggest region for most of a day, and took a surprising number of other AWS services with it.

Kinesis has a fleet of front-end servers that receive requests and route them to the right back-end shard. To route, each front-end server keeps a map of where every shard lives. Part of building that map involved messages exchanged with every other front-end server, and each front-end server ran an operating-system thread for each of its peers.

Early that morning, AWS added a small amount of capacity to the front-end fleet. Every existing server started additional threads for the new peers, and every server went over the operating system's configured thread limit. Building the shard map failed, and servers couldn't route requests.

The fix sounds simple: remove the new capacity and restart. But the fleet had thousands of servers, and each one had to rebuild its map while also serving traffic, so they could only be brought back a few hundred per hour. Errors began falling around noon, and full recovery took until late in the evening.

Meanwhile, services that used Kinesis internally failed too: Cognito sign-ins, CloudWatch metrics and logs (so customers' own alarms went quiet), EventBridge, and through those, parts of Lambda, ECS and EKS. AWS's status page was slow to update, because the tool used to post to it depended on Cognito.

Lessons for Chirp's queues and logs:

• All-to-all coordination has a size limit, even when nobody wrote it down. It's the κ term from lesson 2.1. AWS's fixes were to use bigger servers (fewer peers), alarm on threads, and split the fleet into cells that never need to know about each other.
• A queue is a dependency of everything that publishes to it. If publishing blocks when the log is down, the log's outage becomes your outage. Publish asynchronously with a bounded buffer, and decide what to drop when it's full.
• Know how long a cold restart takes at your current size, before you need one.

Step 10 of 15 · 1 min

Going deeper: rebalance storms

Within a consumer group, each partition is read by exactly one consumer. When a consumer joins or leaves, the group rebalances: partitions are reassigned. That sounds like a rare, small event. Under load, it can be neither.

Kafka decides a consumer has left if it doesn't heartbeat in time, or if it doesn't come back to poll for more messages within max.poll.interval.ms (five minutes by default). A consumer that's processing a slow batch, or stuck in a long garbage-collection pause, misses that window. The group rebalances without it. When it recovers and tries to poll, it rejoins, which triggers another rebalance.

With Kafka's original eager rebalance protocol, every rebalance makes every consumer stop and give up all its partitions before any are reassigned. So one slow consumer pauses the whole group, twice. If the cause is load-related (big batches that are slow to process), the pauses add to the backlog, which makes batches bigger and slower, which causes more rebalances: a rebalance storm, where the group spends more time rebalancing than consuming.

The fixes, in order:

• Process batches that fit comfortably within the poll interval. Lower max.poll.records, or hand slow work to a separate thread pool and keep polling.
• Use cooperative (incremental) rebalancing, added in Kafka 2.4. Only the partitions that need a new owner move; everyone else keeps working.
• Use static membership (group.instance.id) so a consumer that restarts quickly, for a deploy, reclaims its own partitions without triggering a rebalance at all.
• Alert on rebalance rate, not just lag. A group rebalancing every few minutes is sick even if lag looks fine for now.

Step 11 of 15 · 1 min

What Chirp changed

• The like endpoint now does two things: inserts the like and updates the count, in one transaction. Then it returns. Typical time: 6 ms.
• Notifications, pushes, analytics and search updates happen in the background, driven by events.
• Every queue uses acknowledgements; nothing is removed before it's done.
• Every consumer is idempotent: message IDs are recorded, and push sends are deduplicated for a day.
• Retries use backoff with jitter. After five failures, jobs go to a dead-letter queue, which pages someone if it grows.
• Dashboards show the age of the oldest message in every queue and consumer group.

The next time Apple's push service had a slow afternoon, push notifications arrived a minute or two late. Nobody's like button noticed.

One problem was left over, though: the like transaction commits, and then the event about it is sent. What if the server crashes between those two things? That's lesson 4.2.

Step 12 of 15 · 1 min

In the simulation

Stage 3 lets you move notification, push and fan-out work into a queue, and choose the delivery guarantee. A slow push provider appears during the cricket final. Without a queue, it slows down likes for everyone. With a queue but without idempotent consumers, watch the duplicate-notification complaints rise when workers restart.

Open the simulation →

Step 13 of 15 · 1 min

If you're here for an interview

Queues appear in nearly every design: "and then a worker sends the email." Strong answers say a little more:

• What goes in the request, and what goes to the queue, and why.
• At-least-once delivery and idempotent consumers, without being asked. "Messages can be delivered twice, so the consumer records message IDs" is a sentence interviewers like to hear.
• Retries with backoff, and a dead-letter queue for messages that keep failing.
• Queue or log, with a reason: "Kafka, because search and analytics both need every event, and we may want to replay."
• The partition key, if order matters, and what happens with a hot key.

Step 14 of 15 · 2 min

Try these yourself

Step 15 of 15 · 1 min

Further reading

• Martin Kleppmann, Designing Data-Intensive Applications, Chapter 11, "Stream Processing," on message brokers and logs.
• Jay Kreps, "The Log: What every software engineer should know about real-time data's unifying abstraction," LinkedIn Engineering blog, 2013.
• Amazon SQS documentation on visibility timeouts and dead-letter queues.
• The Kafka documentation's design section, on partitions, offsets and consumer groups.
• Amazon Web Services, "Summary of the Amazon Kinesis Event in the Northern Virginia (US-EAST-1) Region," November 2020.
• Apache Kafka KIP-429, "Kafka Consumer Incremental Rebalance Protocol," and KIP-345, "Static Membership."