This gives away every stage. For a clean attempt, practise it first, then come back.
The short version
- Never send a notification inside the request that caused it. Put it on a queue and let workers send it.
- Urgent messages like login codes need their own queue, or they wait behind millions of likes.
- Retries cause duplicates. A dedupe key built from the event stops the same notification going out twice.
- A celebrity post becomes millions of sends. Split it into small batches across many workers.
- Group similar ones (“Priya and 23 others…”) and respect quiet hours, checked at the moment you send.
The 45 minutes
Before we start
Think about your phone right now. A friend liked your photo, a group chat is buzzing, your bank sent a one-time password, and an app you haven’t opened in months wants you back. Each of those is a notification, and each one came from a system like the one we’re about to design.
It looks simple: something happens, send a message. The interview gets interesting when you ask how many, how fast and what if it fails. A celebrity posts and ten million people need to hear about it. Your login code must arrive in seconds. Apple’s push service has a bad minute. A retry sends you the same alert twice.
We’ll go in the usual order: requirements, quick maths, the data and API, then a design that starts simple and grows, and finally the deep dives.
Minutes 0–5Requirements
Interviewer: Design the notification system for Chirp. When something happens, like a like, a comment, a new follower or a login code, the right people should hear about it.
You: Which channels do we need: push notifications, email, SMS, the in-app inbox?
Interviewer: All four. Push and the inbox are the main ones. Email for weekly digests, SMS only for login codes.
You: Roughly how many notifications a day, and what does the busiest moment look like?
Interviewer: About a billion a day, across 50 million users. When a celebrity with 10 million followers posts, all of them might need a notification.
You: Are some more urgent than others?
Interviewer: Yes. Login codes must arrive within seconds. A like can wait a few minutes.
You: Can users choose what they get?
Interviewer: Yes, per type and per channel, and they can set quiet hours at night.
You: What’s worse: sending twice or missing one?
Interviewer: Both are bad. Missing a login code locks someone out. Getting the same “someone liked your post” three times is just annoying.
You: Should we group similar ones, like lots of likes on one post?
Interviewer: Yes. “Priya and 23 others liked your post” is much better than 24 alerts.
You: Great. Let me write down what we’re building.
Functional requirements
- Send notifications by push, email, SMS and the in-app inbox when events happen.
- Respect each user’s preferences: which types, which channels, and quiet hours.
- Keep an in-app inbox for each user, with read and unread.
- Group similar notifications, like many likes on one post.
Out of scope today: tools for the marketing team to build campaigns, and testing different message wording.
Non-functional requirements
- Urgent ones are fast: login codes and security alerts reach the provider within a few seconds.
- Never lost: once an event happens, its notification is eventually sent, even if something crashes along the way.
- Never twice: the same notification never reaches a user twice.
- Big and bursty: about a billion a day, with bursts of 10 million from a single post.
- One failure stays contained: if the SMS provider is down, push and email keep working.
Minutes 5–10Back-of-the-envelope estimates
You: A billion a day is about 12,000 a second on average. At the busiest hour, maybe four times that, so 50,000 a second.
You: If one worker can send about 500 pushes a second to Apple or Google, that’s around 100 workers at peak.
You: The celebrity post is the interesting part: 10 million notifications at once. Even with 100 workers, that’s over three minutes of sending. Anything stuck behind it waits three minutes too.
You: For the inbox: 50 million users, keeping 500 items each at about 300 bytes, is roughly 7.5 TB.
What do these numbers tell us? The average is easy. The burst is the real problem, and anything urgent must never wait behind it. Also worth saying: SMS costs real money per message, while push is almost free. So SMS is for login codes only, never for likes.
Minutes 10–15Core entities and the API
Core entities
- Event: something that happened. “User 42 liked post 881.”
- Notification: one message to one user on one channel, built from an event.
- Preference: what a user wants to receive, on which channels, and their quiet hours.
- Device: a phone or browser that can receive push, identified by a token from Apple or Google.
- Inbox item: a notification stored for the in-app inbox, with read or unread.
The API
Most of this API is used by our own services, not by the app directly:
/v1/notificationsInternal: another service asks for a notification to be sent. The key stops the same event sending twice.
Idempotency-Key: like:post_881:user_42
{ "user_id": 9001, "type": "like", "priority": "normal", "data": { "actor": 42, "post": 881 } }- 202Accepted. It will be sent in the background.
/v1/inbox?cursor=…A user’s in-app inbox, newest first.
- 200
{ "items": [ { "id": "n_77", "text": "Priya and 23 others liked your post", "read": false } ], "next": "…" }
/v1/preferencesTurn types and channels on or off, and set quiet hours.
{ "like": { "push": true, "email": false }, "quiet_hours": "23:00-07:00" }- 200Saved.
/v1/devicesRegister a phone for push. Apple or Google gives the app this token.
{ "token": "f3a9…", "platform": "ios" }- 201Registered.
Two small details matter. The send API answers 202 Accepted, not 200, because the message goes out later, in the background. And the idempotency key is built from the event itself, so if another service retries the request, we can tell it’s the same notification. (Lesson 1.5 covers idempotency keys.)
Minutes 15–25The design, built up
We’ll start with the simplest thing that works, then let growth break it.
Stage 1Send it inside the request
You: The simplest version: when someone likes a post, the app server saves the like, then calls Apple’s push service, waits for the answer, and then replies to the user.
It works, and for a small app it’s fine. But two problems show up fast. If Apple is slow, liking a post becomes slow, because the user is waiting for our push call. And if the push call fails, the notification is simply gone. Nobody tries again.
Stage 2A queue and background workers
You: So the app should never send notifications itself. When someone likes a post, the service saves the like and writes a “liked” event to a queue. That’s quick. A notification service reads events, works out who should be told and how, and hands each message to channel workers that talk to Apple, Google, the email provider and so on. If a send fails, the worker retries a bit later.
You: Now a slow provider only slows the workers, never the app. And nothing is lost, because events wait in the queue until they’re handled.
One detail: saving the like and writing the event must happen together. If the server crashes in between, we’d have a like with no notification. The outbox pattern from lesson 4.2 solves exactly this: write the event into the same database transaction as the like, and a relay moves it to the queue.
The difference between doing work inline and queueing it, from lesson 4.1:
Stage 3Bursts and priorities: fan-out, separate queues, no duplicates
You: Now the celebrity posts. One event needs 10 million notifications. Three changes.
You: First, fan-out workers. Instead of one job looping over 10 million followers, split the follower list into batches of 1,000 and queue each batch as its own job. Many workers share the load, and if one crashes, only its batch is repeated.
You: Second, separate queues by priority. Login codes go to a small urgent queue with its own workers. A burst can fill the normal queues for minutes, and the urgent queue never notices.
You: Third, a dedupe check. Because we retry, the same message can occasionally be sent twice. Before sending, a worker records the notification’s key in Redis. If the key was already there, this one was already sent, so skip it.
Here’s what the second change is worth, played out:
We also check preferences here, from a cache, so a user who has turned off likes, or muted this account, never gets a job created at all. That alone removes a large share of a celebrity burst.
Stage 4The inbox, grouping and tracking
You: Last, the in-app inbox. It’s the one channel we store ourselves. I’d use a wide-column store partitioned by user, newest first, and keep about 500 items per user.
You: Grouping happens here too. When the first like on a post arrives, we send it. For the next few minutes, more likes on the same post update that same inbox item and its count, “Priya and 23 others”, instead of creating new ones.
You: And workers report what happened (sent, delivered, failed) to a separate tracking stream, so support can answer “did my notification go out?” without slowing sending down.
Checking the design against the requirements
| Requirement | Where the design handles it |
|---|---|
| Push, email, SMS, in-app | Channel workers per channel; the inbox store for in-app |
| Respect preferences and quiet hours | Preferences read from a cache before any job is created, and again at send time |
| In-app inbox with read and unread | Inbox store, one partition per user |
| Group similar notifications | Group key per post; later likes update the same item |
| Urgent ones are fast | Separate urgent queue with its own workers |
| Never lost | Outbox plus a durable queue; workers retry until a provider accepts |
| Never twice | Dedupe key checked before every send |
| Big and bursty | Fan-out in batches of 1,000 across many workers |
| One failure stays contained | Separate queues and workers per channel; a circuit breaker per provider |
Deep diveNever lost, never twice
Interviewer: You said both losing and duplicating are bad. How do you get both right?
You: They pull against each other. If we retry until something works, some messages will be sent twice. If we never retry, some will be lost. So we retry, and then remove duplicates.
Bad solutionSend once and hope
Call the provider once. If it fails, log an error and move on. Duplicates are impossible, but every failure is a lost notification, and providers do fail.
Good solutionRetry until it works
Keep the message in the queue until a provider accepts it, retrying with growing gaps between attempts. Nothing is lost. But picture this: a worker sends a push, Apple accepts it, and the worker crashes before it can mark the job done. Another worker picks the job up and sends it again. The user gets two.
Great solutionRetry, but check a dedupe key first
Every notification has a key built from its event, like like:post_881:user_42. Before sending, the worker tries to save that key in Redis with a rule that only works if it isn’t there yet (Redis’s SET key NX), and an expiry of a day. If the save fails, someone already sent this one, so skip it.
There’s still a tiny gap: save the key, then crash before sending, and that one notification is lost. Or send first, then crash before saving, and it’s duplicated. You choose per type. A login code would rather arrive twice than not at all; a like would rather be lost than doubled. Saying that out loud shows you know the gap exists. Lesson 4.2 shows the same pattern for consumers:
Deep diveThe celebrity burst
Interviewer: A celebrity with 10 million followers posts. Walk me through it.
Bad solutionOne job loops over every follower
One worker reads 10 million followers and sends to each. It takes hours, uses one machine, and if it crashes at follower 6 million, it starts again from zero, and everyone gets a duplicate.
Good solutionSplit into batches
The fan-out step splits followers into batches of 1,000 and queues each as a separate job. 10,000 small jobs spread across all the workers. A crash only repeats one batch, and the dedupe keys catch those repeats.
Great solutionBatches, plus skip early, plus respect limits
Before creating a batch, drop people who won’t get this notification anyway: those who’ve muted the account, turned off this type, or haven’t opened the app in months. Then pace the sending. Apple and Google slow down senders who push too hard, so each channel’s workers are rate limited (lesson 6.3). And all of this runs on the normal queues, never the urgent one.
Deep diveWhen a provider fails
Interviewer: The SMS provider goes down. What happens?
You: Only SMS should notice. Each channel has its own queue and workers, so stuck SMS messages can’t block push or email.
You: A circuit breaker on each provider stops workers from hammering one that’s down; messages simply wait in the queue until it recovers. For login codes, I’d keep a second SMS provider and switch to it, because a login code that arrives an hour late is useless.
You: And urgent messages get an expiry. A login code that couldn’t be sent within five minutes should be dropped, not delivered late, because by then the user has already asked for a new one.
One more failure that catches people out: phones get uninstalled and their push tokens stop working. When Apple or Google says a token is no longer valid, delete it, or you’ll keep paying to send to devices that don’t exist. The circuit breaker from lesson 6.1:
Deep diveGrouping and quiet hours
Interviewer: How does “Priya and 23 others liked your post” work? And what happens at 2 a.m.?
You: For grouping, every notification gets a group key, like like:post_881. The first like sends straight away. For the next few minutes, more likes with the same key update the existing inbox item and its count instead of making new ones. On phones, Apple and Google let us replace an earlier notification with a newer one using a collapse identifier, so the user sees one alert that updates, not a stack of 24.
You: For quiet hours, check at send time, in the user’s own time zone. If it’s 2 a.m. for them, don’t drop the notification; schedule it for the morning. Login codes ignore quiet hours, of course.
You: Checking preferences at send time, not when the event happened, matters for another reason too. If someone turns notifications off, anything already waiting in the queue stops immediately.
The complete architecture
Here’s everything together: the full design from the last stage, with every part we added along the way. If you only remember one picture from this question, make it this one.
How one request travels through it:
- A service saves a like and writes a ‘liked’ event to the event log in the same step (using the outbox pattern, so the event can’t be lost).
- The notification service reads the event, works out who should hear about it, and checks their preferences from a cache.
- For a big audience (a celebrity post), it hands the job to fan-out workers, which split the follower list into batches of 1,000.
- Before queueing anything, it checks the dedupe store: if this exact notification was already sent, it’s skipped.
- Login codes go to the urgent queue; everything else goes to normal queues per channel. Urgent work never waits behind a burst.
- Channel workers send to Apple, Google, the email provider or the SMS provider, retrying with backoff if they fail. In-app notifications go to the inbox store.
How this answer gets scored
Interviewers don’t tick boxes, but their notes on this question tend to land on the same few moments. Here’s what each one sounds like from the other side of the table.
| Moment | What earns credit | What costs you |
|---|---|---|
| The first five minutes | Asking about urgency, duplicates and the busiest moment | Listing channels and moving straight to boxes |
| Sending | Taking sending out of the request, onto a queue | Calling Apple’s push service while the user waits |
| Urgent messages | A separate queue and workers for login codes | One queue with a “priority” field |
| Duplicates | Retries plus a dedupe key, and naming the remaining gap | “We’ll use exactly-once delivery” |
| The celebrity post | Batches, skipping muted users, pacing per provider | One job looping over 10 million followers |
| Failures | Separate channels, circuit breakers, a backup SMS provider | One provider’s outage stopping everything |
What the left column has in common: every choice points back to a requirement or a number from earlier. If you can say where each box came from, the interviewer will notice.
Stay in the loop
Get new problems and walkthroughs by email
One short email when something new is published: a deep-dive walkthrough, an LLD problem, or a real outage retold. No spam, unsubscribe any time.