Step 2 of 17 · 1 min

The query that worked fine for a year

The first version of the Chirp feed was one SQL query:

SELECT p.*
FROM posts p
JOIN follows f ON f.followee_id = p.author_id
WHERE f.follower_id = :me
ORDER BY p.created_at DESC
LIMIT 20;

"Give me the newest 20 posts from people I follow." At a few thousand users, it ran in a few milliseconds and nobody thought about it again.

It's worth seeing why this can't last, because the reason shapes everything after it. To answer that query, the database has to find recent posts from every account you follow, merge them, and sort them. If you follow 500 people, that's 500 authors' worth of posts, every single time you open the app. And people open the feed constantly; it's by far the most frequent thing they do. Once the posts table gets big enough to split across several databases, those 500 authors live in different places, and a single feed load has to ask all of them.

Somebody has to do the work of putting a feed together. The only real question is when.

Step 3 of 17 · 1 min

Two ways to pay the bill

You can do the work when someone reads their feed, or when someone writes a post.

Building the feed at read time (usually called fan-out on read, or pull) is what the SQL query does. Posting is cheap: you save one row and you're done. Reading is expensive, and it gets more expensive the more accounts you follow.

read_feed(user):
    followees = get_following(user)              # say, 200 accounts
    lists     = fetch recent posts for each followee
    return newest 20 after merging the lists

Building the feed at write time (fan-out on write, or push) flips it around. Every user gets a precomputed feed, basically an inbox of post IDs. When you post, your post's ID gets dropped into the inbox of each of your followers. Opening the feed becomes one quick lookup.

on_post_created(post):
    for follower in get_followers(post.author):
        add post.id to feed:{follower}

read_feed(user):
    return newest 20 ids from feed:{user}

Now reading is cheap and posting is expensive, and it gets more expensive the more followers you have. You also take on two new costs. Posts arrive in feeds a little after they're written, since the copying happens in the background. And you're storing a copy of every post ID in every follower's feed, which adds up to a lot of memory.

Most explanations stop here and tell you to "use a hybrid." That's the right answer, but it's not very satisfying until you've seen the numbers that force it.

Step 4 of 17 · 3 min

Doing the math before it does it to you

Here's Chirp around the time the singer showed up. All of these are assumptions, so feel free to swap in your own and redo the arithmetic.

• 20M registered users, 10M of them active on a given day
• About 14M users active in the last month
• 5M posts per day
• Each active user opens the feed about 10 times a day
• On average, a user has 200 followers
• Peak traffic is roughly 3× the daily average

That "average of 200 followers" needs a second look. Follower counts are extremely lopsided. A typical Chirp user has maybe 40 followers. A handful of accounts have millions, and they drag the average way up. Keep that in mind, because this lopsidedness causes almost all of the trouble later. If every user had exactly 200 followers, pure push would work fine and this article would be much shorter.

What pull costs

Feed reads:  100M per day / 86,400 s  ≈ 1,160 per second   (peak ≈ 3,500)
Lookups:     1,160 × 200 followees    ≈ 232,000 per second (peak ≈ 700,000)

That's a lot of lookups, but the lookup count isn't what makes pull feel bad. Latency is.

Those 200 lookups land on many different cache servers, and the feed can't render until the slowest one answers. Say each cache server is slow (over 50ms) just 1% of the time. Spread a feed load across 50 of them:

chance at least one is slow = 1 − 0.99^50 ≈ 39%

So about four in ten feed loads hit the slow path, even though every individual server is "fast 99% of the time." You can stare at a dashboard full of healthy servers and still have a sluggish app. Google's Jeff Dean and Luiz Barroso wrote a well-known paper about this effect called The Tail at Scale. It's worth reading.

What push costs

5M posts per day × 200 followers = 1B feed inserts per day
                                 ≈ 11,600 per second (peak ≈ 35,000)

About 30% of Chirp's users haven't been active in a month, and there's no point filling their feeds (more on that later). Skip them and you're at roughly 8,000 inserts per second.

So push spends about 8,000 writes per second to avoid about 232,000 reads per second, and the reads are the part a human is sitting there waiting for. For an ordinary user, it's not close. Push wins.

Then the singer posts

Ten million followers, seven million of them active. One post means seven million inserts, all at once.

A single fan-out worker writing to Redis in batches might manage around 20,000 inserts per second (rough figure; yours will differ). If one worker gets the whole job:

7,000,000 / 20,000 ≈ 350 seconds, or about 6 minutes

For those six minutes, every post queued behind hers waits too. Even with the job spread across 20 workers, it's about 17 seconds of the platform's full fan-out capacity. Normal peak load is about 35,000 inserts per second, so this one post is roughly three minutes of the entire platform's peak traffic arriving in a single burst.

That was Friday night. Chirp was using pure push, the singer posted during a live show, and a few other big accounts posted around the same time. The queue backed up, and regular people's posts took over twenty minutes to show up.

And the memory bill

Feeds usually live in Redis sorted sets. Each entry is a post ID, sorted by timestamp. In Redis's normal storage format, each entry costs very roughly 80 bytes once you include Redis's own bookkeeping.

800 entries per feed × 80 bytes     ≈ 64 KB per user
14M active users × 64 KB            ≈ 900 GB
× 2 for a replica                   ≈ 1.8 TB of RAM

That's an expensive amount of memory. There's a nice trick, though. Redis stores small sorted sets in a much more compact format called a listpack. By default, a sorted set stays compact up to 128 entries. Cap feeds at 128 instead of 800 and the per-entry cost drops to something like 20–25 bytes:

128 entries × ~22 bytes  ≈ 3 KB per user
14M users × 3 KB         ≈ 40 GB

That's about a twentieth of the memory. The catch is that 128 entries is only about six pages of feed, so anyone who scrolls further needs a fallback (we'll get to it). In practice very few people scroll that far, so it's usually an easy trade.

Exact sizes depend on your Redis version and how big your IDs are, so check with MEMORY USAGE on real data before you plan hardware around these numbers.

Step 5 of 17 · 3 min

So we split the difference

Put the numbers side by side and the hybrid pretty much designs itself:

• Most users get push. Their follower counts are small, so it's cheap to copy their posts, and it makes everyone's feed fast to read.
• Huge accounts get pull. Their posts aren't copied anywhere. Instead, when you open your feed, the app fetches the latest posts from the few big accounts you follow and mixes them in.

Pulling big accounts sounds like it brings back the slow-read problem, but it mostly doesn't. Millions of people read the same handful of celebrity post lists, so those lists sit in cache constantly, and you can even keep a copy in each app server's own memory for a second or two.

Here's what actually happens when you open your feed:

1. Read your precomputed feed from Redis.
2. Look up which big accounts you follow. Keep this as its own small list and update it when you follow or unfollow someone. Don't work it out from scratch on every read.
3. Fetch each big account's recent posts.
4. Merge everything by time. Each list is already sorted, so this is quick.
5. Remove duplicates. An account that recently crossed the threshold can show up in both places.
6. Filter out anything you shouldn't see: deleted posts, people you've blocked, posts that became private. This step isn't optional, and the edge-case section below explains why.
7. Load the actual post content and author info, in one batch.
8. Ask for about 30 IDs to fill a page of 20, since filtering usually removes a few.

Where's the line?

"Big account" needs an actual number. Three things push on it.

How quickly posts need to show up. If you want posts in feeds within 10 seconds, and a single post can use at most 50,000 inserts per second without starving everyone else:

threshold ≤ 50,000 × 10 = 500,000 followers

How much pulling your readers can handle. Every big account you follow adds one fetch to your feed load. Set the threshold too low and someone who follows 300 "big" accounts is back to slow pull. Watch the worst cases here (how many accounts the heaviest 1% of readers pull), not the average.

How often they post. An account with 200K followers that posts 50 times a day can cause more trouble than one with 2M followers that posts once a week. Some teams score accounts on followers × posting rate instead of followers alone.

Nobody gets this number right on day one. Start somewhere conservative, in the tens or hundreds of thousands, then adjust it using two metrics: how long posts take to reach feeds, and how slow feed loads get.

Accounts that sit right on the line

Chirp ran into this the week after the hybrid launched. One account kept bouncing between 99,998 and 100,003 followers, so it flipped between push and pull a dozen times a day. The fix is to use two lines instead of one: switch to pull at 100K, but only switch back to push below 80K. Thermostats use the same idea, and the term for it is hysteresis.

The switch itself has a small wrinkle. When an account moves from push to pull, its older posts are still sitting in followers' feeds, and its new posts get pulled. The duplicate removal in step 5 takes care of the overlap. Going the other direction, from pull to push, followers' feeds don't have that account's older posts in them. Either copy a few recent posts in when you switch, or keep pulling the account for a short grace period.

Step 6 of 17 · 1 min

The boring plumbing that holds it together

The loop for follower in followers: insert hides a few problems that only show up in production.

Don't lose posts

Saving a post and sending the "new post" event to the fan-out queue are two writes to two systems, the exact problem from lesson 4.2. Without care, a crash between them means a post that never reaches anyone's feed. So the post and its event are written together through the outbox, and the relay publishes it. Every post is sent at least once.

Make retries harmless

"At least once" means some messages will be processed twice (lesson 4.1). A worker crashes halfway, a chunk times out and gets retried. Feeds need to handle that without showing the same post twice.

This is one of the main reasons feeds use Redis sorted sets rather than plain lists. Push the same post ID onto a list twice and it appears twice. Add the same post ID to a sorted set twice and nothing changes; the entry is just already there. Retries become free.

Don't let one account block everyone

If seven million inserts are one job, one worker is stuck on it for minutes and everything behind it waits. That was Chirp's Friday night.

• Break big jobs into chunks, say 5,000 followers each, so many workers can share the load.
• Give big accounts their own queue with their own workers. Then a celebrity storm can't delay your post reaching your 40 friends.
• Check whether the post still exists before each chunk. If it was deleted halfway through, stop.

Keep feeds from growing forever

Every insert comes with a trim, so a feed never grows past its cap:

ZADD feed:{user} <timestamp> <post_id>
ZREMRANGEBYRANK feed:{user} 0 -(CAP+1)    # keep only the newest CAP entries

Send these in big batches (Redis calls it pipelining) instead of one command per network round trip. Batching is where that 20,000-per-second figure comes from. Without it, you'd be lucky to get a tenth of that.

Step 7 of 17 · 2 min

The small things that bite

The overall design fits on a whiteboard. Most of the actual engineering time goes into the cases below.

People who stopped using the app. Filling the feed of someone who hasn't opened Chirp in a year wastes writes and memory. During fan-out, skip anyone inactive for more than 30 days and let their feed expire. If they come back, their feed is missing, so build it once using pull and then go back to pushing. Their first load is a bit slower, and nobody notices. With Chirp's numbers, this alone cut fan-out work by about 30%.

Deleting a post. Its ID might be sitting in millions of feeds. You could send a "remove this" to every one of them, but that's as expensive as the original post and just as slow, and people expect a deleted post to disappear right now. Instead, rely on step 6 of the read path: when the app loads post details and finds the post deleted, it drops it. You can clean up the leftover IDs in the background whenever convenient. The read-time filter is what guarantees the post is gone. This is also why the read path asks for 30 IDs to fill a page of 20.

Editing a post. This one costs nothing, because feeds store post IDs, not the posts themselves. The text gets loaded fresh every time. If Chirp had copied post text into feeds, every typo fix would mean rewriting millions of entries. (If you played Stage 2, you hit this exact problem with display names copied into posts. It's the same mistake in a different place.)

Unfollowing. That person's posts are still in your feed. Usually you either filter them out at read time by checking whether you still follow the author, or clean them up in the background. Seeing an unfollowed account's post for a few seconds is mildly annoying at worst.

Blocking someone, or a post going private. Here the rule is different. Showing someone a post from a person they blocked is a real failure, and possibly a safety issue. You can't let this catch up "eventually." Check blocks and privacy on every single read, against the current data.

Following someone new. You follow an account and nothing from them appears until they post again, which feels broken. Copy their last few posts into your feed in the background, and until that's done, the read path can fetch their posts directly, exactly the way it does for big accounts.

Brand-new users. They follow nobody, so their feed is empty, which is a terrible first impression. You need something to show them: popular posts, suggested accounts, topics. This is as much a product decision as a technical one, but the feed system has to be able to mix in posts from people you don't follow.

Things happening at the same time. Someone follows an account in the middle of that account's fan-out. They might get the post, they might not. That's fine; the backfill covers it. A post is deleted while it's being fanned out. The chunk check stops it, and the read-time filter catches anything already delivered.

Underneath all of these is one principle: the write path tries its best, and the read path makes sure the result is correct.

Step 8 of 17 · 1 min

Ordering, and a bug that took two days to find

IDs that sort by time

Chirp's post IDs are Snowflake IDs (lesson 4.3), which sort by creation time to the millisecond, so the ID alone tells you which post is newer. One side effect matters for feeds: IDs from servers whose clocks differ by a few milliseconds sort slightly out of creation order. For a feed, a post landing a few milliseconds out of place is invisible.

The bug

When feeds moved to Redis, Chirp's team used the post ID as the sorting score. It seemed obvious: the ID already sorts by time.

A few days later, people reported posts appearing in slightly the wrong order. Not often, and never by much. It took two days to find the cause.

Redis stores sorted-set scores as floating-point numbers, which can only represent whole numbers exactly up to about 9 quadrillion (2⁵³). Snowflake IDs are bigger than that: the same limit that broke Chirp's website in lesson 4.3. So two IDs created a few milliseconds apart could get rounded to the same score, and Redis would order them however it liked.

The fix is small. Use the timestamp in milliseconds as the score, which fits comfortably, and the post ID as the entry itself. When two posts have the same score, Redis sorts them by the entry, which gives a stable tiebreak.

Paging through the feed

Lesson 1.5's rule applies: page with a cursor, not "skip 20." The app holds the timestamp and ID of the last post it showed and asks for posts older than that, so new posts at the top never shift later pages.

Step 9 of 17 · 1 min

When things go wrong

One lesson from that Friday: watch how old the oldest waiting message is, not how many messages are waiting. A million small chunks can clear in seconds, while ten celebrity jobs can hold things up for minutes. The age of the oldest message tells you how late posts are, which is the thing users feel. Chirp's alert was on queue length, and it didn't fire until people were already complaining.

Step 10 of 17 · 1 min

A note on ranked feeds

Everything above assumes the feed shows the newest posts first. Most big apps now rank the feed by what they think you'll care about, and that pushes the design toward pull.

Ranking depends on you: your interests, what you've already seen, maybe the time of day. That's hard to work out in advance when a post is written. So the precomputed feed becomes a pool of candidates. When you open the app, the system gathers candidates (your pushed feed, big accounts you follow, recommendations), scores them, and shows the best few.

The question then becomes which candidates are worth preparing ahead of time, and how much scoring you can do before the user gets impatient. Ranking is big enough to deserve its own article.

Step 11 of 17 · 1 min

From the real world: five seconds, or five minutes

The "how other teams did it" section mentioned Twitter's precomputed timelines. The numbers from Raffi Krikorian's talk about them, around 2012, show why the hybrid was necessary rather than clever.

Twitter was serving about 300,000 timeline reads a second, against several thousand tweets a second. Reads dominated by about two orders of magnitude, so it made sense to do the work at write time: when someone tweets, insert the tweet's ID into each follower's timeline, stored in Redis (and replicated, three copies of each timeline). Reading a timeline was then a single fetch.

The cost landed on accounts with huge followings. A tweet from someone with 31 million followers meant 31 million timeline inserts, each on three replicas. Twitter's goal was to deliver a tweet to followers' timelines within about five seconds. For the largest accounts, Krikorian said it could take up to around five minutes. And while one celebrity's fan-out was running, everyone else's tweets competed with it for the same workers.

The answer was the hybrid from this lesson: for accounts above a threshold, don't fan out at all. When a timeline is read, merge in recent tweets from the few large accounts that person follows. Fan-out work for the biggest accounts disappears, and reads get slightly more work, which is cheap because there are only a handful of such accounts per reader.

Step 12 of 17 · 1 min

Going deeper: why the threshold works

The hybrid works so well because of how followers are distributed. It isn't a bell curve. It's a power law: almost everyone has few followers, a small number have many, and a tiny number have a huge number. That shape shows up everywhere in social systems (followers, page views, sales, city sizes).

The consequence is that a tiny number of accounts generate a large share of all fan-out writes. Here's a simple model: 50 million accounts, with follower counts following a Zipf-like distribution so that the biggest account has 30 million followers.

In that model, about 500 accounts have more than 100,000 followers, roughly one in 100,000 accounts. Yet those accounts cause about a third of all fan-out writes. Lower the threshold to 10,000 followers and you're treating about 7,000 accounts as celebrities, covering about 40% of writes.

How to pick Chirp's threshold, in practice:

• Measure your own distribution. Plot "share of fan-out writes from accounts above X followers" for real data. The exact numbers will differ from the model, but the shape won't.
• Bound the read side. Each celebrity a user follows adds a small merge at read time. Pick the threshold so that a typical reader follows only a few, and cap how many are merged (for example, the 50 most-engaged celebrities they follow).
• Remember it's not just size. An account with 50,000 followers that posts 200 times a day costs more fan-out than one with 500,000 that posts weekly. The real measure is followers × posts per day.
• Make the threshold soft. Accounts near the line can flip back and forth; give them a band (switch to pull above 120,000, back to push below 80,000), so their followers' feeds don't keep changing behaviour.

Step 13 of 17 · 1 min

If you're designing one from scratch

Some rough rules of thumb:

• Under about 100K users: just use the SQL query. Seriously. You'll have bigger problems before this one.
• Read-heavy, and nobody has a huge following yet: push.
• Any real social network, where a few accounts have enormous followings: the hybrid.
• A heavily ranked, personalized feed: mostly pull, with some precomputed candidates.
• Posts must appear instantly: pull that content, or set the threshold low.
• Short on memory: push with small capped feeds, plus pull for people who scroll a long way.

Step 14 of 17 · 1 min

How other teams did it

Twitter, around 2012. In a 2012 conference talk, Raffi Krikorian described Twitter's home timelines as precomputed lists kept in Redis, filled at write time and capped in length, with special handling for accounts with very large followings. Martin Kleppmann uses Twitter's published numbers from that period as the opening example in Designing Data-Intensive Applications, and it's one of the clearest write-ups of this whole trade-off.

Facebook. Facebook has described its News Feed backend (called Multifeed) as gathering and ranking posts mostly at read time. That fits a feed that's heavily ranked rather than chronological.

Pinterest. Pinterest has written about building a "smart feed" that mixes precomputed content with ranking.

Snowflake. Twitter announced Snowflake on its engineering blog in 2010, as a way to create time-ordered IDs without the servers having to coordinate.

Step 15 of 17 · 1 min

If you're here for an interview

Interviewers love this problem because every answer leads to another question. Here's how it usually goes.

"Push or pull?" Don't pick one. Say hybrid, and explain why: follower counts are lopsided, push is cheap for normal users, and big accounts are the exception.

"A celebrity with 50 million followers posts. What happens?" Their posts are pulled at read time. If the interviewer insists on push, talk about splitting the work into chunks, a separate queue, and a target for how quickly posts must arrive.

"Now they delete it." Filter at read time, so it disappears immediately. Clean up the copies in the background if you want.

"How much memory do the feeds need?" Give an actual number: entries per feed × bytes per entry × active users × replicas. Then mention capping feeds and skipping inactive users. Most candidates don't give a number at all.

"How does pagination work?" Use a cursor, not skip-and-count, and say why: new posts would shift everything down.

"What if the feed cache dies?" Rebuild with pull, throttled so you don't overload the database, and serve pull-only feeds until it recovers.

"How would you know it's working?" Name the age of the oldest queued message, slow-end feed load time, how many accounts each read has to pull, and the cache hit rate.

Mistakes that come up a lot: storing post content in feeds instead of IDs, forgetting about inactive users, using plain lists and then getting duplicates on retries, treating a block like an unfollow, and saying "Twitter uses push" without mentioning the celebrity exception.

Step 16 of 17 · 1 min

Try these yourself

Step 17 of 17 · 1 min

Further reading

• Martin Kleppmann, Designing Data-Intensive Applications, Chapter 1. The Twitter timeline example.
• Raffi Krikorian, "Timelines at Scale," QCon San Francisco 2012 (on InfoQ).
• Twitter Engineering, "Announcing Snowflake" (2010).
• Jeffrey Dean and Luiz André Barroso, "The Tail at Scale," Communications of the ACM, 2013.
• Chris Richardson, "Transactional outbox" pattern, on microservices.io.
• Redis docs on sorted sets and memory optimization.
• Raffi Krikorian, "Timelines at Scale," QCon San Francisco 2012 (video and slides on InfoQ).
• Lada Adamic and Bernardo Huberman, "Zipf's law and the Internet," Glottometrics 3, 2002, a short introduction to power laws in online systems.