Step 2 of 14 · 1 min

Why files need their own place

It's worth being clear about why neither of the obvious places works.

Not on app servers' disks. Lesson 2.1: servers should hold nothing you'd miss. A file on one server's disk is invisible to the others and lost when that server is replaced.

Not in the database. Postgres can store bytes in a column, and for a few small files that's fine. At Chirp's scale it's a bad trade. Photos would make the database enormous: lesson 1.3's estimate was over 10 TB a year, compared to under 100 GB for everything else. Backups would take hours, replicas (lesson 2.5) would have to copy every photo, and the database's precious memory cache would fill with image data instead of the rows that make queries fast. Databases are expensive per gigabyte because they're built for fast, consistent queries. Photos don't need any of that.

Step 3 of 14 · 2 min

How object storage works

Object storage services (Amazon S3, Google Cloud Storage, Azure Blob Storage, and many S3-compatible ones) all follow the same simple model:

• You create buckets, which are just named containers.
• Inside a bucket, you store objects. Each object is a key (a name, like photos/8f3a9c/feed.webp), the bytes, and a little metadata (content type, size, custom tags).
• You talk to it over HTTP: PUT an object to store it, GET it to read it, DELETE to remove it.

That's nearly the whole interface. What makes it valuable is what happens underneath: each object is stored redundantly, across many disks and usually across several data centers. Amazon, for example, says S3 is designed for eleven nines of durability: 99.999999999% per object per year. In practice, a stored photo is far more likely to be deleted by a bug in your own code than lost by the storage service.

Storage is also cheap. Standard storage costs in the region of two cents per gigabyte per month, with cheaper classes for files that are rarely read. Chirp's first year of photos, about 11 TB, would cost around $250 a month to store. Storing isn't the expensive part. Sending is.

What it isn't

Object storage looks like a file system, but it isn't one, and assuming it is causes bugs:

• There are no real folders. photos/8f3a9c/feed.webp is one key that happens to contain slashes. Tools show "folders" by grouping keys that share a prefix. Listing a "folder" means asking for every key with that prefix, page by page, which is slow when there are millions.
• Objects are written whole. You can't append to an object or change bytes in the middle. To change a file, you upload a complete new version.
• There's no rename. "Renaming" means copying the object to a new key and deleting the old one. For a large object, that's a large copy.
• Rate limits are per prefix. Services publish guidance on how many requests per second a single key prefix can handle (for S3, thousands per second). Most apps never get near it, but if every key starts with the same prefix, you can.

On consistency: S3 now guarantees that once a PUT succeeds, any following GET returns the new object. Not every object store gives that guarantee, so check yours before relying on it.

Naming keys

Chirp's first version used the uploaded file name as the key: uploads/IMG_4021.jpg. That's a mistake in several ways. Two users can upload files with the same name, one overwriting the other. File names can contain strange characters. And names can reveal private information.

A better scheme uses an ID the server generates:

photos/{photo_id}/original.jpg
photos/{photo_id}/feed.webp
photos/{photo_id}/thumb.webp

Each object, once written, is never changed. If the user edits a photo, it gets a new photo_id. This rule, immutable objects, seems minor, but it's what makes the caching later in this lesson simple.

Step 4 of 14 · 2 min

Uploading without the app server

The fix for the busy app servers is to stop routing photo bytes through them at all. The phone uploads directly to object storage, using a signed URL, also called a presigned URL.

Here's how it works:

1. The phone asks Chirp's API: "I want to upload a 3.2 MB JPEG."
2. The app server decides on a key (photos/8f3a9c/original.jpg), and creates a URL for exactly that upload, signed with Chirp's storage credentials. The signature covers the key, the method (PUT), the content type, and an expiry time a few minutes away. This takes milliseconds and involves no network call to storage at all; it's just cryptography.
3. The phone uploads the bytes straight to that URL. The storage service checks the signature and accepts the upload. Nobody can reuse the URL for a different key, a different kind of request, or after it expires.
4. When the upload finishes, the phone tells the API "done," or the storage service sends Chirp an event saying a new object arrived.

The app server now spends a few milliseconds per upload instead of several seconds. The storage service, which is built for exactly this, deals with slow connections from millions of phones.

Things the server still has to do

Direct uploads move the bytes, not the responsibility. After an upload arrives, Chirp still has to:

• Check what was actually uploaded. The phone said it was a 3 MB JPEG. Don't trust that. Check the size, and check the first bytes of the file to confirm it's really an image. Set a size limit in the upload policy so nobody can upload 5 GB.
• Remove location data. Photos from phones usually contain EXIF metadata, which can include the exact GPS coordinates where the photo was taken. Publishing that by accident tells strangers where someone lives. Chirp strips it from every photo it publishes.
• Make the sizes it actually serves. A 3 MB original is wasteful to show in a feed on a phone screen. A background worker (Part 4 covers background work properly) creates a 100 KB feed version and a 20 KB thumbnail, in a modern format like WebP or AVIF, which are much smaller than JPEG for the same quality. The post shows a placeholder until they're ready, usually a second or two.

Big files and bad networks

For large files, like video, uploading in one piece is fragile: a dropped connection at 95% means starting over. Object stores support multipart uploads: the file is split into parts (for S3, at least 5 MB each, except the last), each part is uploaded separately and can be retried on its own, and the service assembles them at the end. Most mobile upload libraries use this automatically for anything large.

Step 5 of 14 · 2 min

Getting photos to people: the CDN

Uploads are solved. Now for the other direction, which is where the latency and the money are.

Think about how photos are viewed on a social app. A popular post's photo might be viewed 10,000 times, by people in a few dozen cities. Without anything in between, every one of those views is a separate trip to the storage service in Virginia, which sends the same 100 KB 10,000 times, and charges for each one.

A content delivery network (CDN) is a network of servers spread around the world, often in hundreds of locations, called points of presence or edges. Each edge keeps a cache of files that people nearby have requested.

When Priya's phone asks for a photo:

1. Her request goes to the nearest edge, say one in Mumbai. CDNs usually do this with DNS or anycast (the same IP address announced from every location, as mentioned in lesson 2.2), so her phone doesn't need to know where the edges are.
2. If the Mumbai edge has the photo cached, that's a hit. It sends it straight back. The round trip is maybe 20 ms instead of 200.
3. If it doesn't, that's a miss. The edge fetches the photo from Chirp's object storage, called the origin, sends it to Priya, and keeps a copy so the next person in Mumbai gets a hit.

The CDN helps in three separate ways:

Latency. A hit avoids the long trip entirely. Even on a miss, things are faster than you'd expect: Priya's TLS handshake (lesson 1.1) happens with the nearby edge, not with Virginia, and the edge usually keeps warm, already-open connections back to the origin.

Load on the origin. If 95% of requests are hits, the origin only sees 5% of them. This number, the cache hit ratio, is the one to watch.

Cost. Most of the expensive traffic, from the origin out to the internet, is replaced by traffic from the CDN, which providers usually charge less per gigabyte for, especially at volume. With a 95% hit ratio, Chirp's terabyte a day from storage becomes about 50 GB a day from storage, plus the CDN's charges for the rest.

The long tail

Hit ratios aren't the same for every file. A trending photo gets cached everywhere within minutes. A photo from three years ago, viewed once a month by the owner's mum, is a miss almost every time, because the edge has long since evicted it to make room for popular files.

On most social apps, a large share of photos are this kind of rarely viewed "long tail." Two things help:

• Tiered caching (sometimes called an origin shield). Edges that miss don't go straight to the origin; they ask a regional mid-tier cache first. Lots of edges with a few requests each turn into one mid-tier cache with a decent hit ratio.
• Accepting it. The long tail is cheap per photo, because it's rarely requested. What matters is that the popular photos, which make up most of the traffic, are served from the edge.

Step 6 of 14 · 3 min

Telling caches what to do

A CDN (and the browser, and the phone's HTTP library) decides how long to keep a file based on the Cache-Control header the origin sends with it. A few patterns cover most needs:

Cache-Control: public, max-age=31536000, immutable

For files that never change, such as Chirp's photos with ID-based keys. public means any cache may store it. max-age=31536000 means it can be reused for a year without asking the origin again. immutable tells browsers not to even check back when the user hits refresh.

Cache-Control: private, no-store

For anything personal, like API responses with someone's private messages. private means shared caches like the CDN must not store it, and no-store means nothing should keep a copy at all.

Cache-Control: public, max-age=60, stale-while-revalidate=300

For things that change but can be a little out of date, like a public trending page. Caches use it for 60 seconds, then may keep serving the old copy for up to five more minutes while they fetch a new one in the background. Lesson 2.4 explains why serving slightly stale data while refreshing is such a useful trick.

When a cached copy expires, the cache doesn't have to download the whole file again. If the origin sent an ETag (a version identifier) with the file, the cache can ask "has this changed since version "a1b2c3"?" and get a tiny 304 Not Modified reply if it hasn't.

The avatar that wouldn't change

Chirp learned the most important caching rule from profile pictures.

Avatars were stored at a fixed key, avatars/{user_id}.jpg, and served with a one-day cache lifetime. When users changed their avatar, the new image was uploaded to the same key. The CDN, and everyone's phones, kept serving the old one for up to a day. People changed their picture, saw nothing happen, and changed it again. Some filed bug reports. One asked whether Chirp was deliberately showing her ex-boyfriend's photo, because her old avatar was a picture of the two of them.

CDNs do let you purge (invalidate) a cached file. But purges take time to reach every edge, there are often limits on how many you can do, and the phone's own cache doesn't know about them at all.

The reliable fix is to never change what a URL points to. Change the URL instead. Each new avatar is uploaded to a new key, avatars/{user_id}/{version}.webp, and the user's profile in the database stores the current key. A new avatar means a new URL, which no cache has ever seen, so everyone fetches it immediately. The old URL stays cached until it expires and nobody asks for it again.

This trick is called cache busting, and it works for any static file. Website code is often served the same way, with a hash of the file's contents in the name (app.3f9c2a.js), so new releases are fetched immediately and old ones can be cached forever.

Private photos

Some Chirp accounts are private, so their photos shouldn't be viewable by anyone who happens to get the URL. CDNs and object stores support signed URLs for reading too: Chirp's API checks that the viewer is allowed to see the photo, then gives them a URL that only works for a few minutes. The CDN checks the signature before serving.

Two things to keep in mind: anyone who has the URL can use it until it expires, so keep expiry times short. And configure the CDN to cache the file without including the signature in its cache key, or every viewer's unique signature turns into a separate cached copy and the hit ratio collapses.

Step 7 of 14 · 1 min

What the CDN can't help with

The CDN is excellent for files that are the same for everyone. It does much less for Chirp's API responses, because most of them are personal: Priya's feed is different from Meera's. Those can't be shared from an edge cache, and they're marked private.

Even then, routing API traffic through a CDN's network still helps a little: the TLS handshake happens nearby, and the edge's connection back to Chirp's servers is already warm. But the real fix for slow personalized responses in faraway countries is running servers closer to users, which is lesson 5.5.

Step 8 of 14 · 2 min

From the real world: the morning S3 went away

Object storage is so reliable that it's easy to forget it can fail as a whole. On 28 February 2017, Amazon S3 in its largest region, us-east-1 in northern Virginia, was unavailable for about four hours, and took a large part of the internet with it.

At 9:37 in the morning (Pacific time), an engineer was debugging a slow billing system. Following an established playbook, they ran a command to take a small number of servers out of service. One input was mistyped, and a much larger set of servers was removed, including large parts of two core subsystems:

• the index subsystem, which keeps track of every object: its name and where its data lives. Without it, S3 can't serve any GET, LIST, PUT or DELETE.
• the placement subsystem, which decides where new objects' data should go. It depends on the index.

Both subsystems needed a full restart, and that was the second problem. They hadn't been completely restarted in the larger regions for many years, and S3 had grown enormously since. The checks that make sure metadata is intact before serving took far longer at that size than anyone expected. The index subsystem was serving again by early afternoon, and S3 was fully recovered around four hours after the command.

Everything built on S3 in that region failed too: new EC2 instances (which boot from images stored in S3), EBS volumes restored from snapshots, Lambda, and thousands of websites and apps. AWS's own status dashboard couldn't show the problem properly at first, because the dashboard's icons were stored in S3.

What AWS changed, and what's worth copying:

• Tools that remove capacity now do it slowly, and refuse to go below a minimum. A safe tool makes the wrong command fail, instead of relying on people not to type it.
• They split the subsystems into smaller "cells", so restarting one is fast and affects only part of the system.
• Recovery paths need exercising at real scale. "We've never had to restart it" means "we don't know how long it takes."

For Chirp: S3 is still far more reliable than anything Chirp would build. But know what happens if the bucket's region is unavailable: the CDN keeps serving popular cached photos, uploads fail, and anything that reads from S3 on the request path breaks. Cross-region replication of the bucket, and an app that can fall back to it, turns that into a degraded hour instead of a dead one.

Step 9 of 14 · 1 min

Going deeper: the cache key decides your hit ratio

A CDN can only reuse a cached response if the next request asks for the same thing. "The same thing" is defined by the cache key: by default, the full URL, including the query string, plus any request headers the origin listed in its Vary header. Most hit-ratio problems are cache-key problems.

Chirp found two of them in its first month on a CDN:

• Tracking parameters. When people shared photo links, apps added things like ?utm_source=whatsapp or ?igshid=.... Each variant was a separate cache entry for the same file. Popular photos were being fetched from S3 again and again.
• Vary: User-Agent. One endpoint served WebP to browsers that support it and JPEG to others, and signalled that with Vary: User-Agent. There are thousands of distinct user-agent strings (every browser version, every phone model), so practically every request was a new cache key.

The fixes:

• Normalize the key. Configure the CDN to drop query parameters that don't change the response (or to keep only an allow-list of ones that do), and to sort the rest.
• Vary on something small. If you need different formats, have the CDN reduce the Accept header to two or three values ("supports WebP", "supports AVIF", "neither") and vary on that, or put the format in the URL itself (photo.webp).
• Never vary on headers with many values: User-Agent, Cookie, Authorization. If a response depends on who the user is, it shouldn't be in a shared cache at all.

When you look at a CDN dashboard, check hit ratio by bytes and by requests, and look at the most-requested URLs that miss. Those are almost always one cache-key mistake away from hitting.

Step 10 of 14 · 1 min

What Chirp changed

• Photos upload directly from phones using signed URLs, with a size limit and a content-type check in the upload policy.
• A background worker checks each upload, strips location data, and creates feed and thumbnail versions in WebP.
• Every photo key is generated by the server and immutable. Avatars got versioned keys.
• Photos are served through a CDN with public, max-age=31536000, immutable, plus tiered caching.
• Photos from private accounts use short-lived signed URLs, with the signature excluded from the cache key.

App server CPU at peak dropped by about a third once upload bytes stopped passing through. Priya's feed photos started arriving in tens of milliseconds instead of hundreds. And the next month's bandwidth bill was about 70% lower, even though traffic had grown.

Step 11 of 14 · 1 min

In the simulation

Stage 0's decision D2 (where to store images) has consequences that play out in Stage 1 and beyond. If you kept images on the app server's disk, the second server makes half of them disappear. If you put them in the database, watch the backup time and replica lag in Stage 2. Stage 1 also lets you turn a CDN on and see the effect on latency and cost for users in different regions.

Open the simulation →

Step 12 of 14 · 1 min

If you're here for an interview

Any design with images or video needs this lesson, and interviewers expect it to come up without being asked:

• Files go in object storage; the database stores only the key and metadata.
• Uploads go directly to storage with a signed URL. Saying this unprompted shows you've thought about app server load.
• Downloads go through a CDN. Mention hit ratio and what it does to origin load and cost.
• Versioned or content-hashed URLs, never in-place updates, so caching works. This is a frequent follow-up: "How do you update a cached image?"

For video, add multipart (resumable) uploads, and transcoding into several sizes and bitrates. Part 7's video scenario goes much deeper.

Step 13 of 14 · 2 min

Try these yourself

Step 14 of 14 · 1 min

Further reading

• Amazon S3 documentation: the data consistency model and the performance design guidelines.
• RFC 9111, HTTP Caching. The authoritative reference for Cache-Control, ETag and revalidation.
• Ilya Grigorik, High Performance Browser Networking, for how CDNs and connection reuse change real load times.
• Cloudflare Learning Center, "What is a CDN?" and its pages on cache hit ratio and tiered caching.
• Amazon Web Services, "Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region," March 2017.
• Fastly, "Best practices for using the Vary header," and Cloudflare's documentation on cache keys and query-string sorting.