Step 2 of 12 · 2 min

Three kinds of data

Systems produce three main kinds of telemetry. Each answers different questions, and each has a different cost.

Metrics

Numbers measured over time: requests per second, error count, latency histogram (lesson 6.2), queue age (lesson 4.1), CPU. Each metric is stored as a series of values, often with a few labels (endpoint, region, status code).

Metrics are cheap to store and fast to query, because they're pre-aggregated: "1,204 requests to /v1/feed with status 200 in this 10-second window" is one number, whether that was 10 requests or 10 million. They're what dashboards and alerts are built on.

Their main limit is cardinality, the number of distinct label combinations. Labels like region (3 values) and status_code (a dozen) are fine. A label like user_id (50 million values) creates 50 million separate series, which will overload or bankrupt most metrics systems. Chirp learned this when a well-meaning engineer added post_id as a label to a latency metric, and the metrics bill for that month tripled.

Logs

Records of individual events: "request 7f3a from user 4821 to /v1/uploads failed with invalid_content_type after 82 ms." Logs carry detail that metrics throw away, which makes them essential for working out why something happened.

Two habits make logs far more useful:

• Structured logs: write each event as fields (JSON, say) rather than a free-text sentence, so they can be searched and counted: {"event": "upload_rejected", "reason": "invalid_content_type", "platform": "android", "app_version": "7.2.0"}.
• A request ID in every log line. The edge assigns each incoming request an ID and passes it to every service involved. Then one search finds everything that happened for that request, across every service.

Logs are expensive at scale: storing every event from 50 million users adds up quickly. Teams usually keep detailed logs for a short time, sample high-volume success logs, and keep every error.

Traces

A trace follows one request through every service it touches. Each piece of work (an API handler, a database query, a cache lookup, a call to another service) is a span with a start time, a duration and a parent. Laid out on a timeline, the spans show exactly where the time went.

For lesson 6.2's slow home feed requests, traces were what found the causes: a span showing a 400 ms wait on one database shard, or a 200 ms gap where a packet had been resent.

Traces work by propagating context: each service passes the trace ID and its own span ID to the services it calls, in a header (the W3C Trace Context standard defines one, traceparent). Libraries such as OpenTelemetry do this automatically for most frameworks.

Recording every span of every request is too much at Chirp's scale, so traces are sampled. Head-based sampling decides at the start of a request whether to record it (say, 1 in 100), which is cheap but misses most rare slow or failed requests. Tail-based sampling collects all spans briefly and keeps a trace only if it turned out interesting (slow, or an error), which is more expensive to run but keeps exactly the traces you want.

How they fit together

A typical investigation uses all three: a metric shows that something is wrong (errors rising on uploads), a trace shows where (a span in the upload service failing fast), and logs show why (the invalid_content_type error, all from Android, all on app version 7.2.0).

Step 3 of 12 · 1 min

The numbers worth watching

With 400 dashboards, the problem wasn't missing data. It was knowing which numbers mattered. Two short lists, both widely used, cover almost everything.

For anything that serves requests (an API, a service), the RED method:

• Rate: requests per second.
• Errors: how many fail.
• Duration: how long they take, as percentiles from histograms (lesson 6.2), never averages.

For anything that's a resource (a server, a database, a queue, a disk), the USE method:

• Utilization: how busy it is (CPU %, disk throughput).
• Saturation: how much work is waiting (queue length, threads waiting for a connection, lesson 4.1's queue age).
• Errors: failed operations (disk errors, dropped packets).

Google's SRE book describes a similar set, its four golden signals: latency, traffic, errors and saturation.

Chirp rebuilt its dashboards around these: one page per service with its RED numbers and its dependencies' health, and one page per shared resource with its USE numbers. Most of the 400 old dashboards were deleted; nobody missed them.

Step 4 of 12 · 1 min

SLOs: how reliable is reliable enough?

The deeper fix was deciding what "working" means, in numbers, from the user's point of view. That's what service level objectives do.

A service level indicator (SLI) is a measurement of something users care about, as a ratio of good events to total events. For example:

• Upload success: the share of photo uploads that succeed.
• Feed latency: the share of home feed loads that complete in under 500 ms.

A service level objective (SLO) is a target for an SLI over a period of time: 99.9% of uploads succeed, measured over 30 days.

Two things are important about how these are chosen.

They're measured where users feel them. Upload success is measured at the API edge, per request, broken down by platform, not inferred from CPU graphs. If Chirp had measured "upload success rate for Android" on that Sunday, it would have dropped from 99.8% to about 60% within minutes.

They're not 100%. No complex system is 100% reliable, and trying costs enormous effort. More importantly, users can't tell the difference between 99.99% and 100%, because their own phone, Wi-Fi and mobile network fail far more often than that. The question is what level users are happy with, and that's a product decision as much as an engineering one.

Error budgets

An SLO comes with something useful: an error budget. If the target is 99.9% over 30 days, then 0.1% of requests are allowed to fail. For a target measured in time, 0.1% of 30 days is about 43 minutes of complete failure (or more time at partial failure).

That budget turns arguments into arithmetic. When the budget is healthy, teams can move fast, ship features and take risks. When most of it has been used, the team slows down: fewer risky deploys, more time on reliability work, until it recovers. It gives product and engineering a shared, objective way to decide between speed and stability, instead of arguing about whether things "feel" reliable enough.

Step 5 of 12 · 2 min

Alerts that mean something

Chirp's 900 alerts had three problems: most were about causes (CPU high, a disk 80% full, one server down) rather than user pain; many fired for things that fixed themselves; and the thresholds were guesses. The result was alert fatigue: when most alerts don't matter, people stop treating any of them as urgent.

The new approach:

Page on symptoms, not causes. Wake someone up when users are hurting, measured by an SLO, not when a server's CPU is high. A server at 95% CPU that's still serving every request quickly isn't an emergency. An upload success rate dropping is, whatever the cause. Cause-based signals (disk filling up, certificate expiring next week) still matter, but they become tickets for working hours, not pages at 3 am.

Alert on how fast the budget is burning. Rather than a fixed threshold like "error rate above 5%," alert when errors are using up the error budget dangerously fast. The burn rate is how many times faster than "exactly on budget" you're spending it. A burn rate of 1 uses the whole 30-day budget in exactly 30 days. A burn rate of 14.4 would use 2% of it in one hour.

Google's SRE Workbook describes a widely copied scheme: page someone if the burn rate is above about 14 over both the last hour and the last five minutes (a fast, severe burn that's still happening); open a ticket for slower burns over longer windows. Requiring both a long and a short window means alerts fire for real problems and stop soon after they're fixed.

On that Sunday, an Android-specific upload SLO would have been burning at roughly 400 times the allowed rate. A burn-rate alert would have fired within about two minutes.

Every page needs an action. If the on-call engineer can't do anything about an alert, or it fixes itself, it shouldn't page. Chirp attached a short runbook (what this alert means, how to check, what to try first) to every alert that could page someone.

Check from the outside too

Everything above is measured from inside Chirp. It's worth also having synthetic checks: small scripts, run from outside Chirp's network in several countries, that act like a user every minute. Log in, load a feed, upload a small photo from an Android-like client. They catch problems that internal metrics can miss: DNS mistakes (lesson 1.1), certificate errors, a CDN misconfiguration (lesson 2.3), or a whole region being unreachable.

Step 6 of 12 · 2 min

From the real world: 73 hours without eyes

Roblox's outage from 28 to 31 October 2021 lasted 73 hours, one of the longest at any large consumer service, and its postmortem is a long, honest account of how hard diagnosis gets when observability shares a failure with production.

Roblox ran on its own data centers: over 18,000 servers and 170,000 containers. At the centre was HashiCorp Consul, used for service discovery (how services find each other), health checks and a key-value store for configuration. Nearly everything depended on it.

A day before the outage, Roblox enabled a new Consul feature, streaming, on a heavily used service. Under very high load, streaming caused contention inside Consul, on a single Go channel, and Consul's writes slowed dramatically: the median write went from under 300 ms to about 2 seconds. Services that needed Consul began failing, and player numbers fell by half within hours.

Then the part that made it last three days: Roblox's monitoring system also depended on Consul. The telemetry engineers needed to understand Consul's behaviour was unavailable or unreliable because of Consul. Teams spent many hours testing plausible theories. Upgrading the Consul servers to machines with twice the CPU cores didn't help.

A second problem was found along the way. Consul stored its Raft log in BoltDB, which never shrinks its file on disk. The file had grown to 4.2 GB while holding only 489 MB of data, and BoltDB's list of free pages had become so large that every append rewrote it, slowing writes on some Consul leaders. Disabling streaming and compacting the BoltDB files finally restored the cluster, and players were brought back gradually, by region, until full service on the 31st.

Roblox's follow-ups included removing the monitoring system's dependency on Consul. That's the main lesson here: your observability must not share a failure domain with what it observes. Put telemetry, alerting and incident tooling on separate infrastructure, with its own simple way to find its components (static configuration is fine), and test that you can still see when the core platform is down.

Step 7 of 12 · 1 min

Going deeper: sample at the end, not the start

Traces are expensive to keep, so most teams keep only a fraction: say 1%. The usual way, head sampling, decides at the very start of a request: flip a coin, and pass the decision to every service so they all record or skip the same request. It's cheap and simple.

The problem is in the timing. When the coin is flipped, nobody knows whether this request will be slow or fail. So you keep 1% of everything, including 1% of the errors and slow requests. If 200 out of 100,000 requests go wrong, you keep about 2 of them: precisely the traces you need during an incident.

Tail-based sampling moves the decision to the end. Every service sends every span to a collector tier, which holds each trace in memory for a short while (say 30 seconds) until it's complete. Then it decides with full knowledge:

• keep every trace with an error;
• keep every trace above a latency threshold, or from an important customer;
• keep a small random share of everything else, for baselines.

Storage costs are similar to head sampling; usefulness is far higher. The cost moves elsewhere: the collectors must receive and buffer every span, so they need real capacity, and all spans of one trace must reach the same collector (usually by load-balancing on trace ID). The OpenTelemetry Collector's tail-sampling processor implements this, and several tracing vendors do it as a service.

A pragmatic middle ground many teams use: head-sample at a modest rate for baselines, and always record a trace when a service sees an error or a slow request, flagging it so downstream services keep their spans too.

Step 8 of 12 · 1 min

What Chirp changed

• SLIs and SLOs for the things users do most: open the app, load the feed, post, upload a photo, send a message. Each is measured at the edge, broken down by platform and region.
• Error budgets reviewed weekly with the product team. Twice that year, a team paused feature work for a sprint because its budget was nearly spent.
• Burn-rate alerts for paging; cause-based alerts turned into tickets. 900 alerts became 41 that could page someone, each with a runbook.
• RED dashboards for every service and USE dashboards for every resource, replacing 400 old ones.
• Structured logs with request IDs everywhere, and traces with tail-based sampling that keeps every slow and failed request.
• A cardinality budget for metrics labels, checked automatically.
• Synthetic checks from five countries, including an Android-like upload test.

The next time a bad configuration broke something for one platform, the burn-rate alert fired in three minutes, the trace pointed at the config change, and it was rolled back in eleven. Most users never noticed.

Step 9 of 12 · 1 min

In the simulation

The "incident day" epilogue scores you partly on detection: how long until you know something is wrong, and whether your alerts would have woken someone for the right reason. With only cause-based alerts (CPU, disk), some failures go unnoticed until the "user complaints" counter climbs. With SLOs and burn-rate alerts, you find out from the dashboard, not the complaints.

Open the simulation →

Step 10 of 12 · 1 min

If you're here for an interview

Monitoring often comes up near the end of a design ("how would you know it's working?"). Strong answers are specific:

• Name the SLIs for the system you designed, from the user's side: "the share of messages delivered within 2 seconds," not "server health."
• Set SLOs with a reason, and mention error budgets.
• Describe alerting on SLO burn rate, not raw thresholds or CPU.
• Mention all three telemetry types and what each is for, including request IDs and tracing across services.
• Watch for cardinality if you're labelling metrics with user or item IDs.

Step 11 of 12 · 2 min

Try these yourself

Step 12 of 12 · 1 min

Further reading

• Site Reliability Engineering (Google), chapters 4 ("Service Level Objectives") and 6 ("Monitoring Distributed Systems"). Free online.
• The Site Reliability Workbook, chapter 5, "Alerting on SLOs," for burn rates and multiwindow alerting.
• Brendan Gregg, "The USE Method," and Tom Wilkie, "The RED Method."
• Cindy Sridharan, Distributed Systems Observability (O'Reilly, free report).
• The OpenTelemetry documentation, and the W3C Trace Context specification.
• Roblox, "Roblox Return to Service 10/28–10/31, 2021," January 2022.
• OpenTelemetry documentation, "Sampling" (head and tail sampling) and the Collector's tailsamplingprocessor.