Step 2 of 15 · 1 min

Money is never a floating-point number

The creator's ₹3,91,140.00000000006 came from storing amounts as floating-point numbers (double in most languages, float in some databases). Floating point can't represent most decimal fractions exactly. 0.1 + 0.2 is 0.30000000000000004 in nearly every programming language. Add up 40,000 tips in floating point and tiny errors accumulate.

The rule is simple and universal: store money as an integer number of the smallest unit: paise for rupees, cents for dollars. ₹100 is stored as 10000. Arithmetic on integers is exact. Convert to "₹100.00" only when displaying. Some databases have exact decimal types (numeric in Postgres) that also work, but integers in minor units are simpler and harder to get wrong. Always store the currency next to the amount, even if you only support one today.

Step 3 of 15 · 2 min

A ledger, not a balance

The first version of tips did two things per tip: insert a row into tips, and UPDATE creators SET balance = balance + amount. That's the counter pattern from lesson 3.3, and for money it has two serious problems:

• The balance is the only record. If it's ever wrong (a bug, a partial failure, a bad manual fix), there's no way to tell what it should have been.
• History can be rewritten. Anyone with database access can change a balance, and nothing shows that it happened.

Every serious financial system, from banks to payment companies, uses a ledger instead, built on an idea that's centuries old: double-entry bookkeeping.

Double entry

Money is never created or destroyed; it moves between accounts. So every movement is recorded as a set of entries, at least one taking money out of an account (a debit) and at least one putting money into another (a credit), and the entries for each movement always add up to zero.

A fan tipping a creator ₹100, with Chirp keeping a 10% fee, is one transaction with three entries:

Rules that make the ledger trustworthy:

• Entries are append-only. They're never updated or deleted. A mistake is corrected by adding a new transaction that reverses it, so the history shows both the mistake and the correction.
• Every transaction sums to zero, checked by the database (in a constraint or in the code that writes the transaction, inside one database transaction).
• A balance is derived from the entries: the sum of all entries for that account. For speed, it can be cached in a balance column, updated in the same database transaction as the entries (lesson 3.3), and regularly checked against the sum.
• Every entry links to its cause: the tip, the payment, the payout.

With a ledger, the question "what should this creator's balance be?" always has an answer: add up their entries. And "how did it get to this number?" has an answer too: list them.

Step 4 of 15 · 2 min

Calling the payment provider

Chirp doesn't move money between bank accounts itself. It uses a payment provider (a payment gateway), which talks to card networks, banks and UPI. The provider's API is where the double charges came from.

Here's what happened during the match. Chirp's server sent "charge this fan ₹100" to the provider. The provider was overloaded, took 12 seconds, and Chirp's request timed out at 10. Chirp's code treated the timeout as a failure and retried. But the first charge had gone through; the response simply arrived too late. So the fan was charged twice.

This is lesson 1.5's problem exactly (the train in the tunnel), and lesson 6.1's warning about retries. A timeout from a payment provider means "I don't know whether it happened", never "it failed."

Idempotency keys, end to end

The fix starts before the first call. When a fan taps "Tip ₹100," Chirp creates a payment record with a unique ID before calling the provider, and uses that ID as an idempotency key in every request to the provider for this payment. Most payment providers support idempotency keys for exactly this reason: if Chirp retries with the same key, the provider recognizes it and returns the original result instead of charging again.

The key also goes further up the chain: the fan's app sends its own idempotency key when the fan taps "Tip" (lesson 1.5), so a double tap or an app retry finds the same payment record rather than creating a new one.

When you still don't know

Even with idempotency keys, Chirp sometimes doesn't know the outcome: the provider was unreachable, or the response was lost. The payment record then stays in an unknown state, and Chirp asks instead of guessing:

• Query the provider's status API for that payment ID, a few seconds later, with backoff, until there's a definite answer.
• Listen for webhooks: providers call a Chirp endpoint when a payment's status changes (succeeded, failed, refunded). Webhooks can arrive late, twice or out of order, so they're processed idempotently, keyed by the provider's event ID (lesson 4.2's processed-events table).
• Never assume. A tip isn't shown as sent, and the creator isn't credited, until the provider has confirmed the payment succeeded.

The fans who were charged with no tip recorded hit the opposite failure: the provider charged them, then Chirp's server crashed before recording the success. With a payment record created first and a status check that resolves unknown payments, those payments are found and completed within minutes. And the daily reconciliation (below) catches anything that slips through.

Step 5 of 15 · 1 min

A payment is a state machine

A tip isn't one event; it passes through several states, and each move between them is a deliberate, recorded transition:

created → pending at provider → succeeded → credited to creator → (later) paid out
                              ↘ failed
           succeeded → refunded (rare)

Writing it as an explicit state machine (a status column, with only certain transitions allowed, each one done as a conditional update such as UPDATE payments SET status = 'succeeded' WHERE id = :id AND status = 'pending') has real benefits:

• Retries and duplicate webhooks become harmless. A second "succeeded" for a payment that's already succeeded changes nothing (the conditional update matches no rows).
• Every payment is always in a known state. A monitoring job can list payments stuck in "pending" for more than 10 minutes and resolve them.
• The audit trail is clear. Each transition is logged with who or what caused it and when.

Many providers also split card payments into two steps: authorize (reserve the money on the card) and capture (actually take it). That's useful when there's a gap between ordering and fulfilling, such as a ride whose final fare isn't known yet (lesson 7.2). For instant tips, Chirp authorized and captured in one step.

Step 6 of 15 · 1 min

Steps that span systems: sagas

Crediting a creator after a successful payment involves several steps across systems: write the ledger entries, update the creator's dashboard, send a notification, report the transaction for tax purposes. They can't all be one database transaction (lesson 4.2), and some steps can fail after others have succeeded.

The usual pattern is a saga: a sequence of local steps, each committed on its own and each triggering the next (typically through the outbox, lesson 4.2), where every step has a compensating action that undoes its effect if a later step fails for good. For example, if a payout to a creator's bank account is rejected (wrong account number), the compensating action adds ledger entries that return the money to the creator's Chirp balance, and notifies them.

Two rules make sagas safe: every step must be idempotent (it will sometimes run twice), and compensations are new actions, not deletions. A reversed transaction appears in the ledger as its own entries, consistent with the ledger's append-only rule.

Step 7 of 15 · 1 min

Reconciliation

However carefully everything is built, the only way to know your records are right is to compare them with someone else's.

Every day, Chirp's payment provider sends a settlement report: every payment it processed, refunds, fees it charged, and the total it's depositing into Chirp's bank account. A reconciliation job compares that report, line by line, with Chirp's payment records and ledger:

• A payment the provider has but Chirp doesn't: money taken from a fan with no tip recorded. Fix it (complete the tip) and investigate why.
• A payment Chirp has as succeeded but the provider doesn't: Chirp credited a creator with money that never arrived. Reverse it in the ledger and investigate.
• Amounts or fees that differ: investigate.

The first reconciliation after the match day found 312 payments that had succeeded at the provider with no tip on Chirp's side. All were completed that afternoon, and their creators were credited. After the fixes above, daily reconciliation usually finds zero differences, and a non-zero result pages someone. Teams also reconcile the ledger with itself: every account's cached balance must equal the sum of its entries, and all entries across the whole ledger must sum to zero.

Step 8 of 15 · 1 min

The hot account

The fourth bug was lesson 4.5's hot row, in a new place. Every tip added an entry to Chirp's fee account and to the commentator's earnings account, and with a cached balance column, each tip updated those same two rows. During the innings break, that's thousands of updates a second on two rows, each waiting for the row lock.

The fixes are the same family as lesson 4.5's, adapted for money, which must stay exact:

• Don't update a cached balance on every entry for very busy accounts. Append entries (inserts don't contend), and compute the balance by summing, with a periodic snapshot (the balance as of a point in time) so the sum only covers entries since the last snapshot.
• Split the account into sub-accounts (like sharded counters): the fee account becomes 32 sub-accounts, each tip writes to one at random, and the real balance is the sum of all 32. Totals stay exact; contention drops 32 times.
• Batch internal transfers where timing allows, for example moving fee entries into the main fee account once a minute as one summarizing transaction.

Step 9 of 15 · 1 min

Security, briefly

Two points that shape the architecture, even in a lesson that isn't about security:

• Don't touch card numbers. Chirp's app sends card details straight to the provider (using the provider's own payment form or SDK), and Chirp only ever sees a token that refers to the card. That keeps Chirp's own systems largely out of scope for the strict card-security standard (PCI DSS) and keeps card numbers out of its databases and logs.
• Restrict and audit access to the ledger. Very few people and services can write to it, every write is logged, and corrections go through the same "add a reversing transaction" path as everything else, never a manual UPDATE.

Step 10 of 15 · 1 min

From the real world: the night the batch didn't run

Much of the world's money still moves in overnight batches: the day's payments, collected and applied to accounts in a long, strictly ordered run of jobs. In June 2012, that machinery failed at RBS, NatWest and Ulster Bank in the UK and Ireland, and it shows why "make every step safe to re-run" is a payments rule, not a nice-to-have.

The banks' overnight processing was orchestrated by a job scheduler, which runs thousands of jobs in the right order, each depending on the ones before. An upgrade to the scheduler software was applied, ran into problems, and was backed out. In the process, the schedule of jobs, and the order they had to run in, was lost.

The night's batch couldn't complete normally, so millions of accounts weren't updated. Wages didn't arrive, bills and transfers failed, and balances shown to customers were wrong. Worse, each night's batch depended on the previous night's results, so the backlog compounded. Engineers had to reconstruct the schedule and re-run the backlog in the correct order. Most customers were back to normal within days; some Ulster Bank customers waited weeks. Regulators later fined the group £56 million.

What it says about payment systems in general:

• Know what must happen in order, and record progress durably, so you can resume from the last completed step rather than starting over.
• Every step must be safe to run twice. Recovery always involves re-running something; idempotency keys and ledger entries with unique IDs (as in this lesson) are what make that safe.
• Rehearse recovering from a backlog, not just from a crash. Processing three days of payments at once has its own limits: throughput, ordering, and providers' rate limits.
• Treat upgrades of core infrastructure as risky changes, with a tested way back that preserves state.

Step 11 of 15 · 1 min

Going deeper: authorize now, capture later

When Chirp charges a card for Chirp Plus, the card networks split "take the money" into two steps, and a payments design that ignores the gap between them produces confusing bugs.

• Authorization asks the customer's bank: can this card pay ₹499? If yes, the bank places a hold for that amount. The money is still in the customer's account, but no longer available to spend. Nothing has moved.
• Capture tells the bank to actually take the held money. Shops typically capture when they ship; subscription services capture when the subscription is activated. Only now is the money really owed to the merchant, and only now should Chirp's ledger record revenue.
• If the merchant never captures, the authorization should be voided, which releases the hold. If nobody voids it, it expires on its own after a number of days, during which the customer sees "pending" money that never arrives, and contacts support.

So the payment state machine from this lesson needs states for this: authorized, captured, voided, expired, refunded, partially_refunded, each with its own ledger entries. Some details that catch teams out:

• Capturing less than you authorized is usually allowed (a partial capture, say an item was out of stock); capturing more usually isn't.
• Refunds are new money movements, not the deletion of a payment. They take days to reach the customer and can fail.
• Webhooks from the provider can arrive late, twice, or out of order (lesson 3.3's versioning problem). Process them idempotently, and let reconciliation be the final word.

Step 12 of 15 · 1 min

What Chirp built

• Amounts in paise, as integers, with a currency on every amount.
• A double-entry, append-only ledger: every transaction sums to zero, balances derive from entries, corrections are reversing transactions.
• A payment record created before calling the provider, used as the idempotency key for every provider call. The app sends its own idempotency key too.
• Unknown outcomes resolved by status queries and webhooks, never by guessing. Webhooks processed idempotently.
• An explicit state machine for payments and payouts, with conditional transitions.
• Sagas with compensating actions for multi-step flows, driven by the outbox.
• Daily reconciliation with the provider's settlement report, and continuous self-checks of the ledger.
• Sub-accounts and snapshots for the busiest accounts.
• Tokenized card handling; card numbers never reach Chirp's servers.

The next big match, fans sent 70,000 tips in half an hour. Reconciliation the next morning found no differences.

Step 13 of 15 · 1 min

If you're here for an interview

"Design a payment system" (or a wallet, or Stripe) is common, and interviewers are looking for specific instincts:

• Integers in minor units for money, with currency.
• A double-entry, append-only ledger, and balances derived from it.
• Idempotency keys on every request that moves money, generated before the first attempt and reused on every retry.
• "Timeout means unknown": resolve with status checks and webhooks, never by retrying blindly.
• State machines, sagas with compensations, and reconciliation against the provider.
• Hot accounts, and how to keep them exact while spreading the load.

Step 14 of 15 · 2 min

Try these yourself

Step 15 of 15 · 1 min

Further reading

• Martin Fowler's writing on accounting patterns (account, entry, posting) in Analysis Patterns, and on event sourcing.
• Stripe's engineering blog and documentation on idempotent requests and on webhooks.
• Hector Garcia-Molina and Kenneth Salem, "Sagas," SIGMOD 1987: the original paper.
• Your payment provider's documentation on payment states, status queries, webhooks and settlement reports.
• UK Financial Conduct Authority, Final Notice to The Royal Bank of Scotland plc, National Westminster Bank plc and Ulster Bank Ltd (November 2014), which describes the June 2012 IT incident.
• Stripe documentation, "Place a hold on a payment method" (authorize and capture separately) and "Payment Intents: lifecycle."