A field manual for the day your own write vanished
The Lag.
CAP told you what you lose when the cable snaps. This is the accounting for the other 99.9% — because the “C” you kept is not a switch but a ladder, and most systems sit two rungs lower than their teams believe. You will post a comment and watch it vanish, watch time run backward on a healthy wire, then buy the promises back one millisecond at a time — and meet CAP again at the bottom of a quorum dial.
the truth left the primary a moment ago. two copies have it. the third is still in the air.
BEGINFive promises, sold separately.
Ask a team what their replicated database promises and they will say “consistent.” That word covers at least five different contracts, each bought separately, each with its own price — and the load balancer in front of your replicas silently chooses which one you actually get. The Cache treated staleness as a budget on copies: how old may one answer be? This essay is the other axis: staleness as a question about order — which writes must a read have seen, and must your own reads make sense to you?
- Eventual
- Replicas converge — eventually. Until then, any read may see any recent past. The floor everyone gets for free, and the rung everyone assumes is higher.
- Read-your-writes
- You always see your own writes. The comment you just posted does not vanish on refresh. The rung users notice missing first.
- Monotonic reads
- Once you’ve seen version X, you never see older. Time, for you, moves forward. The rung that stops your own feed from un-happening.
- Causal
- You see everything that caused what you saw: the reply implies you’ve seen the comment. Across sessions, users, services.
- Linearizable
- One copy, one order, as if the replicas never existed. The top rung — and you already know its price from The Cache and The Lock.
You saw it. Then you didn’t.
Below: one primary, three replicas doing their asynchronous best, and you — a session whose every read is bounced to whichever replica the load balancer likes today. Post a comment, then hit READ a few times. Two crimes are possible, and both leave every dashboard green:
Crime one — the vanish. Your write is acknowledged at the primary; your next read lands on a replica that hasn’t heard yet; the comment you are looking at is gone. That is a read-your-writes violation, and it has a human name: “the avatar that wouldn’t update.” Users uploaded the new picture, saw the old one for an hour, and filed it as “the site is haunted.” Crime two — the bend. Read twice, quickly. If the second read lands on a different, laggier replica, you can be served a version older than one you already saw. Your feed un-happens. Time runs backward. Nothing failed.
Read the trace strip like a cardiogram: each dot is a read, positioned at the version it was shown, connected in the order your session experienced them. Green dots are current; amber are behind; a red ring means a violation just happened. And the segment that runs leftward — the one stamped ◀ TIME BEND — is the moment your own history rewrote itself. The lag meters up top never alerted anyone. They report distance; violations are about order.
MODEL NOTES — versions are a simple counter; each replica applies updates on its own jittered timer; reads choose a replica uniformly; background users keep the world writing.
Buy the rung back.
Same world, same lag — but now your session carries separate guarantees you can combine. Sticky sessions pin reads to one replica and prevent backward reads, though that replica may still lag behind your own writes. A session watermark records acknowledged writes and versions already read; route each read to a replica that has reached it. Read from primary is another way to see acknowledged writes, at a different load cost.
Toggle one, SOAK, watch the violation counters stay flat — then read the cost line, which never blinks.
Read the price list like an engineer, not a shopper. Sticky is the only free rung — and it buys monotonic, never read-your-writes, because your pinned replica may lag behind your own writes forever. The watermark is the interesting purchase: a few bytes of session state, a few milliseconds of occasional rerouting, and two violations die at once. That little token — invented in the Bayou system in 1994 — is the difference between a team that configures replication and a team that operates it.
R + W > N, drawn in packets.
Leaderless stores such as original Dynamo and Cassandra expose N, W, and R. If R + W > N, every successful read set overlaps every acknowledged write set under fixed membership. The reader must still choose the newest version correctly; overlap alone does not prove linearizability. The figure shows a worst-case placement against the configured membership.
Drag the dials. Then kill a replica: some operations become unavailable, but the guarantee for a write already acknowledged still uses configured N. The failed node might have held the only copy of that write. Changing membership safely takes a separate protocol; simply counting survivors would turn a lost write into a false green light.
Dynamo turned the quorum into a menu: N, R, and W as dials, consistency priced per read and per write.THE DESIGN, PARAPHRASED FROM DYNAMO — SOSP 2007
Set W=1, R=1 on N=3 and you have pure speed — and zero handshake, which is the polite term for “stale reads are now someone else’s incident.” Set R=3, W=3 and you have certainty priced at a full round trip on every operation. Neither is wrong. What’s wrong is not knowing which one you shipped — and that, in an interview, is a sentence worth saying out loud.
MODEL NOTES — N is fixed at three for this figure. The highlighted survivor sets are one placement, while the formula gives the historical worst-case overlap. A completed write could sit on the failed node; changing N safely requires a membership protocol. The model does not simulate read repair or version conflict resolution.
The field guide.
- Session token
- The watermark: the highest version your session has read or had acknowledged as a write. Hand it back with every read and routing becomes arithmetic — any replica at or past your token will do.
- Sticky routing
- Pin a session to one replica. Monotonic reads for free; RYW only if that replica stays caught up. Costs nothing until the replica dies or lags.
- Quorum
- Under fixed membership, R + W > N makes every successful read set overlap every acknowledged write set. Correct version selection is still needed to return the latest value.
- Leader-based replication
- One primary takes writes, followers trail it. Rungs are bought in the routing layer — tokens, stickiness, or “just read the primary.”
- Leaderless replication
- No primary; every replica takes reads and writes subject to the quorum. Rungs are bought in the storage layer — the R and W dials.
- Replication lag
- The distance between the primary’s version and a replica’s. Meters report it everywhere; it tells you nothing about order, which is the thing users actually notice.
WHY MONOTONIC READS NEED A TOKEN
Replicas move forward independently, at their own pace. Nothing in the system remembers what you have seen — that memory exists nowhere except your session.
So a load balancer that picks “a” replica is picking from states with no ordering guarantee relative to your last read. Backward is always one bounce away.
The session token smuggles the missing memory back into the system: “I have seen v12” turns routing into arithmetic, and any replica at or past v12 makes time flow forward again. Three lines of client code, one stateless header — and the haunt is over.
Three scenarios.
From these figures, you can state which reads a quorum or a session watermark actually guarantees. Try changing one assumption and check whether your explanation still holds.
The sealed sheetThree questions are sealed inside this sheet. Nobody is asked to open it — the replicas will catch up eventually.Break the seal
A user uploads a new avatar, gets “saved ✓,” refreshes, and sees the old picture for a while. Which rung broke?