LEADER ELECTIONDOWNTIME PRESSAN INTERACTIVE ESSAY

A field manual for the day the king died

The Vote.

Consensus needs a leader — one node that orders the writes — and the leader is mortal. The whole trick is making the crown replaceable: any node may wear it, so a dead king costs one election, not the system. But the vote itself is the easy part. The hard parts are ending the election and keeping it honest. You will freeze a cluster in an infinite storm, decrown an isolated leader, watch a coup come home from exile — and crown a ghost that erases a committed fact.

three candidates, one empty throne, and a vote that never came home.

BEGIN
№ 01THE CROWN

The vote is the easy part.

The Lock ended with a coordinator who could not be allowed to die — and died anyway. Consensus is the redemption arc: make the coordinator disposable. A leader is just the node everyone currently agrees to route writes through, and an election is the procedure for handing the crown on when its head detaches. The crown matters; the head under it is interchangeable. That is why a CP cluster survives machine death with seconds of unavailability instead of a forensic investigation.

Here is the misconception this essay exists to dismantle: that the election is the vote — one node, one ballot, majority wins, done. The vote is three lines of code. The discipline is in two duties the naive vote ignores. Safety: at most one leader per term — two kings means The Lock’s split-brain, wearing ermine. Liveness: the election must actually finish — and you will see that naive democracy cannot guarantee even that. Three pieces of machinery carry the whole load: randomized timers to reduce repeated split votes, the log check so the winner is safe to obey, and pre-vote so exiles don’t burn the village on their return. Kill things. Slide things. Read the log — it narrates the politics.

Consensus is not agreement about who is best. It is agreement about who is next.THE ONLY CAMPAIGN PROMISE THAT MATTERS
№ 02THE STORM

Democracy cannot terminate.

When the leader dies, every follower’s election timer starts running. The naive design gives them all the same timeout — and so, when it expires, all of them campaign in the same instant. Each voted for itself before any request arrived; each request is declined; nobody reaches a majority. They reset their timers — to the same value — and do it again. Forever. The counters below climb while the cluster serves nothing: term inflation as performance art.

The fix is the most counterintuitive sentence in this series: plant disagreement about when to ask. Each node’s timeout is its own — base plus randomness — so one node’s timer almost always fires first, its requests arrive while the others still follow, and it wins before the field even forms. You fix consensus by making the nodes worse at synchronizing. Slide the jitter while the storm is running and watch it break.

FIG. 01 — THE STORM · SYNCHRONIZED TIMERSKILL THE LEADER · THEN FEED THE STORM SOME JITTER

Let it hit round six, seven, eight — the term counter racing while every request is declined. Then drag jitter to 40% without resetting and watch the contenders separate. Randomized timeouts reduce repeated split votes; this run may elect on the next round, but no particular round is guaranteed. Correlated stalls such as GC pauses can still bunch timers together.

MODEL NOTES — heartbeats modeled as a synchronized broadcast (real clusters get partial de-synchronization from network jitter, and still storm after correlated stalls) · vote flight 180 ms each way · base timeout 3 s · majority = 2 of 3 · the dead king stays dead until RESET.

№ 03THE COUP

Exile, and the return of the exile.

Storms happen without any network trouble. Partitions are where elections earn their salary — and where two delusions must be armed against. Delusion one, the leader’s: cut off from its majority, it still wears the crown and doesn’t know the world moved on. check-quorum is the cure: my crown is real only while a majority answers my heartbeat. Delusion two, the exile’s: an isolated follower keeps timing out, and without protection keeps spending terms on elections it cannot win — inflating its term like a soldier mailing promotions to himself. When the cable heals, it marches home senior to everyone. pre-vote is the cure: never spend a term without evidence of support.

FIG. 02 — THE COUP · PARTITION, HEAL, CONCEDECLICK A CABLE TO CUT · CLICK AGAIN TO HEAL

MODEL NOTES — heartbeats 1.05 s · election timeouts 2.4–3.8 s randomized per reset · check-quorum after 3.2 s of majority silence · pre-vote declines any peer that heard a leader within one timeout · followers adopt the leader’s log on heartbeat (simplified truncation).

№ 04THE BALLOT

Why the winner may be obeyed.

When someone wins, why is it safe to obey them? Raft voters compare a candidate’s last log term, then its last index, with their own. Together with majority intersection, log matching, and the rule for committing entries from the current leader term, that check supports leader completeness. The ballot lab isolates the log comparison; the full theorem needs more than one sentence about overlapping majorities. Scenario B lets an up-to-date IAD win while blocking a stale SIN.

FIG. 03 — THE BALLOT · WHO IS SAFE TO CROWNPICK A SCENARIO · CAMPAIGN · THEN TRY NAIVE RULES

Scenario C shows the other boundary: FRA’s extra entry sits on one node only — uncommitted — and can be overwritten when IAD wins. Scenario B under full rules can still elect IAD, which holds committed entry 3; it rejects SIN’s stale candidacy. The rule refuses a leader that could erase a committed prefix, while allowing an eligible leader to restore progress.

WHY A WINNER PRESERVES COMMITTED ENTRIES — THE INTUITION

Raft counts an entry from the current leader term as committed after replication on a majority. An older-term entry is not committed merely because it appears on a majority; a later current-term commit can establish that prefix.

An election also needs a majority, so at least one voter overlaps a committed entry’s replication set. The voter checks the candidate’s last log term and index, not just its length.

The formal argument also uses log matching and induction across terms to show that an eligible winning candidate preserves the committed prefix. See Raft §5.4 and Figure 8 for the old-term counterexample and proof boundary.

MODEL NOTES — a single-threaded election lab: each campaign advances the term and spends every voter’s ballot for that term · committed = confirmed on a majority at setup, and stays committed (that permanence is the point) · LET IT LEAD models leader-completeness: followers adopt the leader’s log, tails included.

№ 05FIELD GUIDE

The field guide.

Term
The era counter. One election attempt, one term; a higher term outranks everything — seniority, history, the incumbent’s feelings. The unit of political time.
RequestVote
The campaign message: my term, my log’s last entry. Carries no data — just the claim and the credentials for checking it.
Split vote
Multiple candidates campaign before any wins; each node votes once per term, so a three-way race can yield 1–1–1. Random timeouts make repeated ties less likely.
Randomized timeout
A newly sampled election delay reduces synchronized campaigns. It improves the chance of progress under a stable network; it does not guarantee the next round succeeds.
Pre-vote
A secret ballot before the real one: poll for support without spending a term. Stops partitioned nodes from inflating their terms and disrupting healthy clusters on return.
Check-quorum
The leader steps itself down after an election timeout of majority silence. Protects clients from a king whose kingdom has quietly left.
Log up-to-dateness
The ballot rule: compare last entry’s term, then length. The voter’s veto that makes the winner safe before the winner exists.
Leader completeness
The guarantee that falls out: every crowned leader’s log contains all committed entries. Committed facts cannot be voted out of existence.
№ 06WANNA TRY OUT WHAT YOU HAVE LEARNED?

Three scenarios.

From these figures, you can read the election log and explain why a stale candidate cannot take leadership. Try changing one assumption and check whether your explanation still holds.

The sealed sheetThree questions are sealed inside this sheet. Nobody is asked to open it — the cluster already has a king.Break the seal
question 1 of 3

A three-node Raft cluster is configured with identical fixed election timeouts (no randomization). The leader dies. What do you expect?