Putting It All Together

Time to assemble every mechanism you have built into one working replicated KV store — and to see how the pieces interlock into a single safety argument.

The safety chain

Each property you implemented exists to prop up the next:

• Election Safety — at most one leader per term. From: one vote per node per term + strict-majority wins.
• Log Matching — same index + same term ⇒ same command and same prefix. From: the AppendEntries consistency check.
• Leader Completeness — every committed entry is in every future leader's log. From: the up-to-date vote restriction + the Figure-8 rule (only current-term entries commit by counting).
• State Machine Safety — the payoff: no two nodes ever apply different commands at the same index. Your replicas can differ only in how far behind they are, never in content.

Break any link and the chain fails: clear voted_for on an equal term and Election Safety dies; skip the consistency check and Log Matching dies; count prior-term replicas toward commit and Leader Completeness dies. Every one of those bugs ends the same way — two state machines disagreeing about the same index.

Raft in the wild

etcd (all of Kubernetes' control-plane state), Consul, TiKV (Rust), CockroachDB (thousands of Raft groups, one per key range), Apache Ratis (Java), hashicorp/raft (Vault, Nomad). Production systems add performance layers that change none of the rules:

• PreVote — a would-be candidate first polls "would you vote for me?" without incrementing its term, so a node returning from a partition cannot term-bomb a healthy leader.
• Pipelining — send entry N+1 before N's ack returns.
• Batching — many client commands per AppendEntries and per fsync; throughput is fsync-bound without it.
• Leadership transfer, learners (non-voting catch-up members), and witnesses (vote-only members).

Then read the real thing: "In Search of an Understandable Consensus Algorithm" (Ongaro & Ousterhout, 2014) — 16 pages (18 in the extended version) and genuinely readable; raft.github.io collects visualizations and 100+ implementations.

The capstone model

Your simulation compresses the protocol honestly: TIMEOUT elects a node only if a majority of the FULL cluster is alive — dead nodes cast no votes, so a 5-node cluster with 3 crashed elects nobody. Each election starts a higher term. WRITE commits only when a majority is reachable. Committed state survives leader changes — the new leader's log already holds every committed entry; that is Leader Completeness doing its job. READ is linearizable, so it too requires a live leader with quorum.

Keep the two failure answers distinct:

• NO_LEADER — nobody currently leads: before any election, or after the leader crashed and no one has timed out yet.
• NO_QUORUM — a leader exists but cannot assemble a majority of live nodes. (A real minority-partitioned leader keeps appending but never commits — clients just time out; your simulation refuses up front.)

Your exercise

Build the mini Raft KV. The grader-caught mistake is conflating those two failures: WRITE x 5 before any TIMEOUT prints NO_LEADER, but after INIT 5, TIMEOUT 1, then crashing nodes 2, 3, and 4, WRITE x 1 prints NO_QUORUM. Committed data must survive re-election: WRITE x 5 → OK, then CRASH 1, TIMEOUT 2, and READ x still prints 5. A missing key on a healthy cluster prints NIL. And STATE formats exactly: an empty store is term=1 leader=1 kv=-; otherwise keys sort alphabetically — term=1 leader=1 kv=x=5,y=hi.