Persistence and Crash Recovery

A Raft node WILL crash. Power loss, OOM, kernel panic. The protocol's safety guarantees depend on three pieces of state surviving every crash:

1. current_term — the highest term the node has seen
2. voted_for — who got the vote in current_term (or null)
3. log[] — every entry the leader has shipped us

These three fields MUST be on stable storage (fsync) BEFORE responding to any RPC. Lose them and Raft's election safety property breaks: a node could vote twice in the same term, or "forget" entries it acknowledged.

Volatile state (commit_index, last_applied, leader_id, role): can be reconstructed from heartbeats after restart. Always restart as a FOLLOWER.

Why each is critical

• Lose current_term: you might step backward in time, accept a stale leader, overwrite committed entries.
• Lose voted_for: you might vote a second time in the same term — two leaders! Split-brain.
• Lose log entries: a leader's commit decision relied on your ack; losing it means losing a committed write.

Two writes per RPC

Both RequestVote and AppendEntries demand a durable write before reply:

• RequestVote: persist (current_term, voted_for) on YES.
• AppendEntries: persist log[] additions; persist current_term if updated.

Restart sequence

1. Read persistent fields from disk.
2. Set role = follower; leader_id = null.
3. Wait for heartbeat; if none after election_timeout, become candidate.

Optimization: batch fsyncs (group commit); use WAL with a header per record. etcd's bbolt and CockroachDB's Pebble do this.

Inline visualization

Disk:   [ term=5 | vote=c1 | log=[5:x=1, 5:y=2] ]
RAM:    [ role=leader | commit=2 | applied=2 ]
                |
              CRASH
                |
                v
RAM after restart:
        [ role=follower | commit=0 | applied=0 ]
Disk after restart (unchanged):
        [ term=5 | vote=c1 | log=[5:x=1, 5:y=2] ]

State machine reapplies the log entries up to leader's commit_index after rejoining.