Checkpointing and Log Truncation

The log can't grow forever

If every write ever made stays in the WAL, two problems compound over time: the log file grows without bound, and recovery after a crash has to replay the entire history of the database from the very first write, which gets slower the longer the database has been running. Neither is acceptable for a real system. The fix is checkpointing: periodically, take everything the log describes, make sure it's durably reflected in the actual data pages, and then treat everything up to that point as no longer needed in the log.

What a checkpoint actually does

1. Flush dirty pages. "Dirty" means a page has been modified in the in-memory buffer pool (from the Page Cache lesson) but the modification hasn't been written to the on-disk copy yet. A checkpoint writes every dirty page out, so the durable, on-disk state now matches everything the WAL has recorded up to this point.
2. Record how many pages were flushed, and report it: CHECKPOINT outputs OK <pages_flushed>.
3. Truncate the log up to the checkpoint — once you're certain every operation up to sequence number N is safely reflected in the on-disk pages, log entries with sequence number ≤ N are no longer needed for crash recovery, because recovery's job is only to reproduce what the pages should contain, and the pages already correctly contain it.

function checkpoint():
    dirty = buffer_pool.dirty_pages()
    for page in dirty:
        write_to_disk(page)
        page.mark_clean()
    last_checkpoint_seq = wal.latest_sequence_number()
    wal.truncate_up_to(last_checkpoint_seq)
    return OK, len(dirty)

Why this changes recovery

Without checkpointing, RECOVER (previous lesson) replays the WAL from sequence number 0 every time — correct, but wasteful once the log is large. With checkpointing, recovery only needs to replay entries after the last checkpoint, because everything before it is guaranteed to already be durably on disk. This is the actual payoff: checkpointing bounds both the log's size and recovery time, at the cost of the (occasional, batched) work of flushing dirty pages.

WAL SIZE in this exercise reports the number of log entries currently present — which should drop after a CHECKPOINT truncates old entries, and this is a good way to verify your truncation logic actually removed entries rather than just recording a checkpoint marker without shrinking anything.

The ordering constraint that makes this safe

A checkpoint is only valid if the pages are flushed before the corresponding log entries are discarded — never the other way around. If you truncated the log first and then a crash happened mid-flush, you'd have neither the (incompletely written) pages nor the log entries needed to redo the rest — silent, unrecoverable data loss. So the write order must always be: flush pages to disk → confirm the flush succeeded → only then truncate the log. (Real systems handle the case where a crash happens during the checkpoint itself with extra bookkeeping — e.g., ARIES's fuzzy checkpoints — but for this exercise, treating CHECKPOINT as an atomic step, done in that order, is the right model.)

Edge cases

• Checkpoint with no dirty pages — should still succeed, reporting OK 0, and (if there's nothing new since the last checkpoint) may leave the log unchanged or empty, depending on whether there were any entries to truncate.
• Checkpoint truncating entries that recovery would need if a crash happened during the checkpoint — conceptually, make sure your truncation only ever removes entries whose pages are confirmed flushed, never entries for writes still in flight.
• Multiple checkpoints in a row — the second one should mostly be a no-op (nothing new to flush) if no writes happened in between.
• WAL SIZE immediately after CHECKPOINT should reflect only entries logged after the checkpoint, not the pre-checkpoint history.

Checkpointing is the piece that makes the WAL sustainable for a long-running database rather than a debugging tool that only works for short REPL sessions — real systems checkpoint on a timer or once the log exceeds some size threshold, precisely to keep recovery time and log size both bounded.