Log Compaction

After this lesson your log has a cleaner: it rewrites the closed segments so that only the latest record for each key survives, keeps every surviving record at its original offset, and deletes keys with tombstones.

Two kinds of retention

Delete retention (the default) throws away whole old segments by age or size. That is right for events: nobody needs last month's page views. It is wrong for state. Take a topic that mirrors a table, one record per user keyed by user id. A user who has not changed for a year has exactly one record, and deleting it by age would erase that user.

cleanup.policy=compact keeps a different promise: for every key, the latest record is never removed. Older records for the same key are garbage. Used for __consumer_offsets (one record per group and partition), Kafka Streams changelog topics and Kafka Connect configuration.

What the cleaner may touch

Compaction is a background job (the log cleaner thread), and it never touches the active segment, the one still being written. It works on the sealed ones.

segments:  [ 0 .. 3 ]  [ 4 .. 6 ]  [ 7 .. ]   <- active, untouched
            \_______ cleaner works here _____/

The cleaner first scans the closed segments to build a map key -> offset of the latest record in the closed segments. Then it copies the closed segments, keeping a record only if its offset is the one in the map.

Two consequences that surprise people:

• Offsets are not renumbered. Survivors keep their original offsets, so the log now has gaps. A consumer asking for offset 5 when 5 was cleaned simply gets the next record that exists. Offsets are identities, not positions.
• A newer value in the active segment protects nothing. If a=1 sits in a closed segment and a=2 is still in the active one, the map built from the closed segments does not know about a=2, so a=1 survives this pass. After the next roll and pass it goes.

Tombstones

A record with a key and a null value is a tombstone: it means "delete this key". The cleaner treats it like any other record, so it replaces older records of the key. After that pass the tombstone is itself the only trace of the key, and it must stay for a while (delete.retention.ms, one day by default) so that a consumer that is rebuilding its state from the log sees the delete instead of an old value that happens to still be on disk. Once that time has passed, a later pass drops the tombstone too and the key is gone entirely.

A tombstone that is not the latest record for its key (a PUT came after) is just shadowed and goes like any stale record.

Worked trace

Different data from your exercise. Commands: PUT p 1, PUT q 1, ROLL, PUT p 2, DEL q, ROLL, PUT r 1, then COMPACT.

segment 0: 0 p=1   1 q=1                         (closed)
segment 2: 2 p=2   3 q=TOMBSTONE                 (closed)
segment 4: 4 r=1                                 (active)

map from closed segments:  p -> 2,  q -> 3
keep 2 p=2 and 3 q=TOMBSTONE; drop 0 and 1.

after:     segment 2: 2 p=2   3 q=TOMBSTONE      segment 4: 4 r=1 (active)

Segment 0 had nothing left, so it disappears. Segment 2 keeps its name even though it no longer starts at its base offset's first record. A later COMPACT PURGE (the tombstone is now old enough) would also drop 3 q=TOMBSTONE.

Common mistakes

• Compacting the active segment. The cleaner never rewrites a file that is still being appended to.
• Renumbering survivors. Offsets must not change; consumer positions and committed offsets refer to them.
• Dropping the tombstone on the first pass. It has to survive until the purge pass, or slow consumers miss the delete.
• Keeping a stale record because a newer one is in the active segment. Do exactly that, on purpose: only closed segments are inspected.

Your exercise

The starter reads the commands for you. Implement the log, the roll, the two kinds of pass and the dump. The most likely mistake is letting the active segment take part in the pass.