Log Compaction: Tombstones, Key-Latest Semantics, and the Cleaner

Most Kafka topics use delete cleanup — segments older than retention.ms get unlinked. Some topics need a different shape: only the latest value per key has to survive forever. That's log compaction, and it's what powers __consumer_offsets, Kafka Streams' state stores, and any "current state" topic.

The Idea

A compacted topic is still an append-only log, but the cleaner is allowed to delete old records as long as the key has been overwritten by a newer one. After compaction:

• For every key that has ever been written, the most recent value is retained.
• A record with a null value is a tombstone: it means "delete this key". After tombstones have been retained for delete.retention.ms, the cleaner is allowed to drop both the tombstone and the key entirely.

Offsets are preserved — compaction never renumbers records. A consumer that reads from the start sees holes (skipped offsets) but never sees a stale value for a key that has been overwritten.

Why It Matters

__consumer_offsets is a compacted topic. Imagine if it weren't: every offset commit would be retained forever, and the broker would store TBs of historical commit positions. With compaction, the topic is effectively a key-value store: (group, topic, partition) -> latest committed offset.

Kafka Streams uses the same pattern: every state-store update is published to a compacted changelog topic, so a crashed instance can rebuild its state by reading the changelog from the beginning.

How the Cleaner Works

There's a background thread per broker called the log cleaner. For each compacted partition it:

1. Scans the older "tail" of the log and builds an in-memory map key -> latest_offset_in_tail.
2. Copies the tail forward, dropping any record whose offset is less than the latest offset for its key.
3. Tombstones older than delete.retention.ms are dropped during this copy.

The head (the actively written segment) is never compacted — that would race with producers. The cleaner only touches sealed segments.

Compaction is a throughput-vs-disk tradeoff: more frequent cleaning gives a tighter view of "current state" but burns more CPU and I/O. The min.cleanable.dirty.ratio setting (default 0.5) decides when a partition is dirty enough to clean.

The Two-Step Cleanup Policies

cleanup.policy can be:

• delete — classic time- or size-based retention. Drop whole segments older than retention.ms.
• compact — only the latest value per key survives.
• compact,delete — both: compact for key-latest, and additionally drop the whole log older than retention.ms. Useful for stores that you want to bound in time and size.

What This Lesson Asks

The exercise is to implement compaction's user-visible behaviour. Given a stream of (key, value) writes interleaved with tombstones, you'll output the survivors after a "compact" pass — which is exactly what a consumer of a freshly-compacted partition would see if it read from offset 0.