Read

Log Compaction

Kafka's default retention policy deletes old segments after a time or size limit. That's fine for event streams, but it's wrong for data that represents current state — think of a topic that mirrors a key-value table, where each record is (key, value) and a consumer wants to rebuild the latest value for every key by replaying the log. If you deleted records by age, you might lose the only write for a key that hasn't been touched in months, even though it's still the "current" value.

Log compaction solves this by guaranteeing something different from time-based retention: instead of promising "keep everything for N days," it promises "keep at least the last known value for every key, forever." Kafka runs a background cleaner thread per partition that rewrites segments, dropping any record whose key has since been overwritten by a later record.

The rules

1. Only the latest write per key survives. If key=a is written at offset 5 and again at offset 40, the record at offset 5 is eligible for removal — offset 40 is now the only truth for a.
2. Order is preserved. Compaction never reorders the surviving records; it only removes the shadowed ones. A consumer replaying a compacted partition still sees records in offset order.
3. Tombstones delete keys. A record with a null value is a tombstone — it marks the key as deleted. After compaction, a tombstoned key produces no surviving record at all (the tombstone itself is also eventually removed, after a configurable delay long enough for consumers to see it).

Working through it

To compact a partition, you need to know, for every key, the offset of its last write. Then walk the log once more, keeping only those records whose offset matches that "latest" table — and among those, dropping the ones that turned out to be tombstones.

Because survivors is built by iterating in original order and only keeping items whose offset already equals the "final" offset, it comes out already sorted by offset — no extra sort needed, though sorting again is a harmless safety net.

Edge cases worth tracing by hand

• A key that is deleted and then re-written (DEL a followed later by PUT a 9) survives with the new value — the tombstone is shadowed just like any other stale write.
• A key written once and never touched again always survives.
• An input consisting only of tombstones compacts to nothing.
• Compaction is idempotent: running it twice in a row produces the same result as running it once, since the second pass has nothing left to shadow.

This is exactly the mechanism Kafka uses under the hood for things like consumer-offset topics (__consumer_offsets) and KTable changelog topics in Kafka Streams — both need "latest value per key forever," not "everything for 7 days."