Why Compaction? Read Amplification & Space Amplification

Every memtable flush creates a new SSTable. Without compaction, the number of SSTables grows without bound. Two things go wrong:

1. Read amplification. A point lookup must check every SSTable that could contain the key. With 1000 L0 SSTables, that's 1000 bloom-filter checks and potentially many disk reads per query.
2. Space amplification. A key updated 100 times exists in 100 SSTables until something deletes the old versions. The on-disk footprint balloons.

Compaction merges multiple SSTables into fewer, larger SSTables, discarding obsolete versions in the process.

The Three Amplifications

Compaction trades write amplification (paying to rewrite data) for lower read and space amplification. Different strategies put this knob in different places.

Universal Truth: There's No Free Lunch

You can have:

• Low read amplification (few SSTables to check), at the cost of high write amplification (constant rewriting).
• Low write amplification (write once, leave alone), at the cost of high read amplification (many SSTables).
• Low space amplification (eagerly drop old versions), at the cost of high write amplification (constant merging).

The two main families — size-tiered (Cassandra default) and leveled (LevelDB / RocksDB default) — sit at different points on this triangle. Universal compaction (RocksDB option) is a third style that's often best for write-heavy workloads.

Compaction Operates In The Background

A compaction job:

1. Picks a set of input SSTables (the policy).
2. Reads them as sorted iterators.
3. Merges them via a k-way merge (Ch 6).
4. Drops obsolete versions (older than min-snapshot, with a newer version present).
5. Writes one or more new output SSTables.
6. Atomically updates the MANIFEST to swap inputs for outputs.
7. Deletes the input files (after no reader pins them).

The whole process is sequential reads + sequential writes — no random I/O. That's why LSM compaction is bandwidth-bound, not seek-bound.

Write Amplification Math

If each compaction rewrites the average byte W times across all levels, write amplification = W + 1 (the +1 is the initial WAL write). For leveled compaction with 7 levels and a 10x size ratio per level, each byte gets rewritten ~7 times. Plus the WAL: WA ≈ 8x.

For size-tiered, WA is typically 3-5x (each byte is rewritten only when its tier reaches the merge threshold). Lower WA = higher write throughput at the cost of more SSTables = higher read/space amp.

Why You Care

Compaction is the dominant operational concern in any LSM-based database. Get it wrong and you face:

• Write stalls (compaction can't keep up; writes are throttled or blocked)
• Disk full (space amplification spikes)
• Read latency spikes (too many files to merge)
• I/O storms (compaction soaks bandwidth; user traffic suffers)

Cassandra's reputation for "GC pauses" and "STW compactions" comes from compaction pressure, not the JVM.