Compression Trade-offs

Compression saves I/O but costs CPU on read. Modern column stores layer multiple algorithms:

Zstd / Snappy / LZ4 / Gzip (block-level after encoding):

• Snappy: fastest, lowest ratio. Good for hot data.
• LZ4: similar to snappy, slightly better ratio.
• Zstd: modern default. Close to gzip ratio at snappy speed.
• Gzip: best ratio, slowest.

Combined with encodings:

1. RLE / dict / bit-pack reduces logical size.
2. Then snappy/zstd reduces physical size further.

Why stack? Each encoding exploits different structure:

• RLE: runs.
• Dict: low cardinality.
• Snappy: byte-level patterns.

Compression ratio expectations:

• Sorted timestamps with delta + bit-pack + zstd: 50-100x.
• Low-cardinality strings (countries, status): 50-200x dict.
• Random UUIDs: ~1-2x (essentially incompressible).
• Floats: 2-5x typically; Gorilla's float compression for time-series gets 5-10x.

Decompression cost:

• Snappy: ~3 GB/s per core.
• Zstd: ~1.5 GB/s per core.
• Gzip: ~300 MB/s per core.

With CPU 100x faster than disk, decompression is rarely the bottleneck.

Encoding-aware operations: skip decompress entirely.

• Equality on dict-encoded string: lookup target in dict, scan integer indices.
• COUNT on RLE: sum the counts.
• Sum on FOR: sum(deltas) + min * count.

Most modern engines (DuckDB, ClickHouse) implement these for hot paths.

Trade-offs in choice:

• Real-time analytics: lighter compression, faster scans.
• Cold archival: heavy compression, slower decompression OK.
• ML training data: optimize for sequential read throughput.