Putting It All Together

You've built every layer of a modern compressor:

Where to go from here:

• Arithmetic coding / range coding: encodes a stream as a single fractional number. Achieves entropy more closely than Huffman (which is integer-bits).
• Asymmetric Numeral Systems (ANS): modern alternative to range coding. Used in zstd, bzip3, JPEG XL. Simpler than range coding, similar performance.
• Burrows-Wheeler Transform: reorders bytes to make them clusterable. Used by bzip2.
• PPM (prediction by partial matching): model context (previous N bytes), encode current byte by its conditional probability.
• Neural compressors: language models (ByT5) achieve below-state-of-art compression on text — but slow.
• Custom dictionaries: brotli ships with a built-in dictionary of common web content. zstd lets you train custom dictionaries from sample data.

The compression-speed tradeoff is fundamental:

• lz4 / snappy: very fast (~500 MB/s); ratio ~50%
• gzip: medium (~150 MB/s encode); ratio ~33%
• zstd -3: similar speed to gzip but tighter
• zstd -22 / xz / brotli -11: very tight (~20%) but slow encode

The decoder is almost always 5-10x faster than the encoder. Asymmetric speed is intentional — for archives, you encode once and decode many times.