Chunks & Compression

Storage organizes logs into chunks: append-only blocks of N entries (or T seconds) for one stream.

Stream {service=api,env=prod}:
  chunk_001 [12:00 .. 12:15] 8000 entries (compressed)
  chunk_002 [12:15 .. 12:30] 7500 entries (compressed)
  ...

Compression: gzip / snappy / zstd. Logs compress 5-10x because of repetition (timestamps, levels, common substrings).

Snappy is fast (faster than gzip) but lower ratio. Zstd is the modern default — close to gzip ratio at snappy speed.

Why chunks not individual entries?

• Compression efficiency: per-block compression vs per-line.
• Index granularity: index = (stream_id, time_range, chunk_offset). Tiny.
• Object store friendliness: S3 charges per request; read 1 chunk vs 8000 entries.
• Immutability: closed chunks are read-only, easy to cache.

Chunk lifecycle:

1. Active (in-memory): receiving new entries.
2. Closed (sealed when 8000 entries or 15 min): compressed, written to object store, removed from memory.
3. Indexed: chunk metadata (stream_id, time range, file path) added to index.
4. Cold (after retention period): moved to cheaper storage (Glacier) or deleted.

For our system, each chunk = a gzipped JSON file {stream_id}/{ts_start}.gz with one log entry per line.