Retention & Compaction

Storage costs scale with log volume. Strategies:

Tiered storage:

• Hot (last 7 days): SSD, fast queries, expensive.
• Warm (8-30 days): HDD or object store, slower.
• Cold (30-365 days): S3 Glacier / Tape. Minute-to-hour retrieval.
• Drop after retention.

Compaction: small chunks merge into larger ones nightly.

• Reduces total file count → less metadata, faster lookups.
• Better compression ratio (more data per block).
• Requires reading + rewriting chunks → I/O cost.

Sampling / drop policies:

• DEBUG logs: drop after 1 day.
• INFO: drop after 7 days.
• ERROR/WARN: keep 30 days.

Configured via labels: retention_period set per {level="debug"}, etc.

Audit logs are different: must keep for years (compliance). Move them to a separate retention pipeline.

Right-sizing: most teams over-retain. 90% of debug logs aren't read after first day. Reducing retention is the biggest cost lever.