Time Partitioning

TSDB storage shards data by time. Each shard = a fixed time range.

Prometheus block layout:

data/
  01F.../    # 2h block
    chunks/
    index/
    meta.json
  01G.../    # next 2h block
    ...
  wal/       # active in-memory data

Each block holds 2h of data (configurable). After 2h, the WAL data flushes to a new block. Old blocks are immutable.

Why time partition?

• Pruning: query "last 1h" reads 1 block. Query "yesterday" reads ~12 blocks.
• Retention: drop old blocks = drop directory. O(1) deletion.
• Compaction: merge several 2h blocks into a 24h block; less metadata.
• Backup: each block self-contained; tar + S3.

Compaction levels (Prometheus):

• L1: 2h blocks (raw flushes from WAL).
• L2: 1d blocks (merged from 12 × 2h).
• L3: 1w blocks (merged from 7 × 1d).

Each level reduces metadata + improves compression (more data per block).

TimescaleDB hypertables:

• Postgres tables partitioned by time.
• One partition per chunk (e.g., 1 day).
• Optional space-partitioning by hash of host.
• Old chunks compressed to columnar format automatically.

Cross-block queries:

• Query "last 7 days" merges results from 1 daily block + 7 hourly blocks (or 1 weekly + ...).
• Sorted output via k-way merge per series.

Retention policies:

• Drop blocks beyond retention (e.g., 30 days).
• Per-metric retention possible (debug logs: 1 day; audit: 1 year).
• Tiered: recent in fast SSD, old in S3.