Downsampling & Retention

Raw data at 1-second resolution × 1 year × 1M series = petabytes. Solution: downsample old data.

Tiered resolution:

• Last 7 days: 1s resolution (raw).
• 7-30 days: 1m resolution (downsampled).
• 30-365 days: 1h resolution.

• 1 year: 1d resolution.

Each tier stores aggregations: avg, sum, min, max, count, p50, p95, p99 per bucket.

Downsampling:

Run as background job. Old block → multiple downsampled blocks per resolution.

Storage savings:

• Raw 1s = 86400 points/day.
• 1m downsample = 1440 points/day.
• 1h downsample = 24 points/day.
• 60x reduction per tier.

Query routing:

• "last 1h" → raw tier.
• "last 1y avg" → 1d tier.
• "last 30d p99" → 1h tier.
• Picks lowest-resolution tier that has the data.

Downsampling pitfalls:

• Percentiles don't average: avg(p99 of N buckets) != p99 of all data. Approximations via t-digest, HDR histogram.
• Counter resets: rate calculation across downsample boundaries needs care.
• Stale data: downsampling should be incremental — process new buckets, leave old ones alone.

Retention by metric:

• Metrics on critical paths: longer retention.
• Debug metrics: shorter.
• Configurable per metric/label.