Production TSDB

Operational concerns at scale:

Cardinality control:

• Limit max series per tenant (10M typical).
• Drop high-cardinality labels at ingest (relabeling rules).
• Alert on cardinality spike (often = bug).
• "Walgreens example": every customer ID as label = millions of series.

Cardinality bombs:

• Bug deploys add request_id as label → series cardinality explodes.
• Dashboards lag or fail.
• TSDB OOMs.
• Defense: rate limit, label policies, automated alerts.

Storage cost:

• 1M series, 15s scrape, 30 days, ~30 bytes compressed = 5TB/month.
• Retention is the biggest cost lever. 90 days vs 30 days = 3x cost.

Query cost:

• "last 7 days p99 across all hosts" = scan many series × points.
• Pre-aggregation (recording rules) eliminates this on hot dashboards.

Recording rules (Prometheus):

- record: api:request_rate:5m
  expr: sum(rate(http_requests_total[5m])) by (service)

Run every 30s. Persist as a new series. Dashboards query the rule, fast.

Alerting:

• Alert manager evaluates expressions periodically.
• Pending → firing transitions.
• Inhibition + grouping for sane notifications.

Service discovery integration:

• K8s, Consul, EC2, file_sd.
• Auto-discovers new targets; drops old ones.

Federation patterns:

• Hub-spoke: regional clusters → global summary.
• Mesh: cross-cluster queries via Thanos query layer.

Common gotchas:

• Counter resets (process restart): rate function handles, but spikes possible.
• Clock skew: timestamps from different machines may not align.
• Late-arriving data: ignored or replayed; depends on system.

Don't roll your own. Use Prometheus + Thanos / Cortex / Mimir / VictoriaMetrics for self-hosted; Datadog / Grafana Cloud / AWS Timestream for managed.