HA & Replication

Single Prometheus is a SPOF. Modern setups:

Federation (Prometheus → Prometheus):

• Central Prom scrapes summary metrics from regional Proms.
• Hierarchical aggregation.

Thanos:

• Sidecar uploads Prom blocks to S3.
• Query layer reads from S3 + live Proms.
• Compactor downsamples + drops old blocks.
• Global view + long-term storage.

Cortex / Grafana Mimir:

• Push-based, multi-tenant TSDB.
• Distributors → ingesters (in-memory, replicated) → store (S3) via compactor.
• HA: each metric written to 3 ingesters; quorum of 2 for reads.
• Used by Grafana Cloud, AWS Managed Prometheus.

VictoriaMetrics Cluster:

• vminsert → vmstorage shards → vmselect.
• Linearly scalable to billions of series.

InfluxDB Enterprise / Cloud:

• Multi-node clustering with replication factor.
• TSI (Time-Series Index) shards by series.

Replication challenges:

• Two scraper Proms scraping same target produce slightly different timestamps. Dedup at query.
• WAL replication for in-flight writes.
• Cross-DC replication: async, eventual consistency.

Backup:

• TSDB blocks are immutable → trivial to S3.
• WAL needs careful replay on restore.
• Snapshots: filesystem-level (LVM, ZFS) or app-level.

Sharding:

• By time: each shard = a time window. Easy retention.
• By series: each shard = subset of series IDs (consistent hash). Even load.
• Hybrid: most production systems shard by both.

Tenant isolation in multi-tenant TSDBs:

• Per-tenant cardinality limits.
• Per-tenant query rate limits.
• Per-tenant retention.