Monitoring & Operations

Critical metrics:

Throughput:

• Bytes in/out per topic.
• Messages per second.
• Per-broker network usage.

Latency:

• Producer P99 latency: time from send → ack.
• Consumer lag: end_offset - committed_offset (per partition).

Resource:

• Disk usage per broker.
• Heap usage (JVM).
• File handles open.
• Network bandwidth.

Health:

• Under-replicated partitions (URPs).
• Offline partitions.
• Failed broker count.

Tools:

• Kafka itself exposes JMX metrics.
• Prometheus + Kafka exporters.
• Confluent Control Center (commercial).
• Kowl, AKHQ (open-source UIs).

Common operational issues:

Slow consumer:

• Lag grows.
• Investigate: processing speed, downstream system slowness, partition imbalance.

Hot partition:

• One partition has 10x traffic.
• Check key distribution.
• Mitigate: change keying scheme, or increase parallelism.

Disk full:

• Retention not aggressive enough.
• Add brokers, decrease retention, or delete old topics.

ISR shrinking:

• Followers can't keep up.
• Network or disk I/O bottleneck.
• Add brokers, scale up hardware.

Rebalance storms:

• Consumers churn (rejoin frequently).
• Causes: long processing exceeding session.timeout, OOMs.
• Static membership helps.

Schema changes:

• Producer publishes new schema; old consumers can't parse.
• Use Avro/Protobuf with Schema Registry.
• Forward/backward compatible schema evolution rules.

Capacity planning:

• Throughput × retention × replication factor = disk needed.
• Network: throughput × replication.
• Reserve 30-50% headroom.

Backups:

• MirrorMaker 2 to backup cluster.
• Snapshot topics to S3 (Confluent Cloud, custom tools).
• Restore: replay from snapshot.