Monitoring and Operations

After this lesson your program reads the output of kafka-topics.sh --describe exactly as an operator would, and computes the four partition health numbers every Kafka dashboard is built around.

What an operator watches

A broker exposes hundreds of JMX metrics. A handful decide whether to page someone:

The first three are about durability and availability, and they are the ones that should wake a person. The fourth is about load balance: leaders concentrate on the brokers that stayed up, and the "preferred" replica (the first in the replica list) should lead again once it is back.

The source: kafka-topics.sh --describe

The tool prints a header line per topic, then one indented line per partition. Fields are Key: value pairs separated by tabs:

Topic: payments	TopicId: Q2JvT3xWQ3ewV4mS2nL0ZA	PartitionCount: 2	ReplicationFactor: 3	Configs: min.insync.replicas=2,cleanup.policy=delete
	Topic: payments	Partition: 0	Leader: 4	Replicas: 4,5,6	Isr: 4,5,6
	Topic: payments	Partition: 1	Leader: none	Replicas: 5,6,4	Isr: 

Details that bite:

• The partition line starts with a tab, so splitting on tabs gives an empty first piece. Skip pieces without a colon.
• Split each piece at the first colon only, then trim both sides. Config values can contain = and commas.
• An offline partition prints Leader: none (older versions print -1) and an Isr: with nothing after it.
• Newer versions append more fields (for example Elr:). Ignore keys you do not need.
• min.insync.replicas lives in the header's Configs, among other settings, and applies to every partition of that topic. If it is absent the default is 1.

Turning rows into flags

For one partition with R replicas and I in-sync replicas:

• OFFLINE if there is no leader.
• If there is a leader: URP if I < R; UNDER_MIN_ISR if I < min.insync.replicas; NOT_PREFERRED if the leader is not the first replica.

Why do URP, UNDER_MIN_ISR and NOT_PREFERRED need a leader? The real metrics are reported by the leader of each partition. A partition without a leader is not counted by any broker as under-replicated; it shows up only in OfflinePartitionsCount. Mirror that, or one dead partition would be counted twice.

Worked trace

Different data from your exercise. A topic with min.insync.replicas=2 (rf 3):

partition 0  Leader 4  Replicas 4,5,6  Isr 4,5,6     healthy
partition 1  Leader 5  Replicas 4,5,6  Isr 5,6       URP;  I=2 is not below min 2;  leader 5 is not replica 4: NOT_PREFERRED
partition 2  Leader 6  Replicas 4,5,6  Isr 6         URP, UNDER_MIN_ISR (1 < 2), NOT_PREFERRED
partition 3  Leader none                              OFFLINE only

Counters: URP 2, UnderMinIsr 1, Offline 1, Imbalance 2.

Common mistakes

• Counting an offline partition as under-replicated (its ISR is also smaller than its replica list).
• Comparing the ISR to the replication factor in the header instead of to that partition's replica list. A partition mid-reassignment can have more replicas than the factor.
• Splitting on ": " or on spaces. Tabs separate fields; values like Replicas: 4,5,6 and configs contain spaces or colons in other tools' output.
• Applying one topic's min.insync.replicas to another topic's partitions.

Your exercise

The starter reads the text and prints your result. Implement the parsing and the counters. The most likely mistake is mishandling the empty Isr: of an offline partition.