Consumer Groups and Range Assignment

After this lesson your program does the job of a consumer group leader: it reads what every member says it wants to consume, and decides which member owns which partition, with the same answer Kafka's default range assignor gives.

What a group is

A consumer group is a set of consumers that share the work of a set of topics under one group.id. The rule that makes it work: each partition is owned by exactly one member of the group at a time. Members of different groups are independent, each group has its own committed offsets, so two services can read the same topic without disturbing each other. A group with more members than partitions leaves the extra members idle.

One broker is the group's coordinator: it tracks membership by heartbeats and stores the group's committed offsets. When membership changes (a member joins, leaves, crashes and misses its session.timeout.ms, or a subscribed topic gains partitions) the group rebalances.

The join protocol, in bytes

During a rebalance every member sends a JoinGroup request. Along with its member id it sends protocol metadata: for the consumer protocol that is a ConsumerProtocolSubscription:

version     int16     0, 1, 2 ... (later versions append fields)
topics      array     int32 count, then each topic: int16 length + UTF-8 bytes
user_data   bytes     int32 length (-1 for none) + that many bytes
...         v1+ adds owned partitions, which this lesson ignores

The coordinator picks one member as leader and hands it everyone's metadata. The leader (a client, not the broker) runs the assignment algorithm and returns the result in SyncGroup. Because the algorithm runs in client code, it is a client-side strategy: partition.assignment.strategy.

Example: a member subscribed to ["a"], with no user data, sends

00 00             version 0
00 00 00 01       one topic
00 01 61          length 1, "a"
ff ff ff ff       user_data = null

The range assignor

For every topic, independently:

1. Take the members subscribed to this topic, sorted by member id (plain string order, so consumer-10 comes before consumer-2).
2. With n partitions and m such members, every member gets n div m consecutive partitions, and the first n mod m members get one more.
3. Member number i (counting from 0) starts at partition (n div m) * i + min(i, n mod m).

7 partitions, 3 members:  7 div 3 = 2, 7 mod 3 = 1
member 0: start 0, 3 partitions  -> 0 1 2
member 1: start 3, 2 partitions  -> 3 4
member 2: start 5, 2 partitions  -> 5 6

Topics do not know about each other. With 3 topics of 3 partitions and 2 members, member 0 gets the extra partition of every topic: 6 partitions against 3. This skew is the reason round-robin and sticky assignors exist.

Common mistakes

• Sorting members numerically or by arrival. The order is by member id as a string.
• Treating the metadata as text. It is binary: lengths are big-endian signed integers.
• Assigning a topic nobody has metadata for. The leader only knows partition counts for topics in the cluster metadata.
• Giving a member a topic it did not subscribe to. Range assignment is per topic, over that topic's subscribers only.
• Forgetting members with nothing assigned. They still exist in the group.

Your exercise

The starter reads the topics and members and prints the result. Implement the subscription decoder and the range assignment. The most likely mistake is computing one global assignment across topics instead of one per topic.