Putting It Together: a Mini Broker

After this lesson you have one program that behaves like a small Kafka: it routes records by key to partition logs, assigns partitions to the members of a consumer group, serves their polls from committed offsets, reports lag, and tells you how much disk the whole thing will need.

How the pieces fit

Every earlier lesson built one part of this picture. Here they run together:

PRODUCE  key -> murmur2 -> partition -> append to that partition's log -> offset
JOIN     member joins -> rebalance -> range assignment of the topic's partitions
POLL     member -> its partitions -> from the GROUP's committed offset -> records
LAG      per partition: log end offset - committed offset

Three ideas carry the integration:

• Offsets are per partition, and every partition has its own log. orders-2@1 is the second record of partition 2, and nothing else is comparable to it.
• Committed offsets belong to the group, not to the member. That is what makes a rebalance cheap: when partition 3 moves from member A to member B, B simply continues at the group's committed offset for partition 3. Nothing is replayed and nothing is skipped.
• Different groups are independent. Each has its own assignment and its own committed offsets over the same logs.

A scenario: a topic has 4 partitions; one member joins and owns all four. After it polls, a second member joins; the assignment becomes 0,1 and 2,3 (range assignment over sorted ids); the second member's first poll starts at the committed offsets of partitions 2 and 3 as the first member left them.

Running it for real

What changes between this exercise and a production cluster is mostly scale and failure handling, which the earlier lessons covered one at a time:

• Replication factor 3 and min.insync.replicas=2 with acks=all: an acknowledged write survives the loss of one broker. A shrinking ISR stops writes rather than silently reduce safety.
• Idempotent producers (the default in current clients) so retries cannot duplicate; transactions where a pipeline must write several partitions and commit offsets atomically.
• Partition count chosen with growth in mind. Adding partitions later changes where keys land. A common rule of thumb: enough partitions to reach your target throughput divided by the per-partition throughput you measure, and at least as many as the largest consumer group you expect.
• Retention and compaction per topic: time or size retention for events, compaction for state.
• Monitoring: UnderReplicatedPartitions, OfflinePartitionsCount, UnderMinIsrPartitionCount and consumer lag, every one with an alert.

Sizing the disk

Capacity planning starts with one product:

bytes on disk = message size * messages per second * retention in seconds * replication factor

Worked example, different numbers from yours: 1,500 byte messages at 2,000 per second, kept 3 days, replication factor 3:

1500 * 2000 = 3,000,000 bytes per second
3,000,000 * (3 * 86400) = 777,600,000,000 bytes for one copy
* 3 replicas = 2,332,800,000,000 bytes  (about 2.3 TB across the cluster)

That is the floor. Real clusters add headroom (30 to 50 percent is common) for index files, compaction copies and uneven partitions, and they do the multiplication in 64-bit integers: this product overflows 32 bits at once.

Common mistakes

• Keeping committed offsets per member. After a rebalance the new owner restarts from 0, or from a stale value.
• Polling partitions in assignment-hash order. Go in ascending partition order, and stop reading when the budget is used up.
• Recomputing the assignment from join order. Members are sorted by id, whatever order they joined in.
• Letting LEAVE of an unknown member change anything.
• 32-bit sizing arithmetic.

Your exercise

The starter reads the commands and has the hash and the range assignor from your earlier work. Write the state and the six operations. The most likely mistake is the one named in the hint: where committed offsets live.