Consumer Groups

A consumer group is a set of consumers that share the work of consuming a topic.

Topic: 4 partitions
Group: 2 consumers
Result: each consumer reads 2 partitions

Each partition is consumed by exactly ONE consumer in the group at a time.

If consumers > partitions: some are idle. If consumers < partitions: some consumers handle multiple.

Rebalance triggers:

• Consumer joins.
• Consumer leaves (graceful or crash).
• Partition added.
• Subscription changes.

During rebalance:

1. Group leader (a consumer) computes new assignment.
2. All consumers stop processing.
3. New assignment distributed.
4. Resume.

Rebalance cost:

• Pause processing.
• New consumer starts with last committed offset (may reprocess recent batch).
• Heavy if frequent.

Modern: incremental rebalance — minimize disruption.

Static membership (Kafka 2.3+):

• Set group.instance.id explicitly.
• Quick reconnect doesn't trigger rebalance.

Multiple groups on same topic:

• Each group has independent offset tracking.
• "Service A" group + "Service B" group both consume same events.
• Common pattern: log-as-database.

Group coordinator:

• One broker is coordinator for each group.
• Manages membership + offset commits.
• Selected by hash(group_id) % brokers.

kafka-consumer-groups.sh --describe --group my-group --bootstrap-server kafka1:9092

Shows partitions, current offsets, log-end, lag, consumer ID.

Stuck consumer signs:

• Lag growing.
• Same consumer ID for too long (no rebalance happening but processing lagging).
• Consumer logs show errors.