Consumers

A consumer reads messages from a topic.

Pull-based (Kafka):

• Consumer asks broker: "give me messages from offset X."
• Broker returns batch (configurable size).
• Consumer commits its progress (or doesn't, for replay).

Auto-commit vs manual:

• Auto: every N seconds, current offset committed.
• Manual: app calls commit() when ready (e.g., after writing to DB).
• Manual gives stronger delivery guarantees.

At-least-once vs exactly-once:

• At-least-once: process before commit; crash → reprocess on restart. Idempotent ops OK.
• Exactly-once: transactional commits. Process + write atomically. Kafka transactions support.
• At-most-once: commit before process; crash → message lost. Avoid usually.

Offset commits:

• Stored in special __consumer_offsets topic.
• Per (group_id, partition) → offset.
• Commit cadence: throughput vs replay cost on crash.

Backpressure:

• Consumer slow → lag accumulates.
• Lag = end_offset - committed_offset.
• Monitor: kafka-consumer-groups.sh --describe --group my-group.

Long-running message processing:

• Default: consumer must heartbeat periodically.
• If processing > heartbeat timeout (5 min default), broker thinks consumer dead → rebalance.
• Solution: separate worker threads from poll thread.