Partitions

A topic is a logical name. Internally, a topic is split into partitions: each is an independent log.

Why partition?

• Parallelism: multiple consumers can read different partitions simultaneously.
• Scalability: spread storage across brokers.
• Ordering: messages within a partition are ordered; across partitions, no global order.

Partition assignment:

• Producer picks partition for each message:
  • Specified directly.
  • Hash of key (default): same key → same partition. Preserves per-key order.
  • Round-robin if no key.

Cardinality choice:

• More partitions: more parallelism but more overhead (file handles, offset tracking).
• Default 6-12 per topic; high-throughput topics: 50-100.

Adding partitions:

• Possible but breaks key→partition mapping.
• Old messages stay in old partitions; new keys may hash to new partitions.
• Apps depending on per-key ordering must handle.

Brokers:

• Each broker hosts a subset of partitions.
• Adding brokers: trigger rebalance; partitions move.
• Cluster of 3-100 brokers typical.

ZooKeeper / KRaft:

• Old: ZooKeeper holds cluster metadata + leader election.
• New (Kafka 3.3+): KRaft (Kafka Raft) — built-in consensus, no ZK.

Operations:

• Create topic: kafka-topics.sh --create --partitions N --replication-factor R.
• Describe topic: list partitions + leaders + replicas.

Partition keys:

• Must be stable for ordering guarantees.
• Hot key: one partition gets all traffic. Mitigate via salting (key + random suffix mod K).