What Kafka Solves

Apache Kafka is a distributed event streaming platform. Think: a database with one table called "log" that you can only append to.

Use cases:

• Event streaming: capture every user action as an event.
• Microservice communication: services produce/consume events instead of REST.
• Data integration: source-of-truth log; downstream systems materialize their views.
• Log aggregation: replace tail -f from many machines.
• Metrics pipeline: collect → process → store.

Why a log?

• Append-only: simple, fast, sequential I/O.
• Ordered: consumers see events in order.
• Replayable: catch up by re-reading from offset.
• Multi-consumer: each consumer has independent position.

Real systems:

• Apache Kafka: original, JVM, biggest install base.
• Redpanda: C++ rewrite, Kafka API-compatible, faster.
• AWS Kinesis: managed.
• Pulsar: Apache, multi-tenant, BookKeeper-backed.
• NATS JetStream: lightweight alternative.

Architecture:

Producer → Broker (cluster) → Consumer
              ↓
            Topic (logical channel)
              ↓
            Partition 0, 1, 2, ...
              ↓
            Replicas (leader + followers)

Topics = named logs. Partitioned for parallelism. Replicated for HA.

Consumers can be:

• Single (one consumer reads all messages).
• Group (multiple consumers split partitions among themselves).

Throughput: 1M+ messages/sec per broker on commodity hardware.

We'll build a Kafka-style log: append, fetch by offset, partition, replicate, consumer offsets.