The Log: Append-Only Storage

Each Kafka partition is a sequence of immutable messages.

Offset:  0    1    2    3    4    5    6
Message: A    B    C    D    E    F    G
                              ^         ^
                              old       new

Each consumer maintains an offset. Reads are sequential from that offset.

On disk: segmented log files.

00000000000000000000.log    (offsets 0..1000)
00000000000000001001.log    (offsets 1001..2500)
00000000000000002501.log    (offsets 2501..current)

Each segment is ~1 GiB by default. Once full, sealed (read-only) and a new one opened.

Per segment, an index file maps offset → byte position:

.index file: [(offset_in_segment, byte_offset), ...]

Sparse index: not every offset, just every Nth. Fast lookup with bounded I/O.

Append:

1. Append message to active log file.
2. Update active index entry.
3. Flush to OS page cache (default).
4. fsync periodically (configurable).

Read by offset:

1. Find segment containing that offset (filename = base offset).
2. Look up index for nearest position.
3. Scan forward to find exact offset.

Retention:

• Time-based: delete segments older than N days.
• Size-based: cap total size; delete oldest.
• Compaction: keep only latest message per key (KV-style log).

Why fast?

• Sequential I/O matches modern disk strengths.
• Page cache amortizes reads.
• Zero-copy (sendfile) for sending to consumers — kernel passes data NIC-side without user copy.

Throughput: hundreds of MB/sec per disk on commodity hardware.