The Log: Segments and the Sparse Index

After this lesson your log rolls into new segment files when one fills up, builds the same sparse index a real broker builds, and answers "where is offset N?" with a file name, a starting position and the number of records it has to scan.

One partition, many files

A partition's log is not one big file. It is a directory of segments. Each segment is a pair of files named after the base offset, the offset of the first record it holds, left-padded with zeros to 20 digits:

orders-0/
  00000000000000000000.log        records 0..3      (sealed)
  00000000000000000000.index
  00000000000000000000.timeindex
  00000000000000000004.log        records 4..      (active: the only one being written)
  00000000000000000004.index
  00000000000000000004.timeindex

Only the last segment, the active one, is ever appended to. When the next batch would not fit (log.segment.bytes, 1 GiB by default), the broker rolls: it seals the active segment and starts a new one whose base offset is the next offset to be assigned. Sealed files never change again, which is what makes retention cheap (delete whole files) and reads safe (nobody is rewriting what you read).

The rolling rule: if the active segment already holds bytes and size + incoming > segment_bytes, roll first. A batch that is bigger than a whole segment still lands in an empty one, alone; an empty segment never rolls.

Finding a record: two steps

Given offset N:

1. Pick the segment: the one with the greatest base offset that is <= N. File names are sorted, so a binary search over base offsets works.
2. Pick the position inside it: use that segment's .index.

A full index would need one entry per record. Kafka's is sparse: one entry every log.index.interval.bytes (4096 by default). Each entry is 8 bytes: a 4-byte offset relative to the segment base (so it fits in 32 bits no matter how large the real offset is) and a 4-byte byte position in the .log file.

rule applied while appending a batch at byte position P:
    if bytes_since_last_entry > index_interval:
        add entry (offset - base, P);  bytes_since_last_entry = 0
    write the batch;  bytes_since_last_entry += batch_size

The comparison is strictly greater, and the entry is for the batch about to be written. The first record of a segment therefore never gets an entry.

To look up N: find the index entry with the greatest relative offset <= N - base (or start at position 0 if there is none), seek there, and scan forward record by record until you reach N. The scan is short by construction: it covers at most one index interval of bytes.

Worked trace

Different numbers from your exercise: SEGMENT_BYTES 90, INDEX_INTERVAL 30, record sizes 20, 20, 20, 20, 35, 10.

offset 0  size 20  segment 0  pos  0   since=0   no entry       since becomes 20
offset 1  size 20  segment 0  pos 20   since=20  no entry       since becomes 40
offset 2  size 20  segment 0  pos 40   since=40 > 30  ENTRY (2,40)   since 0 -> 20
offset 3  size 20  segment 0  pos 60   since=20  no entry       segment holds 80 bytes
offset 4  size 35  80 + 35 = 115 > 90  ROLL: new segment, base 4
                   segment 4  pos  0   since=0   no entry       since becomes 35
offset 5  size 10  segment 4  pos 35   since=35 > 30  ENTRY (1,35)

Lookup of offset 3: segment 00000000000000000000.log, relative offset 3, the best entry is (2,40), so start at position 40 and scan 2 records (offsets 2 and 3) to reach position 60. Lookup of offset 4: segment 4, no entry applies, start at position 0, scan 1 record.

Common mistakes

• Using >= for the index rule. With interval 30, a gap of exactly 30 bytes must not create an entry.
• Absolute offsets in the index. Entries are relative to the segment base. Entry (2,40) in segment 4 means offset 6.
• Rolling an empty segment. A single 200-byte batch with segment_bytes 100 is stored alone, not rejected and not preceded by an empty segment.
• Forgetting to reset the byte counter on roll. A new segment starts counting bytes since the last entry from zero.

Your exercise

The starter reads the configuration lines and routes each command. Implement the segment bookkeeping and the lookup. The most likely mistake is the comparison in the index rule, so trace the 30-byte-gap case by hand first.