Consumers: Fetch, Decode and Lag

After this lesson your consumer takes the bytes a broker returns for a fetch, checks them, turns them back into records starting exactly at the offset it asked for, never goes past the high watermark, and reports its position and its lag.

Consumers pull

A broker never pushes. A consumer sends a Fetch request: "partition P, starting at offset N, at most M bytes". The broker answers with raw bytes from its log: whole record batches, the same bytes the producer wrote (plus the offsets the broker assigned). It does not decode or rewrite them, which is what lets it hand file contents straight to the network.

Two rules follow from "whole batches":

• A batch cannot be split on the broker. If offset N is in the middle of a batch, the response starts at the start of that batch, and the consumer throws away the records below N.
• The response may end in the middle of a batch (the byte limit M cuts it off). A consumer ignores a truncated final batch and fetches it again next time.

The high watermark

Each partition has a high watermark (HW): the offset below which every in-sync replica has the data. A consumer may only see records with offset below the HW, because anything above it could still be lost if the leader fails. The offset of the next record to be written is the log end offset; the HW trails it.

offsets:       0  1  2  3  4  5  6  7
                           ^        ^
                      HW = 5       log end = 8
consumer sees 0..4; records 5..7 are written but not yet visible

Position and lag

The consumer's position is the offset of the next record it will ask for: the last offset it received plus one (or the fetch offset when nothing came back). Lag is how far behind it is: HW - position, never below zero. Lag that grows means the consumer is slower than the producers, which is why it is the metric to alert on.

Decoding a record batch

Parse the 61-byte header first (the layout is the one you wrote in the producer lesson), then recordsCount records. Before trusting any of it, recompute CRC-32C over the bytes from attributes to the end of the batch and compare it with the stored crc; a mismatch means the data is corrupt and nothing in the batch may be used. The offset of a record is baseOffset + offsetDelta. After log compaction the deltas have gaps; the batch header's lastOffsetDelta still covers the whole original range.

Varints are zigzag: read bytes while the top bit is set, collecting 7 bits each, low group first, then undo the zigzag: (v >> 1) XOR -(v AND 1). A key or value length of -1 means null.

Worked trace, one record (different from your exercise):

1c 00 d8 04 02 04 6b 31 0a 68 65 6c 6c 6f 00

1c          length: zigzag 0x1c = 28 -> 14 bytes follow
00          attributes
d8 04       timestamp delta: (0x58) + (4 << 7) = 600 -> zigzag -> 300 ms
02          offset delta: 2 -> 1
04          key length: 4 -> 2
6b 31       key "k1"
0a          value length: 10 -> 5
68 65 6c 6c 6f   value "hello"
00          header count 0

In a batch with baseOffset 40 this record has offset 41.

What to skip and where to stop

Given a log segment's bytes and a fetch offset, a consumer-side decoder in this exercise follows the rules the real protocol implies:

1. Ignore a final batch that is cut off.
2. Skip batches whose last offset (baseOffset + lastOffsetDelta) is below the fetch offset: the broker would not have sent them, so they are not even checksummed.
3. Stop at the first batch whose base offset is at or above the HW.
4. For every other batch verify the CRC, then deliver records with fetch <= offset < HW.

Common mistakes

• Delivering the whole first batch. Records below the fetch offset belong to the broker's batch granularity, not to the consumer.
• Reading past the HW just because the bytes are there.
• Skipping the CRC and silently handing corrupt records to the application.
• Unsigned varints. The key length -1 for a null key is byte 01 after zigzag.
• Counting position by records. After a gap in offsets the position is last offset + 1, not old position + n.

Your exercise

The starter parses the input and prints the final line. Implement the CRC, the varint reader and the batch walk. The most likely mistake is forgetting that a fetch can start in the middle of a batch.