Wire Framing: How Kafka Speaks on the Wire

A Kafka broker doesn't know what a "message" is until it sees how many bytes belong to it. Every request, every response, and every record-batch starts with the same recipe: declare the length, then send the payload. Get this wrong and the broker sees garbage; get it right and you can read brokers from twenty years ago.

The Outer Frame

Every Kafka API call over TCP is wrapped in a tiny outer frame:

+--------+-----------------------------+
| len(4) |   message bytes (len bytes) |
+--------+-----------------------------+

len is a big-endian 4-byte signed integer. Read it, then read exactly that many bytes, then start over. That's the entire framing protocol for the socket.

This is the same trick used by gRPC (HTTP/2 DATA frame length), Postgres (Int32 message length), MySQL (3-byte length + 1-byte seq), and Redis RESP ($<len>\r\n<bytes>\r\n for bulk strings). It's the most boring protocol decision in distributed systems — and the most universally correct one.

Why Length Prefixes, Not Delimiters?

Newline-delimited works for text (HTTP, SMTP) until the body contains a newline. Then you need quoting, escaping, transfer-encoding. Length prefixes side-step all of it: you say "here come N bytes" and the receiver doesn't have to interpret a single byte until it has all N.

In particular Kafka records are binary — they contain arbitrary user payloads (often compressed and CRC'd). Newlines would be a non-starter.

The Record Batch Inside

Inside a Produce request the payload is a RecordBatch, which has its own length prefix and CRC. Simplified layout:

+-------------+----------+-----------+--------+-----+--------+
| baseOffset  | length   | leaderEpoch | magic | CRC | attrs  |
| (8 bytes)   | (4 bytes)| (4 bytes)   | (1)   | (4) | (2)    |
+-------------+----------+-----------+--------+-----+--------+
| lastOffsetDelta | firstTimestamp | maxTimestamp | producerId |
| (4)             | (8)            | (8)          | (8)        |
+-----------------+----------------+--------------+------------+
| producerEpoch (2) | baseSequence (4) | records (length-prefixed) |
+-------------------+------------------+---------------------------+

A few things to notice:

1. length is the length of the rest of the batch (everything after length field). That's why a corrupted length field will make the whole batch un-parseable — and that's why the CRC covers everything from attrs onward, so corruption is detected before any record is interpreted.

2. producerId + producerEpoch + baseSequence are the three fields that power idempotent producer dedup. The broker remembers the last (producerId, partition) -> baseSequence it accepted; a retry sending the same baseSequence is a no-op.

3. magic is a version byte. Magic 2 is the current format (since Kafka 0.11); magic 0 and 1 are the pre-RecordBatch formats. New clients still have to know about old magic numbers to read old segments.

4. The records payload uses varint (zig-zag) encoded length fields inside the batch, because most record sizes fit in 1-2 bytes and Kafka deals with billions of records.

Reading a Stream

The receive loop on a broker is essentially:

loop:
    n = read_be_int32(socket)
    if n < 0 or n > max_request_size: drop connection
    bytes = read_exactly(socket, n)
    handle(bytes)

The read_exactly is critical: TCP is byte-oriented. A single recv() can return fewer bytes than asked, and a single recv() can fuse two messages together. Always loop until you have exactly n bytes.

Putting It Into Practice

The exercise that follows asks you to parse this exact framing: a list of (length, payload) frames packed into one byte stream. Once you can do this you can talk to a real Kafka broker — the rest of the protocol is just nested length-prefixed structures all the way down.