Wire Framing and the Request Header

After this lesson your program does what the front door of a Kafka broker does: it takes the bytes that arrive from a socket in arbitrary pieces, cuts them into complete requests, and reads the header of each one.

TCP has no messages

A socket is a stream. One read can return half a request, exactly one, or three and a half. Kafka solves this with length-prefixed framing (the Kafka protocol guide, "Common Request and Response Structure"): every request and every response is

+-----------------+------------------------------+
| size  (int32)   | payload  (exactly size bytes) |
+-----------------+------------------------------+

All integers in the protocol are big-endian and signed. The size does not count its own 4 bytes. There is no delimiter, because payloads are arbitrary binary and any delimiter byte could appear inside one.

So the reader's job is a loop over a growing buffer:

append the new bytes to the buffer
while the buffer holds at least 4 bytes:
    size = int32 at the front
    if size is invalid: close the connection
    if the buffer holds fewer than 4 + size bytes: stop, wait for the next read
    cut those 4 + size bytes off the front and handle the payload

Whatever is left in the buffer after the loop belongs to a frame that has not fully arrived yet. Never discard it, never treat it as an error.

Validating the size

The size is read before anything else is known about the peer, so it is hostile input. A negative size is invalid. A huge size would make a broker allocate gigabytes for one request; the real limit is socket.request.max.bytes, 104857600 (100 MiB) by default. This exercise also rejects any size below 10, because the smallest request header is 10 bytes (next section). Check the size as soon as the 4 bytes are there, before waiting for the payload: a bogus size must not make you wait for bytes that will never come.

The request header

Every request payload starts with a header (header version 1 for classic API versions):

Everything after the header is the API-specific body. Newer "flexible" versions use header version 2, which is the same fields plus a tagged-fields section at the end; reading the first four fields works for both.

Worked trace

A Heartbeat request (api key 12), different from your exercise, arriving in two reads:

read 1 (6 bytes):  00 00 00 1b 00 0c
read 2 (the rest): 00 00  00 00 00 07  00 05 61 70 70 2d 31  <12 body bytes>

after read 1:  size = 0x1b = 27, buffered 6 - 4 = 2 payload bytes -> wait
after read 2:  buffered 4 + 27 bytes -> cut the frame
               api_key        00 0c          = 12   (Heartbeat)
               api_version    00 00          = 0
               correlation_id 00 00 00 07    = 7
               client_id      00 05 + "app-1" (5 bytes)
               header = 2 + 2 + 4 + 2 + 5 = 15 bytes, body = 27 - 15 = 12 bytes

Common mistakes

• One read, one request. Works on a laptop, breaks on a real network. Two frames in one read must both be handled; half a frame must wait.
• Counting the size field. size excludes its own 4 bytes. A frame occupies 4 + size bytes in the buffer.
• Unsigned parsing. ff ff ff ff is -1, not 4294967295. Correlation ids and sizes are signed; a null client_id has length -1, which is only visible if you read it signed.
• Losing leftover bytes. After cutting a frame, the bytes behind it are the start of the next one.

Your exercise

The starter reads the chunks and appends them to a buffer for you. Implement the frame loop and header decoding. The most likely mistake is treating the end of a chunk as the end of a frame.