What Kafka Solves

After this lesson your program is the smallest honest Kafka: one append-only log, offsets handed out by the log, and any number of readers that each remember their own position.

The problem: two systems, one stream of facts

A shop writes every order to its database. The search index, the email sender and the analytics job all need to know about each order. Wiring each one to the database means three queries, three failure modes, and no answer to "what did the email sender already see?"

Kafka's answer is to stop treating the facts as rows and treat them as a log: a sequence of records that is only ever added to at the end, never edited, never reordered. Writers append. Readers read. Nobody asks anybody else for permission.

Offsets

The log assigns every record a number when it is appended: its offset. The first record is offset 0, the next is 1, and so on. Offsets are never reused.

offset:   0        1        2        3
record:   order-a  order-b  order-c  order-d
                                      ^
                             log end offset = 4
                             (the offset the NEXT record will get)

The number one past the last record is the log end offset. An empty log has end offset 0. Do not confuse "the last offset" (3) with "the log end offset" (4); consumer lag, replication and the high watermark are all defined on the second one.

A reader is just a number

A consumer does not take records out of the log. It keeps a position: the offset of the next record it wants. To read, it asks for records starting at its position, then moves the position forward by however many it received.

consumer "email" position 0  ->  reads offsets 0,1   ->  position 2
consumer "stats" position 0  ->  reads offset  0     ->  position 1

Two consequences make Kafka different from a queue:

• Readers are independent. "email" moving to 2 does nothing to "stats". Adding a fifth consumer costs the log nothing.
• Reading is repeatable. Setting the position back to 0 (a seek) replays everything. Fixing a bug in the analytics job means rewinding it, not asking producers to resend.

A consumer that has never read starts at position 0 in this exercise (real consumers choose with auto.offset.reset).

Worked trace

Different data from your exercise, so you see the mechanism rather than the answer:

APPEND view:home        -> offset 0
APPEND view:cart        -> offset 1
READ   a 1              -> a reads offset 0, position becomes 1
APPEND view:pay         -> offset 2
READ   b 5              -> b reads offsets 0,1,2, position becomes 3
SEEK   a 0              -> a position becomes 0 (replay)
READ   a 2              -> a reads offsets 0,1 again, position becomes 2

The third record was appended after a had already read: the log grew under an active reader and a simply finds more when it asks again. Asking for more records than exist returns what is there, never an error.

Common mistakes

• Removing records on read. A queue-style pop makes the second consumer see nothing. Reading only moves that consumer's position.
• Sharing one position. One global "next" counter means b starts where a stopped instead of at 0.
• Returning the last offset instead of the next. The end offset of a log with 3 records is 3, not 2.
• Trimming values. A record value is everything after the command word and one space, including any spaces inside it.

Your exercise

The starter already parses each command line. Implement the log, per-consumer positions, and the four command behaviours. The most likely mistake is advancing the position by the number of records you asked for instead of the number you actually got.