Exactly-Once: the Idempotent Producer

After this lesson your broker can tell a producer's retry from a new record: it accepts each batch once, answers a replay with the original offset, and rejects batches that arrive out of order or from a producer that has been replaced.

The problem: retries create duplicates

A producer sends a batch and the broker appends it, but the acknowledgement is lost on the network. The producer cannot know whether the batch was written, so it retries. If the first copy had been written, the log now has the batch twice. Retrying is required for durability, so the cure has to be on the broker: recognise the replay.

Three numbers in every batch

With enable.idempotence=true the producer first asks the broker for a producer id (PID) and epoch (InitProducerId). Every batch it sends then carries, in its header:

Sequence numbers count records per (producer, partition), starting at 0. A batch of three records with base sequence 10 covers sequences 10, 11, 12, so the next batch must have base sequence 13. They are int32: after 2147483647 the next one is 0.

What the broker remembers

For every (partition, producer id) the broker keeps the epoch and a small window: the last five batches it accepted, each as (first sequence, last sequence, base offset). Five matches the default max.in.flight.requests.per.connection, because a producer with five requests in flight can only ever need to retry one of the last five batches.

The checks, in order

For a batch with epoch e, base sequence s and last sequence l:

1. No state for this producer. It must start at 0, otherwise the broker has lost track of it and refuses (OUT_OF_ORDER).
2. Older epoch than the remembered one: a zombie from before a restart. Refuse with FENCED; it must never write again.
3. Newer epoch: the producer restarted and has begun a new numbering. It must start at s = 0. The window is cleared.
4. Same epoch:
   • if a remembered batch has exactly the same (s, l), it is a duplicate: do not append, answer with the base offset the first copy got;
   • otherwise s must be exactly one more than the newest remembered l; a gap or an old sequence outside the window is OUT_OF_ORDER.

Anything accepted is appended at the partition's log end offset and written into the window.

Worked trace

Different data from your exercise. Producer 31, epoch 0, partition p-0 (log end offset starts at 0):

batch seq 0, 3 records      no state, s = 0           ->  OK base=0     log end 3    window [0..2]
batch seq 3, 2 records      3 = 2 + 1                 ->  OK base=3     log end 5    window [0..2] [3..4]
batch seq 3, 2 records      same range remembered     ->  DUPLICATE base=3 (nothing appended)
batch seq 7, 1 record       expected 5, got 7         ->  OUT_OF_ORDER
same batch, epoch 1         newer epoch, but s = 7    ->  OUT_OF_ORDER
batch seq 0, epoch 1        restart                   ->  OK base=5     window [0..0]
batch seq 5, epoch 0        older epoch               ->  FENCED

What this does and does not give you

Idempotence makes a single producer session write each record once, per partition. It does not make multi-partition writes atomic, and it does not survive a producer restart with a new PID. Those need transactions, the next lesson; idempotence is the foundation they build on.

Common mistakes

• Treating any old sequence as a duplicate. Only a remembered batch with the same first and last sequence is one; everything else behind the window is an error.
• One sequence counter for all partitions. State is per (producer, partition).
• Appending, then rejecting. Check before you touch the log or the log end offset.
• Forgetting the wrap. The sequence after 2147483647 is 0.
• Answering a duplicate with a new offset. The producer must be told where the first copy lives.

Your exercise

The starter reads the batches and prints the replies. Implement the checks and the window. The most likely mistake is mixing up the three outcomes of a sequence number that is not the expected one.