Transactions: How "Exactly-Once" Actually Works on the Wire

Idempotent producers (Lesson on Producers) get you safe retries — the broker dedups a single producer's records. But a stream processor needs more: it reads from one topic, processes, and writes to another, and it wants the whole operation to be atomic. That's what transactions give you, and they're the on-wire foundation of Kafka Streams' "exactly-once" guarantee.

The Players

Three new actors appear when transactions are in play:

1. The producer sets transactional.id and calls initTransactions(), beginTransaction(), commitTransaction(), abortTransaction().
2. The transaction coordinator is one broker per transactional.id (assigned by hashing it). It owns the internal compacted topic __transaction_state.
3. The consumer sets isolation.level=read_committed so it sees only committed records.

The transactional id is durable across producer restarts. A restarted producer with the same transactional.id is recognised by the coordinator, its in-flight transaction is aborted, and a producer epoch bump fences off any zombie instance still running with the old epoch.

The Two Marker Records

Inside a transaction the producer just sends regular records, but each batch is tagged with the producer id and epoch. When the producer calls commitTransaction(), the transaction coordinator writes a special marker record into every topic-partition the producer touched:

• COMMIT marker -> consumers in read_committed mode reveal the previous records.
• ABORT marker -> consumers in read_committed mode skip the previous records.

Consumers see the markers in the log but never expose them to the application. They're how the broker communicates "the transaction touching this partition is now resolved" without needing a coordinator round-trip per consume.

The State Machine

A transactional id moves through this state machine, stored in __transaction_state:

EMPTY
  |  beginTransaction (first send)
  v
ONGOING
  |  commitTransaction
  v
PREPARE_COMMIT
  |  coordinator writes COMMIT markers
  v
COMPLETE_COMMIT
  |  (back to)
  v
EMPTY

The ABORT path mirrors it: ONGOING -> PREPARE_ABORT -> COMPLETE_ABORT -> EMPTY. The two PREPARE states are durable, so even if the coordinator crashes mid-commit, on recovery it knows it had a commit in flight and finishes writing the markers — there's no "in-doubt" outcome from a consumer's perspective.

"Read-Process-Write" Atomicity

The whole reason transactions exist:

producer.beginTransaction()
records = consumer.poll()
for r in records:
    out = transform(r)
    producer.send(out)
producer.sendOffsetsToTransaction(consumer.offsets(), consumer.groupId())
producer.commitTransaction()

Notice sendOffsetsToTransaction: the consumer's offset commit is inside the producer's transaction. Either the new records and the new offset commit both become visible (COMMIT), or neither does (ABORT). On crash + restart, the consumer rewinds to the last committed offset and the work is replayed; the previous in-flight attempt's records were either committed (will be skipped) or aborted (already hidden).

Cost vs Benefit

Transactions are not free:

• One extra round-trip per beginTransaction() and commitTransaction() (to the coordinator).
• Extra writes to __transaction_state and one marker per touched partition per commit.
• Consumers in read_committed mode have higher end-to-end latency because they can't deliver records past an open transaction.

For low-frequency commits (Kafka Streams batches every ~100 ms) the overhead is fine. For per-record commits the marker storm becomes the bottleneck.

What This Lesson Asks

You'll model the transaction state machine: given a sequence of BEGIN / SEND / COMMIT / ABORT events on a single transactional id, output what a read_committed consumer would see. Records inside a committed transaction surface in order; records inside an aborted transaction vanish entirely.