Exactly-Once Semantics

Naive Kafka is at-least-once: messages may be reprocessed on consumer crash.

Exactly-once requires:

1. Idempotent producer: retries don't create duplicates.
2. Transactional commits: process + offset commit in one atomic step.

Idempotent producer:

• Each producer has a unique PID (producer ID) assigned by broker.
• Each (PID, partition, sequence_num) → broker tracks; deduplicates retries.
• Configured: enable.idempotence=true.

Transactions:

What happens:

1. Producer marks transaction in coordinator.
2. Sends + offset commit are tagged with transaction ID.
3. Commit: write COMMIT marker. All readers see all messages atomically.
4. Abort: write ABORT marker. Readers skip messages.

Consumer side:

• isolation.level=read_committed: skip messages from open or aborted transactions.
• Default read_uncommitted: see everything.

Use cases:

• Stream processing: read from input topic, write to output topic, commit offsets atomically.
• Avoid double-charging customers in payment processing.

Limitations:

• Exactly-once within Kafka. End-to-end (Kafka → DB) needs DB-level transactions.
• Throughput cost: ~10-30% slower than at-least-once.
• Complexity: more failure modes to debug.

Kafka Streams + Kafka Transactions:

• Stream processing API on top of transactions.
• Built-in EOS.

Modern alternatives:

• Pulsar Functions with idempotent processing.
• Flink with checkpointing + exactly-once sinks.
• Kafka Connect with EOS for sinks (Kafka 3.0+).