Production Kafka

Real-world Kafka deployments:

Cluster size:

• 3 brokers minimum (RF=3).
• 6-12 brokers for moderate load.
• 100+ for big tech (LinkedIn, Netflix run thousands of brokers).

Hardware:

• SSD disks (NVMe ideal).
• 32+ GB RAM (most goes to page cache).
• 10 Gbit network minimum.
• Many cores; Kafka is multi-threaded.

Topic design:

• Partition count: 6-12 per topic typical, more for high-throughput.
• Replication factor: 3 standard.
• Retention: time-based or size-based per topic.
• Compaction for KV-style topics.

Data formats:

• JSON: simple, large. Avoid for high-volume.
• Avro + Confluent Schema Registry: compact, schema evolution.
• Protobuf: similar to Avro, more language support.
• MessagePack: less common in Kafka.

Schema evolution:

• Forward compatibility: new producer + old consumer (consumer ignores new fields).
• Backward compatibility: new consumer + old producer (consumer fills defaults).
• Full compatibility: both directions.
• Schema Registry enforces.

Connectors (Kafka Connect):

• Source connectors: import from DB (Debezium for CDC), files, MQTT, etc.
• Sink connectors: write to S3, Elasticsearch, JDBC.
• Hundreds available.

Stream processing:

• Kafka Streams (Java): topology-based.
• ksqlDB: SQL on Kafka.
• Apache Flink: more powerful, separate cluster.
• Spark Streaming: batch-style micro-batches.

Cloud offerings:

• Confluent Cloud: managed Kafka by Kafka's creators.
• AWS MSK: managed Kafka.
• Aiven, Upstash, Redpanda Cloud.

Alternatives:

• AWS Kinesis: similar concept.
• Pulsar: more flexible (multi-tenancy).
• NATS JetStream: lighter weight.
• Redis Streams: in-process.

Common mistakes:

• Too few partitions (limits parallelism).
• Too many partitions (overhead).
• Long-running consumer processing without heartbeat.
• Forgetting acks=all for critical data.
• Not monitoring lag.

Don't roll your own Kafka. Use upstream Kafka, Redpanda, or managed cloud service.