Why Centralized Logs?

When your service runs on 1 box, tail -f /var/log/app.log works. When it runs on 200 containers across 10 hosts, you need centralized log aggregation: ship all logs to a central system, search them, alert on patterns.

Real systems: Splunk (commercial, expensive), Elasticsearch + Logstash + Kibana (ELK stack), Loki (Prometheus team), Datadog Logs, AWS CloudWatch Logs.

The core pipeline:

[App] -> [Agent (Fluentd/Vector/Filebeat)] -> [Buffer (Kafka)] -> [Indexer] -> [Storage] -> [Query API] -> [UI]

Each stage matters:

• Agent on host: tails files (or reads stdout), parses, batches, ships. Fail-tolerant — if the central system is down, buffer locally.
• Buffer (Kafka or similar): smoothes spikes. App can suddenly do 10x logs; downstream can't always handle.
• Indexer: parses, extracts fields, builds index entries.
• Storage: object store (S3) or columnar files. Old logs are read-only.
• Query: full-text search + filters; aggregations like "count errors per minute".

Volume reality: a single API server can produce GB/day of logs at scale. 100 servers = 100GB/day = 36TB/year. Hot search vs cold archive matters.

We'll build a simplified version: ingest log lines via HTTP, index them by time + labels, serve LogQL-style queries.