Tailing & Batching

The agent (Fluentd, Vector, Filebeat) on each host:

1. Tails log files (tail -F-style) — handles rotation, truncation.
2. Parses lines (regex / JSON / structured).
3. Adds labels (host, pod, env from environment).
4. Batches (e.g. 5000 lines or 1 second, whichever first).
5. POSTs to log server.

Tailing implementation:

Batch policy:

buffer = []
last_flush = now()
for line in tail(path):
    buffer.append(line)
    if len(buffer) >= 5000 or now() - last_flush > 1.0:
        ship(buffer); buffer = []; last_flush = now()

Persistence: if buffer can't be sent (network down), spill to local disk so we don't lose logs on restart.