Access Logs & Metrics

A server that works is not the same as a server you can operate. The moment yours is running somewhere you cannot attach a debugger, two questions arrive that only instrumentation answers: what happened to this one request the customer is complaining about, and is the whole thing getting slower. Those are different questions with different answers — logs for the first, metrics for the second — and conflating them is the most common observability mistake there is.

The access log line

One line per completed request, in the Common Log Format that Apache defined and everything since has copied:

192.168.1.5 - - [17/Mar/2025:09:15:23 +0000] "GET /api/users HTTP/1.1" 200 482 "-" "curl/8.5.0" 23ms

The format is worth following exactly rather than inventing your own, because GoAccess, AWStats, Splunk and every log pipeline already parse it for free. Reading it left to right: client IP; the RFC 1413 ident field, dead for thirty years and always a dash; the authenticated username, a dash when there is none; the timestamp; the request line quoted verbatim as received; the status code; the bytes in the response body; the Referer; the User-Agent; and — the extension everyone adds — the latency.

Three of those fields carry traps. The timestamp uses a fixed English month abbreviation and an explicit offset, not the machine's locale: a server that logged 17/mars/2025 because someone set LC_TIME would be unparseable by every tool listed above. The request line is echoed as it arrived, which means it contains attacker-controlled bytes — quotes and control characters need escaping before they reach a log file, or a crafted request forges an extra log line and covers its tracks. And the dashes are placeholders precisely so that every line has the same field count; emitting an empty string instead silently shifts every downstream parser by one column.

Log what you cannot reconstruct

The status code and the byte count are cheap and they answer most questions. Two more fields repay their cost immediately:

• A request id, generated at accept time, logged on every line the request produces and returned in a response header. It is what turns "the customer says it failed at 09:15" into one exact line, and then into every downstream service's line for the same request.
• The route pattern, not just the path. /api/users/8fe3 and /api/users/2a91 are the same route; logging the raw path makes them two things and hides the fact that the route is failing.

And log what you must not: an access log is a database of who read what and when. Query strings carry tokens, and full IPs are personal data in most of the world. Truncating the IP and stripping known-sensitive query parameters at write time is far easier than purging a year of logs later.

Metrics: the aggregate question

Logs scale with traffic; metrics do not. A counter incremented a billion times is still one number, which is why the "are we getting slower" question is answered with metrics rather than by grepping:

• http_requests_total{method, route, status} — a counter, only ever increasing. Rate of change gives you throughput; the ratio of status=5xx to total gives you the error rate.
• http_request_duration_seconds — a histogram of bucketed latencies, which is the one that matters. Averages are actively misleading here: a mean of 40 ms is equally consistent with every request taking 40 ms and with 95% at 5 ms alongside 5% at 700 ms. Only the buckets let you compute a p99, and the p99 is what your users actually feel.

Keep the label sets small. Labels multiply — method times route times status is fine, but adding user id or raw path makes a new time series per value, and cardinality explosion is how a monitoring system gets taken down by the service it monitors.

Your exercise: Format Common Log Lines

Pipe-separated request records in, exact Common Log Format lines out. The work is in the timestamp: 2025-03-17T09:15:23 has to become 17/Mar/2025:09:15:23 +0000, which means splitting the ISO form, mapping the month number onto a fixed English abbreviation table, and keeping the zero padding the source already gives you. The ladder covers a mid-year date, a January and a December to catch an off-by-one in the month index, midnight and a 23:59:59 to catch time-of-day slips, and a zero-byte response so the body-size field is not quietly special-cased. Every field is placed exactly where a standard parser expects it — that is the whole value of using the format at all.