Metrics and Observability

A load balancer sees every request, which makes it the best vantage point in your whole system. Instrument it well and you can answer "is the site up?", "which backend is slow?" and "when do we need capacity?" without touching an application. In this lesson your balancer starts publishing what it knows in the format Prometheus scrapes from GET /metrics.

Three kinds of number

• A counter only goes up: requests served, bytes moved. You never read it raw; you ask for its rate. Because rates come from differences, a counter that resets to 0 when the balancer restarts does no harm. By convention counters end in _total.
• A gauge moves both ways and is read directly: whether a backend is up, connections open now.
• A histogram records a distribution in buckets, which is how you get p50 and p99 across many balancer instances. Averages hide everything interesting: a 50 ms average is compatible with a 4 second p99.

If you remember one framework, make it RED: Rate, Errors, Duration. All three come straight from the counters and the histogram below.

The exposition format

It is plain text, one sample per line: a metric name, optional {label="value"} pairs, a space, a number. Each metric family is introduced by a # HELP line and a # TYPE line:

# HELP lb_requests_total Requests proxied, by backend and response status.
# TYPE lb_requests_total counter
lb_requests_total{backend="10.0.0.11:8080",status="200"} 4823
lb_requests_total{backend="10.0.0.11:8080",status="500"} 12

Rules that are easy to get wrong:

• Quote label values and escape three characters inside them: a backslash becomes \\, a double quote becomes \", a newline becomes \n. A backend named we"ird is written backend="we\"ird".
• Use base units: durations in seconds, sizes in bytes. A latency of 55 ms is 0.055. Write seconds without trailing zeros: 0.5, 1, 0.055, never 0.500 or 1.0.
• Labels multiply. Every distinct label combination is a separate time series. Label by backend (tens) and status (a handful), never by user id or raw URL path. A cardinality explosion is the classic way to kill a Prometheus server.

Histograms in detail

A histogram is a family of lines per series: one _bucket per upper bound, then _sum and _count. Buckets are cumulative: the bucket with le="0.1" counts every observation of at most 0.1 seconds, including those already counted in smaller buckets, and the bound is inclusive (an observation of exactly 100 ms lands in le="0.1"). The last bucket, le="+Inf", always equals _count.

5 observations: 8 ms, 30 ms, 100 ms, 120 ms, 2000 ms          bounds 0.01 0.05 0.1 0.5 1 +Inf

lb_request_duration_seconds_bucket{backend="a",le="0.01"} 1
lb_request_duration_seconds_bucket{backend="a",le="0.05"} 2
lb_request_duration_seconds_bucket{backend="a",le="0.1"} 3
lb_request_duration_seconds_bucket{backend="a",le="0.5"} 4
lb_request_duration_seconds_bucket{backend="a",le="1"} 4
lb_request_duration_seconds_bucket{backend="a",le="+Inf"} 5
lb_request_duration_seconds_sum{backend="a"} 2.258
lb_request_duration_seconds_count{backend="a"} 5

A histogram with no observations still lists every bucket with count 0.

Common mistakes

• Per-bucket instead of cumulative counts, or < instead of <= at the boundary.
• Skipping zero series. Every pool backend gets its bytes, duration and up lines even when idle, so a graph can tell "no traffic" from "no data". The request counter is the exception: it only has the (backend, status) pairs that actually occurred.
• Unescaped label values, and sums printed as 2.2580.
• 32-bit byte counters. A busy balancer passes 4 GB in minutes.

Your exercise

The starter parses commands and keeps the byte and health numbers. You write escape_label, seconds_text, the recording of a request (REQUEST <backend> <status> <ms> counts the pair and observes the duration) and the METRICS rendering: four families in the order lb_requests_total, lb_bytes_total{direction="in"|"out"}, lb_request_duration_seconds and lb_backend_up, each with its HELP and TYPE lines, backends sorted by name.