Alerting on Logs

Alerts fire when a query result crosses a threshold:

• "More than 100 errors per minute"
• "No INFO logs from service:api for 5 minutes" (means it's down)
• "An ERROR matching 'OutOfMemory' anywhere"

Alert manager pattern:

- name: high-error-rate
  expr: sum(rate({level="error"}[5m])) > 100
  for: 5m   # must be true for 5 min before firing
  annotations:
    summary: "Error rate spike"

Evaluation:

1. Every 30s, run the expression.
2. If true, start a "pending" timer.
3. If still true after for duration, transition to "firing" → notify.
4. If becomes false, reset.

Notifications:

• Slack / PagerDuty / email.
• De-dupe: same alert + same labels → group, send once.
• Inhibition: if "host_down" fires, suppress "high_error_rate" for that host.

Watching for ABSENCE of logs is critical — a service that stops logging is often a service that crashed but didn't restart cleanly.

Real systems: Prometheus AlertManager (used with Loki/Cortex), Datadog Monitors.

Implementation: