Observability & Operations
Alerting & On-Call
An alert should fire only when a human needs to act right now — every alert that does not meet that bar trains the on-call engineer to ignore alerts, which is how real incidents get missed.
- Alert on symptoms (user-facing latency or error rate breaching an SLO) rather than causes (CPU at 80%) — causes change constantly and do not always mean users are affected
- Every page should be actionable: if there is nothing a human can do about it right now, it belongs on a dashboard, not in a page
- Alert fatigue is a real failure mode — a system that pages constantly for non-issues trains engineers to acknowledge-and-ignore, which is exactly when a real incident slips through
- Multi-window, multi-burn-rate alerting (a fast burn over a short window plus a slow burn over a long window) catches both sudden and slow-building SLO violations without paging on every brief blip
- On-call rotation design — length, handoff process, escalation policy — is itself a system that needs designing; an unsustainable rotation causes burnout and worse incident response, not better
- Runbooks attached to alerts turn a 3am page into a 10-minute fix instead of a 45-minute cold investigation
| Symptom-based | Cause-based | |
|---|---|---|
| Fires on | user-facing SLO breach | internal metric crossing a threshold |
| False positive rate | lower — tied to actual impact | higher — many causes never affect users |
| Actionability | usually yes | often "wait and see" |