Observability & Operations
Monitoring & Metrics
Metrics are numeric time series — counters, gauges, histograms — that report the aggregate health of a system in real time, and are the foundation every alert, dashboard, and capacity decision is built on.
- Counter: monotonically increasing (requests served, errors) — useful as a rate (per second) far more often than as a raw total
- Gauge: a value that goes up or down (queue depth, memory used, active connections) — the current state, not a rate
- Histogram: distribution of observed values (request latency) — enables percentiles, which matter far more than the average for user-facing latency
- The four golden signals (latency, traffic, errors, saturation) form a minimal dashboard for any service
- Cardinality — the number of distinct label combinations — is the hidden cost of metrics: tagging a counter by user ID turns it into millions of time series and can take down the monitoring system itself
- Averages hide the worst experience: a 50ms average latency can still mean 1% of users wait 5 seconds — look at p95/p99, not just the mean
| Type | Measures | Example | Typical query |
|---|---|---|---|
| Counter | cumulative count | total requests served | rate() over 5 minutes |
| Gauge | current value | queue depth, memory used | current value, or min/max over a window |
| Histogram | distribution of values | request latency | p50 / p95 / p99 |
Counter.builder("http.requests").tag("status", "200").register(registry).increment();
Gauge.builder("queue.depth", queue, Queue::size).register(registry);
Timer.builder("http.request.duration").publishPercentileHistogram().register(registry);