Metrics and Alerting

Measure what users feel

Not every number deserves an alert. The most useful metrics track what your users actually experience: request success rate, latency, and throughput. A widely used shorthand is the four golden signals — latency, traffic, errors, and saturation. If those four look healthy, the system is usually fine; if one degrades, you have a clear starting point.

Resist the urge to alert on every internal counter. CPU at 80% is not itself a problem if requests are still fast and succeeding. Alert on the outcomes users care about, and keep cause-level metrics as context you consult once an outcome alert fires.

Percentiles, not averages

Averages hide pain. If most requests take 50 ms but one in twenty takes five seconds, the average still looks acceptable while a real fraction of users suffer. Track percentiles instead: the 95th and 99th percentile latency reveal the tail that averages smooth away.

latency_p50: 48ms
latency_p95: 210ms
latency_p99: 1900ms   # the tail that hurts

Set latency objectives against a percentile — for example, "99% of requests under 500 ms" — so your alerting reflects the experience of nearly all users, not a misleading mean. Pick the percentile to match how much of the tail you care about: the 95th catches widespread slowness, while the 99th surfaces the smaller but real group of users having the worst time. Tracking both gives you a fuller picture than either alone, and it stops a healthy median from hiding pain at the edges.

Alert on symptoms

Page humans for symptoms, things the user can feel, such as elevated error rates or breached latency objectives. Treat causes, like a full disk or a slow query, as diagnostic detail you investigate after a symptom alert, not as separate pages. Cause-based alerts multiply quickly and rarely map cleanly to user impact.

This keeps the number of alerts small and each one meaningful. When the pager goes off, it should mean users are affected and action is required, never "a number moved a little". Fewer, sharper alerts are answered faster than a flood of noisy ones.

Tune to avoid fatigue

An alert that fires constantly is quickly ignored, and an ignored alert is worse than none — it trains people to dismiss the pager. Tune thresholds and durations so an alert means a sustained, real problem: require a condition to persist for a few minutes rather than firing on a single noisy data point.

Review every alert periodically. If one never fires, it may be dead weight; if one fires often and is routinely dismissed, it needs retuning or retiring. Our status page reflects platform-side health you can cross-reference before assuming the fault is yours.