How do you choose alert thresholds to avoid alert fatigue and false positives?

Question

Accepted Answer

The core rule: **alert on symptoms, not causes**, and **page only on what is actionable and urgent**. A noisy alert that fires every night gets muted or ignored — so the real risk isn't a missing alert, it's a desensitized on-call who sleeps through the real one.

## Symptoms over causes

Alert on **user-facing SLOs** (error rate, latency, availability), not on internal causes like "CPU > 80%". High CPU may be harmless; what matters is whether users are affected.

```text
BAD  (cause)     CPU > 80% for 5m            → fires constantly, often no impact
GOOD (symptom)   error-rate SLO burn is fast → fires only when users hurt
```

## Multi-window burn-rate alerts

A single static threshold is either too twitchy or too slow. Instead, alert on how fast you are burning the **error budget**, using two windows so a fast burn pages immediately and a slow burn pages only if sustained.

```yaml
# Page: 2% of monthly budget burned in 1h AND still burning over 5m
- alert: HighErrorBudgetBurn
  expr: |
    (slo:error_ratio_1h > 14.4 * 0.001)   # fast window: catch big outages
    and
    (slo:error_ratio_5m > 14.4 * 0.001)   # short window: confirm it's still happening
  for: 2m
  labels: { severity: page }
```

The short window prevents paging for an issue that already self-resolved; the long window prevents paging on a brief blip.

## Page vs ticket vs dashboard

```text
PAGE       actionable + urgent  → wake a human now (SLO at risk, checkout down)
TICKET     actionable, not urgent → fix in business hours (disk 70%, cert in 20 days)
DASHBOARD  informational         → no alert, just visible (per-endpoint traffic)
```

If an alert isn't actionable, it shouldn't page — turn it into a ticket or a dashboard panel. Then **tune over time**: review every page, delete the ones nobody acted on.

## Why it matters

Alert fatigue is a reliability risk, not just an annoyance: humans mentally filter out noisy channels, so the more false positives you send, the more likely a genuine outage is missed. Alerting on symptoms with burn-rate windows, and reserving the page for actionable urgency, keeps every page meaningful — which is what keeps on-call responsive.