How do you set up monitoring and observability to catch problems early?

Question

Accepted Answer

The goal is to **know your system is unhealthy before your users tell you**. Good observability lets you answer questions you didn't anticipate, not just check a fixed set of dashboards. As a tech lead, you set this up *before* the incident, not during it.

## The three pillars

- **Metrics** — cheap numeric time series (request rate, error rate, latency, queue depth). Great for trends, alerting, and SLOs.
- **Logs** — detailed event records for *why* something happened. Make them **structured** (JSON) and attach a **correlation ID** so you can follow one request across services.
- **Traces** — the path of a single request across services, showing where time is actually spent. Essential in distributed systems.

## Alert on symptoms, not noise

Page on what the **user feels**, not on internal jitter. Anchor alerts to **SLOs**: error rate, latency (p95/p99), and availability. A spike in CPU is not an incident; checkout failing for 2% of users is.

| Alert on | Don't page on |
| --- | --- |
| Error rate breaching SLO | Single CPU spike |
| p99 latency over budget | One slow request |
| Failed health checks | Disk at 60% |

## Make alerts actionable

Every alert should be **urgent, real, and actionable** — it names what's wrong and points to a next step. Alerts that fire constantly train the team to ignore them; **alert fatigue** is how real incidents get missed. Add **health checks** and dashboards that show the golden signals at a glance.

## Why it matters

Teams that observe only after an incident are flying blind: they learn about outages from angry customers and debug with guesswork. Investing in observability up front turns 3 a.m. mysteries into 5-minute diagnoses, shrinks downtime, and frees the team to ship instead of firefight.

Alert on	Don't page on
Error rate breaching SLO	Single CPU spike
p99 latency over budget	One slow request
Failed health checks	Disk at 60%