What are the common monitoring tools and how do you choose between them?

Question

Accepted Answer

The landscape splits by pillar — metrics, logs, traces — plus all-in-one managed platforms. The choice comes down to **self-hosted vs managed**, driven by team size, budget, and scale.

## Tools by pillar

```text
METRICS    Prometheus  → pull-based scraping, time-series DB, PromQL query language
           Grafana     → dashboards on top of Prometheus (and many other sources)

LOGS       ELK         → Elasticsearch + Logstash + Kibana (powerful, heavy to run)
           Loki        → "Prometheus for logs": cheap, indexes labels not full text

TRACES     Jaeger      → distributed tracing, OpenTelemetry-compatible
           Tempo       → trace backend that pairs with Grafana/Loki

ALL-IN-ONE Datadog     → managed metrics + logs + traces + APM in one product
```

## Why Prometheus + Grafana is the common open-source core

Prometheus **pulls** metrics from targets it scrapes (the service exposes `/metrics`), stores them in a time-series DB, and queries with **PromQL**. Pull makes targets easy to discover and health-check. Grafana sits on top for visualization. Loki and Tempo extend the same stack to logs and traces, all queryable in one Grafana UI.

```promql
# PromQL: per-route error ratio over 5 minutes
sum by (route) (rate(http_requests_total{status=~"5.."}[5m]))
  / sum by (route) (rate(http_requests_total[5m]))
```

## Self-hosted vs managed tradeoffs

```text
SELF-HOSTED (Prometheus/Grafana/Loki/Tempo)
  ✓ no per-host bill, full control, no data leaves your infra
  ✗ you run, scale, and patch it; storage & HA are your problem

MANAGED (Datadog, Grafana Cloud)
  ✓ no ops burden, integrations out of the box, scales for you
  ✗ costs grow fast with hosts/volume; some vendor lock-in
```

## How to pick

```text
Small team / tight budget   → self-hosted Prometheus + Grafana (+ Loki)
Fast-moving, ops-light team  → managed (Datadog / Grafana Cloud)
Large scale / strict cost    → self-host the core, but budget for the ops team
```

## Why it matters

There's no universally best tool — only the right fit for your constraints. A 5-person startup drowns trying to operate an HA Elasticsearch cluster; a large enterprise can burn six figures a year on a managed platform it could self-host. Knowing what each tool does and the self-hosted-vs-managed tradeoff lets you match observability to team size, budget, and scale instead of cargo-culting whatever's popular.