An API's p99 latency is 3s — how do you debug and optimize it?

Question

IT Interview Practice · Accepted Answer

p99 = **tail latency**: the slowest 1% of requests take 3s. The first move is to **measure and break it down, not guess**. Compare p50 vs p99 — if p50 is fast but p99 is 3s, the path isn't uniformly slow; something bites only *under load or in edge cases* (pool saturation, GC pauses, N+1, cold cache, an un-timed external call).

**Mental model:** p99 is the *longest checkout line*, not the average wait — speeding up the average does nothing for the customer stuck at the back.

## The process (from where to where)
1. **Measure & break down** — RED metrics (Rate, Errors, Duration) + distributed tracing; read p50/p95/p99. The *gap* between them tells the story.
2. **Localize the stage** — instrument each step to see where slow requests spend time; don't optimize on a hunch.

```js
// Time each stage to see where p99 piles up (don't guess)
async function handler(req, res) {
  const t0 = process.hrtime.bigint();
  const mark = (label) =>
    req.log.info({ label, ms: Number(process.hrtime.bigint() - t0) / 1e6 });
  const user = await db.user.find(req.id);   mark('db.user');
  const courses = await loadCourses(user);   mark('loadCourses'); // suspect
  res.json(courses);                         mark('total');
}
```

3. **Identify the tail suspect** — usually *waiting*, not computing: connection-pool exhaustion, N+1 queries, un-timed external calls, GC pauses, missing index, cold cache.

```js
// BEFORE: N+1 — N sequential round-trips, p99 blows up as N grows
const courses = await db.course.findByUser(userId);
for (const c of courses) c.lessons = await db.lesson.byCourse(c.id);

// AFTER: collapse to 1 query -> the tail drops (no more N sequential trips)
const courses = await db.course.findByUser(userId);
const lessons = await db.lesson.byCourseIds(courses.map((c) => c.id)); // one IN(...)
const byCourse = Map.groupBy(lessons, (l) => l.courseId);
for (const c of courses) c.lessons = byCourse.get(c.id) ?? [];
```

Here p50 (small N) looks fine, but under load N grows and the pool saturates, so **p99 detonates**. For external calls, always set a **timeout + circuit breaker** so one slow dependency can't drag the whole endpoint's p99.

4. **Fix one thing → re-measure p99 → repeat.** One variable at a time, or you won't know what helped.
5. **Lock it in** — set an **SLO** (e.g. p99 < 300ms) + alert so it doesn't regress.

## Interview lens
The interviewer is probing a *methodical* process, not a lucky guess: do you distinguish **tail from mean**, measure before changing, and look at **queues/locks (waiting) before CPU**. Weak answer: "add caching / scale up the servers." Strong answer: measure → localize with tracing → fix the specific tail cause → verify with numbers.

## Principle
Optimize the **tail (p99), not the average**. Most "3s p99" comes from *waiting* (pool/queue/lock) rather than *computing* — so look at queues before CPU.

The process (from where to where)

Interview lens

Principle

Related questions