p99 latency na API ya kai 3s — yaya kake debug da inganta shi?

Question

IT Interview Practice · Accepted Answer

p99 = **tail latency**: mafi jinkirin 1% na requests suna ɗaukar 3s. Mataki na farko shi ne **auna da rarraba shi, ba tsammani ba**. Kwatanta p50 da p99 — idan p50 mai sauri ne amma p99 ya kai 3s, hanyar ba jinkiri ba ce iri ɗaya; wani abu yana ciji kawai *ƙarƙashin load ko a edge cases* (pool saturation, GC pauses, N+1, cold cache, external call marar auna lokaci).

**Tunanin fahimta:** p99 shi ne *layin checkout mafi tsawo*, ba matsakaicin jira ba — hanzarta matsakaici ba ya yin komai ga abokin ciniki da ya makale a baya.

## Tsarin aiki (daga ina zuwa ina)
1. **Auna & rarraba** — RED metrics (Rate, Errors, Duration) + distributed tracing; karanta p50/p95/p99. *Gibin* dake tsakaninsu yana ba da labarin.
2. **Gano matakin** — auna kowane mataki don ganin inda requests masu jinkiri ke ɓata lokaci; kada ka inganta bisa zato.

```js
// Auna lokacin kowane mataki don ganin inda p99 ke taruwa (kada ka yi tsammani)
async function handler(req, res) {
  const t0 = process.hrtime.bigint();
  const mark = (label) =>
    req.log.info({ label, ms: Number(process.hrtime.bigint() - t0) / 1e6 });
  const user = await db.user.find(req.id);   mark('db.user');
  const courses = await loadCourses(user);   mark('loadCourses'); // wanda ake zargi
  res.json(courses);                         mark('total');
}
```

3. **Gano wanda ake zargi na tail** — yawanci *jira* ne, ba ƙididdiga ba: connection-pool exhaustion, N+1 queries, external calls marasa auna lokaci, GC pauses, index da ya ɓace, cold cache.

```js
// KAFIN: N+1 — N sequential round-trips, p99 yana fashewa yayin da N ke girma
const courses = await db.course.findByUser(userId);
for (const c of courses) c.lessons = await db.lesson.byCourse(c.id);

// BAYAN: rushe zuwa query 1 -> tail yana faɗuwa (babu sauran N sequential trips)
const courses = await db.course.findByUser(userId);
const lessons = await db.lesson.byCourseIds(courses.map((c) => c.id)); // IN(...) guda ɗaya
const byCourse = Map.groupBy(lessons, (l) => l.courseId);
for (const c of courses) c.lessons = byCourse.get(c.id) ?? [];
```

A nan p50 (ƙaramin N) yana kama da kyau, amma ƙarƙashin load N yana girma kuma pool yana cika, don haka **p99 yana fashewa**. Ga external calls, koyaushe saita **timeout + circuit breaker** don kada dependency ɗaya mai jinkiri ya iya jan p99 na dukan endpoint.

4. **Gyara abu ɗaya → sake auna p99 → maimaita.** Variable ɗaya a lokaci guda, ko kuma ba za ka san abin da ya taimaka ba.
5. **Kulle shi** — saita **SLO** (misali p99 < 300ms) + alert don kada ya koma baya.

## Mahangar hira
Mai hirar yana bincike tsari *na tsari*, ba tsammani na sa'a ba: ko kana banbanta **tail da mean**, ka auna kafin canjawa, kuma ka duba **queues/locks (jira) kafin CPU**. Amsa mai rauni: "ƙara caching / scale up servers." Amsa mai ƙarfi: auna → gano da tracing → gyara takamaiman dalilin tail → tabbatar da lambobi.

## Ƙa'ida
Inganta **tail (p99), ba matsakaici ba**. Yawancin "3s p99" na fitowa daga *jira* (pool/queue/lock) maimakon *ƙididdiga* — don haka duba queues kafin CPU.

Tsarin aiki (daga ina zuwa ina)

Mahangar hira

Ƙa'ida

Tambayoyi masu alaƙa