Ƙera high-volume event-tracking pipeline don rahotanni da ML — wane stack?

Question

IT Interview Practice · Accepted Answer

A ɗaruruwan miliyoyin events/rana (~dubbai kaɗan/daƙiƙa, mafi girma a lokutan kololuwa) matakin nasara shi ne raba ingestion da serving: sha writes cikin log mai ɗorewa, sarrafa stream sau ɗaya, sannan a rarraba zuwa stores biyu da aka daidaita don ayyuka masu adawa — pre-aggregated don iyaye (karatu mai sauri), raw da full-fidelity don ML.

ɗakin wasiƙa — collector yana karɓar wasiƙa, Kafka shi ne conveyor belt, kuma ana kwafa kowane abu zuwa shiryuna biyu: da iyaye ke duban, da da ƙungiyar ML ke tono.

**Tunanin fahimta:** ɗakin wasiƙa — collector yana karɓar wasiƙa, Kafka shi ne conveyor belt, kuma ana kwafa kowane abu zuwa shiryuna biyu: *allon taƙaitawa* da iyaye ke duban, da *cikakken archive* da ƙungiyar ML ke tono.

## Hanyar rubutu (ingest)

```text
Client ─(batch)─▶ Collector ─▶ Kafka ─▶ Stream proc ─┬─▶ OLAP rollup  (daily summaries)
 (learning app)    (HTTP)     (buffer,   (read once)  └─▶ S3 / Parquet (raw events)
                              replay)
```

- **Collector** yana ɗaukar events da aka **batch** ta HTTP — batching a client yana juya ɗaruruwan miliyoyin ƙananan writes zuwa requests kaɗan sosai.
- **Kafka** shi ne durable buffer: yana shan spikes, yana raba producers da consumers, kuma yana ba ka damar **replay** na tarihi lokacin da ML ya ƙara feature ko ka gabatar da sabon rollup.
- **Stream processor** yana karanta Kafka sau ɗaya kuma yana rubuta sinks ɗin biyu — ƙara raw zuwa lake, upsert rollups don rahotanni.

```js
// Consumer ɗaya yana karanta Kafka sau ɗaya kuma yana rubuta sinks ɗin BIYU
for await (const batch of kafka.consume('events')) {
  await lake.appendParquet(batch);                      // raw -> S3/Parquet (ML)
  for (const e of batch)
    rollup.add({ childId: e.childId, day: dayOf(e.ts) },// tara -> OLAP (rahotanni)
               { lessons: 1, studiedMs: e.studiedMs ?? 0 });
  await rollup.flush();
}
```

## Hanyar karatu (serving)

```text
Parent app ─▶ API ─▶ OLAP rollup                 (pre-aggregated: report = O(1), no scan)
ML / Data  ─▶ Spark / warehouse ─▶ S3 / Parquet  (raw, flexible, ad-hoc)
```

Iyaye suna bugun **pre-aggregated summaries**, don haka rahoto key lookup ne, ba taɓa scan a kan biliyoyin rows ba. ML yana query **raw lake**, yana riƙe kowane field don feature engineering.

## Stack da dalili
- **Kafka** — buffer, raba, replay.
- **ClickHouse / daily summary tables** — an inganta don karatu ga dashboards na iyaye.
- **S3 + Parquet** — mai arha, columnar, raw storage mai sassauci don ML.
- **Schema registry** — tabbatar da event contract don kada producers su karya consumers a shiru.

## Mahangar hira
Mai hirar yana bincike ko ka raba **write-optimized ingest da read-optimized serving**, ka guji scanning raw events don kowane rahoton iyaye, kuma ka riƙe raw data don ML. **Mai rauni:** "sanya duka a Postgres kuma ka query." **Mai ƙarfi:** durable buffer + hanyoyin serving guda biyu + replay + schema contract, tare da fahimtar girman.

## Fa'ida da hasara
Pre-aggregation yana ba da karatu nan take amma yana ƙayyade dimensions tun farko; raw lake yana da sassauci amma a hankali. Riƙe **duka biyun** — rarrabuwar gargajiya ta write-once, read-many.

Hanyar rubutu (ingest)

Hanyar karatu (serving)

Stack da dalili

Mahangar hira

Fa'ida da hasara

Tambayoyi masu alaƙa