Buni pipeline ya kufuatilia matukio (event-tracking) yenye kiasi kikubwa kwa ripoti na ML — stack gani?

Question

IT Interview Practice · Accepted Answer

Katika mamia ya mamilioni ya matukio/siku (~maelfu machache/sekunde, zaidi wakati wa kilele) hatua ya ushindi ni kutenganisha ingest na serving: fyonza writes kwenye log ya kudumu, chakata stream mara moja, na sambaza kwa stores mbili zilizosanidiwa kwa kazi zinazopingana — zilizojumlishwa awali kwa wazazi (reads za haraka), ghafi na kamili kwa ML.

chumba cha barua — collector inapokea barua, Kafka ni ukanda wa kusafirisha, na kila kitu kinanakiliwa kwenye rafu mbili: ambao wazazi wanaangalia, na ambayo timu ya ML inachimba.

**Mfano wa kufikiri:** chumba cha barua — collector inapokea barua, Kafka ni ukanda wa kusafirisha, na kila kitu kinanakiliwa kwenye rafu mbili: *ubao wa muhtasari* ambao wazazi wanaangalia, na *kumbukumbu kamili* ambayo timu ya ML inachimba.

## Njia ya kuandika (ingest)

```text
Client ─(batch)─▶ Collector ─▶ Kafka ─▶ Stream proc ─┬─▶ OLAP rollup  (daily summaries)
 (learning app)    (HTTP)     (buffer,   (read once)  └─▶ S3 / Parquet (raw events)
                              replay)
```

- **Collector** inachukua matukio yaliyo-**batch** kupitia HTTP — kufanya batch upande wa client hugeuza mamia ya mamilioni ya writes ndogo kuwa maombi machache zaidi.
- **Kafka** ni buffer ya kudumu: inafyonza spikes, inatenganisha producers na consumers, na inakuruhusu **kufanya replay** ya historia wakati ML inaongeza feature au unaanzisha rollup mpya.
- **Stream processor** inasoma Kafka mara moja na kuandika sinks zote mbili — ongeza ghafi kwenye lake, upsert rollups kwa ripoti.

```js
// Consumer mmoja anasoma Kafka mara moja na kuandika sinks ZOTE MBILI
for await (const batch of kafka.consume('events')) {
  await lake.appendParquet(batch);                      // raw -> S3/Parquet (ML)
  for (const e of batch)
    rollup.add({ childId: e.childId, day: dayOf(e.ts) },// aggregate -> OLAP (reports)
               { lessons: 1, studiedMs: e.studiedMs ?? 0 });
  await rollup.flush();
}
```

## Njia ya kusoma (serving)

```text
Parent app ─▶ API ─▶ OLAP rollup                 (pre-aggregated: report = O(1), no scan)
ML / Data  ─▶ Spark / warehouse ─▶ S3 / Parquet  (raw, flexible, ad-hoc)
```

Wazazi wanafikia **muhtasari uliojumlishwa awali**, kwa hivyo ripoti ni key lookup, kamwe si scan juu ya safu bilioni. ML inauliza **lake ghafi**, ikihifadhi kila uga kwa feature engineering.

## Stack na kwa nini
- **Kafka** — buffer, tenganisha, replay.
- **ClickHouse / daily summary tables** — zimeboreshwa kwa reads kwa dashboards za wazazi.
- **S3 + Parquet** — hifadhi ghafi nafuu, columnar, inayonyumbulika kwa ML.
- **Schema registry** — timiza mkataba wa tukio ili producers wasiweze kuvunja consumers kimya kimya.

## Mtazamo wa mahojiano
Mhojaji anachunguza kama unatenganisha **ingest iliyoboreshwa-kwa-kuandika na serving iliyoboreshwa-kwa-kusoma**, unaepuka kufanya scan ya matukio ghafi kwa kila ripoti ya mzazi, na unahifadhi data ghafi kwa ML. **Dhaifu:** "weka yote kwenye Postgres na uulize." **Imara:** buffer ya kudumu + njia mbili za serving + replay + mkataba wa schema, ukiwa na hisia ya scale.

## Ubadilishano (trade-offs)
Pre-aggregation inatoa reads za papo hapo lakini inaweka dimensions mapema; lake ghafi inanyumbulika lakini ni polepole. Weka **zote mbili** — mgawanyo wa kawaida wa write-once, read-many.

Njia ya kuandika (ingest)

Njia ya kusoma (serving)

Stack na kwa nini

Mtazamo wa mahojiano

Ubadilishano (trade-offs)

Maswali yanayohusiana