Rancangen pipeline event-tracking volume dhuwur kanggo laporan lan ML — stack apa?

Question

IT Interview Practice · Accepted Answer

Ing atusan yuta event/dina (~sawetara ewu/detik, luwih dhuwur nalika puncak) langkah sing menang yaiku misahake ingestion saka serving: nyerep tulisan menyang log tahan lama, ngolah stream sepisan, lan nyebar menyang rong store sing disetel kanggo tugas sing walikan — pra-agregat kanggo wong tuwa (maca cepet), mentah lan full-fidelity kanggo ML.

ruang pos — collector nampa surat, Kafka iku ban berjalan, lan saben item disalin menyang rong rak: sing dideleng sepintas dening wong tuwa, lan sing digali dening tim ML.

**Model mental:** ruang pos — collector nampa surat, Kafka iku ban berjalan, lan saben item disalin menyang rong rak: *papan ringkesan* sing dideleng sepintas dening wong tuwa, lan *arsip kebak* sing digali dening tim ML.

## Jalur tulis (ingest)

```text
Client ─(batch)─▶ Collector ─▶ Kafka ─▶ Stream proc ─┬─▶ OLAP rollup  (daily summaries)
 (learning app)    (HTTP)     (buffer,   (read once)  └─▶ S3 / Parquet (raw events)
                              replay)
```

- **Collector** njupuk event **batch** liwat HTTP — batching ing client ngubah atusan yuta tulisan cilik dadi request sing luwih sithik.
- **Kafka** iku buffer tahan lama: nyerep lonjakan, misahake producer saka consumer, lan ngidini kowe **replay** sejarah nalika ML nambah fitur utawa kowe ngenalake rollup anyar.
- **Stream processor** maca Kafka sepisan lan nulis loro-lorone sink — append mentah menyang lake, upsert rollup kanggo laporan.

```js
// Siji consumer maca Kafka sepisan lan nulis LORO-LORONE sink
for await (const batch of kafka.consume('events')) {
  await lake.appendParquet(batch);                      // mentah -> S3/Parquet (ML)
  for (const e of batch)
    rollup.add({ childId: e.childId, day: dayOf(e.ts) },// agregat -> OLAP (laporan)
               { lessons: 1, studiedMs: e.studiedMs ?? 0 });
  await rollup.flush();
}
```

## Jalur maca (serving)

```text
Parent app ─▶ API ─▶ OLAP rollup                 (pre-aggregated: report = O(1), no scan)
ML / Data  ─▶ Spark / warehouse ─▶ S3 / Parquet  (raw, flexible, ad-hoc)
```

Wong tuwa ngakses **ringkesan pra-agregat**, mula laporan iku lookup key, ora tau scan milyaran baris. ML nge-query **lake mentah**, nyimpen saben field kanggo feature engineering.

## Stack & sebabe
- **Kafka** — buffer, misahake, replay.
- **ClickHouse / tabel ringkesan saben dina** — dioptimalake kanggo maca ing dashboard wong tuwa.
- **S3 + Parquet** — murah, columnar, panyimpenan mentah fleksibel kanggo ML.
- **Schema registry** — meksa kontrak event supaya producer ora bisa ngrusak consumer tanpa ana sing weruh.

## Sudut pandang wawancara
Pewawancara nyelidiki apa kowe misahake **ingest dioptimalake-tulis saka serving dioptimalake-maca**, ngindhari scan event mentah kanggo saben laporan wong tuwa, lan nyimpen data mentah kanggo ML. **Lemah:** "lebokna kabeh ing Postgres lan query." **Kuwat:** buffer tahan lama + rong jalur serving + replay + kontrak schema, kanthi rasa marang skala.

## Trade-off
Pra-agregasi menehi maca sanalika nanging netepake dimensi ing ngarep; lake mentah fleksibel nanging alon. Simpen **loro-lorone** — pamisah klasik nulis-sepisan, maca-akeh.

Jalur tulis (ingest)

Jalur maca (serving)

Stack & sebabe

Sudut pandang wawancara

Trade-off

Pitakon sing gegandhengan