Design a high-volume event-tracking pipeline for reports and ML — which stack?

Question

IT Interview Practice · Accepted Answer

At hundreds of millions of events/day (~a few thousand/sec, higher at peaks) the winning move is to decouple ingestion from serving: absorb writes into a durable log, process the stream once, and fan out to two stores tuned for opposite jobs — pre-aggregated for parents (fast reads), raw and full-fidelity for ML.

a mailroom — the collector receives mail, Kafka is the conveyor belt, and every item is copied onto two shelves: a parents glance at, and a the ML team digs through.

**Mental model:** a mailroom — the collector receives mail, Kafka is the conveyor belt, and every item is copied onto two shelves: a *summary board* parents glance at, and a *full archive* the ML team digs through.

## Write path (ingest)

```text
Client ─(batch)─▶ Collector ─▶ Kafka ─▶ Stream proc ─┬─▶ OLAP rollup  (daily summaries)
 (learning app)    (HTTP)     (buffer,   (read once)  └─▶ S3 / Parquet (raw events)
                              replay)
```

- **Collector** takes **batched** events over HTTP — batching on the client turns hundreds of millions of tiny writes into far fewer requests.
- **Kafka** is the durable buffer: it absorbs spikes, decouples producers from consumers, and lets you **replay** history when ML adds a feature or you introduce a new rollup.
- **Stream processor** reads Kafka once and writes both sinks — append raw to the lake, upsert rollups for reports.

```js
// One consumer reads Kafka once and writes BOTH sinks
for await (const batch of kafka.consume('events')) {
  await lake.appendParquet(batch);                      // raw -> S3/Parquet (ML)
  for (const e of batch)
    rollup.add({ childId: e.childId, day: dayOf(e.ts) },// aggregate -> OLAP (reports)
               { lessons: 1, studiedMs: e.studiedMs ?? 0 });
  await rollup.flush();
}
```

## Read path (serving)

```text
Parent app ─▶ API ─▶ OLAP rollup                 (pre-aggregated: report = O(1), no scan)
ML / Data  ─▶ Spark / warehouse ─▶ S3 / Parquet  (raw, flexible, ad-hoc)
```

Parents hit **pre-aggregated summaries**, so a report is a key lookup, never a scan over billions of rows. ML queries the **raw lake**, keeping every field for feature engineering.

## Stack & why
- **Kafka** — buffer, decouple, replay.
- **ClickHouse / daily summary tables** — read-optimized for parent dashboards.
- **S3 + Parquet** — cheap, columnar, flexible raw storage for ML.
- **Schema registry** — enforce the event contract so producers can't silently break consumers.

## Interview lens
The interviewer probes whether you separate **write-optimized ingest from read-optimized serving**, avoid scanning raw events for every parent report, and keep raw data for ML. **Weak:** "put it all in Postgres and query it." **Strong:** durable buffer + two serving paths + replay + a schema contract, with a feel for the scale.

## Trade-offs
Pre-aggregation gives instant reads but fixes the dimensions upfront; the raw lake is flexible but slow. Keep **both** — the classic write-once, read-many split.

Write path (ingest)

Read path (serving)

Stack & why

Interview lens

Trade-offs

Related questions