Real-time revenue analytics juu ya mamilioni ya transactions/siku: vipi?

Question

IT Interview Practice · Accepted Answer

**Stream transactions kupitia Kafka kwenda stream processor inayofanya pre-aggregate kwenda OLAP store, ili dashboards zisome rollups ndogo zilizohesabiwa mapema badala ya kuscan raw events.** Kwenye mamilioni ya txns/siku kamwe huendeshi `SUM()` juu ya raw table kwa kila page load — unaaggregate mara moja, wakati wa write.

## Usanifu

```text
Payment svc ──▶ Kafka topic ──▶ Flink/Kafka Streams ──┬─▶ OLAP (ClickHouse/Druid)  ◀── Dashboard
 (txn events)   (buffer,replay)   (windowed pre-agg)    │      (per-minute rollups)     (read)
                                                         └─▶ S3/Parquet (raw, replay/audit)
                                     state store ◀──────────┘  (exactly-once checkpoints)
```

## Jinsi inavyofanya kazi

- **Kafka** ni durable log. Kila payment huchapisha event (`amount`, `currency`, `ts`, `merchant`). Hubuffer spikes, hutenganisha producers na consumers, na hukuruhusu **replay** history ili kujenga upya aggregates.
- **Stream processor (Flink / Kafka Streams)** huweka **tumbling/sliding windows** (kwa dakika/saa) na hukunja kila event kuwa running total katika **state store** yake. Output ni row iliyoaggregatiwa mapema kwa kila (dakika, dimension).
- **OLAP store (ClickHouse, Druid, Pinot)** huhifadhi rollups hizo columnar. Dashboard inayoquery "revenue saa 24 zilizopita" huscan ~1,440 rows, si mamilioni — chini ya sekunde.
- **Materialized views** katika ClickHouse zinaweza kufanya rollup ile ile ndani ya DB (`AggregatingMergeTree`) ikiwa unataka moving parts chache.

## Mtazamo wa mahojiano

Mhojiwa huchunguza kama **unapre-aggregate wakati wa write vs scan wakati wa read**, na kama unaelewa **exactly-once**. Majibu mazuri hutaja Kafka + OLAP; majibu bora sana hueleza **kwa nini late/duplicate events huharibu totals** na jinsi ya kuyarekebisha: idempotent keys, Flink **checkpointing + transactional sink** kwa exactly-once, na **watermarks** kufunga windows licha ya late arrivals. Mtego wa kawaida ni kupendekeza `SELECT SUM()` kwenye Postgres — sawa kwa maelfu/siku, huporomoka kwenye mamilioni.

## Maelewano na hitilafu

- **Lambda vs Kappa.** Lambda huendesha batch layer (Spark ya usiku hulinganisha source of truth) kando ya stream layer (haraka lakini ya makadirio) — sahihi lakini codebases mbili. **Kappa** hutumia stream layer pekee na **hufanya replay ya Kafka** kurekebisha bugs — rahisi, codebase moja, sasa default ya kawaida.
- **Exactly-once si bure.** Transactional sinks huongeza latency na gharama ya throughput; timu nyingi hukubali **at-least-once + idempotent upserts** zilizowekwa key kwa `txn_id` badala yake.
- **Late data.** Payment inayosettle saa 2 kwa kuchelewa hufika baada ya window yake kufungwa. Watermarks + allowed-lateness hufungua windows tena, au batch job ya usiku hurekebisha namba za jana.
- **Multi-currency / refunds** lazima zimodeliwe kama signed events, la sivyo "revenue" hupotoka.
- **Wakati usifanye:** maelfu machache ya txns/siku hazihitaji lolote la haya — Postgres rollup table iliyoupdatiwa na cron hushinda Kafka+Flink cluster unayolazimika kuendesha.

Usanifu

Jinsi inavyofanya kazi

Mtazamo wa mahojiano

Maelewano na hitilafu

Maswali yanayohusiana