Real-time revenue analytics kan miliyoyin transactions/rana: ta yaya?

Question

IT Interview Practice · Accepted Answer

**Stream ɗin transactions ta Kafka zuwa stream processor da ke pre-aggregate zuwa OLAP store, domin dashboards su karanta ƙananan rollups da aka riga aka ƙididdige maimakon scan raw events.** A miliyoyin txns/rana ba ka taɓa gudanar da `SUM()` kan raw table a kowace page load — kana aggregate sau ɗaya, a write.

## Gine-gine

```text
Payment svc ──▶ Kafka topic ──▶ Flink/Kafka Streams ──┬─▶ OLAP (ClickHouse/Druid)  ◀── Dashboard
 (txn events)   (buffer,replay)   (windowed pre-agg)    │      (per-minute rollups)     (read)
                                                         └─▶ S3/Parquet (raw, replay/audit)
                                     state store ◀──────────┘  (exactly-once checkpoints)
```

## Yadda yake aiki

- **Kafka** shi ne durable log. Kowane biyan kuɗi yana buga event (`amount`, `currency`, `ts`, `merchant`). Yana buffer spikes, yana raba producers daga consumers, kuma yana bar ka **replay** tarihi don sake gina aggregates.
- **Stream processor (Flink / Kafka Streams)** yana ajiye **tumbling/sliding windows** (kowane minti/sa'a) kuma yana naɗa kowane event zuwa running total a **state store** ɗinsa. Fitarwa row ne da aka riga aka aggregate ga kowane (minti, dimension).
- **OLAP store (ClickHouse, Druid, Pinot)** yana adana waɗannan rollups columnar. Dashboard da ke query "revenue last 24h" yana scan kusan rows 1,440, ba miliyoyin ba — ƙasa da daƙiƙa.
- **Materialized views** a ClickHouse na iya yin rollup ɗin a cikin DB (`AggregatingMergeTree`) idan kana son ƙananan sassa masu motsi.

## Ra'ayin hira

Mai yin hirar yana bincike ko kana **pre-aggregate a write vs scan a read**, kuma ko ka fahimci **exactly-once**. Amsoshi masu kyau suna ambaton Kafka + OLAP; amsoshi masu kyau sosai suna bayanin **me ya sa late/duplicate events ke ɓata jimloli** da yadda za a gyara su: idempotent keys, Flink **checkpointing + transactional sink** don exactly-once, da **watermarks** don rufe windows duk da late arrivals. Tarkon gama gari shi ne ba da shawarar `SELECT SUM()` a Postgres — yayi kyau a dubbai/rana, yana rushewa a miliyoyin.

## Sulhu & kurakurai

- **Lambda vs Kappa.** Lambda yana gudanar da batch layer (nightly Spark yana sulhunta source of truth) kusa da stream layer (mai sauri amma approximate) — daidai amma codebases biyu. **Kappa** yana amfani da stream layer kawai kuma yana **replay Kafka** don gyara bugs — mafi sauƙi, codebase ɗaya, yanzu default gama gari.
- **Exactly-once ba kyauta ba ne.** Transactional sinks suna ƙara latency da throughput cost; tawagogi da yawa suna karɓar **at-least-once + idempotent upserts** da aka key da `txn_id` maimakon.
- **Late data.** Biyan kuɗi da ke settling sa'o'i 2 daga baya yana sauka bayan window ɗinsa ya rufe. Watermarks + allowed-lateness suna sake buɗe windows, ko nightly batch job yana gyara lambobin jiya.
- **Multi-currency / refunds** dole a model su a matsayin signed events, in ba haka ba "revenue" yana kauda.
- **Lokacin da ba za a yi ba:** 'yan dubban txns/rana ba sa buƙatar wannan — Postgres rollup table da cron ke update yana doke Kafka+Flink cluster da za ka gudanar.

Gine-gine

Yadda yake aiki

Ra'ayin hira

Sulhu & kurakurai

Tambayoyi masu alaƙa