Ta yaya za ka adana log events miliyan 100 kowace rana?

Question

IT Interview Practice · Accepted Answer

Gina **append-only pipeline: agents → Kafka → tiered storage (hot a ClickHouse/ELK, cold a object storage)** da retention policy da ke index recent data kuma yana archive sauran cikin arha. Events miliyan 100/rana kusan 1,200/daƙiƙa ne a matsakaici (spikes masu girma sosai) da GB da yawa a rana — kana optimize don rubuce-rubuce masu arha da iyakantaccen query cost, ba OLTP semantics ba.

```text
App/agents ─▶ Kafka ─▶ Consumers ─┬─▶ ClickHouse / ELK   (hot: last 7–30d, indexed, fast search)
 (Fluent Bit)  (buffer,            │
                replay)            └─▶ S3/Parquet         (cold: 90d–years, cheap, scan-on-demand)
                                          │
                                   Lifecycle/tiering ─▶ Glacier (archive) ─▶ delete at retention
```

## Append pipeline

Logs ana rubuta su sau ɗaya, ba a taɓa updating ba — don haka ingest path shi ne **append**. Agents kamar **Fluent Bit / Vector** suna aika lines zuwa **Kafka**, wanda ke buffer bursts kuma yana raba producers daga storage. Idan ClickHouse ya faɗi don maintenance, events suna queue a Kafka kuma suna replay — ba ka zubar da logs ba. Batch writes (dubban rows kowane insert) domin per-row inserts suna rushe throughput a columnar stores.

## Hot store: index don search

Recent logs suna buƙatar fast search ("errors a sa'a ta ƙarshe ga service X"). **ClickHouse** (columnar, yana compress 10x, yana scan biliyoyin rows/daƙiƙa) ko **ELK stack** (Elasticsearch inverted index) suna bauta wannan. Partition da day/service domin queries su prune zuwa 'yan partitions. Wannan tier mai tsada ne ga kowane GB, don haka ka ajiye shi ƙarami — kwana 7–30 na ƙarshe kawai.

## Cold store: object storage + tiering

Tsofaffin logs suna matsawa zuwa **S3 a matsayin Parquet** — mai arha, durable, mai iya query a buƙata ta Athena/Trino. **Lifecycle policy** yana canja S3 → Glacier → sharewa a iyakar retention (misali shekara 1). Wannan *tiering* shi ne ainihin cost lever: hot storage na iya zama sau 50 na farashin archival.

## Ra'ayin hira

Masu yin hira suna bincike **index-vs-cost trade-off**: indexing komai yana ba da fitaccen search amma yana ɓata kasafi; indexing babu komai mai arha ne amma ba mai iya query ba. Mai kyau: "amfani da ELK." Mai kyau sosai: kana tier — index recent hot slice, archive sauran cold — kuma ka ambaci Kafka a matsayin buffer da ke mai da ingestion mai juriya ga downstream outages.

**Tarkon gama gari:** sanya duk logs a Elasticsearch har abada. Shi ne mafi tsada kuma cluster ɗinsa yana faɗuwa a kan cardinality/retention da ba ka shirya ba.

## Sulhu & hanyoyin gazawa

- **Retention vs cost:** retention mai tsawo = growth na storage na linear; yi tier da ƙarfi.
- **Kafka backpressure:** idan consumers suka yi lag, disk yana cika — yi size na retention kuma ka monitor lag.
- **Cardinality explosion:** high-cardinality fields (user_id) suna kumbura ES indexes; sample ko drop su.
- **Query cost a cold data:** Athena scans suna kashewa ga kowane TB — partition Parquet da date/service domin scans su kasance iyakantattu.
- **At-least-once dupes:** dedup a kan event id idan ainihin counts suna da muhimmanci.

Append pipeline

Hot store: index don search

Cold store: object storage + tiering

Ra'ayin hira

Sulhu & hanyoyin gazawa

Tambayoyi masu alaƙa