How do you store 100M log events per day?

Question

IT Interview Practice · Accepted Answer

Build an **append-only pipeline: agents → Kafka → tiered storage (hot in ClickHouse/ELK, cold in object storage)** with a retention policy that indexes recent data and cheaply archives the rest. 100M events/day is ~1,200/sec average (spikes far higher) and tens-to-hundreds of GB daily — you optimize for cheap writes and bounded query cost, not OLTP semantics.

```text
App/agents ─▶ Kafka ─▶ Consumers ─┬─▶ ClickHouse / ELK   (hot: last 7–30d, indexed, fast search)
 (Fluent Bit)  (buffer,            │
                replay)            └─▶ S3/Parquet         (cold: 90d–years, cheap, scan-on-demand)
                                          │
                                   Lifecycle/tiering ─▶ Glacier (archive) ─▶ delete at retention
```

## Append pipeline

Logs are write-once, never updated — so the ingest path is an **append**. Agents like **Fluent Bit / Vector** ship lines to **Kafka**, which buffers bursts and decouples producers from storage. If ClickHouse is down for maintenance, events queue in Kafka and replay — you don't drop logs. Batch writes (thousands of rows per insert) because per-row inserts destroy throughput on columnar stores.

## Hot store: index for search

Recent logs need fast search ("errors in the last hour for service X"). **ClickHouse** (columnar, compresses 10x, scans billions of rows/sec) or the **ELK stack** (Elasticsearch inverted index) serve this. Partition by day/service so queries prune to a few partitions. This tier is expensive per GB, so keep it small — only the last 7–30 days.

## Cold store: object storage + tiering

Older logs move to **S3 as Parquet** — cheap, durable, queryable on demand via Athena/Trino. A **lifecycle policy** transitions S3 → Glacier → deletion at your retention limit (e.g. 1 year). This *tiering* is the core cost lever: hot storage might be 50x the price of archival.

## Interview perspective

Interviewers probe the **index-vs-cost trade-off**: indexing everything gives great search but blows the budget; indexing nothing is cheap but unqueryable. Good: "use ELK." Great: you tier — index the recent hot slice, archive the rest cold — and name Kafka as the buffer that makes ingestion resilient to downstream outages.

**Common trap:** putting all logs in Elasticsearch forever. It's the most expensive option and its cluster falls over on cardinality/retention you didn't plan for.

## Trade-offs & failure modes

- **Retention vs cost:** longer retention = linear storage growth; tier aggressively.
- **Kafka backpressure:** if consumers lag, disk fills — size retention and monitor lag.
- **Cardinality explosion:** high-cardinality fields (user_id) bloat ES indexes; sample or drop them.
- **Query cost on cold data:** Athena scans cost per TB — partition Parquet by date/service so scans stay bounded.
- **At-least-once dupes:** dedup on an event id if exact counts matter.

Append pipeline

Hot store: index for search

Cold store: object storage + tiering

Interview perspective

Trade-offs & failure modes

Related questions