Ta yaya za ka gina full-text search kan records miliyan 100+?

Question

IT Interview Practice · Accepted Answer

Yi amfani da dedicated search engine — **Elasticsearch/OpenSearch da aka gina a kan inverted index** — da ake ciyar da shi daga source DB ta **CDC**, kuma ka shard index don sikeli. Primary database ɗinka (Postgres/MySQL) ba zai iya yin relevance-ranked text search kan ɗarurruwan miliyoyin rows ba; `LIKE '%term%'` yana scan kowace row. Search engine yana juya matsalar.

```text
Source DB ─▶ CDC (Debezium) ─▶ Kafka ─▶ Indexer ─▶ Elasticsearch/OpenSearch
(Postgres)   (change stream)                        ├─ Shard 0 (inverted index) + replicas
                                                     ├─ Shard 1
Query ─▶ Coordinator ─▶ scatter to shards ─▶ gather ─┴─ Shard N ── ranked by BM25
```

## Inverted index — me ya sa yake da sauri

Index na yau da kullum yana taswira row → values. **Inverted index** yana taswira *term → jerin documents da ke ɗauke da shi*. Nemi "database scaling" yana lookup posting lists guda biyu masu gajarta kuma yana intersect su — O(matching docs), ba O(all docs) ba. Wannan shi ne abin da ke sa ƙasa da 100ms search kan documents miliyan 100+ mai yiwuwa. Text ana **analyze** shi da farko: tokenized, lowercased, stemmed ("running" → "run"), an zubar da stopwords — domin queries su daidaita da variants.

## Ingestion ta CDC

Kada ka dual-write (app yana rubuta DB *da* ES) — gazawa yana barin su inconsistent. Maimakon haka, stream canje-canje da **Change Data Capture**: **Debezium** yana bin write-ahead log na DB, yana buga row changes zuwa **Kafka**, kuma indexer yana amfani da su ga Elasticsearch. Wannan yana ajiye search **eventually consistent** da source kuma yana tsira daga ES downtime (events suna replay daga Kafka).

## Sharding & relevance

Index ɗin an raba shi zuwa **shards** (kowanne self-contained Lucene index), an baza a kan nodes; **replicas** suna ƙara read throughput da HA. Query yana scatter zuwa dukan shards, kowanne yana dawo da top-k ɗinsa, kuma coordinator yana merge. Relevance yana amfani da **BM25** ta tsohuwa — term frequency × inverse document frequency — mai iya tune da boosts, synonyms, da field weights.

## Ra'ayin hira

Masu yin hira suna bincike ko ka (1) nemi kayan aiki daidai maimakon `LIKE`, (2) san *yadda* inverted index ke aiki, kuma (3) warware **sync problem** ba tare da dual-writes ba. Mai kyau: "amfani da Elasticsearch." Mai kyau sosai: ka ambaci CDC/Debezium don ingestion, BM25 don ranking, kuma ka karɓi eventual consistency a matsayin sulhu don sikeli.

**Tarkon gama gari:** ɗaukar Elasticsearch a matsayin source of truth ɗinka. Derived index ne — ba durable/transactional ba kamar DB ɗinka. Sake gina shi daga source lokacin da ake buƙata.

## Sulhu & hanyoyin gazawa

- **Eventual consistency:** search yana bin writes bayan latency na CDC pipeline (yawanci ƙasa da daƙiƙa).
- **Reindex pain:** canje-canjen mapping/analyzer suna buƙatar full reindex — yi amfani da aliases don zero-downtime swaps.
- **Deep pagination:** page 10,000 mai tsada ne (scatter-gather) — yi amfani da `search_after`, ba `from/size` ba.
- **Hot shard / skew:** rarraba doc mara daidai; zaɓi routing key mai kyau da isassun shards tun farko (an gyara su a ƙirƙira).
- **Relevance tuning na ci gaba ne:** BM25 defaults ba safai suke daidaita da business intent ba — sa ran iteration.

Inverted index — me ya sa yake da sauri

Ingestion ta CDC

Sharding & relevance

Ra'ayin hira

Sulhu & hanyoyin gazawa

Tambayoyi masu alaƙa