Unajengaje full-text search juu ya records 100M+?

Question

IT Interview Practice · Accepted Answer

Tumia search engine maalum — **Elasticsearch/OpenSearch iliyojengwa juu ya inverted index** — inayolishwa kutoka source DB yako kupitia **CDC**, na shard index kwa scale. Database yako ya msingi (Postgres/MySQL) haiwezi kufanya relevance-ranked text search juu ya mamia ya mamilioni ya rows; `LIKE '%term%'` huscan kila row. Search engine hubadilisha tatizo.

```text
Source DB ─▶ CDC (Debezium) ─▶ Kafka ─▶ Indexer ─▶ Elasticsearch/OpenSearch
(Postgres)   (change stream)                        ├─ Shard 0 (inverted index) + replicas
                                                     ├─ Shard 1
Query ─▶ Coordinator ─▶ scatter to shards ─▶ gather ─┴─ Shard N ── ranked by BM25
```

## Inverted index — kwa nini ni haraka

Index ya kawaida huramani row → values. **Inverted index** huramani *term → orodha ya documents zenye term hiyo*. Kutafuta "database scaling" hutafuta posting lists mbili fupi na kuziintersect — O(matching docs), si O(all docs). Hii ndiyo hufanya search ya chini ya 100ms juu ya documents 100M+ iwezekane. Text huchanganuliwa (analyzed) kwanza: tokenized, lowercased, stemmed ("running" → "run"), stopwords hudondoshwa — ili queries zilingane na variants.

## Ingestion kupitia CDC

Usifanye dual-write (app huandika DB *na* ES) — kushindwa huacha zikiwa hazilingani. Badala yake, stream mabadiliko kwa **Change Data Capture**: **Debezium** hufuatilia write-ahead log ya DB, huchapisha row changes kwenda **Kafka**, na indexer huzitumia kwenye Elasticsearch. Hii huweka search **eventually consistent** na source na huokoka ES downtime (events hufanya replay kutoka Kafka).

## Sharding na relevance

Index hugawanywa kuwa **shards** (kila moja Lucene index inayojitosheleza), zilizosambazwa kwenye nodes; **replicas** huongeza read throughput na HA. Query huscatter kwenye shards zote, kila moja hurudisha top-k yake, na coordinator huunganisha. Relevance hutumia **BM25** kwa default — term frequency × inverse document frequency — inayoweza kutunwa kwa boosts, synonyms, na field weights.

## Mtazamo wa mahojiano

Wahojiwa hukagua kama (1) unakimbilia chombo sahihi badala ya `LIKE`, (2) unajua *jinsi* inverted index inavyofanya kazi, na (3) unatatua **tatizo la sync** bila dual-writes. Nzuri: "tumia Elasticsearch." Bora sana: unataja CDC/Debezium kwa ingestion, BM25 kwa ranking, na kukubali eventual consistency kama maelewano ya scale.

**Mtego wa kawaida:** kutendea Elasticsearch kama source of truth yako. Ni index inayotokana — si durable/transactional kama DB yako. Ijenge upya kutoka source inapohitajika.

## Maelewano na aina za hitilafu

- **Eventual consistency:** search huchelewa nyuma ya writes kwa latency ya CDC pipeline (kwa kawaida chini ya sekunde).
- **Reindex pain:** mabadiliko ya mapping/analyzer yanahitaji full reindex — tumia aliases kwa zero-downtime swaps.
- **Deep pagination:** page 10,000 ni ghali (scatter-gather) — tumia `search_after`, si `from/size`.
- **Hot shard / skew:** usambazaji wa docs usio sawa; chagua routing key nzuri na shards za kutosha mapema (zimewekwa wakati wa creation).
- **Relevance tuning ni ya kuendelea:** BM25 defaults mara chache hulingana na business intent — tarajia iteration.

Inverted index — kwa nini ni haraka

Ingestion kupitia CDC

Sharding na relevance

Mtazamo wa mahojiano

Maelewano na aina za hitilafu

Maswali yanayohusiana