Ungeshard vipi database kwa watumiaji bilioni 1?

Question

IT Interview Practice · Accepted Answer

Shard kwa key ambayo **ipo karibu kila query na husambaza load kwa usawa** — kwa kawaida `user_id` iliyohashiwa — kisha panga resharding na cross-shard queries tangu siku ya kwanza. Watumiaji bilioni 1 hawatatoshea wala kufanya kazi kwenye node moja; unagawanya rows kwenye databases nyingi na kuelekeza kila query kwenye shard inayoimiliki.

```text
                 ┌──────────────┐
Request ─▶ Router/│ shard_of(key)│─▶ Shard 0  (users 0–…)
           Proxy  └──────────────┘   Shard 1
           (Vitess)      │           Shard 2
                         └─────────▶  Shard N   ← each = its own primary+replicas
Directory/lookup table (which key → which shard) ── for range or resharding
```

## Chaguo la shard key — uamuzi mgumu kutengua

Chagua key yenye **cardinality ya juu, usambazaji sawa, na iliyopo katika queries nyingi**. `user_id` kwa kawaida hushinda: user-scoped queries ("profile yangu, orders zangu") hupiga shard moja tu. Key mbaya (mfano `country`) hutengeneza hotspots — shard moja hushikilia India yote wakati nyingine imekaa bila kazi.

## Hash vs range

- **Hash sharding** (`shard = hash(user_id) % N`): husambaza kwa usawa, huua hotspots — lakini range queries ("watumiaji walioundwa wiki iliyopita") lazima zifanye scatter-gather kwenye shards zote.
- **Range sharding** (`user_id 0–1M → shard 0`): huweka ranges karibu na hurahisisha resharding, lakini huhatarisha hotspots (watumiaji wapya zaidi wote huingia shard ya mwisho).
- Jibu la kawaida: **hash kwa load sawa**, na **lookup/directory table** ili uweze kuhamisha ranges bila kurehash kila kitu.

## Resharding — panga kabla huhitaji

`% N` ni mtego: kuongeza shard moja huremap karibu kila key. Tumia **consistent hashing** au **virtual buckets** (ramani keys → 4096 logical buckets → physical shards). Kugawa shard kisha huhamisha buckets, si keyspace nzima. Zana kama **Vitess** (inayotumiwa na YouTube/Slack) huautomate live resharding kwa cutover.

## Cross-shard queries

Chochote kisichowekwa key kwa shard key ni chungu: global "top posts" query hufanya fan-out kwenye kila shard na kuunganisha. Punguza kwa **denormalizing** au kuweka index/search store tofauti (Elasticsearch) kwa reads za cross-cutting.

## Mtazamo wa mahojiano

Wahojiwa huchunguza kama unaelewa chaguo la shard-key ni **la nusu-kudumu** na kama umefikiria kuhusu **resharding na cross-shard joins** — mambo mawili yanayouma production. Nzuri: "shard kwa user_id." Bora sana: unaibua consistent hashing, upunguzaji wa hot-shard, na ukweli kwamba distributed transactions kuvuka shards ni bora kuepukwa kabisa.

**Mtego wa kawaida:** `hash % N` yenye N iliyowekwa — hufanya ukuaji wa baadaye kuwa full data migration.

## Maelewano na aina za hitilafu

- **Kupoteza cross-shard ACID:** hakuna joins/transactions nafuu kuvuka shards; tumia sagas au denormalize.
- **Hot shard:** user wa viral; sub-shard au cache entity hiyo.
- **Hatari ya rebalancing:** live migration lazima ifanye double-write na kuthibitisha kabla ya cutover.
- **Gharama ya uendeshaji:** N shards × replicas = databases nyingi za kufuatilia, ku-back up, na ku-fail over.

Chaguo la shard key — uamuzi mgumu kutengua

Hash vs range

Resharding — panga kabla huhitaji

Cross-shard queries

Mtazamo wa mahojiano

Maelewano na aina za hitilafu

Maswali yanayohusiana