Apa iku data partitioning lan sharding?

Question

Accepted Answer

**Data partitioning** (sharding) mbagi data ing sawetara server/database dadi saben siji ngandhut subset — ngidupake horizontal scaling data lan beban ngluwihi siji server. Milih carane partisi (partition key lan strategy) iku kritis.

## Apa iku partitioning/sharding

```text
PARTITIONING / SHARDING → divide data into pieces (partitions/shards) across multiple
servers, each holding a SUBSET:
  → no single server holds (or is overwhelmed by) all the data
  → scales STORAGE and LOAD horizontally (each shard handles its portion)
  → enables handling data/throughput beyond one machine's capacity
```

## Strategi partitioning

```text
HASH-based → hash the partition key → assign to a shard:
  ✓ EVEN distribution (avoids hotspots)  ✗ range queries hard; resharding is tricky
RANGE-based → partition by value ranges (e.g. A-M, N-Z; date ranges):
  ✓ efficient range queries  ✗ risk of HOTSPOTS (uneven load if data/access is skewed)
DIRECTORY/lookup → a lookup table maps keys to shards (flexible, but the lookup is overhead)
GEOGRAPHIC → partition by region (data locality)
```

## Pilihan kritis: partition key

```text
The PARTITION KEY (shard key) is the most important decision:
  ✓ HIGH CARDINALITY + EVEN distribution → spreads data/load evenly (no hot shards)
  ✓ Aligns with QUERY patterns → queries hit one shard (efficient) vs all (scatter-gather)
  ✗ A BAD key → hotspots (one shard overloaded), uneven data, or queries hitting all shards
  → hard to change later → choose carefully
```

## Tantangan

```text
⚠️ CROSS-SHARD queries/joins are hard (data spread across shards) and slow (scatter-gather)
⚠️ REBALANCING / adding shards is complex (moving data)
⚠️ Transactions across shards are difficult; hotspots; operational complexity
→ powerful for scale, but adds significant complexity → use when truly needed
```

## Napa iku penting

Mangerteni data partitioning lan sharding iku berharga amarga iku **teknik utama kanggo scaling data ngluwihi siji server**, tantangan kritis kanggo sistem gede, mula penting banget ilmu system-design.

Partitioning/sharding — mbagi data ing sawetara server dadi saben siji ngandhut subset — ngidupake **horizontal scaling penyimpanan lan beban** ngluwihi kapasitas mesin siji, penting nalika data utawa throughput ngluwihi apa sing siji server bisa tanggung.

Mangerteni **strategi** lan trade-off-e — **hash-based** (distribusi rata ngindhari hotspot, nanging nggawe range query lan resharding susah), **range-based** (efficient range query nanging risiko hotspot saka skew), directory-based (fleksibel karo overhead lookup), lan geographic — perlu kanggo milih carane partisi.

Paling kritis, mangerteni manawa **partition key iku keputusan paling penting** iku insight kunci: key sing apik (high cardinality, distribusi rata, align karo query pattern dadi query mengkene siji shard tinimbang kabeh) nyebar beban rata lan ngidupake query efficient, dene key sing ala nimbulaké hotspot (siji shard kebanjiran), data ora rata, utawa scatter-gather query — lan amarga susah diowah mengko, milih nang-nang-nangan iku penting.

Mangerteni **tantangan** — manawa cross-shard query lan join susah lan alon, rebalancing lan nambahi shard rumit, lan cross-shard transaction susah — penting amarga iki ndadikake sharding kuat nanging rumit banget, mula kudu dienggo nalika bener-bener perlu (sawise scaling luwih gampang kaya caching lan replication).

Amarga scaling data ngluwihi siji server iku tantangan kritis kanggo sistem gede lan partitioning/sharding (karo keputusan partition-key kritis lan trade-off lan tantangan-e) iku teknik-e, lan amarga mangerteni strategi, pentinge partition-key, lan tantangan-e penting kanggo ndesain sistem gede, mangerteni data partitioning lan sharding iku berharga, praktis-relevan ilmu system-design — teknik kunci kanggo horizontal data scaling, butuh pangertian teliti babagan partition strategy, pilihan partition-key kritis, lan tantangan signifikan, lan pusat kanggo ndesain sistem sing scaling data ngluwihi kapasitas server siji.