How do you fix data skew and hot partitions when sharding?

Question

IT Interview Practice · Accepted Answer

**First identify what is uneven: stored data, request traffic, or updates to one record.** Adding servers helps only when the overloaded work can move to them.

Suppose orders are split by merchant ID. Most merchants are small, but one merchant runs a major campaign. Its shard becomes overloaded while others remain quiet. This is a hot partition: a partition receiving more work than it can handle.

## 1. Measure the imbalance before changing the key

Compare data size, request rate, latency and CPU or disk pressure across shards. Then break down the overloaded shard by merchant and operation. A shard holding old records may be large but idle; a small shard can be overwhelmed by repeated updates to one popular item.

For our campaign, determine whether the load comes from reading product details, creating many independent orders, or repeatedly changing the same inventory record. Those three cases need different remedies.

## 2. Understand why the current distribution stays uneven

The shard key decides where records go. Hashing merchant IDs can spread different merchants across shards, but all orders with the same merchant key still follow the same placement rule. Hashing does not divide one merchant's load automatically.

Keys with few distinct values limit opportunities to split data. A time range can concentrate new writes in the newest partition. A more scattered key can improve distribution but make common queries contact many shards. MongoDB's [shard-key guidance](https://www.mongodb.com/docs/manual/core/sharding-choose-a-shard-key/) explains these considerations.

## 3. Match the remedy to the overloaded work

| Evidence | Direction to evaluate | Important limit |
|---|---|---|
| Many movable partitions sit on one shard | Rebalance them across available capacity | Moving data helps only if the associated work moves too |
| Repeated reads of the same product | Cache or read replicas, if acceptable freshness allows | This does not distribute writes to that product |
| One merchant creates many independent orders | Split that merchant using an additional order-based key or bucket | Merchant-wide queries may need multiple reads and result merging |
| Many writes contend on the same inventory record | Reduce contention or redesign how stock is allocated | More shards do not remove the need to coordinate correctness |

Isolating the large merchant can protect other tenants and buy time. It does not provide unlimited capacity for that merchant, so still measure its individual bottleneck.

## 4. Check the price of splitting a merchant

Imagine routing orders using merchant ID plus a bucket derived from order ID. The campaign's independent orders can then be placed across several partitions.

```text
Merchant A + bucket 0 -> Partition 1
Merchant A + bucket 1 -> Partition 2
Merchant A + bucket 2 -> Partition 3
```

Fetching one order is easier when its bucket can be computed. Fetching all of Merchant A's recent orders now requires reading several partitions and merging them in order. Pagination, totals and transactions across those partitions become more complex. Choose the split only after checking that this extra read and coordination cost is acceptable.

## 5. Migrate with a way to prove correctness

Changing the routing rule while writes continue requires a migration plan. Copy existing records, capture changes made during the copy, and reconcile both record counts and business totals before switching traffic. Counts alone can miss incorrect order values.

Define how in-flight writes are handled at cutover and how routing can be rolled back without losing writes accepted by the new layout. Dual writes need failure handling; simply writing to two places does not guarantee they stay identical.

Finally, test with the actual skewed workload, including the large merchant, rather than uniformly generated traffic. Success means lower pressure on the overloaded shard, acceptable query costs and unchanged business results. **The right key balances workload distribution with how the application reads and updates data.**

Evidence	Direction to evaluate	Important limit
Many movable partitions sit on one shard	Rebalance them across available capacity	Moving data helps only if the associated work moves too
Repeated reads of the same product	Cache or read replicas, if acceptable freshness allows	This does not distribute writes to that product
One merchant creates many independent orders	Split that merchant using an additional order-based key or bucket	Merchant-wide queries may need multiple reads and result merging
Many writes contend on the same inventory record	Reduce contention or redesign how stock is allocated	More shards do not remove the need to coordinate correctness

1. Measure the imbalance before changing the key

2. Understand why the current distribution stays uneven

3. Match the remedy to the overloaded work

4. Check the price of splitting a merchant

5. Migrate with a way to prove correctness

相关问题