How do you speed up bulk indexing without overloading the cluster?

Question

IT Interview Practice · Accepted Answer

Bulk sends several indexing operations in one request. Improve the rate of successful operations while keeping the cluster responsive, rather than maximizing the number of requests sent.

## Tune one dimension at a time

Start with modest batches of representative documents. Increase batch bytes until throughput stops improving or memory and latency worsen. Then tune the number of concurrent senders separately. A thousand small events and a thousand large documents are very different batches.

## A successful HTTP response can contain failed items

Consider this simplified illustrative bulk response:

```json
{
  "errors": true,
  "items": [
    {"index":{"_id":"a","status":201}},
    {"index":{"_id":"b","status":429}},
    {"index":{"_id":"c","status":400}}
  ]
}
```

- Item a succeeded; do not resubmit it merely because another item failed.
- Inspect b's error details. A temporary overload rejection calls for slowing down and a bounded retry after a delay.
- Inspect c's details too. A permanent mapping error needs data or mapping correction, not endless retries.

## Keep retries from becoming another overload

Increase retry delays progressively and add a random offset so clients do not all retry together. Bound the queue and slow producers when it fills; this is backpressure.

After an uncertain network outcome, stable document IDs help avoid duplicate records, but scripted increments and stale updates need additional safeguards. Monitor rejected writes, memory, disk and search latency. Reduce refresh frequency only if the product accepts less immediate search visibility.

[Official documentation](https://www.elastic.co/docs/deploy-manage/production-guidance/optimize-performance/indexing-speed).

Tune one dimension at a time

A successful HTTP response can contain failed items

Keep retries from becoming another overload

Related questions