How do you approach processing a large volume of data?

Question

IT Interview Practice · Accepted Answer

**Start with the result, its deadline and the correctness rules before choosing a big-data tool.** A large dataset does not automatically require a distributed processing framework.

Suppose the business needs yesterday's revenue by merchant each morning. That differs from detecting a suspicious payment before approving it. Both may involve many records, but one allows scheduled processing while the other needs a timely decision.

## 1. Define what must be processed

Clarify how much data exists, how quickly new data arrives, which fields are needed and how long data must be retained. For the revenue report, agree whether refunds change the original day or the day of the refund. Without that rule, a faster pipeline can still produce the wrong total.

Also define the source of truth and how late or corrected records are handled. These decisions determine whether an old result can be finalized or must be revised later.

## 2. Reduce the work before distributing it

Measure where time and resources go: reading files, database queries, computation, network transfer or writing results. Then filter irrelevant records early and read only the columns needed for the report.

Process bounded chunks instead of loading everything into memory. If one worker can finish within the deadline at acceptable cost, that may be the easiest system to operate. More machines introduce coordination and data-transfer costs, so add them to solve an observed limit.

## 3. Choose batch or streaming from the deadline

| Need | Starting approach | Reason and responsibility |
|---|---|---|
| A complete report each morning | Scheduled batch | A finite input is easier to reconcile and rerun; it must finish before the deadline |
| A dashboard updated throughout the day | Incremental batches or streaming | Process new data more frequently; define how corrections update earlier totals |
| Decisions shortly after each event | Streaming | Reduce delay, while handling duplicates, late events and recovery |

Streaming means processing arriving data continuously, not assuming that every event arrives once and in order. For our daily report, I would start with a batch unless the business actually needs fresher results.

## 4. Make partial failure safe

Imagine the worker writes a revenue total and crashes before recording its progress. On restart, it may process the same records again. If it simply adds their amounts again, revenue is overstated.

Use stable record identifiers and writes that can be repeated safely, or coordinate output and progress tracking transactionally where supported. A checkpoint records completed progress; saving it before the output is safely stored can skip work after a crash. Saving it afterward requires safe replay of the gap.

- Retain source data long enough to replay the required period.
- Separate invalid records for investigation rather than silently dropping them.
- Limit how quickly workers feed a slower destination. This backpressure prevents an ever-growing memory buffer or queue.

Recovery behavior is part of the design, not a cleanup task after the first failure.

## 5. Distribute only the work that needs it

When a single worker is insufficient, partition the input into pieces that can run independently. Check actual data distribution: one very large merchant can leave one worker busy while the rest finish early.

Grouping and joining may require moving records between workers, called a shuffle. Reduce data before that movement when possible and verify that the destination can accept the combined output. Spark offers relevant techniques, but their applicability depends on the query and version; see its [performance tuning guide](https://spark.apache.org/docs/3.5.6/sql-performance-tuning.html).

Storage should follow access patterns too. Column-oriented files such as Parquet can suit analytical scans over selected fields, while an indexed database may better serve individual operational lookups. Choose based on measured access needs rather than file size alone.

## 6. Prove speed and correctness together

Compare the result with a trusted small dataset, checking both record counts and business totals. Include duplicate events, refunds, late records and a crash between writing output and saving progress.

Then test realistic volume and skew while measuring runtime, peak memory, backlog and cost. For the revenue report, success means the correct, reproducible total arrives before the agreed deadline. **Choose the simplest pipeline that meets those conditions and can recover without losing or double-counting data.**

Need	Starting approach	Reason and responsibility
A complete report each morning	Scheduled batch	A finite input is easier to reconcile and rerun; it must finish before the deadline
A dashboard updated throughout the day	Incremental batches or streaming	Process new data more frequently; define how corrections update earlier totals
Decisions shortly after each event	Streaming	Reduce delay, while handling duplicates, late events and recovery

1. Define what must be processed

2. Reduce the work before distributing it

3. Choose batch or streaming from the deadline

4. Make partial failure safe

5. Distribute only the work that needs it

6. Prove speed and correctness together

متعلقہ سوالات