How would you design a scheduler for 10M timed tasks?

Question

IT Interview Practice · Accepted Answer

Store tasks in a **time-bucketed / priority queue** keyed by due time, poll only the tasks due *now*, and hand them to idempotent workers with **at-least-once** delivery. Never scan 10M rows every tick — index by fire time so each poll touches only what is due.

## Architecture

```text
API ─▶ Task store (DB/Redis ZSET, score = due_ts)  ── 10M tasks, sharded
             │
     Poller (leader per partition) ── ZPOPMIN due<=now ──▶ Ready queue (Kafka)
             │ leader election (etcd/ZK)                       │
             ▼                                                 ▼
     other pollers idle/standby                       Worker pool (idempotent)
                                                        └─ ack ✔ / retry ✖ ─▶ DLQ
```

## Components and flow

- **Task store indexed by fire time**: a Redis sorted set (`ZADD tasks score=due_ts`) or a DB table indexed on `(run_at)`. `ZPOPMIN`/`ZRANGEBYSCORE now` pulls only tasks due this second — O(log n), not a table scan. **Sharding/partitioning** by task-id spreads 10M tasks across nodes so no single store is hot.
- **Poller with leader election**: each partition has one active poller elected via etcd/ZooKeeper; standbys take over on failure. One leader per partition avoids two pollers dispatching the same task, while partitioning keeps throughput high.
- **Ready queue (Kafka)**: due tasks are enqueued, decoupling *scheduling* from *execution*. Workers scale independently of the poller.
- **Idempotent workers** execute and **ack**. Because delivery is at-least-once, a task may run twice (poller crashed after dispatch, before marking done) — so the work itself must be idempotent, keyed on `task_id`, or you double-charge a customer. Failures retry with backoff, then land in a DLQ.

## Interview perspective

The interviewer probes two things: **efficient time lookup** and **exactly-once vs at-least-once**. "Good" is a cron loop over a table; "great" is the sorted-set index plus the honest admission that distributed exactly-once is impractical, so you choose **at-least-once + idempotent workers**. A classic trap: relying on a single scheduler node — it is a SPOF and a throughput ceiling. Another: polling too coarsely (every minute) when tasks need second precision, or too finely and hammering the store.

## Trade-offs and failure

- **Precision vs load**: 1s tick gives second-level accuracy but more polls; bucket tasks into per-minute then per-second sub-queues to cut scan cost (the "timing wheel" / hashed-wheel approach used by Kafka and Quartz).
- **At-least-once vs exactly-once**: you cannot get true exactly-once across a crash boundary; idempotency keys make double execution harmless — the right trade.
- **Failure**: if a poller dies mid-dispatch, leader election promotes a standby and un-acked tasks redeliver — nothing is lost, some may repeat. Watch **clock skew** across nodes (NTP) or tasks fire early/late; and cap retries so a permanently failing task doesn't loop forever — send it to the DLQ for inspection.

Architecture

Components and flow

Interview perspective

Trade-offs and failure

Related questions