Ta yaya za ka tsara scheduler ga ayyuka miliyan 10 masu lokaci (timed tasks)?

Question

IT Interview Practice · Accepted Answer

Adana ayyuka a **time-bucketed / priority queue** da aka key da lokacin da suka kamata, ka poll ayyukan da suka kamata *yanzu* kawai, kuma ka miƙa su ga idempotent workers da **at-least-once** delivery. Kada ka taɓa scan rows miliyan 10 kowane tick — yi index da fire time domin kowane poll ya taɓa abin da ya kamata kawai.

## Gine-gine

```text
API ─▶ Task store (DB/Redis ZSET, score = due_ts)  ── 10M tasks, sharded
             │
     Poller (leader per partition) ── ZPOPMIN due<=now ──▶ Ready queue (Kafka)
             │ leader election (etcd/ZK)                       │
             ▼                                                 ▼
     other pollers idle/standby                       Worker pool (idempotent)
                                                        └─ ack ✔ / retry ✖ ─▶ DLQ
```

## Sassa da gudanawa

- **Task store da aka index da fire time**: Redis sorted set (`ZADD tasks score=due_ts`) ko DB table da aka index a kan `(run_at)`. `ZPOPMIN`/`ZRANGEBYSCORE now` yana ja ayyukan da suka kamata a wannan daƙiƙa kawai — O(log n), ba table scan ba. **Sharding/partitioning** da task-id yana baza ayyuka miliyan 10 a kan nodes domin babu store ɗaya da ke zama mai zafi.
- **Poller da leader election**: kowace partition tana da active poller ɗaya da aka zaɓa ta etcd/ZooKeeper; standbys suna karɓa a gazawa. Leader ɗaya ga kowace partition yana hana pollers biyu dispatch ɗin task ɗaya, yayin da partitioning ke ci gaba da throughput mai yawa.
- **Ready queue (Kafka)**: ayyukan da suka kamata ana shigar da su cikin queue, suna raba *scheduling* daga *execution*. Workers suna scale ba tare da poller ba.
- **Idempotent workers** suna aiwatarwa kuma suna **ack**. Domin delivery at-least-once ne, task na iya gudana sau biyu (poller ya faɗi bayan dispatch, kafin markawa done) — don haka aikin kansa dole ya zama idempotent, da aka key a kan `task_id`, ko za ka caje abokin ciniki sau biyu. Gazawa tana retry da backoff, sannan tana sauka a DLQ.

## Ra'ayin hira

Mai yin hirar yana bincike abubuwa biyu: **efficient time lookup** da **exactly-once vs at-least-once**. "Mai kyau" shi ne cron loop kan table; "mai kyau sosai" shi ne sorted-set index tare da amincewa da gaskiya cewa distributed exactly-once ba mai amfani ba ne, don haka ka zaɓi **at-least-once + idempotent workers**. Tarkon gargajiya: dogaro da scheduler node ɗaya — SPOF ne kuma rufin throughput. Wani kuma: polling mai kauri sosai (kowane minti) lokacin da ayyuka suke buƙatar daidaiton daƙiƙa, ko mai kyau sosai da ke bugun store.

## Sulhu da gazawa

- **Precision vs load**: 1s tick yana ba da daidaiton matakin daƙiƙa amma polls masu yawa; yi bucket na ayyuka zuwa per-minute sannan per-second sub-queues don rage scan cost (hanyar "timing wheel" / hashed-wheel da Kafka da Quartz ke amfani da su).
- **At-least-once vs exactly-once**: ba za ka iya samun ainihin exactly-once a kan iyakar crash ba; idempotency keys suna mai da double execution mara illa — sulhun daidai.
- **Gazawa**: idan poller ya mutu a tsakiyar dispatch, leader election yana ɗaga standby kuma ayyukan da ba a ack redeliver — babu abin da ya ɓace, wasu na iya maimaituwa. Kula da **clock skew** a kan nodes (NTP) ko ayyuka za su fara da wuri/lokacinsu ya wuce; kuma ka iyakance retries domin task da ke gazawa har abada kada ya yi loop har abada — tura shi zuwa DLQ don bincike.

Gine-gine

Sassa da gudanawa

Ra'ayin hira

Sulhu da gazawa

Tambayoyi masu alaƙa