Unaendeshaje kazi za usuli imara (Celery/ARQ)?

Question

Accepted Answer

Kwa **kazi za usuli imara na za kudumu** (zaidi ya `BackgroundTasks` nyepesi ya FastAPI), unatumia foleni halisi ya kazi (task queue): **Celery** (kiwango kilichoanzishwa) au **ARQ** (foleni ya kisasa ya async-native). Hizi huendesha kazi nzito, zinazoweza kujaribiwa tena, na zilizopangwa katika michakato tofauti ya wafanyakazi (worker processes), zikiungwa mkono na broker kama Redis.

## Kwa nini si BackgroundTasks tu?

```text
BackgroundTasks runs in the web process → NO persistence (lost on crash), NO retries,
NO scheduling, and heavy work ties up the worker. Fine only for quick fire-and-forget.
For durable/critical/heavy/scheduled jobs → use a real task queue (Celery / ARQ).
```

## Celery — fafanua na panga kazi kwenye foleni

```python
# tasks.py
from celery import Celery
celery_app = Celery("tasks", broker="redis://localhost:6379")

@celery_app.task(bind=True, max_retries=3)
def process_video(self, video_id: int):
    try:
        do_heavy_processing(video_id)        # runs in a WORKER, not the web process
    except Exception as exc:
        raise self.retry(exc=exc, countdown=60)   # automatic retry on failure
```

```python
# in a FastAPI endpoint — QUEUE the task, return immediately
@app.post("/videos/{id}/process")
def process(id: int):
    process_video.delay(id)                  # enqueue → returns at once (doesn't block)
    return {"status": "processing started"}
# run workers separately: celery -A tasks worker
```

Endpoint huweka kazi kwenye foleni na hujibu mara moja; mchakato tofauti wa **worker** huiendesha ukiwa na uthabiti (persistence) na majaribio upya (retries).

## ARQ — async-native (inafaa modeli ya async ya FastAPI)

```python
# ARQ is built on asyncio — a natural fit for async FastAPI codebases
async def process_video(ctx, video_id: int):
    await do_async_processing(video_id)

# enqueue from an endpoint
await redis.enqueue_job("process_video", video_id)
```

ARQ ni mbadala mwepesi, wa async-kwanza — wenye mvuto wakati msingi wako wa msimbo tayari ni async.

## Foleni halisi ya kazi hutoa nini

```text
✓ Persistence — jobs survive crashes/restarts (stored in the broker)
✓ Retries with backoff — resilience against transient failures
✓ Scheduling — periodic/cron-like jobs (Celery Beat)
✓ Distribution & scaling — workers scale independently of the web servers
✓ Concurrency & rate limiting, task routing/priorities, result tracking
```

## Kwa nini ni muhimu

Kuendesha kazi za usuli imara ni jambo la lazima kwa kujenga programu za FastAPI za kiwango cha production zinazoshughulikia kazi nzito au zilizoahirishwa kwa uaminifu — na kuelewa wakati na jinsi ya kutumia foleni halisi ya kazi ni maarifa muhimu ya kiwango cha senior.

Uamuzi wa msingi ni kutambua **mipaka ya `BackgroundTasks`** (hakuna persistence, retries, au scheduling — inafaa tu kwa kazi za haraka, zisizo muhimu) na kuelekea **Celery au ARQ** wakati kazi lazima iwe ya kudumu, inayoweza kujaribiwa tena, iliyopangwa, nzito, au iliyosambazwa (kuchakata faili/video, kutuma barua pepe nyingi, kusawazisha data, kutengeneza ripoti).

Foleni hizi hutoa uwezo muhimu wa production: **persistence** (kazi hudumu baada ya ajali kupitia broker, hivyo kazi haipotei), **majaribio upya ya kiotomatiki** (uimara dhidi ya kushindwa kwa muda), **scheduling** (kazi za mara kwa mara), na **upanuzi wa kujitegemea** wa workers kutoka kwa web servers.

Kujua jinsi ya kufafanua na kuweka kazi kwenye foleni (`.delay()` ya Celery, API ya async-native ya ARQ), usanifu (broker + workers), na hasa **uamuzi wa wakati wa kutumia foleni halisi dhidi ya BackgroundTasks**, ni jambo la thamani kwa kubuni programu za kuaminika zinazohamisha kazi nzito bila kuipoteza — hitaji la kawaida na muhimu linalotofautisha mifumo imara ya production na ile dhaifu.