Yaya za ka tsara sarrafa bayanai masu yawa?

Question

IT Interview Practice · Accepted Answer

**Fara da sakamakon da ake so, lokacin da ake buƙatarsa da ƙa’idojin correctness kafin zaɓar big-data tool.** Babban dataset ba ya nufin distributed processing framework ya zama dole.

A ce kasuwanci yana buƙatar revenue na jiya bisa merchant kowace safiya. Wannan ya bambanta da gano suspicious payment kafin amincewa da shi. Dukansu na iya amfani da records da yawa, amma na farko yana barin scheduled processing yayin da na biyu yake buƙatar shawara cikin lokaci.

## 1. Bayyana abin da za a sarrafa

San yawan data, saurin zuwan sabuwa, fields da ake buƙata da tsawon retention. Ga revenue report, a amince ko refunds suna sauya ranar asalin sayayya ko ranar refund. Ba tare da wannan ƙa’ida ba, pipeline mai sauri na iya bada total marar kyau.

Bayyana source of truth da yadda za a kula da late ko corrected records. Waɗannan shawarar suna nuna ko tsohon result zai iya zama final ko sai an sake gyara shi daga baya.

## 2. Rage aiki kafin rarraba shi

Auna inda lokaci da resources suke tafiya: file reads, database queries, computation, network transfer ko rubuta results. Sannan tace irrelevant records da wuri kuma karanta columns da ake buƙata kawai.

Process bounded chunks maimakon loda komai a memory. Idan worker ɗaya zai iya kammala kafin deadline da kuɗin da ya halatta, wannan na iya zama tsarin mafi sauƙin gudanarwa. Ƙarin machines suna ƙara coordination da data-transfer costs, don haka ƙara su don warware iyakar da aka gani.

## 3. Zaɓi batch ko streaming bisa deadline

| Buƙata | Hanyar farawa | Dalili da alhaki |
|---|---|---|
| Cikakken rahoto kowace safiya | Scheduled batch | Finite input ya fi sauƙin reconcile da rerun; dole ya gama kafin deadline |
| Dashboard mai sabuntawa a rana | Incremental batches ko streaming | Sarrafa sabuwar data akai-akai; bayyana yadda corrections za su sauya tsofaffin totals |
| Shawara jim kaɗan bayan kowane event | Streaming | Rage delay tare da kula da duplicates, late events da recovery |

Streaming yana nufin ci gaba da processing yayin zuwan data, ba ɗaukar cewa kowane event yana zuwa sau ɗaya a tsari ba. Ga daily report, zan fara da batch sai idan kasuwanci yana buƙatar fresher results da gaske.

## 4. Sanya partial failure ya zama safe

Ka ɗauka worker ya rubuta revenue total sannan ya crash kafin adana progress. Bayan restart, zai iya sake process records ɗin. Idan ya sake ƙara amounts, revenue zai yi yawa fiye da gaske.

Yi stable record identifiers da writes da ake iya maimaitawa lafiya, ko haɗa output da progress tracking transactionally inda aka tallafa. Checkpoint yana rubuta aikin da aka gama; adana shi kafin output ya zauna lafiya na iya sa a tsallake aiki bayan crash. Adana shi daga baya yana buƙatar safe replay na gibin.

- Riƙe source data tsawon lokacin da ya isa replay na zangon da ake bukata.
- Ware invalid records don bincike maimakon jefar da su a ɓoye.
- Takaita saurin workers zuwa destination mai jinkiri. Wannan backpressure yana hana memory buffer ko queue ƙaruwa ba tare da iyaka ba.

Recovery behavior ɓangare ne na design, ba aikin gyara bayan failure na farko ba.

## 5. Rarraba aikin da yake bukatar hakan kawai

Idan worker ɗaya bai isa ba, raba input zuwa sassa da za su iya gudana da kansu. Duba data distribution na gaske: merchant ɗaya mai girma zai iya barin worker ɗaya yana aiki yayin da sauran sun gama.

Grouping da joining na iya buƙatar motsa records tsakanin workers, abin da ake kira shuffle. Rage data kafin wannan motsi inda zai yiwu kuma a tabbatar destination zai iya karɓar combined output. Spark yana da dabaru masu amfani amma dacewarsu ya danganta da query da version; duba [performance tuning guide](https://spark.apache.org/docs/3.5.6/sql-performance-tuning.html).

Storage ma ya bi access patterns. Column-oriented files kamar Parquet na iya dacewa da analytical scans kan fields da aka zaɓa, yayin da indexed database zai iya fi dacewa da individual operational lookups. Zaɓi bisa access needs da aka auna, ba file size kawai ba.

## 6. Tabbatar da speed da correctness tare

Kwatanta result da ƙaramin trusted dataset, ka duba record counts da business totals. Haɗa duplicate events, refunds, late records da crash tsakanin rubuta output da adana progress.

Sannan gwada realistic volume da skew tare da auna runtime, peak memory, backlog da cost. Ga revenue report, nasara ita ce correct total da za a iya sake samarwa kafin deadline da aka amince da shi. **Zaɓi pipeline mafi sauƙi da ya cika waɗannan sharuɗɗa kuma zai iya recovery ba tare da rasa data ko lissafa ta sau biyu ba.**

Buƙata	Hanyar farawa	Dalili da alhaki
Cikakken rahoto kowace safiya	Scheduled batch	Finite input ya fi sauƙin reconcile da rerun; dole ya gama kafin deadline
Dashboard mai sabuntawa a rana	Incremental batches ko streaming	Sarrafa sabuwar data akai-akai; bayyana yadda corrections za su sauya tsofaffin totals
Shawara jim kaɗan bayan kowane event	Streaming	Rage delay tare da kula da duplicates, late events da recovery

1. Bayyana abin da za a sarrafa

2. Rage aiki kafin rarraba shi

3. Zaɓi batch ko streaming bisa deadline

4. Sanya partial failure ya zama safe

5. Rarraba aikin da yake bukatar hakan kawai

6. Tabbatar da speed da correctness tare

Tambayoyi masu alaƙa