Describe your worst production incident and the first 30 minutes.

Question

IT Interview Practice · Accepted Answer

**A strong answer shows how you reduced customer harm, made decisions with incomplete evidence and prevented recurrence.** Describe your own role clearly rather than presenting the team's work as entirely your own.

Use an incident you actually experienced. The scenario below is fictional and demonstrates the structure; replace it with your real actions and verified results in an interview. Its time windows are illustrative, not a universal incident-response deadline.

## 1. Establish the situation and your responsibility

Suppose I was the on-call backend lead when checkout failures increased shortly after a release. Customers could browse but some could not complete orders. At this point, the timing made the release a hypothesis, not a proven cause.

I would explain my responsibility: coordinate technical recovery, assign investigation work and keep the incident lead informed. If I also led the incident, I would say so explicitly and delegate communication or hands-on investigation to avoid becoming a bottleneck.

## 2. First five minutes: confirm impact and organize the response

Check checkout errors, latency and successful orders to establish who is affected and whether the problem is ongoing. Open one incident channel, assign responsibilities, pause unrelated changes and start a timeline of observations and actions.

The reason for this organization is practical: several people making independent changes can obscure the cause or worsen the incident. An update should distinguish confirmed facts from hypotheses and state when the next update will arrive. See Google's [incident-response guidance](https://sre.google/workbook/incident-response/).

## 3. Next ten minutes: choose a safe way to reduce harm

Suppose investigation finds that database connections are saturated and the release added an expensive reporting path. I would compare rollback with temporarily disabling that path while keeping checkout available.

- Check whether rollback is compatible with any schema or data changes already made. A previous application version is not automatically safe against the current database.
- If a tested switch can disable optional reporting, use it to reduce load and observe whether checkout improves.
- Assign someone to examine the connection pressure while another verifies the recovery path, so investigation continues without delaying a reasonable mitigation.

The decision should follow available evidence and reversibility. I would not wait for complete root-cause certainty while customers remain blocked, but I would record why the selected action was expected to help.

## 4. Up to thirty minutes: verify useful service is restored

A falling CPU graph is insufficient. Confirm that real checkout attempts succeed, latency returns to an acceptable range and queued work is draining without creating new pressure. Watch the system after mitigation rather than declaring recovery after a single successful request.

For ambiguous payment outcomes, reconcile with the payment system before retrying charges. A timeout can hide a completed payment. Communicate what has recovered, what remains uncertain and what customers should do, without inventing a recovery time that the evidence cannot support.

## 5. After recovery: change the conditions that allowed the failure

In this fictional scenario, the investigation might show that reporting and checkout shared a connection budget and reports had no effective work limit. The follow-up should address that mechanism rather than end with “be more careful.”

- Bound report work and concurrency so one feature cannot consume the entire shared resource.
- Evaluate resource isolation where needed and test reporting alongside checkout under representative load.
- Alert on checkout success and latency, and rehearse the mitigation or rollback procedure.
- Give each corrective action an owner, a deadline and a way to verify completion.

In a real interview, close with the actual recovery result and evidence from follow-up tests or later operation. If a measurement was not collected, say that rather than supplying an impressive number. The lesson should connect directly to a change you helped deliver and verify.

1. Establish the situation and your responsibility

2. First five minutes: confirm impact and organize the response

3. Next ten minutes: choose a safe way to reduce harm

4. Up to thirty minutes: verify useful service is restored

5. After recovery: change the conditions that allowed the failure

相关问题