What is mutation testing?

Question

Accepted Answer

**Mutation testing** evaluates the **quality of a test suite** by deliberately introducing small bugs ("mutants") into the code and checking whether the tests **catch them**. It measures how good tests actually are at detecting defects — going beyond coverage, which only measures execution.

## How mutation testing works

```text
1. A tool makes small changes (MUTATIONS) to the code, e.g.:
   → change + to -, > to >=, true to false, remove a statement, change a constant
   → each mutation = a deliberate "bug" (a MUTANT version of the code)
2. Run the TEST SUITE against each mutant:
   → tests FAIL → the mutant is "KILLED" (good — tests caught the bug) ✓
   → tests PASS → the mutant "SURVIVED" (bad — tests DIDN'T catch the bug) ✗
3. MUTATION SCORE = % of mutants killed → measures how good the tests are at catching bugs
```

## Why it's better than coverage

```text
COVERAGE → measures if code was EXECUTED (not if behavior was verified) → can be high with
  weak tests (no real assertions)
MUTATION TESTING → measures if tests actually DETECT bugs (changes in behavior):
  → a surviving mutant reveals a GAP — tests execute that code but don't verify it well
    (weak/missing assertions, untested behavior)
→ A much stronger signal of TEST QUALITY than coverage.
```

## Considerations

```text
✓ Reveals weak tests (high coverage but surviving mutants = false confidence)
⚠️ COMPUTATIONALLY EXPENSIVE (runs the suite many times — once per mutant) → slow
⚠️ "Equivalent mutants" (mutations that don't change behavior) → false survivors to ignore
→ Tools: Stryker (JS), PIT (Java), mutmut (Python). Use on critical code; run periodically.
```

## Why it matters

Understanding mutation testing is valuable senior-level knowledge because it provides a **much stronger measure of test quality than coverage**, addressing coverage's key weakness, so it's useful for evaluating and improving test suites.

The fundamental insight is that **coverage measures execution, not verification** — code can be "covered" (executed) by tests that don't actually verify its behavior (weak or missing assertions), giving false confidence. **Mutation testing** addresses this by measuring whether tests actually **detect bugs**: it introduces small deliberate bugs (mutants) into the code and checks if the tests catch them (a mutant "killed" by failing tests is good; a "surviving" mutant means the tests didn't catch that bug).

The **mutation score** (percentage of mutants killed) is a far stronger signal of test quality than coverage because it reveals whether tests genuinely verify behavior — a **surviving mutant exposes a real gap** (tests execute the code but don't verify it well, e.g. weak assertions or untested behavior) that coverage would miss.

Understanding this — that mutation testing reveals weak tests hiding behind high coverage — is the key value, addressing the limitation that high coverage doesn't guarantee good tests.

Understanding the **considerations** — that it's **computationally expensive** (running the suite once per mutant, making it slow), that "equivalent mutants" (mutations not changing behavior) are false survivors to ignore, and that tools exist (Stryker, PIT, mutmut) — reflects practical awareness of when and how to use it (on critical code, periodically, rather than constantly).

Since test quality genuinely matters (coverage alone can give false confidence) and mutation testing provides a much stronger measure by checking whether tests actually catch bugs, and since understanding it, its advantage over coverage, and its practical considerations is useful for truly evaluating and improving test suites, understanding mutation testing is valuable senior-level knowledge — a sophisticated technique for measuring real test quality, addressing coverage's key weakness, and reflecting the deeper understanding of test effectiveness expected for senior roles concerned with genuine test quality, not just coverage numbers.