Definition
Mutation testing generates variants of a program (mutants), each carrying one deliberately introduced defect, and measures whether the existing tests fail on them. What is being measured is not the product code but the defect-detection strength of the tests.
Why it matters
Coverage counts whether a line was executed. A line that was executed but never checked still counts as 100%.
it("creates an order", async () => {
await createOrder({ items });
// no assertion — coverage goes up, verification is zero
});
Mutation testing inverts the question. Instead of "did the test run this code?", it asks "does the test turn red when this code is wrong?". Flip > to >=, && to ||, return null instead of a value — if everything stays green, that is direct evidence the tests are not guarding that spot.
It is especially good at finding tests whose assertions were quietly loosened under pressure to make the build pass. Unlike coverage, you cannot game it by adding lines.
How it works
- Apply mutation operators to the original code to generate many single-defect variants.
- Run the relevant tests against each variant.
- If a test fails, the mutant is killed; if all pass, it survived.
- Compute
mutation score = killed / (total − equivalent).
Representative mutation operators:
| Operator | Example mutation | What a survivor implies |
|---|---|---|
| Conditional boundary | a > b → a >= b | No boundary-value test |
| Logical operator | && → || | Some branch of a compound condition is unverified |
| Return value | return x → return null | Nobody checks the return value |
| Call removal | delete logAudit() | Side effects are not verified |
| Constant change | 0 → 1 | No check on initial values or offsets |
An equivalent mutant changes the code but produces observably identical behavior, so it can never be killed. Detecting them automatically is impossible (equivalent to the halting problem), so a human must exclude them with a stated reason.
Applying it
Most tools support running only against changed files. Given how slow a full run is, this option is effectively mandatory.
# JS/TS: Stryker
npx stryker run --incremental --mutate "src/domain/**/*.ts"
When a mutant survives, add a test that guards that spot, then revert the defect and confirm the suite goes green again.
// survived: nobody caught `total > limit` → `total >= limit`
it("allows an amount exactly equal to the limit", () => {
expect(isOverLimit(1000, 1000)).toBe(false);
});
it("blocks anything above the limit", () => {
expect(isOverLimit(1001, 1000)).toBe(true);
});
A workable setup is to run it on changed files in CI and keep the full run as a nightly job.
Trade-offs
The dominant cost is time: mutants × test duration, which is trivially tens to hundreds of times your suite. Parallelism, change-scoping, and selecting only the tests that cover a mutant reduce it, but it stays heavy.
The second cost is human time spent classifying equivalent mutants. Because of them the score never reaches 100%, and treating 100% as the goal sends the team into meaningless work.
What you get is information no other metric provides. Coverage tells you where you did not look; mutation testing tells you where you looked but cannot catch anything, down to the line.
When not to use it
- A codebase with flaky tests. You cannot tell whether a failure came from the defect or the flake, which makes the whole result meaningless. Stabilize first.
- Before the judgment criteria are settled. Killing mutants against wrong expectations only cements wrong behavior more firmly. test-oracle is a prerequisite.
- UI rendering, log formats, experimental code — areas that change often and where correctness costs little. The runtime cost outweighs the benefit.
- An organization that wants to enforce the score as a KPI. Padding the number with meaningless tests reproduces exactly the distortion coverage caused.
Common mistakes
- Running against the whole codebase at once. Results take hours and nobody reads them. Start narrowly, at high-risk boundaries.
- Skipping the revert check after killing a mutant. This is where tests that also fail on correct code get introduced.
- Writing off every survivor as equivalent. Without a stated reason, the next person repeats the whole judgment from scratch.
- Treating mutation score as a coverage replacement. The useful output is not the score but the list of surviving mutants — that list is the map of your testing gaps.
Related concepts
- test-oracle — the question of what correct means; mutation testing verifies that the judgment actually works
- react-render-count-isolation-testing — another case of making assertions concrete enough to detect defects