Definition
A test oracle is the criterion that decides whether the result of an execution is correct for a given input. It is not the test code itself, but the source of truth the test code refers to.
Every test has two halves: driving the program into a particular state, and judging whether the outcome is right. The first half automates easily; the second does not. Software engineering calls that difficulty the oracle problem.
Why it matters
When the oracle is left implicit, a very common accident follows: someone writes the implementation first, runs it once, and pastes whatever it produced into the expected value.
// Expected value copied from a single run of the implementation
expect(formatDiscount(1000, 0.1)).toBe("900원");
This test always passes, and it verifies nothing. Whether the rounding rule or the currency format is wrong, the expectation says what it says only because the implementation said it first. The result is a test that catches regressions but can never catch the original defect.
The problem grows when code generation tools are involved. A tool can read the existing code, the existing tests, and the current screen behavior, so it easily concludes "this is how it behaves, so this must be the requirement." That is why it pays to fix, up front, where the authority to define correctness lives.
How it works
Oracles differ by where the answer comes from.
| Kind | Source of truth | Limitation |
|---|---|---|
| Specified oracle | Approved requirements or acceptance criteria | Unless the document location and version are pinned, the baseline drifts again |
| Human oracle | A person inspects the result and judges | Slow and hard to reproduce; the judgment must be written down to become automatable |
| Derived oracle | Previous release output, reference implementation, production logs | Risks promoting an old bug to the status of correct behavior |
| Implicit oracle | Crashes, unhandled exceptions, deadlocks — obviously wrong outcomes | Says nothing about domain correctness |
| Metamorphic oracle | Derives relations between outputs from relations between inputs | Only needed where the exact answer is not computable |
In domains where you cannot compute the expected value, metamorphic relations are the practical fallback.
// Verifiable without knowing the correct ordering
const once = sortByScore(items);
const twice = sortByScore(shuffle(once));
expect(twice).toEqual(once); // input order must not change the result
Applying it
The core practice is to write the judgment criteria before the tests, and attach a source to each row.
| # | Given | When | Then | Never | Source |
| --- | ------------------ | ----------- | ------------------------- | ---------------------- | ---------------------- |
| 1 | List is empty | Page loads | Empty-state message shown | Spinner keeps spinning | Spec §3.2 |
| 2 | Request exceeds 5s | Auto-cancel | Retry button shown | Indefinite wait | User's explicit answer |
Rows with no attributable source stay blank and get escalated as a question. The moment you fill one in by convention, it stops being verification and becomes a guess.
For long-running work, save the table as a file and record its content hash. If the baseline shifts mid-flight, tooling notices before anyone else does.
shasum -a 256 docs/acceptance/checkout.md
Trade-offs
Writing the criteria first slows the opening phase. If requirements are still exploratory, filling in the table may itself be waste.
What you get in return is twofold. The expectations are independent of the implementation, so you can throw the implementation away and keep the tests. And when an argument starts, it ends by pointing at a line in a document rather than at the code.
Implicit oracles (no crashes) cost almost nothing but catch no domain errors; specified oracles cost the most and detect the most. Mixing them by risk level is the realistic answer.
When not to use it
- Prototype exploration. If what to build is itself undecided, freezing the criteria gets in the way.
- Areas with genuinely many correct answers. Layout micro-adjustments, copy, recommendation ordering — pinning a single expected value makes every legitimate change break a test. Use invariants (no overlap, item count preserved) or visual regression instead.
- When an external system defines correctness. Freezing a third-party response as your expectation means their change breaks your suite. Verify the shape of the contract only.
Common mistakes
- Copying expected values from a run of the implementation. The most common and quietest failure. The test for it is simple: "if I deleted the implementation and rewrote it, could I still use this expectation?"
- Reflexively editing the test when it turns red. Red may signal that requirement and implementation have diverged. Determine which side is wrong first.
- Keeping the criteria only in someone's head. When that person is away, nobody on the team knows what correct means.
- Filling unresolved rows by convention. "That's how it's usually done" comes back later as a requirement violation.
- Confusing the oracle with the harness. A failure caused by a broken locator is a tooling problem, not a correctness problem. Without classifying the cause, you fix the wrong thing.
Related concepts
- mutation-testing — verifies, in reverse, whether the judgment actually catches wrong implementations
- playwright-flaky-vs-failed-triage — the separate question of whether a judgment result can be trusted at all