What this covers
While mapping out how frontend work gets handed to a code generation tool, I looked at two recurring failures: requirements quietly shifting mid-implementation, and "the tests pass" turning out to be a fake green. What follows is only the correctness-criteria design principles that apply anywhere, not any particular tool or product.
The same day I also revisited some earlier notes on restoring text anchors, and the two topics converged on the same principle in a way I did not expect.
Concepts to know first
An oracle is the criterion that decides whether a result is right or wrong — the answer key of an exam. Staring at the answer sheet (the code) never grades it. A policy source is the place that has the authority to define what correct means. VALID_RED describes a newly written test failing for the reason we intended; mistaking a typo or a misconfiguration for a real failure sends you off fixing the wrong thing.
Restrict who gets to define correctness
The most striking design decision was limiting the sources of correctness to exactly two: the user's explicit answer, and an approved specification. Product code, existing tests, observed browser behavior, and reviewer opinions are all demoted to investigative evidence.
The counterexample makes the point. A tool can read the existing code, so it easily concludes "this is how it behaves, so this must be the requirement" — and at that moment a bug is promoted to a specification. One rule, that a conflict between implementation and requirement means stopping to ask rather than conforming to the code, blocks most of that failure mode.
The criteria go into a table of rows: given this situation, doing this should produce this result, and this must never happen. Rows with no attributable source are left unresolved rather than filled in by convention.
→ Test oracles and where correctness comes from
Lock the agreement as text, verify it by hash
Requirements agreed only in conversation reliably blur over a long session. Save the criteria table as a file and record its content hash, and the fact that it blurred gets noticed by tooling before any human notices.
One rule attaches here: when the hash does not match, you do not re-lock to make it pass. You discard the evidence gathered so far and return to waiting on a human decision. Changing the criteria to make verification pass means it is no longer verification.
Name the fake greens and ban them
There are only a few ways to manufacture "it passes": loosening assertions, converting a failing test to skipped, and inserting an arbitrary wait to paper over timing. Writing those three down as an explicit ban held up far better than "please don't cut corners."
The deeper mechanism is moving the definition of done. If passing tests is done, then passing becomes the goal and assertions start eroding. Define done as evidence from real interaction in a browser plus an independent review, and there is nothing to gain from loosening anything.
To check mechanically whether tests actually catch defects, you can break the code on purpose.
→ Measuring test strength with mutation testing
Stop when confidence runs out
While tidying older notes the same day I noticed a symmetry. When restoring a highlight in a web document, one rule scores the candidate ranges and then refuses to attach anything if the gap between first and second place is small — a signal about competition, separate from whether a threshold was cleared.
That is exactly the same judgment as "if the policy is unresolved, do not proceed." One escalates a question to a human, the other gives up on a highlight, but both exist to avoid producing a quietly wrong result. It is the principle that attaching to the wrong place is worse than not attaching, translated into numbers.
→ Re-finding a text range in a document that changes
Do not trust exit code 0
One incidental lesson. A command meant to create a note exited with code 0 and no output, yet no file appeared. Exit code 0 does not mean "it succeeded"; it means "it did not report an error." For commands whose purpose is a side effect, verify the side effect itself.