Filtering non-deterministic AI code through deterministic checkpoints — a step-by-step workflow breakdown

What this post is about

Ask an AI to do the same task twice and something interesting happens. The request is identical, but the code comes out different every time. The approach that passed yesterday shows up in a subtly different shape today, and even inside a single conversation it will hand you code that contradicts a promise it made a moment ago.

We call this non-deterministic: there's no guarantee the same input produces the same output. The opposite — same input, always the same output — is deterministic.

The problem is review. If you inspect something that changes every time using only human eyes, the human misses something different every time. So I use the inverse strategy.

Leave the non-deterministic producer (the AI) alone, and turn the inspecting side entirely into deterministic instruments. Then split the work small enough that inspection actually works.

In this post I'll walk through how the frontend-oracle-design workflow I use implements that strategy, one step at a time. The philosophical background is covered in Lock the correctness criteria before you write code; this post focuses on how the procedure actually runs.

First: the chef varies, the thermometer doesn't

Let me start with an analogy.

You hire a skilled chef, but this chef makes the same order slightly differently every time. Some days it's excellent, some days the seasoning is off. Standing beside them and watching won't give you control, because the process is different every time.

So what do you do? You install instruments in the kitchen.

• A thermometer measures the internal temperature — by the same standard every time
• A timer measures the cook time — by the same standard every time
• A scale measures the quantities — by the same standard every time

No matter what order the chef does things in, the plate has to clear every instrument before it goes out. Instruments don't have moods. Measure the same plate twice and you get the same value twice. That's deterministic verification.

Back in code, the instruments look like this.

There's one key point here. One instrument isn't enough. A thermometer can't catch over-salting, and a scale can't catch undercooking. Each verifier catches a different class of defect, so you have to layer them to close the gaps. Safety engineering calls this the Swiss cheese model — one slice has holes, but stack enough slices and the odds of a hole going all the way through drop sharply.

The whole map of the workflow

Here's the procedure at a glance. It splits into two stretches: Design and Delivery.

[Design stretch]
1. Investigate — fix the sources allowed to define correctness
2. Question    — ask about ambiguous boundaries up front (BVA)
3. Oracle Card — write requirements row by row
4. Lock        — check with a script, seal with a hash

[Delivery stretch]
5. Red    — write tests first and make them fail "for the intended reason" (VALID_RED)
6. Green  — pass them with the minimum implementation
7. Drill  — plant a bug on purpose and confirm the tests catch it (mutation testing)
8. Retrial — an independent reviewer re-checks holding only the card

Two rules run through the entire thing: routing that assigns exactly one cause per observation, and caps on retry counts (budgets). Let's go through them one by one.

Step 1 — Investigate: admit the evidence before the trial

The first step isn't reading code. It's fixing where the authority to define correctness lives.

Think about a trial: before the real argument starts, you decide which evidence is admissible. Anything not admitted can't be used in the ruling, however plausible it looks. Same here.

• Admissible: the user's explicit answers, the exact location of approved specs, requirement docs, designs
• Not admissible: the product code as it currently runs, existing tests, "everyone does it this way these days" best practices

Code that currently runs tells you "what is happening now," not "what is correct." Because an AI can read the whole repository, it very naturally makes the mistake of treating current behavior as the requirement. That's the moment a bug gets promoted to a specification.

What if the sources contradict each other, or a required document is inaccessible? You don't pick the plausible-looking side — you stop in the NEEDS_DECISION state and ask a human. This "stop" will keep showing up throughout the workflow.

Step 2 — Question: write down the boundaries like an insurance rider

Once the sources are fixed, you go looking for the ambiguous boundaries and ask up front.

It's like an insurance contract. "We'll cover you if you get hurt" sounds great, but every actual dispute happens at the boundary. Does the commute count? What about a bicycle? That's why good contracts spell out boundary cases as riders in advance.

Requirements are the same. "Show a list" is easy; the accidents live at the edges.

• What if the list has 0 items? 1 item? 10,000 items?
• What if the request fails? What if it takes 5 seconds?
• What if someone mashes the same button?

Pulling cases from around value boundaries is called Boundary Value Analysis (BVA). It rests on a long-standing observation: bugs cluster overwhelmingly at the edges rather than in the middle of a value range.

Questions in this step have a cap too — at most 2 rounds. Asking forever means never progressing, so within 2 rounds you ask only the questions that change the outcome.

Step 3 — The correctness table: build the rubric first

Once answers are in, you organize them into a table called an Oracle Card. In exam terms, you're writing the grading rubric first, not the answer key.

| #   | Given                   | When                   | Then                        | Never                         | Source      |
| --- | ----------------------- | ---------------------- | --------------------------- | ----------------------------- | ----------- |
| O1  | List is empty           | Entering the page      | Show an empty-state message | A spinner that keeps spinning | Spec §3.2   |
| O2  | Request exceeds 5s      | Auto-cancel            | Show a retry button         | Waiting forever               | User answer |
| O3  | Same item clicked twice | Ignore from the second | Exactly 1 request fires     | Duplicate requests            | User answer |

Each row later becomes one test. And a row with an empty Source column can't proceed. The moment you fill an unsourced row with "well, this is how it's usually done," verification turns into guesswork.

When the table is done, attack it yourself once: "Is a still-wrong implementation possible that satisfies this table?" That's an adversarial self-review. If the answer is yes, you're missing rows — add them.

Step 4 — Lock: get the contract notarized

This is where the deterministic instruments really enter. A human wrote the table; now a machine checks and seals it.

1. Structure check — a script (oracle-verify) checks the table's format. Rows without sources, statements that can't be verified, and so on get filtered mechanically. A human waves through "eh, looks fine"; the machine doesn't.
2. Hash seal — record the SHA-256 hash of the table file. A hash is like a fingerprint that changes completely if a single character changes, so comparing fingerprints before each step tells you immediately whether the table was quietly altered.

It's the paper-contract equivalent of notarization and a seal across the page edges. With the seal in place, swapping a page later becomes impossible.

And if the fingerprint doesn't match — you don't re-lock to make it pass. You discard all the evidence gathered so far and go back to step 1. Changing the verification criteria in order to pass verification isn't verification anymore.

Why go this far? Because a non-deterministic worker often subtly reinterprets the criteria themselves during a long task. Humans do it too. The hash physically blocks that reinterpretation.

Step 5 — Red: test the fire alarm first

Only now do you write tests. The order matters — tests come before production code.

Move each row of the table into a test, run it, and first confirm it fails for the intended reason. That state is called VALID_RED.

Picture installing a new smoke alarm. The first thing you do after mounting it is press the test button, right? You can't sleep trusting an alarm you never confirmed would sound. VALID_RED is exactly that test button. There's no implementation yet, so the test should fail — and you confirm the failure reason is "the feature doesn't exist." If it's red because of a typo or a config problem, the alarm itself is broken.

A few detail rules keep this step deterministic.

• Executions are recorded in a ledger. Test runs go through a dedicated script, and results are written to an append-only ledger. When reporting, instead of saying "it passed" you cite the run ID in the ledger. A run that isn't in the ledger doesn't count as a pass. Words are non-deterministic; the ledger isn't.
• State transitions have a script as referee. Order violations — implementing before writing tests, quietly flipping a failing test to skip — get checked by the transition script, and a violation means the transition itself is rejected. It's like a football referee holding the match sheet: a player can insist "I scored back there," but if it's not on the sheet, it's not a goal.
• The network gets blocked with MSW. A real server is a lump of non-determinism whose responses can vary, so you swap it for a tool (MSW) that intercepts requests and returns fixed responses. You're removing non-determinism from the places that aren't under test.

Step 6 — Green: implement the minimum

Once red is confirmed, implement. The goal is the minimum code that passes the locked table. No features not in the table, no "for later" abstractions.

If it doesn't pass, you fix and retry — and that has a budget too: at most 3 rounds of implementation improvement. And you don't count those rounds in your head; a script counts them. Leave "wait, which attempt was this?" to a non-deterministic worker and the count always comes out generous.

If it's still red after the budget is spent, you honestly report failure and stop, along with the last real failure. That's faster than "let me keep trying until it works," and more importantly, the failure survives as data.

Step 7 — Drill: start a fire on purpose

For high-risk work (payments, permissions), you add one more layer: mutation testing.

You plant small bugs in the implementation on purpose — change a < to <=, flip a condition. Then run the tests and see whether red lights up. If it does, that's evidence the tests can catch that class of bug (we say the mutant was "killed"); if it doesn't, there's a hole in your tests. After checking, revert the bug and confirm green again.

It's a fire drill. Before a real fire, you stage a fake situation and confirm the alarm and evacuation procedure work. "We installed an alarm" and "we confirmed it actually sounds" are completely different levels of confidence.

Step 8 — Retrial: give a different referee only the card

Finally, you add an independent reviewer. The point is "independent."

Review your own code and you repeat the same misconception. It's the same reason you can't find typos in your own writing. So you give a reviewer in a separate session from the main work only the correctness table and the result. You don't give them "the story of how hard this was." Hear the story and the reviewer gets talked into it too.

When the reviewer raises something, you don't just fix it — you classify it first using the routing table in the next section. Then, after applying the valid findings, you re-run the entire test suite and re-confirm green. That re-verified state (REVIEW_VERIFIED) is this workflow's definition of "done." Passing tests is not done — once you move "done" behind the review, there's no longer anything to gain from loosening a test.

Across all steps — the ER triage board and the alarm clock

When something odd turns up between these steps, you don't reflexively fix code — you classify first. It's ER triage: you don't treat in arrival order, you look at the symptom and route it down a predetermined path.

Without this classification you get the reflex "the test went red, so fix the product code." Fixing perfectly fine code when red was actually a tooling problem happens more often than you'd think.

And each path's budget (2 rounds of questions, 2 tooling adjustments, 3 implementation rounds) is non-transferable, and a script counts it. Just like an alarm clock: leave "five more minutes" to your own willpower and you will oversleep, so you hand it to a machine.

Modularity — the checkpoint only works if the cargo is small

That was the checkpoint story. Now for why you split the work small. The short version: checkpoints only work properly when the cargo is small.

Think about customs inspection

If one container arrives with electronics, food, and clothing all mixed together, how is the inspector supposed to inspect it? They get worn out opening it and land on "looks fine, I guess." Ship it split into boxes by category instead, and each box gets exactly the inspection criteria for its category.

Code is the same. A 500-line diff exhausts the reviewer (human or machine), and when a test fails you can't pin down which part caused it. Ten 50-line diffs each get judged cleanly. The Oracle Card from earlier is exactly that "box." One card = one small contract = one reviewable unit.

You don't build every module from scratch

Here's the principle that really matters. Splitting small does not mean building every piece anew and reviewing every piece from scratch each time.

Think about LEGO. Nobody starts a spaceship by injection-molding the bricks. You assemble bricks whose specs are already verified, and the only things you design fresh are the few special parts this particular build needs. Your review focuses on those new parts. For existing bricks, you just confirm they fit the spec (the public API) — done.

The workflow does the same.

• Look for existing components, utilities, and patterns first. Rebuilding something that already exists in the repository is the most common failure mode of AI code, so you search before you write. Code that's already there is a brick that already passed inspection.
• Reuse the verification pipeline too. Type check, lint, test runner, CI — you use the checkpoints already installed. You don't invent a new verification tool per task. The only new things are this task's correctness table and its tests.
• Focus only on the problem in front of you. No abstractions or configuration "in case we need it later." That's code with no contract (no card) to review against, which means it's cargo that sails straight through the checkpoint.

The split pieces live with their ownership boundary

There's a rule for where the split pieces go, too. Tests and fake server responses (MSW handlers) live next to the folder that owns the feature. Collecting everything into root-level e2e/ and mocks/ folders looks convenient, but later, when you delete a feature, the tests don't get deleted with it and you accumulate dead checks nobody can attribute. Letting a feature and its tests be born together and die together is what completes modularity.

Wrapping up — the strategy is one sentence

To control a producer whose output varies, don't try to fix the producer. Turn the judging side entirely into machines, and split the work into judgeable sizes.

Condensed by step:

And modularity is the precondition for that entire table working. Small review units make each instrument's verdict sharp, and reusing already-verified bricks shrinks the surface you have to review anew.

Will these procedures become unnecessary as AI gets better? I think the opposite. The faster production gets, the faster and the more wrong things get made too. Which means you need more instruments that don't have moods.

Further reading

• Lock the correctness criteria before you write code — designing AI-assisted frontend work — the philosophy half of this workflow
• Test oracles and the sources of correctness
• Measuring test detection power with mutation testing