Reuse the tests instead of rewriting them
This CLI reads Playwright tests and asks Hermes Agent to check the same flows on a live staging site. It does not replace the existing E2E suite. It fills the gap between deterministic tests built around mocks and the screen users actually see.
Replaying mocked values and selectors verbatim is brittle: a small copy change can break the flow. The tool extracts the user intent from the test and leaves the contextual screen reading to the Agent.
Playwright spec → QA scenario → staging exploration → verdict and evidence
Mark the safe scope with annotations
There is no separate DSL. A few annotations are added to the existing test.
// @qa-page: dashboard
import { expect, test } from "@playwright/test";
// @qa-live-policy: readonly
test("shows the current plan", async ({ page }) => {
await expect(page.getByTestId("plan-name")).toBeVisible();
});
The CLI turns the spec into a scenario and records the Agent's screen evidence with one of four verdicts: pass, fail, manual_review, or skip.
Design philosophy
A test is the source of intent before it is a replay script
The tool does not replay Playwright code against staging as-is. Mock values and selectors are useful in CI, but they age quickly against live data. Instead, it extracts what the user should be able to see and turns that into a live scenario.
Deterministic tests stay in place
Playwright CI and API contract tests remain the primary regression gates. The Agent is a supporting layer for data, copy, and DOM states that can vary between runs. This is not a choice between one approach or the other.
Uncertainty is a valid result
The biggest risk in AI judgment is a plausible but wrong answer. When evidence is weak, the tool returns manual_review instead of forcing pass or fail. An honest unknown is safer than a false pass.
Safety must be explicit, not inferred
Each test declares a policy such as readonly or safe-interaction-no-confirm. Billing changes, subscription mutations, and mocked authentication flows are blocked. Credentials are passed through environment variables rather than config files or CLI arguments.
Evidence matters more than the verdict
Results are saved as Markdown and JSON so a person can see what the Agent observed and why it reached that conclusion. The review step reads the saved judgment without opening the browser again.
What this tool does not own
The spec → abstract-ai → judge → review → slack flow is available as an npm package and CLI. It can run as nightly QA, but it does not replace deterministic regression tests, API contract tests, or a QA engineer.