Playwright flaky vs failed triage

Definition

A Playwright flaky test fails on the first run but passes on automatic retry. A failed test is a reproducible (deterministic) defect where the first run and all retries fail for the same reason. Even when a CI job is red, separating these two first changes the fix direction (test stabilization vs app/build bug).

Why it matters

E2E (End-to-End) tests are sensitive to network latency, SSE (Server-Sent Events — a connection where the server pushes events to the browser), portal UI, and hydration timing (the process of attaching client events to server HTML). When a team confuses "fix flaky" with "fix app bug," they work at cross purposes from the same red CI log.

With failOnFlakyTests: true, the CI exit code is 1 even when retry succeeds, so GitHub conclusion alone does not distinguish flaky from failed. You must read per-attempt results in the Playwright report and stdout.

How it works

1. passed: First test() run succeeds.
2. flaky: Attempt 1 fails → retry succeeds. Cause is usually ordering race (timer, focus, network predicate).
3. failed: All attempts share the same assertion/exception. Cause is usually app logic, environment, or production build output (e.g. CSS loaded via <script>).
4. Diagnostic depth: pageerror often has message only. CDP (Chrome DevTools Protocol) Runtime.exceptionThrown provides exception URL and lineNumber.
5. Reproduction: Run failing tests against production build + next start (or equivalent), not the dev server. Validate hypotheses with on/off A/B builds of suspect config.

Common flaky patterns:

Practical application

// ❌ Loose predicate
page.waitForRequest((req) => req.url().includes("/api/items"));

// ✅ Only the intended request
page.waitForRequest((req) => {
  const u = new URL(req.url());
  return u.pathname === "/api/items" && u.searchParams.get("limit") === "51";
});

CI investigation procedure:

1. Collect all push workflow logs in the period (do not sample only a few).
2. Tag each test as passed / flaky / failed.
3. Top flaky → test stabilization; 100% failed → production reproduction + CDP.
4. After custom webpack splitChunks changes, check HTML script tags and build manifest for CSS mixed into JS entries.

Trade-offs

• More retries let flaky pass but real instability remains. failOnFlakyTests exposes that in CI.
• CDP helps local reproduction but is hard to attach in every CI — maintain separate stdout parsing and artifact policy.
• networkidle and sleep are easy to write but mask flake and recurrence.

When not to use

• Unit-test-level logic does not need Playwright triage — isolate with Vitest, etc.
• Do not estimate flaky rate when infrastructure failure (artifact upload failure, disk full) leaves no logs.
• Deleting assertions from flaky tests to turn CI green hides defects.

Common mistakes

• Assuming GitHub job conclusion = flaky.
• Calling 123/123 identical SyntaxError (e.g. group-delete) flaky — that is deterministic failed.
• Running only the dev server and missing production build defects.
• Mocking SSR auth requests with page.route() only, causing server/browser mock mismatch.

Related concepts

• webpack-splitchunks-vendor-shared-chunk — trap where CSS enters JS entry and <script src="*.css"> appears
• ssr-hydration-mismatch — networkidle ≠ hydration complete
• react-render-count-isolation-testing — render-count isolation unit tests