Definition
A Playwright flaky test fails on the first run but passes on automatic retry. A failed test is a reproducible (deterministic) defect where the first run and all retries fail for the same reason. Even when a CI job is red, separating these two first changes the fix direction (test stabilization vs app/build bug).
Why it matters
E2E (End-to-End) tests are sensitive to network latency, SSE (Server-Sent Events — a connection where the server pushes events to the browser), portal UI, and hydration timing (the process of attaching client events to server HTML). When a team confuses "fix flaky" with "fix app bug," they work at cross purposes from the same red CI log.
With failOnFlakyTests: true, the CI exit code is 1 even when retry succeeds, so GitHub conclusion alone does not distinguish flaky from failed. You must read per-attempt results in the Playwright report and stdout.
How it works
- passed: First
test()run succeeds. - flaky: Attempt 1 fails → retry succeeds. Cause is usually ordering race (timer, focus, network predicate).
- failed: All attempts share the same assertion/exception. Cause is usually app logic, environment, or production build output (e.g. CSS loaded via
<script>). - Diagnostic depth:
pageerroroften has message only. CDP (Chrome DevTools Protocol)Runtime.exceptionThrownprovides exception URL and lineNumber. - Reproduction: Run failing tests against
production build+next start(or equivalent), not the dev server. Validate hypotheses with on/off A/B builds of suspect config.
Common flaky patterns:
| Pattern | Symptom | Alternative |
|---|---|---|
Fixed setTimeout + SSE refetch | Percent/state assertion order flips | Promise gate to control event order |
| Radix Select keyboard | Key input before portal option focus | Wait for option visible and focused |
networkidle | Click no response | Semantic wait: dialog open, button enabled |
Loose waitForRequest | Catches a different API request first | Predicate matching method, pathname, query |
Practical application
CI investigation procedure:
- Collect all push workflow logs in the period (do not sample only a few).
- Tag each test as passed / flaky / failed.
- Top flaky → test stabilization; 100% failed → production reproduction + CDP.
- After custom webpack
splitChunkschanges, check HTML script tags and build manifest for CSS mixed into JS entries.
Trade-offs
- More retries let flaky pass but real instability remains.
failOnFlakyTestsexposes that in CI. - CDP helps local reproduction but is hard to attach in every CI — maintain separate stdout parsing and artifact policy.
networkidleandsleepare easy to write but mask flake and recurrence.
When not to use
- Unit-test-level logic does not need Playwright triage — isolate with Vitest, etc.
- Do not estimate flaky rate when infrastructure failure (artifact upload failure, disk full) leaves no logs.
- Deleting assertions from flaky tests to turn CI green hides defects.
Common mistakes
- Assuming GitHub job conclusion = flaky.
- Calling 123/123 identical SyntaxError (e.g. group-delete) flaky — that is deterministic failed.
- Running only the dev server and missing production build defects.
- Mocking SSR auth requests with
page.route()only, causing server/browser mock mismatch.
Related concepts
- webpack-splitchunks-vendor-shared-chunk — trap where CSS enters JS entry and
<script src="*.css">appears - ssr-hydration-mismatch —
networkidle≠ hydration complete - react-render-count-isolation-testing — render-count isolation unit tests