Browser-Compat Interaction Smoke Test

Definition

To catch old-browser compatibility regressions, a smoke that only checks whether the screen "draws" is not enough. You must run real interactions like click, toggle, and submit while promoting runtime errors to test failures via a broad console-error filter.

Why It Matters

A runtime TypeError only fires at the moment the code actually runs. If a smoke opens the page and inspects only rendering — like "are N rows visible, is a certain text present" — the event handler that actually breaks never runs once, so the test passes green. That is how you get "the test passes but login doesn't work on a real old browser."

One more thing: a browser more than five years old can't even be reproduced on the modern engine that current automation tools bundle. You must download and run that version's browser/driver binaries directly at pinned versions, and operate the production build on that actual engine, to truly verify syntax/runtime compatibility.

How It Works

• Render-only smoke: existence/text assertions. Passes even if all handlers are dead — powerless against compatibility regressions.
• Interaction assertion: run real click/toggle/submit/routing to prove hydration + handler behavior.
• Console-error promotion: treat syntaxerror, referenceerror, is not a function, is not defined as failures. But exclude the network-flavored Failed to fetch as a false positive.
• Separate from API existence checks: whether the polyfill loaded (coverage) and whether the interaction works (behavior) are different roles, so verify them separately.
• Pinned-binary operation: download the old browser + driver at pinned versions, enforce the version, then run.

// console-error filter example: promote only compatibility signals, exclude network-flavored ones
function isCompatError(msg) {
  const m = msg.toLowerCase();
  if (m.includes("failed to fetch")) return false; // exclude network false positive
  return (
    m.includes("syntaxerror") ||
    m.includes("referenceerror") ||
    m.includes("is not a function") ||
    m.includes("is not defined")
  );
}

Practical Application

1. Add assertions that actually run each page's core interactions (click, toggle, dialog, routing, images, etc.).
2. Tune the console filter as above.
3. Download the old browser/driver at pinned versions and run the production build on that engine.
4. Wrap the old driver's known crashes (e.g. a defective automation action endpoint) in a workaround helper.

Trade-offs

When Not to Use

• Constantly running a heavy old-engine smoke on a project that supports only modern browsers — over-investment.
• Widening the console filter so far that network/third-party noise also becomes a failure, inducing flake.

Common Mistakes

• Checking only rendering and concluding "old browser passes."
• The console filter watching only syntaxerror and missing missing APIs (is not a function).
• Mistaking a network Failed to fetch for a compatibility failure.
• Believing you verified old browsers by mimicking them with a modern engine.

Related Concepts

• syntax-transpilation-vs-runtime-polyfill — The two kinds of compatibility error the smoke must catch
• hydration-failure-dead-handlers — What the interaction test proves (are the handlers alive)
• playwright-flaky-vs-failed-triage — Smoke triage that separates real failures from flake