Definition
Reach for this concept when you're trying to judge "I made a performance improvement — is it really faster?" but the numbers jump around every time you reload the same page, so you can't tell whether it's an improvement or a coincidence. It's the method of re-measuring several times instead of once and comparing by the median.
Lab performance measurement variance (lab measurement — measuring directly on your own device rather than from real user data — and how much the value jumps each time) is the phenomenon where LCP, TTFB, and CLS differ significantly across dev-server reload traces (performance recordings on reload) even for the same page and same code. Here, LCP (Largest Contentful Paint, the time until the largest element on screen is painted), TTFB (Time To First Byte, the time until the first byte of the server response arrives), and CLS (Cumulative Layout Shift, the cumulative score of how suddenly the screen shifted) are representative web-performance metrics. The comparison protocol is the procedure that fixes environment metadata and judges before/after not by a single run but by an N-run reload median (the median of values reloaded several times — unlike the mean, it's less swayed by outliers).
Why it matters
Environments like next dev have on-demand compile (compiling that page just-in-time when a request comes), HMR (Hot Module Replacement, a dev feature that swaps only the changed part without a reload when you save code), source maps, and unoptimized bundles, so TTFB and LCP jump from run to run. If you conclude "regression" (performance got worse) or "improvement" from a single trace, you confuse environment noise with an actual change. Conversely, you might miss a real regression as a single outlier (a measured value far off from the others). Before release, it's good to re-confirm with the same protocol on a production build.
How it works
LCP roughly decomposes into TTFB + load delay + load duration + render delay. If TTFB in dev swings between 300ms and 1000ms, the whole LCP moves proportionally. CLS, too, can have a single late shift in one run raise the score substantially.
Metadata to record: URL, auth state, data scenario, dev vs start, git commit, throttle, measurement tool.
Aggregation: in the same Chrome session, 3 reload traces → median(LCP), median(CLS), median(TTFB). Record any run 2× or more of the median separately as an outlier, but don't use it in the headline.
Cross-environment comparison: comparing the absolute values of a dev median and a production median is meaningless — look only at the before/after delta within each environment.
Practical application
# baseline header (example)
url: https://example.com/app
build: pnpm dev | pnpm start
auth: logged-in fixture
tool: Chrome Performance trace (reload)
runs: 3
run1 LCP / CLS / TTFB
run2 ...
run3 ...
median ...
outlier (if any) + tag: cold | compile | network
Lighthouse's Performance 0–100 and a trace's LCP ms are different instruments. Don't substitute an A11y/SEO category score for LCP (lighthouse-measurement-accuracy).
Trade-offs
| Choice | Benefit | Cost |
|---|---|---|
| dev 3× median | fast iteration | abs ms differs from prod |
| prod only | high confidence | slow measurement/auth setup |
| single run | fastest | high risk of misjudgment |
When not to use
- When you already have a sufficient sample from RUM (Real User Monitoring, field performance data collected from real users' browsers) and lab only needs a regression smoke test (the protocol can be simplified).
- When you compare only single runs while changing the throttle/network profile.
Common mistakes
- Concluding PR performance from a single dev run.
- Interpreting a Lighthouse category score of 100 as optimal LCP.
- Measuring only the cold first run and saving it as the baseline.
- Directly mapping dev median figures to a production SLA.
Related concepts
- lighthouse-measurement-accuracy — avoiding score vs. ms confusion
- largest-contentful-paint — LCP composition/improvement
- critical-rendering-path — TTFB/render pipeline