What I wanted to do
Several jobs run in parallel in separate folders, but the pre-push hook and the PR CI ran the unit tests for hundreds of files every time, so the same work kept repeating. I wanted to cut that down to "only the tests related to what changed" while keeping the safety nets as they were.
Here are a few concepts that are good to know beforehand.
- import graph: A map of how files pull in other files with
import. If you follow this map backward from a changed file, you can find the tests that are affected. - fail closed: A design where, when a judgment is uncertain, you don't let things pass and instead go to the safer side (here, the full test suite). It is like a traffic light turning every direction red when it breaks.
What I did
I didn't build my own tool. I used the --changed <base> option built into Vitest. It collects the commit diff, staged and unstaged changes, and even untracked files, and picks the related tests.
- I don't guess the base point. PR CI uses the actual base commit that the platform reports, and the push hook uses the "previous remote SHA of that branch" that Git passes to the hook. For a new branch, Git doesn't record the branch it split from, so I made a person state
TEST_BASEexplicitly. - When uncertain, I go to the full run. File deletions and renames remove the starting point from the graph, so they switch to the full test suite, and global inputs such as config, lockfile, tsconfig, translation JSON, and shared assets are rerun in full through
forceRerunTriggers. I block a dirty working tree or a ref that is not the current HEAD. - I kept the final candidate behind a separate gate. A PR to the development branch gets a selective run, and the main candidate gets the full suite plus E2E.
- I decided the worker count by measuring it myself. Adding workers inside one process was different from launching several processes at the same time. But I couldn't measure the throughput of a real parallel environment, so I didn't generalize.
- I added regression tests for the hook (push range, base for a first push, dirty working tree, missing base, deleted-file fallback).
What I learned
A narrowing rule alone is risky. There must also be a rule that sends situations where the graph can't be trusted to a full run. I also confirmed firsthand that when you change a shared module, the related tests are close to the whole suite and the benefit of selective runs disappears. → Affected test selection
Take the base point from the situation, not from the tool. If there is no value you know, don't guess; ask a person for it.
For E2E tests that only flake in a parallel environment, suspect the environment first. In local parallel runs the failures wobbled and moved around, but when I ran only the failed files on their own they passed, and when I ran with the same settings as CI everything passed. I chose to use the CI settings as they are, without adding sleep or changing app code.
What I haven't confirmed yet: the improvement in total elapsed time and resource usage across real parallel jobs, and cases where tests are missed because of dynamic imports. I left the latter to the final full gate.