Why a config change runs your whole test suite, and what to show instead

When a pull request changes a config, data, or build file, graph-based test selection has to run the full suite: the file has no node in the code graph, so nothing can prove which tests depend on it. What it can do is tell the reviewer which code reads the file and which tests that code reaches. We tried letting those readers narrow the run, measured it, and shipped them as advice instead, because the version strict enough to be safe narrowed nothing.

The problem

Blastline, our open-source test-impact tool, picks the tests a diff reaches by querying a CGraph code graph. Its contract is "run at least these": when the graph can't vouch for a diff, it falls back to the full suite and says why.

Across the last 60 pull requests in six repositories running it, every full-suite verdict had the same cause: a changed file with no graph node. A fixture, a YAML profile, a stylesheet, a build file. That fallback is correct, but it's also unhelpful. A reviewer who sees "run everything" learns nothing about which tests actually matter for the change, and many of these files are read by code that says so in plain text: a test opening fixtures/rows.csv, a migration test walking migrations/, a Spring test loading application-production.yml.

What we did

The obvious move was to let those readers replace the full suite: find the code that names the file, take the tests it reaches, run those. The first version did exactly that, and on 30 recorded full-suite PRs it turned 9 into targeted runs with no regressions in the replay.

Then an independent review went looking for ways it could silently drop a real test, and found them:

• a pytest fixture in conftest.py, which has no graph edge to the tests that use it
• subclasses of a @SpringBootTest base class, and classes carrying a meta-annotation
• paths built from a stem, like loadFixture("rows")
• Go's os.ReadDir, Rust's #[files], and Spring profiles activated through a constant

Every rule added to close a hole made the resolver stricter. The version strict enough to be safe vouched for none of the 30 PRs, because build files, CI workflows, and package.json mention folder names that a text search can't tell apart from real readers.

So the verdict stays the full suite, and the readers become advice. Under a full-suite verdict, the pull request comment now lists, for each unmapped file, the code that reads it, the tests those readers reach, and what it could not vouch for. Here is the shape of one from a Next.js app, where a PR touched the global stylesheet:

File	Read by	Tests reached	Not vouched for
app/globals.css	the root layout, a walkthrough component, a setup spec, and 10 more files	11	a config file mentions it, but nothing found runs it; 5 more

The result

Advice changes nothing about what runs, so the measurement is about whether it is accurate and useful. Replayed over the same 30 PRs, the verdicts are identical on all 30, and 17 of the 29 full-suite PRs now get a reader table. On the Spring PR that motivated the work, the table named 4 readers reaching 6 tests, computed in under a second.

The first release was too generous, and real PRs showed it. Matching a folder name counted every module that imported a sibling, so the next release follows a folder mention segment by segment and stops the moment the path disagrees:

Narrowing still happens where it can be proven rather than guessed. A package.json change that only adds scripts nothing runs is checked structurally against the manifest at both ends of the diff, and on a real web-app PR adding two scripts that took the run from all 395 test files to 1. The difference is the evidence: a structural proof can shrink the suite, a text match can only point.

Takeaway

When your tooling can't prove which tests a change touches, don't let it guess. Run everything and hand the reviewer the evidence it found. If flaky, slow, or untrustworthy CI is costing your team, that is the kind of developer platform work we take on, and the tools are MIT-licensed if you'd rather start on your own.