Test impact across a TypeScript/Python boundary: one fused graph, one PR comment
A one-language code graph stops at the wire. We fused our TypeScript app and Python worker through a 19-line seam spec so one PR comment covers both sides.
A code graph built from one language stops at the wire. When a TypeScript API and a Python worker share an HTTP contract, test selection on either side alone is blind to the other, and the same two-file diff can get two opposite verdicts from two graphs of the same repository. We closed that gap on this site with a 19-line seam spec that joins the two graphs at the contract, and a CI job that comments the cross-language blast radius on every pull request that touches it.
The problem
This site has a small Python side. The Factory feed's drain, a script that runs
on a developer machine, composes activity notes and POSTs them to
/api/factory/notes, a Next.js route whose ingest schema lives in a TypeScript
file. That is the contract: one schema, one endpoint, one caller, in two
languages, in one repository. (The gate that decides whether those notes may
publish is
its own post.)
Every pull request here already gets
graph-based test selection:
a job builds a code graph of src/, hands the diff to Blastline, and runs the
tests the diff can reach. That job is TypeScript-only by construction, because
src/ is. The Python files are not in its graph, and nothing in a graph of the
TypeScript tells it that a one-line change to the ingest schema breaks a
function six hundred lines into a Python script.
That blindness fails in two directions. A change to the Python drain alone
reaches the TypeScript job as two files with no graph node, which is a correct
fail-open: run everything. A change to the schema is worse, because the job
can vouch for it. It selects the TypeScript tests that import the schema and
says nothing about the Python caller that just broke. The edge from the schema
to the drain's post() function is in neither language's graph. It is in the
contract between them.
What we did
We declared the contract once, as a seam spec, and let CGraph resolve it against both graphs. The whole file:
{
"provider": "webapp",
"api_version": "v1",
"error_codes": [400, 401, 429, 500],
"services": [
{"name": "webapp", "role": "provider", "owned": true},
{"name": "factory", "role": "consumer", "owned": true}
],
"schemas": [
{"name": "FactoryNoteIngest", "canonical": "webapp/src/lib/factory/schema.ts"}
],
"endpoints": [
{"method": "POST", "path": "/api/factory/notes", "response_schema": "FactoryNoteIngest", "path_params": [], "query_params": []}
],
"consumes": [
{"service": "factory", "method": "POST", "path": "/api/factory/notes", "call_site": {"graph": "factory", "file": "factory-drain.py", "line": 600}}
],
"mirrors": []
}
Three decisions are in those nineteen lines, and each was made against an alternative we rejected.
Anchor to real code, or emit nothing. The schema is anchored to the file
that defines it; the consumer is anchored to a file and a line inside the
function that makes the request. cgraph seam gen resolves both against the
real per-repo graphs and emits a cross-service edge only when a node is there.
The alternative is guessing: match URL strings across repositories and draw an
edge wherever /api/factory/notes appears. That produces exactly the
plausible-but-wrong edge you cannot build a test selector on, and
the open-source seam design
rejects it for the same reason.
Build both graphs on every pull request, then fuse. The job builds a graph
of src/ and a graph of tools/factory/, runs seam gen against the committed
spec, seam fuse to produce one graph, and hands that to the Blastline action.
It triggers only on paths that can touch the contract: the factory library and
route, the Python tools directory, and the MongoDB client the route writes through.
The ignore list is a set of claims that changes there cannot affect which
tests should run: Markdown, the documentation and drafts directories, posts,
images, JSON, plists, shell scripts and YAML. Getting that list wrong in the
other direction is how a test-impact tool starts lying, so it stays short,
and it already holds one entry we would tighten: the JSON pattern also
matches the seam spec itself, so a pull request that changes only
seam.json gets no selection from it.
Advisory here, enforced there. This job posts a comment. It does not skip a single test. The repo-wide job is the one that runs the selected subset, and it runs the full suite whenever it fails open. We wanted to read the cross-language selection against reality for a while before letting it gate anything. You cannot learn whether a test-impact tool is honest by trusting it. This is the same posture we take on the developer platforms we build for clients: a new signal earns its authority by being right in public first.
The result
The artifact is the set of comments the job has actually posted. Four pull requests have triggered it since August.
| PR | What the diff touched | Seam job verdict |
|---|---|---|
| #38, the end-to-end exercise | a comment added to the ingest schema | Subset: 4 TypeScript test files, with the Python caller in the blast radius |
| #41, pseudonym auto-publish | factory library and route, plus .env.example |
Full suite: .env.example has no graph node |
| #42, Mongo database name | the MongoDB client, plus .env.example |
Full suite: .env.example has no graph node |
| #46, dry-run cursor fix | the Python drain and its test file | Subset: 1 of 12 test files |
The first row is the one the seam exists for. A two-line comment added to the TypeScript schema, in a pull request opened only to exercise the job and closed without merging, produced a comment that selected the four TypeScript test files the graph reaches from it, and listed, in the same blast radius, the contract and the Python code on the other side of it:
Blast radius (12 dependents)
- endpoint POST /api/factory/notes
- file factory/factory-drain.py (tools/factory/factory-drain.py:1)
- file factory/feed.test.ts (src/lib/factory/feed.test.ts:1)
- file factory/feed.ts (src/lib/factory/feed.ts:1)
- file factory/notes.test.ts (src/lib/factory/notes.test.ts:1)
- file factory/notes.ts (src/lib/factory/notes.ts:1)
- file factory/schema.test.ts (src/lib/factory/schema.test.ts:1)
- file lib/mongodb.test.ts (src/lib/mongodb.test.ts:1)
- file lib/mongodb.ts (src/lib/mongodb.ts:1)
- function main (tools/factory/factory-drain.py:620)
- function post (tools/factory/factory-drain.py:596)
- service factory
That is the comment as posted, with the runner's path prefix removed. No graph
of the TypeScript alone contains the second line, the two function lines,
or the last one. The walk
went from the schema node through the endpoint to the consumer's call site and
on to the function that calls it, across the language boundary, because the
seam put that edge in the graph.
The two middle rows are correct and unexciting. Both pull requests also edited
.env.example, a file with no node in any code graph, so the selector refused
to vouch and ran everything. That is the behavior we want, and
why a config change runs your whole suite
has the full reasoning.
The last row is the clearest picture of what the fusion buys, because by September the repo-wide job existed too, and both jobs commented on the same pull request two seconds apart. The diff touched two Python files and nothing else.
The repo-wide job, with its TypeScript graph, reported that two files had no
graph node and fell open to its full run, which is bun test across the 11
TypeScript test files; the Python test is not in its suite at all. The seam job, with the Python graph
fused in, mapped the same two files to five symbols, found that nothing in
the TypeScript depended on the changed functions, and selected one test file
of twelve: the Python one. Same diff, same repository, same tool. The only
difference was whether the graph knew the other language existed.
Two limits are worth stating plainly, because they are the kind a vendor page leaves out.
The Python test file drives the drain through a subprocess, which no static
graph can see. So a schema change puts post() and main() in the blast
radius but does not select the Python test; a second consumer anchor at the
test's drive site would. And the consumer anchor is a line number. The spec
says 600. When the first comment was posted, def post started at line 596
and line 600 fell inside its body, so the anchor resolved to post(); the
September fix shifted the file, and def post now sits at line 600 exactly.
Neither state was arranged. Nobody has re-anchored the spec, and the next
edit above that line will move it again: the anchor then lands on whatever
sits at line 600, or, if nothing does, seam gen fails naming the anchor and
the job stops before it comments. Newer CGraph releases
discover HTTP routes and their callers, for the frameworks they read,
without a hand-written spec;
this job still pins the release it was verified with.
Takeaway
If your tests live on both sides of a wire, your test selection has to as well. Declare the contract once, anchor it to real code rather than to a URL string, let the graph refuse when it cannot find the anchor, and run the selection as advice until its comments have been right in public for a while. A per-language graph is not wrong about the language it covers. It is silent about the boundary, and the boundary is where the change that hurts usually crosses.