testscope

Map a set of changed files in a Go module to the minimal set of test packages that transitively cover them — by following the import graph, not directory names.

Built as a working prototype for the diff-to-test-scope mapper described in kyverno/kyverno#16665 (Phase 2).

Install

$ go install github.com/manshusainishab/testscope@latest

This drops the binary in $(go env GOPATH)/bin, which is not on PATH by default. If testscope comes back as command not found, add it:

$ echo 'export PATH="$PATH:$(go env GOPATH)/bin"' >> ~/.zshrc && exec zsh

Or run it from a clone without installing anything:

$ go run . -repo ../kyverno <files...>

---

$ testscope -repo ../kyverno pkg/webhooks/resource/validation/validation.go
module     github.com/kyverno/kyverno
changed    1 file(s)
selected   2 of 206 test packages (99.0% avoided)

test packages:
  github.com/kyverno/kyverno/pkg/webhooks/resource
  github.com/kyverno/kyverno/pkg/webhooks/resource/validation

go test github.com/kyverno/kyverno/pkg/webhooks/resource ...

---

Why not just map paths to tests

The obvious approach — glob changed paths onto test directories — is unsound, and unsound in a way that fails silently. Two cases break it:

Transitive dependents. A change in a leaf utility package breaks the packages that import it, not itself. Path globbing selects the leaf's own tests (often none) and skips every test that would actually have caught the regression.

External test packages. Go's _test packages import their subject from outside it. On Kyverno, behaviour in cmd/cli/kubectl-kyverno/commands/jp/query is asserted from sibling packages — so a path map skips the tests that cover the change. (Raised by @omlahore on the issue thread.)

testscope walks the actual import graph, including TestImports and XTestImports, so both cases resolve correctly.

Soundness argument

A test package T is selected for a changed package X iff X is reachable from T's test binary, where the roots are T itself plus everything imported by T, T's in-package tests, and T's external _test package.

Graph traversal is restricted to packages inside the module under analysis. That restriction is sound because a package outside the module cannot import one inside it — so no path from a test binary to a changed package can leave the module and re-enter it.

Recall-first by construction

Selection optimises for provably no missed coverage first, fewest jobs second. On an admission controller a false negative is a security incident, not a slow build, so every ambiguous case resolves toward running more tests:

Situation	Behaviour
Changed file is in a generated tree	Raises the codegen flag and still selects dependents' tests — regenerating a clientset can break its consumers
Changed file maps to no package (docs, manifests, testdata)	Reported, never used to justify skipping
Changed .go file maps to no package	Reported separately as a fault — see below

The failure mode this tool is most careful about

0 test packages selected and the diff was not understood are identical on the wire and opposite in meaning. A stale index or a wrong path would otherwise render as "100% avoided" — which in CI reads as skip everything.

So an unresolved .go file is tracked apart from unresolved docs and manifests: every Go file in a module belongs to some package, so a hit there means the index is stale or the path is wrong. -strict turns it into a non-zero exit.

$ testscope -repo ../kyverno -strict pkg/nonexistent/foo.go
WARNING    1 changed .go file(s) resolved to no package. Every Go file in
           the module should belong to one, so the index is stale or the path
           is wrong. Do not treat the selection below as complete.

no Go test packages cover this diff
testscope: 1 changed .go file(s) resolved to no package; selection is not trustworthy:
  pkg/nonexistent/foo.go
$ echo $?
1

---

Usage

testscope [flags] [file ...]
testscope [flags] -diff <ref>

  -repo string        path to the Go module to analyse (default ".")
  -diff string        take changed files from 'git diff --name-only <ref>'
  -generated string   comma-separated module-relative prefixes holding generated code
  -json               emit the selection as JSON
  -explain            report which changed package selected each test package
  -quiet              print only the selected package list, one per line
  -strict             exit non-zero if any changed .go file could not be resolved

Wiring it into CI:

PKGS=$(testscope -diff origin/main -strict -quiet) || exit 1
[ -n "$PKGS" ] && go test $PKGS

-json for programmatic consumers:

{
  "testPackages": ["github.com/kyverno/kyverno/pkg/webhooks/resource"],
  "codegen": false,
  "totalTestPackages": 206
}

Measured on Kyverno

github.com/kyverno/kyverno at 8e53789ec — 564 packages, 206 with tests.

Changed file	Selected	Avoided
docs/dev/README.md	0 / 206	100%
pkg/webhooks/resource/validation/validation.go	2 / 206	99.0%
pkg/engine/handlers/mutation/mutate_resource.go	43 / 206	79.1%
pkg/client/clientset/versioned/clientset.go (generated)	48 / 206 + codegen	76.7%
api/kyverno/v1/* (24-file API commit)	130 / 206	36.9%

The last row is the tool working correctly, not failing: changing a core API type genuinely does require running most of the suite. A selector that reported a big reduction there would be wrong.

Cold run ≈ one go list -json ./...; warm ≈ 2.6s on this module.

Design

One go list -e -json ./... invocation supplies every package's direct imports; the transitive closure is computed in-process rather than shelling out to go list -deps per package, which would re-walk the same graph hundreds of times.

internal/index/index.go   go list parsing, import graph, reverse-dependency index
internal/index/select.go  changed files -> packages -> covering test packages
main.go                   CLI

Status

Prototype. Covers Go unit tests only.

The harder half of the Kyverno problem is conformance: 1,040 chainsaw-test.yaml cases across 52 suites, with no import graph and essentially no metadata linking a suite to the source it exercises. That mapping can't be derived from the filesystem. The approach I'd take is coverage instrumentation — Kyverno's go.mod is on Go 1.26, so go build -cover + GOCOVERDIR allow a conformance run against an instrumented binary to yield an empirical, regenerable suite→package map, rather than a hand-curated one that rots.

Before any selector is allowed to skip anything, it should be backtested against historical CI: replay past PRs where conformance failed and confirm the selector would have run the suite that caught it. Target miss rate: zero.

Tests

$ go test ./...
ok  github.com/manshusainishab/testscope/internal/index

14 tests over a synthetic fixture module — transitive dependents, external _test imports, generated-path routing, unresolved-file classification, and the reduction arithmetic.

License

Apache 2.0