perf(compilers/openapi): index the source tree once per compile - #340
Merged
Conversation
One compile parsed the same source bytes twice — once in the pre-parse cycle scan, once in the loader — and then walked the resulting node tree four times: the recursive-anchor descent, the pure-$ref collection, the raw node count, and the alias weigher, each starting from the root with no shared state. internal/sourceindex walks the decoded tree once and answers what the anchor descent and the node count each walked it to ask. The loader decodes, indexes, and hands the index to scan, so a source is parsed once and the two walks that only ever asked something about the tree itself become one. The walk is bounded by MaxIndexedNodes rather than by the input. A tree past it is refused under a new openapi/source-too-large code rather than half-counted: every answer in a truncated index is partial, including the node count the alias-expansion allowance is derived from, and an allowance computed from a count that stopped early would refuse documents on a bound they never crossed. Behaviour is unchanged. Every golden and conformance snapshot is byte-identical without regeneration, and the whole testdata corpus compiles to identical documents, diagnostics and exit codes before and after.
The conflict this branch predicted, resolved as it said: #304's TestDetectCycles_EmptyPointerSegmentIsRefused calls Cycles(0, []byte(src)), and Cycles now takes a sourceindex.Index, so the call becomes scanBytes(t, ...). Two the branch could not predict. cycles_test.go: #304 and #310 added a test where this branch adds a scanIndex helper, both at the same offset sharing a closing brace — kept both. And #328 consolidated the openapi test scaffolding into internal/openapitest after this branch was written, so its new entry_internal_test.go calls sourceOf by the package-local name that no longer exists; repointed, with the import added. Byte-identical output re-proven against the merged tree rather than carried over from the branch: 115 sources under testdata compiled through binaries built from main and from this merge, capturing document, stderr and exit code each. diff -r over the two trees is empty.
The pre-parse index was built through a package-level function variable so a test could shrink its node bound and count its calls. That is mutable package-level state in a pipeline stage, which the compiler has nowhere else, and it forced the three tests that used it to run sequentially: rebinding a value the parallel tests around them also read would otherwise race. Carry the builder as an unexported field of load.Options instead. The bound becomes an input to the stage like every other option, a test's choice is visible only to the load it passes it to, and the three tests are now parallel. Loads that leave it nil — everything outside this package's tests — index under the compiler's own bound exactly as before.
TestBuild_TracksAncestorsOnlyToItsDepthBound places its alias one level past maxTrackedDepth, so it holds only that a node beyond the bound is out of the walk's reach. Narrowing the comparison to f.depth < maxTrackedDepth left the whole suite green: an alias past the bound is unreachable either way, and no fixture puts one at the bound itself. That off-by-one is a real regression against the recursive descent this walk replaced, which tracked to depth 10000 inclusive and would still have caught a recursive anchor there. Add the control that separates the two — an alias at exactly the bound is still found — so the bound is pinned from both sides.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
One compile parsed the same source bytes twice and then walked the resulting node tree four times.
Counted by instrumenting a copy of the tree at
dbf0054and compilingtestdata/golden/openapi/petstore.yaml(280 YAML nodes):
scan.scanCycles,load.decodeanchorCycle,refScan.collect,rawNodeCount,aliasWeigher.weighload.decodesourceindex.Build,refScan.collect,aliasWeigher.weighcompilers/openapi/internal/sourceindexwalks a decoded tree once and returns an immutable valueholding what the pre-parse refusals used to walk it separately to ask: the document's own node
count, and the first alias whose target is one of its own ancestors. The loader decodes, indexes,
and hands the index to
scan, which reads those answers instead of deriving them. The two walksthat only ever asked something about the tree itself become one;
refScan.collectand the aliasweigher stay, because each asks something the tree alone does not answer — the first reads mappings
through the merge- and alias-aware view, the second needs the allowance the node count produces.
The index is a value the loader threads, never a package-level cache, so stages stay pure and
reentrant and nothing is memoized across
Compilecalls. The bound it walks under is anunexported
load.Optionsfield, so the tests that shrink it or count its builds hand it to oneload rather than rebinding shared state, and run in parallel with everything else. Its walk is bounded by a named
MaxIndexedNodesrather than by the input, and a tree past that bound is refused under a newopenapi/source-too-largecode rather than half-counted: every answer in a truncated index ispartial, including the node count the alias-expansion allowance is derived from, and an allowance
computed from a count that stopped early would refuse documents on a bound they never crossed. The
bound sits far above anything that could have been decoded in the first place, so no document in
any corpus reaches it.
decodeis now the compile's only parse, and so the only place the yaml.v3 alias budget is spent —which is what bounds a billion-laughs expansion before anything walks it. The two parses were
byte-identical calls on the same input, so folding them does not widen that bound; the doc comment
that claimed the scan had already decoded under a barrier is corrected rather than left to rot.
What this does not do. The other half of the same problem is the lookups performed during
lowering:
annotation.RawChildNode(442 calls on petstore) andRawPropertyNode(430) linearlyrescan a mapping's
Contentfor a key the parse already located, andnodeview.PointerPathrestarts from the document root per pointer. Those counts are unchanged here. They are read from
schema,operation,authandannotationitself — roughly thirty call sites, none of which canreach the index without widening
lowering.Ctxand the signatures beneath it — so they are theirown change, filed as #338.
docs/micro-compiler-design.md§10, which recorded the whole thing asblocked on
$refdefects that have since closed, is updated to say which half landed and which didnot.
CLAUDE.mdcarried a count of the compiler's internal packages that this would have made stale byadding one, filed as #339 rather than edited here. #301 has since removed it — the layering section
now says the diagram deliberately neither names nor counts them, and points at
archtest'srulesmap instead — so #339 is closed and there is nothing left to edit.
Merge order. #304 landed first, against
internal/scanandinternal/nodeviewfor a differentdefect. It added a
TestDetectCycles_EmptyPointerSegmentIsRefusedcallingCycles(0, []byte(src));this change makes
Cyclestake asourceindex.Index, so that one call is nowscanBytes(t, []byte(src))— already carried here by the merge ofmain. Nothing else overlapped —PointerPath, which #304 rewrote, is untouched here.Test plan
gofmt,go vet,golangci-lint,go build, and the coverage gate at exactly100%.
no
-updaterun. Beyond that, both binaries compiled every spec source undertestdata/— theset
find testdata -type f \( -name '*.yaml' -o -name '*.yml' -o -name '*.json' \) ! -name '*.golden.json'derives, 115 files at 3bc0692 — capturing the emitted document, stderrdiagnostics and exit code for each;
diff -rover the two output trees is empty.cmd/morphic-harness testdata— the six oracles, including the two-ordercomparison — produces identical output before and after.
internal/sourceindextests cover the count, the anchor-cycle answer and which of several isreported, legal anchor reuse, an alias with no target, the node bound, and that a truncated index
reports truncation rather than a clean document. A pair pins the ancestor-tracking depth to what
the recursive descent it replaces reached, from both sides: an alias past the bound is out of the
walk's reach while the nodes there are still counted, and an alias at exactly the bound is still
found.
TestLoad_IndexesTheSourceOnceasserts the loader builds exactly one index per source, andTestLoad_IndexesAPatchedTreeAgainthat an overlay makes it exactly two. Planting a secondrefusalscall inLoadreddens the first; without that assertion nothing would have caught are-added walk, since the refusals would still be correct.
the truncation tests in both packages, never recording an anchor cycle reddens the sourceindex
tests plus
TestDetectCycles_ReproducersandTestCompile_CyclicSpecDoesNotCrash, miscountingalias nodes reddens
TestAliasAmplification_BoundaryPairand the flat-fan-out refusal, andremoving the depth gate reddens the tracking-bound tests. Narrowing that gate by one, to
f.depth < maxTrackedDepth, is the case the at-the-bound test exists for — it is the only thingin the suite that reddens on it.
Compileon petstore, run interleaved on both trees: 15,017 → 13,885allocs/op and 1,400,625 → 1,322,837 B/op (both stable across runs), best-of-nine wall time
1.60 ms → 1.51 ms. The benchmark was not kept — it measured the change rather than guarding it,
and the index-count tests are what hold the shape.
Closes #179