Skip to content

perf(compilers/openapi): index the source tree once per compile - #340

Merged
OmarAlJarrah merged 4 commits into
mainfrom
perf/openapi-index-source-tree-once
Aug 9, 2026
Merged

perf(compilers/openapi): index the source tree once per compile#340
OmarAlJarrah merged 4 commits into
mainfrom
perf/openapi-index-source-tree-once

Conversation

@OmarAlJarrah

@OmarAlJarrah OmarAlJarrah commented Aug 9, 2026

Copy link
Copy Markdown
Member

Summary

One compile parsed the same source bytes twice and then walked the resulting node tree four times.
Counted by instrumenting a copy of the tree at dbf0054 and compiling testdata/golden/openapi/petstore.yaml
(280 YAML nodes):

parses raw-tree walks
before 2 — scan.scanCycles, load.decode 4 — anchorCycle, refScan.collect, rawNodeCount, aliasWeigher.weigh
after 1 — load.decode 3 — sourceindex.Build, refScan.collect, aliasWeigher.weigh

compilers/openapi/internal/sourceindex walks a decoded tree once and returns an immutable value
holding what the pre-parse refusals used to walk it separately to ask: the document's own node
count, and the first alias whose target is one of its own ancestors. The loader decodes, indexes,
and hands the index to scan, which reads those answers instead of deriving them. The two walks
that only ever asked something about the tree itself become one; refScan.collect and the alias
weigher stay, because each asks something the tree alone does not answer — the first reads mappings
through the merge- and alias-aware view, the second needs the allowance the node count produces.

The index is a value the loader threads, never a package-level cache, so stages stay pure and
reentrant and nothing is memoized across Compile calls. The bound it walks under is an
unexported load.Options field, so the tests that shrink it or count its builds hand it to one
load rather than rebinding shared state, and run in parallel with everything else. Its walk is bounded by a named
MaxIndexedNodes rather than by the input, and a tree past that bound is refused under a new
openapi/source-too-large code rather than half-counted: every answer in a truncated index is
partial, including the node count the alias-expansion allowance is derived from, and an allowance
computed from a count that stopped early would refuse documents on a bound they never crossed. The
bound sits far above anything that could have been decoded in the first place, so no document in
any corpus reaches it.

decode is now the compile's only parse, and so the only place the yaml.v3 alias budget is spent —
which is what bounds a billion-laughs expansion before anything walks it. The two parses were
byte-identical calls on the same input, so folding them does not widen that bound; the doc comment
that claimed the scan had already decoded under a barrier is corrected rather than left to rot.

What this does not do. The other half of the same problem is the lookups performed during
lowering: annotation.RawChildNode (442 calls on petstore) and RawPropertyNode (430) linearly
rescan a mapping's Content for a key the parse already located, and nodeview.PointerPath
restarts from the document root per pointer. Those counts are unchanged here. They are read from
schema, operation, auth and annotation itself — roughly thirty call sites, none of which can
reach the index without widening lowering.Ctx and the signatures beneath it — so they are their
own change, filed as #338. docs/micro-compiler-design.md §10, which recorded the whole thing as
blocked on $ref defects that have since closed, is updated to say which half landed and which did
not. CLAUDE.md carried a count of the compiler's internal packages that this would have made stale by
adding one, filed as #339 rather than edited here. #301 has since removed it — the layering section
now says the diagram deliberately neither names nor counts them, and points at archtest's rules
map instead — so #339 is closed and there is nothing left to edit.

Merge order. #304 landed first, against internal/scan and internal/nodeview for a different
defect. It added a TestDetectCycles_EmptyPointerSegmentIsRefused calling Cycles(0, []byte(src));
this change makes Cycles take a sourceindex.Index, so that one call is now
scanBytes(t, []byte(src)) — already carried here by the merge of main. Nothing else overlapped —
PointerPath, which #304 rewrote, is untouched here.

Test plan

  • Full gate passes: gofmt, go vet, golangci-lint, go build, and the coverage gate at exactly
    100%.
  • Byte-identical output, proven by execution. Every golden and conformance snapshot passes with
    no -update run. Beyond that, both binaries compiled every spec source under testdata/ — the
    set find testdata -type f \( -name '*.yaml' -o -name '*.yml' -o -name '*.json' \) ! -name '*.golden.json' derives, 115 files at 3bc0692 — capturing the emitted document, stderr
    diagnostics and exit code for each; diff -r over the two output trees is empty. cmd/morphic-harness testdata — the six oracles, including the two-order
    comparison — produces identical output before and after.
  • New internal/sourceindex tests cover the count, the anchor-cycle answer and which of several is
    reported, legal anchor reuse, an alias with no target, the node bound, and that a truncated index
    reports truncation rather than a clean document. A pair pins the ancestor-tracking depth to what
    the recursive descent it replaces reached, from both sides: an alias past the bound is out of the
    walk's reach while the nodes there are still counted, and an alias at exactly the bound is still
    found.
  • TestLoad_IndexesTheSourceOnce asserts the loader builds exactly one index per source, and
    TestLoad_IndexesAPatchedTreeAgain that an overlay makes it exactly two. Planting a second
    refusals call in Load reddens the first; without that assertion nothing would have caught a
    re-added walk, since the refusals would still be correct.
  • Each new test was checked to bite by planting the defect it names: dropping the node bound reddens
    the truncation tests in both packages, never recording an anchor cycle reddens the sourceindex
    tests plus TestDetectCycles_Reproducers and TestCompile_CyclicSpecDoesNotCrash, miscounting
    alias nodes reddens TestAliasAmplification_BoundaryPair and the flat-fan-out refusal, and
    removing the depth gate reddens the tracking-bound tests. Narrowing that gate by one, to
    f.depth < maxTrackedDepth, is the case the at-the-bound test exists for — it is the only thing
    in the suite that reddens on it.
  • Throwaway benchmark over Compile on petstore, run interleaved on both trees: 15,017 → 13,885
    allocs/op and 1,400,625 → 1,322,837 B/op (both stable across runs), best-of-nine wall time
    1.60 ms → 1.51 ms. The benchmark was not kept — it measured the change rather than guarding it,
    and the index-count tests are what hold the shape.

Closes #179

One compile parsed the same source bytes twice — once in the pre-parse cycle
scan, once in the loader — and then walked the resulting node tree four times:
the recursive-anchor descent, the pure-$ref collection, the raw node count, and
the alias weigher, each starting from the root with no shared state.

internal/sourceindex walks the decoded tree once and answers what the anchor
descent and the node count each walked it to ask. The loader decodes, indexes,
and hands the index to scan, so a source is parsed once and the two walks that
only ever asked something about the tree itself become one.

The walk is bounded by MaxIndexedNodes rather than by the input. A tree past it
is refused under a new openapi/source-too-large code rather than half-counted:
every answer in a truncated index is partial, including the node count the
alias-expansion allowance is derived from, and an allowance computed from a
count that stopped early would refuse documents on a bound they never crossed.

Behaviour is unchanged. Every golden and conformance snapshot is byte-identical
without regeneration, and the whole testdata corpus compiles to identical
documents, diagnostics and exit codes before and after.
The conflict this branch predicted, resolved as it said: #304's
TestDetectCycles_EmptyPointerSegmentIsRefused calls Cycles(0, []byte(src)), and
Cycles now takes a sourceindex.Index, so the call becomes scanBytes(t, ...).

Two the branch could not predict. cycles_test.go: #304 and #310 added a test
where this branch adds a scanIndex helper, both at the same offset sharing a
closing brace — kept both. And #328 consolidated the openapi test scaffolding
into internal/openapitest after this branch was written, so its new
entry_internal_test.go calls sourceOf by the package-local name that no longer
exists; repointed, with the import added.

Byte-identical output re-proven against the merged tree rather than carried over
from the branch: 115 sources under testdata compiled through binaries built from
main and from this merge, capturing document, stderr and exit code each. diff -r
over the two trees is empty.
The pre-parse index was built through a package-level function variable so a
test could shrink its node bound and count its calls. That is mutable
package-level state in a pipeline stage, which the compiler has nowhere else,
and it forced the three tests that used it to run sequentially: rebinding a
value the parallel tests around them also read would otherwise race.

Carry the builder as an unexported field of load.Options instead. The bound
becomes an input to the stage like every other option, a test's choice is
visible only to the load it passes it to, and the three tests are now parallel.
Loads that leave it nil — everything outside this package's tests — index under
the compiler's own bound exactly as before.
TestBuild_TracksAncestorsOnlyToItsDepthBound places its alias one level past
maxTrackedDepth, so it holds only that a node beyond the bound is out of the
walk's reach. Narrowing the comparison to f.depth < maxTrackedDepth left the
whole suite green: an alias past the bound is unreachable either way, and no
fixture puts one at the bound itself.

That off-by-one is a real regression against the recursive descent this walk
replaced, which tracked to depth 10000 inclusive and would still have caught a
recursive anchor there. Add the control that separates the two — an alias at
exactly the bound is still found — so the bound is pinned from both sides.
@OmarAlJarrah
OmarAlJarrah merged commit 3a6cfd0 into main Aug 9, 2026
1 check passed
@OmarAlJarrah
OmarAlJarrah deleted the perf/openapi-index-source-tree-once branch August 9, 2026 13:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

compilers/openapi: index the source tree once instead of rediscovering it

1 participant