perf(merge): read only new catalog commits when opening branch state

[azimafroozeh]

What & why

Second PR of the merge-latency series, following #540. With the walk removed there, merge cost tracks commit-history depth: opening a branch's state for a merge re-derives its commit projection from a full __manifest scan every time (~1.5 s p50 and ~24.6k catalog GET calls per merge on a 3500-commit store), and history only grows. This PR makes that read incremental: reuse the already-decoded state when it is provably current, read only the catalog commits appended since, and degrade to exactly today's full scan on anything unprovable.

• Per-branch state cache on the engine handle, revalidated per use by the existing manifest-incarnation probe (branch-identifier fenced, so delete/recreate forces a full re-open; entries purge on delete, evict on refresh failure).
• On probe mismatch, diff the catalog's own fragment lists and fold ONLY the appended fragments' live rows into retained accumulators; the fold and the full scan share one reduction implementation. Head-row updates arrive as deletion-vector growth and drop exactly the superseded heads; any other dead row, fragment rewrite, or inconsistency fails closed to the full scan.
• Debug builds verify every fold against a full scan of the same version and fail loudly on divergence, so the whole merge suite is the fold's oracle; two probe counters make the path structurally testable.

Measurement (local FS; merge runs via #537's harness)	before	after	reduction
One branch-state refresh, 1,500-commit catalog (isolated, release)	141.7 ms	3.5 ms	40x, growing with depth
OuterPrepare p50, 3,500-commit store, repeat merges	1576 ms	815 ms	48%
Catalog GET calls per merge, same shape	24,721	14,169	43%

The repeat-merge shape re-diverges one long-lived branch pair per rep (the pattern the cache serves; fresh-branch runs cannot produce a hit). The residual is the non-FF base snapshot's historical scan, the publish path's per-attempt scan, and the publishing coordinator's post-publish rebuild: named follow-ups.

Backing issue / RFC

• None filed yet; motivated by the measured O(history) branch-state cost (RFC document owed as a follow-up).

Checklist

• Change is focused (one mechanism: incremental branch-state reads for merge)
• Tests added/updated for behavior changes (merge_projection_cache.rs: incremental refresh pinned by probe counters, delete/recreate fenced via merge outcome; fold oracled against the full scan across the whole merge suite in debug; ignored release micro-benchmark of the refresh)
• Public docs updated if user-facing surface changed (internal, no public surface; vocabulary inventory updated for the moved and new diagnostics)
• Reviewed against docs/dev/invariants.md: no Hard Invariant weakened, no deny-list item hit (nearest edge: state reuse rides the same incarnation currency the bound-branch fast path already trusts, fail-closed otherwise)

Local verification

• cargo test -p omnigraph-engine (debug): green; one pre-existing sandbox-only failure (external-blob special-file test, unrelated)
• cargo test -p omnigraph-engine --release --lib projection_refresh_deep_catalog -- --ignored --nocapture: the 40x row above
• cargo clippy -p omnigraph-engine --all-targets, cargo fmt --all, vocabulary guard (all three surfaces): clean

Notes for reviewers

• No semantic change: the captured branch state is byte-equivalent to a fresh open (debug-oracled); only the read cost moves.
• Cache is bounded by live branch count; its mutex serializes merge captures, acceptable because the schema serial queue already serializes merges there.
• Memory: one extra lineage-row copy per cached coordinator.
• The publishing coordinator still pays one full rebuild per merge (fence-cleared by its own publishes); folding the publish delta instead is a follow-up whose red test ships here.
• Deliberately untouched: the publish path's per-attempt scan and the non-FF base snapshot's historical scan.

Greptile Summary

The PR adds a per-branch merge-authority cache and incrementally refreshes retained manifest projections by folding newly appended fragments and graph-head deletion-vector changes, while falling back to a full scan for unrecognized transitions.

• Refactors manifest-state reduction into reusable projection accumulators.
• Adds branch-incarnation fencing and cache eviction for stale or deleted branches.
• Adds probe counters, structural merge tests, and a deep-catalog microbenchmark.

Confidence Score: 5/5

The PR appears safe to merge; the incremental path is fenced by branch identity and fragment-shape checks and falls back to a full manifest scan when the delta cannot be proven.

The retained projection remains coupled to its exact manifest version, normal Lance merge-insert updates match the modeled new-fragment and immutable deletion-vector shape, and compaction, restoration, branch recreation, malformed deltas, or refresh failures cannot silently reuse the cached state.

Important Files Changed

Filename	Overview
crates/omnigraph/src/db/manifest.rs	Adds retained projection state, incremental fragment/deletion-vector refresh, branch-incarnation fencing, and fail-closed full-scan fallback.
crates/omnigraph/src/db/manifest/state.rs	Consolidates manifest reduction into projection accumulators and supports fragment-restricted delta decoding.
crates/omnigraph/src/db/omnigraph.rs	Adds per-branch merge-authority coordinator caching with probe-based refresh, eviction, and branch-delete purging.
crates/omnigraph/src/instrumentation.rs	Adds test-scoped counters distinguishing incremental and full projection refreshes.
crates/omnigraph/tests/merge_projection_cache.rs	Verifies incremental refresh after repeated merges and correct fencing after branch deletion and recreation.
crates/omnigraph/src/db/manifest/tests.rs	Adds an ignored release microbenchmark comparing full and incremental refresh costs.
tools/omnigraph-vocabulary-guard/graph-vocabulary-inventory.tsv	Updates generated vocabulary inventory entries for the new diagnostics and documentation.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Merge requests branch authority] --> B{Cached coordinator exists?}
  B -- No --> C[Open branch and fully scan manifest]
  C --> D[Retain state and projection accumulators]
  B -- Yes --> E[Probe latest manifest incarnation]
  E --> F{Incarnation current?}
  F -- Yes --> G[Reuse cached authority]
  F -- No --> H{Branch identifier unchanged and delta provable?}
  H -- Yes --> I[Remove superseded graph heads]
  I --> J[Fold live rows from new fragments]
  J --> K[Install refreshed state and lineage]
  H -- No --> L[Full manifest scan]
  L --> D
  D --> G
  K --> G
  G --> M[Capture snapshot and commit graph for merge]

Loading

_{Reviews (1): Last reviewed commit: "perf(merge): read only new catalog commi..." | Re-trigger Greptile}

Context used:

• Knowledge Base — OmniGraph core storage: manifest, table store, and Lance access
• Knowledge Base — Branching and merge

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.