perf(merge): classify branch merges from fragment lineage

[azimafroozeh]

What & why

Branch merges now read only what changed. The classifier used to run full ordered scans of base, source, and target for every diverged table, so a merge cost O(rows) no matter how small the delta. Candidates now come from a Lance manifest diff per (base, side) pair: byte-identical fragments are skipped unread, new or rewritten fragments contribute their live rows, changed deletion files contribute exactly the deletion-vector difference. The classification loop itself is unchanged and runs over the candidates through chunked exact id IN filters, so merge cost tracks the delta.

Correctness is structural: Lance never rewrites a file behind its name, so every changed row must sit in a changed fragment or a deletion-vector diff, and false positives simply classify as unchanged. A fail-closed gate falls back to the full scan whenever any precondition is unprovable (Blob schema, schema/path/version-pin mismatch, missing stable row ids or the exact-id primary-key contract, non-linear history, candidate byte budget).

Measured with the end-to-end benchmark harness from #537 (release, local FS, warm, 5 reps, frozen validated fixtures, A/A floor under 1%):

scenario	full scan	lineage	change
m3 T=8 N=100k d=1	751.9 ms	79.1 ms	9.5x
m3 T=8 N=100k d=50	2910.8 ms	130.2 ms	22.4x
m3 T=8 N=100k d=5000	2955.9 ms	312.5 ms	9.5x

Phase attribution puts the entire improvement on the removed scan phase, and write-path counters are byte-identical between modes: the classification route is the only behavioral change.

Backing issue / RFC

• Delivers the goal of docs/rfcs/0027-lineage-merge-deltas.md (merge reads O(delta)) and unblocks both of its named blockers via fragment plus deletion-vector metadata, no new Lance primitives needed. RFC text update to follow.

Checklist

• Change is focused (candidate discovery plus its gate; classification, staging, and publish untouched)
• Tests: debug builds default to verify mode, which runs both paths per merge and fails loudly on any divergence, so the whole merge suite is a scan-vs-lineage oracle: 72/72 green, zero divergences
• Public docs: docs/user/reference/constants.md documents OMNIGRAPH_MERGE_LINEAGE and the gate
• Reviewed against docs/dev/invariants.md — no Hard Invariant weakened, no deny-list item hit

Local verification

• cargo fmt --all --check — clean
• cargo clippy --workspace --all-targets — clean, no new warnings
• cargo test --workspace --locked — green
• engine merge suites — 72 passed, all as verify-mode oracle runs

Notes for reviewers

• Release builds run lineage by default (unset = on); OMNIGRAPH_MERGE_LINEAGE=off restores today's behavior exactly and is the documented fallback. A mistyped value warns and behaves as off, so the kill switch cannot misfire on.
• Engine footprint: exec/merge.rs (discovery, gate, dispatch) plus one pub(crate) scan-tuning door in table_store.rs.
• On history-deep stores the remaining cost is O(commit-history) prepare/publish, already measured and tracked for a follow-up PR.

Greptile Summary

The PR changes divergent branch-merge classification to discover candidate entities from Lance fragment and deletion-vector lineage, while retaining a fail-closed full-scan fallback and a debug verification mode.

• Adds lineage-based candidate discovery and exact-id filtered classification.
• Preserves the existing three-way classification and atomic publication paths.
• Documents OMNIGRAPH_MERGE_LINEAGE and updates vocabulary inventory metadata.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains.

No blocking failure remains.

Important Files Changed

Filename	Overview
crates/omnigraph/src/exec/merge.rs	Adds gated lineage candidate discovery, filtered cursor execution, verification mode, and shared three-way classification without an eligible follow-up defect.
crates/omnigraph/src/table_store.rs	Exposes the existing exact-id primary-key validation helper for the lineage gate.
docs/user/reference/constants.md	Documents the lineage merge mode, defaults, verification behavior, and full-scan fallback.
tools/omnigraph-vocabulary-guard/graph-vocabulary-inventory.tsv	Registers new physical-storage terminology introduced by candidate discovery.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Base, source, and target snapshots] --> B{Lineage preconditions proven?}
  B -- No --> C[Full ordered three-way scan]
  B -- Yes --> D[Diff fragment and deletion metadata]
  D --> E[Collect and chunk candidate ids]
  E --> F[Exact id filtered cursors]
  F --> G[Shared three-way classifier]
  C --> G
  G --> H[Stage deltas and deletions]
  H --> I[Validate and atomically publish merge]

Loading

_{Reviews (3): Last reviewed commit: "review" | Re-trigger Greptile}

Context used:

• Knowledge Base — Branching and merge
• Knowledge Base — OmniGraph core storage: manifest, table store, and Lance access

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.