feat(optimize): surface vector index layout so coverage answers can-it-prune

[memmmmike]

Resolves #486.

Coverage currently answers "does an index exist over these rows" but not "can it prune" — the exact blindspot from the #432 incident, where pending_indexes: [] reported a healthy index while every nearest() flat-read ~10 GB.

What this adds (reporting only; the repair remains #485):

• TableOptimizeStats.vector_index_layouts: per built vector index — name, partition summary, indexed rows, worst rows-per-partition across delta segments, and a degenerate flag.
• The flag is computed per segment: Lance searches delta segments independently, so a well-partitioned later segment cannot repair a mono-partition earlier one. It fires on rows-per-partition (> 4096, generous slack over ~sqrt(N) healthy sizing), which catches both the mono-partition case and the incident's own 8-partitions-over-852k shape — partition count alone would miss the latter.
• CLI: layouts print under each table; degenerate ones loudly, with the worst segment's rows/partition. --json carries the full struct.
• Statistics missing the expected fields are a loud error, not a silent skip — a silent skip would recreate the exact failure mode this issue is about. lance_surface_guards now pins num_indexed_rows and num_indexed_rows_per_delta (aligned with indices[]) beside the existing num_partitions pin, so a Lance upgrade that changes the shape fails the guard instead of degrading the stats.

Scope checked: skipped_for_drift intentionally reports no layout (that HEAD is not manifest-accepted); TableOptimizeStats is not serialized by the server, so no OpenAPI drift.

Tests: classifier unit tests including the incident shape and the bad-segment-behind-good-segment case; an end-to-end maintenance test asserting layouts surface through optimize() on a real vector index; the guard pins. Maintenance suite 34/34. CLI smoke on a local graph verified both text and --json output, and a no-vector-column graph stays clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01E4S9Xpe9G7pLEv9JaoXUAb

Greptile Summary

The PR extends optimize statistics with physical vector-index layouts so callers can distinguish index coverage from actual pruning capability.

• Collects per-segment Lance partition and indexed-row statistics.
• Classifies layouts using the worst rows-per-partition segment.
• Exposes layout details through library statistics and CLI JSON/human output.
• Adds classifier, Lance-surface, and end-to-end maintenance coverage.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains.

No blocking failure remains.

Important Files Changed

Filename	Overview
crates/omnigraph/src/table_store.rs	Parses Lance vector-index statistics into aligned per-segment partition and row counts with loud shape validation.
crates/omnigraph/src/db/omnigraph/table_ops.rs	Defines the public layout report and classifies degeneracy from the worst independently searched delta segment.
crates/omnigraph/src/db/omnigraph/optimize.rs	Threads vector-index layouts through the optimize result paths.
crates/omnigraph-cli/src/main.rs	Adds vector-index layout details to optimize JSON and human-readable output.
crates/omnigraph/tests/maintenance.rs	Verifies that optimize surfaces a real vector index's physical layout.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Lance[Lance index statistics] --> Store[TableStore layout parser]
  Store --> Classifier[Per-segment degeneracy classifier]
  Classifier --> Stats[TableOptimizeStats]
  Stats --> JSON[CLI JSON output]
  Stats --> Human[CLI human output]

Loading

_{Reviews (2): Last reviewed commit: "chore: satisfy the fmt and clippy gates" | Re-trigger Greptile}

Context used:

• Knowledge Base — OmniGraph core storage: manifest, table store, and Lance access
• Knowledge Base — CLI and Config

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

[greptile-apps]

Document vector layout output

optimize --json now emits vector_index_layouts, and the human output reports healthy and degenerate layouts, but the maintenance documentation still enumerates the old field set. Update docs/user/operations/maintenance.md in this PR so operators and JSON consumers can discover the new layout, partition, and degeneracy fields.

Context Used: CLAUDE.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[memmmmike]

Heads-up while this waits for a CI approval: both gates would have failed on the first run, so I've fixed them in 9a44f04 (no behavior change).

• cargo fmt --all --check: three files this PR touches were unformatted.
• cargo clippy --workspace --all-targets -- -D warnings: manual_contains on the degenerate-partition check — segment_partitions.iter().any(|p| *p == 0) is now segment_partitions.contains(&0).

Since CI on this branch is action_required and has never executed (0s), neither gate had reported, so this was invisible from the PR page. fmt, clippy, and the lib / maintenance / lance_surface_guards suites (296 / 32 / 34) are green locally on 1.97.1.