[prrao87]

Why

refresh() applies Lance writes to the CSR as a delta over its base, and rebuilds on its own only once pending changes pass 25% of the base. Below that threshold every lookup also reads the delta, and there was no way to fold it in short of rebuilding the engine. #93 added the reporting half of #75 (base_version, unindexed_edges); this adds the explicit optimize step, so users choose when to rebuild. Closes #75.

What changed

• optimize_csr(), in Python and Rust (engine.optimize_csr().await), rebuilds every CSR that carries a delta at the version the engine is pinned to. Afterwards csr_stats() reports update "rebuild", delta_edges, tombstones and unindexed_edges at 0, and base_version equal to version. A CSR without a delta, node indexes and everything else are reused, and without use_csr=True it does nothing. Query results never change.
• It doesn't move to newer versions; refresh() stays the call for that. Both share one state-building path, and optimize_csr() only adds a flag that sends delta-carrying CSRs straight to a full build. It takes refresh()'s lock, so the two run one at a time, and the replacement is built in full before it's swapped in.
• Why a separate call: refresh() is meant to be cheap enough to run on a timer, while a rebuild scans every edge's endpoint columns again. Keeping them apart follows Lance, where picking up new data and optimize_indices() are separate steps.

The CSR guide's new "Folding the delta in" section says this mirrors Lance's index design and maps one onto the other:

Lance index	CSR
The fragments an index was built from	The base CSR, at base_version
Unindexed rows, scanned alongside the index	The delta, read by every lookup
num_unindexed_rows in index statistics	unindexed_edges in csr_stats()
optimize_indices()	optimize_csr()

It links Lance's indexing guide and index format, runs an example, and adds a tip on when to reach for optimize_csr() rather than refresh(): after a bulk load, before a latency-sensitive run of queries, or on a schedule, not after every refresh. The update field description, the "How to keep the CSR fresh" steps, the Python and Rust comparison table and the docstring (which feeds the API reference) mention it too.

User impact: one new method on CypherEngine in Python and Rust. Existing calls behave as before.

Validation

• Rust: an append that stays a delta, then optimize_csr(), checking every stats field and the query results, and that a second call reuses the CSR.
• Python: mixed appends and deletes folded, with results equal to a fresh join-path engine and a fresh CSR engine; an unrefreshed write stays unseen after optimize_csr(); no-op without use_csr.
• cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace, make test (1660 passed) and make lint pass locally, as do docs/check_snippets.py (the new example prints rebuild 0), zensical build --strict and typos. CI is pending.
• No benchmark runs: no query path changes, and the new code runs only when optimize_csr() is called.

🤖 Generated with Claude Code

[cloudflare-workers-and-pages]

Deploying lance-csr with    Cloudflare Pages

Latest commit:	8ee2ca3
Status:	✅  Deploy successful!
Preview URL:	https://d679b183.lance-csr.pages.dev
Branch Preview URL:	https://feat-csr-optimize.lance-csr.pages.dev

View logs