feat: integrate cypher, csr and engine work for 0.3.0 by prrao87 · Pull Request #55 · hyperdimensionalcomputingai/lance-csr

prrao87 · 2026-10-26T17:32:02Z

Why

A large set of Cypher, CSR and engine improvements was developed on a stack of branches and reviewed as PRs #30–#52. Most of those PRs merged into intermediate branches rather than main, and they predate the crate rename and the move of CypherEngine into the Rust crate (#53). This PR brings the final integrated version of that work onto main as one reviewed change, before the 0.3.0 release.

The work arrives as a merge of codex/issue-resolution-current-main, the final integration branch, so every original commit and its authorship is kept. Fixes for problems found in review follow as separate commits.

What changed

Cypher

Typed list and map parameters, and parameters in SKIP/LIMIT.
Numeric functions (abs, round, floor, ceil, sqrt) and string and list functions (left, right, substring, replace, split, trim, reverse, last, tail, size, range with a step).
keys(), labels(), reduce, and the any/all/none/single predicates.
UNION / UNION ALL, named path values with relationships(), startNode() and endNode(), and read-only CALL { ... } subqueries.
Schema-aware type errors that name the expression and its operand types.

CSR and execution

Iterative variable-length traversal, including zero-hop paths and mixed relationship types.
Filter-column reuse during target materialization, plus faster size([x IN list WHERE ...]).
Cooperative query deadlines through a keyword-only timeout= on execute, execute_batches and CypherQuery.execute.
Per-snapshot plan templates for repeated queries.
target_partitions on CypherEngine.
Python type stubs updated for the new parameters.

Integration with the Rust engine. The Python engine changes from the original branches are ported into lance_csr::CypherEngine (engine.rs):

The plan template cache.
target_partitions, through a new CypherEngine::with_target_partitions.
Lower-cased parameter names.

The Python class keeps argument validation, timeouts and the deadline-aware batch reader. Rust callers get the same cooperative deadlines through lance_csr::query::with_execution_deadline, now a documented API with a "Timeouts" section in the Rust guide, a compiled example and a test. The engine unit tests moved into engine.rs, where CI runs them; tests in the Python crate never run in CI. refresh() keeps the lock-based fix from #53 instead of the branch's retry loop; both fix #12.

Fixes for problems found in review (each with a regression test):

String-number comparisons such as p.age = '30' run again. The new type check had started rejecting them. DataFusion's coercion decides, as before.
Unlabeled variable-length targets inherit the source label again. MATCH (a:Person)-[:KNOWS*1..2]->(b) failed with "No mapping found for node label: Node" when relationships have no endpoint labels.
to_sql works again for Spark, PostgreSQL, MySQL and SQLite on queries with typed nulls (coalesce(x, null), CASE ... ELSE null). Those dialects get a bare NULL, as before.
Mixing UNION and UNION ALL in one query raises ValueError, as openCypher requires.
Named paths filtered on the start node work. MATCH p = (a)-[...]->(b) WHERE a.id = 1 failed to plan, because DataFusion copied the filter into the path operator's inventory inputs.
target_partitions is capped at 1024. Huge values used to be accepted and could exhaust memory instead of raising.
A test now covers the relationship join-key lookup from refactor(graph): expose read-only mapping accessors #51.

Known limitations (not changed here)

count(DISTINCT p) on a path, and count(var) for variables without an id column, raise an error rather than counting.
On main too: a relationship variable reused across comma-separated pattern parts falls back to a cross join, and (a)-[r1]->(b)<-[r2]-(c) can bind r1 and r2 to the same relationship. Ascending ORDER BY puts nulls first.
CALL subqueries plan and scan per outer row, so they get slow beyond a few thousand outer rows. The docs say so.
A WHERE on a path's start node is applied above the path operator rather than inside the start scan.

Validation

cargo fmt --check and cargo clippy --all-targets -- -D warnings pass in both crates.
cargo test --workspace and cargo check -p lance-csr-benches --benches pass. cargo run -p lance-csr --example embedded_query prints the documented output.
In python/: make install, make test and make lint pass, with the full Python suite including the new feature suites. docs/check_snippets.py reports no failures.
zensical build --strict and typos pass.
Each reviewer-reported bug was reproduced before its fix and checked after it, with and without the CSR.

Benchmarks

I compared the published 0.2.0 wheel with a release wheel of this branch (same thin-LTO release profile) on both suites, using the published settings, on an Apple M5. To cancel drift, each suite and mode ran in ABBA order (0.2.0, branch, branch, 0.2.0), and the table compares averages. All 88 result assertions passed in every run. Values are the geometric mean of the per-query median ratio (branch / 0.2.0; below 1 is faster):

Suite	In memory	Lazy
IC1–IC14 (GDC Interactive complex reads)	0.75	0.78
30-query (GDC SNB SF1)	0.97	1.00

Faster: IC12 is about 7× faster (152 → 21 ms in memory, 190 → 33 ms lazy), IC13 about 4×, IC14 about 1.35×, and IC3 and IC11 about 1.2–1.3×.
Slower: two queries regress by more than run-to-run noise. A focused re-run with 15+ rounds, ABBA order and non-overlapping ranges confirmed both:
- 30-query Q30: about 10–14% slower (41.7 → 45.9 ms in memory, 41.2 → 46.9 ms lazy).
- IC6 in memory: about 14% slower (22.5 → 25.8 ms).
The cause isn't established. Per-batch overhead from bounded-chunk CSR expansion on high fan-out traversals is a candidate, but I haven't verified it. The remaining queries are within noise.
Engine lifecycle: engine_cache.py (20 tables, 5 interleaved runs per side) shows no change. Construction, first query, warm query and unchanged refresh are all within ±2%.

Published benchmark tables are unchanged. The crates.io publish and the release wheels follow after this merges.

🤖 Generated with Claude Code

Keep integer lists as Int64 values and convert numeric lists only at vector boundaries. Add singleton-row planning so standalone RETURN examples execute on configured engines. Issue #3. Implemented with assistance from Codex; coordinated runtime and benchmark validation pending.

Preserve integer IDs in nested parameter expressions and round-trip containers through CypherQuery.parameters. Validate keys, integer bounds and nesting; document Arrow list coercion. Part of #20; based on #3. Implemented with assistance from Codex. Runtime and benchmark validation pending.

Retry refresh when another replacement has been published since its starting snapshot. Add deterministic publication coverage and preserve failed-refresh and active-reader behavior. Resolves #12. Implemented with assistance from Codex; runtime and benchmark validation pending.

Migrate mapping readers while keeping public fields for source compatibility until a breaking release. Document migration and test normalized keys and original labels. Part of #23. Implemented with assistance from Codex; runtime and benchmark validation pending.

Use catalog schemas and scoped aliases to diagnose incompatible operands before planning. Preserve numeric and temporal coercions and defer unknown types to DataFusion. Add Rust semantic and Python execution coverage. Part of #23. Implemented with assistance from Codex; runtime tests queued and benchmark validation pending.

@stugorf

Reuse unoptimized plans in the engine, binding fresh scalar values before optimization. Bound the cache to 128 templates and retain ordinary planning for containers and value-dependent output names. Include result, schema, refresh and concurrency regressions and scoped timing harnesses. Part of #14. Isolated core Rust suite: 344 passed, 1 optional timing diagnostic ignored. Formatting, focused Ruff and strict docs passed. Python runtime, bridge test linking, benchmark and Clippy evidence remain pending. Submitted by David Hughes (@stugorf), with AI Collaboration.

Resolve literal-or-parameter row counts before logical planning, including chained WITH limits. Reject invalid counts without truncation or wraparound. Part of #20. Implemented with assistance from Codex; coordinated runtime and benchmark checks pending.

@stugorf

SKIP and LIMIT parameters determine graph plan shape, including chained WITH limits. Keep these queries on ordinary planning and document the boundary. Add a regression for every row-count scope. Part of #14, dependent on #20. Submitted by David Hughes (@stugorf), with AI Collaboration.

@stugorf

Release the GIL during template lookup/build, cover repeated parameterized limits, and document exact local stage measurements and benchmark proof. Coordinated Python, benchmark and SF1 validation remain pending. Part of #14. Submitted by David Hughes (@stugorf), with AI Collaboration.

@stugorf

Keep the existing refresh test array downcast in scope after sharing its real snapshot fixture with cache tests. Submitted by David Hughes (@stugorf), with AI Collaboration.

Resolve all RETURN expressions before applying scalar aliases to the ORDER BY binding scope. Preserve whole-node projection behavior. Add focused planner and CSR/join Python regressions for issue #21. Document alias scope in the Cypher guide. Validation: Rust formatting, Python lint, strict docs build, spelling, and whitespace checks pass. Published 0.2.0 reproduction: 14 failures, 4 existing passes. Current-source runtime tests remain pending the coordinated build. Implemented by David Hughes with assistance from Codex.

@stugorf

DataFusion named_struct requires arguments. Use an empty Struct scalar for empty literal and parameter maps. Covered by the container parameter round-trip regression. Part of #20. Submitted by David Hughes (@stugorf), with AI Collaboration.

@stugorf

Specialize size() of unmapped list comprehensions to filtered COUNT(1), preserving null elements, duplicates, padding and input-row multiplicity. Keep mapped and pattern comprehensions on their existing path. Document the non-nullable Arrow count field and add focused plan/result tests plus a reference benchmark. Part of #18. Isolated Rust suite: 349 passed, one optional timing diagnostic ignored. Formatting, focused Ruff and strict docs passed. Python, benchmark and SF1 measurements pending; no performance claim. Submitted by David Hughes (@stugorf), with AI Collaboration.

@stugorf

Implement the bounded issue #20 scalar-function group with strict argument kinds, Unicode string operations, list operations, and stepped ranges. Preserve Cypher null, invalid-length, and rounding behavior. Accept empty quoted string literals and document the supported signatures and limits. Validation: 343 Rust library tests, 176 focused Python tests, workspace all-target clippy, Python lint, strict docs, spelling, and executable docs pass. Full Python run: 581 pass, two prerequisite empty-map parameter failures; the prerequisite correction is being integrated separately. Submitted by David Hughes (@stugorf), with AI Collaboration.

@stugorf

Source and runtime checks confirm literals and arithmetic already use the stable expr column name. Remove the unnecessary fallback and correct its regression expectation and docs. Fresh-versus-cached Arrow schema/result tests pass for unaliased scalar parameters. Part of #14. Submitted by David Hughes (@stugorf), with AI Collaboration.

@stugorf

Use an explicitly unknown function in the rejection regression now that replace() is supported. Submitted by David Hughes (@stugorf), with AI Collaboration.

@stugorf

Address the read-only CALL portion of #20 with explicit imports and per-row streaming Apply execution. Preserve entity aliases, child aggregates and top-k semantics, and keep child variables scoped. Bypass snapshot templates for CALL and yield during long runs of empty children. Validation: workspace Rust tests, focused CALL regressions, 724 Python tests on the rebuilt extension, Clippy, executable docs, strict docs build, Ruff, rustfmt and typos 1.26.0. Submitted by David Hughes (@stugorf), with AI Collaboration.

@stugorf

Parse UNION and UNION ALL as independent branches, validate matching output columns and compatible Arrow types, and preserve distinct versus bag semantics. Include all branches in query metadata and document direct Rust AST migration. Part of #20. Submitted by David Hughes (@stugorf), with AI Collaboration. Full combined validation and SF1 measurements remain pending.

@stugorf

Seed branches from the same available imports while requiring each branch to declare its own WITH. Preserve entity outputs, scope catalog checks, and bypass cached shapes in every nested query. Use DataFusion UNION coercion before capturing child schemas and the final physical result schema when collecting batches. Validation: 10 focused Rust tests and 88 CALL/UNION Python cases pass, including streaming null/numeric coercion and repeated parameter changes. Combined feature validation follows integration. Submitted by David Hughes (@stugorf), with AI Collaboration.

Resolve #11 by resuming neighbor traversal between output batches and storing immutable deletion occurrence offsets. Add focused traversal/snapshot tests, a reusable hub memory scenario, and documentation. Assisted by Codex.

@stugorf

Resolve #13 with bounded typed traversal over CSR snapshots, retaining mixed relationship types without unrolling every length or type sequence. Implement zero-hop identity in indexed and join paths while preserving the 20-hop limit. Submitted by David Hughes (@stugorf), with AI Collaboration.

@stugorf

Resolve #16 by retaining filter columns needed by the projection and reconstructing target IDs in predicates. Fetch only remaining properties for surviving rows, without a persistent property cache. Submitted by David Hughes (@stugorf), with AI Collaboration.

Resolve #17 with complete-layer bidirectional search for single-target CSR queries, preserving all shortest paths and duplicate edges. Keep shared forward search for multiple targets. Developed by David Hughes with assistance from Codex.

@stugorf

Implement keys for maps and exposed entity property columns, preserving null map fields while omitting absent entity properties. Resolve labels from canonical configured node bindings and preserve optional nulls and WITH aliases. Reject mixed-label bindings explicitly. Validation: 344 Rust library tests, 32 focused Python cases, all 615 Python tests, workspace all-target clippy, Python lint, executable documentation, strict docs build, spelling, formatting, and whitespace checks pass. Part of #20; mixed-label labels() remains an explicit engine boundary. Submitted by David Hughes (@stugorf), with AI Collaboration.

@stugorf

Retain actual edge occurrences and typed endpoint records through named paths, collect, and UNWIND. Keep length/node-only shortest queries on CSR and reuse cooperative bidirectional search for rich path values. Part of #20. Submitted by David Hughes (@stugorf), with AI Collaboration.

Part of #20. Keep fold variables local, preserve list order and null semantics, and correlate graph predicates with iterator bindings. Document supported forms and explicit body limits.

Add PEP 561 stubs matching the extension graph module, strict consumer checks, and runtime signature parity tests. Document the typing workflow and check wheel inclusion. Runtime and benchmark validation remain pending the coordinated build.

@stugorf

Add keyword-only timeout to the three Cypher execution entrypoints. Keep one deadline through streamed reads and release the owned stream while idle; preserve native Arrow reader errors and cooperative traversal cancellation. Issue #19. Submitted by David Hughes (@stugorf), with AI Collaboration. External benchmark runner adoption and performance measurements remain pending.

@stugorf

Keep one cooperative scheduling helper, bind and restore lambda-local types, and bypass cached templates for extension-owned path predicates and query-local inventories. Add CALL/UNION/path/fold, refresh isolation, parameter rebinding and deadline regressions. Combined runtime validation follows the inventory partition fix. Submitted by David Hughes (@stugorf), with AI Collaboration.

Preserve inventory source coalescing and refresh output partitioning after optimizer rewrites. Cover Arrow and Lance scans beyond the first source partition and the unmodified IC14 query over real relationship records.

The expression type check rejected comparisons such as p.age = '30', which DataFusion has always run by casting the string side. Let DataFusion's coercion decide scalar compatibility again, so only genuinely incompatible operands (string arithmetic, booleans against numbers, entity arithmetic) fail before planning. Update the tests that asserted the old rejection and document the behaviour.

Variable-length expansion started reading the target label from the logical plan, which defaults an unlabeled node to "Node". With relationships that record no endpoint labels (the deprecated builder), MATCH (a:Person)-[:KNOWS*1..2]->(b) then failed with "No mapping found for node label: Node". Fall back to the source variable's label in that case, as the planner did before, in both the analysis and label collection.

SQL export rejected every typed null outside the Default dialect, so ordinary queries such as coalesce(p.name, null) or CASE ... ELSE null stopped exporting for Spark, PostgreSQL, MySQL and SQLite. Those dialects have no arrow_cast; emit a bare NULL there, as exports did before, and keep the arrow_cast wrapper for the Default dialect.

openCypher allows one union mode per query; the planner folded a mix left to right instead. Report it during semantic analysis, update the tests that expected the folded result and document the rule.

DataFusion copies a predicate pushed past an extension node into every input. PathMatch's extra inputs (node and edge inventories) lack the start node's columns, so MATCH p = (a)-[...]->(b) WHERE a.id = 1 failed to plan with "No field named a__id". Only allow pushing predicates on columns every input carries.

target_partitions is now validated to 1..=1024 in the engine. Say so where the option is documented, and show the Rust CypherEngine::with_target_partitions constructor in the Rust guide.

cloudflare-workers-and-pages · 2026-10-26T17:32:04Z

Deploying lance-csr with Cloudflare Pages

Latest commit:	`bec6758`
Status:	✅ Deploy successful!
Preview URL:	https://cf2d4dab.lance-csr.pages.dev
Branch Preview URL:	https://feat-integrate-cypher-and-en.lance-csr.pages.dev

View logs

with_execution_deadline was hidden plumbing for the Python bridge. Make it a documented API: nested-loop joins inside it check the deadline and fail with "Query timeout exceeded", and cross joins yield so an outer tokio::time::timeout_at can stop them. Add a test, a Timeouts section to the Rust guide with a compiled example, and a link from the Python deadline docs.

stugorf added 30 commits October 25, 2026 09:58

test(graph): restore refresh assertion array import

9f82994

Keep the existing refresh test array downcast in scope after sharing its real snapshot fixture with cache tests. Submitted by David Hughes (@stugorf), with AI Collaboration.

test(cypher): keep unsupported-function coverage after replace support

b118562

Use an explicitly unknown function in the rejection regression now that replace() is supported. Submitted by David Hughes (@stugorf), with AI Collaboration.

fix(csr): stream expansion in bounded chunks

67912be

Resolve #11 by resuming neighbor traversal between output batches and storing immutable deletion occurrence offsets. Add focused traversal/snapshot tests, a reusable hub memory scenario, and documentation. Assisted by Codex.

feat(cypher): add ordered reduce and list predicates

8e1563f

Part of #20. Keep fold variables local, preserve list order and null semantics, and correlate graph predicates with iterator bindings. Document supported forms and explicit body limits.

feat(python): ship public API type stubs

ae1218f

Add PEP 561 stubs matching the extension graph module, strict consumer checks, and runtime signature parity tests. Document the typing workflow and check wheel inclusion. Runtime and benchmark validation remain pending the coordinated build.

fix(cypher): read every rich path source partition

97ce7d6

Preserve inventory source coalescing and refresh output partitioning after optimizer rewrites. Cover Arrow and Lance scans beyond the first source partition and the unmodified IC14 query over real relationship records.

prrao87 added 6 commits October 26, 2026 10:19

fix(cypher): reject queries that mix UNION and UNION ALL

ff2123f

openCypher allows one union mode per query; the planner folded a mix left to right instead. Report it during semantic analysis, update the tests that expected the folded result and document the rule.

docs: document the target_partitions range and Rust constructor

4d0ed86

target_partitions is now validated to 1..=1024 in the engine. Say so where the option is documented, and show the Rust CypherEngine::with_target_partitions constructor in the Rust guide.

prrao87 marked this pull request as ready for review October 26, 2026 17:33

prrao87 merged commit afdf746 into main Oct 26, 2026
8 checks passed

prrao87 deleted the feat/integrate-cypher-and-engine-work branch October 26, 2026 18:01

Navigation Menu

Repository navigation

feat: integrate cypher, csr and engine work for 0.3.0 - #55

feat: integrate cypher, csr and engine work for 0.3.0#55
prrao87 merged 68 commits into
mainfrom
feat/integrate-cypher-and-engine-work

prrao87 commented Oct 26, 2026 •

edited

Loading

Uh oh!

cloudflare-workers-and-pages Bot commented Oct 26, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

prrao87 commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What changed

Known limitations (not changed here)

Validation

Benchmarks

Uh oh!

cloudflare-workers-and-pages Bot commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying lance-csr with Cloudflare Pages

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

prrao87 commented Oct 26, 2026 •

edited

Loading

cloudflare-workers-and-pages Bot commented Oct 26, 2026 •

edited

Loading