Skip to content

fix(planner): plan optional match pattern predicates that read earlier variables - #107

Merged
prrao87 merged 2 commits into
mainfrom
fix/optional-match-pattern-variables
Nov 2, 2026
Merged

prrao87 merged 2 commits into
mainfrom
fix/optional-match-pattern-variables

Conversation

@prrao87

@prrao87 prrao87 commented Nov 1, 2026

Copy link
Copy Markdown
Member

Why

A pattern predicate in an OPTIONAL MATCH ... WHERE that reads a variable bound by an earlier clause fails to plan. This is how Ladybug's runner phrases LDBC IC10, and it surfaced while profiling that query:

MATCH (person:Person)-[:KNOWS]->(friend:Person)
WITH DISTINCT friend, person
OPTIONAL MATCH (friend)<-[:CREATED]-(post:Post)
WHERE (post)-[:HAS_TAG]->(:Tag)<-[:INTEREST]-(person)
RETURN person.id, friend.id, count(post)

On main this returns PlanError: Failed to build projection: Schema error: No field named person. The same profile also showed physical planning spending time formatting whole input plans, which this PR removes while it is in the same area.

What changed

  • OPTIONAL MATCH pattern predicates can read earlier variables. The optional side is planned from a seed that carries the earlier variables its WHERE reads. Those were collected by a helper that deliberately skips pattern predicates (it was written for selectivity scoring), so person above was never carried. The planner now also collects the variables named in pattern predicates, keeping only those bound by earlier clauses, so variables local to the predicate's pattern are never carried. A pattern predicate nested inside a value (CASE WHEN (a)-->(b) ...) is still not visited; that is noted in the code.
  • Extension nodes format without their input plans. DataFusion's physical planner formats every extension node with Debug while planning it. The derived Debug of CsrExpandNode, NodeLookupNode, ShortestPathNode and ApplyNode printed the entire input plan, which made this quadratic in plan depth. They now print their EXPLAIN line, as PathMatchNode already did. Nothing relied on the old output.

User impact: queries of the shape above now run instead of failing. No API changes and no documentation changes; the docs already list WHERE inside OPTIONAL MATCH as supported.

Validation

  • New regression test optional_match_where_pattern_reads_an_earlier_variable, with and without the CSR. It fails on main with the error above and passes here. It covers two people with different interests (so each row must use its own person), a post matching through two tags (counted once), and a friend with no posts (count 0).

  • cargo test --workspace --locked, make test (1665 passed), cargo clippy --workspace --all-targets -- -D warnings, cargo fmt --all --check and uvx typos@1.26.0: all pass.

  • Expected answers in both benchmark suites, with a release wheel of this branch: graph-benchmark-ldbc 30/30 and graph-benchmark-ldbc-complex 14/14, in both memory and lazy mode.

  • Planning time, release build, SF1 in memory, mean of 50 plan-only calls, two alternating runs of main and this branch. Apple M5, 24 GiB, macOS 26.6.2:

    Query main (ms) branch (ms) Change
    IC1 18.3 / 18.5 17.3 / 17.4 -5.5%
    IC10 38.1 / 38.1 36.9 / 36.9 -3.2%
    IC14 26.6 / 26.9 25.6 / 25.8 -3.7%

    End-to-end execute times for these three queries moved by less than run-to-run noise.

Not yet run: the full main-versus-branch benchmark harness from CONTRIBUTING.md (both suites, both modes, alternating runs). This stays a draft until those tables are added.

🤖 Generated with Claude Code

…rn predicate

OPTIONAL MATCH (friend)<-[:CREATED]-(post) WHERE (post)-->()<--(person) failed
to plan with 'No field named person' when person was bound by an earlier
clause. The optional side's seed carries the earlier variables its WHERE
reads, but they were collected with boolean_variables, which skips pattern
predicates. Also collect the variables named in pattern predicates, keeping
only those bound earlier.
DataFusion's physical planner formats every extension node with Debug while
planning it. The derived Debug of CsrExpandNode, NodeLookupNode,
ShortestPathNode and ApplyNode printed the whole input plan, which made this
quadratic in the plan's depth. Print the node's explain line instead, as
PathMatchNode already does.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Deploying lance-csr with  Cloudflare Pages  Cloudflare Pages

Latest commit: e5d026f
Status: ✅  Deploy successful!
Preview URL: https://5653a8cf.lance-csr.pages.dev
Branch Preview URL: https://fix-optional-match-pattern-v.lance-csr.pages.dev

View logs

@prrao87
prrao87 marked this pull request as ready for review November 1, 2026 23:36
@prrao87
prrao87 merged commit d3c0192 into main Nov 2, 2026
7 checks passed
@prrao87
prrao87 deleted the fix/optional-match-pattern-variables branch November 2, 2026 00:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant