perf(gfql): dispatch indexed fixed-hop bindings before the canonical traversal#1776
Open
lmeyerov wants to merge 1 commit into
Open
perf(gfql): dispatch indexed fixed-hop bindings before the canonical traversal#1776lmeyerov wants to merge 1 commit into
lmeyerov wants to merge 1 commit into
Conversation
…traversal The resident-index fixed-hop path (`rows(binding_ops=...)`, the Cypher multi-alias MATCH...RETURN shape) was only consulted inside row materialization — after the engine had already executed the full canonical traversal — so its compact path bag was built on top of the graph-sized work it exists to avoid. Both boundaries now ask the same shared structural gate first: - pandas/cuDF: `_handle_boundary_calls` in compute/chain.py - native Polars: `chain_polars` in gfql/lazy/engine/polars/chain.py Only when the helper can serve the WHOLE middle exactly do they skip the canonical traversal and hand the compact state to the unchanged row materializer through an internal graph copy; a decline is memoized so the rows path does not re-attempt (or re-record) the same decision. Engagement remains operator/index/dtype/cost based — no query, schema, or hop-count recognition — and seeded, prefiltered, policy-bearing, shortest-path, unsupported, and cost-gated shapes still fall back with identical results. Also in this change: - indexed seed lookup now covers a unique node id PLUS extra scalar constraints (gather the one indexed row, then filter it) instead of scanning the whole node table; - the seeded typed-hop property projection keeps its fast path through trailing DISTINCT / ORDER BY / SKIP / LIMIT by delegating those plain frame ops to the canonical chain; - fix a pre-existing cuDF dtype divergence in that projection: the pandas rows-pivot upcast (int -> float64, bool -> object) was applied on every engine, but cuDF's canonical pivot preserves source dtypes; - keep the empty edge frame's alias marker columns in each engine's own canonical column order (pandas puts them first, polars appends them). Tests are standard-derived (LDBC-shaped, names only in test ids): exact value/order/dtype/index/schema parity vs the same-engine canonical path, one trace decision per serve/decline, no-canonical-traversal proofs for both pandas and polars, an unnamed-middle regression that pins the bypass gate, and a mixed-dtype projection parity case across pandas/polars/cuDF. Validated on dgx-spark under the capped guard: full CPU `graphistry/tests/compute` 6232 passed with only master's pre-existing dask failure, cuDF lane 30 passed, ruff clean, mypy no new errors vs master. On a standard LDBC SNB SF1 interactive-short profile the redundant two-hop typed-mask and 16-merge buckets disappear and the profiled call drops ~11.9x (1263 ms -> 106 ms) with the exact 19-row oracle preserved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NxxTFpKiqA4FQa9Pjy8yR1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The resident-index fixed-hop path (
rows(binding_ops=...)— the Cypher multi-aliasMATCH … RETURNshape) was only consulted inside row materialization, after the engine had already executed the full canonical traversal. It therefore built its compact path bag on top of the graph-sized work it exists to avoid.Both boundaries now ask the same shared, structural gate first:
_handle_boundary_calls(compute/chain.py)chain_polars(gfql/lazy/engine/polars/chain.py)Only when the helper can serve the whole middle exactly do they skip the canonical traversal and hand the compact state to the unchanged row materializer via an internal graph copy. A decline is memoized so the rows path does not re-attempt (or re-record) the same decision.
Engagement stays operator / index / dtype / cost based — no query, schema, or hop-count recognition. Seeded, prefiltered, policy-bearing, shortest-path, unsupported, and cost-gated shapes still fall back to canonical execution with identical results.
Also included:
{id, …extra scalar constraints}— gather the one indexed row, then filter it, instead of scanning the whole node table. Missing ids and non-matching constraints return the same empty result; duplicate-id graphs cannot build the index and are unaffected.DISTINCT/ORDER BY/SKIP/LIMITby delegating those plain frame ops to the canonical chain.float64, bool →object) on every engine, but cuDF's canonical pivot preserves source dtypes — so the fast path returnedfloat64/objectwhere cuDF's own canonical path returnsint64/bool. The cast rule is now engine-aware; the dtype-class decline guard is unchanged.Not in this PR
Engine.pyAUTO source-native routing is unchanged — the deferred#1743policy is deliberately out of scope, and the two tests that pinned it were dropped.Tests
Standard-derived (LDBC-shaped; benchmark names appear only in test ids/comments), all comparing against the same-engine canonical path:
offindex lifecycle, named-edge schema, policy hooks, shortest-path exclusion, exception propagation;ASTEdge.executeforbidden) and polars (_chain_traversal_polarsforbidden);test_out_of_shape_declines_with_parityreconciled:DISTINCTandORDER BY … LIMITmoved to a new engages-with-parity case, since they are now served with proven parity.Validation (dgx-spark, capped guard, read-only source)
graphistry/tests/compute: 6232 passed, 1 failed — master's pre-existingtest_chain_dask_edgesruff: clean ·mypy: 0 new errors vs master🤖 Generated with Claude Code
https://claude.ai/code/session_01NxxTFpKiqA4FQa9Pjy8yR1