Skip to content

perf(gfql): dispatch indexed fixed-hop bindings before the canonical traversal#1776

Open
lmeyerov wants to merge 1 commit into
masterfrom
perf/gfql-general-indexed-bindings
Open

perf(gfql): dispatch indexed fixed-hop bindings before the canonical traversal#1776
lmeyerov wants to merge 1 commit into
masterfrom
perf/gfql-general-indexed-bindings

Conversation

@lmeyerov

Copy link
Copy Markdown
Contributor

What

The resident-index fixed-hop path (rows(binding_ops=...) — the Cypher multi-alias MATCH … RETURN shape) was only consulted inside row materialization, after the engine had already executed the full canonical traversal. It therefore built its compact path bag on top of the graph-sized work it exists to avoid.

Both boundaries now ask the same shared, structural gate first:

  • pandas / cuDF — _handle_boundary_calls (compute/chain.py)
  • native Polars — chain_polars (gfql/lazy/engine/polars/chain.py)

Only when the helper can serve the whole middle exactly do they skip the canonical traversal and hand the compact state to the unchanged row materializer via an internal graph copy. A decline is memoized so the rows path does not re-attempt (or re-record) the same decision.

Engagement stays operator / index / dtype / cost based — no query, schema, or hop-count recognition. Seeded, prefiltered, policy-bearing, shortest-path, unsupported, and cost-gated shapes still fall back to canonical execution with identical results.

Also included:

  • Indexed seed lookup covers {id, …extra scalar constraints} — gather the one indexed row, then filter it, instead of scanning the whole node table. Missing ids and non-matching constraints return the same empty result; duplicate-id graphs cannot build the index and are unaffected.
  • Seeded typed-hop property projection keeps its fast path through trailing DISTINCT / ORDER BY / SKIP / LIMIT by delegating those plain frame ops to the canonical chain.
  • Fix (pre-existing, cuDF): that projection applied the pandas rows-pivot artifact (int → float64, bool → object) on every engine, but cuDF's canonical pivot preserves source dtypes — so the fast path returned float64/object where cuDF's own canonical path returns int64/bool. The cast rule is now engine-aware; the dtype-class decline guard is unchanged.
  • Empty edge-frame alias marker columns now follow each engine's own canonical column order (pandas first, polars appended).

Not in this PR

Engine.py AUTO source-native routing is unchanged — the deferred #1743 policy is deliberately out of scope, and the two tests that pinned it were dropped.

Tests

Standard-derived (LDBC-shaped; benchmark names appear only in test ids/comments), all comparing against the same-engine canonical path:

  • exact value / order / dtype / index / schema parity, one trace decision per serve/decline, missing / stale / off index lifecycle, named-edge schema, policy hooks, shortest-path exclusion, exception propagation;
  • no-canonical-traversal proofs for both pandas (ASTEdge.execute forbidden) and polars (_chain_traversal_polars forbidden);
  • an unnamed-middle regression pinning the bypass gate — reverting the gate fails it on both engines;
  • mixed int / float / bool / string / id projection dtype parity across pandas / polars / cuDF;
  • test_out_of_shape_declines_with_parity reconciled: DISTINCT and ORDER BY … LIMIT moved to a new engages-with-parity case, since they are now served with proven parity.

Validation (dgx-spark, capped guard, read-only source)

  • full CPU graphistry/tests/compute: 6232 passed, 1 failed — master's pre-existing test_chain_dask_edges
  • cuDF lane (indexed + seeded fast-path files): 30 passed
  • ruff: clean · mypy: 0 new errors vs master
  • standard LDBC SNB SF1 interactive-short profile: the redundant two-hop typed-mask and 16-merge buckets disappear; profiled call 1263 ms → 106 ms (~11.9×), exact 19-row oracle preserved (diagnostic measurement, not a published comparison number)

🤖 Generated with Claude Code

https://claude.ai/code/session_01NxxTFpKiqA4FQa9Pjy8yR1

…traversal

The resident-index fixed-hop path (`rows(binding_ops=...)`, the Cypher
multi-alias MATCH...RETURN shape) was only consulted inside row
materialization — after the engine had already executed the full canonical
traversal — so its compact path bag was built on top of the graph-sized work
it exists to avoid.

Both boundaries now ask the same shared structural gate first:

- pandas/cuDF: `_handle_boundary_calls` in compute/chain.py
- native Polars: `chain_polars` in gfql/lazy/engine/polars/chain.py

Only when the helper can serve the WHOLE middle exactly do they skip the
canonical traversal and hand the compact state to the unchanged row
materializer through an internal graph copy; a decline is memoized so the
rows path does not re-attempt (or re-record) the same decision. Engagement
remains operator/index/dtype/cost based — no query, schema, or hop-count
recognition — and seeded, prefiltered, policy-bearing, shortest-path,
unsupported, and cost-gated shapes still fall back with identical results.

Also in this change:

- indexed seed lookup now covers a unique node id PLUS extra scalar
  constraints (gather the one indexed row, then filter it) instead of
  scanning the whole node table;
- the seeded typed-hop property projection keeps its fast path through
  trailing DISTINCT / ORDER BY / SKIP / LIMIT by delegating those plain frame
  ops to the canonical chain;
- fix a pre-existing cuDF dtype divergence in that projection: the pandas
  rows-pivot upcast (int -> float64, bool -> object) was applied on every
  engine, but cuDF's canonical pivot preserves source dtypes;
- keep the empty edge frame's alias marker columns in each engine's own
  canonical column order (pandas puts them first, polars appends them).

Tests are standard-derived (LDBC-shaped, names only in test ids): exact
value/order/dtype/index/schema parity vs the same-engine canonical path,
one trace decision per serve/decline, no-canonical-traversal proofs for both
pandas and polars, an unnamed-middle regression that pins the bypass gate,
and a mixed-dtype projection parity case across pandas/polars/cuDF.

Validated on dgx-spark under the capped guard: full CPU
`graphistry/tests/compute` 6232 passed with only master's pre-existing dask
failure, cuDF lane 30 passed, ruff clean, mypy no new errors vs master. On a
standard LDBC SNB SF1 interactive-short profile the redundant two-hop
typed-mask and 16-merge buckets disappear and the profiled call drops ~11.9x
(1263 ms -> 106 ms) with the exact 19-row oracle preserved.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NxxTFpKiqA4FQa9Pjy8yR1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant