fix(gfql): index AUTO preserves resident polars frames (kills the polars-hop O(E) tax)#1767
Open
lmeyerov wants to merge 5 commits into
Open
fix(gfql): index AUTO preserves resident polars frames (kills the polars-hop O(E) tax)#1767lmeyerov wants to merge 5 commits into
lmeyerov wants to merge 5 commits into
Conversation
…E) tax) resolve_engine(AUTO) maps polars frames to PANDAS, so create_index coerced-and-replaced the user's polars frames with pandas copies; every later hop(engine='polars') re-converted the full frames per call (O(E), ~220ms at 4M edges) and the pandas-engine index could never fingerprint- match. Resolve AUTO over resident polars frames to POLARS in the index API only (explicit engines unchanged; NOT the global auto-policy change). dgx (warm median 30, parity vs pandas indexed oracle): polars seeded 1-hop 221 -> 0.80ms at 4M (276x); ladder 0.17/0.35/0.80/0.83ms at 0.25M/1M/4M/16M edges vs previously linear 16.6 -> 1751ms. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
…rialize-first Review findings (plans/review-pr-1767/review.md): B1 edges-only polars graph crashed in the NODE_ID build (materialize under AUTO synthesizes pandas; polars gathers ran on a pandas frame) — gfql_index_all now materializes + engine-aligns BEFORE any build so all indexes land valid, and create_index re-aligns for direct calls. M1 LazyFrame frames under AUTO crashed — the gate now requires EAGER polars DataFrames on ALL present frames (LazyFrame/mixed keep the legacy pandas path). Spy test asserts the index SERVED (non-None), not merely was called. AUTO-query trade-off documented in CHANGELOG (deferred to #1743 by policy hold). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
lmeyerov
added a commit
that referenced
this pull request
Jul 22, 2026
…1755) With gfql_index_all() resident, the seeded helpers replace their three full-frame scans (O(N) seed row, O(E) frontier isin, O(N) endpoint gather) with node-id searchsorted + CSR adjacency gathers. Decline-gated to the identical scan body: absent/stale index (get_valid fingerprint + identity), non-numeric id families, seed not id-filtered. Local 50k/200k covered shape: pandas 3.11->1.85ms, polars 1.74->1.10ms; win scales with graph size. Pinned: parity+engagement (pandas/polars), string-id decline, stale-index decline. Stacked on perf/gfql-index-preserve-polars-frames (#1767) — polars index residency requires it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
lmeyerov
added a commit
that referenced
this pull request
Jul 22, 2026
…1755) With gfql_index_all() resident, the seeded helpers replace their three full-frame scans (O(N) seed row, O(E) frontier isin, O(N) endpoint gather) with node-id searchsorted + CSR adjacency gathers. Decline-gated to the identical scan body: absent/stale index (get_valid fingerprint + identity), non-numeric id families, seed not id-filtered. Local 50k/200k covered shape: pandas 3.11->1.85ms, polars 1.74->1.10ms; win scales with graph size. Pinned: parity+engagement (pandas/polars), string-id decline, stale-index decline. Stacked on perf/gfql-index-preserve-polars-frames (#1767) — polars index residency requires it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
This was referenced Jul 22, 2026
lmeyerov
added a commit
that referenced
this pull request
Jul 22, 2026
Refresh the GFQL performance and engine-selection docs to lead with numbers measured on the 0.58.0 release tag (DGX Spark GB10, warm medians N=30, results verified identical across engines): - Seeded typed-hop Cypher fast path, all four engines (pandas 29.9->2.46ms 12.1x, polars 13.8->2.28ms 6.1x, cuDF 30.1->4.89ms 6.1x, polars-gpu 25.2->2.49ms 10.1x; native chain pandas 21.1->1.65ms 12.8x, cuDF 23.2->3.84ms 6.0x) - Resident-index covered-shape lookups (pandas 1.74 / polars 1.59 / polars-gpu 1.91 / cudf 5.78 ms) with the 0.58.0 caveat that Polars frames need gfql_index_all(engine='polars') explicitly (fix in PR #1767) - Flat seeded-hop scaling on pandas: 0.159-0.164ms from 0.25M to 32M edges (pandas-only; polars hop not yet index-routed) - One-keyword engine='polars' LDBC SNB SF1 seed-lookup: 1,299.6->106.1ms (12.3x, same build) — replaces the stale "up to ~38x" headline in engines/performance/quick/about/overview/index - LDBC SNB interactive SF1 vs Neo4j 5.26, same box, warm: GFQL wins 4 of 5 clean pairs; Neo4j wins recent-replies (reported as-is) - OLAP multi-join vs embedded Kuzu (100k scale, engine='polars'): q8 5.0ms vs 1,004ms (200x), q9 14.2x — with the honest inverse (Kuzu wins single-table aggregates 2-4x, seeded property-projection lookups 2.4-64x) Prior Orkut/LiveJournal bulk-sweep tables are retained but relabeled as prior-release measurements; methodology disclosures kept and extended with the 0.58.0 tag-sweep provenance. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
pull Bot
pushed a commit
to admariner/pygraphistry
that referenced
this pull request
Jul 22, 2026
New docs/source/gfql/indexing.rst: lifecycle guide to GFQL resident indexes (gfql_index_all / gfql_index_edges / create_index / show_indexes / drop_index) — node-id + CSR in/out adjacency sidecars, what engages them on 0.58.0 (seeded typed-hop fast paths incl. property RETURNs and property-seeded lookups per graphistry#1768/graphistry#1770, direct g.hop(); general polars chain traversal honestly noted as not yet covered), the identity + fingerprint staleness contract with safe declines, engine notes (polars needs gfql_index_all(engine='polars') until graphistry#1767), 0.58.0-tag measured numbers, and a runnable end-to-end example (executed). Wired into the GFQL toctree + recommended paths next to index_adjacency; CHANGELOG entry under Development > Documentation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
lmeyerov
commented
Jul 22, 2026
| return eng | ||
|
|
||
|
|
||
| def _is_resident_index_valid( |
Contributor
Author
There was a problem hiding this comment.
can we split out autoindexing from enabling polars by default?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem (the C3 "polars hop is linear" mystery — root-caused)
g.gfql_index_all()(engine=auto) on a polars graph silently replaced the polars frames with pandas copies:resolve_engine(AUTO)maps polars input frames toEngine.PANDAS(legacy input-format policy), andcreate_indexcoerces the graph into the resolved engine before building.Consequences for every later
hop(engine='polars')call:_coerce_input_formatsre-converts the FULL nodes+edges frames pandas→polars per call — O(E), ~208ms of a 223ms hop at 4M edges (cProfile:pl.from_pandasnew_stron the 4M-row string column + pandas_isna_string_dtype), ~1.7s at 32M.get_validrequires engine match AND frame identity, and the per-call conversion breaks both — so the hop was also a scan.This is why the C3 ladder showed pandas flat at 0.16ms while polars scaled linearly 16.6→1751ms.
Fix
_resolve_index_engine: when the caller said AUTO and the resident frames are polars, the index API resolves POLARS and indexes the frames in place (the index layer is already engine-polymorphic: numpy sidecar arrays + polars row-gather). Explicit engines are honored unchanged. Deliberately scoped to the index API — this is NOT the globalengine=autorouting-policy change (#1743, on hold).Measured (dgx GB10, container polars 1.35.2, warm median of 30, parity vs pandas indexed oracle at every rung)
Frames verified to remain polars after
gfql_index_all(); parity OK at every rung.Tests
test_index.py::TestIndexAutoPreservesPolarsFrames(file already inPOLARS_TEST_FILES):hop(engine='polars'), parity vs pandas indexed oracleengine='pandas'still coerces (contract unchanged)Full
bin/test-polars.shlocally: failures identical to master baseline (pre-existing local polars-version drift in the cypher conformance matrix; zero new failures). Full index suite (non-GPU) 85/85.🤖 Generated with Claude Code
https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL