You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Read target columns once when a CSR hop fetches a large share of a node table #86
The CSR's fused target lookup fetches the target node's columns with take_rows, one batch of expanded rows at a time. In lazy mode, a batch of rows scattered across a large node table touches most of each column's pages, so a hop that expands a large share of a relationship pays roughly one column read per batch. A hash join reads the node table once.
Measured on GDC SF1 (release build, lazy mode, Apple M5) with IC5's hasMember hop from the 3,016 friends and friends-of-friends of person 1131: 824,130 expanded edges reaching 58k distinct forums, returning count(DISTINCT forum.title):
Plan
Time
Unnamed hop on the CSR, with fused target lookup (main)
45 ms
Same hop as hash joins
20 ms
IC5 avoided this on main only because it names its relationship variable (membership), which forced the join path. Once #73 puts named hops on the CSR, IC5 pays it: about 190 ms in lazy mode against 100 ms on main. Memory mode is affected far less, since an Arrow take is an in-memory gather.
Idea
Apply the switch that #87 (the pull request for #73) uses for relationship-column filters to target columns. Count the target rows a CSR expansion fetches; once they pass a share of the node table, read the needed columns once, sequentially, into memory, and gather from there for the rest of the query. Small hops keep their per-batch fetches.
To watch
Memory: the projected node columns stay resident for the query's duration.
The switch only matters for Lance sources; in-memory tables already gather cheaply.
Measure both suites in both modes against main. IC5 is the query to watch, along with any large unnamed hops that read target properties.
Activity