[prrao87]

Why

The CSR's fused target lookup fetched the target's columns with take_rows, one chunk of expanded rows at a time. When a hop reaches a large share of a Lance node table, each chunk of scattered rows touches most of the column's pages, so the hop reads the column about once per chunk, while a hash join reads it once. Since named relationship hops moved onto the CSR, this made IC5 about 69% slower in lazy mode, because its hasMember hop reads forum titles across about 800k expanded edges.

This PR closes #86.

What changed

• Read target columns once for broad hops. An expansion counts the target rows it fetches, shared across the query's partitions. Once the count passes 1/8 of the node table and one batch, it reads the columns the hop needs (output columns plus target-filter columns) in one sequential scan, then gathers from memory for the rest of the query.
• Small hops keep per-chunk fetches. The one-batch floor matters for small tables: without it, a hop into Place (1,460 rows) switched after a few hundred rows, and the scan's setup cost more than the fetch it replaced.
• Row ids are mapped, not used as offsets. The scan keeps each row's Lance row id, so tables split across files, deleted rows and stable row ids stay exact. A row missing from the scan is an error rather than a wrong answer.
• In-memory tables don't change. They already gather without I/O.
• Known limit: the scanned columns stay in memory until the query ends, with no memory budget, since the engine doesn't configure a DataFusion memory pool. A shortcut: comment marks where a reservation would go.

Validation

• Rust: 671 tests pass, plus clippy with -D warnings and cargo fmt --check. A new test checks that the scanned columns match fetches by row id, with and without stable row ids, over three fragments with a deleted row, and that an unknown row id fails.
• Python: 1,647 tests pass, plus ruff and pyright. A new test crosses the switch (about 16k target fetches against a threshold of 8,193) on a fragmented Lance table with deleted rows and null names, and compares CSR and join answers for output columns, a target filter and count(DISTINCT ...).
• Docs: no user-facing behavior changed; strict Zensical build and typos pass.

Benchmarks. Release wheels, run with the harness in CONTRIBUTING.md, main and branch alternating twice. Every run passed against the expected answers. Apple M5, 24 GB, macOS 26.6.2, on AC power. Commits: main 0b77246; graph-benchmark-ldbc 5f528da; graph-benchmark-ldbc-complex f3e63fd.

Suite, mode	Branch vs main (geomean)	Median gap between main runs
IC1 to IC14, lazy	-15.6%	0.8%
Custom 30, lazy	+0.5%	1.2%
IC1 to IC14, memory	-0.3%	1.1%
Custom 30, memory	+0.9%	2.4%

Queries that changed beyond the gap between main runs, lazy mode:

Query	main	Branch	Change
IC9	812.6 ms	310.3 ms	-62%
IC3	432.7 ms	165.5 ms	-62%
IC5	175.7 ms	111.1 ms	-37% (0.4.0: 104.0 ms)
IC12	25.7 ms	20.3 ms	-21%
Custom Q28	2.31 ms	2.06 ms	-11%
Custom Q12	7.10 ms	6.50 ms	-8%
IC10	128.5 ms	141.0 ms	+10%, noise: see below

IC10's +10% comes from one branch run taken at load average 8 (mean 153 ms, fastest round 132 ms). The other three branch runs measured 128.7 to 131.7 ms, the same as main. Every other query is within 7% of main.

The memory-mode runs used a build without the one-batch floor. In-memory tables never scan, so the floor can't change them. Without the floor, custom Q17, Q18 and Q27 were 13 to 15% slower in lazy mode; with it, they match main.

Still open after this PR: IC5 in memory mode remains about 20% slower than in 0.4.0 (95.7 ms against 79.8 ms). This change doesn't affect memory mode, so that slowdown ships in 0.5.0 as a known trade-off and is tracked in #92.

🤖 Generated with Claude Code