Right-size DataFusion partitions for selective graph queries #15

Problem

The engine's DataFusion session uses default settings (build_engine_state()), so plans repartition data to the number of CPU cores even when the intermediate results are a few rows. Graph queries are mostly selective (start from one person, expand a few hops), so repartitioning and merging adds overhead without parallel work to share.

Evidence

RepartitionExec is one of the most common operators in the IC plans on SF1: 49 in IC14, 34 in IC10, 26 in IC1 and 23 in IC3, on an Apple M5 with 10 cores. These are the queries where lance-csr is slowest relative to the other engines (IC10 is 7.7x slower than Ladybug).

Suggested fix

Measure first: run the IC and 30-query workloads with datafusion.execution.target_partitions set to 1, 2 and the default, and with enable_round_robin_repartition off. Then pick a rule, for example a low default for the Cypher engine, or using statistics from KeyStatisticsExec to keep partitions at 1 below a row-count threshold. A user-facing setting may be warranted for analytical queries that do scan large tables.

Acceptance criteria

• A table of per-query latency for each setting, for both workloads, in the pull request.
• The chosen default improves the geometric mean on both workloads without making any query more than 10% slower, or the trade-off is stated.
• If a setting is exposed, docs/api-reference.md documents it.

Identified by profiling the LDBC IC benchmark; no setting has been measured yet.