diff --git a/README.md b/README.md index 21d1e56..418c734 100644 --- a/README.md +++ b/README.md @@ -108,7 +108,7 @@ each of the 30 queries run in the benchmark. **These are not LDBC Benchmark Results.** These independent, unaudited measurements apply to the configurations below. -Neo4j, Kuzu, Ladybug and lance-graph were measured back to back on **2026-10-24** with `./run_benchmarks.sh`. lance-csr 0.3.1 was re-run on its own with `./run_benchmarks.sh lance_csr` on **2026-10-27** ([`results/environment.yml`](results/environment.yml)). DuckPGQ and FalkorDB results are from the earlier run on **2026-10-16**. All runs used the same machine: +Neo4j, Kuzu, Ladybug and lance-graph were measured back to back on **2026-10-24** with `./run_benchmarks.sh`. lance-csr 0.5.1 was run on its own with `./run_benchmarks.sh lance_csr` on **2026-10-31** ([`results/environment.yml`](results/environment.yml)). DuckPGQ and FalkorDB results are from the earlier run on **2026-10-16**. All runs used the same machine: | | | | --- | --- | @@ -125,7 +125,7 @@ Engine versions: - **Kuzu 0.11.3** (unchanged) - **Ladybug 0.21.2** (previously 0.21.1). All 30 queries use plain `conn.execute(query)`; `ANALYZE`, result cleanup and the six ART secondary indexes remain enabled - **lance-graph 0.5.4** (unchanged) -- **lance-csr 0.3.1** (previously 0.2.0), a fork of lance-graph with a CSR adjacency index, running lance-graph's 30 queries over the same Lance datasets in two modes: tables in memory, and Lance datasets read lazily from disk. See the [lance-csr README](lance_csr/README.md) +- **lance-csr 0.5.1** (previously 0.3.1), a fork of lance-graph with a CSR adjacency index, running lance-graph's 30 queries over the same Lance datasets in two modes: tables in memory, and Lance datasets read lazily from disk. See the [lance-csr README](lance_csr/README.md) - **DuckPGQ** on DuckDB 1.5.6 (extension build `61f5d082`), new in this run, with the six ART indexes that mirror Ladybug's - **FalkorDB 6.0.2**, new in this run, running Neo4j's 30 Cypher queries unmodified, with ID indexes plus seven secondary range indexes similar to Ladybug's ART indexes @@ -158,38 +158,38 @@ Every Neo4j, Kuzu, Ladybug, DuckPGQ and FalkorDB query has a 10-second timeout; Times are in milliseconds. The ratio in parentheses is Neo4j mean / engine mean, so a value above 1 means the engine was faster than this Neo4j setup. -| Query | neo4j-2026.09.0 (ms) | kuzu-0.11.3 (ms) | ladybug-0.21.2 (ms) | lance-graph-0.5.4 (ms) | duckpgq-1.5.6 (ms) | falkordb-6.0.2 (ms) | lance-csr-0.3.1-lazy (ms) | lance-csr-0.3.1-memory (ms) | +| Query | neo4j-2026.09.0 (ms) | kuzu-0.11.3 (ms) | ladybug-0.21.2 (ms) | lance-graph-0.5.4 (ms) | duckpgq-1.5.6 (ms) | falkordb-6.0.2 (ms) | lance-csr-0.5.1-lazy (ms) | lance-csr-0.5.1-memory (ms) | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| q1 | 1.277 | 1.734 (0.74x) | 1.218 (1.05x) | 1.297 (0.98x) | 0.978 (1.31x) | 0.440 (2.90x) | 1.128 (1.13x) | 0.656 (1.95x) | -| q2 | 3.260 | 1.322 (2.47x) | 1.123 (2.90x) | 2.337 (1.39x) | 3.450 (0.94x) | 4.682 (0.70x) | 1.331 (2.45x) | 0.512 (6.37x) | -| q3 | 0.665 | 1.034 (0.64x) | 0.789 (0.84x) | 1.862 (0.36x) | 1.120 (0.59x) | 4.420 (0.15x) | 0.773 (0.86x) | 0.616 (1.08x) | -| q4 | 2.656 | 0.848 (3.13x) | 0.393 (6.75x) | 2.982 (0.89x) | 1.545 (1.72x) | 0.215 (12.35x) | 0.751 (3.54x) | 0.481 (5.52x) | -| q5 | 2.785 | 3.440 (0.81x) | 4.004 (0.70x) | 2.044 (1.36x) | 2.014 (1.38x) | 1.211 (2.30x) | 2.229 (1.25x) | 0.771 (3.61x) | -| q6 | 2.597 | 0.664 (3.91x) | 0.787 (3.30x) | 0.727 (3.57x) | 0.554 (4.69x) | 0.193 (13.44x) | 0.710 (3.66x) | 0.413 (6.29x) | -| q7 | 0.430 | 27.735 (0.02x) | 25.112 (0.02x) | 15.677 (0.03x) | 2.666 (0.16x) | 0.649 (0.66x) | 1.053 (0.41x) | 0.982 (0.44x) | -| q8 | 7.287 | 2.723 (2.68x) | 3.375 (2.16x) | 1.141 (6.39x) | 1.825 (3.99x) | 1.801 (4.05x) | 1.753 (4.16x) | 0.613 (11.90x) | -| q9 | 0.758 | 1.728 (0.44x) | 1.342 (0.56x) | 1.888 (0.40x) | 1.158 (0.65x) | 0.235 (3.23x) | 1.319 (0.57x) | 0.809 (0.94x) | -| q10 | 3.788 | 1.553 (2.44x) | 1.778 (2.13x) | 26.693 (0.14x) | 2.728 (1.39x) | 0.201 (18.84x) | 0.908 (4.17x) | 0.700 (5.41x) | -| q11 | 10.282 | 7.960 (1.29x) | 4.635 (2.22x) | 3.084 (3.33x) | 2.690 (3.82x) | 6.742 (1.53x) | 1.893 (5.43x) | 1.466 (7.01x) | -| q12 | 2.549 | 17.156 (0.15x) | 7.840 (0.33x) | 18.684 (0.14x) | 4.340 (0.59x) | 1.125 (2.27x) | 6.801 (0.37x) | 0.693 (3.68x) | -| q13 | 6.654 | 42.661 (0.16x) | 5.837 (1.14x) | 9.616 (0.69x) | 3.382 (1.97x) | 2.522 (2.64x) | 0.995 (6.69x) | 0.736 (9.04x) | -| q14 | 0.460 | 1.556 (0.30x) | 0.902 (0.51x) | 2.909 (0.16x) | 1.439 (0.32x) | 0.204 (2.26x) | 0.654 (0.70x) | 0.477 (0.96x) | -| q15 | 2.369 | 2.379 (1.00x) | 1.164 (2.04x) | 3.023 (0.78x) | 1.295 (1.83x) | 0.226 (10.49x) | 0.703 (3.37x) | 0.505 (4.69x) | -| q16 | 1.260 | 1.775 (0.71x) | 1.036 (1.22x) | 4.329 (0.29x) | 3.400 (0.37x) | 4.415 (0.29x) | 1.056 (1.19x) | 0.624 (2.02x) | -| q17 | 3.086 | 2.616 (1.18x) | 1.738 (1.78x) | 2.301 (1.34x) | 1.365 (2.26x) | 1.199 (2.57x) | 1.369 (2.25x) | 0.869 (3.55x) | -| q18 | 3.108 | 1.508 (2.06x) | 1.221 (2.55x) | 1.763 (1.76x) | 1.047 (2.97x) | 0.628 (4.95x) | 1.074 (2.89x) | 0.631 (4.93x) | -| q19 | 4.856 | 11.657 (0.42x) | 3.433 (1.41x) | 19.927 (0.24x) | 4.327 (1.12x) | 2.129 (2.28x) | 1.332 (3.65x) | 0.900 (5.40x) | -| q20 | 421.921 | 13.247 (31.85x) | 14.154 (29.81x) | 3.132 (134.71x) | 0.956 (441.15x) | 0.455 (926.79x) | 16.448 (25.65x) | 2.047 (206.08x) | -| q21 | 0.511 | 0.441 (1.16x) | 0.500 (1.02x) | 1.459 (0.35x) | 0.662 (0.77x) | 0.346 (1.48x) | 0.788 (0.65x) | 0.700 (0.73x) | -| q22 | 2.657 | 23.338 (0.11x) | 7.456 (0.36x) | 13.413 (0.20x) | 5.143 (0.52x) | 0.670 (3.97x) | 0.884 (3.01x) | 0.647 (4.11x) | -| q23 | 3.133 | 1.187 (2.64x) | 0.791 (3.96x) | 2.419 (1.30x) | 1.131 (2.77x) | 0.299 (10.49x) | 1.175 (2.67x) | 0.662 (4.73x) | -| q24 | 0.717 | 1.233 (0.58x) | 0.812 (0.88x) | 1.754 (0.41x) | 0.705 (1.02x) | 0.201 (3.56x) | 1.001 (0.72x) | 0.627 (1.14x) | -| q25 | 2.858 | 1.453 (1.97x) | 1.269 (2.25x) | 1.468 (1.95x) | 0.850 (3.36x) | 0.175 (16.30x) | 1.127 (2.54x) | 0.719 (3.98x) | -| q26 | 0.649 | 3.372 (0.19x) | 1.147 (0.57x) | 2.932 (0.22x) | 1.489 (0.44x) | 0.299 (2.17x) | 1.019 (0.64x) | 0.665 (0.98x) | -| q27 | 2.219 | 17.960 (0.12x) | 3.708 (0.60x) | 24.840 (0.09x) | 2.881 (0.77x) | 0.767 (2.89x) | 1.236 (1.79x) | 0.801 (2.77x) | -| q28 | 2.369 | 1.513 (1.57x) | 1.776 (1.33x) | 2.498 (0.95x) | 1.171 (2.02x) | 4.895 (0.48x) | 2.246 (1.05x) | 1.359 (1.74x) | -| q29 | 2.621 | 1.022 (2.56x) | 0.718 (3.65x) | 2.963 (0.88x) | 1.231 (2.13x) | 4.379 (0.60x) | 0.967 (2.71x) | 0.616 (4.25x) | -| q30 | 1,359.978 | 146.520 (9.28x) | 97.766 (13.91x) | 34.313 (39.63x) | 26.809 (50.73x) | timeout | 47.496 (28.63x) | 43.670 (31.14x) | +| q1 | 1.277 | 1.734 (0.74x) | 1.218 (1.05x) | 1.297 (0.98x) | 0.978 (1.31x) | 0.440 (2.90x) | 1.200 (1.06x) | 0.730 (1.75x) | +| q2 | 3.260 | 1.322 (2.47x) | 1.123 (2.90x) | 2.337 (1.39x) | 3.450 (0.94x) | 4.682 (0.70x) | 1.397 (2.33x) | 0.532 (6.13x) | +| q3 | 0.665 | 1.034 (0.64x) | 0.789 (0.84x) | 1.862 (0.36x) | 1.120 (0.59x) | 4.420 (0.15x) | 0.810 (0.82x) | 0.654 (1.02x) | +| q4 | 2.656 | 0.848 (3.13x) | 0.393 (6.75x) | 2.982 (0.89x) | 1.545 (1.72x) | 0.215 (12.35x) | 0.785 (3.38x) | 0.501 (5.30x) | +| q5 | 2.785 | 3.440 (0.81x) | 4.004 (0.70x) | 2.044 (1.36x) | 2.014 (1.38x) | 1.211 (2.30x) | 2.162 (1.29x) | 0.780 (3.57x) | +| q6 | 2.597 | 0.664 (3.91x) | 0.787 (3.30x) | 0.727 (3.57x) | 0.554 (4.69x) | 0.193 (13.44x) | 0.730 (3.56x) | 0.427 (6.08x) | +| q7 | 0.430 | 27.735 (0.02x) | 25.112 (0.02x) | 15.677 (0.03x) | 2.666 (0.16x) | 0.649 (0.66x) | 1.102 (0.39x) | 1.034 (0.42x) | +| q8 | 7.287 | 2.723 (2.68x) | 3.375 (2.16x) | 1.141 (6.39x) | 1.825 (3.99x) | 1.801 (4.05x) | 1.740 (4.19x) | 0.628 (11.61x) | +| q9 | 0.758 | 1.728 (0.44x) | 1.342 (0.56x) | 1.888 (0.40x) | 1.158 (0.65x) | 0.235 (3.23x) | 1.348 (0.56x) | 0.815 (0.93x) | +| q10 | 3.788 | 1.553 (2.44x) | 1.778 (2.13x) | 26.693 (0.14x) | 2.728 (1.39x) | 0.201 (18.84x) | 0.962 (3.94x) | 0.737 (5.14x) | +| q11 | 10.282 | 7.960 (1.29x) | 4.635 (2.22x) | 3.084 (3.33x) | 2.690 (3.82x) | 6.742 (1.53x) | 2.004 (5.13x) | 1.615 (6.37x) | +| q12 | 2.549 | 17.156 (0.15x) | 7.840 (0.33x) | 18.684 (0.14x) | 4.340 (0.59x) | 1.125 (2.27x) | 6.761 (0.38x) | 0.760 (3.35x) | +| q13 | 6.654 | 42.661 (0.16x) | 5.837 (1.14x) | 9.616 (0.69x) | 3.382 (1.97x) | 2.522 (2.64x) | 1.099 (6.05x) | 0.801 (8.31x) | +| q14 | 0.460 | 1.556 (0.30x) | 0.902 (0.51x) | 2.909 (0.16x) | 1.439 (0.32x) | 0.204 (2.26x) | 0.735 (0.63x) | 0.534 (0.86x) | +| q15 | 2.369 | 2.379 (1.00x) | 1.164 (2.04x) | 3.023 (0.78x) | 1.295 (1.83x) | 0.226 (10.49x) | 0.877 (2.70x) | 0.561 (4.23x) | +| q16 | 1.260 | 1.775 (0.71x) | 1.036 (1.22x) | 4.329 (0.29x) | 3.400 (0.37x) | 4.415 (0.29x) | 1.134 (1.11x) | 0.659 (1.91x) | +| q17 | 3.086 | 2.616 (1.18x) | 1.738 (1.78x) | 2.301 (1.34x) | 1.365 (2.26x) | 1.199 (2.57x) | 1.599 (1.93x) | 0.908 (3.40x) | +| q18 | 3.108 | 1.508 (2.06x) | 1.221 (2.55x) | 1.763 (1.76x) | 1.047 (2.97x) | 0.628 (4.95x) | 1.161 (2.68x) | 0.700 (4.44x) | +| q19 | 4.856 | 11.657 (0.42x) | 3.433 (1.41x) | 19.927 (0.24x) | 4.327 (1.12x) | 2.129 (2.28x) | 1.393 (3.49x) | 0.935 (5.19x) | +| q20 | 421.921 | 13.247 (31.85x) | 14.154 (29.81x) | 3.132 (134.71x) | 0.956 (441.15x) | 0.455 (926.79x) | 16.771 (25.16x) | 2.026 (208.27x) | +| q21 | 0.511 | 0.441 (1.16x) | 0.500 (1.02x) | 1.459 (0.35x) | 0.662 (0.77x) | 0.346 (1.48x) | 0.846 (0.60x) | 0.778 (0.66x) | +| q22 | 2.657 | 23.338 (0.11x) | 7.456 (0.36x) | 13.413 (0.20x) | 5.143 (0.52x) | 0.670 (3.97x) | 0.950 (2.80x) | 0.721 (3.69x) | +| q23 | 3.133 | 1.187 (2.64x) | 0.791 (3.96x) | 2.419 (1.30x) | 1.131 (2.77x) | 0.299 (10.49x) | 1.236 (2.54x) | 0.731 (4.28x) | +| q24 | 0.717 | 1.233 (0.58x) | 0.812 (0.88x) | 1.754 (0.41x) | 0.705 (1.02x) | 0.201 (3.56x) | 1.078 (0.67x) | 0.697 (1.03x) | +| q25 | 2.858 | 1.453 (1.97x) | 1.269 (2.25x) | 1.468 (1.95x) | 0.850 (3.36x) | 0.175 (16.30x) | 1.207 (2.37x) | 0.813 (3.51x) | +| q26 | 0.649 | 3.372 (0.19x) | 1.147 (0.57x) | 2.932 (0.22x) | 1.489 (0.44x) | 0.299 (2.17x) | 1.163 (0.56x) | 0.744 (0.87x) | +| q27 | 2.219 | 17.960 (0.12x) | 3.708 (0.60x) | 24.840 (0.09x) | 2.881 (0.77x) | 0.767 (2.89x) | 1.535 (1.45x) | 0.875 (2.54x) | +| q28 | 2.369 | 1.513 (1.57x) | 1.776 (1.33x) | 2.498 (0.95x) | 1.171 (2.02x) | 4.895 (0.48x) | 2.691 (0.88x) | 1.408 (1.68x) | +| q29 | 2.621 | 1.022 (2.56x) | 0.718 (3.65x) | 2.963 (0.88x) | 1.231 (2.13x) | 4.379 (0.60x) | 1.061 (2.47x) | 0.691 (3.79x) | +| q30 | 1,359.978 | 146.520 (9.28x) | 97.766 (13.91x) | 34.313 (39.63x) | 26.809 (50.73x) | timeout | 42.024 (32.36x) | 40.900 (33.25x) | ![](results/benchmark_heatmap.png) diff --git a/lance_csr/README.md b/lance_csr/README.md index 6ce785c..6b6cada 100644 --- a/lance_csr/README.md +++ b/lance_csr/README.md @@ -6,7 +6,7 @@ This directory has no queries or data of its own. `query.py` loads [`../lance_gr ## Setup -lance-csr is embedded: it runs inside the Python process, with no server. This directory is its own uv project, because lance-csr 0.3.1 needs pylance 13 while the root project's lance-graph 0.5.4 uses pylance 1.0.4. Build the Lance datasets first (see [lance-graph](../lance_graph/README.md)), then install: +lance-csr is embedded: it runs inside the Python process, with no server. This directory is its own uv project, because lance-csr 0.5.1 needs pylance 13 while the root project's lance-graph 0.5.4 uses pylance 1.0.4. Build the Lance datasets first (see [lance-graph](../lance_graph/README.md)), then install: ```sh uv sync --frozen @@ -35,4 +35,4 @@ From the repository root, `./run_benchmarks.sh lance_csr` runs both modes with t LANCE_CSR_MODE=memory uv run --frozen pytest benchmark_query.py --benchmark-min-rounds=5 --benchmark-disable-gc ``` -The runner doesn't pass lance-csr's `timeout` argument, so `QUERY_TIMEOUT_SECONDS` doesn't apply and queries run without a limit. The shared lance-graph query module uses the deprecated `with_node_label` and `with_relationship` builder methods, so each run prints deprecation warnings. +The runner doesn't pass lance-csr's `timeout` argument, so `QUERY_TIMEOUT_SECONDS` doesn't apply and queries run without a limit. `query.py` replaces the shared module's `build_config()`, which uses lance-graph's builder methods, with one that uses lance-csr's `node_table` and `rel_table`. It reads each relationship's endpoint node tables from its dataset name (`person_likes_post` goes from `Person` to `Post`). diff --git a/lance_csr/pyproject.toml b/lance_csr/pyproject.toml index b422d5b..b77af70 100644 --- a/lance_csr/pyproject.toml +++ b/lance_csr/pyproject.toml @@ -4,7 +4,7 @@ version = "0.1.0" description = "lance-csr runs of the lance-graph query suite" requires-python = ">=3.13" dependencies = [ - "lance-csr==0.3.1", + "lance-csr==0.5.1", "polars==1.37.1", "pyarrow==23.0.0", "pytest>=9.0.2", diff --git a/lance_csr/query.py b/lance_csr/query.py index 93f7247..813e521 100644 --- a/lance_csr/query.py +++ b/lance_csr/query.py @@ -1,7 +1,8 @@ """Run lance_graph/query.py's 30 queries on lance-csr. -The queries, graph mapping and data (lance_graph/graph_lance) are shared with lance-graph, so both -engines answer identical Cypher over identical Lance datasets. +The queries and data (lance_graph/graph_lance) are shared with lance-graph, so both engines answer +identical Cypher over identical Lance datasets. The graph mapping is the same, but built with +lance-csr's node_table and rel_table, which also name each relationship's endpoint node tables. LANCE_CSR_MODE picks how the datasets are read; both use CSR traversal (use_csr=True): - memory (default): every table loaded into memory as Arrow, as lance-graph's benchmark does @@ -29,6 +30,22 @@ _spec.loader.exec_module(q) q.CypherEngine = functools.partial(lance_csr.CypherEngine, use_csr=True) + + +def build_config() -> lance_csr.GraphConfig: + # lance-csr's rel_table names the node tables at both ends; read them from the dataset names, + # such as person_likes_post for Person to Post. + labels = {label.lower(): label for label in q.NODE_LABELS} + builder = lance_csr.GraphConfig.builder() + for label in q.NODE_LABELS: + builder = builder.node_table(label, "id") + for rel_type, dataset_name in q.REL_DATASETS.items(): + parts = dataset_name.split("_") + builder = builder.rel_table(rel_type, labels[parts[0]], labels[parts[-1]], "src", "dst") + return builder.build() + + +q.build_config = build_config if MODE == "lazy": def load_datasets(root: Path) -> dict[str, lance.LanceDataset]: names = {label: label for label in q.NODE_LABELS} | q.REL_DATASETS diff --git a/lance_csr/uv.lock b/lance_csr/uv.lock index 5fccd31..ba589e0 100644 --- a/lance_csr/uv.lock +++ b/lance_csr/uv.lock @@ -34,7 +34,7 @@ dependencies = [ [package.metadata] requires-dist = [ - { name = "lance-csr", specifier = "==0.3.1" }, + { name = "lance-csr", specifier = "==0.5.1" }, { name = "polars", specifier = "==1.37.1" }, { name = "pyarrow", specifier = "==23.0.0" }, { name = "pytest", specifier = ">=9.0.2" }, @@ -52,16 +52,16 @@ wheels = [ [[package]] name = "lance-csr" -version = "0.3.1" +version = "0.5.1" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "pyarrow" }, { name = "pylance" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/62/3d/b08698f76db0f3842b6782a773e2f5001fd11f3b3a548d1c2401285bfa28/lance_csr-0.3.1.tar.gz", hash = "sha256:c4b1e2b24434181a1c1ef0a2436ece3292e7c41b83af22705e9cb28582491139", size = 377368, upload-time = "2026-10-26T22:03:13.763Z" } +sdist = { url = "https://files.pythonhosted.org/packages/c6/7d/b2bde4142b9ac7f2ca61593ba3f1db151d7580a394eeabbd3626572720c6/lance_csr-0.5.1.tar.gz", hash = "sha256:e6a707530176b591cb70185bbef3f95ad2b53c4641347056e9d903332634d40e", size = 407023, upload-time = "2026-10-31T17:39:48.981Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/15/95/8eaa71e3a3cbab7a3dddeaf37250da783a093b6f3dfb8c83ffae0f431d26/lance_csr-0.3.1-cp312-abi3-macosx_11_0_arm64.whl", hash = "sha256:5c5b0060ecd81fc5832a994f179a53ae2876a4f6730ac5a9edc077bb69ed73f6", size = 58388379, upload-time = "2026-10-26T22:03:04.265Z" }, - { url = "https://files.pythonhosted.org/packages/9a/e7/3a70facacd665f635c86bedee72c5e185993c5469725a373f9311e8c4b3b/lance_csr-0.3.1-cp312-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:71d90e097069d5b344c1b3c27871108c4414b5c6ba7663b6bdd1a4b1775f414c", size = 65787618, upload-time = "2026-10-26T22:03:10.951Z" }, + { url = "https://files.pythonhosted.org/packages/59/e2/25afc58f33bf29a4df0f2114a146706027048d04ce1680050259a48845c4/lance_csr-0.5.1-cp312-abi3-macosx_11_0_arm64.whl", hash = "sha256:08325e1c712412e216014f81f9dbaba1bb8d7238fab4da0ecc65c4febdca7531", size = 58486525, upload-time = "2026-10-31T17:39:40.198Z" }, + { url = "https://files.pythonhosted.org/packages/94/b3/1a8f0bd219dd26d1b4c5ce829cfaab3f36de42ad7471ccff45d6fa4d57ab/lance_csr-0.5.1-cp312-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:773cd2d4ddea405cfada08789a32988e814da1e91981f3ba0cb1219a4a170a24", size = 65892908, upload-time = "2026-10-31T17:39:46.336Z" }, ] [[package]] diff --git a/results/lance-csr-0.3.1-lazy.txt b/results/archived/lance-csr-0.3.1-lazy.txt similarity index 100% rename from results/lance-csr-0.3.1-lazy.txt rename to results/archived/lance-csr-0.3.1-lazy.txt diff --git a/results/lance-csr-0.3.1-memory.txt b/results/archived/lance-csr-0.3.1-memory.txt similarity index 100% rename from results/lance-csr-0.3.1-memory.txt rename to results/archived/lance-csr-0.3.1-memory.txt diff --git a/results/benchmark_heatmap.png b/results/benchmark_heatmap.png index 214a079..b730de4 100644 Binary files a/results/benchmark_heatmap.png and b/results/benchmark_heatmap.png differ diff --git a/results/environment.yml b/results/environment.yml index f0ab5fe..150f9a6 100644 --- a/results/environment.yml +++ b/results/environment.yml @@ -1,4 +1,4 @@ -date: 2026-10-27 +date: 2026-10-31 machine: Apple M5, 24 GB os: macOS 26.6.2 power: AC Power diff --git a/results/lance-csr-0.5.1-lazy.txt b/results/lance-csr-0.5.1-lazy.txt new file mode 100644 index 0000000..0743806 --- /dev/null +++ b/results/lance-csr-0.5.1-lazy.txt @@ -0,0 +1,50 @@ +============================= test session starts ============================== +platform darwin -- Python 3.13.14, pytest-9.1.1, pluggy-1.7.0 +benchmark: 5.3.0 (defaults: timer=time.perf_counter disable_gc=True min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=5) +rootdir: /Users/prrao/code/graph-benchmark-ldbc/lance_csr +configfile: pyproject.toml +plugins: benchmark-5.3.0 +collected 30 items + +benchmark_query.py .............................. [100%] + + +------------------------------------------------------------------------------------ benchmark: 30 tests ------------------------------------------------------------------------------------ +Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- +test_benchmark_query1 1.1068 (1.68) 1.5375 (1.77) 1.1999 (1.64) 0.0747 (3.08) 1.1805 (1.63) 0.0694 (2.91) 15;5 833.4242 (0.61) 99 1 +test_benchmark_query10 0.8906 (1.35) 4.7358 (5.46) 0.9615 (1.32) 0.2251 (9.28) 0.9424 (1.30) 0.0307 (1.29) 4;42 1,040.0712 (0.76) 834 1 +test_benchmark_query11 1.8800 (2.85) 2.2941 (2.64) 2.0038 (2.75) 0.0607 (2.50) 1.9931 (2.75) 0.0685 (2.87) 75;12 499.0605 (0.36) 309 1 +test_benchmark_query12 6.1168 (9.28) 7.6960 (8.87) 6.7610 (9.26) 0.4897 (20.20) 6.5025 (8.99) 0.9520 (39.98) 65;0 147.9080 (0.11) 148 1 +test_benchmark_query13 1.0054 (1.52) 1.5330 (1.77) 1.0991 (1.51) 0.0669 (2.76) 1.0834 (1.50) 0.0667 (2.80) 108;29 909.8535 (0.66) 584 1 +test_benchmark_query14 0.6594 (1.0) 2.2205 (2.56) 0.7346 (1.01) 0.0653 (2.69) 0.7235 (1.0) 0.0414 (1.74) 72;52 1,361.2446 (0.99) 1110 1 +test_benchmark_query15 0.7156 (1.09) 4.5540 (5.25) 0.8766 (1.20) 0.3225 (13.30) 0.8046 (1.11) 0.0520 (2.18) 51;85 1,140.7989 (0.83) 982 1 +test_benchmark_query16 1.0255 (1.56) 2.8543 (3.29) 1.1341 (1.55) 0.1643 (6.78) 1.1033 (1.52) 0.0484 (2.03) 29;47 881.7275 (0.64) 678 1 +test_benchmark_query17 1.2902 (1.96) 5.4670 (6.30) 1.5995 (2.19) 0.3964 (16.35) 1.4542 (2.01) 0.2591 (10.88) 54;47 625.1876 (0.46) 495 1 +test_benchmark_query18 1.0848 (1.65) 5.4825 (6.32) 1.1615 (1.59) 0.2395 (9.88) 1.1421 (1.58) 0.0320 (1.35) 3;26 860.9307 (0.63) 607 1 +test_benchmark_query19 1.2945 (1.96) 1.7630 (2.03) 1.3934 (1.91) 0.0627 (2.59) 1.3835 (1.91) 0.0756 (3.18) 136;10 717.6550 (0.52) 514 1 +test_benchmark_query2 1.2604 (1.91) 1.5870 (1.83) 1.3975 (1.91) 0.0458 (1.89) 1.3953 (1.93) 0.0615 (2.58) 182;4 715.5822 (0.52) 550 1 +test_benchmark_query20 15.8822 (24.09) 17.5501 (20.22) 16.7709 (22.98) 0.3703 (15.27) 16.8396 (23.28) 0.4892 (20.54) 15;0 59.6270 (0.04) 55 1 +test_benchmark_query21 0.8027 (1.22) 1.0811 (1.25) 0.8456 (1.16) 0.0345 (1.42) 0.8382 (1.16) 0.0238 (1.0) 24;11 1,182.6216 (0.86) 204 1 +test_benchmark_query22 0.8798 (1.33) 1.6702 (1.92) 0.9501 (1.30) 0.0596 (2.46) 0.9368 (1.29) 0.0453 (1.90) 64;37 1,052.5614 (0.77) 761 1 +test_benchmark_query23 1.1363 (1.72) 1.4525 (1.67) 1.2357 (1.69) 0.0409 (1.69) 1.2317 (1.70) 0.0453 (1.90) 149;21 809.2882 (0.59) 604 1 +test_benchmark_query24 0.9944 (1.51) 1.5253 (1.76) 1.0776 (1.48) 0.0550 (2.27) 1.0662 (1.47) 0.0429 (1.80) 101;38 927.9515 (0.68) 677 1 +test_benchmark_query25 1.1357 (1.72) 1.3586 (1.57) 1.2067 (1.65) 0.0365 (1.50) 1.2007 (1.66) 0.0507 (2.13) 201;10 828.7400 (0.60) 651 1 +test_benchmark_query26 1.0215 (1.55) 3.0645 (3.53) 1.1630 (1.59) 0.1880 (7.76) 1.1263 (1.56) 0.0671 (2.82) 30;46 859.8797 (0.63) 677 1 +test_benchmark_query27 1.2326 (1.87) 17.2890 (19.92) 1.5347 (2.10) 1.0284 (42.42) 1.3357 (1.85) 0.0895 (3.76) 19;47 651.6080 (0.48) 455 1 +test_benchmark_query28 1.9332 (2.93) 28.8359 (33.22) 2.6913 (3.69) 1.6562 (68.31) 2.4330 (3.36) 0.6988 (29.35) 15;24 371.5676 (0.27) 397 1 +test_benchmark_query29 0.9771 (1.48) 1.9362 (2.23) 1.0609 (1.45) 0.0867 (3.57) 1.0408 (1.44) 0.0533 (2.24) 46;43 942.5568 (0.69) 631 1 +test_benchmark_query3 0.7461 (1.13) 0.9754 (1.12) 0.8101 (1.11) 0.0356 (1.47) 0.8071 (1.12) 0.0423 (1.78) 24;4 1,234.4895 (0.90) 102 1 +test_benchmark_query30 39.3043 (59.61) 48.5372 (55.92) 42.0236 (57.57) 1.9553 (80.65) 42.2449 (58.39) 2.2818 (95.83) 5;1 23.7961 (0.02) 24 1 +test_benchmark_query4 0.7112 (1.08) 0.9522 (1.10) 0.7849 (1.08) 0.0242 (1.0) 0.7832 (1.08) 0.0285 (1.20) 259;25 1,273.9761 (0.93) 955 1 +test_benchmark_query5 1.9825 (3.01) 2.4624 (2.84) 2.1618 (2.96) 0.0725 (2.99) 2.1501 (2.97) 0.0863 (3.63) 98;10 462.5759 (0.34) 339 1 +test_benchmark_query6 0.6599 (1.00) 0.8680 (1.0) 0.7299 (1.0) 0.0282 (1.16) 0.7277 (1.01) 0.0343 (1.44) 265;21 1,369.9784 (1.0) 942 1 +test_benchmark_query7 1.0491 (1.59) 1.2558 (1.45) 1.1019 (1.51) 0.0323 (1.33) 1.0952 (1.51) 0.0329 (1.38) 34;9 907.5534 (0.66) 153 1 +test_benchmark_query8 1.5128 (2.29) 2.1154 (2.44) 1.7397 (2.38) 0.0750 (3.09) 1.7217 (2.38) 0.0930 (3.91) 130;9 574.8120 (0.42) 463 1 +test_benchmark_query9 1.2541 (1.90) 1.5054 (1.73) 1.3479 (1.85) 0.0491 (2.02) 1.3398 (1.85) 0.0569 (2.39) 44;5 741.8973 (0.54) 161 1 +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + +Legend: + Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile. + OPS: Operations Per Second, computed as 1 / Mean +============================= 30 passed in 22.59s ============================== diff --git a/results/lance-csr-0.5.1-memory.txt b/results/lance-csr-0.5.1-memory.txt new file mode 100644 index 0000000..cdc615c --- /dev/null +++ b/results/lance-csr-0.5.1-memory.txt @@ -0,0 +1,50 @@ +============================= test session starts ============================== +platform darwin -- Python 3.13.14, pytest-9.1.1, pluggy-1.7.0 +benchmark: 5.3.0 (defaults: timer=time.perf_counter disable_gc=True min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=5) +rootdir: /Users/prrao/code/graph-benchmark-ldbc/lance_csr +configfile: pyproject.toml +plugins: benchmark-5.3.0 +collected 30 items + +benchmark_query.py .............................. [100%] + + +------------------------------------------------------------------------------------------------ benchmark: 30 tests ------------------------------------------------------------------------------------------------ +Name (time in us) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- +test_benchmark_query1 634.3330 (1.66) 932.2500 (1.42) 729.8911 (1.71) 75.0810 (4.30) 709.4585 (1.69) 68.7920 (3.40) 5;3 1,370.0674 (0.58) 26 1 +test_benchmark_query10 669.3330 (1.75) 1,887.3340 (2.88) 737.3078 (1.73) 71.0321 (4.07) 726.5420 (1.73) 24.8645 (1.23) 31;65 1,356.2856 (0.58) 1011 1 +test_benchmark_query11 1,469.9590 (3.84) 5,476.3330 (8.36) 1,615.0940 (3.78) 417.4248 (23.93) 1,555.3755 (3.71) 80.7510 (4.00) 1;5 619.1590 (0.26) 90 1 +test_benchmark_query12 707.5000 (1.85) 942.5420 (1.44) 760.4058 (1.78) 32.8495 (1.88) 752.3335 (1.79) 34.7080 (1.72) 64;10 1,315.0873 (0.56) 226 1 +test_benchmark_query13 729.0420 (1.90) 1,057.5830 (1.61) 800.8536 (1.88) 32.4599 (1.86) 796.0420 (1.90) 35.7930 (1.77) 178;24 1,248.6676 (0.53) 710 1 +test_benchmark_query14 487.4170 (1.27) 654.8750 (1.0) 534.0363 (1.25) 17.9638 (1.03) 532.5830 (1.27) 20.2080 (1.0) 356;53 1,872.5319 (0.80) 1449 1 +test_benchmark_query15 518.5420 (1.35) 788.2500 (1.20) 560.6848 (1.31) 19.2686 (1.10) 558.5000 (1.33) 21.7712 (1.08) 332;36 1,783.5333 (0.76) 1367 1 +test_benchmark_query16 620.2910 (1.62) 769.3750 (1.17) 658.5908 (1.54) 18.2096 (1.04) 657.0830 (1.57) 21.6043 (1.07) 268;26 1,518.3936 (0.65) 1027 1 +test_benchmark_query17 849.5420 (2.22) 1,234.4160 (1.88) 908.0775 (2.13) 39.6692 (2.27) 902.3125 (2.15) 38.6035 (1.91) 59;11 1,101.2276 (0.47) 264 1 +test_benchmark_query18 650.5420 (1.70) 852.0420 (1.30) 700.3193 (1.64) 24.2825 (1.39) 696.9170 (1.66) 28.1040 (1.39) 255;29 1,427.9200 (0.61) 996 1 +test_benchmark_query19 863.7910 (2.26) 1,212.2920 (1.85) 935.4496 (2.19) 42.3268 (2.43) 924.3120 (2.20) 49.3330 (2.44) 160;19 1,069.0046 (0.46) 630 1 +test_benchmark_query2 485.7500 (1.27) 1,049.9170 (1.60) 532.1240 (1.25) 32.3186 (1.85) 527.1250 (1.26) 24.9685 (1.24) 58;19 1,879.2613 (0.80) 479 1 +test_benchmark_query20 1,856.3750 (4.85) 2,819.2080 (4.30) 2,025.7905 (4.75) 163.5517 (9.38) 1,957.5420 (4.67) 131.2918 (6.50) 65;53 493.6345 (0.21) 419 1 +test_benchmark_query21 687.0000 (1.79) 988.8750 (1.51) 778.0500 (1.82) 44.8463 (2.57) 770.9170 (1.84) 27.6665 (1.37) 14;11 1,285.2644 (0.55) 80 1 +test_benchmark_query22 669.5840 (1.75) 1,104.3750 (1.69) 720.8336 (1.69) 25.0228 (1.43) 717.1045 (1.71) 24.6260 (1.22) 189;26 1,387.2828 (0.59) 890 1 +test_benchmark_query23 684.6250 (1.79) 915.5830 (1.40) 731.1930 (1.71) 20.1686 (1.16) 727.7290 (1.74) 20.4165 (1.01) 215;46 1,367.6280 (0.58) 1052 1 +test_benchmark_query24 657.0420 (1.72) 790.2080 (1.21) 696.9850 (1.63) 17.4434 (1.0) 695.3750 (1.66) 20.5208 (1.02) 283;34 1,434.7511 (0.61) 1117 1 +test_benchmark_query25 755.8330 (1.97) 4,335.0000 (6.62) 813.4423 (1.91) 127.4248 (7.31) 799.5830 (1.91) 27.9168 (1.38) 15;59 1,229.3435 (0.52) 963 1 +test_benchmark_query26 694.3330 (1.81) 961.8750 (1.47) 743.5708 (1.74) 20.8127 (1.19) 741.1250 (1.77) 22.0517 (1.09) 263;36 1,344.8618 (0.57) 1067 1 +test_benchmark_query27 821.0840 (2.14) 1,159.8750 (1.77) 874.6335 (2.05) 32.6918 (1.87) 868.3960 (2.07) 36.3335 (1.80) 153;27 1,143.3360 (0.49) 696 1 +test_benchmark_query28 1,334.1670 (3.48) 1,721.0830 (2.63) 1,407.7025 (3.30) 41.4662 (2.38) 1,400.1250 (3.34) 48.8438 (2.42) 146;9 710.3774 (0.30) 569 1 +test_benchmark_query29 640.5410 (1.67) 3,305.8340 (5.05) 691.4326 (1.62) 81.3623 (4.66) 687.2295 (1.64) 21.0410 (1.04) 2;28 1,446.2726 (0.62) 1086 1 +test_benchmark_query3 599.3750 (1.57) 829.7080 (1.27) 654.0271 (1.53) 37.6271 (2.16) 643.4590 (1.53) 36.8015 (1.82) 52;15 1,528.9886 (0.65) 223 1 +test_benchmark_query30 38,606.5000 (100.81) 44,217.0830 (67.52) 40,899.5560 (95.81) 1,381.0011 (79.17) 40,732.6250 (97.14) 1,600.3020 (79.19) 8;1 24.4501 (0.01) 23 1 +test_benchmark_query4 452.3340 (1.18) 660.0000 (1.01) 501.1263 (1.17) 19.6150 (1.12) 498.0625 (1.19) 21.6255 (1.07) 337;52 1,995.5048 (0.85) 1352 1 +test_benchmark_query5 727.3330 (1.90) 1,011.1250 (1.54) 779.6713 (1.83) 32.7196 (1.88) 771.4580 (1.84) 31.5730 (1.56) 21;7 1,282.5918 (0.55) 145 1 +test_benchmark_query6 382.9590 (1.0) 1,307.8750 (2.00) 426.8827 (1.0) 52.1038 (2.99) 419.3125 (1.0) 20.7080 (1.02) 70;106 2,342.5642 (1.0) 1704 1 +test_benchmark_query7 974.8340 (2.55) 1,210.8750 (1.85) 1,034.2585 (2.42) 31.1440 (1.79) 1,030.4585 (2.46) 32.5210 (1.61) 59;13 966.8763 (0.41) 260 1 +test_benchmark_query8 592.6250 (1.55) 699.4170 (1.07) 627.8437 (1.47) 19.7342 (1.13) 624.6250 (1.49) 22.8952 (1.13) 40;7 1,592.7532 (0.68) 161 1 +test_benchmark_query9 761.3340 (1.99) 1,138.3750 (1.74) 815.0558 (1.91) 30.2278 (1.73) 808.8750 (1.93) 30.1670 (1.49) 191;44 1,226.9098 (0.52) 951 1 +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + +Legend: + Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile. + OPS: Operations Per Second, computed as 1 / Mean +============================= 30 passed in 18.82s ============================== diff --git a/run_benchmarks.sh b/run_benchmarks.sh index 9833046..52c44c0 100755 --- a/run_benchmarks.sh +++ b/run_benchmarks.sh @@ -31,8 +31,8 @@ RUNS=( "lance_graph lance-graph-0.5.4" "duckpgq duckpgq-1.5.6" "falkordb falkordb-6.0.2" - "lance_csr lance-csr-0.3.1-memory memory" - "lance_csr lance-csr-0.3.1-lazy lazy" + "lance_csr lance-csr-0.5.1-memory memory" + "lance_csr lance-csr-0.5.1-lazy lazy" ) for run in "${RUNS[@]}"; do read -r dir name mode <<< "$run"