Native-library perf benchmark for the driverless-only formats - #15
Merged
Conversation
…ueries Add make_mbtiles to make_perf_fixtures.py (stdlib sqlite3 + gzip, ~10k gzip-compressed synthetic vector tiles, --mbtiles-tiles scale arg), closing the last gap in the driverless-only pairing table. Driverless queries for the script-library benchmark: - mbtiles-bulk: tile count + decompressed blob byte sum (exercises the reader's gzip auto-decompression; matched by sqlite3 + gzip on the native side). - stata-data / spss-data: fold the Data table only. The full nav-doc fold also counts the Variables metadata table (+16 / +24 cells) which no library reproduces, so the fair comparison decodes the same data matrix. - evtx-count: event count for the Tier 2 structural parity gate (the reader flattens each event into its own columns; pyevtx-rs yields raw XML).
run-script-benchmark.ps1 pairs each driverless-only reader against the best native library, driven through the Power Query engine's Python/R ADO.NET script providers (the Python.Execute / R.Execute path, Route A). Same PQTest.exe harness as run-odbc-benchmark.ps1: one process per timed run, median of 5, the driverless reader on the embedded fixture vs a script reading the identical file from disk and printing one parity integer that must match (Tier 1 exact cell/element count; Tier 2 event count for evtx). Two floors are isolated, mirroring the ODBC process floor: a trivial query for PQTest startup (driverless eval), and a per-pairing imports-only script for startup + interpreter launch + import + CSV/RData marshalling (script eval). Results to results/<host>-scripts[-<label>].json+md. Scripts under tests/perf/scripts/<format>.<engine>.py|.R with a __PERF_OUT__ fixture-path placeholder; each keeps only a one-cell result data frame in the namespace so the marshalling stays cheap. Generated wrapper queries land in the gitignored scripts/out/.
Nine pairings measured back-to-back in one session, median of 5. Parity 9/9 (eight Tier 1 exact, evtx Tier 2 event count); the runner aborts on any mismatch. At 10x the native libraries win decode-only time by 4.2x (avro/fastavro) to 276x (evtx/pyevtx-rs), with gpkg/pyogrio at 133x — a far wider gap than ODBC's 2-15x, since these are compiled bulk decoders with no per-row bridging. Import + CSV/RData marshalling floor is ~3.4-4.5 s per eval, the script-side analog of the ODBC setup floor; at 1x it swamps every C-backed decode, so the 10x figures are the meaningful ones.
Extend the report from an ODBC-only writeup to two parts. Part 2 covers the nine script-engine pairings (avro, evtx, stata×2, spss×2, matlab, gpkg, mbtiles) with the 10x results table, the imports-floor observation, the 1x-is-floor-bound note, and the updated environments (Python 3.12/R 4.6 and library versions). Retitle the report and mark the seven formerly driverless-only formats as measured in the status table.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extends the ODBC-vs-driverless benchmark (part 1, PR #13) to the seven formats
with no mainstream ODBC driver — avro, evtx, stata, spss, matlab, gpkg,
mbtiles — by pairing each driverless reader against the best native library
through the Power Query engine's Python/R script providers (the
Python.Execute/R.Executepath). SamePQTest.exeharness as part 1:one process per timed run, median of 5, driverless reader on the embedded
fixture vs a script reading the identical file from disk and printing one
parity integer that must match.
What's here
make_mbtilesinmake_perf_fixtures.py(stdlib sqlite3 + gzip, ~10k gzip-compressed synthetic vector tiles,
--mbtiles-tilesscale arg) plusqueries/mbtiles-bulk.query.pq.run-script-benchmark.ps1— nine pairings (stata/spss paired againstboth pandas/pyreadstat and R haven). Isolates two floors: a trivial query
for PQTest startup (driverless eval) and a per-pairing imports-only script
for interpreter launch + import + CSV/RData marshalling (script eval).
tests/perf/scripts/<format>.<engine>.py|.R; each keepsonly a one-cell result data frame in the namespace so the marshalling stays
cheap.
stata-data,spss-data) and theevtx-countstructural gate for Tier 2 parity.results/WOOM-scripts[-10x].json+md) and a two-partREPORT.md.
Result
At 10x the native libraries win decode-only time by 4.2x (fastavro) to
276x (pyevtx-rs), with gpkg/pyogrio at 133x — a far wider gap than part 1's
2–15x, because these are compiled bulk decoders with no per-row bridging.
Parity is 9/9 (eight Tier 1 exact cell/element counts, evtx Tier 2 event
count); the runner aborts on any mismatch. The ~3.4–4.5 s import/marshalling
floor is the script-side analog of the ODBC setup floor and swamps every
C-backed decode at 1x, so the 10x figures are the meaningful ones.
Setup for the script-provider route is documented in
tests/perf/SETUP.md(§3, PR #14). Fixtures and generated wrapper queries stay gitignored; results
files are committed per machine.
Note for follow-up
The 1x→10x→25x scaling screen (the same one that caught the xlsb quadratic)
flags evtx as mildly superlinear (k≈1.2) while gpkg is cleanly linear.
The likely cause is whole-file retention of all parsed binxml trees before
row extraction. That fix is intentionally not in this PR — it's reader
work, separate from the benchmark — and will follow on its own branch.