ODBC vs driverless benchmark: five pairings, 10x scaling runs, xlsb linear grid fix - #13
Merged
Conversation
tests/perf/run-odbc-benchmark.ps1 times the same full-decode query through PQTest.exe against the driverless Sqlite3.Database reader (fixture embedded in the mez) and Odbc.Query via an installed SQLite3 ODBC driver reading the identical bulk.db from disk. Both sides must return the same value. A trivial-query case captures PQTest process overhead so eval-only ratios can be derived. Results (median of 5, plus hardware/software environment) land in tests/perf/results/<hostname>.json+md, one file per machine. First machine (WOOM, i7-1260P, no Office, sqliteodbc 0.99991): driverless 22.4 s wall / 18.4 s eval, ODBC 4.7 s wall / 0.8 s eval on 200k rows x 4 cols; eval-only ratio ~24x in favour of the native driver.
Office (64-bit, ACE 16.0.20131.20012) unlocks the remaining driver
counterparts. The runner is now pairing-driven - sqlite3, xlsb, xls, access,
dbf - each with driver detection so partial environments record skipped
pairings instead of failing, and per-pairing run-count overrides.
ACE rejects the synthetic workbook fixtures ("External table is not in the
expected format"), so make_ace_fixtures.ps1 authors real ones through the
installed engines: bulk-ace.xlsb/.xls via Excel COM automation, bulk-ace.accdb
via DAO (first bulk Access fixture in the suite). The workbooks carry a header
row because the ACE Excel ODBC driver unconditionally consumes row 1 as
column names; the driverless pairing queries Table.Skip(1) to fold the same
data cells. Output parity holds exactly on all five pairings.
WOOM results (eval-only, median of 5): sqlite3 11.0x in favour of the native
driver, xlsb 2.2x, xls 1.3x, access 1.3x - and dbf 0.78x, the driverless
reader beating the ACE dBASE driver outright.
Runner gains -Label so scaled runs write results/<hostname>-<label> beside the baseline; the Excel fixture writer now writes in 10k-row blocks (one SAFEARRAY per 100k-row sheet is fragile) and rejects >65536 BIFF8 rows. Fixtures: sqlite3 2M rows, xlsb 100k, access 200k, dbf 200k, xls capped at 65k by the BIFF8 row limit. Output parity exact on all five. The 1x near-parity was the ODBC setup floor (~1 s/driver) masking per-row cost. At 10x the native drivers win everywhere: dbf 2.7x, access 8.9x, xls 10.8x, xlsb 14.7x, sqlite3 18.2x (eval-only). Driverless scaling is linear for xls (8.1x time for 8.1x rows) and access (9.7x for 10x), mildly super- linear for sqlite3 (14.3x) and dbf-sublinear (7.4x); xlsb is 17.6x time for 10x rows and worth a separate look.
The cells->table stage built a Record-as-dictionary rowMap
(Record.FromList with one field per row + Record.FieldOrDefault per
lookup) whose field-lookup cost grows with field count: row assembly was
~1.8 s at 10k rows but ~29 s at 100k (full decode 3.5 s -> 43.4 s eval,
12.4x time for 10x rows; record walk, cell decode and grouping all
scaled linearly).
Replace the Table.Group/rowMap machinery with a single walk over the
buffered cell list: cut it at row boundaries, keep each run as an [A, B)
index range (List.Range on a list enumerates its offset like a
BinaryFormat Skip record - an early version sliced runs out with it and
got 3x slower), and assemble each row with a column-pointer walk reading
cells{i} directly. Out-of-order or repeated row headers still assemble
via a rare-path sort and merge, first cell winning per column, as
before.
Same-session medians (bulk-ace fixtures, PQTest eval ms):
10k 3502 -> 3328, 30k 9789 -> 9830, 60k 26282 -> 19473,
100k 43438 -> 18669. Full suite passes (24 baselines, 10 perf).
Both scales re-measured back-to-back on 21 Jul with the 61cb250 xlsb fix, so scaling multiples are finally drift-free. Parity exact on all ten measurements. The fix verifies in this harness: xlsb driverless is linear (10.1x time for 10x rows, was 17.6x) and 1.8x faster at 100k rows (40.7 -> 22.9 s eval), pulling its 10x ratio from 14.7x to 10.0x. The first pass''s mild sqlite3 superlinearity (14.3x) measures 9.7x same-session - cross-session drift, not code. Every driverless reader now scales linearly or better. REPORT.md compiles the full analysis: 1x results (ACE 1.9-2.9x, dbf at parity, sqlite3 11.1x), 10x scaling (2.3-15.2x, ODBC per-row rates), the two-regime conclusion, environments, and method.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Benchmarks the driverless readers against their native ODBC driver counterparts — same full-decode query, same file bytes, same PQTest harness — and fixes a real scaling bug the benchmark exposed.
What's here
tests/perf/run-odbc-benchmark.ps1— pairing-driven runner (sqlite3, xlsb, xls, access, dbf). Per pairing: N timed runs + warmup on each side, trivial-query overhead case for eval-only figures, output-parity gate (aborts on mismatch), driver detection so machines without Office record skipped pairings,-Labelfor non-default fixture scales.tests/perf/make_ace_fixtures.ps1— ACE rejects the synthetic workbook fixtures, so this authors real ones through the installed engines:bulk-ace.xlsb/.xlsvia Excel COM (block writes, BIFF8 row-limit guard),bulk-ace.accdbvia DAO — the suite's first bulk Access fixture.tests/perf/odbc/(templates get absolute paths substituted at run time; nothing machine-specific is committed).tests/perf/results/(WOOM,WOOM-10x), and the compiled analysis intests/perf/REPORT.md.xlsb/Xlsb.Workbook.pqfix (61cb250) — the 10x scaling run caught the reader at 17.6x time for 10x rows: the cells→table stage used a Record-as-dictionary row map whose field lookup grows with field count. Replaced with a linear run walk over the buffered cell list. Verified: 10.1x for 10x rows, 100k-row decode 40.7 → 22.9 s, full suite passes.Results (WOOM, i7-1260P, eval-only = median of 5 minus process overhead)
Ratios > 1 favour the native driver. Output parity exact on all ten measurements.
Two regimes: on small files ODBC's ~1 s per-connection setup dominates, so driverless sits at 1.0–2.9x against ACE (dbf at parity) and can win on wall clock; at scale the native drivers win everywhere, 2.3x (dbf) to 15x (sqlite3). Full analysis, method, per-row rates, and measurement discipline (settle-time, drift, same-session A/B rules) in
tests/perf/REPORT.md.