Halve sequential read-path allocations#303
Merged
Merged
Conversation
The repository fixtures are a handful of rows, so index benchmarks need generated data: BenchmarkTableGenerator writes a 50,000-row DBF plus a matching compound index (two tags: unique integer IDs and duplicated character codes) using the simplest structure the library's strict CDX reader accepts - uncompressed leaf entries, max-key interior entries, and a two-tag directory. Generated files are verified for index/scan result equivalence through the public API before anything is measured. IndexQueryBenchmarks pairs the same SQL with UseIndexes on and off: equality seek 459x faster (50us vs 23ms), range scan 233x, character seek 141x, top-10 descending order 20x, filtered COUNT(*) 24x, and the whole-table COUNT(*) status scan cuts allocations from 16.8MB to 10KB. Results recorded in benchmarks.md. The benchmarks project now references the published DbfDataReader 2.1.0 package. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
….0 results Recreates the Sylvan / NDbf / DbfDataReader comparison in the style of MarkPflug/Benchmarks (reading every field of every record of the tl_2019_01_place census fixture), with the same method names as the historical v0.5.x tables. The DbfDataReader package version is now an MSBuild property (-p:DbfDataReaderVersion=x.y.z) so any release can be benchmarked; the 2.x-only index benchmarks are excluded from compilation for 1.x versions. A project README documents the commands. benchmarks.md gains v1.1.0 and v2.1.0 comparison sections: the sequential read path is unchanged between the two (~1.15x NDbf, identical allocations), confirming the SQL and index features are additive, and both improve markedly on the v0.5.x-era ratios. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cuts per-row allocations on the sequential read path by half, from three sources measured on the tl_2019_01_place benchmark fixture: - Character fields: trim padding from the raw bytes before decoding, so each field allocates one right-sized string instead of a full-width string plus a trimmed copy. Byte-level trimming only applies to single-byte and UTF-8 encodings, where 0x00/0x20 can never be part of a multi-byte sequence; other encodings keep the decode-then-trim path. - Numeric and date fields: parse straight from the record buffer via stackalloc char spans (AsciiFieldParser) instead of allocating an intermediate ASCII string per field. Bytes above 0x7f decode to '?' exactly as Encoding.ASCII.GetString did, preserving parse results. - Null checks and typed getters: IDbfValue gains IsNull, so IsDBNull no longer boxes the value it inspects, and the typed getters read through DbfValue<T?> directly instead of boxing in GetValue() and unboxing. GetBoolean on a null cell now throws SqlNullValueException like every other typed getter (previously NullReferenceException). Full-row read of the 587-row/16-column fixture: 620 KB -> 299 KB allocated, 455 us -> 413 us. The benchmark loop (read + IsDBNull + field access) drops from 675 KB to 327 KB, and a fully typed consumer reads rows with zero allocation beyond the parsed strings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dates parse with the invariant culture (the field is fixed-format ASCII digits, so the current culture's calendar should never influence it), and GetBoxedValue is static. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the three read-path optimizations identified while decomposing the ~2× sequential-read gap to Sylvan.Data.XBase (measured on
tl_2019_01_place.dbf, 587 rows × 16 columns: the padded-then-trimmed double string per character field, intermediate ASCII strings for numeric parsing, and double boxing inIsDBNull+ typed getters).Changes
Character fields — trim before decoding (
DbfValueString)Padding is trimmed from the raw bytes and the right-sized result decoded once, instead of decoding the full field width and allocating a second trimmed string. Byte-level trimming applies only when
0x00/0x20can never be part of a multi-byte sequence (single-byte encodings and UTF-8); other encodings keep the decode-then-trim path. Trim order matches the previous implementation exactly (NULs off both ends first, then spaces perStringTrimmingOption), pinned by parity tests against the old algorithm across cp1252/UTF-8/UTF-16.Numeric and date fields — parse from the buffer (
AsciiFieldParser)DbfValueDecimal/DbfValueInt/DbfValueInt64/DbfValueFloat/DbfValueDateparse viastackallocchar spans with the sameNumberStyles/format arguments as before, eliminating one string allocation per numeric field per row. Bytes above0x7fdecode to'?'exactly asEncoding.ASCII.GetStringdid, so malformed content fails to parse the same way (covered by tests, including overflow, exponents,*fill, and high-byte content).Null checks and typed getters — no boxing (
IDbfValue.IsNull,DbfRecord)IsDBNullpreviously boxed every value it inspected viaGetValue(), and typed getters boxed the same value a second time insideGetValue<T>.IDbfValuegains anIsNullproperty (note: minor source-breaking addition for externalIDbfValueimplementors), and the typed getters read throughDbfValue<T?>directly.DbfQueryDataReader.IsDBNullroutes live rows through the same path.One behavioral fix rides along:
GetBooleanon a null cell now throwsSqlNullValueExceptionlike every other typed getter (previouslyNullReferenceExceptionfrom unboxing null).Measurements
Release build, min-of-5 interleaved Stopwatch rounds, full read of every field of
tl_2019_01_place.dbf(Apple M3 Max; same-session before/after):Read+IsDBNull+ field access)Read()only (full-row parse)Read+IsDBNullevery fieldAllocations on the comparative benchmark drop 675 KB → 327 KB (−52%), reaching parity with Sylvan; wall-clock improves ~9% (1.93× → 1.81× vs Sylvan). The remaining time gap is structural — eager whole-row parsing with per-field virtual dispatch vs Sylvan's lazy decode-on-access — and is tracked separately (#296 covers the selective-column side). A typed consumer now reads rows with zero allocation beyond the parsed strings themselves.
All 480 tests pass; 41 new tests pin trimming parity, span-parse parity,
IsNull, and typed-getter/IsDBNullconsistency across three fixtures.🤖 Generated with Claude Code