Repository navigation
feat: add rowMode option to keep columns with duplicate names - #591
vuanhphung wants to merge 3 commits into
Conversation
Rows keyed by column name lose data when columns share a name (e.g. SELECT carat AS a, color AS a). rowMode: 'array' returns each row as values in schema order on the Thrift JSON, Arrow, CloudFetch and kernel paths. The default object mode is unchanged but now warns once per result set when it detects duplicate column names. Fixes #118 Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
There was a problem hiding this comment.
Verdict: 1 Low
Looks good, with one low concern: there is no e2e coverage for the new public option. I checked the array-mode Arrow path (index-based children, per-field types, nested struct/list handling, precision passthrough, row-count slicing), the JSON path, staging re-keying, option plumbing on both backends, and the default object path. All are consistent with the existing object-mode conversion.
Cover duplicate-alias rows in array mode on Thrift column-based, inline Arrow (LZ4) and CloudFetch results, staging statements in array mode, and the kernel and Thrift backends in the kernel parity spec. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
There was a problem hiding this comment.
Verdict: 1 Low
Looks good, with one low finding: the new RowMode type isn't exported from the package entry point. I traced the option through the Thrift (JSON, inline Arrow, CloudFetch) and kernel paths, plus staging and validation. The array path reads cells by index and converts types the same way the object path does, and the default object path is unchanged apart from the duplicate-name warning.
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
Rows are returned as objects keyed by column name, so columns that share a name overwrite each other:
SELECT carat AS a, color AS acomes back as{ a: 'E' }and thecaratvalue is lost. This affects every result path (Thrift column-based, inline Arrow, CloudFetch, and the kernel backend).This adds an opt-in
rowModeoption toexecuteStatement(), like node-postgres'srowMode. WithrowMode: 'array',fetchChunk/fetchAll/ iterators / streams return each row as an array of values in result-schema order, so every column is kept. Column names are still available fromgetResultMetadata()/getSchema(). The default stays'object', so existing callers see no change apart from a single warning per result set when duplicate column names are detected. The Arrow array path reads cells by column index and takes each column's type from the batch data, because apache-arrow'sRecordBatch.schemamerges same-named fields into one type. Staging (PUT/GET/REMOVE) responses are still read by name in array mode, and an unknownrowModevalue throws before the statement runs.Verification: new unit tests (
tests/unit/result/rowMode.test.ts) cover duplicate names on JSON, Arrow and kernel paths, per-column type conversion, precision preservation, the compatibility fixtures in array form, the warning, staging and validation. Full unit suite, lint, prettier, type-check and build pass. Live on a SQL warehouse, the issue query returns[[0.23, 'E'], [0.21, 'I']]on Thrift JSON, inline Arrow, CloudFetch, and kernel (sync and async), and a 2M-row duplicate-column query was checked row by row on Thrift inline Arrow, CloudFetch and kernel.The default object path is unchanged. Offline converter benchmark (500k rows × 6 columns, median of 5 fresh-process runs): Arrow object mode 2543 ns/row on main vs 2491 on this branch, JSON object mode 330.9 vs 336.3 (both within run-to-run noise). Array mode is 893 ns/row on Arrow (≈2.8× faster than object rows) and 340 on JSON. Live fetch time on a warehouse (6 columns, median of 5 fresh-process runs, main vs branch object, then branch array): Thrift inline Arrow 500k rows 2009 / 1922 / 1172 ms, Thrift CloudFetch 3M rows 9014 / 9148 / 4764 ms, kernel 3M rows 7786 / 7763 / 3050 ms. The Thrift and kernel e2e result specs also pass.
Fixes #118
This pull request and its description were written by Isaac.