fix(snowflake-driver): map TIMESTAMP_TZ/TIMESTAMP_LTZ to generic timestamp so they can be pre-aggregated - #11404
Open
npwalker wants to merge 3 commits into
Open
Conversation
Leaving them unmapped made BaseDriver.toGenericType() fall through to its `|| columnType` default, handing Cube Store the literal Snowflake type name. Cube Store rejects that when creating the pre-aggregation table: "Custom type 'timestamp_tz' is not supported".
…variants Credential-free unit tests, so they run without a Snowflake connection.
Adds non-UTC-offset columns, documenting that all TIMESTAMP variants carry the same UTC instant - which is what makes the shared generic mapping correct.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Check List
Issue Reference this PR resolves
Fixes #11403
Description of Changes Made
TIMESTAMP_TZandTIMESTAMP_LTZcolumns can't be used as pre-aggregation time dimensions on Snowflake — the rollup build fails in Cube Store withCustom type 'timestamp_tz' is not supported.SnowflakeToGenericTypemaps onlytimestamp_ntz, andBaseDriver.toGenericType()falls through asDbTypeToGenericType[columnType.toLowerCase()] || columnType— i.e. unknown types are returned verbatim.DbTypeToGenericTypehas notimestamp_tz/timestamp_ltzentry either, so the raw Snowflake type name is handed to Cube Store as though it were a generic type. Both paths that type a pre-aggregation column are affected:queryColumnTypes()(INFORMATION_SCHEMA.COLUMNS.DATA_TYPE→TIMESTAMP_TZ) and the snowflake-sdk path (column.getType()→timestamp_tz). Both normalise throughtoLowerCase(), so one map entry covers each.This adds the two missing entries:
Why mapping all three to the same generic type is correct
This is a type-map omission, not a change in value semantics:
src/type-parsers.tsalready liststimestamp_tzandtimestamp_ltzin its hydrator and normalises both viaformatUtcTimestamp.test/SnowflakeDriver.test.tsalready asserts that aTIMESTAMP_TZcolumn hydrates to the same string as the equivalentTIMESTAMP_NTZcolumn.initConnection()pinsTIMEZONE = 'UTC'on every connection, so all three variants render deterministically as UTC.The driver already treats
TIMESTAMP_TZas a UTC timestamp everywhere except the type map.I also confirmed Cube Store handles the offset the unload emits (
TIMESTAMP_FORMAT = 'YYYY-MM-DD"T"HH24:MI:SS.FF3TZH:TZM'). Againstcubestoredv1.7.8, via bothINSERTand the CSV-location ingest path used by the export bucket:2025-08-02T17:00:00.000(no offset)2025-08-02T17:00:00.0002025-08-02T17:00:00.000+00:002025-08-02T17:00:00.0002025-08-02T17:00:00.000-07:002025-08-03T00:00:00.0002025-08-02T17:00:00.000+05:302025-08-02T11:30:00.000date_trunc('day', ts)grouped correctly throughout — the parser converts offsets to UTC rather than truncating wall-clock time, so a+00:00value is byte-equivalent to theTIMESTAMP_NTZcase that already works.Precedent
timestamp_ntz: 'timestamp'was itself added to this exact map to fix the same class of breakage (#2592 / #2601);timestamp_tzandtimestamp_ltzwere never added alongside it. Same remedy in #1796 (int8: 'bigint'), #6458 (Firebolttimestamptz) and #10175. The class is still open elsewhere: #7017 and #5277 (identical Cube Store error, Postgres), #7127 (DuckDBhugeInt), #10363 (BigQuery precision/scale), #10289 (ClickHouse).Tests
New
test/type-mapping.test.ts— 19 assertions, no Snowflake connection required, so they run without credentials:TIMESTAMP_NTZ/TIMESTAMP_LTZ/TIMESTAMP_TZmaps totimestamp;INFORMATION_SCHEMA(upper) and snowflake-sdk (lower) spellings;TIMESTAMPvariant is returned as its raw Snowflake name — the specific fall-through this PR fixes;NUMBER,fixed,object,DATE,TEXT,VARCHAR,BOOLEAN) are unchanged, so the additions can't shadow anything.I verified all 19 pass with the patch and that the 4 new timestamp assertions fail without it.
test/SnowflakeDriver.test.tsgainsTIMESTAMP_TZandTIMESTAMP_LTZcolumns carrying a non-UTC offset (-0700), asserting they hydrate to the UTC instant (13:07:42.123 -07:00→20:07:42.123) rather than the wall clock. That pins down the property that makes the shared mapping correct.One caveat, in the interest of transparency: I have no Snowflake account to run the integration suite against, so
test/SnowflakeDriver.test.tshas not been executed locally — the expected values there are derived fromformatUtcTimestampformatting the UTC components of the SDK'sDate. The new unit tests were run and pass. Please flag if CI disagrees and I'll correct them.Note
realhas the same gap — present in thetype-parsers.tshydrator next tofixed, absent from both type maps, sotoGenericType('real')returns'real'. Left out to keep this focused; happy to fold it in.