Skip to content

Always use iso format and test timezone impact #75

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
theory merged 1 commit into from
Aug 28, 2026
Merged

Always use iso format and test timezone impact #75

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 26 additions & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,31 @@ All notable changes to this project will be documented in this file. It uses the
[Semantic Versioning]: https://semver.org/spec/v2.0.0.html
"Semantic Versioning 2.0.0"

## [v0.1.1] — Unreleased

### 🐞 Bug Fixes

* Added the `date_time_output_format='iso'` setting to each chDB query to
always format `timestamp` and `timestamptz` values in plain text formats
with the ISO-8601 format in UTC, converting `timestamp` values from the
current `timezone` setting.

### 💅🏻 Quality

* Added comprehensive tests for timestamp and timestamptz `COPY` output and
round-tripping, including the impact of the `timezone` setting on
`timestamp` values.

### 📚 Documentation

* Added the "Timestamp Conversion" section to the [chdb_hook docs] to
document the ISO-8601 format of plain text exports of `timestamp` and
`timestamptz` values, as well as the impact of the `timezone` setting on
exported and imported values.

[v0.1.1]: https://github.com/clickhouse/pg_chdb/compare/v0.1.0...v0.1.1
[chdb_hook docs](./doc/chdb_hook.md)

## [v0.1.0] — 2026-08-25

The theme of this release is *Shakedown.*
Expand Down Expand Up @@ -48,7 +73,7 @@ The theme of this release is *Shakedown.*
* chdb extension reference documentation in [doc/chdb.md](doc/chdb.md)
* chdb_hook module reference documentation in [doc/chdb_hook.md](doc/chdb_hook.md)

[v0.1.0]: https://github.com/clickhouse/pg_chdb/compare/a1487bd...v0.1.0
[v0.1.0]: https://github.com/clickhouse/pg_chdb/compare/fca4dc1...v0.1.0
[chDB]: https://clickhouse.com/chdb "chDB - fast, reliable, and scalable in-process database"
[formats]: https://github.com/chdb-io/chdb/blob/main/refs/clickhouse-formats-settings.md#complete-format-names-table
"chDB Docs: Complete Format Names Table"
Expand Down
38 changes: 36 additions & 2 deletions doc/chdb_hook.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -477,8 +477,8 @@ those you need.
| date | Date32 | |
| time | Time64(6) | Override with `String` for formats that don't support times. |
| timetz | String | |
| timestamp | DateTime64(6) | Declared with the `UTC` time zone; values cross as UTC instants. |
| timestamptz | DateTime64(6) | Declared with the `UTC` time zone; values cross as UTC instants. |
| timestamp | DateTime64(6) | Declared with the `UTC` time zone, converted from session time zone. |
| timestamptz | DateTime64(6) | Declared with the `UTC` time zone. |
| numeric | Decimal | |
| uuid | UUID | |
| point | `Point` | Same two coordinates as Postgres. |
Expand All @@ -497,6 +497,40 @@ No Postgres type maps to `Map` or `Tuple`, but [structure](#structure) may
name one. A `Map` can convert to an array of key value pairs, and a `Tuple`
converts to an array. Use `text[]` for heterogeneous support.

### Timestamp Conversion

In plain text formats (TSV, CSV, etc.), the `COPY` hook emits DateTime and
DateTime64 values in ISO-8601 format, `YYYY-MM-DDThh:mm:ssZ`, without regard
to the current `datestyle` setting. This ensures that timestamptz values
remain consistent, even if a source importing the values uses a different time
zone. Using a different type in the `structure` output, such as `Datetime64(3,
'America/Los_Angeles')`, has no impact on the offset of the output, but does
change the precision.

Timestamp TZ Examples:

| timestamptz | `DateTime64(6, 'UTC')` | `DateTime64(3 'Japan')` |
| ----------------------------------------- | ----------------------------- | -------------------------- |
| `2026-08-28T12:00:00Z` | `2026-08-28T12:00:00.000000Z` | `2026-08-28T12:00:00.000Z` |
| `2026-08-28T11:00:00 America/Los_Angeles` | `2026-08-28T18:00:00.000000Z` | `2026-08-28T18:00:00.000Z` |
| `2026-08-28T10:00:00.723923 Asia/Tokyo` | `2026-08-28T01:00:00.723923Z` | `2026-08-28T01:00:00.723Z` |

The `COPY` hook also converts timestamp values from the session time zone to
UTC, thus ensuring that they're output relative to that time zone. When loaded
into a new system, it should convert them to its local time zone. Thus the
values will differ if the time zone differs, but will be the same relative to
the time zone difference.

Example of the effect of the `timezone` setting on the timestamp
`2026-08-28T12:00:00`:

| timezone setting | `DateTime64(6, 'UTC')` | `DateTime64(3 'Japan')` |
| --------------------- | ----------------------------- | -------------------------- |
| `UTC` | `2026-08-28T12:00:00.000000Z` | `2026-08-28T12:00:00.000Z` |
| `America/Los_Angeles` | `2026-08-28T19:00:00.000000Z` | `2026-08-28T19:00:00.000Z` |
| `America/New_York` | `2026-08-28T16:00:00.000000Z` | `2026-08-28T16:00:00.000Z` |
| `Japan` | `2026-08-28T03:00:00.000000Z` | `2026-08-28T03:00:00.000Z` |

### chDB to Postgres

chdb_hook maps the ClickHouse types reported by [`DESCRIBE`] to these Postgres
Expand Down
1 change: 1 addition & 0 deletions src/helper/chdb_helper.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -319,6 +319,7 @@ setup_session(
"output_format_json_quote_denormals,"
"output_format_native_write_json_as_string,"
"output_format_native_encode_types_in_binary_format=0,"
"date_time_output_format='iso',"
"max_threads=%" PRIu16 ",max_parsing_threads=%" PRIu16
",max_memory_usage=%" PRIu64,
max_threads,
Expand Down
7 changes: 0 additions & 7 deletions src/hook/copy.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,13 +138,6 @@ chdb_copy(chdbCopyContext* ctx) {
*/
int nestlevel = NewGUCNestLevel();

set_config_option(
"datestyle", "ISO", PGC_USERSET, PGC_S_SESSION, GUC_ACTION_SAVE, true, 0, false
);
set_config_option(
"timezone", "UTC", PGC_USERSET, PGC_S_SESSION, GUC_ACTION_SAVE, true, 0, false
);

/* We always need a structure. */
if (ctx->structure[0] == '\0') {
TupleDesc desc = RelationGetDescr(ctx->rel);
Expand Down
15 changes: 10 additions & 5 deletions test/expected/settings.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ PREPARE show_chdb(bool) AS
'max_memory_usage', 'max_threads', 'max_parsing_threads',
'allow_experimental_nullable_tuple_type',
'output_format_native_encode_types_in_binary_format',
'output_format_native_write_json_as_string'
'output_format_native_write_json_as_string',
'date_time_output_format'
)
ORDER BY name
$$,
Expand All @@ -44,12 +45,13 @@ EXECUTE show_chdb(false);
name | value
----------------------------------------------------+-------
allow_experimental_nullable_tuple_type | 1
date_time_output_format | iso
max_memory_usage | 0
max_parsing_threads | auto
max_threads | auto
output_format_native_encode_types_in_binary_format | 0
output_format_native_write_json_as_string | 1
(6 rows)
(7 rows)

-- Set integer values.
SET chdb.max_memory = 5000;
Expand All @@ -69,12 +71,13 @@ EXECUTE show_chdb(false);
name | value
----------------------------------------------------+------------
allow_experimental_nullable_tuple_type | 1
date_time_output_format | iso
max_memory_usage | 5242880000
max_parsing_threads | 12
max_threads | 42
output_format_native_encode_types_in_binary_format | 0
output_format_native_write_json_as_string | 1
(6 rows)
(7 rows)

-- And again using size syntax for the memory
SET chdb.max_memory = '100MB';
Expand All @@ -94,12 +97,13 @@ EXECUTE show_chdb(false);
name | value
----------------------------------------------------+-----------
allow_experimental_nullable_tuple_type | 1
date_time_output_format | iso
max_memory_usage | 104857600
max_parsing_threads | 10
max_threads | 88
output_format_native_encode_types_in_binary_format | 0
output_format_native_write_json_as_string | 1
(6 rows)
(7 rows)

-- Set max values.
SET chdb.max_memory = 65535;
Expand All @@ -119,12 +123,13 @@ EXECUTE show_chdb(true);
name | value
----------------------------------------------------+-------------
allow_experimental_nullable_tuple_type | 1
date_time_output_format | iso
max_memory_usage | 68718428160
max_parsing_threads | 65535
max_threads | MAX
output_format_native_encode_types_in_binary_format | 0
output_format_native_write_json_as_string | 1
(6 rows)
(7 rows)

-- Set invalid values.
SET chdb.max_memory = -1;
Expand Down
15 changes: 10 additions & 5 deletions test/expected/settings_1.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ PREPARE show_chdb(bool) AS
'max_memory_usage', 'max_threads', 'max_parsing_threads',
'allow_experimental_nullable_tuple_type',
'output_format_native_encode_types_in_binary_format',
'output_format_native_write_json_as_string'
'output_format_native_write_json_as_string',
'date_time_output_format'
)
ORDER BY name
$$,
Expand All @@ -44,12 +45,13 @@ EXECUTE show_chdb(false);
name | value
----------------------------------------------------+-------
allow_experimental_nullable_tuple_type | 1
date_time_output_format | iso
max_memory_usage | 0
max_parsing_threads | auto
max_threads | auto
output_format_native_encode_types_in_binary_format | 0
output_format_native_write_json_as_string | 1
(6 rows)
(7 rows)

-- Set integer values.
SET chdb.max_memory = 5000;
Expand All @@ -69,12 +71,13 @@ EXECUTE show_chdb(false);
name | value
----------------------------------------------------+------------
allow_experimental_nullable_tuple_type | 1
date_time_output_format | iso
max_memory_usage | 5242880000
max_parsing_threads | 12
max_threads | 42
output_format_native_encode_types_in_binary_format | 0
output_format_native_write_json_as_string | 1
(6 rows)
(7 rows)

-- And again using size syntax for the memory
SET chdb.max_memory = '100MB';
Expand All @@ -94,12 +97,13 @@ EXECUTE show_chdb(false);
name | value
----------------------------------------------------+-----------
allow_experimental_nullable_tuple_type | 1
date_time_output_format | iso
max_memory_usage | 104857600
max_parsing_threads | 10
max_threads | 88
output_format_native_encode_types_in_binary_format | 0
output_format_native_write_json_as_string | 1
(6 rows)
(7 rows)

-- Set max values.
SET chdb.max_memory = 65535;
Expand All @@ -119,12 +123,13 @@ EXECUTE show_chdb(true);
name | value
----------------------------------------------------+-------------
allow_experimental_nullable_tuple_type | 1
date_time_output_format | iso
max_memory_usage | 68718428160
max_parsing_threads | 65535
max_threads | MAX
output_format_native_encode_types_in_binary_format | 0
output_format_native_write_json_as_string | 1
(6 rows)
(7 rows)

-- Set invalid values.
SET chdb.max_memory = -1;
Expand Down
Loading