Skip to content

Native insert_many: date-times, Decimals, bytes, numpy scalars and pandas NA stay on the JSON bulk entry; an uncarriable value moves its own row - #532

Closed
tae898 wants to merge 1 commit into
native-columnar-stringsfrom
mixed-chunk-bulk-native
Closed

tae898 wants to merge 1 commit into
native-columnar-stringsfrom
mixed-chunk-bulk-native

Conversation

@tae898

@tae898 tae898 commented Oct 25, 2026 •

Copy link
Copy Markdown

Stacked on #514 (base native-columnar-strings, so the diff shows only this change; GitHub retargets it to native-preview when #514 merges). It touches _native/database.py::_insert_chunk, the function #514 edits, and one #514 test that asserted the old fallback.

On the native wheel, insert_many sent a chunk through INSERT INTO t CONTENT :row for each row (arcadedb_n_exec_many) as soon as one value made json.dumps raise: a datetime, date, Decimal, bytes, a numpy scalar, a pandas NA/NaT. #514 does not change that: columnar.pack refuses date-times and Decimals, and _insert_chunk then fell to the statement path (its test test_values_json_cannot_carry_still_go_row_by_row_in_one_crossing asserted exactly that for a datetime). Json.java already decodes the tagged values {"\u0001": [kind, payload]} (T, D, n, b), so no native-image change is needed (backlog rows N-04, J-04, J-05).

What changes

  • insert_many writes those values as tagged values inside the arcadedb_n_insert_rows text (json_bulk_dumps_tagged, same function and semantics as the JVM wheel's PR). A chunk with such a value loses the columnar entry (columnar.pack still refuses it) but stays on the bulk entry.
  • A row with a value the text still cannot carry (NaN and Infinity, an integer beyond 64 bits, a date-time or Decimal inside a list or dict, a non-str key, a lone surrogate, a datetime.time) goes to the statement on its own: runs of carriable rows go to the bulk entry (columnar first, then the JSON text), runs of uncarriable rows go to arcadedb_n_exec_many in one crossing, in the order given.
  • insert_many(parallel=True) already writes every type as tagged JSON (extras.insert_parallel) and is unchanged.
  • Native: columnar bulk insert by address for strings, nulls and vectors (no JSON text) #514's test is split in two: a datetime now stays on the JSON entry (and the columnar entry is not used), and the old assertion (statement path, one crossing, no INSERT_ROWS) now uses a datetime inside a list, which is still uncarriable.
  • A quirk found on the way, not changed here: Json.java decodes any one-key map keyed by the tag marker, on the statement path and the bulk entry alike, so a user map {"\u0001": [...]} is read as a tag in both. The test pins that the two paths agree.

Stored values

tests/test_native_mixed_chunk.py (native backend only, collected through conftest.py like #514's file) stores 17 value kinds (naive and UTC+9 datetime, date, Decimal with and without exponent, bytes, bytearray, np.int64/int8/uint32/float64/float32/bool_, None) through the statement path (_command_many with INSERT ... CONTENT :row, what the chunk used to take) and through insert_many, in a declared type (DATETIME, DATE, DECIMAL, BINARY, LONG, INTEGER, DOUBLE, FLOAT, BOOLEAN properties) and a schemaless one, and compares the stored value and its Python type. A spy on lib.call proves the entry (arcadedb_n_insert_rows once, arcadedb_n_exec_many never). It also covers pandas.Timestamp (nanoseconds kept, as the statement path keeps them; aware), pandas.NA/NaT and np.datetime64 (the statement refuses these; the bulk entry stores null / a naive datetime), a 10,000-row chunk with one datetime, one uncarriable row among 1,000 (one statement run, two bulk calls, stored rows equal the statement path's), insert order around statement runs, commit_every, and rollback. 14 of the 17 tests failed before the change.

Untimed 1M-row figure

One run each (not through ab.py, so no claim), fresh process, 1,000,000 rows of six properties with a Python datetime per row, row building inside the timed call, #514's native library, through quiet_build: insert_many took 15.2 s before (100 arcadedb_n_exec_many crossings, INSERT ... CONTENT per row) and 11.2 s after (100 arcadedb_n_insert_rows crossings). The same rows without the ts column take 6.1 s on the columnar entry (arcadedb_n_insert_colp), so a date-time column still costs the columnar entry; carrying date-times in columnar.pack (an epoch-microsecond int64 column) is the next step and is not in this PR.

Local checks

Docs changed: the insert_many docstring in _native/database.py, docs/api/database.md, docs/development/testing.md.

🤖 Generated with Claude Code

…as NA stays on the JSON bulk entry; an uncarriable value moves its own row, not the chunk

Json.java already decodes the tagged values (T, D, n, b), so a datetime, date, Decimal, bytes, numpy scalar, pandas NA or NaT sitting directly in a row now crosses in the arcadedb_n_insert_rows text instead of sending the whole 10,000-row chunk through INSERT INTO t CONTENT :row for each row. A value the text still cannot carry sends only its own run of rows to the statement; the rows around it keep the bulk entry and the insert order. Python only, no native-image change.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@tae898

tae898 commented Oct 26, 2026

Copy link
Copy Markdown
Author

Closing here: the bindings moved to ArcadeData/arcadedb-python. The branch stays for the queued A/B insert-many-temporal-ab-laptop-20261025; if it passes, the change is ported there.

@tae898 tae898 closed this Oct 26, 2026
tae898 added a commit to ArcadeData/arcadedb-python that referenced this pull request Nov 4, 2026
…as NA stays on the JSON bulk entry; an uncarriable value moves its own row, not the chunk

Port of humemai/arcadedb-embedded-python#532, which was closed unmerged in the move to this repo. The JVM build has done this since #82.

_insert_chunk called the untagged json_bulk_dumps, which raises on a datetime, date, Decimal, bytes, numpy scalar, pandas NA or NaT, so the whole 10,000-row chunk went through INSERT INTO t CONTENT :row for each row (arcadedb_n_exec_many). Json.java already decodes the tagged values, so the rows now cross inside the arcadedb_n_insert_rows text via json_bulk_dumps_tagged. A row with a value the text still cannot carry (NaN or Infinity, an integer beyond 64 bits, a date-time or Decimal inside a list or dict, a non-str key, a lone surrogate, a datetime.time) goes to the statement on its own; runs of carriable rows keep the bulk entry (columnar first), and insert order is kept. Python only, no native-image change.

tests/test_native_mixed_chunk.py stores 17 value kinds through the statement path and through insert_many, declared and schemaless, and compares the stored value and its Python type; a spy on lib.call proves the entry. test_native_columnar_insert.py's datetime case is split in two: a datetime stays on the JSON entry, a datetime inside a list is still a statement run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tae898 added a commit to ArcadeData/arcadedb-python that referenced this pull request Nov 5, 2026
…ndas NA stay on the JSON bulk path; an uncarriable value moves its own row, not the chunk (#218)

* Native insert_many: a date-time, Decimal, bytes, numpy scalar or pandas NA stays on the JSON bulk entry; an uncarriable value moves its own row, not the chunk

Port of humemai/arcadedb-embedded-python#532, which was closed unmerged in the move to this repo. The JVM build has done this since #82.

_insert_chunk called the untagged json_bulk_dumps, which raises on a datetime, date, Decimal, bytes, numpy scalar, pandas NA or NaT, so the whole 10,000-row chunk went through INSERT INTO t CONTENT :row for each row (arcadedb_n_exec_many). Json.java already decodes the tagged values, so the rows now cross inside the arcadedb_n_insert_rows text via json_bulk_dumps_tagged. A row with a value the text still cannot carry (NaN or Infinity, an integer beyond 64 bits, a date-time or Decimal inside a list or dict, a non-str key, a lone surrogate, a datetime.time) goes to the statement on its own; runs of carriable rows keep the bulk entry (columnar first), and insert order is kept. Python only, no native-image change.

tests/test_native_mixed_chunk.py stores 17 value kinds through the statement path and through insert_many, declared and schemaless, and compares the stored value and its Python type; a spy on lib.call proves the entry. test_native_columnar_insert.py's datetime case is split in two: a datetime stays on the JSON entry, a datetime inside a list is still a statement run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* tests: pandas NA/NaT and numpy datetime64 are accepted by the native insert_many bulk path now, so both builds assert the same stored values

test_insert_many_mixed_values.py asserted that the native build refuses pd.NA, pd.NaT and np.datetime64 with a TypeError, so that the two tests would turn red the day native accepted them. With json_bulk_dumps_tagged on the native path they do (null, null, naive datetime), as the JVM build stores them. Both tests now run the JVM assertions on both builds. Found by the dispatched native parity run 41660808672 (passed=2015 failed=2, exactly these two).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: bulk-loads page lists the values insert_many keeps on the bulk path

Native insert_many now sends date-times, Decimals, bytes, numpy scalars and pandas NA/NaT inside the bulk text (as the JVM build has since #82); rows the text cannot carry go in with their own statement, in order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@tae898
tae898 deleted the mixed-chunk-bulk-native branch November 8, 2026 11:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant