Repository navigation
Native insert_many: date-times, Decimals, bytes, numpy scalars and pandas NA stay on the JSON bulk entry; an uncarriable value moves its own row - #532
Closed
tae898 wants to merge 1 commit into
Conversation
…as NA stays on the JSON bulk entry; an uncarriable value moves its own row, not the chunk Json.java already decodes the tagged values (T, D, n, b), so a datetime, date, Decimal, bytes, numpy scalar, pandas NA or NaT sitting directly in a row now crosses in the arcadedb_n_insert_rows text instead of sending the whole 10,000-row chunk through INSERT INTO t CONTENT :row for each row. A value the text still cannot carry sends only its own run of rows to the statement; the rows around it keep the bulk entry and the insert order. Python only, no native-image change. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Author
|
Closing here: the bindings moved to ArcadeData/arcadedb-python. The branch stays for the queued A/B insert-many-temporal-ab-laptop-20261025; if it passes, the change is ported there. |
This was referenced Nov 3, 2026
tae898
added a commit
to ArcadeData/arcadedb-python
that referenced
this pull request
Nov 4, 2026
…as NA stays on the JSON bulk entry; an uncarriable value moves its own row, not the chunk Port of humemai/arcadedb-embedded-python#532, which was closed unmerged in the move to this repo. The JVM build has done this since #82. _insert_chunk called the untagged json_bulk_dumps, which raises on a datetime, date, Decimal, bytes, numpy scalar, pandas NA or NaT, so the whole 10,000-row chunk went through INSERT INTO t CONTENT :row for each row (arcadedb_n_exec_many). Json.java already decodes the tagged values, so the rows now cross inside the arcadedb_n_insert_rows text via json_bulk_dumps_tagged. A row with a value the text still cannot carry (NaN or Infinity, an integer beyond 64 bits, a date-time or Decimal inside a list or dict, a non-str key, a lone surrogate, a datetime.time) goes to the statement on its own; runs of carriable rows keep the bulk entry (columnar first), and insert order is kept. Python only, no native-image change. tests/test_native_mixed_chunk.py stores 17 value kinds through the statement path and through insert_many, declared and schemaless, and compares the stored value and its Python type; a spy on lib.call proves the entry. test_native_columnar_insert.py's datetime case is split in two: a datetime stays on the JSON entry, a datetime inside a list is still a statement run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tae898
added a commit
to ArcadeData/arcadedb-python
that referenced
this pull request
Nov 5, 2026
…ndas NA stay on the JSON bulk path; an uncarriable value moves its own row, not the chunk (#218) * Native insert_many: a date-time, Decimal, bytes, numpy scalar or pandas NA stays on the JSON bulk entry; an uncarriable value moves its own row, not the chunk Port of humemai/arcadedb-embedded-python#532, which was closed unmerged in the move to this repo. The JVM build has done this since #82. _insert_chunk called the untagged json_bulk_dumps, which raises on a datetime, date, Decimal, bytes, numpy scalar, pandas NA or NaT, so the whole 10,000-row chunk went through INSERT INTO t CONTENT :row for each row (arcadedb_n_exec_many). Json.java already decodes the tagged values, so the rows now cross inside the arcadedb_n_insert_rows text via json_bulk_dumps_tagged. A row with a value the text still cannot carry (NaN or Infinity, an integer beyond 64 bits, a date-time or Decimal inside a list or dict, a non-str key, a lone surrogate, a datetime.time) goes to the statement on its own; runs of carriable rows keep the bulk entry (columnar first), and insert order is kept. Python only, no native-image change. tests/test_native_mixed_chunk.py stores 17 value kinds through the statement path and through insert_many, declared and schemaless, and compares the stored value and its Python type; a spy on lib.call proves the entry. test_native_columnar_insert.py's datetime case is split in two: a datetime stays on the JSON entry, a datetime inside a list is still a statement run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * tests: pandas NA/NaT and numpy datetime64 are accepted by the native insert_many bulk path now, so both builds assert the same stored values test_insert_many_mixed_values.py asserted that the native build refuses pd.NA, pd.NaT and np.datetime64 with a TypeError, so that the two tests would turn red the day native accepted them. With json_bulk_dumps_tagged on the native path they do (null, null, naive datetime), as the JVM build stores them. Both tests now run the JVM assertions on both builds. Found by the dispatched native parity run 41660808672 (passed=2015 failed=2, exactly these two). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: bulk-loads page lists the values insert_many keeps on the bulk path Native insert_many now sends date-times, Decimals, bytes, numpy scalars and pandas NA/NaT inside the bulk text (as the JVM build has since #82); rows the text cannot carry go in with their own statement, in order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #514 (base
native-columnar-strings, so the diff shows only this change; GitHub retargets it tonative-previewwhen #514 merges). It touches_native/database.py::_insert_chunk, the function #514 edits, and one #514 test that asserted the old fallback.On the native wheel,
insert_manysent a chunk throughINSERT INTO t CONTENT :rowfor each row (arcadedb_n_exec_many) as soon as one value madejson.dumpsraise: adatetime,date,Decimal,bytes, a numpy scalar, a pandasNA/NaT. #514 does not change that:columnar.packrefuses date-times and Decimals, and_insert_chunkthen fell to the statement path (its testtest_values_json_cannot_carry_still_go_row_by_row_in_one_crossingasserted exactly that for a datetime).Json.javaalready decodes the tagged values{"\u0001": [kind, payload]}(T,D,n,b), so no native-image change is needed (backlog rows N-04, J-04, J-05).What changes
insert_manywrites those values as tagged values inside thearcadedb_n_insert_rowstext (json_bulk_dumps_tagged, same function and semantics as the JVM wheel's PR). A chunk with such a value loses the columnar entry (columnar.packstill refuses it) but stays on the bulk entry.datetime.time) goes to the statement on its own: runs of carriable rows go to the bulk entry (columnar first, then the JSON text), runs of uncarriable rows go toarcadedb_n_exec_manyin one crossing, in the order given.insert_many(parallel=True)already writes every type as tagged JSON (extras.insert_parallel) and is unchanged.INSERT_ROWS) now uses a datetime inside a list, which is still uncarriable.Json.javadecodes any one-key map keyed by the tag marker, on the statement path and the bulk entry alike, so a user map{"\u0001": [...]}is read as a tag in both. The test pins that the two paths agree.Stored values
tests/test_native_mixed_chunk.py(native backend only, collected throughconftest.pylike #514's file) stores 17 value kinds (naive and UTC+9 datetime, date, Decimal with and without exponent, bytes, bytearray, np.int64/int8/uint32/float64/float32/bool_, None) through the statement path (_command_manywithINSERT ... CONTENT :row, what the chunk used to take) and throughinsert_many, in a declared type (DATETIME, DATE, DECIMAL, BINARY, LONG, INTEGER, DOUBLE, FLOAT, BOOLEAN properties) and a schemaless one, and compares the stored value and its Python type. A spy onlib.callproves the entry (arcadedb_n_insert_rowsonce,arcadedb_n_exec_manynever). It also coverspandas.Timestamp(nanoseconds kept, as the statement path keeps them; aware),pandas.NA/NaTandnp.datetime64(the statement refuses these; the bulk entry stores null / a naive datetime), a 10,000-row chunk with one datetime, one uncarriable row among 1,000 (one statement run, two bulk calls, stored rows equal the statement path's), insert order around statement runs,commit_every, and rollback. 14 of the 17 tests failed before the change.Untimed 1M-row figure
One run each (not through
ab.py, so no claim), fresh process, 1,000,000 rows of six properties with a Pythondatetimeper row, row building inside the timed call, #514's native library, throughquiet_build:insert_manytook 15.2 s before (100arcadedb_n_exec_manycrossings,INSERT ... CONTENTper row) and 11.2 s after (100arcadedb_n_insert_rowscrossings). The same rows without thetscolumn take 6.1 s on the columnar entry (arcadedb_n_insert_colp), so a date-time column still costs the columnar entry; carrying date-times incolumnar.pack(an epoch-microsecondint64column) is the next step and is not in this PR.Local checks
native-lib/colstr, built from the commit whose Java sources equal Native: columnar bulk insert by address for strings, nulls and vectors (no JSON text) #514's head):test_native_mixed_chunk.py17 passed;test_native_columnar_insert.py,test_native_columnar_pack.py,test_bulk_insert.py,test_bulk_json_paths.py,test_insert_columns.py,test_dataframe_ingest.py,test_conversion_fixes.py,test_core.py,test_graph_batch.py,test_async_executor.py,test_check_test_skips.pypass (267 passed, 20 skipped, with the one updated Native: columnar bulk insert by address for strings, nulls and vectors (no JSON text) #514 test)Docs changed: the
insert_manydocstring in_native/database.py,docs/api/database.md,docs/development/testing.md.🤖 Generated with Claude Code