Repository navigation
Native insert_many: date-times, Decimals, bytes, numpy scalars and pandas NA stay on the JSON bulk path; an uncarriable value moves its own row, not the chunk - #218
Conversation
…as NA stays on the JSON bulk entry; an uncarriable value moves its own row, not the chunk Port of humemai/arcadedb-embedded-python#532, which was closed unmerged in the move to this repo. The JVM build has done this since #82. _insert_chunk called the untagged json_bulk_dumps, which raises on a datetime, date, Decimal, bytes, numpy scalar, pandas NA or NaT, so the whole 10,000-row chunk went through INSERT INTO t CONTENT :row for each row (arcadedb_n_exec_many). Json.java already decodes the tagged values, so the rows now cross inside the arcadedb_n_insert_rows text via json_bulk_dumps_tagged. A row with a value the text still cannot carry (NaN or Infinity, an integer beyond 64 bits, a date-time or Decimal inside a list or dict, a non-str key, a lone surrogate, a datetime.time) goes to the statement on its own; runs of carriable rows keep the bulk entry (columnar first), and insert order is kept. Python only, no native-image change. tests/test_native_mixed_chunk.py stores 17 value kinds through the statement path and through insert_many, declared and schemaless, and compares the stored value and its Python type; a spy on lib.call proves the entry. test_native_columnar_insert.py's datetime case is split in two: a datetime stays on the JSON entry, a datetime inside a list is still a statement run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…insert_many bulk path now, so both builds assert the same stored values test_insert_many_mixed_values.py asserted that the native build refuses pd.NA, pd.NaT and np.datetime64 with a TypeError, so that the two tests would turn red the day native accepted them. With json_bulk_dumps_tagged on the native path they do (null, null, naive datetime), as the JVM build stores them. Both tests now run the JVM assertions on both builds. Found by the dispatched native parity run 41660808672 (passed=2015 failed=2, exactly these two). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Timed A/B of this PR (native build; arm A = main's CI native wheel at
By the study's rule (ship as a speed change only if the upper end of the What the PR still does is parity: on the native build |
Closes #217
Port of humemai/arcadedb-embedded-python#532 (closed unmerged in the move) onto this repo's
main. The JVM build has done the same since #82.On the native build,
_insert_chunkcalled the untaggedjson_bulk_dumps, so onedatetime,date,Decimal,bytes, numpy scalar or pandasNA/NaTsent the whole chunk toINSERT ... CONTENT :rowfor each row (arcadedb_n_exec_many).Json.javaalready decodes the tagged values, so this is Python only and needs no native-image rebuild.What changes
src/arcadedb/_native/database.py:_insert_chunkusesjson_bulk_dumps_tagged, so those values cross inside thearcadedb_n_insert_rowstext. Such a chunk loses the columnar entry (columnar.packstill refuses it) but stays on the bulk entry.datetime.time) goes to the statement on its own. Runs of carriable rows go to the bulk entry (columnar first, then the JSON text), runs of uncarriable rows go toarcadedb_n_exec_manyin one crossing, in the order given.insert_many(parallel=True)already wrote every type as tagged JSON and is unchanged.tests/test_native_columnar_insert.py: the datetime case is split in two. A datetime now stays on the JSON entry, and the old assertion (statement path, one crossing, noINSERT_ROWS) uses a datetime inside a list.Tests
tests/test_native_mixed_chunk.py(native backend only, incollect_ignoreelsewhere) is the parity test: it stores 17 value kinds (naive and UTC+9 datetime, whole-second datetime, date, three Decimals, bytes, bytearray,np.int64/int8/uint32/float64/float32/bool_, None, a plain int) through the statement path (INSERT ... CONTENT :row, what the chunk used to take) and throughinsert_many, in declared types and schemaless, and compares each stored value and its Python type. A spy onlib.callproves the entry (arcadedb_n_insert_rowsonce,arcadedb_n_exec_manynever). It also coverspandas.Timestamp(nanoseconds, aware),pandas.NA/NaTandnp.datetime64(null / naive datetime, compared with the same rows holding None), a 10,000-row chunk with one datetime, one uncarriable row among 1,000 (one statement run, two bulk calls, stored rows equal the statement path's), insert order around statement runs,commit_every, rollback, and a user map keyed by the tag marker.Not run locally: the tests need a native wheel, so CI is the check. Locally:
black,isortand the repo pre-commit hooks pass,compile,pyflakes(only three unused-import notes that were already indatabase.py) and Bandit are clean, and a stub of_insert_chunkconfirmed the run order (bulk, statement, bulk, ...) for five scattered uncarriable rows.This PR is a draft: a timed A/B on the main CI wheel with only these source files overlaid comes before any speed claim. The untimed 1M-row figure in #217 is a hypothesis.
🤖 Generated with Claude Code