[tae898]

What

The native wheel's bulk insert no longer has to go through JSON text when every column maps to a supported kind. arcadedb_n_insert_colp is a new entry point that reads the caller's buffers in place (an address table of six longs per column: data, aux, null bitmap) and builds the documents inside the image without a parser, a HashMap per row or a boxed number. arcadedb_n_insert_cols keeps its single-buffer form (old callers and old libraries keep working) and shares the same loop.

The design and the build decision are in the untimed study .notes/bench/repros/native-json-transport-20261025 (JSON text was 43-76% of the native bulk-insert wall on the mixed 1M-document load, rule verdict BUILD). The public API is unchanged and the JVM wheel is untouched.

Column kinds

kind	layout	used for
q, d, b	int64, float64, bool byte	ints, floats, bools (as before)
i, e	int32, float32	only when the property is declared INTEGER / FLOAT; otherwise an int32 or float32 column widens to q / d, as the JSON path stores it
f	float32 matrix, dim per row	numpy 2-D arrays and lists of float arrays (as before), and lists of float lists or Arrow list columns when the property is declared ARRAY_OF_FLOATS
s	int32 offsets[n+1] plus UTF-8 bytes (Arrow string layout; offsets[0] need not be 0, so a sliced Arrow array crosses without a copy)	str columns, Arrow string / large_string
nulls	Arrow validity bitmap, bit per row	None in a list column, Arrow nulls, pandas <NA>

Every length, offset and bitmap size is checked in the image before the first document is built: a wrong table is an exception with a message, never a fault, and nothing is written. Same transaction contract as before (a transaction opened by the call is committed every commit_every rows and rolled back on failure; a caller's transaction is left alone).

Where it is used

• insert_many: a chunk (10,000 rows) of dicts with the same keys whose columns each hold one value type is transposed to columns and sent by address.
• insert_columns, insert_dataframe, insert_arrow: string and nullable Arrow columns stay Arrow arrays and cross in place; the declared types of the type are read once per call (the DataFrame path already has them).
• Fallback to the existing JSON path, per chunk or per call, with identical stored records: nested maps and lists, mixed-type columns (including int with float), rows with different keys, date-times, decimals, bytes, integers past 64 bits, lone surrogates, undeclared list-of-float columns (the JSON path keeps their doubles), parallel=True, vertex types through GraphBatch.
• ARCADEDB_NATIVE_BULK=0 turns the columnar entry off (JSON path), for A/B runs in one image.

Tests

• tests/test_native_columnar_pack.py (all backends, no native library): the layouts read back from the addresses, null bitmaps, which columns are refused, the address table and header, Arrow offsets of sliced arrays.
• tests/test_native_columnar_insert.py (collected on native only, per conftest.py): the same data through the JSON path and through the columnar entry on the probe workload's column types (INTEGER, DOUBLE, 20-value category, variable-length text, float32 vectors) via insert_many, insert_columns, insert_dataframe, insert_arrow, with the stored records compared value by value and type by type; unicode, empty, NUL-containing and 70 KB strings; nulls (list columns and pandas nullable dtypes); int64 edges, NaN and infinities; int32 and float32 columns; vertex types; every fallback case, with a spy on the crossing so a silent fall back cannot pass; duplicate-key failure and rollback on both paths; chunking and commit interval; and calling the entry with a wrong address table, which must give an error and write nothing.
• Local native run against the library built from this branch: 1019 passed, 185 skipped (the existing NOTYET / NA-JVM / no-optional-dependency skips), 9 failed, all in test_docs_examples.py and test_wheel_packages.py, whose child processes cannot import the package or setuptools outside an installed wheel (the parity run installs the wheel, see native-wheel/parity).

Status

Ready for review; the timed check is still to come. Done: the image builds (native-spike at 8a7d4b9, -march=compatibility, about 2 minutes on the laptop), the native tests above pass against it, a wheel 26.10.2.dev1+colstr is built from it, and the study's native arms are re-run on that wheel (.notes/bench/repros/native-json-transport-20261025 README, Step 3). Untimed and indicative, 1M mixed documents, two runs per arm, one process each, GC logs on: with the UNIQUE index on id, insert_many of the same dict rows 43.5 s -> 19.0 s and one insert_columns call 15.1 s (the floor is the engine's per-document work plus the commit with the index check, 13.6 s for the numeric and vector columns that already had a binary path); without the index 34.6 s -> 10.0 s and 6.1 s. In-image allocation with the index 15.2 GB -> 9.4 GB. The JSON path inside the new image is as fast as in dev1, so the image change itself costs nothing. These are not speed claims. The timed ab.py study (dev1 wheel against this wheel, rows and frame loads, rule written first) is prepared in .notes/bench/repros/native-columnar-ab-laptop-20261025 and not started; the prototype goes into a preview only if its result says ok and both ratios have the upper end of the 95% CI below 0.80. CI: the native parity run on this branch is the check that matters.

🤖 Generated with Claude Code

[tae898]

Back to draft on purpose: untimed it is 2.3x (indexed) to 3.5x (no index) faster on 1M mixed documents, but the prototype merges only after the timed laptop A/B (dev1 wheel vs this wheel, ab.py --noise, rule written first: both gating figures need a 95% CI upper end below 0.80). That study is queued right after the running heap study.

[tae898]

Closing here: the bindings moved to ArcadeData/arcadedb-python, and this change is merged there as ArcadeData/arcadedb-python#72 (columnar bulk insert by address). The branch stays.