Repository navigation
Description
Native insert_many sends a chunk whose rows hold lists of ints (and ragged lists of floats) as one JSON text, because the columnar entry arcadedb_n_insert_colp has no layout for them. Today columnar._list_column returns None for a column of lists of ints, so pack refuses the whole chunk and Database._insert_chunk falls through to arcadedb_n_insert_rows. A fixed-length list of floats for an ARRAY_OF_FLOATS property does go as a float32 matrix; a ragged one, or one with a None row, does not.
The shape that hits it is the sparse-vector lane: Doc(id LONG, tokens ARRAY_OF_INTEGERS, weights ARRAY_OF_FLOATS) with an LSM_SPARSE_VECTOR index on (tokens, weights), loaded with insert_many. The probe (untimed orientation, one run per cell, laptop, 100,000 rows, commit_every=500, wheels of 26.11.1.dev0) logged columns_declined_by_columnar={'tokens': 200} for all 200 chunks of every sparse shape. Net microseconds per row is the wall time minus the row generator:
| shape | native net us/row | JVM net us/row | native / JVM | native text chars | JVM text chars | native / JVM chars |
|---|---|---|---|---|---|---|
| sparse30 | 82.80 | 39.77 | 2.08x | 81621820 | 19969590 | 4.09x |
| sparse30_noidx | 35.60 | 14.42 | 2.47x | 81621820 | 19969590 | 4.09x |
| sparse_ragged | 92.62 | 66.89 | 1.38x | 88288760 | 88288760 | 1.00x |
The JVM build sends weights of the fixed-length shapes as a float32 matrix (split_float_vectors) and keeps the rest as JSON text, so on the fixed shape its text is about a quarter of the native text. On the ragged shape both builds send the same text (the two counts are equal) and the native build is 1.38x the JVM per row, so there the native build only gets ahead by leaving the text behind.
Figures are untimed single runs from paper-ingest-probes-20261104/out-20261103 (generated by script from the two logs); a timed wheel-vs-wheel A/B will decide whether the change ships.
Proposal: add four ragged list kinds to the columnar entry, laid out like the existing string column (int32 element offsets in the data buffer, the elements in the aux buffer, optional Arrow null bitmap): I int32 elements stored as int[] (property declared ARRAY_OF_INTEGERS), E float32 stored as float[] (declared ARRAY_OF_FLOATS), Q int64 stored as a list of Long and D float64 stored as a list of Double (undeclared or declared LIST), which is what the JSON text parses to today. A library without the decoder is detected by the absence of a probe entry (arcadedb_n_colp_lists) and keeps such columns on the JSON path, the way lib.NEXT_BIN does for arcadedb_n_next_bin. Anything the kinds cannot carry unchanged (mixed int and float elements, bools, nested lists, NaN or an infinity, an int past 64 bits, an int outside int32 for ARRAY_OF_INTEGERS, another declared array type) stays on the JSON path.
Tests: a spy proves the columnar entry ran and no JSON text was sent, and the stored records (values, Python types, the engine's type(), and vector.sparseNeighbors results through the sparse index) are compared with the JSON path. They run in CI.
Activity
tae898 commentedon Nov 4, 2026
Timed result of PR #232 for this issue: both the first run and the confirmatory run are GRAY by the rule written before the first run (both sparse shapes load faster, but the plain-row control moved too and peak RSS is higher), so the PR stays a draft with no speed claim. Figures for both runs: #232 (comment)
tae898 commentedon Nov 5, 2026
Closing as not planned: #232 was measured twice and came out inconclusive by its pre-registered rule (results above and on #232), and speed was its only motive, so the current path stays.