create_vertices: Java float[] and float32 arrays cross as one matrix per property on both wheels (closes #337) by tae898 · Pull Request #339 · ArcadeData/arcadedb-python

tae898 · 2026-11-09T12:22:52Z

Stacked on #330 (perf/318-single-pass-validation): this branch is cut from its head, because both PRs edit classify_vertex_rows, split_float_vectors and GraphBatch.create_vertices. The base is release/26.11.1 as for the other stacked PRs, so the diff here also shows the two commits of #330 until it merges. Merge order: #330 first.

Closes #337

I searched first: no open PR or issue does this. Issue #220 and PR #221 (JVM float[] as a candidate in type_conversion.py only, closed on 2026-11-05 as not planned after two GRAY timed runs) are the closest. This PR is the same idea for the JVM wheel plus the part #221 did not touch (a float32 array on a property that is not declared, and the whole native encoding), with no speed gate: the issue says every glue drawback gets fixed.

What I found (step 1, read from the code at `26.11.1` and at the #330 head)

GraphBatch.create_vertices on the JVM wheel, per value type, before this change:

row value	property declared `ARRAY_OF_FLOATS`	property not declared
Java `float[]` (`to_java_float_array`)	not a candidate (`is_vector_candidate` is `list` or `ndarray`), `classify_vertex_rows` returns `None`, the whole input goes through `_to_java_property_matrix`	same
numpy `float32`	candidate, `split_float_vectors` takes it as one float32 matrix per chunk	not a candidate (`key in declared` is the only test), whole input through `_to_java_property_matrix`
numpy `float64`	candidate, matrix, cast to float32 (documented; the engine stores floats for that type)	whole input through `_to_java_property_matrix`, stored as a list of doubles
list of floats	candidate, matrix, cast to float32	whole input through `_to_java_property_matrix` (one boxed `double[]` per row)
ragged, `None`, missing	the split declines the column, the chunk's text fails (an array is no JSON), that chunk goes through `_to_java_property_matrix`	same

A correction to the hypothesis in #337: the 26.11.1 scan in _create_vertices_json_bulk does not read every row for these shapes. It returns None at the first Java float[] it meets, which is the first row, so it costs one row. The cost is in _to_java_property_matrix.

Where the 59 to 61 percent go. _to_java_property_matrix runs once for the whole input (not per chunk) and, for each row, does a Python loop over the properties (_to_java_varargs: convert_python_to_java for every value, about ten type tests each), jpype.JArray(jpype.JObject)(row) (JPype converts the 2K-element list: a Java String per key, a boxed Long or Double per scalar), and finally assigns the N Object[] one by one into an Object[][]. In the untimed 26.11.1 profile (cells e2 and l3d, variant cv) the phase counters do not cover this function: boundary (the engine's createVertices alone) is 16.5 to 20.2 percent, the harness's own row building (caller, one to_java_float_array per row) 21.5 to 22.8 percent, and the remainder, rest, is 59.2 and 60.6 percent. So the split inside _to_java_property_matrix is read from the code, not timed: I did not time its parts separately. Peak memory is also bigger on that path, because the Python lists, the JArray wrappers and the Java Object[] of the whole input exist at once; the profile has no memory row for these two shapes, so the post1 A/B should take it (commands below).

NativeGraphBatch.create_vertices on the native wheel (the library does not look at declarations; to_java_float_array returns a float32 ndarray there, so there is no Java array):

row value	before
float32 ndarray, 1-D, non-empty	`encode_gb_args`: `_prep(args)` walks every value of every row in Python, `json.dumps` calls the hook `_gb_default` then `_default` for every array (about a dozen `isinstance` tests, one `ascontiguousarray` and one append per row), one `{"\^A":["B",offset,count]}` tag per row is written and parsed back in Java (a `HashMap`, an `ArrayList` and a `float[]` per row), `np.concatenate` joins one small array per row, then `tobytes()` copies it again. Plus a `[r or {} for r in chunk]` per row. This is the 29.0 and 31.3 percent `prep` of the profile
float64 ndarray, list	number text through `.tolist()` / `json.dumps`

What changed (step 2)

src/arcadedb/type_conversion.py: take_vector_columns(rows, declared, undeclared=False) is the one split of both wheels. A declared column qualifies as before and now also with a Java float[] or double[]. With undeclared=True any other property qualifies when its value in every row is a float32 numpy array or a Java float[] of one length (the stacked column must come out as float32, so the stored value, a float[], is what the property-matrix path stored; a float64 array, a list or a Java double[] on an undeclared property keeps the old path and its doubles). is_vector_candidate and classify_vertex_rows know the Java arrays (_java_vector_arrays, resolved once the JVM runs, () on the native wheel). _without_keys rebuilds the rows without the vector columns from columns by C iterators (itemgetter, zip, dict) when the rows share their keys in one order, and by the dict comprehension otherwise. split_float_vectors keeps its signature and return value (insert_many uses it with the declared columns only, so insert_many now also takes a Java float[] column through the matrix).
src/arcadedb/graph_batch.py: _create_vertices_json_bulk asks for the undeclared float32 columns too; docstrings.
src/arcadedb/_native/codec.py: encode_vertex_args writes the rows without the vector columns with one json.dumps (no hook and no _prep when the other values are plain scalars, decided by a census in C: _plain_rows), puts the matrices at the front of the side buffer and describes them by one "vecs": {"names", "dims", "offs"} entry. The buffer is a ctypes view of the matrix, not a copy (bytes when the array is read-only).
src/arcadedb/_native/extras.py: NativeGraphBatch._create_chunk uses it; the per-row [r or {} for r in chunk] only runs when a row is None.
native/src/main/java/org/humemai/nativelib/NativeExtras.java: withVectors turns the "vecs" entry into one float[] per row (bounds-checked against the side buffer) before GraphBatch.createVertices. The ["B", offset, count] tag stays for edge properties and for a float32 column that does not qualify; such a tag then follows the matrices in the buffer. The Java is not compiled on my laptop; the native CI job builds it.
docs/modules/ROOT/pages/how-to/vectors.adoc: the two paragraphs on create_vertices vectors.

Stored values are the same float32 values on every path: the matrix carries the float32 bits and Matrix.row(i) copies them. A library that predates the "vecs" entry cannot meet this Python (the library and the Python ship in one wheel); lib.GB_BIN false keeps the text path.

Tests (step 3)

CI is my only run environment (a repo hook blocks pytest and java locally), so none of the tests below ran on my laptop. Locally I only ran the pure functions (take_vector_columns, _without_keys, _plain_rows, encode_vertex_args, and a ctypes c_char array as a c_char_p argument) in a plain script, and the lint gates: compile(), pyflakes (its five warnings in extras.py are on the base too), black 26.10.1, isort 9.0.2, bandit==1.9.4 (clean).

tests/test_vertex_vector_loads.py (both wheels, to_java_float_array is the shared call): stored bits for Java float[] (JVM) or float32 ndarray (native), numpy float32, declared and not, 64 and 96 dimensions, with -0.0, the largest, the smallest normal and the smallest subnormal; the matrix against the per-vertex path; numpy float64 and lists (documented values); ragged, None, missing and a None row; a float64 row among float32 rows; other properties and different key orders; two vector properties and a date-time in one call; a load of chunk + 3 rows (the JVM chunk is 100,000, the native chunk is commit_every); insert_many; and the split itself without a database.
tests/test_graph_batch_vector_crossings_jvm.py (JVM only): wraps VertexBatcher, FloatVectors, convert_python_to_java and _to_java_property_matrix. For 250 rows in three chunks of 100 it asserts no fallback, no per-value conversion, ["createVerticesJson"] * 3 and one Matrix per property and chunk, for a Java float[] and a float32 array, declared and not; a ragged row sends only its chunk to the fallback; a Java double[] is a vector only when declared; insert_many does no per-row insert.
tests/test_native_vertex_vector_crossings.py (native only): spies on lib.call and on codec._gb_default, codec._default and codec._prep: three arcadedb_n_gb_bin crossings for three chunks, none of the three hooks ever called, no tag and no number in the text, the buffer is a ctypes view, the "vecs" offsets for two columns, a non-qualifying float32 array following the matrices, the text entry for float64 and lists and for GB_BIN false, and a "vecs" entry outside the buffer refused.
tests/test_native_graph_batch_binary_vectors.py: the one assertion that expected a "B" tag per vertex row now expects "vecs".
tests/conftest.py: the two spy files are registered (test_graph_batch_vector_crossings_jvm.py in NATIVE_NOT_APPLICABLE, test_native_vertex_vector_crossings.py in the native-only list).

What each one proves fails on the base (stated from reading the code; the run of these files on the base of #330 is linked under CI below and I have not read its result when writing this): every JVM spy case that holds a Java float[] or double[] (the matrix cases, the ragged case, the two-property case, insert_many) fails on fallback == [] or on the per-row insert count, the float32 case on a property that is not declared fails the same way, and the same case on a declared property passes on the base (it is the control); the native spy test fails on the hook counts (the base calls _gb_default once per row); the split tests fail because take_vector_columns does not exist. The value tests pin equality with the old path and are expected to pass on both.

Profile commands for the post1 A/B (step 5, nothing timed here)

Untimed phase and memory profile of the ingest shapes, cells e2 and l3d, variant cv, one fresh process per cell, 3 repetitions, once per arm (the 26.11.1.post1 wheel with this change against the one without it), both wheels:

taskset -c 8-11 uv run --no-project --offline --python 3.12 --with numpy==2.5.4 [--with JPype1==1.7.1 for the JVM wheel] \
  python -I profile.py --tree <unzipped wheel> --wheel {jvm,native} --shape {e2,l3d} --variant cv --db <fresh dir> --rep <n> --out <cell>.json
python score.py   # the phase table (caller, prep, validate, boundary, commit, rest) and the memory section (anon MiB, B/row, RSS)

The figure to read is rest on the JVM wheel (was 59.2 and 60.6 percent) and prep on the native wheel (was 31.3 and 29.0 percent), plus anon_MiB and B/row per cell.

CI (the humemai fork, because the upstream runner queue is backed up)

native, test-native.yml with version=26.11.1.post1: https://github.com/humemai/arcadedb-python/actions/runs/42616214876
JVM, test-jvm.yml with arcadedb-tag=26.11.1: https://github.com/humemai/arcadedb-python/actions/runs/42616219493
examples, test-python-examples.yml with arcadedb-tag=26.11.1: https://github.com/humemai/arcadedb-python/actions/runs/42616224280
the new test files on the unchanged insert_many and create_vertices: check the rows in the one json.dumps pass, not in a loop before it (refs #318) #330 head (branch perf/337-tests-on-base, JVM only; the red cases there are the proof that they fail on the base): https://github.com/humemai/arcadedb-python/actions/runs/42616346182

Update (the post1 glue A/B found a regression and a memory excess in the candidate that contains this PR)

The glue A/B of the released 26.11.1 JVM wheel (A) against candidate 3 (B) found j_g1_cv, the l2 Person load (dict rows {id, name, age, city}, 5,000 rows per create_vertices call), 1.307 times slower, and the peak RSS of the Java float[] vertex load (j_l3d_cv) 1.3 times higher. Cause of the first, from cProfile and a per-piece timing of one 5,000-row chunk: _dumps_once of #330 scanned the JSON text of every chunk with _INT64_OVERFLOW (a lookbehind and a 20-way alternation tried at every byte, no literal prefix), 13.8 ms of a 59 ms call, against 1.8 ms for the per-value loop it replaced. This PR's source is the one that candidate ships (I merged #330 7e81170 here, resolved as the integration branch resolves it), so the fix is on this branch.

Fix (head 4cc08cf, Python only, no Java changed):

The columns decide, in C, and the text is not scanned: _rows_need_walk takes the type census and screens each column of exact scalars (int range by min and max, UTF-8 form of the joined strs); both regexes are gone. The walk _check_rows is unchanged and is still the reference, so the first bad row still gives the old error text.
create_vertices screens the whole input once (screen_vertex_rows, one chunk or several). A table of exact scalars is vetted and its chunks are written by json_bulk_dumps(chunk, vetted=True), which is json.dumps and nothing else. A bad scalar anywhere still sends the whole input through the property-matrix path.
Rows with vector columns cross in blocks of about 1 MiB of floats (_VECTOR_BLOCK_FLOATS, at least 512 rows), and the previous chunk's rest rows, text and matrices are released before the next chunk is built.

Measured on this laptop (cores 12-15, JVM 2 cpus and -Xms480m as in the glue study, one fresh process per cell, tree order rotated; A = 26.11.1, B = candidate 3, C = B with the two changed files; medians over the repetitions, from repros/bindings-g1-regression-20261110/results-final/SUMMARY.txt in my notes):

figure	A	B	C	C / A
g1 wall per 5,000-row call (ms, 7 reps)	44.7	58.7	45.4	1.015
`json_bulk_dumps` of one chunk (ms, micro)	3.85	17.02	3.63 (1.49 vetted)	0.94
vector load peak RSS, 10 calls of 10,000 rows (MiB, 15 reps)	366	396	348	0.950
same, above the pre-load level (MiB)	159	174	148	0.929
vector load peak RSS, one call of 10,000 rows (MiB, 10 reps)	228	246	228	1.001

Differential check of the fix against the 26.11.1 validation loop (copied from tests/test_bulk_single_pass_validation.py): 36,000 random and scalar-table row lists, same text or same exception as the reference, whole-input verdicts equal to candidate 3 case by case, and a vetted verdict never wrong. New tests: the regex pattern test is now a column-screen test over the same corpus, spies that a vetted table runs neither the census nor the walk per chunk, the block sizes of a vector load, vector_width, and the release of the previous chunk. I have not run the new tests that need a database; CI does.

🤖 Generated with Claude Code

… pass, not in a loop before it (refs #318) json_bulk_dumps and json_bulk_dumps_tagged walked every value of every row in Python before json.dumps walked the rows again. The rows are now written once and the checks come from what that pass shows: the text (a surrogate escape, an integer token of 2**63 and above) and a census of the types of the keys and values in C. The exact per-row walk (_check_rows) is the reference and runs only when a value may be bad, and when json.dumps itself refuses the rows, so the exception type and message for the first bad row in row order are the old loop's. Fuzzed against the old code: 60000 cases, no difference. JVM GraphBatch.create_vertices: classify_vertex_rows decides the shapes by type census instead of a json_bulk_scalar_ok call per value; a scalar the text would change (NaN, integer beyond 64 bits, lone surrogate, subnormal) now sends its own chunk through the property-matrix path, as a tagged chunk already did. The untagged path now also refuses a subnormal, as the tagged one did. NativeGraphBatch: the dead _create_vertices_json_bulk is deleted, and the row type check is a C census. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…ed query (refs #318) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…per property on both wheels (refs #337) JVM: a Java float[] was no vector candidate, so classify_vertex_rows gave up on the first row and the whole load went through _to_java_property_matrix (convert_python_to_java per value, one Object[] per row, one Object[][] per call): 59 to 61 percent of the wall of the e2 and l3d loads in the untimed ingest profile of 26.11.1. A float32 numpy array on an undeclared property took the same way. take_vector_columns (type_conversion) now picks the columns: declared ARRAY_OF_FLOATS properties as before plus Java float[] and double[], and any other property whose value in every row is a float32 array or a Java float[]; the stored value of those is a float[] on both paths. The rest rows are rebuilt from columns by C iterators when the rows share their keys. Native: the float32 arrays of a row went through json.dumps with a Python hook (_gb_default, _default) per row, after a _prep walk of every value, one ["B", offset, count] tag per row parsed back in Java, and np.concatenate of one small array per row (29 to 31 percent prep in the same profile). encode_vertex_args writes the other properties with one json.dumps (no hook when they are plain scalars) and the matrices are the front of the side buffer, a ctypes view with no copy, described by one "vecs" entry that NativeExtras.withVectors turns into one float[] per row. The old per-row tag stays for edge properties and for a column that does not qualify. Tests: both wheels pin the stored bits and the split; the JVM wheel counts the JPype crossings and the property-matrix fallbacks; the native wheel counts the ctypes crossings and the codec's per-row hooks. The existing native test that expected a tag per vertex row now expects the matrix. The Java is not compiled locally: CI builds the native image. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…(no json.dumps __context__); test the stored type across chunks (refs #318) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…ound the whole batch (refs #337) codec._prep is counted by name, so it also counted the calls the batch makes outside create_vertices: creating the batch (extras._enc of its options runs the original _prep, whose recursion goes through the patched name, once per option) and close() (encode_gb_args({}) once). That is 2 calls, matching the Free leg; they do not depend on the rows. The create_vertices path itself calls neither _prep nor a hook (take_vector_columns, _plain_rows and one json.dumps). The first test now takes the hook counts and the crossings around the create_vertices call (gb_default and default exactly 0, prep at most one per crossing), and a new test runs 250 rows in three chunks and 2,500 rows in three chunks with the same bounds: the old path called the hooks per row. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

… it has several chunks (refs #318) A bad scalar in a later chunk used to send only that chunk through the property-matrix path, which stores a Python int as a Long while the JSON chunks store an Integer: one undeclared property got two Java types in one call. An input of more than one chunk now has its columns of exact scalars screened in C over the whole input first (type census, joined text of a str column, min and max of an int column, one float64 array for a float column, in blocks of 100,000 rows); a column that may hold a bad value is judged value by value as before, so a bad scalar anywhere sends the whole input the matrix way, as before #330. One chunk is decided by its own dumps before anything is sent. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…ctor-vertex-loads The two PRs both edit classify_vertex_rows, split_float_vectors and create_vertices. The conflict in type_conversion.py is resolved as the candidate-3 integration branch resolves it, so the source of these two files here is byte for byte the one the candidate-3 JVM wheel ships (the g1 regression is measured on it). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…f candidate 3), vector rows cross in small blocks (refs #337, #318) Measured cause of the 31 percent g1 regression (5,000-row Person load, JVM wheel, candidate 3 against 26.11.1): _dumps_once ran _INT64_OVERFLOW.search over the 330 KB JSON text of every chunk, 13 ms of a 43 ms call (2.5 us a row; the regex starts with a lookbehind and an alternation at every byte, no literal prefix). cProfile: 60 re.Pattern.search calls, 0.29 s per 20 calls. The 26.11.1 per-value loop it replaced cost 1.7 ms. Fix: the columns decide, in C, and the text is not scanned. - _rows_need_walk takes the type census and screens each column of exact scalars (int range by min and max, UTF-8 form of the joined strs); both regexes are gone. - GraphBatch.create_vertices screens the whole input once (screen_vertex_rows, for one chunk or several). When every column is exact scalars and clean the verdict is vetted and the chunks are written by json_bulk_dumps(vetted=True): json.dumps, nothing else (also no subnormal text search). A bad scalar anywhere still sends the whole input the property-matrix way, and the first bad row still gives the old error text (the walk is unchanged and still the reference). - classify_vertex_rows keeps its signature and result; the census and the screen share one list per column. Memory of the vector vertex load: rows with vector columns cross in blocks of about 1 MiB of floats (_VECTOR_BLOCK_FLOATS, at least 512 rows), and the previous chunk's rest rows, text and matrices are released before the next chunk is built (two float[] buffers were alive at once). Python only; the Java copies each row's vector as before. Tests: the regex pattern test becomes a column-screen test over the same corpus; spies that a vetted table runs neither the census nor the walk per chunk and that other columns still do; the block sizes of a vector load; vector_width; the release of the previous chunk. Differential check against the 26.11.1 validation loop (36,000 cases) is in the .notes harness. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…the screen (the verdict is tagged, not None) screen_vertex_rows returns None for a set (a type the tagged JSON does not carry) but a dict with an int key is a tagged shape whose bad key the chunk's walk refuses, as before; the parametrized case was wrong. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

… walk, the column screen does not (fixes CI row4 of test_a_value_that_may_be_bad_runs_the_walk) Row 4 of the parameter list was {"a": "\U0001f600"}. json.dumps writes a non-BMP character as a surrogate PAIR (😀); the removed _SURROGATE_ESCAPE regex took that for a lone surrogate and sent the rows to the walk, which found nothing: a false alarm that the test had recorded as "may be bad". The 26.11.1 loop accepts the value (UTF-8 encodes it) and writes the same text, and the column screen passes it, so no walk runs. Not an unsound screen: on the old and the new tree the end-to-end text is the 26.11.1 text. The row leaves the "runs the walk" list. New: test_a_non_bmp_character_is_no_bad_value_and_is_not_walked (emoji, U+10000, U+10FFFF, mixed text, untagged and tagged: same text as the 26.11.1 loop, no walk; two adjacent lone surrogates are no pair and still refused by the walk), and non-BMP text in the rows of the not-walked table test and in the vetted-table test. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

tae898 and others added 4 commits November 9, 2026 18:58

Tests: bandit annotations for the seeded corpus and the test-controll…

7fddca0

…ed query (refs #318) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

bulk dumps: raise the first bad row's error outside the except block …

4b317ba

…(no json.dumps __context__); test the stored type across chunks (refs #318) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

tae898 mentioned this pull request Nov 9, 2026

Vector vertex loads: the JVM wheel spends about 60 percent of the wall in unwrapped Python when rows hold Java float[], the native wheel about 30 percent in prep #337

Open

tae898 and others added 2 commits November 9, 2026 22:22

tae898 self-assigned this Nov 9, 2026

tae898 and others added 4 commits November 10, 2026 03:57

Navigation Menu

Repository navigation

create_vertices: Java float[] and float32 arrays cross as one matrix per property on both wheels (closes #337) - #339

create_vertices: Java float[] and float32 arrays cross as one matrix per property on both wheels (closes #337)#339
tae898 wants to merge 10 commits into
release/26.11.1from
perf/337-vector-vertex-loads

tae898 commented Nov 9, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

tae898 commented Nov 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What I found (step 1, read from the code at 26.11.1 and at the #330 head)

What changed (step 2)

Tests (step 3)

Profile commands for the post1 A/B (step 5, nothing timed here)

CI (the humemai fork, because the upstream runner queue is backed up)

Update (the post1 glue A/B found a regression and a memory excess in the candidate that contains this PR)

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

tae898 commented Nov 9, 2026 •

edited

Loading

What I found (step 1, read from the code at `26.11.1` and at the #330 head)