Skip to content

JVM GraphBatch: a list, dict or date-time property sends create_vertices and new_edges to per-element Python #243

Open

Description

@tae898

On the JVM wheel, one list, tuple, dict, date-time, date, Decimal or bytes property in a row moves the whole GraphBatch.create_vertices call, and the whole GraphBatch.new_edges call with properties, off the JSON bulk path onto per-element Python.

_create_vertices_json_bulk in src/arcadedb/graph_batch.py returns None as soon as json_bulk_scalar_ok refuses one value, and the chunk goes through _to_java_property_matrix (one JPype conversion per element). new_edges does the same and then buffers every edge with new_edge (one crossing and one RID build per edge). #82 added tagged values to the JSON text for insert_many (json_bulk_dumps_tagged, decoded by DocumentBatcher.untag), so the text can already carry a date-time, date, Decimal and bytes that sit directly in a row, and any nested list or map of JSON values. The graph batches kept the untagged check.

Evidence so far is UNTIMED orientation only: a list property cost 91.4 us per row against 9.3 us for the same rows without it (one run, no timed lock). Scalar-only properties are engine bound (profile of ingest_graph on the JVM wheel: engine 80.7% of the calling thread, Java glue 1.7%), so only these shapes are affected. #496 (#33) covered declared ARRAY_OF_FLOATS vectors only.

Proposed change, in a draft PR: carry those shapes through json_bulk_dumps_tagged, decode them in VertexBatcher and EdgeBatcher with DocumentBatcher.untag (and toMap(), so a nested list or map is stored as a plain List or Map as insert_many does), and keep everything the text cannot carry on the old path: a NaN, an integer beyond 64 bits or a date-time inside a list or map, a subnormal, a non-str dict key, a set, a numpy or pandas value. The stored values must equal what the per-element path stores, types included; the PR's tests compare both.

A timed A/B of the PR (wheel against wheel, since the bridge jar changes) follows once CI has built the wheels.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions