Skip to content

JVM readers: a record or row to dict, the property names and result-set iteration in one crossing each - #233

Merged
tae898 merged 3 commits into
mainfrom
perf/jvm-reader-crossings
Nov 5, 2026
Merged

tae898 merged 3 commits into
mainfrom
perf/jvm-reader-crossings

Conversation

@tae898

@tae898 tae898 commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #227

The JVM build read a record, a row, a list of property names and each row of a result set through many JPype calls: Document.to_dict() took 18 reads of Java objects for 3 properties, Result.to_dict() 14, the property names 12, and for row in rs two calls per row (counts of an untimed census, see the issue; no timing yet). A new bridge class, ReaderCalls (src/java/com/arcadedb/python/ReaderCalls.java, compiled with the other bridge classes into the same jar), cuts each to one call:

  • Result.to_dict() (results.py) and Document.to_dict() (graph.py): ReaderCalls.rowJson / documentJson write the row or record as the typed JSON of TypedRows (the transport ResultSet.to_list() already uses). Python makes one call, reads one array element and one string, and parses with the C json module; the side array of the engine's own objects is read only when the JSON carries the tag character. Values have the types convert_java_to_python gives them, and the key order is the engine's.
  • get_property_names() and property_names of both classes (_property_names_tuple): rowNames / documentNames return the names as one JSON text instead of one JPype read per name.
  • ResultSet.__next__ (results.py): nextOrNull does hasNext() and next() in one call. A row is still read when it is asked for, an error in the statement surfaces from the call as before, the end is the same None as hasNext() false, and the closed and exhausted states are unchanged.

Without the class (a jar that lacks it) every caller takes the old path. The native build is not touched. TypedRows.appendRow and TypedRows.quote lose private so the new class can use them.

One difference on purpose: Document.to_dict() now stores the names of the read it just made, so a property set after an earlier to_dict() or get_property_names() is no longer missing from the next to_dict() (the cached tuple of names was never refreshed on a mutable record). A record whose wrapped Java object is not a com.arcadedb.database.Document (a binary record) now raises a TypeError from the bridge call where it raised AttributeError for the missing getPropertyNames.

Tests (tests/test_reader_crossings.py, JVM only, listed in NATIVE_NOT_APPLICABLE): a counting proxy on the bridge classes and the Java objects, which also counts the reads of the arrays and the Java set they return, pins the exact calls (to_dict is the call and one array read; the old path is at least 7 for a record and 9 for a row; iteration is one call per row against two), and every value type of the wide table, the values SQL cannot produce, tag-named properties and unusual property names are compared with an independent per-property read of the Java object (same Python types, same key order). Closed-database errors, laziness of iteration and the close semantics are pinned too.

Overlap with #214 (issue #213): #214 edits RowAccess.java, DbCalls.java, results.py (ResultSet.one, ResultSet.count), graph.py (Document.get_raw) and core.py. This PR adds its Java in a new file, edits ResultSet.__next__, Result._property_names_tuple, Result.to_dict, Document._property_names_tuple and Document.to_dict, and one line of tests/conftest.py (next to the line #214 adds). The hunks do not touch, so the two merge in either order; the only shared line region is the NATIVE_NOT_APPLICABLE table in tests/conftest.py, which may need a trivial rebase.

Status: the pre-registered timed A/B of the main CI JVM wheel against this PR's CI JVM wheel shipped by its rule. Figures and scope are in this comment and in the docs passage this PR now adds.

Docs: the Results section of how-to/databases-and-queries.adoc has the four timed figures (Document.to_dict at 3 and 12 properties, iterating a result into dicts at 100 and 10,000 rows) with their 95% intervals and the control, added in 7cacef9.

🤖 Generated with Claude Code

Edited 2026-11-05: the status line now gives the timed result; the PR is ready for review.

@tae898

tae898 commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

CI on head 3403dd2: workflow_dispatch run 41716185993 of test-jvm.yml, 18 of 18 jobs succeeded, including all 10 test legs (the new tests/test_reader_crossings.py ran on each). No timed run exists yet; the PR stays draft. The timed A/B is prepared with its rule and the GRAY rerun clause written before any run.

@tae898

tae898 commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

Timed A/B of this PR on the JVM wheel (laptop timed lane, tools/ab.py --noise, ABBA, round 0 dropped, an A/A control). Arm A = main's CI JVM wheel 1b315c1318 (run 41716288781), arm B = this PR's CI JVM wheel at 3403dd2764 (run 41716185993). B/A median with its 95% interval (below 1 = this PR is faster):

workload B/A median [95% CI] A/A median [95% CI]
record to dict, 1,000 records of 3 properties 0.307 [0.303, 0.311] 0.997 [0.964, 1.002]
record to dict, 1,000 records of 12 properties 0.133 [0.132, 0.135] 1.000 [0.991, 1.005]
iterate a result into dicts, 100 rows 0.657 [0.649, 0.664] 0.995 [0.976, 1.014]
iterate a result into dicts, 10,000 rows 0.474 [0.454, 0.480] 1.001 [0.992, 1.005]

Clean pairs per figure: 12. By the rule written before the run (every gating upper end below 0.90, every control median inside [0.95, 1.05], a clean A/A) this is a SHIP. The control times a pure-Python json.dumps of rows fetched before the timer (no query, no JPype call, no code this PR changes); its B/A medians are 0.996, 0.987, 0.993, 1.000 on the four workloads.

This is the third run. The first two were invalid by the same rule and report nothing: their control was to_json_list over the same rows, whose timed region includes the query, and its A/A did not settle. The new control was declared in the study before this run started. to_json_list is still measured and reported (it never decides); on the 3-property records its B/A is 1.046 [1.012, 1.067].

The PR leaves draft at the end of the current batch of timed studies that pin main, with its docs, and merges on green. I can re-run the study on the branch after any change.

@tae898
tae898 marked this pull request as ready for review November 4, 2026 21:47
@tae898
tae898 force-pushed the perf/jvm-reader-crossings branch from 7cacef9 to 64efea3 Compare November 4, 2026 22:57
@tae898 tae898 added this to the 26.11.1 milestone Nov 5, 2026
@tae898 tae898 added jvm The arcadedb-jvm package (JPype build) performance Speed or memory of the bindings, with a measurement labels Nov 5, 2026
@tae898
tae898 force-pushed the perf/jvm-reader-crossings branch from 64efea3 to c896a38 Compare November 5, 2026 01:29
tae898 added a commit that referenced this pull request Nov 5, 2026
…e skip list

#214 and #233 each add one NATIVE_NOT_APPLICABLE entry at the same place, so merging one made the other conflict and cost a full CI run per rebase. Both branches now carry the same two entries in the same order; an entry for a test file that is not on this branch is only a collect_ignore name and changes nothing here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tae898 and others added 3 commits November 5, 2026 11:56
…et iteration in one crossing each (#227)

Document.to_dict took 18 reads of Java objects for 3 properties, Result.to_dict 14, a
property-names call 12, and each row of a result set two calls (hasNext, next). The new
bridge class ReaderCalls writes a record or a row as the typed JSON of TypedRows (one call,
one string read, the C json decoder), returns the names as one JSON text, and hands over
the next row in one call. Values keep the types convert_java_to_python gives them; the side
array is read only when the JSON refers to it; a result set is still read one row per call.
Without the class the old paths run. Document.to_dict now also refreshes the cached names,
so a property set after an earlier read is no longer missing from get_property_names().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lt iteration)

The study that adopts this PR timed record to dict at 3 and 12 properties and iteration into dicts at 100 and 10,000 rows on the arcadedb-jvm package. The Results section gets those four gating figures with their 95% intervals, the unit (time per record or row), the control that the change does not touch, and the scope (JVM build, these workloads, one laptop). The numbers are generated from the study's ab-score.md by script.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e skip list

#214 and #233 each add one NATIVE_NOT_APPLICABLE entry at the same place, so merging one made the other conflict and cost a full CI run per rebase. Both branches now carry the same two entries in the same order; an entry for a test file that is not on this branch is only a collect_ignore name and changes nothing here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tae898
tae898 force-pushed the perf/jvm-reader-crossings branch from c896a38 to cad8e65 Compare November 5, 2026 02:56
tae898 added a commit that referenced this pull request Nov 5, 2026
…by_key, transaction mark) (#214)

* Document.get_raw: one crossing and one scan of the record instead of has() then get()

get() answers null for a property the record does not have, as it does for one set to null; Python reads None either way, so the has() call only repeated the scan. has_property() still tells the two apart.

test_read_crossings.py counts the calls into the JVM on a proxy of the Java object and pins that the answers are the plain path's (absent, null-valued, mutable, vertex and edge). It is JVM-only (NATIVE_NOT_APPLICABLE); the native backend is unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* ResultSet.one() and count(): one crossing through RowAccess, like first()

one() read hasNext()/next() per row and closed afterwards; count() crossed twice per row. RowAccess.upToTwoAndClose and countAndClose do the same work Java-side with the same errors and the same result-set state; the plain path stays for a missing bridge class.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* lookup_by_key: the key names and values cross as scalars or varargs, no Java arrays

DbCalls.lookupFirstKey takes one key as scalars and lookupFirstKeys takes any number as the varargs of one call, so neither side builds a String[] and an Object[] through JPype per lookup. The engine receives the same arrays, and a different number of names and values reaches it as before. lookupFirst, which takes the arrays, stays.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Transaction-stack mark in one crossing

The mark taken before every begin() (with db.transaction(), run_in_transaction) asked isTransactionActive() and getNestedTransactions() separately. DbCalls.stackMark returns both as one number, so with db.transaction() goes from 4 crossings to 3 and run_in_transaction from 6 to 5. The mark is still taken before begin(), so the interrupt and refused-begin rules of #9970 and #9813 are unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Docs: the concurrency figures of the JVM read-crossing cuts (mixed window at 2 Python threads)

The study that adopts this PR is a mixed read/write window at 2 Python threads on the arcadedb-jvm package. The page gets the gating figure with its 95% interval, the 1-thread guard, the reader and writer shares and the Cypher control, all generated from the study's ab-score.md by script, and says what was not measured (4 or more threads, native, other hardware). The one-thread row is stated plainly: the gain is not specific to concurrency.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* tests: list the read- and reader-crossing tests together in the native skip list

#214 and #233 each add one NATIVE_NOT_APPLICABLE entry at the same place, so merging one made the other conflict and cost a full CI run per rebase. Both branches now carry the same two entries in the same order; an entry for a test file that is not on this branch is only a collect_ignore name and changes nothing here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
@tae898
tae898 merged commit 53c4213 into main Nov 5, 2026
41 checks passed
@tae898
tae898 deleted the perf/jvm-reader-crossings branch November 5, 2026 04:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jvm The arcadedb-jvm package (JPype build) performance Speed or memory of the bindings, with a measurement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JVM wheel: record and row readers cross into the JVM many times (to_dict, property names, result-set iteration)

1 participant