Repository navigation
Native point reads: shorter call path (one-parameter text, flat exec call, first row, record wrap) - #525
Closed
tae898 wants to merge 3 commits into
Closed
Native point reads: shorter call path (one-parameter text, flat exec call, first row, record wrap)#525tae898 wants to merge 3 commits into
tae898 wants to merge 3 commits into
Conversation
…call, first row, record wrap) The untimed decomposition of the native point read (bindings-profile-20261025 and native-point-call-ab-laptop-20261025/decomp.out) puts about 15 of the 34 us of db.query(...).first() and 21 of the 53 us of lookup_by_key in Python outside the engine. The ctypes crossing itself is not the cost (the call minus the image's own timers is 0.6 us), so this change leaves the signatures alone and takes the measurable pieces of the Python side: encode_params writes a lone int or str parameter directly (the text json.dumps gives), lib.exec_query is one function for arcadedb_n_exec (no getattr by name, no argument tuple, cached bytes for the language and the statement), NativeResultSet.first() takes a buffered first row without the iterator protocol, lookup_by_key remembers its statement text per (type, keys), and the record wrapper no longer imports a module or makes an object per row. New tests first: 6 of them fail on native-preview and all pass here. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This was referenced Oct 25, 2026
The native lookup_by_key ran SELECT ... WHERE k = ? LIMIT 1, which works without an index and scans the whole type. The JVM engine refuses (IllegalArgumentException: No index has been created on type ... properties [...]), and the docs call the method an index lookup. Native now reads schema:indexes once per (type, keys) pair and database handle and raises ArcadeDBError with the engine's wording unless an LSM_TREE or HASH index covers exactly those properties in that order, on the type or a supertype (a FULL_TEXT index gets the engine's "No key index" text). The result is remembered per handle next to the statement text, so a repeated lookup is still one crossing; DROP/ALTER (and any sqlscript) through command() clears it. A missing type keeps the schema error it had. Tests: the point-call test creates the composite index its composite lookup needs; new parity tests (both wheels) cover keys no index covers, order and prefix of a composite, a missing type, another database with the same type name, DROP INDEX, a supertype index and a FULL_TEXT index; a native-only test pins the single schema read per pair. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…collected on JPype test_native_point_call.py kept the tests of the native call helpers (crossing counts, the schema-read cache, the parameter codec) next to the point-read tests both wheels run, and marked them skipif. On the JPype legs that was 33 skips, and scripts/check_test_skips.py fails the job on any skip. The native-only tests move to test_native_point_call_native.py, which conftest.py does not collect on the JPype backend (as for test_native_startup_runtime.py). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Author
|
Closing here: the bindings moved to ArcadeData/arcadedb-python. The branch stays for the queued A/B native-point-call-ab-laptop-20261025; if it passes, the change is ported there. |
tae898
added a commit
to ArcadeData/arcadedb-python
that referenced
this pull request
Oct 30, 2026
#147) Port of humemai/arcadedb-embedded-python#525 (closed when the bindings moved here) onto today's _native code. An untimed per-call decomposition on the laptop put about 15 of 34 us of `SELECT .. WHERE id = ? .first()` in Python outside the engine. What changes, behaviour identical: - codec.encode_params: one exact-int or exact-str parameter is written as '{"pos": [v]}' directly (the bytes json.dumps gives) without the set of types and the encoder call; bool, int/str subclasses, floats and everything else keep the general path. - lib.exec_query: arcadedb_n_exec for a text-only parameter list without the by-name getattr, the thread-handle call and the argument tuple; same reply reader as call_bin (a float32 tail still decodes). database._exec uses it, with cached language and statement bytes. - NativeResultSet.first(): when the first batch holds the row and the result is readable, take it and close, without the iterator protocol. Same state afterwards as the iterator path (closed, not marked drained). - Record wrap: the import in NativeResult._record and records._java_object is done once, and _wrap looks the class up in a module dict. Dropped from #525: the lookup_by_key statement cache (lookup_by_key now calls the engine's lookupByKey through arcadedb_n_lookup; it still gains from the first() change), and the shared Java-object stand-in per process (each record keeps its own, only the import is cached). Untimed numbers from the old study (us per call, whole call incl. two get() calls): SELECT .. WHERE id = ? .first() 34.36 -> 29.60; encode_params 2.39 -> 0.58, json_loads 4.86 -> 4.49 (unchanged by this, see the follow-up note on a binary first batch). These are a decomposition on the previous code, not a speed claim: the timed A/B on the current wheel is still to run. tests/test_native_point_read.py (native backend only, collected in conftest): the one-parameter text equals json.dumps and the general encoder for 13 int/str values and 17 other kinds, the point-read rows equal the iterator and to_list() rows (float32 tail, escapes, multi-byte text), first() state and errors, record wrapping per kind, lookup_by_key, and a spy that the short path is the one taken. The two spy tests fail on the previous code, the others pass on both. Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The native wheel's point reads cost 49.6 us per
lookup_by_keyand 33.7 us perSELECT ... WHERE id = ?plus.first()in the untimed profile (1M-document database, 100k calls each), against 20.1 and 17.5 us on the JVM wheel. The image's own timers account for about 55% of that wall; this change takes the Python side that was measurable and leaves every signature, result and error as it was.encode_paramswrites a lone int or str parameter directly (byte for byte the textjson.dumpsgives; bool, subclasses and numpy scalars keep the general path),lib.exec_queryis one function forarcadedb_n_exec(no getattr by name, no argument tuple, cached bytes for language and statement),NativeResultSet.first()takes a buffered first row without the iterator protocol,lookup_by_keyremembers its statement text per (type, keys), and the record wrapper no longer imports a module and makes a stand-in object per row. It does not touchcall_bin,_fill,loadsor the Java side, so it does not overlap #520 or #521.What was measured, untimed (one run each on the dev1 image, perf_counter_ns around each step on cores 8-15, the timer read subtracted; a decomposition, not a speed claim), microseconds per call before and after, from
repros/native-point-call-ab-laptop-20261025/decomp.outanddecomp-after.outin the notes repository:The ctypes crossing itself is not the cost: the call minus the image's own timers is 0.6 us. The largest piece left on the Python side is the reply decode (
json_loads, 4.9 and 5.7 us), which belongs to the result-transport work, and I left it alone.What was not measured: any timed speed. The timed A/B is queued as
native-point-call-ab-laptop-20261025(arm A the published dev1 wheel, arm B dev1 plus these four Python files, arm Z the A/A control; ab.py with noise log, ABBA, cores 6,7); its rule is written before any timed run: ship only if the upper end of the 95% CI of B/A is below 0.95 for both point-read figures, no figure is slower (lower CI end above 1.03), and the A/A control is clean. Until it reports, nothing here is a speed claim.Tests first:
bindings/python/tests/test_native_point_call.py(38 tests, 32 behaviour tests that pass before and after on both backends where they apply, and 6 helper tests that fail on native-preview and pass here). With the dev1 image the new file plustest_resultset,test_results_after_close,test_graph_api,test_graphandtest_vectorgive 116 passed, 24 skipped; a wider run oftest_core,test_parameter_binding,test_percall_calls,test_typed_rows,test_bulk_insertandtest_cypherbefore the last comment-only edits gave 255 passed, 43 skipped, 1 failed (a test of mine that was wrong, fixed).Parity fix found by this PR's CI (commit b2d6481). The JVM leg failed
test_lookup_by_key_returns_the_record_for_int_str_and_composite_keyswithArcadeDBError: Failed to lookup by key in 'P': java.lang.IllegalArgumentException: No index has been created on type 'P' properties [id, name]: the test looked up the composite key["id", "name"]while_docs()created only single-property indexes. The test was wrong, and the JVM wheel was right.lookup_by_keyis documented as an index lookup (docs/api/database.md) and the engine refuses keys no index covers. The native wheel ranSELECT FROM type WHERE k = ? AND ... LIMIT 1, which works without an index and silently scans the whole type, so the same test passed there for the wrong reason._docs()now createsCREATE INDEX ON P (id, name) UNIQUE, and nativelookup_by_keymakes the engine's check: on the first lookup of a (type, keys) pair on a database handle it readsschema:indexesfor the type once and raisesArcadeDBError("Failed to lookup by key in 'P': java.lang.IllegalArgumentException: No index has been created on type 'P' properties [id, name]")unless anLSM_TREEorHASHindex covers exactly those properties in that order, on the type or a supertype (aFULL_TEXTindex on the keys gets the engine'sNo key index has been created ...text; a type that does not exist keeps its schema error). A pair that passed is remembered on the handle next to the statement text, so every later lookup is still the one crossing this PR measured; the remembered set is per handle (not in the module-wide_LOOKUP_SQLtext cache, because whether an index exists is a fact about one database) andDROPorALTERthroughcommand(), and any sqlscript, clears it. Limit: an index dropped by another connection to the same database is not noticed until the handle's next DDL; that falls back to the old unchecked statement, never to a false error, and only refusals are never cached.Tests for the parity fix:
test_native_point_call.pyis now 43 tests. Four new ones run on both wheels (keys no index covers: a property with no index, the composite in the wrong order, a prefix of the composite, a superset; three calls each; a missing type; another database with a type of the same name;DROP INDEXthen a re-created index; an index on a supertype and aFULL_TEXTone) and one is native-only (exactly oneschema:indexesread per pair, and 2 x 5 lookups plus 2 reads in total). On the unfixed native source the four parity tests fail (DID NOT RAISE ArcadeDBError) and the cache test fails; on the fixed source all pass. Run with the nnext image: this file plustest_graph_api,test_conversion_fixes,test_percall_calls,test_vector,test_graph,test_core,test_resultsetandtest_results_after_closegive 198 passed, 46 skipped. On the JVM main wheel (78e97b1, Python 3.12): this file gives 10 passed, 33 skipped (the skips are the native-only helper tests), and the same wider set gives 208 passed, 35 skipped, 1 failed:test_vector.py::TestLSMVectorIndex::test_lsm_vector_search_uses_database_rid_conversion, a file this PR does not touch and a code path (_native) the JVM wheel does not load.🤖 Generated with Claude Code