Skip to content

Native: record ids go to the engine as a parameter, not in the statement text (N-09) - #529

Closed
tae898 wants to merge 1 commit into
native-previewfrom
perf/native-rid-param
Closed

tae898 wants to merge 1 commit into
native-previewfrom
perf/native-rid-param

Conversation

@tae898

@tae898 tae898 commented Oct 25, 2026 •

Copy link
Copy Markdown

Row N-09 of the bindings backlog: on the native wheel, a record id was pasted into the SQL text, so each traversal step and each lookup_by_rid was a new statement text. The id now goes over as a parameter (the RID the codec already tags as R).

Statement cache finding: the claim holds. The engine's parsed-statement cache is StatementCache over SQLLiteralParameterizer/LiteralParameterizer, a SegmentedLRUCache<String, Lookup> keyed by the statement text (LiteralParameterizer.java:65, :134), sized by arcadedb.sqlStatementCache, default 300 (GlobalConfiguration.java:765). The literal extraction that lets texts share an entry only takes integer, floating point and non-empty string tokens (SQLLiteralParameterizer.java:112, isCandidate) and keeps every literal in a FROM target (class comment and classify), so SELECT FROM #1:0 and SELECT FROM #1:1 are two entries. The native image reaches the same path: ArcadeNative.java:258-262 calls db.query and db.command, which end in SQLQueryEngine.parseParameterized (SQLQueryEngine.java:285). Checked on ArcadeData/arcadedb main at 26f0bf8. So 25 records meant 25 texts, and a loop over more than 300 records evicts the statements the application re-runs.

What changed (Python only, no image rebuild): NativeDatabase.lookup_by_rid (SELECT FROM ?), and in records.py the Vertex.get_out_edges/get_in_edges/get_both_edges, Vertex.new_edge, Edge.get_in/get_out, save() (UPDATE :rid MERGE :row) and delete() (DELETE FROM :rid). The id validation and the error messages are unchanged, and a text without the leading # is still accepted as before. Docs: the Parameters section of guide/core/queries.md and lookup_by_rid in api/database.md.

Untimed counts (test_native_statement_texts.py, 25 records): lookup_by_rid gave 25 distinct texts before and gives 1 after (the test failed on exactly this before the change); get_out_edges("Knows"), get_in_edges() and get_both_edges() over the 24 linked vertices are 72 calls and now 3 texts, Edge.get_in/get_out over 24 edges are 48 calls and now 2 texts, save() and delete() are one text each, and new_edge is two texts (with and without CONTENT) per label instead of one per pair of records. Before the change every one of these carried the record id in its text (the second test failed on CREATE EDGE Likes FROM #1:0 TO #1:1). No timing was run.

Local checks, all against the image of native-lib/nnext (the engine of the current native-preview) with ARCADEDB_BACKEND=native: the two new tests fail before the change and pass after; test_native_statement_texts.py, test_graph_api.py, test_graph.py, test_core.py, test_resultset.py, test_graph_batch.py, test_schema.py, test_cypher.py, test_parameter_binding.py, test_restore_sql.py: 228 passed, 26 skipped (the NOTYET skips), 0 failed. The new behaviour tests in test_graph_api.py (traversal over a 12-vertex chain, save and delete by id, new_edge with and without properties, the refusals of lookup_by_rid) pass on the JPype wheel too: test_graph.py, test_core.py, test_graph_api.py, 64 passed. Pre-commit (black, isort) ran and its changes are in the commit.

Overlap with the open native PRs, checked with git merge-tree against each head: independent of #520, #521, #523, #525 (which also edits records.py, database.py and codec.py but in other hunks) and #526. Two adjacent-line conflicts, no semantic overlap: #514 in the import block of _native/database.py (it adds from .bulk import insert_rows_columnar next to my from .codec import RID) and #522 in tests/conftest.py (both add a collect_ignore.append line at the same place). Whichever merges second needs a trivial resolution; I did not stack on any of them.

🤖 Generated with Claude Code

…ent text (N-09)

lookup_by_rid, Vertex.get_out_edges/get_in_edges/get_both_edges, new_edge, Edge.get_in/get_out, save and delete pasted the record id into the SQL text. The engine keeps parsed statements in a cache keyed by statement text (default 300 entries) and leaves a record id in a FROM target in the text, so each record was a new text to parse and a new cache entry. The id now goes as a RID parameter (the "R" tag the library already decodes), so one operation has one text. Results and error messages are unchanged.

Tests: behaviour tests in test_graph_api.py run on both backends; test_native_statement_texts.py (native only) counts the statement texts for 25 records.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@tae898

tae898 commented Oct 26, 2026

Copy link
Copy Markdown
Author

Closing here: the bindings moved to ArcadeData/arcadedb-python. This change has no counterpart there yet and no queued A/B. If it is wanted, it goes there as its own PR: it is now a line in the tracking issue ArcadeData/arcadedb-python#19. The branch stays.

@tae898 tae898 closed this Oct 26, 2026
tae898 added a commit to ArcadeData/arcadedb-python that referenced this pull request Nov 3, 2026
…ent text (#202)

* Native: record ids go to the engine as a parameter, not in the statement text

Port of humemai/arcadedb-embedded-python#529 (branch perf/native-rid-param, never merged there) onto current main.

On the native build a record id was pasted into the SQL text, so each traversal step and each lookup_by_rid was a new statement text. The engine caches a parsed statement by its text (arcadedb.sqlStatementCache, 300 entries) and its literal parameterizer keeps a literal in a FROM target, so 25 records meant 25 cache entries. The id now goes over as a parameter (the RID the codec already tags as R): lookup_by_rid (SELECT FROM ?), Vertex.get_out_edges/get_in_edges/get_both_edges, Vertex.new_edge, Edge.get_in/get_out, save() (UPDATE :rid MERGE :row) and delete() (DELETE FROM :rid). Id validation and error messages are unchanged.

test_native_statement_texts.py counts the distinct statement texts (native only, not collected on JPype); the new behaviour tests in test_graph_api.py run on both builds. Untimed counts only: a timed A/B decides it before merge.

Refs #19

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Native: keep lookup_by_rid's literal id; the parameter path measured 2 % slower there

First timed A/B (ab.py, laptop, native wheel of main vs this PR, 12 clean pairs): lookup_rid_us 1.022 [1.019, 1.025] (slower), out_edges_us 0.792 [0.790, 0.794], edge_ends_us 0.792 [0.789, 0.794], save_us 0.805 [0.801, 0.809] (faster). Guards flat, A/A clean. Narrow the PR to the paths that got faster; lookup_by_rid returns to main's text. database.py is now identical to main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@tae898
tae898 deleted the perf/native-rid-param branch November 8, 2026 11:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant