Skip to content

Native point operations: cut the repeated Python work around the one crossing - #230

Merged
tae898 merged 2 commits into
mainfrom
perf/native-point-op-glue
Nov 5, 2026
Merged

tae898 merged 2 commits into
mainfrom
perf/native-point-op-glue

Conversation

@tae898

@tae898 tae898 commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #229

On the arcadedb-jvm build, insert_many now pauses Python's cyclic garbage collector for the duration of a call when rows is a list or a tuple, and restores the previous state in finally (re-enabled only if it was enabled on entry). A generator, an iterator, a DataFrame or any other lazy input is not paused. The native build is not changed.

The change is the arm C of the study python-gc-ingest-ab-laptop-20261025 (HumemAI benchmark notes), adopted by the decision in DECISIONS #238 addendum on TA-2 and #239: the study's rule was written before the first timed run and C passed it. Wheel measured: arcadedb-jvm 26.11.1.dev0 from CI run 41646452646 on main 413f54ffe1; 500,000 dict rows in slices of 100,000, ABBA order, ab.py --noise, 95% CI of the median ratio against the default:

Figure Candidate Clean pairs Median ratio vs default 95% CI
insert_many wall library pause (this change) 12 0.860 [0.819, 0.903]
insert_many wall caller gc.disable() 13 0.854 [0.797, 0.875]
whole load wall (rows built too) library pause (this change) 12 0.876 [0.861, 0.923]
whole load wall (rows built too) caller gc.disable() 13 0.881 [0.847, 0.895]
peak RSS library pause (this change) 12 0.976 [0.924, 1.057]
peak RSS caller gc.disable() 13 0.940 [0.895, 1.076]
insert_many wall A/A control 13 1.080 [0.957, 1.105]
whole load wall A/A control 13 1.044 [0.974, 1.084]

The shipped form follows the risk section of the study README: the guard to a list or a tuple is what keeps a generator that builds reference cycles from piling up garbage for the whole load. gc.disable() is process wide, so with overlapping calls from several threads the pause does not last for the whole of each call, and it is never left stuck off.

Code: Database.insert_many in src/arcadedb/core.py records gc.isenabled(), calls gc.disable() for a list or tuple, runs the unchanged body (moved to _insert_many_nogc) and restores in finally. Docs: the docstring and a new section on the bulk-loads page, with no speed figure. Tests: tests/test_insert_many_gc_pause.py (JVM only, listed in NATIVE_NOT_APPLICABLE) observes gc.isenabled() from inside the call for a list and a tuple, several chunks, the parallel mode, an exception, a collector the caller had disabled, and a generator, an iterator and a generator expression.

CI (test-native.yml dispatched, linux-x86_64 and the other legs): tests only, commit 1d64df8 on the branch perf/native-point-op-glue-tests-first: https://github.com/ArcadeData/arcadedb-python/actions/runs/41715698870 ; with the cuts, head 2c1aa6a: https://github.com/ArcadeData/arcadedb-python/actions/runs/41715725208 . Results are added here when the runs finish (the macOS image build leg gates the test jobs and was still running at the time of writing).

Edited 2026-11-05: the closing reference was #235 (an issue another PR closed); this PR closes #229.

🤖 Generated with Claude Code

@tae898

tae898 commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

Test runs for this PR, both read on 2026-11-04:

Run Branch Code Result
41715725208 perf/native-point-op-glue (head 2c1aa6af83) this PR every job green
41715698870 perf/native-point-op-glue-tests-first (head 1d64df8359) main plus only the new test file test_native_point_op_glue.py: 41 failed, 41 passed; every other file passes (linux-x86_64 py3.12 parity: 2039 passed, 41 failed, 0 errors)

The 41 tests that fail on main are the ones that check the changed path; the 41 that pass on main check behaviour that must not change. The tests-only branch is deleted after this read, and its remaining macOS legs were cancelled to free the shared runners.

@tae898

tae898 commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

Timed A/B (native wheel; arm A = main's CI native wheel, arm B = the same tree with only this PR's files, A/A control; laptop timed lane, tools/ab.py --noise, ABBA, round 0 dropped). Per-operation time, B/A median with 95% CI (below 1 = this PR is faster):

operation role B/A A/A
lookup_by_key gating 0.777 [0.772, 0.784] 1.001 [0.995, 1.007]
out edges of a vertex gating 0.843 [0.836, 0.849] 1.001 [0.996, 1.007]
in edges of a vertex gating 0.842 [0.837, 0.845] 1.001 [0.994, 1.014]
ends of an edge gating 0.893 [0.887, 0.897] 0.999 [0.995, 1.005]
point SELECT (statement path) control 0.998 [0.996, 1.002] 0.999 [0.992, 1.006]
both-direction edges info 0.848 [0.843, 0.849] 1.005 [0.993, 1.009]
lookup_by_rid info 0.991 [0.988, 1.000] 1.006 [0.994, 1.007]

Clean pairs per figure: 12; every run passed its arm check, and the A/B output differential passed before the run. By the rule written before the run this is a SHIP: every gating figure is below 1.00 with the control flat.

The merge waits for the end of the current batch of timed studies that pin main (merging now would move their pins); the PR leaves draft at that point.

@tae898
tae898 marked this pull request as ready for review November 4, 2026 21:42
@tae898
tae898 force-pushed the perf/native-point-op-glue branch from 2c1aa6a to 890555f Compare November 4, 2026 22:58
@tae898 tae898 added this to the 26.11.1 milestone Nov 5, 2026
@tae898 tae898 added native The arcadedb package (native library build) performance Speed or memory of the bindings, with a measurement labels Nov 5, 2026
tae898 and others added 2 commits November 5, 2026 09:50
…dge ends, steps) (refs #229)

Spies on the codec, the cursor object, the row object and the statement text objects, plus parity checks against the general encoder and
the long result path. The spy tests fail on main (the cuts come in the next commit); the parity tests pass on both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…crossing (closes #229)

- encode_params: a lone named map of plain values and RIDs is written by the cached encoder (same text; no _prep, hook closure or encoder object). Every edge end and step sends {rid: RID}.
- lookup_by_key: JSON by encode_lookup (plain values take the cached encoder, no per-call import of extras), and the first row of a reply that holds the whole result is read without building a NativeResultSet.
- Edge.get_in/get_out and the steps build their statement text once (per function, per label list, bounded), a step reads its rows as (props, meta) pairs instead of a NativeResult per row, and a record built from a row nobody else holds takes the property dict as it is.
- lookup_by_rid keeps its text (test_native_statement_texts pins it); it only skips the property copy.

Results, statement texts and errors are unchanged; the crossings per call are unchanged (spied in the new tests).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tae898
tae898 force-pushed the perf/native-point-op-glue branch from 890555f to 60ac1a8 Compare November 5, 2026 00:50
@tae898
tae898 merged commit fc1ff19 into main Nov 5, 2026
38 checks passed
@tae898
tae898 deleted the perf/native-point-op-glue branch November 5, 2026 02:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

native The arcadedb package (native library build) performance Speed or memory of the bindings, with a measurement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native point operations: cut the repeated Python work around the one crossing

1 participant