[tae898]

Closes #172

Native save() of a record object overwrote a record that another transaction changed after the object was read (a lost update). The JVM build refuses it with ConcurrentModificationException. The engine keeps no record version, so the native build now compares the values the record held as read with the values stored at save() time, in the transaction of the update, and raises the same error (retryable by run_in_transaction) when they differ. Nothing is written in that case.

The fingerprint is lazy. A first design fingerprinted the stored bytes in Java for every record row a read returned, and a timed A/B (rule written before the run) showed native reads 5 to 8 percent slower (for example a 10k-row SELECT, ratio 95% CI [1.075, 1.079]; a lookup by RID [1.063, 1.075]), see the comment on #172. This head does no fingerprint work on any read path. It takes the fingerprint at modify(), at the first set() of a record that is already saved, or at save(), from the values the record holds then, which are the read-time snapshot, and caches it on the record. After a successful save it moves to the stored values (UPDATE .. MERGE .. RETURN AFTER), so an object's own earlier save is not another transaction's change. A record that is read and never saved pays nothing. There is no Java change.

Parity with the JVM package: save() of an existing record is a write, so it needs an active transaction on both builds. Outside one it raises ArcadeDBError ("Transaction not begun"), as on main and as Document.save() does through the engine; an earlier head of this PR opened and committed a transaction of its own when none was active, which made native accept what the JVM package refuses, and that is removed. The native build tracks set_auto_transaction() (the library has no getter), so with auto-transaction on, the engine's own per-statement transactions apply as before.

The cost moves to the write: save() of an existing record now runs one SELECT before and gets the stored record back from the UPDATE (one extra crossing). Reads are unchanged by design.

Tests: test_save_of_an_existing_record_outside_a_transaction_is_refused in tests/test_graph_api.py runs on both builds (it raises outside a transaction, leaves no transaction behind, writes nothing, and the same object saves inside one). The four refusal tests in tests/test_graph_api.py run on both builds. tests/test_native_save_lazy_fingerprint.py (native only) counts calls through the records._fingerprint seam: no read path (query rows, lookup_by_rid, get_element, edges, Cypher) calls it; modify() calls it once and caches it; a modify-then-save after a concurrent change still raises, also when modify() runs after the change; a record created by the object and changed elsewhere is refused. The new refusal tests fail on main (DID NOT RAISE) and pass here. I ran them and test_graph.py, test_core.py, test_concurrency.py, test_cross_model_atomicity.py and test_bridge_fixes.py locally against a native dev wheel with this records.py on top (51 passed, 1 skipped, plus 25 passed in the first two files).

Fast fingerprint: the first version of the fingerprint built a canonical copy of the record (nested lists) and hashed its repr, about 390 us for a record with 20 properties and a 768-float vector, taken three times per save() (at modify(), on the re-read stored row, on the RETURN AFTER row). It now writes a type tag and a fixed-width or length-prefixed form of each value straight into one bytearray (a list of floats or of ints goes in one array call) and hashes that once, about 23 us for the same record (17 times less, measured with a script that also checks that every pair of a 60-value corpus is equal or different exactly as before, and that 3000 random nested records give one fingerprint in any dict order). What the guard detects is unchanged: it still fingerprints the whole record, the token still comes from the RETURN AFTER row (what the engine stored), and it is taken at the same places. Equal values still give equal fingerprints whatever the dict or set order, a tuple is a list, and 1, 1.0 and True stay three values. One class is wider on purpose: a float ndarray equals the list of its doubles. New tests in tests/test_native_save_lazy_fingerprint.py: one medium-like record has one fingerprint through lookup_by_rid, a SELECT and the RETURN AFTER row, and any one changed property or vector element changes it. The timed A/B on this head is still to be run.

Draft: the same timed A/B (ab.py --noise, native wheel, interleaved pairs against main) runs on this head before it leaves draft, with the same pre-written rule.

🤖 Generated with Claude Code

[tae898]

A finding from preparing the timed A/B of this head (7008eb4), before it runs: with this PR, native save() of an existing record works outside a transaction, because _save_update opens and commits its own when none is active (own = not db.is_transaction_active(), then db.begin()). On main, native save() outside a transaction raises Transaction not begun, and so does the JVM package (Document.save() calls the engine's MutableDocument.save(), which refuses a write without an active transaction while autoTransaction is false, the engine default). So this head makes the two backends behave differently, which the bindings should not do.

Plan: drop the own-begin branch so that native and JVM both raise outside a transaction, and add a test that checks it on both backends. I will push that after the timed A/B of 7008eb4 finishes. The A/B wraps every save in an explicit transaction on all arms (as users on main must do today), so the own-begin path is not part of what it times, and its figures carry over to the fixed head.

[tae898]

Timed A/B of this PR's head 3492acb against main 87ecaf0 (laptop timed lane, ab.py --noise, ABBA pairs, an A/A control, the rule written in the study before the run). Every save in all arms runs inside an explicit with db.transaction():, so the own-begin path that this head removed is not part of it.

Result by that rule: VERDICT: reads **SHIP**; save path **GRAY** (save_medium_us: median 5.882, CI [5.86, 5.938]). Result: ok (all 16 B/A figures have >= 10 clean pairs and CI width <= 0.08, the A/A control is clean).

Reads: the lazy fingerprint costs nothing measurable. All 12 read figures and both controls are within 1.006x of main (largest median ratio), so the read side can ship as it is.

The save path is where it costs:

figure	clean pairs	median ratio PR / main	95% CI	A/A median (control)
save_small_us	14	1.273	[1.268, 1.276]	1.000
save_medium_us	14	5.882	[5.860, 5.938]	0.998

Each operation is lookup_by_rid(rid).modify(), one set("name", ...), then save(). The small documents have a handful of scalar properties; the medium ones have 20 properties plus a 768-float vector. The overhead grows with the size of the record, which points at the save-time work on the whole record: the extra read of the stored row and the fingerprint over every value, the vector included, while the change itself is one string.

So the PR stays a draft. Options I see, cheapest first, before measuring again:

• compare only what the save writes: check the properties being set (and the ones read through modify()) instead of a fingerprint of the whole record, for example as a conditional UPDATE ... MERGE ... WHERE on the old values of those properties, which also saves the separate read;
• keep the whole-record check but leave large arrays (vectors) out of the fingerprint, or hash their bytes once at read time;
• move the fingerprint into the native library, so the comparison does not go through Python values.

Of these, the first would make native weaker than JVM: the JVM package's save() is guarded by the engine's MVCC, which refuses the commit on any concurrent change to the record, not only to the properties being written. So the check stays whole-record, and I will try the cheap ways to keep it so (the third option, then the second) and rerun this A/B on them. If someone prefers another contract for a lost-update guard, say so before I build it.

Edited: the last paragraph first said I would try the first option; it now keeps the whole-record check for parity with JVM.

[tae898]

Edit to my comment above: I first wrote that I would try comparing only the properties being saved. That would make native weaker than JVM, whose save() the engine refuses on any concurrent change to the record, so the next try keeps the whole-record check and makes it cheaper (fingerprint in the native library, or vectors hashed once at read time).

[tae898]

Where the medium save goes, from an untimed phase profile of the same operation on the same medium database (2,000 saves per arm, medians in microseconds, alone on 2 cores; the timer overhead is the difference between the unwrapped and the wrapped loop):

phase	main	this PR
whole operation (no timers)	279.9	2089
lookup_by_rid	66.95	74.98
modify()	1.223	559.7
of which the fingerprint	-	556.6
save()	139	1357
re-read of the stored row	-	46.57
fingerprint of the stored row	-	544.2
UPDATE ... MERGE	136.7	164.1
decode of the RETURN AFTER row	-	40.66
fingerprint of the RETURN AFTER row	-	539.6
commit	73.66	79.37

The three fingerprints take 1640 of the 2089 us (79 %). One fingerprint of a medium record splits into canonicalising the values (271.8 us), repr of the result (236.2 us) and the hash itself (17.73 us), so it is the Python canonicalisation and repr of the 768 floats, not the hash and not the extra read (46.57 us).

Plan, keeping the whole-record check: build the fingerprint from the value bytes (floats and vectors packed, not repr), and take the token after the save from the values just written instead of decoding and hashing the RETURN AFTER row. Then the same timed A/B again.

[tae898]

Third timed A/B: this PR's head 4023a03 (with the byte-encoded fingerprint) against main 8031fe7, same harness, protocol and rule as the second run (laptop timed lane, ab.py --noise, ABBA pairs, an A/A control). Every save runs inside an explicit with db.transaction():.

Result by the rule: VERDICT: reads **SHIP**; save path **GRAY** (save_medium_us: median 1.649, CI [1.635, 1.662]).

Reads: unchanged. All 14 read figures and controls are within 1.013x of main (largest median ratio, expand2_medium_us).

Save path, against the second run:

figure	second run, median [95% CI]	this run, median [95% CI]	clean pairs (this run)	A/A median (this run)
save_small_us	1.273 [1.268, 1.276]	1.306 [1.305, 1.309]	12	1.000
save_medium_us	5.882 [5.860, 5.938]	1.649 [1.635, 1.662]	12	0.994

So the byte-encoded fingerprint removed most of the medium-record cost, and the small-record cost is about where it was. The medium save is still above the bar the study set before the run (median at most 1.5 with the 95% upper end below 1.6), so by that rule the PR stays draft and the save cost is discussed here before a merge.

Proposed next step from our side: an untimed profile of the remaining medium-save cost (where the fingerprint time goes for a medium record: encoding, hashing or the extra read), so that the choice is between concrete options rather than a number: accept this cost for the lost-update guarantee, cut it further, or make the check opt-out. We will post that profile here; the merge decision waits for it.

[tae898]

Untimed profile of the medium save on the same two arms as the third A/B (one fresh process each, 2000 saves of one medium record, median microseconds per call; profile only, the timed figures are the ones above):

phase	main	this PR
one iteration (lookup, modify, set, save, commit), library unwrapped	266.9	485.7
lookup_by_rid	67.6	69.5
modify	1.2	39.7
of which: fingerprint of the record as read		37.2
save	125.4	308.7
of which: re-read of the stored row (SELECT)		55.5
of which: fingerprint of the stored row		35.8
of which: the UPDATE	123.0	126.1
of which: reading the updated row back		35.2
of which: fingerprint of the row after the update		36.5
commit (transaction exit)	69.8	71.7

One fingerprint of this record (6655 packed bytes, 21 properties) is about 25.6 us of packing plus 7.3 us of hashing, and a medium save computes three of them (109.5 us together). The other added parts are the re-read of the stored row and reading the row back after the update.

Proposed change on this PR, then a fourth A/B with the same rule:

1. At the first change (modify() or the first set()), keep a copy of the values as read (a dict copy, with nested lists, dicts and arrays copied too, since set() writes into the same dict and a caller can change a nested list in place), and at save() compare that copy with the stored row's decoded values directly instead of fingerprinting both. That replaces two fingerprints with a copy and a comparison.
2. After the update, take the values we wrote as the new baseline instead of reading the row back and fingerprinting it, where the engine stores exactly what we sent; a parity test pins which value kinds are stored as sent, and only the others keep the read-back.
3. Longer term, if the native path can see the record version, a version check in the UPDATE would remove the re-read as well; that needs an engine-side look first, so it is not part of this PR.

If you see a reason against 1 or 2, say so; otherwise we will push them with tests and re-measure.