Repository navigation
Conversation
…l k (B * k lookups -> k) A type with B buckets has B vector indexes; the search loaded and wrapped every hit of every bucket and kept k. GraphCalls.topHits merges the hit lists by score (stable, as the Python sort was), cuts to k, and loads those records in one crossing. Same records, order and scores; a missing record is still named, but now only when it is among the final k. New test file pins the result against a merge of the raw per-bucket lists and counts k wraps and k lookups (the no-bridge-jar fallback too). Port of humemai/arcadedb-embedded-python#534. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
09b90ba to
a352125
Compare
|
Rebased onto main |
|
The timed A/B for this PR is done (laptop, JVM wheel, 2 pinned cores,
Answers: 0 run(s) without CHECK ok (0 with a MISMATCH against the old algorithm, 0 with wrong wrap counts); cross-build comparison: 136 (round, type, k) answers where A and Z agree, 0 where B differs. So the change does what it says where it applies: with several buckets it loads Closing this PR for now. Multi-bucket vector types are not the default; if they become a target, this can come back with a study whose headline is a multi-bucket type and a steadier memory measure (several fresh JVMs per arm). |
VectorIndex.find_neareston a type with B buckets loaded and wrapped every hit of every bucket index (one record lookup and one Python wrapper per hit, B * k of them) and then sorted and kept k. This merges the per-bucket hit lists by score first and loads only the final k: a new bridge callGraphCalls.topHits(db, pairLists, k)takes the Java hit lists, sorts the candidates by score with a stable sort (equal scores keep the bucket order, as the Python sort of the concatenated hits did), cuts to k, and returns the k records, scores and RIDs in one crossing. Without the bridge jar the Python fallback does the same, reading each candidate's score and looking up only the k winners. This is the port of humemai/arcadedb-embedded-python#534 (backlog row B-01, JVM half); the patch applied tosrc/arcadedb,src/javaandtestsunchanged apart from thearcadedb_embeddedtoarcadedbimport rename.Results are identical: the same k records in the same order with the same scores. Checked with a fixed-seed dump of 120
find_nearestcalls (k = 1, 5, 10, 50; 600 vectors, 16 dimensions, 4 buckets) taken on humemaiorigin/mainand on this branch, byte-identical (rid andrepr(score)per hit, plus the hit shape). One edge differs on purpose: a hit whose record is missing now raises only if it is among the final k (before: if it was anywhere in any bucket's list); the message is unchanged.What it saves, untimed: record lookups and wrappers per search are k instead of B * k (40 to 10 at B = 4 and k = 10, counted by the new tests). With B = 1 nothing changes, every hit is a final hit. No timing was run.
Local checks (on the committed bytes, bridge jar rebuilt from this branch's
src/javain eclipse-temurin:25-jdk against the jars of an installedarcadedb-embedded26.11.1.dev0 wheel, tests run withPYTHONPATH=src, not a wheel built from this repo):tests/test_vector_top_k_first.py(7 new tests: result equals a stable merge of the raw per-bucket lists for k = 1, 5, 10, 50, 400; k wraps; k lookups on the no-jar path) passes, and its two count tests fail on the oldvector.py(40 vs 10) while the result tests pass on both; together withtest_vector.py,test_vector_second_pass.py,test_vector_params_verification.py,test_vector_delta_visibility.py,test_vector_sql.pyandtest_typed_rows.pythat is 147 passed; the existing test that countedGraphCalls.hitscalls (test_lsm_vector_search_looks_up_all_hits_in_one_bridge_call) now countstopHits; black, isort and bandit are clean on the changed files. Not run: the full suite,scripts/build.sh, CI, the pre-commit hook set as a whole. No doc page in this repo mentionsGraphCallsorfind_nearest, so no docs changed.🤖 Generated with Claude Code