Skip to content

test(python): migrate index count tests to the 0.8 API - #226

Merged
gouyt13 merged 2 commits into
VectorDB-NTU:mainfrom
leo062644:test/migrate-index-counts-214
Nov 11, 2026
Merged

gouyt13 merged 2 commits into
VectorDB-NTU:mainfrom
leo062644:test/migrate-index-counts-214

Conversation

@leo062644

Copy link
Copy Markdown
Contributor

Summary

The eight lifecycle-count cases in tests/python/test_index_counts.py currently fail when DeprecationWarning is treated as an error. Migrate their setup and insertion calls to the 0.8 API so they can continue checking capacity, stored records, live points, removal, and persistence after the 0.7 call forms are removed.

  • Use the new IVF constructor and check zero initial capacity, then capacity growth after insertion.
  • Keep HNSW's 64-slot preallocation with resize(64) and configure the single supplied centroid explicitly.
  • Replace IVF/HNSW build() with set_centroids() and keyword-based add(), keeping one worker thread.
  • Use SymphonyQG nbits=32 for raw vectors, move ef_construction=40 to construction, and build through add().
  • Preserve the existing count, returned-ID, invalid-removal, empty-search, and save/load checks.

Related to #214. This PR covers only test_index_counts.py; the tracking issue still has other files to migrate.

Testing

Validation on Ubuntu under WSL2, Python 3.14.10, with a source build of this checkout and RABITQ_ENABLE_NATIVE_OPTIMIZATION=OFF:

  • Before: the target file passed normally (8 passed, 26 deprecation warnings), but all 8 cases failed with -W error::DeprecationWarning.
  • After: python -m pytest tests/python/test_index_counts.py -W error::DeprecationWarning -q — 8 passed, no warnings.
  • Python suite using the wheel CI exclusions: 1110 passed, 2 skipped, 1429 warnings. Command: python -m pytest tests/python --ignore=tests/python/test_compare_with_faiss.py --ignore=tests/python/test_cpp_clustering_examples.py --ignore=tests/python/test_benchmark_protocol.py --disable-warnings --tb=short -q.
  • Repository Python formatting/lint script (Ruff 0.16.1) — passed. The committed LF script was executed through git show HEAD:scripts/check-python.sh | bash because the Windows checkout had CRLF line endings.
  • git diff --check — passed.

AI assistance was used to prepare the migration and run local validation.

@leo062644
leo062644 requested a review from gouyt13 as a code owner November 10, 2026 16:29
@github-actions github-actions Bot added the release:maintenance Release category managed from the PR title label Nov 10, 2026

@gouyt13 gouyt13 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this migration looks correct and keeps what the test checks. I ran the file with -W error::DeprecationWarning against a build of main (8 passed, stable over repeated runs), and scripts/check-python.sh passes.

Two small follow-ups, as suggestions you can commit directly:

  1. The two later add() calls used the default thread count before. The rule in #214 to pass num_threads=1 applies only where build() is replaced, since build() defaulted to one thread, so these two calls can keep the default.
  2. Set the capacity after the first add() in each branch instead of with kind != "hnsw". This matches how the later if kind == "ivf" updates read.

The tests still pass with all three suggestions applied.

Comment thread tests/python/test_index_counts.py Outdated
Comment on lines +45 to +51
if kind == "symqg":
index.build(data[:count], ef_construction=40, num_threads=1)
capacity = count
index.add(data[:count], num_threads=1)
else:
index.build(data[:count], centroid, assignments[:count])
index.set_centroids(centroid)
index.add(data[:count], cluster_ids=assignments[:count], num_threads=1)
if kind != "hnsw":
capacity = count

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if kind == "symqg":
index.build(data[:count], ef_construction=40, num_threads=1)
capacity = count
index.add(data[:count], num_threads=1)
else:
index.build(data[:count], centroid, assignments[:count])
index.set_centroids(centroid)
index.add(data[:count], cluster_ids=assignments[:count], num_threads=1)
if kind != "hnsw":
capacity = count
if kind == "symqg":
index.add(data[:count], num_threads=1)
capacity = count
else:
index.set_centroids(centroid)
index.add(data[:count], cluster_ids=assignments[:count], num_threads=1)
if kind == "ivf":
capacity = count

HNSW keeps the 64 slots from resize(64); IVF and SymphonyQG now hold exactly count.

Comment thread tests/python/test_index_counts.py Outdated
check_counts(index, capacity, count, count - 3)

ids = index.add(data[count : count + 3], assignments[:3])
ids = index.add(data[count : count + 3], cluster_ids=assignments[:3], num_threads=1)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
ids = index.add(data[count : count + 3], cluster_ids=assignments[:3], num_threads=1)
ids = index.add(data[count : count + 3], cluster_ids=assignments[:3])

This call used the default thread count before; only replaced build() calls need num_threads=1.

Comment thread tests/python/test_index_counts.py Outdated

np.testing.assert_array_equal(index.add(data[-1:], assignments[:1]), [stored])
np.testing.assert_array_equal(
index.add(data[-1:], cluster_ids=assignments[:1], num_threads=1), [stored]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
index.add(data[-1:], cluster_ids=assignments[:1], num_threads=1), [stored]
index.add(data[-1:], cluster_ids=assignments[:1]), [stored]

Same here.

Keep default threading for the two later add calls and place capacity
updates in their respective initialization branches.
@gouyt13
gouyt13 merged commit 2f5fedf into VectorDB-NTU:main Nov 11, 2026
38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release:maintenance Release category managed from the PR title

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants