From 3f65150f8df03470209f25e43cd4be73eb240769 Mon Sep 17 00:00:00 2001 From: Max Buckley Date: Tue, 25 Aug 2026 19:14:29 +0200 Subject: [PATCH] perf(brute_force): drop the duplicate bitset/bitmap->CSR conversion on the SDDMM path brute_force_search_filtered builds the CSR structure from the filter via to_csr, then hands the same CSR to raft::sparse::linalg::masked_matmul, which converts the mask into that structure again. masked_matmul's temporary is sparsity-preserving, so it shares indptr/indices with the CSR cuVS just filled and writes identical values over it. Every SDDMM-path search ran the conversion twice. nsys over 21 SDDMM searches (1M x 128, 0.5% selectivity, RTX 5090), before -> after: calc_nnz_by_rows 42 -> 21, repeat_csr 42 -> 21, exclusive-scan 42 -> 21, value fill 42 -> 21, and fill_indices_by_rows -- the sparsity-owning instantiation only cuVS's to_csr could produce -- 21 -> 0. sddmm_ker and csr_to_coo stay at 21, so no real work is removed. Dropping cuVS's call also removes a host sync per search: to_csr on a sparsity-owning matrix reads the computed nnz back to the host and syncs the stream, while masked_matmul's conversion runs on a sparsity-preserving matrix and takes neither branch. 8-byte device-to-host copies per 29 filtered searches fall from 50 to 29 -- exactly the 21 removed readbacks. Paired A/B on a rebuilt libcuvs.so, 10 alternating rounds x 200 reps, dedicated idle RTX 5090, fp16/L2Expanded/k=64/100 queries. The saving is a roughly fixed ~150 us per search, so it is largest where the search is cheapest: N dim sel baseline patched speedup saving (95% CI) 1M 128 0.1% 0.383 ms 0.238 ms 1.61x +136 us [ 93, 180] 1M 1024 0.1% 0.468 ms 0.323 ms 1.45x +146 us [109, 183] 1M 128 1.0% 0.708 ms 0.579 ms 1.22x +133 us [115, 151] 10M 128 0.1% 0.750 ms 0.589 ms 1.27x +158 us [134, 182] 10M 128 3.0% 14.084 ms 13.760 ms 1.02x +335 us [252, 419] All 12 SDDMM-path configs improve with the interval excluding zero (t = 6.7 to 26.5). Three dense-path configs act as a null control and are flat, as they must be: +1.8 us, -57.6 us, +96.7 us, every interval spanning zero. Top-k ids and distances are bit-identical to the previous build across 14 configs spanning both dispatch paths, and the prefiltered brute-force tests pass. Co-Authored-By: Claude Opus 5 (1M context) --- cpp/src/neighbors/detail/knn_brute_force.cuh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/cpp/src/neighbors/detail/knn_brute_force.cuh b/cpp/src/neighbors/detail/knn_brute_force.cuh index 76ed71fe57..e0046f9275 100644 --- a/cpp/src/neighbors/detail/knn_brute_force.cuh +++ b/cpp/src/neighbors/detail/knn_brute_force.cuh @@ -825,8 +825,9 @@ void brute_force_search_filtered( raft::identity_op(), filter_type); } else { + // masked_matmul fills the CSR structure from the mask itself, so it is left uninitialized + // here; converting first would only run the same conversion twice. auto csr = raft::make_device_csr_matrix(res, n_queries, n_dataset, nnz_h); - std::visit([&](const auto& actual_view) { actual_view.to_csr(res, csr); }, *filter_view); // create filter csr view auto compressed_csr_view = csr.structure_view();