Fix stream ordering for brute-force L2 sqrt post-processing - #2509
Open
dantegd wants to merge 2 commits into
Open
Fix stream ordering for brute-force L2 sqrt post-processing#2509dantegd wants to merge 2 commits into
dantegd wants to merge 2 commits into
Conversation
dantegd
added
bug
This was referenced Aug 26, 2026
divyegala
approved these changes
Aug 26, 2026
9 tasks
rapids-bot Bot
pushed a commit
to NVIDIA/raft
that referenced
this pull request
Aug 26, 2026
Assign the cuRAND generator used by `MST_solver::alteration()` to the MST CUDA stream. ## Problem `MST_solver::alteration()` allocates `rand_values` on the supplied stream and launches the alteration kernel on that stream, but the cuRAND generator is left on its default stream. This permits the following ordering: 1. cuRAND starts writing `rand_values` on stream 0. 2. The alteration kernel starts reading `rand_values` on the MST stream. 3. There is no dependency between those streams. Under concurrent HDBSCAN workloads using managed memory on GB300, this can eventually surface as an illegal address in a later operation. The downstream error observed by cuML was: ```cpp copy_if failed on 2nd step: cudaErrorIllegalAddress ``` Related cuML issue: NVIDIA/cuml#8510 Companion cuVS PR: NVIDIA/cuvs#2509 Authors: - Dante Gama Dessavre (https://github.com/dantegd) Approvers: - Divye Gala (https://github.com/divyegala) URL: #3125
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Run the fused brute-force KNN distance post-processing on the same selected CUDA stream as
fusedL2Knn.Problem
In
brute_force_knn_impl(), a stream is selected using:auto stream = raft::resource::get_next_usable_stream(handle, i);fusedL2Knn is launched on this selected stream. For L2SqrtExpanded and L2SqrtUnexpanded, the subsequent square-root map is instead launched using the original handle, whose main stream may differ from the selected stream:
There is no dependency making the main stream wait for the selected stream before the map reads the KNN output. The later stream-pool synchronization does not repair this read-before-write race.
Under concurrent cuML HDBSCAN workloads using managed memory on GB300, the corrupted CUDA context was subsequently reported by RAFT MST as:
Related cuML issue: NVIDIA/cuml#8510
Companion RAFT PR: NVIDIA/raft#3125