Repository navigation
Conversation
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
2 open issues
What changed in this PR
Fixes a race in URMA send completion handling by reclaiming/clearing the completed send buffer before publishing SQ credits, preventing a concurrent writer from reusing a slot that is still being cleared.
Changes:
- Reorders send-buffer reclamation vs. SQ-credit publication (and moves the memory barrier accordingly).
- Adds a capacity guard to reject invalid completions when the SQ window is already full.
- Adds regression tests covering reclamation order, cursor wraparound, and full-window rejection.
| File | Description |
|---|---|
| test/brpc_urma_unittest.cpp | Adds regression tests that validate the new reclamation order and edge cases (wraparound, full window). |
| src/brpc/urma/urma_endpoint.cpp | Reorders buffer clearing and SQ credit publication and adds a capacity check to reject overflow completions. |
Review effort: Lite
| endpoint._sbuf.resize(endpoint._sq_size - | ||
| urma::UrmaEndpoint::RESERVED_WR_NUM); | ||
| endpoint._local_window_capacity = endpoint._sbuf.size(); | ||
| endpoint._sq_window_size.store(0, butil::memory_order_relaxed); | ||
| ASSERT_EQ(0, endpoint._sbuf[0].append_user_data( | ||
| &payload, sizeof(payload), [&](void*) { | ||
| released = true; | ||
| // Observe the credit at the exact point the old buffer is freed. | ||
| // Publishing first lets the writer reuse a slot still being cleared. | ||
| EXPECT_EQ(0, endpoint._sq_window_size.load( | ||
| butil::memory_order_relaxed)); | ||
| })); |
| ASSERT_EQ(0, endpoint.HandleCompletion(SendCompletion())); | ||
| EXPECT_TRUE(released); | ||
| EXPECT_TRUE(endpoint._sbuf[0].empty()); | ||
| EXPECT_EQ(1, endpoint._sq_sent); |
wwbmmm
left a comment
There was a problem hiding this comment.
[replied by brpc-oncall robot] The reclamation-order change in HandleCompletion looks correct and consistent with the RDMA endpoint, but the new unit tests cannot compile when BRPC_WITH_URMA is enabled because they access private UrmaEndpoint members; also the capacity check is now duplicated.
| char payload = 'x'; | ||
| bool released = false; | ||
| urma::UrmaEndpoint endpoint(_socket.get()); | ||
| endpoint._sbuf.resize(endpoint._sq_size - |
There was a problem hiding this comment.
[replied by brpc-oncall robot] These tests cannot compile when the tree is built with WITH_URMA=ON: _sbuf, _sq_size, _sq_sent, _sq_window_size, _local_window_capacity and HandleCompletion() are all private in UrmaEndpoint (see src/brpc/urma/urma_endpoint.h: everything after the private: section that starts at the State enum), and the fixture is not declared as a friend. CI does not catch this because test/brpc_urma_unittest.cpp compiles to an empty translation unit unless BRPC_WITH_URMA is 1. Please grant access explicitly, e.g. add #include <gtest/gtest_prod.h> plus FRIEND_TEST(UrmaSendCompletionTest, reclaim_buffer_before_publishing_credit); (and the other two test names) in UrmaEndpoint, or expose a small test-only accessor, so that the URMA unit-test build actually compiles.
| uint16_t wnd = 1; // We signal every WR (complete_enable=1). | ||
| uint16_t old = | ||
| _sq_window_size.load(butil::memory_order_relaxed); | ||
| if (old >= _local_window_capacity) { |
There was a problem hiding this comment.
[replied by brpc-oncall robot] This capacity check plus its warning is now duplicated: the same check is still inside the while (true) CAS loop below. The in-loop branch is now only reachable after the buffer has already been cleared and _sq_sent advanced, in which case the credit is dropped while the state was already mutated. Since _sq_window_size is only incremented here (writers only decrement it), consider doing the single pre-check and then publishing with _sq_window_size.fetch_add(wnd, butil::memory_order_relaxed) after the barrier, removing the now-redundant retry loop so reclamation and credit publication stay in one place.

What problem does this PR solve?
Issue Number: Related to #3428
Problem Summary:
URMA send completion handling publishes the reclaimed SQ credit before clearing the corresponding IOBuf. A concurrent writer can observe the available credit and reuse the ring slot while the completion handler is still clearing it, potentially causing concurrent IOBuf modification and a crash during buffer release.
What is changed and the side effects?
Changed:
Validation:
WITH_URMA=ONandURMA_USE_MOCK=ONpassed.git diff --checkpassed.Side effects:
Check List: