Skip to content

Fix URMA send buffer reclamation order - #3626

Open
BGQ99 wants to merge 1 commit into
apache:masterfrom
LinQuickDev:fix/urma-send-buffer-reclaim
Open

BGQ99 wants to merge 1 commit into
apache:masterfrom
LinQuickDev:fix/urma-send-buffer-reclaim

Conversation

@BGQ99

@BGQ99 BGQ99 commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: Related to #3428

Problem Summary:

URMA send completion handling publishes the reclaimed SQ credit before clearing the corresponding IOBuf. A concurrent writer can observe the available credit and reuse the ring slot while the completion handler is still clearing it, potentially causing concurrent IOBuf modification and a crash during buffer release.

What is changed and the side effects?

Changed:

  • Clear the completed send buffer and advance the completion cursor before publishing the SQ credit, following the reclamation order used by the RDMA endpoint.
  • Move the existing memory barrier before the credit update.
  • Check window capacity before reclamation so an invalid completion cannot clear a buffer or advance the cursor.
  • Add regression tests for reclamation order, cursor wraparound, and rejection of a completion when the SQ window is already full.

Validation:

  • CI passed in the LinQuickDev/brpc repository.
  • Full CMake build with WITH_URMA=ON and URMA_USE_MOCK=ON passed.
  • All 17 URMA unit tests passed, including the 3 new tests.
  • The reclamation-order test fails as expected with the original upstream implementation.
  • git diff --check passed.
  • Real URMA hardware validation of this upstream-based revision and ASAN were not run locally.

Side effects:

  • Performance effects: adds a capacity check before reclamation. No performance improvement is claimed.
  • Breaking backward compatibility: none intended. No public API or wire protocol changes. Receive processing and ACK-only completions are unchanged.

Check List:

@wwbmmm
wwbmmm requested a balanced review from Copilot October 19, 2026 12:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open issues
What changed in this PR

Fixes a race in URMA send completion handling by reclaiming/clearing the completed send buffer before publishing SQ credits, preventing a concurrent writer from reusing a slot that is still being cleared.

Changes:

  • Reorders send-buffer reclamation vs. SQ-credit publication (and moves the memory barrier accordingly).
  • Adds a capacity guard to reject invalid completions when the SQ window is already full.
  • Adds regression tests covering reclamation order, cursor wraparound, and full-window rejection.
File Description
test/​brpc_urma_unittest.cpp Adds regression tests that validate the new reclamation order and edge cases (wraparound, full window).
src/​brpc/​urma/​urma_endpoint.cpp Reorders buffer clearing and SQ credit publication and adds a capacity check to reject overflow completions.

Review effort: Lite

Comment on lines +75 to +86
endpoint._sbuf.resize(endpoint._sq_size -
urma::UrmaEndpoint::RESERVED_WR_NUM);
endpoint._local_window_capacity = endpoint._sbuf.size();
endpoint._sq_window_size.store(0, butil::memory_order_relaxed);
ASSERT_EQ(0, endpoint._sbuf[0].append_user_data(
&payload, sizeof(payload), [&](void*) {
released = true;
// Observe the credit at the exact point the old buffer is freed.
// Publishing first lets the writer reuse a slot still being cleared.
EXPECT_EQ(0, endpoint._sq_window_size.load(
butil::memory_order_relaxed));
}));
ASSERT_EQ(0, endpoint.HandleCompletion(SendCompletion()));
EXPECT_TRUE(released);
EXPECT_TRUE(endpoint._sbuf[0].empty());
EXPECT_EQ(1, endpoint._sq_sent);

@wwbmmm wwbmmm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[replied by brpc-oncall robot] The reclamation-order change in HandleCompletion looks correct and consistent with the RDMA endpoint, but the new unit tests cannot compile when BRPC_WITH_URMA is enabled because they access private UrmaEndpoint members; also the capacity check is now duplicated.

char payload = 'x';
bool released = false;
urma::UrmaEndpoint endpoint(_socket.get());
endpoint._sbuf.resize(endpoint._sq_size -

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[replied by brpc-oncall robot] These tests cannot compile when the tree is built with WITH_URMA=ON: _sbuf, _sq_size, _sq_sent, _sq_window_size, _local_window_capacity and HandleCompletion() are all private in UrmaEndpoint (see src/brpc/urma/urma_endpoint.h: everything after the private: section that starts at the State enum), and the fixture is not declared as a friend. CI does not catch this because test/brpc_urma_unittest.cpp compiles to an empty translation unit unless BRPC_WITH_URMA is 1. Please grant access explicitly, e.g. add #include <gtest/gtest_prod.h> plus FRIEND_TEST(UrmaSendCompletionTest, reclaim_buffer_before_publishing_credit); (and the other two test names) in UrmaEndpoint, or expose a small test-only accessor, so that the URMA unit-test build actually compiles.

uint16_t wnd = 1; // We signal every WR (complete_enable=1).
uint16_t old =
_sq_window_size.load(butil::memory_order_relaxed);
if (old >= _local_window_capacity) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[replied by brpc-oncall robot] This capacity check plus its warning is now duplicated: the same check is still inside the while (true) CAS loop below. The in-loop branch is now only reachable after the buffer has already been cleared and _sq_sent advanced, in which case the credit is dropped while the state was already mutated. Since _sq_window_size is only incremented here (writers only decrement it), consider doing the single pre-check and then publishing with _sq_window_size.fetch_add(wnd, butil::memory_order_relaxed) after the barrier, removing the now-redundant retry loop so reclamation and credit publication stay in one place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants