Skip to content

Integrate URMA with the common transport handshake - #88

Merged
zchuango merged 17 commits into
LinQuickDev:urma-handshakefrom
Spicy-cream:feature/urma-handshake-followup
Oct 23, 2026
Merged

zchuango merged 17 commits into
LinQuickDev:urma-handshakefrom
Spicy-cream:feature/urma-handshake-followup

Conversation

@Spicy-cream

@Spicy-cream Spicy-cream commented Oct 22, 2026 •

Copy link
Copy Markdown

Summary

  • Integrate URMA v2/v3 negotiation with the common AdapterTransport handshake.
  • Carry the standalone bonding import, ordered SENDs and 4096-byte CTP SEND limit from PR59, retaining 8192-byte buffers.
  • Fix CQ callback lifetime separately: pin the main Socket before endpoint access, synchronize teardown and reject stale reset-generation callbacks.
  • Add URMA compatibility, fallback and CQ lifecycle tests, bounded stream diagnostics and common-handshake documentation.

Scope

The target branch is urma-handshake. This branch includes the common transport-handshake framework from apache#3535 and the URMA follow-up changes.

URMA transport support and its Chinese/English documentation are already in community master through apache#3428. The existing URMA documents and .github/workflows/ci-linux.yml are kept identical to apache/brpc master at 87d498d. The common-handshake design document describes the framework and URMA integration introduced here.

Local validation

  • URMA: 59 tests; fallback: 8; common handshake: 28; RDMA: 92; UBSHM: 43.
  • Legacy/common interoperability and final-ACK/first-packet handoff: 100 shuffled repetitions with loopback TCP and the mock provider.
  • CQ close, last-reference release and reset-generation tests: 100 shuffled repetitions normally and under ASan (detect_leaks=0), covering event/polling and copy/zero-copy. Close tests use the event dispatcher and actual polling workers.
  • Large-attachment tests verify patterned echoes byte-for-byte across the receive/CQ modes.
  • TCP-only fallback: 8 tests. Release static-library build/install and performance-example build passed.

Hardware restart/reconnect validation

User-run node2 client to node1 server, using the system liburma, bonding_dev_0 in standalone mode, v3 common handshake on both peers and 8192-byte buffers. Each run used 4 workers, queue depth 16, target 250 QPS per worker, a 128-byte echoed attachment and a 180-second client duration. The server was stopped with Ctrl+C and restarted during client traffic.

All four combinations recovered to approximately 1000 QPS. The supplied log excerpts show the cumulative error count remaining constant in the final samples; the completion/import/protocol/FATAL/fallback searches returned no matches. Counts below include RPC failures during the intentional server downtime.

CQ mode Receive mode Final cumulative errors Final latency samples
event zero-copy 30080 121-135 us
polling zero-copy 27072 99-109 us
polling copy 21056 59-62 us
event copy 21008 121-165 us

The earlier PR59 hardware record includes 1 MiB echo and five-minute concurrent echo with errors=0. These are separate from the restart/reconnect runs above.

Spicy-cream and others added 11 commits October 8, 2026 23:58
…lity

Resolve UBSHM handshake conflicts using the tested rebased tree. Retain backend-specific format negotiation, remote shared-memory lengths, and upstream IPC V2 tests.
…er-oct15

Resolve current master conflicts and preserve UBSHM IPC V2 negotiation
…ict-oct18

Resolve master conflict and preserve shared memory name safety
…flict-refresh

Resolve modular CMake conflict and preserve handshake protobuf generation
Port URMA v2 and v3 negotiation into AdapterTransport while preserving the existing RDMA and UBSHM implementations and regression tests.

Include standalone bonding import, ordered SEND work requests, the 4096-byte CTP payload limit, bounded diagnostics, mock regression coverage and concise documentation.
@Spicy-cream

Copy link
Copy Markdown
Author

Updated in dc983e8.

  • Recognize URMA v2/v3 on plain TCP servers, including builds without the URMA SDK, and reply with a disabled Hello.
  • Preserve legacy URMA rejection semantics: a disabled Hello switches to TCP without an ACK. A client import failure after an accepted Hello still sends a disabled ACK.
  • Add fragmented/coalesced-frame, legacy no-ACK, resource-failure cleanup and concurrent TCP-fallback RPC regressions. Malformed frames still fail the connection; established URMA errors do not switch data paths.
  • Add WITH_URMA=ON/OFF CI coverage and keep the original RDMA, UBSHM and common-handshake test files unchanged. Legacy implementation removal is left for a separate cleanup.

Validation: URMA 51 tests, fallback 8 tests, common handshake 28 tests, RDMA 92 tests and UBSHM 43 tests passed. URMA and fallback suites passed 100 shuffled repetitions each. TCP-only build passed all 8 fallback tests; Make TCP-only fallback object compilation passed.

@Spicy-cream

Copy link
Copy Markdown
Author

Added in 9f4efc6 (tests and documentation only; no production data-path changes).

  • Execute retained legacy UrmaConnect/client/server state machines against the common handshake over loopback TCP in both directions, for v2/v3 and event/polling modes. These are full baidu_std RPCs with response and attachment checks, not just Hello round-trips.
  • Add barrier-controlled first-packet cases: data before the last ACK byte; data while the completed ACK is still inside the TCP parser; and an ACK-first control. The common-peer cases cover copy and zero-copy receives. Each scenario checks exactly 17 service executions.
  • Five new tests cover 32 configurations. All passed 100 shuffled repetitions (54,400 RPCs). Full URMA suite: 56 passed; fallback: 8 passed; common handshake: 28 passed. TCP-only build and its 8 fallback tests also passed.
  • Mutation check: temporarily registering CQ callbacks during resource allocation made the ACK-parser regression fail with the tcp_fd/urma_jetty concurrent-parsing assertion. The mutation was reverted.

Validation boundary: the tests run actual old/new handshake code but share the current data path and mock provider. Two independent old/new binaries on real devices have not been tested here. The separately identified CQ callback/resource teardown lifetime risk remains open; these handoff tests do not claim to fix or validate that race.

Pin the main Socket through a carrier-owned context before endpoint access. Serialize CQ dispatch with teardown and reject callbacks from an earlier reset generation.

Add barrier-controlled event/polling and copy/zero-copy regressions for concurrent close, last-reference recycling, and stale callbacks after reset. Keep this lifecycle fix separate from handshake wire compatibility.
@Spicy-cream

Copy link
Copy Markdown
Author

Updated to 91bf2d1 with two separate commits:

  • 5323b4f: fix CQ callback lifetime during teardown. Callbacks pin the main Socket before endpoint access; a shared context, mutex and generation check synchronize teardown and reject stale callbacks after reset.
  • 91bf2d1: simplify the Chinese and English URMA documentation (52 fewer lines).

Added three CQ regression tests covering event/polling and copy/zero-copy: close during dispatch, callback release of the last main-Socket reference, and callbacks after reset. The close tests use the event dispatcher and actual polling workers.

Local Linux validation: URMA 59 tests, fallback 8, common handshake 28, RDMA 92, UBSHM 43; TCP-only fallback 8; Release static-library build. CQ tests passed 100 shuffled repetitions both normally and under ASan (detect_leaks=0). Removing each lifetime protection made its corresponding regression fail, including an ASan heap-use-after-free report for endpoint access before pinning the main Socket. All temporary mutations were restored.

The CQ lifecycle change is separate from handshake wire compatibility. Hardware validation of this new lifecycle fix is the next test step. CI is running for the updated head.

@Spicy-cream
Spicy-cream changed the base branch from transport-handshake to urma-handshake October 22, 2026 11:28
@Spicy-cream

Copy link
Copy Markdown
Author

Updated in e894f75:

  • Restore .github/workflows/ci-linux.yml exactly from apache/brpc master (87d498d), including its master-only PR trigger. Remove the added transport-handshake trigger and URMA-specific CI job.
  • Restore docs/cn/urma.md and docs/en/urma.md exactly from community master. URMA support and both documents were introduced by the merged apache/brpc PR3428 on September 18, 2026.
  • Keep docs/cn/handshake_common_design.md for the common handshake and URMA integration implemented by this branch.

The PR file diff now contains no workflow changes and no changes to either URMA document. Runtime code and tests are unchanged by this cleanup. The description now distinguishes the existing URMA transport from the common-handshake integration; the current urma-handshake base also means this PR includes the framework from apache/brpc PR3535.

The Linux workflow retains the community master-only trigger, so it does not run automatically for this fork PR targeting urma-handshake. The previous 24-check result belongs to 91bf2d1.

@Spicy-cream

Copy link
Copy Markdown
Author

Addressed the five comment/style findings in 48640dc:

  • Describe HelloMessage as explicitly serialized network-order fields, independent of struct memory layout; identify its reserved wire bytes.
  • Mark UrmaConnect, the old TCP handshake callback and related helpers/factories as legacy interoperability-test code. Clarify AdapterTransport as the normal handshake orchestrator and UrmaEndpoint as the resource/data-path owner.
  • Replace the inaccurate Stateless label on UrmaHandshakeAdapter with its selected-version and parsed-peer state responsibilities.
  • Align the UrmaTransport friend declarations with AdapterTransport style.
  • Wrap the trace SGE cast and use a named std::min<size_t> prefix length, with an explicit algorithm include.

Validation: URMA 59 tests, fallback 8, common handshake 28, trace tests repeated 100 times with shuffling, Release static-library build, and git diff --check passed. Changes are limited to four headers. CI workflows and the existing Chinese/English URMA documents remain unchanged from community master.

@zchuango
zchuango merged commit 0d5aeb5 into LinQuickDev:urma-handshake Oct 23, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants