Skip to content

Cut fixed per-request costs out of the reshard control plane - #816

Open
copybara-service[bot] wants to merge 1 commit into
mainfrom
test_973205820
Open

Cut fixed per-request costs out of the reshard control plane#816
copybara-service[bot] wants to merge 1 commit into
mainfrom
test_973205820

Conversation

@copybara-service

Copy link
Copy Markdown

Cut fixed per-request costs out of the reshard control plane

Every Stage-3 coordination paid four avoidable fixed costs on its framed
RPCs (coordinate, GET_METADATA, receiver arm):

  1. The 4-byte length prefix and the body went out as two separate send()
    calls with Nagle enabled, on requests and responses alike, exposing
    every hop to the delayed-ACK stall (tens of ms on small RPCs).
  2. The destination controller was asked for every registered unit's full
    pool manifest on every request, although work units register once per
    engine lifetime.
  3. Client sockets carried no keepalive, so a black-holed peer was only
    detected at the full receive timeout.
  4. The framed server never reaped its per-connection threads: one
    std::thread handle and stack per request, held until shutdown.

Changes:

  • framed_rpc: single-buffer framing on both directions; TCP_NODELAY on
    client and accepted sockets; SO_KEEPALIVE plus TCP_USER_TIMEOUT bounded
    by the call's I/O timeout on client sockets; the accept loop joins
    finished connection threads.
  • reshard_coordinator: destination metadata is cached per controller
    address. Staleness (engine replacement) surfaces as a plan-build or
    receiver-arm failure, both side-effect-free beyond the abandoned claim,
    and is repaired by invalidate + fresh query + one replay; a failure
    after the receiver ack is never replayed.

Validation: reshard package builds and reshard_service_test passes in the
glibc-2.36 container, including the new
RemoteMetadataCachedAndRefreshedOnStaleFailure test (cache hit on the
second request, exactly one refetch after a fingerprint-mismatch replay).

GitHub: #808

Every Stage-3 coordination paid four avoidable fixed costs on its framed
RPCs (coordinate, GET_METADATA, receiver arm):

1. The 4-byte length prefix and the body went out as two separate send()
   calls with Nagle enabled, on requests and responses alike, exposing
   every hop to the delayed-ACK stall (tens of ms on small RPCs).
2. The destination controller was asked for every registered unit's full
   pool manifest on every request, although work units register once per
   engine lifetime.
3. Client sockets carried no keepalive, so a black-holed peer was only
   detected at the full receive timeout.
4. The framed server never reaped its per-connection threads: one
   std::thread handle and stack per request, held until shutdown.

Changes:

- framed_rpc: single-buffer framing on both directions; TCP_NODELAY on
  client and accepted sockets; SO_KEEPALIVE plus TCP_USER_TIMEOUT bounded
  by the call's I/O timeout on client sockets; the accept loop joins
  finished connection threads.
- reshard_coordinator: destination metadata is cached per controller
  address. Staleness (engine replacement) surfaces as a plan-build or
  receiver-arm failure, both side-effect-free beyond the abandoned claim,
  and is repaired by invalidate + fresh query + one replay; a failure
  after the receiver ack is never replayed.

Validation: reshard package builds and reshard_service_test passes in the
glibc-2.36 container, including the new
RemoteMetadataCachedAndRefreshedOnStaleFailure test (cache hit on the
second request, exactly one refetch after a fingerprint-mismatch replay).

GitHub: #808
PiperOrigin-RevId: 973205820
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant