drpcstream: bound message size against the flow-control window#75
Draft
suj-krishnan wants to merge 11 commits into
Draft
drpcstream: bound message size against the flow-control window#75suj-krishnan wants to merge 11 commits into
suj-krishnan wants to merge 11 commits into
Conversation
suj-krishnan
force-pushed
the
sujatha/flow-control-max-streams
branch
from
July 13, 2026 07:01
d87ef7a to
02f81c4
Compare
suj-krishnan
force-pushed
the
sujatha/flow-control-message-size
branch
from
July 13, 2026 07:01
799584c to
d290786
Compare
suj-krishnan
force-pushed
the
sujatha/flow-control-max-streams
branch
from
July 13, 2026 07:06
02f81c4 to
96b1758
Compare
suj-krishnan
force-pushed
the
sujatha/flow-control-message-size
branch
from
July 13, 2026 07:06
d290786 to
450281c
Compare
suj-krishnan
force-pushed
the
sujatha/flow-control-max-streams
branch
from
July 13, 2026 08:41
96b1758 to
65d48d6
Compare
suj-krishnan
force-pushed
the
sujatha/flow-control-message-size
branch
from
July 13, 2026 08:41
450281c to
fc096db
Compare
suj-krishnan
force-pushed
the
sujatha/flow-control-max-streams
branch
from
July 13, 2026 08:47
65d48d6 to
b9f1367
Compare
suj-krishnan
force-pushed
the
sujatha/flow-control-message-size
branch
from
July 13, 2026 08:47
fc096db to
dee763b
Compare
suj-krishnan
force-pushed
the
sujatha/flow-control-max-streams
branch
from
July 13, 2026 08:55
b9f1367 to
bfa4cbe
Compare
suj-krishnan
force-pushed
the
sujatha/flow-control-message-size
branch
from
July 13, 2026 08:55
dee763b to
d9f6834
Compare
Define the KindWindowUpdate control frame that carries a flow-control credit grant: a varint byte delta in the body, for the stream named by the frame's stream id. Add WindowUpdateFrame/ParseWindowUpdate helpers to build and read it. Stream id 0 is reserved for a possible future connection-level window and is unused in v1. The control bit marks the frame as out-of-band signaling, so it is emitted without blocking on data backpressure. It is not relied on for backward compatibility: an unknown control frame is only dropped after packet assembly, so it is not safely ignorable on an active stream. Instead, grants are only ever sent once flow control is active cluster-wide, so a peer that predates flow control never receives one. ParseWindowUpdate enforces the whole v1 wire contract in one place, since these frames are intercepted before packet assembly and its message-id checks: it accepts only a self-contained control frame (Control and Done set) naming a real stream (id != 0) with a positive delta and no trailing bytes. A non-conforming frame yields ok=false and is dropped by the caller, so a reserved stream-0 update from a future peer is ignored rather than mishandled. This is dormant scaffolding for the flow-control work: nothing emits or acts on the frame yet. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
Add sendWindow, the per-stream flow-control credit balance on the sender. acquire spends credit, blocking until enough is available; grant adds credit; close terminates the window. acquire returns early on close (with the close error) or context cancellation (with ctx.Err()), consuming no credit in either case; both are checked before any debit, so a canceled context never consumes credit or lets a frame proceed even when credit is available. Grants are no-revoke: grant takes an unsigned delta (the wire type), so a grant can only ever raise the balance, and the addition saturates at math.MaxInt64 so a large or malicious delta can never wrap it negative. The balance is a signed int64; acquire never drives it below zero, but the enablement layer may debit credit for bytes sent before flow control begins enforcing, leaving it negative, in which case acquire blocks until grants restore it. This is the primitive only -- it is not yet wired into the stream write path (that gate comes in a later commit). Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
Wire the sendWindow credit gate into rawWriteLocked: a KindMessage frame now acquires len(frame) bytes of per-stream send credit before it is handed to the writer. Control frames (invoke/metadata) bypass the gate. The window is opt-in: a stream has no send window by default, so data writes stay ungated (unlimited) and behavior is unchanged until one is installed. terminate closes the window, so a send parked on credit wakes with the termination error -- matching the write path's existing preemption semantics. Per-stream only; the connection-level window and the enablement path that installs the window come in later commits. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
Add recvWindow, the per-stream receive side of flow control. It tracks buffered bytes (in-progress reassembly plus completed, not-yet-consumed data) and decides when to return credit to the sender: grants are withheld while buffered is at or above the high-water mark, and the accrued returnable credit is otherwise released once it reaches the threshold, coalescing many frames into a single grant. Consuming reopens a gate that the high-water mark had closed. dispatched/consumed return the credit delta the caller should emit as a KindWindowUpdate. Emitting the frame, wiring into the read path (HandleFrame/MsgRecv), and the connection-level receive budget come in later commits. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
Install a per-stream recvWindow and drive it from the read/recv path:
- HandleFrame intercepts an incoming KindWindowUpdate before the
assembler and applies it to the send window. Intercepting early keeps
a grant that interleaves with an in-progress message (grants are
emitted off the write lock) from disturbing reassembly.
- A dispatched KindMessage frame returns credit via recvWindow, emitting
a KindWindowUpdate once the accrued amount crosses the threshold and
the high-water gate is open.
- RawRecv/MsgRecv call recvWindow after dequeuing; consuming can reopen a
gate the high-water mark had closed and flush the accrued grant.
Grants are control frames, appended without blocking, so emitting them
from the reader goroutine is safe. The window is opt-in: with no recvw
installed no grants are emitted, and an incoming grant with no send
window is ignored, so default behavior is unchanged.
Stream-level only; the connection-level receive budget and the enablement
path that installs the windows come in later commits.
Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
Add Options.FlowControl to enable per-stream flow control. When Enabled, NewWithOptions installs the send window (seeded with StreamWindow as initial credit) and the receive window (HighWater + GrantThreshold), so data writes become credit-gated and grants are emitted -- all driven by configuration rather than injected by tests. This is static, symmetric enablement: both peers must enable it with compatible sizes. Advertise-on-enable (safe under mixed enablement), the manager wiring that turns it on for a whole connection, and the connection-level window come in later commits. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
The manager already threads drpcstream.Options through newStream, so enabling FlowControl in Options.Stream installs windows on every stream the manager creates (client and server) with no additional wiring. Add an end-to-end test over a real net.Pipe connection between two flow-control-enabled managers: a message larger than the send window completes only if credit grants flow back across the connection, exercising the full stream-level credit loop (gate -> grant-on-dispatch -> apply-grant) through the real read/write paths. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
Add an integration test for the production wake path of a sender parked on flow-control credit. A flow-control-enabled client sends a message larger than its window to a server that has no flow control (so it never returns credit); the send parks, and cancelling the RPC context wakes it. This exercises manageStream -> Cancel -> terminate -> sendWindow.close end to end, which the unit tests previously covered only in pieces (the primitive's ctx path, and stream-level Cancel). Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
Add Options.MaxStreams, a per-connection cap on how many concurrent streams a peer may open against this manager. When the cap is reached an inbound stream is refused in the read path -- before any per-stream state is allocated -- with a stream-level KindError, leaving the connection up. 0 means unlimited. This bounds the per-stream bookkeeping (goroutines, stream-table entries, window counters) that the byte-based flow-control windows do not, giving DoS resistance against a peer that opens many zero-byte streams. It is the third flow-control bound alongside the per-stream and connection windows, and is independent of the credit path. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
Flow control bounds a single message to high_water + window (the implicit maximum message size): a larger message can never complete -- the sender runs out of credit while the receiver's buffer sits full with nothing to drain -- so it would deadlock. rawWriteLocked now rejects such a message up front with a clear error rather than parking the sender on a message that can never finish. The bound is checked against the local FlowControl config, which under static symmetric enablement matches the peer's; a zero bound means unlimited (windows not sized). Defending against a non-compliant peer that sends an oversized message is a separate, receiver-side fail-stop (a follow-up). Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
The sender-side guard rejects an oversized message locally, but a peer that does not honor the max-message bound (a non-compliant or older peer, or an asymmetric configuration where the sender's window is larger than ours) can still put one on the wire. Such a message can never complete: once it reaches the receiver's implicit maximum of high_water + window the sender is out of credit with our buffer full and nothing to drain, so the reassembly would stall. HandleFrame now tracks the in-progress inbound message size and, when it grows past the maximum, fails the stream with a data-overflow error and notifies the peer with an abortive cancel. The connection stays up (stream-level isolation). SendCancel is safe from the reader goroutine because it terminates -- waking any parked local sender and releasing the write lock -- before it takes that lock, unlike SendError. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
suj-krishnan
force-pushed
the
sujatha/flow-control-max-streams
branch
from
July 13, 2026 08:59
bfa4cbe to
fd0c66a
Compare
suj-krishnan
force-pushed
the
sujatha/flow-control-message-size
branch
from
July 13, 2026 08:59
d9f6834 to
2f79115
Compare
suj-krishnan
force-pushed
the
sujatha/flow-control-max-streams
branch
3 times, most recently
from
July 20, 2026 15:18
5b303d7 to
88e78a0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #74.
Flow control gives each stream a credit window plus a receive high-water mark. Their sum,
high_water + window, is the largest single message the pair can move: a bigger message runs the sender out of credit while the receiver's buffer is full with nothing to drain, so it deadlocks. This PR bounds a message to that implicit maximum on both sides.Commit 1 — sender-side guard.
rawWriteLockedrejects aKindMessagelarger thanhigh_water + windowup front with a clear error, instead of parking the sender on a message that can never complete. Checked against the local config; v1 configures a uniform window cluster-wide, so local and peer maxima agree. A zero bound means unlimited (windows not sized / flow control off).Commit 2 — receiver-side fail-stop. A peer that does not honor the bound can still put an oversized message on the wire.
HandleFrametracks the in-progress inbound message size and, when it grows past the maximum, fails the stream with a data-overflow error and notifies the peer with an abortive cancel. The connection stays up (stream-level isolation).SendCancelis safe from the reader goroutine because it terminates — waking any parked local sender and releasing the write lock — before it takes that lock, unlikeSendError.Relationship to the current design. The receiver check here is per-message (tracking one in-progress message). The design generalizes this to a single enforced-cap rule — fail the stream once total buffered bytes (assembler + ring) would exceed
high_water + window, before the Enqueue that would block the shared reader — of which the oversized-single-message case is one instance, and specifies the stream-local (non-connection-fatal) error path it runs through. Generalizing this check is a follow-up; the fail-stop direction here is correct (there is no cooperative-tolerance behavior to unwind).Tests
drpcstream: sender-side rejection (fails fast rather than parking); no size limit when flow control is off; receiver-side detection observing the emitted cancel on the wire and the data-overflow error surfaced byRawRecv.drpcmanager: asymmetric end-to-end — a large-window client sends a message that exceeds a small-window server's maximum, and the server rejects it on receipt while the connection stays up.All green under
-race; gofmt/vet/build clean.