proxy-io: Reference-count Connection objects - #336
Draft
ryanofsky wants to merge 8 commits into
Draft
Conversation
Split connection teardown out of ~Connection into an idempotent disconnect() method, with the destructor delegating to it. This is a behavior-neutral refactor: the same steps run in the same order on destruction. Having a separate disconnect() method allows severing a connection while keeping the Connection object alive, which the next commits use to let shutdown code wait for in-flight server call bodies to finish after a disconnect (bitcoin/bitcoin#35845). Two details are new: - disconnect() cancels the m_on_disconnect handlers before severing the connection. Previously they were implicitly canceled when the TaskSet member was destroyed. When disconnect() is called separately from destruction, this is required for correctness: severing the stream completes m_network.onDisconnect(), and the registered handlers (_Serve, ConnectStream) destroy the Connection object out from under the caller. - disconnect() explicitly releases m_thread_pool and m_thread_map so worker thread teardown happens at disconnect time whether or not the object is destroyed right away. Previously this happened implicitly during member destruction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a per-connection ServerObjectTracker counting live ProxyServer objects, incremented in the ProxyServerBase constructor and decremented in its destructor, with Connection::waitDrained() blocking until the count reaches zero and Connection::pendingServerObjects() exposing it for logging. Disconnecting a connection cancels the KJ promise of an in-flight call, but a C++ server method body already dispatched to a worker thread runs to completion. Counting live server objects turns Cap'n Proto's object lifetime rules into a usable quiescence signal: a ProxyServer object is not destroyed until its outstanding calls finish (the target capability is kept alive for the duration of a call and pinned by post()/PassField via thisCap()), so after disconnect() the count drains to zero exactly when no server call body is still executing. Waiting for that lets shutdown code avoid freeing application state that a still-running call body dereferences (bitcoin/bitcoin#35845). The tracker is held via shared_ptr by the Connection and by every ProxyServer object because objects kept alive by in-flight calls can outlive the Connection on some teardown paths (see ~ProxyServerBase), and their destructors must decrement state that is still valid. It must be declared before m_rpc_system, whose construction creates the bootstrap server object that registers itself with the tracker. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a deterministic mptest regression test for bitcoin/bitcoin#35845: hold a server method body in flight on a worker thread, call Connection::disconnect(), and assert that Connection::waitDrained() blocks until the body finishes and its server object is destroyed. Also covers destroying an already-disconnected connection (~Connection noticing disconnect() has run). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Make Connection objects shared_ptr-owned (created via a new Connection::make() factory whose custom deleter destroys the object on the event loop thread), and have every proxy object share ownership of its connection (ProxyContext::connection becomes shared_ptr<Connection>, populated via enable_shared_from_this). A Connection now always outlives its proxy objects and survives disconnect() as an inert husk until the last reference is dropped, which removes the long-standing rule that ~ProxyServerBase must not dereference m_context.connection. Because server-side proxy objects now hold references back to their connection, constructing the bootstrap server object during the Connection constructor would call shared_from_this() before any shared_ptr owner exists. Server-side setup is therefore split in two: make() constructs the connection, and a new Connection::serve(make_client) method starts the RPC system afterwards. (A side effect is that the member-initialization-order constraint on m_server_objects is gone, since make_client no longer runs during construction.) A consequence of the reference cycle m_rpc_system exports -> ProxyServer -> ProxyContext::connection is that dropping references alone never destroys a connected Connection: disconnect() breaks the cycles, and every teardown path now calls it before releasing its reference. The _Serve remote-disconnect handler now looks its connection up through a weak_ptr and removes it from m_incoming_connections by value instead of capturing a list iterator. This fixes a latent use-after-free: the handler runs from the event loop task set, so disconnectIncoming() could destroy the connection and invalidate the captured iterator between the handler being queued and running. Ipc::disconnectIncoming() behavior is unchanged; it now erases connections from the list in its first sync (keeping them alive via collected references), drains them, and then drops the references. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Remove the per-client disconnect tracking from ProxyClientBase: client objects no longer register a cleanup callback with their Connection, and a disconnect no longer eagerly releases their m_client capability handles or nulls their connection pointers. Neither is necessary now that proxy objects share ownership of their Connection. The connection pointer stays valid after a disconnect because the Connection outlives its proxies, and keeping the capability handle is safe: Cap'n Proto's per-connection state is refcounted and outlives the RPC system as long as handles reference it, with calls on handles of a disconnected connection failing cleanly with DISCONNECTED errors. The handle is simply released (on the event loop thread, since capability refcounts are not thread safe) whenever the client object is eventually destroyed, and clientInvoke checks the connection's m_disconnected flag instead of a nulled pointer, throwing the same 'IPC client method called after disconnect' error as before. This deletes the detach machinery from ~ProxyClientBase, including the FIXME'd duplicate-cleanup code path. Connection::addSyncCleanup remains for its one other user, the per-thread connection maps (see SetThread), which the next commit converts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Update comments to reflect that after the previous commit, the sync cleanup callback list has exactly one remaining purpose: eagerly removing a disconnected connection's ProxyClient<Thread> entries from the thread_local per-thread connection maps (ThreadContext::request_threads / callback_threads) via callbacks registered by SetThread. Unlike interface clients, these entries cannot simply be left alive across a disconnect: they are owned by other threads that may never touch their maps again, and a surviving entry would hold the disconnected Connection object -- and through its EventLoopRef the event loop -- alive indefinitely, preventing the loop from ever exiting. (Replacing the callbacks with lazy garbage collection in SetThread was tried and hangs mptest for exactly this reason: entries owned by long-lived threads pin the loop after their connection is gone.) So this per-object disconnect tracking is retained by design, now clearly documented as thread-map-specific rather than a general client-object mechanism. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Now that proxy objects hold shared ownership of their Connection, a ProxyServer object kept alive by an in-flight call can no longer outlive the Connection, so ~ProxyServerBase can always reach the tracker through m_context.connection. Drop the shared_ptr indirection that existed to keep the tracker valid past the Connection's death, and the separate tracker handle member on ProxyServerBase. No behavior change; Connection::waitDrained() semantics are identical. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. ReviewsSee the guideline and AI policy for information on the review process. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Use reference counting to manage Connection object lifetimes. This implements an old idea from #176 (comment) and has two benefits:
disconnectandwaitDrainedmethods #335 for server objects, and this PR extends it to treat client and server objects symmetrically.ProxyClientobjects register with Connections, soConnectionobjects no longer need to store lists ofProxyClientobjects and can just use use counts instead.This is based on #335. The non-base commits are:
d28279bipc: manage Connection lifetime with shared_ptr1557073ipc: keep client capability handles across disconnect05c9a7dipc doc: scope Connection cleanup callbacks to per-thread map entriesa067599ipc: make server object tracker a plain Connection member