Conversation
…struction A helper node whose first graph use lands after rclcpp::shutdown() is left marked as registered while absent from the listener's node list, and its destructor then throws NodeNotFoundError out of a noexcept context.
A discovery pass asks the graph about every name it listed. A name that answers with an rcl error, or with no service, publisher or subscription at all, is not a node any more and is left out of that pass.
The reader owns a private ROS node named after the gateway, so a second instance claims a fully qualified name that is already taken. Plugins take the gateway's own reader through the plugin context.
The shared reader's mutex covers making and destroying a call's callback group and client on its private node, which is what rclcpp does not serialise. The service wait, the request and the inline spin run outside it, so a read of one node no longer waits for a read of another, and the destructor waits on an in-flight count rather than on the mutex. The isolated group comes from ros2_common, where node-entity creation lives. Ros2FaultServiceTransport is the only private client node whose registration a test can falsify: its clients exist before the wait, so a post-shutdown wait reaches the graph. The reader and the parameter beacon create clients per call, where rcl_client_init fails first, so their registration is correct by construction only.
bburda
force-pushed
the
fix/discovery-departed-nodes
branch
from
September 13, 2026 18:26
1a5a98f to
316389d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
Summary
Three defects on the path a node takes when it leaves the graph while the gateway is running.
A private client node could take the gateway down at shutdown. The gateway creates private nodes for its service clients (fault reporting, lifecycle state reads, the parameter beacon). rclcpp registers a node with the context's graph listener lazily, on its first graph wait, and spends the one-shot flag before the registration happens. A shutdown that lands between the two leaves
~NodeGraphcallingremove_nodeon a listener that never saw the node;NodeNotFoundErrorescapes a destructor and the process aborts. The private nodes now register at construction, while the context is known to be valid. A unit test drives a service wait after shutdown on the fault transport and reproduces the abort on the previous code; the lifecycle reader and the beacon create their clients per call, so they cannot reach that wait after shutdown and are fixed by construction only.A node the graph no longer attributes an endpoint to is not an App. The DDS graph cache can keep a node's name after the node is gone while the same cache has already dropped every one of its readers and writers. Discovery already asks the graph for each node's services on every pass; it now treats a name whose per-node queries throw, or that has no services, publishers or subscriptions at all, as absent. A live rclcpp node cannot reach that state: the time source subscribes to
/parameter_eventsunconditionally on every distro, and jazzy and newer add the type-description service.docs/config/discovery-options.rstrecords what makes a node an App, per distro. A new feature test churns a demo node through thirty start/stop cycles at a fast refresh and checks the gateway answers and lists only live nodes afterwards, and asilent_nodedemo with parameter services off pins that a node advertising a single endpoint stays an App.One lifecycle-state reader per gateway. The REST server and the graph watchdog each created their own
<gateway>_lifecycle_state_readernode, two nodes of one name on the graph. The gateway now owns one, hands it to the REST server and to plugins through the plugin context (a virtual accessor with a null default, so out-of-tree plugins build unchanged and fall back to their own reader). Sharing it must not queue/statusbehind the watchdog's seed loop: the reader's mutex covers only client creation and destruction on the shared node, and concurrent reads overlap on their own callback groups. An e2e test counts the reader on the graph once, and a second one bounds/statuslatency with the watchdog armed on nodes that never answer: 6 to 7 ms with this change, against 489 ms when the whole read was serialised.Issue
No issue. Two of the three are crashes or false listings seen in the sanitizer jobs on
main.Type
Testing
test_graph_listener_shutdown_safety: aborts with the CI signature on the previous code, passes now.test_runtime_discovery: a name the graph does not know is reported absent; a live node stays an App; a mutation that returns the empty service list as present fails exactly the absent case.test_departed_node_discovery(churn plus the minimal-node control),test_lifecycle_reader_identity_e2e(fails with "was on the graph 2 times" when the watchdog builds its own reader),test_lifecycle_reader_contention_e2e(fails at 489 ms with the wide mutex).test_combined_introspectionand the graph_watchdog node-death e2e suite green in repeated runs; gateway unit suite 128 targets; full integration suite on jazzy 1591 cases, 0 failures; Sphinx with warnings as errors clean.Checklist