Skip to content

gateway: survive a node leaving mid-pass, and keep one lifecycle reader - #689

Open
bburda wants to merge 6 commits into
mainfrom
fix/discovery-departed-nodes
Open

bburda wants to merge 6 commits into
mainfrom
fix/discovery-departed-nodes

Conversation

@bburda

@bburda bburda commented Sep 13, 2026

Copy link
Copy Markdown
Collaborator

Pull Request

Summary

Three defects on the path a node takes when it leaves the graph while the gateway is running.

A private client node could take the gateway down at shutdown. The gateway creates private nodes for its service clients (fault reporting, lifecycle state reads, the parameter beacon). rclcpp registers a node with the context's graph listener lazily, on its first graph wait, and spends the one-shot flag before the registration happens. A shutdown that lands between the two leaves ~NodeGraph calling remove_node on a listener that never saw the node; NodeNotFoundError escapes a destructor and the process aborts. The private nodes now register at construction, while the context is known to be valid. A unit test drives a service wait after shutdown on the fault transport and reproduces the abort on the previous code; the lifecycle reader and the beacon create their clients per call, so they cannot reach that wait after shutdown and are fixed by construction only.

A node the graph no longer attributes an endpoint to is not an App. The DDS graph cache can keep a node's name after the node is gone while the same cache has already dropped every one of its readers and writers. Discovery already asks the graph for each node's services on every pass; it now treats a name whose per-node queries throw, or that has no services, publishers or subscriptions at all, as absent. A live rclcpp node cannot reach that state: the time source subscribes to /parameter_events unconditionally on every distro, and jazzy and newer add the type-description service. docs/config/discovery-options.rst records what makes a node an App, per distro. A new feature test churns a demo node through thirty start/stop cycles at a fast refresh and checks the gateway answers and lists only live nodes afterwards, and a silent_node demo with parameter services off pins that a node advertising a single endpoint stays an App.

One lifecycle-state reader per gateway. The REST server and the graph watchdog each created their own <gateway>_lifecycle_state_reader node, two nodes of one name on the graph. The gateway now owns one, hands it to the REST server and to plugins through the plugin context (a virtual accessor with a null default, so out-of-tree plugins build unchanged and fall back to their own reader). Sharing it must not queue /status behind the watchdog's seed loop: the reader's mutex covers only client creation and destruction on the shared node, and concurrent reads overlap on their own callback groups. An e2e test counts the reader on the graph once, and a second one bounds /status latency with the watchdog armed on nodes that never answer: 6 to 7 ms with this change, against 489 ms when the whole read was serialised.


Issue

No issue. Two of the three are crashes or false listings seen in the sanitizer jobs on main.


Type

  • Bug fix
  • New feature or tests
  • Breaking change
  • Documentation only

Testing

  • test_graph_listener_shutdown_safety: aborts with the CI signature on the previous code, passes now.
  • test_runtime_discovery: a name the graph does not know is reported absent; a live node stays an App; a mutation that returns the empty service list as present fails exactly the absent case.
  • test_departed_node_discovery (churn plus the minimal-node control), test_lifecycle_reader_identity_e2e (fails with "was on the graph 2 times" when the watchdog builds its own reader), test_lifecycle_reader_contention_e2e (fails at 489 ms with the wide mutex).
  • test_combined_introspection and the graph_watchdog node-death e2e suite green in repeated runs; gateway unit suite 128 targets; full integration suite on jazzy 1591 cases, 0 failures; Sphinx with warnings as errors clean.

Checklist

  • Breaking changes are clearly described (and announced in docs / changelog if needed)
  • Tests were added or updated if needed
  • Docs were updated if behavior or public API changed

…struction

A helper node whose first graph use lands after rclcpp::shutdown() is left
marked as registered while absent from the listener's node list, and its
destructor then throws NodeNotFoundError out of a noexcept context.
A discovery pass asks the graph about every name it listed. A name that
answers with an rcl error, or with no service, publisher or subscription at
all, is not a node any more and is left out of that pass.
The reader owns a private ROS node named after the gateway, so a second
instance claims a fully qualified name that is already taken. Plugins take
the gateway's own reader through the plugin context.
The shared reader's mutex covers making and destroying a call's callback group
and client on its private node, which is what rclcpp does not serialise. The
service wait, the request and the inline spin run outside it, so a read of one
node no longer waits for a read of another, and the destructor waits on an
in-flight count rather than on the mutex. The isolated group comes from
ros2_common, where node-entity creation lives.

Ros2FaultServiceTransport is the only private client node whose registration a
test can falsify: its clients exist before the wait, so a post-shutdown wait
reaches the graph. The reader and the parameter beacon create clients per call,
where rcl_client_init fails first, so their registration is correct by
construction only.
@bburda
bburda force-pushed the fix/discovery-departed-nodes branch from 1a5a98f to 316389d Compare September 13, 2026 18:26
@bburda bburda self-assigned this Sep 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant