Skip to content

opcua: rescan while disconnected, heal PLC_COMMS_LOST on connect, and name a plugin freeze-frame's source #658

Description

@bburda

Summary

Running the OPC UA plugin config-less against an OpenPLC server (OPCUA_DISCOVERY_ENABLED=1,
OPCUA_DISCOVERY_SUBNETS set, no OPCUA_ENDPOINT_URL, no node map) shows four
behaviours that a single deployment hits in one boot.

  1. Discovery is a one-shot scan at start-up. A gateway that powers on with its controller
    scans while the controller is still coming up, finds nothing, falls back to
    opc.tcp://localhost:4840 and retries that endpoint for as long as it runs. Only a
    restart finds the controller. OPCUA_DISCOVERY_INTERVAL_S is parsed and only warned
    about. Follows feat(opcua): read-only PLC network discovery -> auto_browse #509 and Config-less discovery: device-derived identity, A&C faults, AccessLevel writable, runtime fault triggers #544, and the same shape of fix as the ADS re-probe.
  2. PLC_COMMS_LOST raised before a restart stays CONFIRMED after the plugin reconnects.
    The clear is sent only from the poller's reconnect arm and only when this process was
    the one that raised it, so after a restart the first connect succeeds inside
    set_context() and nothing clears the persisted fault. Follows opcua: raise a comms-lost fault when the OPC UA connection drops #496 and gateway: a comms-loss fault can never get a freeze frame #564.
  3. A freeze-frame on a plugin-backed entity carries an empty topic and an empty
    message_type, correctly (the values are not a ROS message), which leaves no field
    at all naming where the values came from. Two such frames from different bridges are
    indistinguishable, and so is a frame from a topic capture whose metadata went missing.
  4. The gateway lists the nodes it runs inside its own process (<gateway>_sub,
    <gateway>_fault_clients, <gateway>_lifecycle_state_reader, and the gateway node
    itself) as apps. None of them starts with an underscore, so the hidden-node rule does
    not cover them, and the self-node predicate used for the start-up peer count is not
    applied to app discovery.

Two more came out of fixing the above and belong with them, because they reach an
operator on the same deployment.

  1. The per-entity SOVD DELETE /{entity}/faults/{code} route lands on the plugin's
    FaultProvider::clear_fault for a plugin-owned entity, which sends
    skip_correlation_auto_clear off. An operator scoped to one entity can therefore
    cascade-clear correlated symptoms reported by apps in other entities, which is the
    opposite of what the ClearFault contract documents for scoped routes.
  2. A start-up discovery sweep is not cancellable. It runs inside set_context(), before
    the executor exists, so a SIGTERM during a wide sweep waits the sweep out.

Proposed solution

What should hold afterwards:

  1. A gateway started before its server adopts that server without a restart, within two
    rescan intervals, and its SOVD component identity is re-derived from the device
    instead of staying pinned to the fallback endpoint.
  2. An explicit interval_s: 0 keeps discovery on with the start-up scan only, an unset
    interval_s takes the 30 s default, and a negative value is refused. The cadence is
    measured from the end of a sweep and the reconnect backoff does not stretch it.
  3. PLC_COMMS_LOST is cleared on every successful connect, including a first connect in
    a process that never raised it.
  4. A link-state clear and a scoped operator clear set skip_correlation_auto_clear. A
    clear the device itself reports still cascades.
  5. A freeze-frame on a plugin entity carries x-medkit.source naming the capture path,
    and the field is absent (not empty) on a fault-manager frame taken from a real topic.
  6. /api/v1/apps on a gateway with the plugin loaded lists none of the gateway's own
    in-process nodes, and a peer named <node>_monitor or <node>2 is still listed.
  7. A SIGTERM ends a start-up or rescan sweep within one probe.

How to prove it:

# Unit and lint
./scripts/test.sh
./scripts/test.sh lint

# Docker, from the plugin's docker directory
bash scripts/run_integration_tests.sh
bash scripts/run_alarm_tests.sh

# The start-up race, from the repo root
bash src/ros2_medkit_plugins/ros2_medkit_opcua/docker/scripts/run_discovery_race_test.sh

The race scenario is the falsifying one. On a build without the fix it stops at the
adoption step with endpoint still 'opc.tcp://localhost:4840' (connected='False') after 50s. With the fix it runs through to the config-less rename, and it also asserts the
container never restarted, since a restart would satisfy the endpoint check while
proving nothing. The existing docker suites all pin OPCUA_ENDPOINT_URL and so
short-circuit discovery, which is why none of them can see the start-up race.

Additional context

Size: M. Plugin lifecycle and the fault path, plus two gateway-side changes that land on
plugin entities, with one breaking config change (interval_s: 0) announced in the
plugin changelog.

Where the work lives: opcua_plugin.cpp and opcua_poller.cpp (rescan on the reconnect
arm, endpoint adoption, identity re-derivation, the connect-time clear, the clear origin,
the pending-dispatch buffer), network_discovery.cpp (the cancel predicate and the
report levelling), gateway_node.cpp (one self-node predicate for the peer count and the
app filter), entity_freeze_frame_capture.cpp and http/handlers/fault_handlers.cpp
(the frame's capture path and x-medkit.source), the plugin README, docs/api/rest.rst,
docs/tutorials/snapshots.rst, the plugin CHANGELOG.rst, unit coverage in the plugin
and gateway suites, and a new docker scenario at
src/ros2_medkit_plugins/ros2_medkit_opcua/docker/scripts/run_discovery_race_test.sh.

Reproduced against an OpenPLC server on a controller reached by network discovery with
no node map. Related closed work: #496 (comms-lost raised), #564 (comms-loss fault
without a freeze frame), #478 (active-condition replay on reconnect), and the merged
discovery work in #509 and #544. Fault storage is untouched.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    discoveryDiscovery endpoints or strategiesenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions