diff --git a/docs/api/messages.rst b/docs/api/messages.rst index 1b796d66e..e6b405ade 100644 --- a/docs/api/messages.rst +++ b/docs/api/messages.rst @@ -294,6 +294,63 @@ Retrieve diagnostic snapshots captured at fault occurrence time. See :doc:`/tutorials/snapshots` for detailed usage. +SetPlannedStop.srv / GetPlannedStop.srv +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Declare, withdraw and read a planned stop on the fault manager. + +.. code-block:: text + + # SetPlannedStop.srv + bool active # true declares a planned stop, false withdraws it + string reason # why the plant is stopped + string declared_by # who declared the transition + --- + bool success # true when the request was applied + string message # status or error description + bool was_active # the state of the switch before this request + + # GetPlannedStop.srv + --- + bool active # whether a planned stop is declared + string reason # the reason given; retained after the withdrawal + string declared_by # who declared it; retained after the withdrawal + builtin_interfaces/Time since # when it was declared + builtin_interfaces/Time ended_at # when it was withdrawn; zero while one is in force + +While a planned stop is on, it owns every fault cycle that *starts* - a new fault, +one raised again after being cleared, or one that fails again after healing. An +owned fault is reported, debounced, confirmed, captured and audited unchanged, and +is marked as muted: absent from the default ``ListFaults`` response, counted in +``muted_count``, and listed under ``muted_faults`` with ``rule_id: planned_stop`` +when ``include_muted`` is set. A cycle that started before the stop is untouched. + +Publication matches a rule-muted symptom exactly: ``EVENT_CONFIRMED`` and +``EVENT_UPDATED`` are withheld whichever kind of report produced them, +``EVENT_CLEARED`` is published as usual. + +Withdrawing the stop does two things that do not cover the same faults. It *unmutes* +every fault it owns whose entry is the stop's own - a fault a hierarchical rule has +since claimed stays muted, and returns to the stop's mute if that rule's root cause +is acknowledged while the stop is still on. It *announces* the subset of those that +is CONFIRMED and that no live cluster is hiding, one ``EVENT_CONFIRMED`` each. + +An auto-cluster rule hides a non-representative member without an entry of its own - +it suppresses that member's events on each report - so such a member is unmuted and +NOT announced, with nothing written in the stop's place. Afterwards the burst matches +one that never met a planned stop in ``muted_faults``, in the counts, in the cluster +listing and in the audit log, but not in the event stream: a confirmation that fell +inside the stop and behind a cluster is never announced. What is announced is the +representative, when the stop owned its cycle. A cluster hides only the reports that +fall inside its ``window_ms``; one after that starts a new burst, and the fault is +announced at the switch-off like any owned fault. Cluster membership is not +persisted, so a stop that spanned a restart releases every fault the store says it +owns and announces the CONFIRMED ones among them. The audit records exist only when +``audit_log.enabled`` is set, which it is not by default. + +See :doc:`/config/fault-manager` for the configuration that decides whether the +declaration survives a restart and whether the transitions are audited. + MedkitDiscoveryHint.msg ~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/config/fault-manager.rst b/docs/config/fault-manager.rst index 400de56c0..3a089cbdb 100644 --- a/docs/config/fault-manager.rst +++ b/docs/config/fault-manager.rst @@ -608,12 +608,143 @@ Fault correlation identifies root causes and filters symptom faults. - Path to YAML file defining correlation rules. * - ``correlation.cleanup_interval_sec`` - ``5.0`` - - Interval for running correlation cleanup tasks. + - Interval for running correlation cleanup tasks. The correlation engine is + constructed on every fault manager, configured rules or not, because the + planned-stop switch needs none - so this timer runs everywhere. With no + rules loaded a tick takes the engine's mutex, walks two empty containers and + re-asserts an empty ownership set: measured at ~190 ns per call, so ~2.3 us + of CPU per minute at the 5 s default. .. seealso:: :doc:`/tutorials/fault-correlation` for correlation rule syntax and examples. +Planned Stop +------------ + +A planned stop is declared at runtime, not configured: an operator calls +``~/set_planned_stop`` (or the matching SOVD operation on the ``fault_manager`` +entity) with a reason and a declarer, and calls it again with ``active: false`` +when the plant is back up. There is no parameter for it - a stop has a reason and +an audit trail, and a parameter has neither. + +.. code-block:: bash + + ros2 service call /fault_manager/set_planned_stop ros2_medkit_msgs/srv/SetPlannedStop \ + "{active: true, reason: 'line 3 quarterly maintenance', declared_by: 'shift_lead'}" + +Two configuration choices decide how much the switch can do: + +.. list-table:: + :header-rows: 1 + :widths: 35 15 50 + + * - Parameter + - Default + - Effect on the planned stop + * - ``storage_type`` + - ``sqlite`` + - The declaration, and the per-fault ownership flags behind it, are stored + beside the faults, so both survive a restart. With ``memory`` they live only + for the process, and a restart inside a stop ends it. A correlation rule's + mute is not persisted on either backend, so a rule-muted fault comes back + unmuted after a restart unless the stop owns it. + * - ``audit_log.enabled`` + - ``false`` + - **Off by default, so a stock manager records no audit row for the switch + at all.** When on, each transition is recorded as ``planned_stop_started`` + / ``planned_stop_ended`` under the ``__audit__`` fault code, with that + transition's own reason and declarer. Recorded even under + ``audit_log.transitions: confirmed_only``. + +While the stop stands, it owns every fault cycle that *starts*: a new fault, one +raised again after being cleared, or one that fails again after healing. An owned +fault is reported, debounced, confirmed, captured and audited unchanged, and is +marked as muted rather than dropped: absent from the default fault list, counted in +``muted_count``, and listed under ``muted_faults`` with ``rule_id: planned_stop`` +when muted entries are requested. A fault that was already up when the stop was +declared is left alone - its confirmation has already been announced, and reporters +re-send FAILED for as long as the condition holds. + +Ownership is a flag on the fault row, written by the report that starts the cycle +and in the same transaction, so it survives a restart. While the manager runs, the +engine holds its own copy of that set and the switch-off works from it; the stored +flags are what a startup reads back, and they are dropped once the switch-off has +announced what it released. The mute is derived from ownership: a hierarchical rule +muting an owned fault overlays the stop rather than taking the fault from it, the +withdrawal leaves a rule-held fault alone, and when the rule lets go - which means +its root cause being acknowledged, the one path that drops a symptom's entry - the +fault is muted by the stop again. A rule whose window has closed keeps its entry +until then. + +A rule outranks the stop at the switch-off as well. A fault an active +``show_as_single`` cluster still hides is not announced, and nothing is written in +the stop's place: a cluster hides a member by suppressing that member's events on +every report, never by an entry in the muted list. + +Afterwards the burst is indistinguishable from one that never met a planned stop in +the muted list, in ``muted_count``, in the cluster listing and in the audit log. It +is NOT indistinguishable in the event stream, which is the point of the switch: a +confirmation that fell inside the stop and behind a cluster is never announced, +where the same fault outside a stop would have been announced if it confirmed before +its cluster reached ``min_count``. What the stop withheld, it withholds for good. + +The representative is announced if the stop owned its cycle; one whose cycle +predates the stop was announced when it confirmed and is not announced again; a +member the cluster is still hiding stays hidden, and the one promoted when the +representative is acknowledged is heard from again from that point on. A fault that +joins a cluster whose membership has since fallen below ``min_count`` is not one of +the members that cluster shows as a single line: it is announced and updated like +any fault of its own. + +*Still hiding* is bounded by the rule's ``window_ms``. A cluster hides the reports +that fall inside its window; a report after it starts a new burst, with the lapsed +cluster dropped and the reporting fault as the representative of a fresh one, so +that fault is no longer folded into the old line and the switch-off announces it +like any owned fault. This is the cluster's own behaviour, not the stop's: with no +stop in force the same late report publishes an update. + +``min_count`` gates whether a cluster FORMS, not how long it hides: once formed, a +cluster folds into the representative the members it formed with, for as long as its +window holds them, until the last of them is acknowledged and it dissolves, so a +burst that shrinks back below the threshold does not start announcing its members +again. A cluster that never reached ``min_count``, +or one configured without ``show_as_single``, hides nobody and its members are +released as usual. + +The cluster hold does not survive a restart. Ownership is persisted and cluster +membership is not, so a cluster only holds faults it formed from reports the running +process saw. If the stop spanned a reboot, the switch-off releases every fault the +store says the stop owns and announces the CONFIRMED ones among them, cluster or no +cluster. + +A muted fault is published exactly as a rule-muted symptom is: ``EVENT_CONFIRMED`` +and ``EVENT_UPDATED`` are withheld whichever kind of report produced them, while +``EVENT_CLEARED`` - the fault healing, or being acknowledged - is published as +usual. + +Withdrawing the stop does two things that do not cover the same faults. It *unmutes* +every fault the stop owns whose muted-list entry is the stop's own, so a fault a +hierarchical rule has since claimed stays muted and everything else leaves +``muted_count``. It *announces* the subset of those that is CONFIRMED and that no +live cluster is hiding, one ``EVENT_CONFIRMED`` each. A cluster-hidden member is +therefore unmuted and not announced. + +``~/get_planned_stop`` keeps serving the declaration after the withdrawal, with +``ended_at`` stamped, so the reason stays readable once the plant is back up. A +request the store cannot record answers ``success: false`` and changes nothing. + +Over HTTP the switch is an operation on the fault manager's App entity +(``POST /apps/fault_manager/operations/set_planned_stop/executions``), which +exists automatically in ``runtime_only`` discovery. Under ``hybrid`` or +``manifest_only`` the fault manager must be declared in the manifest, or the +entity is not there to address. + +.. seealso:: + + :doc:`/design/ros2_medkit_fault_manager/index` for how the switch composes + with correlation rules. + Complete Example ---------------- diff --git a/src/ros2_medkit_fault_manager/CMakeLists.txt b/src/ros2_medkit_fault_manager/CMakeLists.txt index 064cf44f1..cf0513846 100644 --- a/src/ros2_medkit_fault_manager/CMakeLists.txt +++ b/src/ros2_medkit_fault_manager/CMakeLists.txt @@ -208,6 +208,29 @@ if(BUILD_TESTING) medkit_add_launch_test(test_rosbag_entity_scope test/integration/test_rosbag_entity_scope.test.py TIMEOUT 120 LABELS "integration") + # The planned-stop switch end to end: marking, counting, auditing, capture, and + # the release at switch-off. One case raises 200 faults inside a stop, so the + # budget is well above the 60s default. + medkit_add_launch_test(test_planned_stop test/integration/test_planned_stop.test.py TIMEOUT 240 + LABELS "integration") + + # Kills the manager and lets launch bring it back, so the declaration is read + # from the store by a process that never saw the request that made it. + medkit_add_launch_test(test_planned_stop_restart test/integration/test_planned_stop_restart.test.py + TIMEOUT 180 LABELS "integration") + + # A switch-off killed between the declaration write and the flag clearing: the + # store is left as that crash leaves it and the replacement process has to + # finish the release. Its respawn delay is what the timeout is sized for. + medkit_add_launch_test(test_planned_stop_interrupted_release + test/integration/test_planned_stop_interrupted_release.test.py TIMEOUT 180 LABELS "integration") + + # auto_confirm_after_sec confirms from a timer rather than a report, so the + # mute has to be honoured on that path too. Its own launch, because every other + # planned-stop case would then race a timer that confirms behind its back. + medkit_add_launch_test(test_planned_stop_auto_confirm + test/integration/test_planned_stop_auto_confirm.test.py TIMEOUT 120 LABELS "integration") + # The only suite running above max_bags_per_fault=1: drives confirm / clear / # confirm on one code and asserts both recordings survive and stay separately # addressable. Six occurrences plus their post-roll windows, hence the timeout. diff --git a/src/ros2_medkit_fault_manager/README.md b/src/ros2_medkit_fault_manager/README.md index c12e54c8a..de31326cb 100644 --- a/src/ros2_medkit_fault_manager/README.md +++ b/src/ros2_medkit_fault_manager/README.md @@ -43,6 +43,174 @@ ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \ | `~/list_faults` | `ros2_medkit_msgs/srv/ListFaults` | Query faults with filtering | | `~/clear_fault` | `ros2_medkit_msgs/srv/ClearFault` | Clear/acknowledge a fault | | `~/get_snapshots` | `ros2_medkit_msgs/srv/GetSnapshots` | Get topic snapshots for a fault | +| `~/set_planned_stop` | `ros2_medkit_msgs/srv/SetPlannedStop` | Declare or withdraw a planned stop | +| `~/get_planned_stop` | `ros2_medkit_msgs/srv/GetPlannedStop` | Read the planned-stop declaration | + +## Planned Stop + +Maintenance produces faults that nobody wants paged: a robot on a bench raises +`NODE_UNREACHABLE` for every peer it can no longer see, and the alarm reaches +whoever is on call. The planned stop is the operator saying "this is us, not the +plant". + +```bash +ros2 service call /fault_manager/set_planned_stop ros2_medkit_msgs/srv/SetPlannedStop \ + "{active: true, reason: 'line 3 quarterly maintenance', declared_by: 'shift_lead'}" + +ros2 service call /fault_manager/get_planned_stop ros2_medkit_msgs/srv/GetPlannedStop "{}" +``` + +**Marked, never dropped.** While the stop stands, it *owns* every fault cycle that +starts. An owned fault goes through report, debounce, confirmation, snapshot and +rosbag capture and the audit log exactly as it would otherwise. What changes is +only how it is *shown*: it is registered as muted with `rule_id: planned_stop` and +the pseudo root cause `PLANNED_STOP`, so it is absent from the default +`~/list_faults` response, counted in `muted_count`, and listed under +`muted_faults` when `include_muted` is set. + +**Ownership is the truth, and it is persisted.** A flag on the fault row records it, +so a restart reads back exactly which cycles the stop owns - no timestamp +comparison, which a clock step would break and which cannot tell a rule's mute from +the stop's. It is written by the report that starts the cycle, in the same store +call and the same transaction, so there is no moment at which a cycle that STARTED +inside a stop exists unowned - a moment a crash would turn into a confirmed fault +that no switch-off releases and no restart recognises. (The cycle a fault was +already in when the stop was declared is never owned; the fault itself becomes owned +if it heals and fails again inside the stop. See below.) The mute is derived from +ownership: an owned fault is muted unless a rule's mute overlays it, and when that +overlay ends the fault is muted by the stop again. + +While the process runs, the engine keeps the same set in memory and the switch-off +works from it. The stored flags are the durable copy: they are what a startup reads +back, and they are dropped once the switch-off has announced what it released. + +**Only a cycle that starts inside the stop.** A fault that was already up when the +stop was declared keeps its place in the fault list. Reporters are level-triggered +- `FaultReporter::report()` is called for as long as the condition holds, not once +per transition - so marking on any report would take a standing alarm off the list +and then announce it a second time at the switch-off. A cycle starts on a new +fault, on one raised again after being cleared, and on one that fails again after +healing (the heal published the fault's end, so the next confirmation is fresh +news). `occurrence_count` keeps its own, narrower definition and counts the first +two only. + +**Published exactly like a rule-muted symptom.** `EVENT_CONFIRMED` and +`EVENT_UPDATED` are withheld while the fault is muted, whichever kind of report +produced them - a PASSED that leaves the fault CONFIRMED announces nothing either. +`EVENT_CLEARED` is not withheld: a fault that heals past the healing threshold, or +that is acknowledged, publishes it as any muted fault does. Consumers therefore see +the end of a fault whose start they never heard - that is the existing muting +contract, not something the switch changes. + +**Withdrawing unmutes, and separately announces.** Two different things happen, and +they do not cover the same faults. + +*Unmuted* is every fault the stop owns whose entry in the muted list is the stop's +own. A fault a hierarchical rule has since claimed keeps that rule's entry and stays +muted; everything else the stop owned leaves the muted list, `muted_count` included. + +*Announced* is the subset of those that is CONFIRMED and that no live cluster is +hiding: each publishes a single `EVENT_CONFIRMED`, because that confirmation +happened behind the mute and was never heard, while the condition it reports still +stands on the machine. A fault still short of confirmation announces nothing (there +is nothing yet to announce), and one acknowledged during the stop announces nothing +either. A cluster-hidden member is therefore unmuted and NOT announced - it leaves +the muted list and goes on being folded into its cluster's line. + +**Correlation rules and the stop compose, and the rule outranks it.** A hierarchical +rule's mute is an *overlay* on an owned fault, not a transfer: while the rule holds +it, the withdrawal leaves it alone, and when the rule lets go - which means its root +cause being acknowledged, the one path that drops a symptom's entry - the fault goes +back to being muted by the stop. A rule whose window has closed keeps its entry +until then. A fault whose cycle started before the stop is not owned at all, so the +stop neither hides it nor releases it. + +Cluster rules take the same precedence, and they take it without an entry of their +own: a cluster hides a member by suppressing that member's events on every report, +not by writing it into the muted list. So at the switch-off a member an active +`show_as_single` cluster still hides is simply not announced - the stop's entry +goes and nothing is written in its place. Releasing them all would fire, in one +wave, the storm the rule exists to show as a single line. + +Afterwards the burst is indistinguishable from one that never met a planned stop in +the muted list, in `muted_count`, in the cluster listing and in the audit log. It is +NOT indistinguishable in the event stream, and that is the point of the switch: a +confirmation that happened inside the stop and behind a cluster is never announced, +where the same fault outside a stop would have been announced if it confirmed before +its cluster reached `min_count`. What the stop withheld, it withholds for good. + +Which member is announced at the switch-off: the representative, if the stop owned +its cycle. A representative whose cycle predates the stop was announced when it +confirmed and is not announced again. A member the cluster is still hiding stays +hidden, and the one promoted when the representative is acknowledged is heard from +again from that point on. + +*Still hiding* is bounded by the rule's `window_ms`. A cluster hides the reports that +fall inside its window; a report after it is a new burst - the lapsed cluster is +dropped and the reporting fault becomes the representative of a fresh one - so that +fault is no longer folded into the old line, and the switch-off announces it like any +owned fault. Nothing about the stop causes this: with no stop in force the same late +report publishes an update. + +`min_count` gates whether a cluster FORMS, not how long it hides. Once formed, a +cluster folds into the representative the members it formed with, for as long as its +window holds them, until the last of them is acknowledged and it dissolves - a burst +that shrinks back below the threshold does not start announcing its members again, +and a fault that JOINS below the threshold was never one of them and is announced and +updated like any fault of its own. A cluster that never reached `min_count`, or one +configured without `show_as_single`, hides nobody, so every member of it is released +as usual. + +**The cluster hold does not survive a restart.** Ownership is persisted; cluster +membership is not. A cluster only holds faults it formed from reports this process +saw, so if the stop spanned a reboot the switch-off releases every fault the store +says the stop owns and announces the CONFIRMED ones among them, cluster or no +cluster. Persisting cluster state is not part of this. + +**Both transitions are audited** - when the audit log is on. `audit_log.enabled` +is `false` by default, and with it off the switch writes no audit row at all. With +it on, each transition appends `planned_stop_started` / `planned_stop_ended` +carrying that transition's own reason in `description` and its declarer in +`source_id`, under the `__audit__` fault code the log's own lifecycle markers use, +because the transition is about the installation rather than one fault. They are +recorded whatever `audit_log.transitions` says. A request asking for the state the +switch is already in succeeds, changes nothing and records nothing, so a retried +call cannot manufacture evidence of a stop that never started. A request the store +cannot record answers `success: false` with the reason, and changes nothing. + +**The declaration outlives the stop.** `~/get_planned_stop` keeps serving the +reason, the declarer and the start time after the withdrawal, with `ended_at` +stamped, so "why was line 3 quiet on Friday?" is answerable once the plant is back +up without reading the audit database. + +**The declaration survives a restart** when the SQLite backend is in use: it is a +row in the fault store, so a stop declared on Friday still mutes on Monday. On +startup the manager reads the ownership flags back, so the switch-off after a +reboot releases exactly the cycles the stop owned and announces their +confirmations. A *rule's* mute is not persisted anywhere, so a fault a correlation +rule was muting before a restart comes back unmuted unless the stop owns it - a +pre-existing property of correlation, not of the switch. The in-memory backend has +no file behind it and starts every process with no declaration and no ownership. + +**A switch-off interrupted by a crash is finished at startup.** The withdrawal +writes the declaration first, announces the confirmations it was holding back, and +drops the ownership flags last. A process that dies in between leaves faults owned +by a declaration that is already over; the next startup recognises exactly that, +announces them once a consumer is listening, and clears the flags. + +**Over SOVD.** The gateway maps a node's services to operations on its App entity, +so an operator reaches the switch without any new route: + +```bash +curl -X POST http://localhost:8080/api/v1/apps/fault_manager/operations/set_planned_stop/executions \ + -H 'Content-Type: application/json' \ + -d '{"parameters": {"active": true, "reason": "line 3 maintenance", "declared_by": "shift_lead"}}' +``` + +That works out of the box in `runtime_only` discovery, where every node is an +App. In `hybrid` or `manifest_only` discovery the fault manager has to be +declared in the manifest like any other entity, or there is no `fault_manager` +entity to address. ## Features @@ -55,7 +223,8 @@ ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \ - **Snapshot capture**: Captures topic data when faults are confirmed for debugging (snapshots are deleted when fault is cleared) - **Near-miss series**: Appends one entry per FAILED report that moved the debounce counter without confirming, bounded per fault code and retained when the fault is cleared - **Freeze-frame retention**: One compact JSON freeze-frame per fault code, retained across `clear_fault` (see below) -- **Fault correlation** (optional): Root cause analysis with symptom muting and auto-clear +- **Planned stop**: An operator declares the plant deliberately down, and faults raised while it stands are marked rather than announced (see below) +- **Fault correlation** (optional): Root cause analysis with symptom muting and auto-clear. The correlation *engine* is always constructed, because the planned stop needs no configuration; without a `correlation.config_file` it carries no rules, correlates nothing, and its cleanup timer (`correlation.cleanup_interval_sec`, default 5 s) walks two empty containers - **Tamper-evident audit log** (optional): Append-only, hash-chained record of fault state transitions for verifiable history ## Parameters diff --git a/src/ros2_medkit_fault_manager/design/index.rst b/src/ros2_medkit_fault_manager/design/index.rst index e37d62acb..acc0ca643 100644 --- a/src/ros2_medkit_fault_manager/design/index.rst +++ b/src/ros2_medkit_fault_manager/design/index.rst @@ -186,6 +186,18 @@ Clears (acknowledges) a fault by setting its status to CLEARED. - **Idempotent**: Clearing an already-cleared fault succeeds - **Returns**: ``success=true`` if fault existed, ``success=false`` if not found +~/set_planned_stop and ~/get_planned_stop +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Declares, withdraws and reads the planned stop. + +- **One declaration per manager**: the stop is a fact about the installation, not about a fault +- **Idempotent**: a request for the state the switch is already in succeeds, changes nothing and writes no audit record; ``was_active`` tells the caller which of the two happened +- **Store first**: the declaration is written before anything is muted, unmuted, announced or audited, and a store that refuses the write ends the request with ``success=false`` - a stop the manager could not record must not be one a restart comes back believing in +- **Audited**: when ``audit_log.enabled`` is set (off by default), each real transition appends ``planned_stop_started`` / ``planned_stop_ended`` under the ``__audit__`` fault code, with that transition's own reason and declarer +- **Persisted**: stored in the fault store, so the declaration outlives the process (SQLite backend), and ``~/get_planned_stop`` keeps serving it after the withdrawal with ``ended_at`` stamped +- **Returns**: ``success``, a message, and the state the switch was in before the call + Design Decisions ---------------- @@ -246,6 +258,170 @@ merged per-entity config at startup, logs a warning, and falls back to safe defa healing is disabled, any HEALED row left by a previous (healing-enabled) run is reclassified to CLEARED once at startup so it does not behave inconsistently under the latch. +Planned Stop as a Second Mute Source +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Maintenance produces faults that are real, correctly detected, and not news. The +planned stop marks them instead of dropping them: while it stands, a fault whose +cycle *starts* is registered in the correlation engine as muted with +``rule_id = "planned_stop"`` and the pseudo root cause ``PLANNED_STOP``, and +everything else about it - debounce, confirmation, snapshot and rosbag capture, +the audit records - happens exactly as it would otherwise. Dropping the report +instead was rejected because a fault that outlived the stop would then never be +raised again: reporters are level-triggered, so nothing re-announces a condition +whose report was thrown away. + +**Only a cycle that starts inside the stop.** That same level-triggered shape is +why the mark is gated on the cycle boundary rather than on the report. +``ros2_medkit_diagnostic_bridge`` sends one FAILED per incoming +``DiagnosticArray`` - typically 1 Hz for as long as the condition holds - and +``FaultReporter::report()`` is documented to be called the same way. Marking on any +report would take a fault that was confirmed and announced *before* the stop off +the fault list within a second of the declaration, and announce it a second time at +the switch-off. The boundary is ``report_fault_event``'s own ``is_new`` - a new fault, or one +raised again after being cleared - plus a re-fail out of HEALED, because the manager +publishes ``EVENT_CLEARED`` at the heal and the confirmation that follows is +therefore fresh news. ``occurrence_count`` keeps its own, narrower definition. + +**Ownership is the fact; the mute is derived from it.** The stop owns a fault +CYCLE, and the durable record of that is a flag on the fault row in the store - +not a timestamp comparison. The report that starts the cycle carries it: the store +is told whether a stop is in force, decides inside the same transaction whether +this report started a cycle, and writes the flag with the row. Recorded by a call +that follows the report instead, there is a window in which a fault has confirmed +inside a stop and is not owned, and a process that dies there comes back to a +confirmed fault that no switch-off releases, that no startup recognises as +interrupted, and whose next report announces an update mid-stop. + +The engine keeps the same set in the process (``planned_stop_owned_``) and that is +what a running switch-off works from; the stored flags are what a STARTUP reads +back, and ``clear_planned_stop_owned`` drops them once the switch-off has announced +what it released. Several paths move each of them - the report takes a cycle, the +constructor restores from the store, an acknowledgement and the switch-off give it +up - and they stay in step because every one of THOSE runs on the node's single +thread: the constructor before anything is spinning, and the rest as service +callbacks under the ``rclcpp::spin()`` in ``main.cpp``, none of them on a callback +group of its own, so no two are ever in flight at once. (The node does create a +callback group elsewhere - ``SnapshotCapture`` drains one on a thread of its own - +but nothing on it reads or writes either set.) + +Everything else follows: an owned fault is +muted unless a rule's mute overlays it, and the moment the overlay ends the engine +re-asserts the stop's mute (``reassert_planned_stop_mutes``). An overlay ends when a +cycle ends, which is ``process_clear`` in all three of its forms: on the root cause, +which erases its symptoms' entries; on the symptom itself; and on a symptom the +auto-clear loop takes with its root cause. A window closing is not one of them: +``cleanup_expired`` drops pending root causes and pending clusters and never touches +``muted_faults_``, so it calls the re-assert to cover the ownership set rather than +to hand anything back. A rule therefore borrows a fault +rather than taking it, which is what makes the two features compose in both +directions: the withdrawal leaves a rule-held fault muted, and a rule that lets go +mid-stop hands the fault back instead of dropping it out of the stop for good. + +Deriving the mute also fixes what ``should_mute`` means. A repeat report of a fault +whose rule has stopped matching produces no correlation result at all, so the report +path used to treat it as unmuted and announce an update for a fault the list was +hiding. ``process_fault`` now answers with the state of the mute map, not with what +this particular report matched, and the PASSED path asks the same question rather +than defaulting to "not muted". + +Ownership ends in exactly three places: the fault is acknowledged (both +acknowledgement paths - the correlation clear and the scoped per-entity clear that +skips it - and the store clears the flag with the CLEARED status), the fault is +auto-cleared with its root cause, or the stop is withdrawn. + +**The mark survives a restart** because the flags do. Startup reads them back and +re-registers the mute. Without that, a reboot inside a weekend stop would make the +survivors silently visible and the switch-off would announce none of them: their +confirmations, suppressed before the reboot, would be lost for good. A correlation +rule's mute is not persisted anywhere, so a rule-muted fault comes back unmuted +after a restart unless the stop owns its cycle - a pre-existing property of +correlation rather than something the switch introduces. + +**A withdrawal is ordered so a crash cannot swallow an announcement.** The +declaration is written first (so a store that refuses the write changes nothing, and +a restart never re-arms a stop the operator withdrew), the confirmations are +announced next, and the ownership flags are cleared last. A process that dies in +between leaves faults owned by a declaration that is already over, which is a state +the next startup recognises: it announces them and clears the flags of exactly +those faults. Clearing first would have turned a crash into permanent silence. + +The recovery runs synchronously in the constructor, before the node has a service +or a timer. Deferring it - waiting for a subscriber to match, say - opens a window +in which an operator declares a NEW stop over the same services and new cycles +become owned by it; the deferred work then announces a fault the new stop is +holding and, if it clears ownership wholesale rather than the set it captured, +drops that fault's flag too. Running early costs an announcement that may reach no +subscriber, because nothing has matched a publisher that is milliseconds old and +the events topic is volatile. That is the same accepted loss as every other +publication made at startup, and the released faults are in the default fault list +regardless. + +Withdrawing releases the rest and publishes ``EVENT_CONFIRMED`` once for each +released fault that is CONFIRMED. That republication is the point of marking +rather than dropping: the confirmation happened behind the mute, no consumer of +the event stream ever heard it, and the condition still stands on the machine. +What is withheld is exactly what a rule-muted symptom withholds - ``EVENT_CONFIRMED`` +and ``EVENT_UPDATED`` - while ``EVENT_CLEARED`` is published as usual, so a fault +that heals or is acknowledged inside the stop still reports its end. + +**A rule outranks the stop at the switch-off, clusters included.** A hierarchical +rule holds its symptom through the withdrawal because its mute entry is there to +see. A cluster's ``show_as_single`` is different: it sets ``should_mute`` on the +report and writes no entry, so an owned member carries the stop's entry and would be +released with everything else - announcing, in one wave, exactly the storm the rule +exists to fold into a single alarm. The release therefore asks the cluster the same +question the report path asks (``cluster_hides``: an ACTIVE cluster, a +``show_as_single`` rule, and not the representative) and, when the answer is yes, +erases the stop's entry without announcing the fault and without writing anything in +its place. + +A cluster-attributed entry there would be wrong in two ways. It is a remembered +answer to a question whose inputs keep moving - +membership and the representative both change - so acknowledging the representative +afterwards leaves the promoted member hidden behind an entry naming a fault that is +gone, with nothing but clearing it to remove the entry. And it makes ``muted_count`` +and the default fault list depend on whether a stop happened to be in force earlier, +for a cluster that is otherwise identical, because a cluster writes no entry at any +other time. ``cluster_hides`` is a verdict, asked fresh on every report and once +more at the release; it is never stored. + +Ownership ends either way. What is announced is the representative, if the stop +owned its cycle. ``min_count`` gates whether a cluster FORMS, not how long it hides: +a formed cluster holds its members until the last one is acknowledged and it +dissolves, so a burst that shrinks below the threshold keeps hiding. A cluster that +never reached ``min_count``, or one whose rule groups without hiding, has no verdict +to outrank the stop with, and every member of it is released. + +Promotion has to read the ACTIVE cluster, not the pending twin. ``cleanup_expired`` +drops the pending cluster when its window closes and keeps the active one, so once +the window is behind the burst the active cluster is the only record of who is left. +Promoting from a twin that is no longer there leaves ``representative_code`` naming +the acknowledged fault, and every remaining member is then hidden by a cluster whose +representative can never be reported again - at the switch-off nobody is released. +The member severities therefore live on ``ClusterData`` rather than beside the +pending cluster, so a highest-severity rule can still name a replacement. + +**The cluster hold does not survive a restart.** Ownership is persisted; cluster +membership is not, and persisting it is a separate feature this does not build. A +cluster only holds faults it formed from reports the running process saw, so a stop +that spanned a reboot releases and announces every fault the store says it owns. + +The switch is a service rather than a parameter because it carries a reason and a +declarer and produces an audit record, none of which a parameter can do; and it is +not a new HTTP route because the gateway already exposes a node's services as SOVD +operations on its App entity. + +One cost falls on every deployment, correlation or not: the engine is now always +constructed, so the ``correlation.cleanup_interval_sec`` timer (5 s by default) +runs on every fault manager. With no rules loaded a tick takes the engine's mutex, +walks two empty containers - ``pending_root_causes_`` and ``pending_clusters_`` - +and re-asserts an empty ownership set. Measured at ~190 ns per call (1e6 calls on a +rules-free engine, three runs: 191.2, 194.7, 190.3 ns), which at the default +interval is ~2.3 us of CPU per minute. Building the engine lazily when a stop is +first declared would put a second construction path under the service handler to +save that. + Rosbag Black-Box Recording ~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/correlation/correlation_engine.hpp b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/correlation/correlation_engine.hpp index f1092138a..47f8a8b38 100644 --- a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/correlation/correlation_engine.hpp +++ b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/correlation/correlation_engine.hpp @@ -19,6 +19,7 @@ #include #include #include +#include #include #include @@ -66,6 +67,23 @@ struct MutedFaultData { std::string root_cause_code; std::string rule_id; uint32_t delay_ms{0}; + /// Whether the planned stop wrote this entry, as opposed to a correlation rule. + /// Not part of the ListFaults wire shape: it is how the engine tells its own + /// entry from a rule's overlay without matching on the rule id, which an + /// operator's configuration could collide with. + bool by_planned_stop{false}; +}; + +/// What withdrawing a planned stop did. The two are not the same set: every fault the +/// stop alone was muting loses that entry, while one a live cluster is still hiding is +/// unmuted without being announced, because the cluster goes on folding it into its +/// representative's line. +struct EndPlannedStopResult { + /// Fault codes whose suppressed confirmation the caller should publish now. + std::vector to_announce; + + /// How many faults left the muted list, `to_announce` included. + size_t unmuted{0}; }; /// Information about an active cluster (for ListFaults response) @@ -77,6 +95,10 @@ struct ClusterData { std::string representative_code; std::string representative_severity; std::vector fault_codes; + /// Severity of each member, keyed by fault code. Carried on the cluster rather than + /// beside it so a cluster that outlives the window it formed in can still name the + /// member a highest-severity rule would promote. + std::map fault_severities; std::chrono::system_clock::time_point first_at; std::chrono::system_clock::time_point last_at; }; @@ -87,10 +109,22 @@ struct ClusterData { /// - Whether they should be muted (symptoms of a root cause) /// - Whether they are root causes that should collect symptoms /// - Whether they form part of an auto-detected cluster +/// - Whether a planned stop is in force, which mutes every fault whose cycle +/// starts while it is on /// /// Thread-safe: all public methods can be called from multiple threads. class CorrelationEngine { public: + /// Pseudo root cause recorded against a fault the planned stop muted. It is + /// not a fault code any reporter can raise; it names the declaration as the + /// reason the fault is not being announced. + static constexpr const char * kPlannedStopRootCause = "PLANNED_STOP"; + + /// Rule id recorded against a fault the planned stop muted. The switch is not + /// a configured rule, so it carries a fixed id of its own and a consumer of + /// `muted_faults` can tell an operator's declaration from a correlation rule. + static constexpr const char * kPlannedStopRuleId = "planned_stop"; + /// Create correlation engine from configuration /// @param config Correlation configuration (must be enabled and valid) explicit CorrelationEngine(const CorrelationConfig & config); @@ -99,9 +133,17 @@ class CorrelationEngine { /// @param fault_code The fault code /// @param severity The fault severity (for representative selection) /// @param timestamp When the fault occurred + /// @param cycle_started Whether this report STARTED a fault cycle (a new fault, + /// or one raised again after being cleared) as opposed to a repeat of a + /// condition that is already up. Only a cycle that starts while a planned + /// stop is declared is marked by it: reporters are level-triggered and + /// re-send FAILED for as long as the condition holds, so muting on any + /// report would swallow a fault that was announced before the stop began. + /// Correlation rules are unaffected and match on every report. /// @return Processing result indicating whether to mute, correlations, etc. ProcessFaultResult process_fault(const std::string & fault_code, const std::string & severity, - std::chrono::steady_clock::time_point timestamp = std::chrono::steady_clock::now()); + std::chrono::steady_clock::time_point timestamp = std::chrono::steady_clock::now(), + bool cycle_started = true); /// Process a fault being cleared /// @param fault_code The fault code being cleared @@ -131,7 +173,67 @@ class CorrelationEngine { /// Called periodically to remove old state void cleanup_expired(); + /// Declare a planned stop. From here every fault reported through + /// process_fault is muted unless a rule already mutes it, and stays muted + /// until the stop ends or the fault is cleared. Idempotent. + void begin_planned_stop(); + + /// Withdraw the planned stop and release the faults it alone was muting. + /// A fault a hierarchical rule has since claimed keeps that rule's mute and is + /// neither unmuted nor announced; nor is one that was cleared while the stop was on. + /// A fault a live cluster is hiding is unmuted but not announced. + /// @return what was unmuted and what to announce; see EndPlannedStopResult + EndPlannedStopResult end_planned_stop(); + + /// Whether a planned stop is currently declared. + bool planned_stop_active() const; + + /// End the planned stop's ownership of one fault cycle, and take its mute with + /// it. A rule's overlay is left alone. Called when a fault is acknowledged on a + /// path that does not run the correlation clear (a scoped per-entity DELETE), + /// so an acknowledged fault never stays counted as muted. + void release_planned_stop_ownership(const std::string & fault_code); + + /// Record that the planned stop owns a fault cycle, without a report driving it. + /// This is how a restart rebuilds the mute from the flags the store kept: the + /// engine's own record lived in the process that is gone, and without it the + /// switch-off would neither release nor announce those faults. + void restore_planned_stop_ownership(const std::string & fault_code); + + /// Every fault cycle the planned stop currently owns, including those a rule's + /// mute is overlaying. + std::vector planned_stop_owned_codes() const; + private: + /// The correlation half of process_fault: rules, clusters and their muting, + /// with no planned-stop involvement. Caller holds mutex_. + ProcessFaultResult correlate(const std::string & fault_code, const std::string & severity, + std::chrono::steady_clock::time_point timestamp); + + /// Put back the planned stop's mute for every fault it owns that has no entry + /// left - a rule's overlay ended (its root cause was acknowledged, its window + /// closed) and the fault is the stop's again. Ownership is the truth; this is + /// what keeps the mute map derived from it. Caller holds mutex_. + void reassert_planned_stop_mutes(); + + /// Write the planned stop's mute entry for one owned fault. Caller holds mutex_. + void mute_as_planned_stop(const std::string & fault_code); + + /// Whether an ACTIVE cluster is currently folding this fault into its representative: + /// the fault belongs to a cluster that reached min_count, the rule says + /// `show_as_single`, and the fault is not the representative. A verdict, not an entry - + /// the cluster path writes nothing to `muted_faults_`, it suppresses the fault's event + /// on each report. The single place that decides it, so the report path and the + /// planned stop's switch-off cannot answer differently. Caller holds mutex_. + bool cluster_hides(const std::string & fault_code) const; + + /// The configured rule with this id, or nullptr. Caller holds mutex_. + const CorrelationRule * find_rule(const std::string & rule_id) const; + + /// Name the member the cluster's rule would pick as representative, from the members + /// the cluster currently has. A no-op on an empty cluster. Caller holds mutex_. + void promote_representative(ClusterData & cluster); + /// Check if fault matches a root cause pattern in any hierarchical rule /// @return Rule ID if matched, empty optional otherwise std::optional try_as_root_cause(const std::string & fault_code); @@ -177,7 +279,6 @@ class CorrelationEngine { struct PendingCluster { ClusterData data; std::chrono::steady_clock::time_point steady_first_at; - std::map fault_severities; ///< fault_code -> severity }; /// Pending clusters being formed (rule_id -> cluster data) @@ -187,6 +288,16 @@ class CorrelationEngine { /// Counter for cluster ID generation uint64_t cluster_counter_{0}; + /// Whether an operator has declared a planned stop. + bool planned_stop_active_{false}; + + /// Fault cycles the planned stop OWNS: they started while it was declared. This + /// is the runtime mirror of the flag the fault store keeps, and it is the truth + /// the mute map is derived from - a rule's mute overlays an owned fault without + /// taking it, and when the overlay ends the fault is the stop's again. A code + /// leaves the set when the fault is acknowledged or the stop is withdrawn. + std::set planned_stop_owned_; + mutable std::mutex mutex_; }; diff --git a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_audit_log.hpp b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_audit_log.hpp index 967090690..40c9d9457 100644 --- a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_audit_log.hpp +++ b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_audit_log.hpp @@ -54,6 +54,12 @@ struct AuditEvent { int64_t occurred_at_ns{0}; ///< wall-clock timestamp of the transition }; +/// Fault code carried by records that describe the INSTALLATION rather than one +/// fault - the log's own activation markers and the planned-stop transitions. One +/// sentinel for all of them, so a reader filtering them out has a single value to +/// filter on. +constexpr const char * kAuditMarkerFaultCode = "__audit__"; + /// Canonical transition kinds. Stored verbatim, so they are part of the hash. constexpr const char * kTransitionOccurred = "occurred"; constexpr const char * kTransitionConfirmed = "confirmed"; @@ -67,6 +73,14 @@ constexpr const char * kTransitionHealed = "healed"; /// its own start and stop. constexpr const char * kTransitionLoggingActivated = "logging_activated"; constexpr const char * kTransitionLoggingDeactivated = "logging_deactivated"; +/// Planned-stop lifecycle: an operator declared, or withdrew, a deliberate stop +/// of the plant. Like the logging markers these describe the installation rather +/// than one fault, so they carry kAuditMarkerFaultCode and are appended +/// independently of the per-fault transition filter. The reason travels in +/// `description` and the declarer in `source_id` - that transition's own reason +/// and declarer, so a withdrawal records who ended the stop, not who began it. +constexpr const char * kTransitionPlannedStopStarted = "planned_stop_started"; +constexpr const char * kTransitionPlannedStopEnded = "planned_stop_ended"; // NOTE: there is deliberately no "ack" kind. The open fault_manager has no // acknowledge action separate from clearing: ~/clear_fault IS the acknowledge, // and it is recorded as kTransitionCleared (clear == ack). A separate "ack" kind diff --git a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_manager_node.hpp b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_manager_node.hpp index 9157102d9..f46af823d 100644 --- a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_manager_node.hpp +++ b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_manager_node.hpp @@ -18,8 +18,10 @@ #include #include #include +#include #include #include +#include #include "rclcpp/rclcpp.hpp" #include "ros2_medkit_fault_manager/capture_thread_pool.hpp" @@ -32,12 +34,14 @@ #include "ros2_medkit_msgs/msg/fault_event.hpp" #include "ros2_medkit_msgs/srv/clear_fault.hpp" #include "ros2_medkit_msgs/srv/get_fault.hpp" +#include "ros2_medkit_msgs/srv/get_planned_stop.hpp" #include "ros2_medkit_msgs/srv/get_rosbag.hpp" #include "ros2_medkit_msgs/srv/get_snapshots.hpp" #include "ros2_medkit_msgs/srv/list_faults.hpp" #include "ros2_medkit_msgs/srv/list_faults_for_entity.hpp" #include "ros2_medkit_msgs/srv/list_rosbags.hpp" #include "ros2_medkit_msgs/srv/report_fault.hpp" +#include "ros2_medkit_msgs/srv/set_planned_stop.hpp" namespace ros2_medkit_fault_manager { @@ -98,6 +102,12 @@ class FaultManagerNode : public rclcpp::Node { audit_transition(transition, fault, "test", 0); } + /// Test-only: register the planned stop's ownership of a fault in the engine, + /// the way startup does when it reads the flags back. + void restore_planned_stop_ownership_for_test(const std::string & fault_code) { + correlation_engine_->restore_planned_stop_ownership(fault_code); + } + /// Get the storage type being used const std::string & get_storage_type() const { return storage_type_; @@ -119,6 +129,17 @@ class FaultManagerNode : public rclcpp::Node { /// @return true if entity_id matches any source static bool matches_entity(const std::vector & reporting_sources, const std::string & entity_id); + protected: + /// Construct with a storage backend the caller supplies instead of the one the + /// parameters describe. + /// + /// The backend is handed over BEFORE the node builds anything that borrows it - + /// snapshot and rosbag capture hold a raw `FaultStorage *` - so this cannot leave + /// a dangling pointer the way swapping the store at runtime would. Protected + /// rather than public because the only callers are tests driving the failure + /// paths of a store that rejects a write, which a healthy disk never produces. + FaultManagerNode(const rclcpp::NodeOptions & options, std::unique_ptr storage); + private: /// Create storage backend based on configuration std::unique_ptr create_storage(); @@ -151,6 +172,14 @@ class FaultManagerNode : public rclcpp::Node { void handle_list_rosbags(const std::shared_ptr & request, const std::shared_ptr & response); + /// Handle SetPlannedStop service request + void handle_set_planned_stop(const std::shared_ptr & request, + const std::shared_ptr & response); + + /// Handle GetPlannedStop service request + void handle_get_planned_stop(const std::shared_ptr & request, + const std::shared_ptr & response); + /// Handle ListFaultsForEntity service request void handle_list_faults_for_entity(const std::shared_ptr & request, @@ -164,10 +193,31 @@ class FaultManagerNode : public rclcpp::Node { /// @param config SnapshotConfig to populate with loaded values void load_snapshot_config_from_yaml(const std::string & config_file, SnapshotConfig & config); - /// Initialize correlation engine from configuration file - /// @return CorrelationEngine instance if enabled and config is valid, nullptr otherwise + /// Finish a switch-off a previous process did not: announce the faults still + /// owned by a declaration that is already withdrawn, and drop exactly their + /// flags. Runs from the constructor, before this node has a service or a timer + /// that could declare a new stop underneath it. + void finish_interrupted_release(); + + /// Publish EVENT_CONFIRMED for each of these faults that is CONFIRMED, which is + /// what releasing a fault from the planned stop means to a consumer of the + /// stream. + /// @return how many confirmations were announced + size_t announce_released(const std::vector & fault_codes); + + /// Build the correlation engine. + /// + /// Never null. The engine also owns the planned-stop mute, which an operator can + /// declare on any fault manager, so it exists whether or not correlation rules are + /// configured; without a config file (or with one that fails to load) it simply + /// carries no rules and correlates nothing. std::unique_ptr create_correlation_engine(); + /// Parse the correlation config file named by `correlation.config_file`. + /// @return the parsed config, or nullopt when correlation is not configured, + /// explicitly disabled, or the file is missing or invalid + std::optional load_correlation_config(); + /// Publish a fault event to the events topic /// @param event_type One of FaultEvent::EVENT_CONFIRMED, EVENT_CLEARED, EVENT_UPDATED /// @param fault The fault data associated with this event @@ -203,6 +253,26 @@ class FaultManagerNode : public rclcpp::Node { void audit_transition(const char * transition, const ros2_medkit_msgs::msg::Fault & fault, const std::string & source_id, int64_t occurred_at_ns); + /// Append one already-built record to the audit log, carrying the shared + /// failure handling (health signals, fail-closed rethrow). No-op when the log + /// is off. Bypasses the per-fault transition filter, so a caller that has + /// decided a record must be written gets it written. + void append_audit_event(const AuditEvent & event); + + /// Record a planned-stop transition. Written whatever `audit_log.transitions` + /// says, for the same reason the logging markers are: the switch describes the + /// installation, and the evidence chain has to show when the plant was + /// declared stopped. + /// @param state The declaration as it stands after the transition (for its status) + /// @param reason Why THIS transition was made + /// @param declared_by Who made it + void audit_planned_stop(const char * transition, const PlannedStopState & state, const std::string & reason, + const std::string & declared_by, int64_t occurred_at_ns); + + /// Handed to create_storage() when a caller supplied a backend. Empty in every + /// normal construction, and moved from exactly once. + std::unique_ptr storage_override_; + std::string storage_type_; std::string database_path_; int32_t confirmation_threshold_{-1}; @@ -237,6 +307,8 @@ class FaultManagerNode : public rclcpp::Node { rclcpp::Service::SharedPtr get_rosbag_srv_; rclcpp::Service::SharedPtr list_rosbags_srv_; rclcpp::Service::SharedPtr list_faults_for_entity_srv_; + rclcpp::Service::SharedPtr set_planned_stop_srv_; + rclcpp::Service::SharedPtr get_planned_stop_srv_; rclcpp::TimerBase::SharedPtr auto_confirm_timer_; /// Timer for periodic cleanup of expired correlation data @@ -253,9 +325,15 @@ class FaultManagerNode : public rclcpp::Node { /// shared_ptr to allow safe capture-by-value in the pool's capture jobs. std::shared_ptr rosbag_capture_; - /// Correlation engine for fault correlation/muting (nullptr if disabled) + /// Correlation engine for fault correlation/muting. Always present: it also + /// holds the planned-stop mute, which needs no configuration. std::unique_ptr correlation_engine_; + /// The planned-stop declaration in force, mirrored from the store at startup and + /// written back on every transition. Only the service handlers touch it, and they + /// run on the node's single-threaded executor, so it needs no lock of its own. + PlannedStopState planned_stop_; + /// Bounded pool that runs capture jobs off the service thread (issue #441). /// Declared after snapshot_capture_/rosbag_capture_ so it is destroyed first. std::unique_ptr capture_pool_; diff --git a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_storage.hpp b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_storage.hpp index 3bc60dc35..4399cb0e4 100644 --- a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_storage.hpp +++ b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_storage.hpp @@ -98,6 +98,12 @@ struct FaultState { std::string status; std::set reporting_sources; + /// Whether the planned stop owns this fault cycle: it started while a stop was + /// declared. Persisted, because the declaration outlives the process and the + /// switch-off has to know which faults it is releasing. Cleared when the fault + /// is acknowledged and when the stop is withdrawn. + bool planned_stop_owned{false}; + // Debounce state (internal, not exposed in Fault.msg) int32_t debounce_counter{0}; ///< FAILED decrements (-1), PASSED increments (+1) rclcpp::Time last_failed_time{}; ///< Timestamp of last FAILED event @@ -195,6 +201,21 @@ struct RosbagFileInfo { int64_t created_at_ns{0}; ///< Timestamp when bag was created }; +/// The operator's planned-stop declaration, as the store holds it. +/// +/// One declaration per fault manager, not one per fault: it says the plant is +/// deliberately down, which is a fact about the installation. It is persisted so +/// a stop declared on Friday is still in force after a Saturday reboot. +struct PlannedStopState { + bool active{false}; + std::string reason; ///< Why the plant is stopped; retained after the withdrawal + std::string declared_by; ///< Who declared it; retained after the withdrawal + int64_t since_ns{0}; ///< Wall-clock time of the declaration; 0 when none was ever made + /// Wall-clock time the declaration was withdrawn; 0 while one is in force. The + /// row outlives the stop so the reason stays readable after the plant is back up. + int64_t ended_at_ns{0}; +}; + /// Abstract interface for fault storage backends class FaultStorage { public: @@ -214,11 +235,21 @@ class FaultStorage { /// @param source_id Reporting source identifier /// @param timestamp Current time for tracking /// @param config Debounce configuration to apply for this event (resolved per-entity by the node) + /// @param planned_stop_active Whether a planned stop is declared. When it is, a report that STARTS + /// a fault cycle - a new fault, one raised again after being cleared, or one that fails + /// again after healing - marks that cycle as the stop's, in the same write as the report. + /// Ownership recorded by a separate call afterwards has a window in which a confirmed + /// fault exists unowned, and a process that dies there comes back to a fault no + /// switch-off releases. A report that does not start a cycle never takes ownership and + /// never drops it. + /// The default is repeated on both overrides so a call on a concrete backend means the + /// same as one through this interface. /// @return true if this is a new occurrence (new fault or reactivated CLEARED fault), /// false if existing active fault was updated virtual bool report_fault_event(const std::string & fault_code, uint8_t event_type, uint8_t severity, const std::string & description, const std::string & source_id, - const rclcpp::Time & timestamp, const DebounceConfig & config) = 0; + const rclcpp::Time & timestamp, const DebounceConfig & config, + bool planned_stop_active = false) = 0; /// Get faults matching filter criteria /// @param filter_by_severity Whether to filter by severity @@ -454,6 +485,40 @@ class FaultStorage { return {}; } + /// Replace the planned-stop declaration. Writing a default-constructed state + /// withdraws it. A backend keeps exactly one declaration, so this overwrites + /// rather than appends. + virtual void set_planned_stop(const PlannedStopState & state) = 0; + + /// The planned-stop declaration this store holds. A store that has never been + /// given one answers with a default-constructed (inactive) state. + virtual PlannedStopState get_planned_stop() const = 0; + + /// Every fault the planned stop currently owns, as the store has it. This is the + /// durable record: a STARTUP reads it to rebuild the mute in a process that never + /// saw the reports, and a startup that finds faults owned by a declaration already + /// withdrawn finishes that interrupted release from it. A switch-off in a running + /// manager releases from the engine's own copy of the set and only drops the flags + /// here afterwards. Ownership is recorded rather than derived from a time + /// comparison against the declaration, which cannot survive a clock step and cannot + /// tell a rule's mute from the stop's. + virtual std::vector get_planned_stop_owned() const = 0; + + /// Drop every ownership flag. Called once the switch-off has announced what it + /// released, so a crash before this point leaves the flags for the next startup + /// to finish. Safe as an all-or-nothing sweep there: the withdrawal ends the + /// declaration that owns every flagged fault, and it runs to completion before + /// any other request is served. + /// @return how many faults were released + virtual size_t clear_planned_stop_owned() = 0; + + /// Drop the ownership flags of exactly these faults, leaving every other one + /// alone. What a release captured is what it may clear: a fault flagged after + /// the capture belongs to a later declaration, and wiping it would take a fault + /// out of a stop that is still in force. + /// @return how many of them were owned + virtual size_t clear_planned_stop_owned(const std::vector & fault_codes) = 0; + protected: FaultStorage() = default; FaultStorage(const FaultStorage &) = default; @@ -472,7 +537,8 @@ class InMemoryFaultStorage : public FaultStorage { bool report_fault_event(const std::string & fault_code, uint8_t event_type, uint8_t severity, const std::string & description, const std::string & source_id, - const rclcpp::Time & timestamp, const DebounceConfig & config) override; + const rclcpp::Time & timestamp, const DebounceConfig & config, + bool planned_stop_active = false) override; std::vector list_faults(bool filter_by_severity, uint8_t severity, const std::vector & statuses) const override; @@ -520,6 +586,12 @@ class InMemoryFaultStorage : public FaultStorage { std::vector get_all_faults() const override; std::vector reclassify_healed_as_cleared() override; + void set_planned_stop(const PlannedStopState & state) override; + PlannedStopState get_planned_stop() const override; + std::vector get_planned_stop_owned() const override; + size_t clear_planned_stop_owned() override; + size_t clear_planned_stop_owned(const std::vector & fault_codes) override; + private: /// Update fault status based on debounce counter and given config void update_status(FaultState & state, const DebounceConfig & config); @@ -564,6 +636,10 @@ class InMemoryFaultStorage : public FaultStorage { size_t max_snapshots_per_fault_{0}; ///< 0 = unlimited bool retain_snapshots_on_clear_{false}; size_t max_near_misses_per_fault_{0}; ///< 0 = unlimited + /// Held for the life of the process only. This backend has no file behind it, + /// so a restart starts with no declaration - which is why a deployment that + /// needs the stop to outlive a reboot runs on the SQLite backend. + PlannedStopState planned_stop_; }; } // namespace ros2_medkit_fault_manager diff --git a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/sqlite_fault_storage.hpp b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/sqlite_fault_storage.hpp index 3565febf4..3015b147e 100644 --- a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/sqlite_fault_storage.hpp +++ b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/sqlite_fault_storage.hpp @@ -48,7 +48,8 @@ class SqliteFaultStorage : public FaultStorage { bool report_fault_event(const std::string & fault_code, uint8_t event_type, uint8_t severity, const std::string & description, const std::string & source_id, - const rclcpp::Time & timestamp, const DebounceConfig & config) override; + const rclcpp::Time & timestamp, const DebounceConfig & config, + bool planned_stop_active = false) override; std::vector list_faults(bool filter_by_severity, uint8_t severity, const std::vector & statuses) const override; @@ -95,6 +96,12 @@ class SqliteFaultStorage : public FaultStorage { std::vector get_all_faults() const override; std::vector reclassify_healed_as_cleared() override; + void set_planned_stop(const PlannedStopState & state) override; + PlannedStopState get_planned_stop() const override; + std::vector get_planned_stop_owned() const override; + size_t clear_planned_stop_owned() override; + size_t clear_planned_stop_owned(const std::vector & fault_codes) override; + /// Get the database path const std::string & db_path() const { return db_path_; @@ -134,11 +141,18 @@ class SqliteFaultStorage : public FaultStorage { /// still says nobody holds it. std::vector store_rosbag_file_locked(const RosbagFileInfo & info); - /// report_fault_event body without taking mutex_ or opening a transaction. Caller holds mutex_ - /// and wraps the call, so the fault row and any near-miss row commit together. + /// report_fault_event body without taking mutex_ or opening a transaction. Caller holds mutex_. + /// A FAILED report is the one the caller wraps in a transaction, so the fault row, any near-miss + /// row and the planned stop's ownership of the cycle commit together; a PASSED report writes at + /// most one row, starts no cycle, and runs in autocommit. bool report_fault_event_locked(const std::string & fault_code, uint8_t event_type, uint8_t severity, const std::string & description, const std::string & source_id, - const rclcpp::Time & timestamp, const DebounceConfig & config); + const rclcpp::Time & timestamp, const DebounceConfig & config, + bool planned_stop_active); + + /// Mark the fault's current cycle as the planned stop's. Caller holds mutex_ and has already + /// written the fault row inside the same transaction. + void mark_planned_stop_owned_locked(const std::string & fault_code); /// Append one entry to the near-miss series and evict the oldest entries beyond /// max_near_misses_per_fault_. Caller holds mutex_ and has already written the fault row. diff --git a/src/ros2_medkit_fault_manager/src/correlation/correlation_engine.cpp b/src/ros2_medkit_fault_manager/src/correlation/correlation_engine.cpp index c7cd12728..3b3934f21 100644 --- a/src/ros2_medkit_fault_manager/src/correlation/correlation_engine.cpp +++ b/src/ros2_medkit_fault_manager/src/correlation/correlation_engine.cpp @@ -25,9 +25,35 @@ CorrelationEngine::CorrelationEngine(const CorrelationConfig & config) } ProcessFaultResult CorrelationEngine::process_fault(const std::string & fault_code, const std::string & severity, - std::chrono::steady_clock::time_point timestamp) { + std::chrono::steady_clock::time_point timestamp, + bool cycle_started) { std::lock_guard lock(mutex_); + // Correlation runs on every report and may write its own mute entry for this + // code. That entry OVERLAYS the planned stop rather than replacing it: ownership + // is a fact about which cycle the fault is in, not about who is currently + // holding it quiet. + ProcessFaultResult result = correlate(fault_code, severity, timestamp); + + if (planned_stop_active_ && cycle_started) { + planned_stop_owned_.insert(fault_code); + } + + if (planned_stop_owned_.count(fault_code) > 0 && muted_faults_.count(fault_code) == 0) { + mute_as_planned_stop(fault_code); + } + + // Derived, not remembered: a fault stays reported as muted for as long as an + // entry exists, whatever this particular report matched. Without this a repeat + // report of a fault whose rule stopped matching announces an update for a fault + // the list is hiding. + result.should_mute = result.should_mute || muted_faults_.count(fault_code) > 0; + + return result; +} + +ProcessFaultResult CorrelationEngine::correlate(const std::string & fault_code, const std::string & severity, + std::chrono::steady_clock::time_point timestamp) { ProcessFaultResult result; // First, clean up expired entries @@ -118,7 +144,9 @@ ProcessClearResult CorrelationEngine::process_clear(const std::string & fault_co } } - // Clean up muted faults + // The rule's overlay on each symptom goes with the root cause. A symptom the + // stop owns is re-muted by reassert_planned_stop_mutes() below; one that is + // auto-cleared has its cycle ended, so its ownership goes too. for (const auto & symptom_code : it->second) { muted_faults_.erase(symptom_code); } @@ -149,7 +177,7 @@ ProcessClearResult CorrelationEngine::process_clear(const std::string & fault_co auto & codes = pending_cluster.fault_codes; codes.erase(std::remove(codes.begin(), codes.end(), fault_code), codes.end()); - pending_it->second.fault_severities.erase(fault_code); + pending_cluster.fault_severities.erase(fault_code); if (codes.empty()) { pending_it = pending_clusters_.erase(pending_it); @@ -158,46 +186,7 @@ ProcessClearResult CorrelationEngine::process_clear(const std::string & fault_co // Reassign representative if the cleared fault was the representative if (pending_cluster.representative_code == fault_code) { - for (const auto & rule : config_.rules) { - if (rule.id == pending_it->first) { - switch (rule.representative) { - case Representative::FIRST: { - auto & sevs = pending_it->second.fault_severities; - const std::string & first_code = codes.front(); - pending_cluster.representative_code = first_code; - auto sev_it = sevs.find(first_code); - pending_cluster.representative_severity = (sev_it != sevs.end()) ? sev_it->second : ""; - break; - } - case Representative::HIGHEST_SEVERITY: { - auto & sevs = pending_it->second.fault_severities; - std::string best_code = codes.front(); - int best_rank = -1; - for (const auto & code : codes) { - auto sev_it = sevs.find(code); - int rank = (sev_it != sevs.end()) ? severity_rank(sev_it->second) : 0; - if (rank > best_rank) { - best_rank = rank; - best_code = code; - } - } - pending_cluster.representative_code = best_code; - auto best_sev_it = sevs.find(best_code); - pending_cluster.representative_severity = (best_sev_it != sevs.end()) ? best_sev_it->second : ""; - break; - } - case Representative::MOST_RECENT: { - auto & sevs = pending_it->second.fault_severities; - const std::string & most_recent_code = codes.back(); - pending_cluster.representative_code = most_recent_code; - auto sev_it = sevs.find(most_recent_code); - pending_cluster.representative_severity = (sev_it != sevs.end()) ? sev_it->second : ""; - break; - } - } - break; - } - } + promote_representative(pending_cluster); } ++pending_it; @@ -209,18 +198,18 @@ ProcessClearResult CorrelationEngine::process_clear(const std::string & fault_co auto & active_cluster = active_it->second; auto & codes = active_cluster.fault_codes; codes.erase(std::remove(codes.begin(), codes.end(), fault_code), codes.end()); + active_cluster.fault_severities.erase(fault_code); if (codes.empty()) { active_clusters_.erase(active_it); } else if (active_cluster.representative_code == fault_code) { - // Sync representative from pending cluster (already updated above) - for (const auto & [rule_id, pending] : pending_clusters_) { - if (pending.data.cluster_id == cluster_id) { - active_cluster.representative_code = pending.data.representative_code; - active_cluster.representative_severity = pending.data.representative_severity; - break; - } - } + // Promoted from the ACTIVE cluster's own members rather than copied from the + // pending twin, because the twin is gone once its window closes + // (cleanup_expired drops it while the active cluster stays). Copied from a + // twin that is not there, the representative keeps naming the acknowledged + // fault, and every remaining member is then hidden by a cluster whose + // representative can never be reported again. + promote_representative(active_cluster); } } @@ -229,10 +218,189 @@ ProcessClearResult CorrelationEngine::process_clear(const std::string & fault_co // Remove from muted faults if it was a symptom muted_faults_.erase(fault_code); + // A cleared fault has nothing left to announce, so the planned stop must not + // hand it back at switch-off. + planned_stop_owned_.erase(fault_code); + for (const auto & auto_cleared : result.auto_cleared_codes) { + muted_faults_.erase(auto_cleared); + planned_stop_owned_.erase(auto_cleared); + } + + // A symptom that was NOT auto-cleared is still up, and the stop may own it. + reassert_planned_stop_mutes(); return result; } +void CorrelationEngine::begin_planned_stop() { + std::lock_guard lock(mutex_); + planned_stop_active_ = true; +} + +EndPlannedStopResult CorrelationEngine::end_planned_stop() { + std::lock_guard lock(mutex_); + + planned_stop_active_ = false; + + EndPlannedStopResult result; + result.to_announce.reserve(planned_stop_owned_.size()); + for (const auto & fault_code : planned_stop_owned_) { + auto it = muted_faults_.find(fault_code); + // A rule's overlay is not the stop's to lift: that fault stays muted, and + // stays unannounced, for as long as the rule holds it. + if (it == muted_faults_.end() || !it->second.by_planned_stop) { + continue; + } + + // The stop's entry goes whatever else is true: the cycle it owned is over. + const bool held_by_cluster = cluster_hides(fault_code); + muted_faults_.erase(it); + ++result.unmuted; + + // A cluster rule outranks the stop, because it is the narrower promise: the + // stop says "not now", the cluster says "this burst is one line, ever". + // Announcing every owned member would fire, in one wave, exactly the storm + // the rule exists to fold into a single alarm. Nothing is written in the + // stop's place: the cluster suppresses the fault's events through the verdict + // it reaches on each report, which is how it hides a member with no stop in + // force, and a remembered copy of that verdict would outlive the membership + // and the representative it was taken from. + if (!held_by_cluster) { + result.to_announce.push_back(fault_code); + } + } + planned_stop_owned_.clear(); + + return result; +} + +bool CorrelationEngine::cluster_hides(const std::string & fault_code) const { + auto cluster_it = fault_to_cluster_.find(fault_code); + if (cluster_it == fault_to_cluster_.end()) { + return false; + } + + // An ACTIVE cluster is one that reached min_count. min_count gates FORMATION, not + // the hiding: once formed, the cluster folds its members into the representative + // until every one of them is acknowledged and the cluster dissolves. A cluster that + // never formed is a handful of separate faults matching the same pattern. + auto active_it = active_clusters_.find(cluster_it->second); + if (active_it == active_clusters_.end()) { + return false; + } + + const ClusterData & cluster = active_it->second; + if (fault_code == cluster.representative_code) { + return false; // the representative is the line the cluster shows + } + + // Membership is read from the ACTIVE cluster, not from fault_to_cluster_, which is + // written on every join including one to a pending twin that is below min_count - + // active_clusters_ is only refreshed when the burst reaches the threshold. A fault + // that joined a twin two members short maps to a formed cluster it is not part of; + // it is a fault of its own, announced and updated like any other. + if (std::find(cluster.fault_codes.begin(), cluster.fault_codes.end(), fault_code) == cluster.fault_codes.end()) { + return false; + } + + const CorrelationRule * rule = find_rule(cluster.rule_id); + return rule != nullptr && rule->show_as_single; +} + +const CorrelationRule * CorrelationEngine::find_rule(const std::string & rule_id) const { + for (const auto & rule : config_.rules) { + if (rule.id == rule_id) { + return &rule; + } + } + return nullptr; +} + +void CorrelationEngine::promote_representative(ClusterData & cluster) { + if (cluster.fault_codes.empty()) { + return; + } + + const CorrelationRule * rule = find_rule(cluster.rule_id); + const Representative policy = rule != nullptr ? rule->representative : Representative::FIRST; + const auto & severities = cluster.fault_severities; + auto severity_of = [&severities](const std::string & code) { + auto it = severities.find(code); + return it != severities.end() ? it->second : std::string{}; + }; + + std::string promoted = cluster.fault_codes.front(); + switch (policy) { + case Representative::FIRST: + break; + case Representative::MOST_RECENT: + promoted = cluster.fault_codes.back(); + break; + case Representative::HIGHEST_SEVERITY: { + int best_rank = -1; + for (const auto & code : cluster.fault_codes) { + const int rank = severity_rank(severity_of(code)); + if (rank > best_rank) { + best_rank = rank; + promoted = code; + } + } + break; + } + } + + cluster.representative_code = promoted; + cluster.representative_severity = severity_of(promoted); +} + +bool CorrelationEngine::planned_stop_active() const { + std::lock_guard lock(mutex_); + return planned_stop_active_; +} + +void CorrelationEngine::release_planned_stop_ownership(const std::string & fault_code) { + std::lock_guard lock(mutex_); + + if (planned_stop_owned_.erase(fault_code) == 0) { + return; + } + auto it = muted_faults_.find(fault_code); + if (it != muted_faults_.end() && it->second.by_planned_stop) { + muted_faults_.erase(it); + } +} + +void CorrelationEngine::restore_planned_stop_ownership(const std::string & fault_code) { + std::lock_guard lock(mutex_); + + planned_stop_owned_.insert(fault_code); + if (muted_faults_.count(fault_code) == 0) { + mute_as_planned_stop(fault_code); + } +} + +std::vector CorrelationEngine::planned_stop_owned_codes() const { + std::lock_guard lock(mutex_); + return {planned_stop_owned_.begin(), planned_stop_owned_.end()}; +} + +void CorrelationEngine::mute_as_planned_stop(const std::string & fault_code) { + MutedFaultData muted; + muted.fault_code = fault_code; + muted.root_cause_code = kPlannedStopRootCause; + muted.rule_id = kPlannedStopRuleId; + muted.by_planned_stop = true; + muted_faults_[fault_code] = muted; +} + +void CorrelationEngine::reassert_planned_stop_mutes() { + for (const auto & fault_code : planned_stop_owned_) { + if (muted_faults_.count(fault_code) == 0) { + mute_as_planned_stop(fault_code); + } + } +} + std::vector CorrelationEngine::get_muted_faults() const { std::lock_guard lock(mutex_); @@ -315,8 +483,10 @@ void CorrelationEngine::cleanup_expired() { pending_clusters_.erase(it); } } -} + // Whatever a rule just stopped holding, the stop still owns. + reassert_planned_stop_mutes(); +} std::optional CorrelationEngine::try_as_root_cause(const std::string & fault_code) { for (const auto & rule : config_.rules) { if (rule.mode != CorrelationMode::HIERARCHICAL) { @@ -383,6 +553,9 @@ std::optional CorrelationEngine::try_as_symptom(const std::s muted.root_cause_code = prc.fault_code; muted.rule_id = rule.id; muted.delay_ms = result.delay_ms; + // Overlays whatever was there, the planned stop's entry included. The + // stop keeps its ownership, so when this rule lets go the fault is muted + // by the stop again rather than falling out of it. muted_faults_[fault_code] = muted; } @@ -441,7 +614,7 @@ std::optional CorrelationEngine::try_auto_cluster(const std: pending.data.representative_code = fault_code; pending.data.representative_severity = severity; pending.data.fault_codes.push_back(fault_code); - pending.fault_severities[fault_code] = severity; + pending.data.fault_severities[fault_code] = severity; pending.data.first_at = now_system; pending.data.last_at = now_system; @@ -464,15 +637,12 @@ std::optional CorrelationEngine::try_auto_cluster(const std: // Already in cluster - ensure consistent muting for duplicates ProcessFaultResult result; result.cluster_id = cluster.cluster_id; - if (rule.show_as_single && fault_code != cluster.representative_code && - cluster.fault_codes.size() >= rule.min_count) { - result.should_mute = true; - } + result.should_mute = cluster_hides(fault_code); return result; } cluster.fault_codes.push_back(fault_code); - pending.fault_severities[fault_code] = severity; + cluster.fault_severities[fault_code] = severity; cluster.last_at = now_system; fault_to_cluster_[fault_code] = cluster.cluster_id; @@ -523,9 +693,7 @@ std::optional CorrelationEngine::try_auto_cluster(const std: } // Mute non-representative faults - if (rule.show_as_single && fault_code != cluster.representative_code) { - result.should_mute = true; - } + result.should_mute = cluster_hides(fault_code); } return result; diff --git a/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp b/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp index 146f0e3b4..c2222ae05 100644 --- a/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp +++ b/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp @@ -115,7 +115,11 @@ std::string validate_recording_id(const std::string & recording_id) { } // namespace -FaultManagerNode::FaultManagerNode(const rclcpp::NodeOptions & options) : Node("fault_manager", options) { +FaultManagerNode::FaultManagerNode(const rclcpp::NodeOptions & options) : FaultManagerNode(options, nullptr) { +} + +FaultManagerNode::FaultManagerNode(const rclcpp::NodeOptions & options, std::unique_ptr storage) + : Node("fault_manager", options), storage_override_(std::move(storage)) { // Declare and get parameters storage_type_ = declare_parameter("storage_type", "sqlite"); database_path_ = declare_parameter("database_path", "/var/lib/ros2_medkit/faults.db"); @@ -253,6 +257,11 @@ FaultManagerNode::FaultManagerNode(const rclcpp::NodeOptions & options) : Node(" // Create event publisher for SSE streaming event_publisher_ = create_publisher("~/events", rclcpp::QoS(100).reliable()); + // A planned stop outlives the process that declared it, so the declaration and + // the cycles it owns are read back from the store rather than started fresh. + planned_stop_ = storage_->get_planned_stop(); + finish_interrupted_release(); + // Build global debounce config global_config_.confirmation_threshold = confirmation_threshold_; global_config_.healing_enabled = healing_enabled_; @@ -377,6 +386,18 @@ FaultManagerNode::FaultManagerNode(const rclcpp::NodeOptions & options) : Node(" handle_list_faults_for_entity(request, response); }); + set_planned_stop_srv_ = create_service( + "~/set_planned_stop", [this](const std::shared_ptr & request, + const std::shared_ptr & response) { + handle_set_planned_stop(request, response); + }); + + get_planned_stop_srv_ = create_service( + "~/get_planned_stop", [this](const std::shared_ptr & request, + const std::shared_ptr & response) { + handle_get_planned_stop(request, response); + }); + // Initialize snapshot capture auto snapshot_config = create_snapshot_config(); if (snapshot_config.enabled) { @@ -427,21 +448,32 @@ FaultManagerNode::FaultManagerNode(const rclcpp::NodeOptions & options) : Node(" }); } - // Initialize correlation engine (nullptr if disabled or not configured) + // Correlation engine: rules when configured, and in every case the planned-stop mute correlation_engine_ = create_correlation_engine(); - // Create correlation cleanup timer if correlation is enabled - if (correlation_engine_) { - auto cleanup_interval_sec = declare_parameter("correlation.cleanup_interval_sec", 5.0); - if (!(std::isfinite(cleanup_interval_sec) && cleanup_interval_sec > 0.0)) { - RCLCPP_WARN(get_logger(), "correlation.cleanup_interval_sec must be positive, got %.2f. Using default 5.0s", - cleanup_interval_sec); - cleanup_interval_sec = 5.0; + auto cleanup_interval_sec = declare_parameter("correlation.cleanup_interval_sec", 5.0); + if (!(std::isfinite(cleanup_interval_sec) && cleanup_interval_sec > 0.0)) { + RCLCPP_WARN(get_logger(), "correlation.cleanup_interval_sec must be positive, got %.2f. Using default 5.0s", + cleanup_interval_sec); + cleanup_interval_sec = 5.0; + } + auto cleanup_interval_ms = static_cast(cleanup_interval_sec * 1000); + correlation_cleanup_timer_ = create_wall_timer(std::chrono::milliseconds(cleanup_interval_ms), [this]() { + correlation_engine_->cleanup_expired(); + }); + + // A weekend stop must not end because the box rebooted. The cycles it owns come + // back from the flags the store kept, so the switch-off still releases and + // announces exactly them. + if (planned_stop_.active) { + correlation_engine_->begin_planned_stop(); + const auto owned = storage_->get_planned_stop_owned(); + for (const auto & fault_code : owned) { + correlation_engine_->restore_planned_stop_ownership(fault_code); } - auto cleanup_interval_ms = static_cast(cleanup_interval_sec * 1000); - correlation_cleanup_timer_ = create_wall_timer(std::chrono::milliseconds(cleanup_interval_ms), [this]() { - correlation_engine_->cleanup_expired(); - }); + RCLCPP_INFO(get_logger(), + "Planned stop is in force (reason='%s', declared_by='%s'): %zu fault(s) re-marked, new faults marked", + planned_stop_.reason.c_str(), planned_stop_.declared_by.c_str(), owned.size()); } // Create auto-confirmation timer if enabled @@ -490,6 +522,40 @@ FaultManagerNode::FaultManagerNode(const rclcpp::NodeOptions & options) : Node(" } } +void FaultManagerNode::finish_interrupted_release() { + if (planned_stop_.active) { + return; // the declaration still stands; nothing was left half-released + } + + const auto owned = storage_->get_planned_stop_owned(); + if (owned.empty()) { + return; + } + + // Faults owned by a declaration that is already withdrawn: the previous process + // died between writing the withdrawal and dropping the flags, so these + // confirmations were suppressed and announced by nobody. Finish the job. + // + // Synchronously, here, before this node has a service or a timer that could run: + // deferring it opens a window in which an operator declares a NEW stop and new + // cycles become owned by it, and the deferred work would then announce a fault + // the new stop is holding and drop its flag. A publication this early may reach + // no subscriber - the events topic is volatile and nothing has matched a + // publisher this young - and that is the same accepted loss as any other + // publication made at startup. The released faults are in the default fault list + // either way, which is where an operator looks after a restart. + RCLCPP_WARN(get_logger(), + "%zu fault(s) are still owned by a planned stop that was already withdrawn - a previous run did not " + "finish releasing them. Releasing them now.", + owned.size()); + const size_t announced = announce_released(owned); + // Exactly what was captured above: a flag written after that capture belongs to + // a declaration this release knows nothing about. + const size_t released = storage_->clear_planned_stop_owned(owned); + RCLCPP_INFO(get_logger(), "Finished an interrupted planned-stop release: released %zu fault(s), announced %zu", + released, announced); +} + FaultManagerNode::~FaultManagerNode() { // Join capture workers FIRST so no worker is mid-capture when rosbag tears // down. RosbagCapture::stop() is not a barrier against an in-flight @@ -508,7 +574,7 @@ FaultManagerNode::~FaultManagerNode() { if (audit_log_) { try { AuditEvent marker; - marker.fault_code = "__audit__"; + marker.fault_code = kAuditMarkerFaultCode; marker.transition = kTransitionLoggingDeactivated; marker.status = "INACTIVE"; marker.source_id = "fault_manager"; @@ -522,6 +588,11 @@ FaultManagerNode::~FaultManagerNode() { } std::unique_ptr FaultManagerNode::create_storage() { + if (storage_override_) { + RCLCPP_INFO(get_logger(), "Using a caller-supplied fault storage backend"); + return std::move(storage_override_); + } + if (storage_type_ == "memory") { RCLCPP_INFO(get_logger(), "Using in-memory fault storage"); return std::make_unique(); @@ -621,7 +692,7 @@ std::unique_ptr FaultManagerNode::create_audit_log() { // Appended directly, so it is recorded even in confirmed_only mode. // Best-effort: a failure here does not abort startup. AuditEvent marker; - marker.fault_code = "__audit__"; + marker.fault_code = kAuditMarkerFaultCode; marker.transition = kTransitionLoggingActivated; marker.status = "ACTIVE"; marker.source_id = "fault_manager"; @@ -657,6 +728,14 @@ void FaultManagerNode::audit_transition(const char * transition, const ros2_medk event.description = fault.description; event.occurred_at_ns = occurred_at_ns; + append_audit_event(event); +} + +void FaultManagerNode::append_audit_event(const AuditEvent & event) { + if (!audit_log_) { + return; + } + try { audit_log_->append(event); } catch (const std::exception & e) { @@ -670,7 +749,7 @@ void FaultManagerNode::audit_transition(const char * transition, const ros2_medk audit_healthy_.store(false, std::memory_order_relaxed); const uint64_t dropped = audit_dropped_writes_.load(std::memory_order_relaxed); RCLCPP_ERROR(get_logger(), "Failed to append audit record for '%s' (%s): %s [audit dropped_writes=%" PRIu64 "]", - fault.fault_code.c_str(), transition, e.what(), dropped); + event.fault_code.c_str(), event.transition.c_str(), e.what(), dropped); if (audit_fail_closed_) { // Explicit fail-FAST: surface the broken audit to the caller so a // compliance-strict deployment learns the chain is now incomplete and an @@ -681,12 +760,30 @@ void FaultManagerNode::audit_transition(const char * transition, const ros2_medk "audit_log.fail_closed is set: audit append failed for '%s' (%s); the audit chain is now " "incomplete and requires operator action. The already-committed fault-state change is NOT " "rolled back (the fault store is a separate database).", - fault.fault_code.c_str(), transition); + event.fault_code.c_str(), event.transition.c_str()); throw; } } } +void FaultManagerNode::audit_planned_stop(const char * transition, const PlannedStopState & state, + const std::string & reason, const std::string & declared_by, + int64_t occurred_at_ns) { + AuditEvent event; + // The same sentinel the log's own lifecycle markers use: this transition is + // about the installation, not about one fault. + event.fault_code = kAuditMarkerFaultCode; + event.transition = transition; + event.status = state.active ? "ACTIVE" : "INACTIVE"; + // The reason and the declarer of THIS transition: who withdrew a stop, and why, + // is not who declared it. + event.source_id = declared_by; + event.description = reason; + event.occurred_at_ns = occurred_at_ns; + + append_audit_event(event); +} + void FaultManagerNode::capture_on_confirm(const std::string & fault_code) { // Capture snapshots/rosbag when a fault confirms, via the bounded pool. // Both callers - the report handler and the auto-confirm timer - run on the @@ -811,10 +908,22 @@ void FaultManagerNode::handle_report_fault( // TODO(#276): warn when different entities resolve different configs for the same fault_code auto resolved_config = resolve_config(request->source_id); - // Report the fault event (use wall clock time, not sim time, for proper timestamps) + // Report the fault event (use wall clock time, not sim time, for proper timestamps). + // + // Ownership is persisted, not inferred: the switch-off, and a restart, both read it back from + // the store rather than comparing timestamps. The switch travels WITH the report, so the store + // writes the flag inside the transaction that writes the fault row. + // + // The cycle boundary is therefore decided twice: in the store for the persisted flag, and again + // below for the engine's in-process set. Both read the same two facts - whether the report was a + // new occurrence, and the status the fault held before it - so they are one formula evaluated in + // two places. They cannot drift apart because this handler is the only writer of either, and the + // node's services run one at a time on the single-threaded executor rclcpp::spin() gives it + // (main.cpp), with no callback group taking any of them off that thread. const rclcpp::Time event_time = get_wall_clock_time(); - bool is_new = storage_->report_fault_event(request->fault_code, request->event_type, request->severity, - request->description, request->source_id, event_time, resolved_config); + bool is_new = + storage_->report_fault_event(request->fault_code, request->event_type, request->severity, request->description, + request->source_id, event_time, resolved_config, planned_stop_.active); response->accepted = true; @@ -824,9 +933,19 @@ void FaultManagerNode::handle_report_fault( // Process through correlation engine (if enabled) // Only process FAILED events with correlation bool should_mute = false; - if (correlation_engine_ && request->event_type == ros2_medkit_msgs::srv::ReportFault::Request::EVENT_FAILED) { + if (request->event_type == ros2_medkit_msgs::srv::ReportFault::Request::EVENT_FAILED) { + // The cycle boundary the planned stop needs. `is_new` covers a new fault and + // one raised again after being cleared; a re-fail out of HEALED is the third + // case, because the manager publishes EVENT_CLEARED at the heal, so the + // confirmation that follows is a fresh announcement. Reporters are + // level-triggered, so without this boundary a condition that was already up + // - and already announced - would be swallowed by a stop declared after it + // started. occurrence_count keeps its own, narrower definition. + const bool cycle_started = is_new || status_before == ros2_medkit_msgs::msg::Fault::STATUS_HEALED; + auto correlation_result = - correlation_engine_->process_fault(request->fault_code, correlation::severity_to_string(request->severity)); + correlation_engine_->process_fault(request->fault_code, correlation::severity_to_string(request->severity), + std::chrono::steady_clock::now(), cycle_started); should_mute = correlation_result.should_mute; @@ -846,6 +965,11 @@ void FaultManagerNode::handle_report_fault( correlation_result.cluster_id.c_str(), correlation_result.retroactive_mute_codes.size()); } } + } else { + // A PASSED report can still produce an event - the fault stays CONFIRMED and + // its data is updated - and a muted fault must not announce that either. The + // mute is a property of the fault, not of the report that happened to arrive. + should_mute = correlation_engine_->is_muted(request->fault_code); } // Determine event type based on status transition @@ -1011,9 +1135,15 @@ void FaultManagerNode::handle_clear_fault( // correlated symptom fault codes. Per-entity DELETE routes set it to true // so they cannot reach across entity boundaries via the correlation graph. std::vector auto_cleared_codes; - if (correlation_engine_ && !request->skip_correlation_auto_clear) { + if (!request->skip_correlation_auto_clear) { auto clear_result = correlation_engine_->process_clear(request->fault_code); auto_cleared_codes = clear_result.auto_cleared_codes; + } else { + // Scoped per-entity acknowledgement: the correlation graph is deliberately not + // walked, because it reaches across entity boundaries. The planned stop is not + // a graph - it is this one fault's mute - so it is released either way, or an + // acknowledged fault would stay counted as muted until the stop ended. + correlation_engine_->release_planned_stop_ownership(request->fault_code); } bool cleared = storage_->clear_fault(request->fault_code); @@ -1073,6 +1203,102 @@ void FaultManagerNode::handle_clear_fault( } } +void FaultManagerNode::handle_set_planned_stop( + const std::shared_ptr & request, + const std::shared_ptr & response) { + response->was_active = planned_stop_.active; + response->success = true; + + if (request->active == planned_stop_.active) { + // Asking for the state the switch is already in is not an error and is not a + // transition: nothing changes and the audit chain records nothing, so a client + // that retries a call cannot manufacture evidence of a stop that never started. + response->message = planned_stop_.active ? "Planned stop already in force" : "No planned stop in force"; + return; + } + + const int64_t transition_at_ns = get_wall_clock_time().nanoseconds(); + + // What the declaration becomes. A withdrawal keeps the reason, the declarer and + // the start time of the stop it ends, and stamps when it ended, so an operator + // can still read why the plant was quiet once it is back up. + PlannedStopState next = planned_stop_; + if (request->active) { + next.active = true; + next.reason = request->reason; + next.declared_by = request->declared_by; + next.since_ns = transition_at_ns; + next.ended_at_ns = 0; + } else { + next.active = false; + next.ended_at_ns = transition_at_ns; + } + + // The store goes first, and a store that refuses the write ends the request + // here: nothing is muted, unmuted, announced or audited on a declaration the + // manager could not record, and a restart cannot come back believing in it. + try { + storage_->set_planned_stop(next); + } catch (const std::exception & e) { + response->success = false; + response->message = std::string("Failed to record the planned stop: ") + e.what(); + RCLCPP_ERROR(get_logger(), "Planned stop %s rejected: %s", request->active ? "declaration" : "withdrawal", + e.what()); + return; + } + + planned_stop_ = next; + + if (request->active) { + correlation_engine_->begin_planned_stop(); + audit_planned_stop(kTransitionPlannedStopStarted, planned_stop_, request->reason, request->declared_by, + transition_at_ns); + + response->message = "Planned stop declared"; + RCLCPP_INFO(get_logger(), "Planned stop declared by '%s': %s", planned_stop_.declared_by.c_str(), + planned_stop_.reason.c_str()); + return; + } + + const auto withdrawal = correlation_engine_->end_planned_stop(); + + // The switch-off is the moment those faults become news. A CONFIRMED one was + // never announced - the confirmation happened behind the mute - so publish it + // now, once, or the alarm standing on the controller is invisible to every + // consumer of the stream. PREFAILED has nothing to announce yet, and a HEALED + // one already published its end. + // + // Announcing covers fewer faults than unmuting: a member a live cluster is still + // hiding leaves the muted list here and goes on being folded into its + // representative's line, so it is counted as unmuted and never announced. + const size_t announced = announce_released(withdrawal.to_announce); + + // The flags go LAST. A crash between the declaration write and this point + // leaves owned faults behind an ENDED declaration, which the next startup + // recognises and finishes - announcing what this loop did not get to. Clearing + // first would turn a crash into permanent silence for those faults. + storage_->clear_planned_stop_owned(); + + // The withdrawal carries its own reason and declarer into the audit record; the + // stored declaration keeps the ones that started the stop. + audit_planned_stop(kTransitionPlannedStopEnded, planned_stop_, request->reason, request->declared_by, + transition_at_ns); + + response->message = "Planned stop withdrawn"; + RCLCPP_INFO(get_logger(), "Planned stop withdrawn by '%s': unmuted %zu fault(s), announced %zu confirmation(s)", + request->declared_by.c_str(), withdrawal.unmuted, announced); +} + +void FaultManagerNode::handle_get_planned_stop( + const std::shared_ptr & /*request*/, + const std::shared_ptr & response) { + response->active = planned_stop_.active; + response->reason = planned_stop_.reason; + response->declared_by = planned_stop_.declared_by; + response->since = rclcpp::Time(planned_stop_.since_ns, RCL_SYSTEM_TIME); + response->ended_at = rclcpp::Time(planned_stop_.ended_at_ns, RCL_SYSTEM_TIME); +} + bool FaultManagerNode::is_valid_severity(uint8_t severity) { return severity <= ros2_medkit_msgs::msg::Fault::SEVERITY_CRITICAL; } @@ -1364,18 +1590,47 @@ void FaultManagerNode::load_snapshot_config_from_yaml(const std::string & config } } +size_t FaultManagerNode::announce_released(const std::vector & fault_codes) { + // A released fault that is CONFIRMED was never announced - the confirmation + // happened behind the mute - so it is announced now, once, or the alarm standing + // on the controller is invisible to every consumer of the stream. PREFAILED has + // nothing to announce yet, and a HEALED one already published its end. + size_t announced = 0; + for (const auto & fault_code : fault_codes) { + auto fault = storage_->get_fault(fault_code); + if (fault && fault->status == ros2_medkit_msgs::msg::Fault::STATUS_CONFIRMED) { + publish_fault_event(ros2_medkit_msgs::msg::FaultEvent::EVENT_CONFIRMED, *fault); + ++announced; + } + } + return announced; +} + std::unique_ptr FaultManagerNode::create_correlation_engine() { + auto config = load_correlation_config(); + if (config) { + RCLCPP_INFO(get_logger(), "Correlation engine enabled (default_window=%ums, patterns=%zu, rules=%zu)", + config->default_window_ms, config->patterns.size(), config->rules.size()); + return std::make_unique(*config); + } + + // No rules, but still an engine: the planned stop is a mute source an operator + // declares at runtime and it needs no configuration to exist. + return std::make_unique(correlation::CorrelationConfig{}); +} + +std::optional FaultManagerNode::load_correlation_config() { // Get correlation config file path from parameter auto config_file = declare_parameter("correlation.config_file", ""); if (config_file.empty()) { - RCLCPP_DEBUG(get_logger(), "Correlation disabled: no config_file specified"); - return nullptr; + RCLCPP_DEBUG(get_logger(), "Correlation rules disabled: no config_file specified"); + return std::nullopt; } if (!std::filesystem::exists(config_file)) { RCLCPP_ERROR(get_logger(), "Correlation config file not found: %s", config_file.c_str()); - return nullptr; + return std::nullopt; } try { @@ -1384,7 +1639,7 @@ std::unique_ptr FaultManagerNode::create_correla if (!config.enabled) { RCLCPP_INFO(get_logger(), "Correlation explicitly disabled in config"); - return nullptr; + return std::nullopt; } // Validate the config @@ -1396,17 +1651,14 @@ std::unique_ptr FaultManagerNode::create_correla for (const auto & error : validation.errors) { RCLCPP_ERROR(get_logger(), "Correlation config error: %s", error.c_str()); } - return nullptr; + return std::nullopt; } - RCLCPP_INFO(get_logger(), "Correlation engine enabled (default_window=%ums, patterns=%zu, rules=%zu)", - config.default_window_ms, config.patterns.size(), config.rules.size()); - - return std::make_unique(config); + return config; } catch (const std::exception & e) { RCLCPP_ERROR(get_logger(), "Failed to load correlation config: %s", e.what()); - return nullptr; + return std::nullopt; } } diff --git a/src/ros2_medkit_fault_manager/src/fault_storage.cpp b/src/ros2_medkit_fault_manager/src/fault_storage.cpp index 34f9053e8..1382c6175 100644 --- a/src/ros2_medkit_fault_manager/src/fault_storage.cpp +++ b/src/ros2_medkit_fault_manager/src/fault_storage.cpp @@ -121,7 +121,8 @@ void InMemoryFaultStorage::update_status(FaultState & state, const DebounceConfi bool InMemoryFaultStorage::report_fault_event(const std::string & fault_code, uint8_t event_type, uint8_t severity, const std::string & description, const std::string & source_id, - const rclcpp::Time & timestamp, const DebounceConfig & config) { + const rclcpp::Time & timestamp, const DebounceConfig & config, + bool planned_stop_active) { std::lock_guard lock(mutex_); const bool is_failed = (event_type == EventType::EVENT_FAILED); @@ -156,6 +157,13 @@ bool InMemoryFaultStorage::report_fault_event(const std::string & fault_code, ui record_near_miss(state, config, severity, source_id, timestamp); } + // A new fault is always the start of a cycle. The flag is set with the rest of the state, + // under the one lock, because the two backends have to make the same fault owned by the same + // report: which faults a switch-off releases must not depend on which store a deployment + // picked. Nothing here outlives the process, so this is parity with the durable backend's + // transaction rather than a crash window being closed. + state.planned_stop_owned = planned_stop_active; + faults_.emplace(fault_code, std::move(state)); return true; } @@ -194,6 +202,10 @@ bool InMemoryFaultStorage::report_fault_event(const std::string & fault_code, ui if (is_near_miss(true, state.status)) { record_near_miss(state, config, severity, source_id, timestamp); } + // A fault raised again after being cleared starts a new cycle. + if (planned_stop_active) { + state.planned_stop_owned = true; + } return true; // Reactivation treated as new occurrence for event publishing } @@ -205,6 +217,15 @@ bool InMemoryFaultStorage::report_fault_event(const std::string & fault_code, ui state.debounce_counter = clamp_debounce_counter(state.debounce_counter, config); if (is_failed) { + // A failure out of HEALED starts a cycle: the heal published the fault's end, so the + // confirmation that follows is fresh news. A repeat report of a condition that is already up + // is the same cycle and takes nothing, so no report ever lowers the flag. It is lowered + // elsewhere: by acknowledgement (clear_fault), by the healed-to-cleared reclassification, and + // by the switch-off (clear_planned_stop_owned). + if (planned_stop_active && state.status == ros2_medkit_msgs::msg::Fault::STATUS_HEALED) { + state.planned_stop_owned = true; + } + // last_occurred tracks occurrences only. A PASSED event is the fault ENDING, not // occurring; bumping it there makes a long-stale CONFIRMED fault look freshly // active to operators. The PASSED instant is kept in last_passed_time. @@ -333,6 +354,9 @@ bool InMemoryFaultStorage::clear_fault(const std::string & fault_code) { } it->second.status = ros2_medkit_msgs::msg::Fault::STATUS_CLEARED; + // An acknowledged cycle is over, so the planned stop no longer owns it: it has + // nothing left to release or to announce for this fault. + it->second.planned_stop_owned = false; return true; } @@ -856,6 +880,7 @@ std::vector InMemoryFaultStorage::reclassify_healed_as_cleared() { for (auto & [code, state] : faults_) { if (state.status == ros2_medkit_msgs::msg::Fault::STATUS_HEALED) { state.status = ros2_medkit_msgs::msg::Fault::STATUS_CLEARED; + state.planned_stop_owned = false; // CLEARED ends the cycle the stop owned reclassified.push_back(code); } } @@ -875,4 +900,50 @@ std::vector InMemoryFaultStorage::reclassify_healed_as_cleared() { return reclassified; } +std::vector InMemoryFaultStorage::get_planned_stop_owned() const { + std::lock_guard lock(mutex_); + std::vector owned; + for (const auto & [code, state] : faults_) { + if (state.planned_stop_owned) { + owned.push_back(code); + } + } + return owned; +} + +size_t InMemoryFaultStorage::clear_planned_stop_owned() { + std::lock_guard lock(mutex_); + size_t cleared = 0; + for (auto & [code, state] : faults_) { + if (state.planned_stop_owned) { + state.planned_stop_owned = false; + ++cleared; + } + } + return cleared; +} + +size_t InMemoryFaultStorage::clear_planned_stop_owned(const std::vector & fault_codes) { + std::lock_guard lock(mutex_); + size_t cleared = 0; + for (const auto & fault_code : fault_codes) { + auto it = faults_.find(fault_code); + if (it != faults_.end() && it->second.planned_stop_owned) { + it->second.planned_stop_owned = false; + ++cleared; + } + } + return cleared; +} + +void InMemoryFaultStorage::set_planned_stop(const PlannedStopState & state) { + std::lock_guard lock(mutex_); + planned_stop_ = state; +} + +PlannedStopState InMemoryFaultStorage::get_planned_stop() const { + std::lock_guard lock(mutex_); + return planned_stop_; +} + } // namespace ros2_medkit_fault_manager diff --git a/src/ros2_medkit_fault_manager/src/sqlite_fault_storage.cpp b/src/ros2_medkit_fault_manager/src/sqlite_fault_storage.cpp index e466ea06a..0523e9168 100644 --- a/src/ros2_medkit_fault_manager/src/sqlite_fault_storage.cpp +++ b/src/ros2_medkit_fault_manager/src/sqlite_fault_storage.cpp @@ -164,7 +164,8 @@ void SqliteFaultStorage::initialize_schema() { debounce_counter INTEGER NOT NULL DEFAULT 0, last_failed_ns INTEGER NOT NULL DEFAULT 0, last_passed_ns INTEGER NOT NULL DEFAULT 0, - confirmed_at_ns INTEGER NOT NULL DEFAULT 0 + confirmed_at_ns INTEGER NOT NULL DEFAULT 0, + planned_stop_owned INTEGER NOT NULL DEFAULT 0 ); )"; @@ -197,6 +198,29 @@ void SqliteFaultStorage::initialize_schema() { } } + // Migration: the planned stop records which fault CYCLES it owns, so the flag + // lives with the fault rather than being inferred from timestamps. Rows written + // before it arrive unowned, which is what a database that predates the switch + // means. + { + bool has_owned = false; + SqliteStatement info(db_, "PRAGMA table_info(faults)"); + while (info.step() == SQLITE_ROW) { + if (info.column_text(1) == "planned_stop_owned") { + has_owned = true; + break; + } + } + if (!has_owned) { + if (sqlite3_exec(db_, "ALTER TABLE faults ADD COLUMN planned_stop_owned INTEGER NOT NULL DEFAULT 0", nullptr, + nullptr, &err_msg) != SQLITE_OK) { + std::string error = err_msg ? err_msg : "Unknown error"; + sqlite3_free(err_msg); + throw std::runtime_error("Failed to add planned_stop_owned column: " + error); + } + } + } + // Migration: releases that advanced last_occurred_ns on PASSED events left // inflated rows behind, and a latched CONFIRMED fault that only ever heals // would keep the wrong timestamp forever. last_failed_ns holds the true @@ -316,6 +340,48 @@ void SqliteFaultStorage::initialize_schema() { } } + // Create planned_stop table: the operator's declaration that the plant is + // deliberately down. A fault manager holds exactly one, so the table is pinned + // to a single row by a constant primary key and written with INSERT OR REPLACE. + const char * create_planned_stop_table_sql = R"( + CREATE TABLE IF NOT EXISTS planned_stop ( + id INTEGER PRIMARY KEY CHECK (id = 1), + active INTEGER NOT NULL, + reason TEXT NOT NULL, + declared_by TEXT NOT NULL, + since_ns INTEGER NOT NULL, + ended_at_ns INTEGER NOT NULL DEFAULT 0 + ); + )"; + + if (sqlite3_exec(db_, create_planned_stop_table_sql, nullptr, nullptr, &err_msg) != SQLITE_OK) { + std::string error = err_msg ? err_msg : "Unknown error"; + sqlite3_free(err_msg); + throw std::runtime_error("Failed to create planned_stop table: " + error); + } + + // Migration: the row outlives the stop now, so it records when the stop ended. + // A database written before that reads as "never withdrawn", which is what a + // row with no end time means. + { + bool has_ended_at = false; + SqliteStatement info(db_, "PRAGMA table_info(planned_stop)"); + while (info.step() == SQLITE_ROW) { + if (info.column_text(1) == "ended_at_ns") { + has_ended_at = true; + break; + } + } + if (!has_ended_at) { + if (sqlite3_exec(db_, "ALTER TABLE planned_stop ADD COLUMN ended_at_ns INTEGER NOT NULL DEFAULT 0", nullptr, + nullptr, &err_msg) != SQLITE_OK) { + std::string error = err_msg ? err_msg : "Unknown error"; + sqlite3_free(err_msg); + throw std::runtime_error("Failed to add ended_at_ns column: " + error); + } + } + } + // Create rosbag_files table. One row = one LINK (a fault claiming a recording): // several faults of a burst link to one bag, and one fault links to several bags // over time. Bytes belong to file_path, not to the row. @@ -665,26 +731,30 @@ std::string SqliteFaultStorage::serialize_json_array(const std::vector lock(mutex_); - // Only a FAILED report can write two rows, and only those two have to land together: written as - // separate autocommit statements, a failure on the second would leave the debounce counter + // Only a FAILED report can write more than one row, and those rows have to land together: written + // as separate autocommit statements, a failure on the second would leave the debounce counter // already advanced, so the caller's retry would advance it a second time and the near miss it - // retried for would still be missing from the series. + // retried for would still be missing from the series. The planned stop's ownership of a cycle is + // in the same transaction for the same reason, one step stronger: a cycle that confirms inside a + // stop and is not owned is a fault the switch-off never releases. // - // A PASSED report writes one row at most and never appends a near miss, so it keeps the plain - // autocommit path. BEGIN IMMEDIATE takes the writer lock up front, which would make a heal - // heartbeat - including one that turns out to write nothing at all - contend for that lock and - // fail with SQLITE_BUSY where before it could not. + // A PASSED report writes one row at most, never appends a near miss and never starts a cycle, so + // it keeps the plain autocommit path. BEGIN IMMEDIATE takes the writer lock up front, which would + // make a heal heartbeat - including one that turns out to write nothing at all - contend for that + // lock and fail with SQLITE_BUSY where before it could not. if (event_type != EventType::EVENT_FAILED) { - return report_fault_event_locked(fault_code, event_type, severity, description, source_id, timestamp, config); + return report_fault_event_locked(fault_code, event_type, severity, description, source_id, timestamp, config, + planned_stop_active); } exec_or_throw("BEGIN IMMEDIATE"); try { - const bool is_new_occurrence = - report_fault_event_locked(fault_code, event_type, severity, description, source_id, timestamp, config); + const bool is_new_occurrence = report_fault_event_locked(fault_code, event_type, severity, description, source_id, + timestamp, config, planned_stop_active); exec_or_throw("COMMIT"); return is_new_occurrence; } catch (...) { @@ -693,9 +763,18 @@ bool SqliteFaultStorage::report_fault_event(const std::string & fault_code, uint } } +void SqliteFaultStorage::mark_planned_stop_owned_locked(const std::string & fault_code) { + SqliteStatement stmt(db_, "UPDATE faults SET planned_stop_owned = 1 WHERE fault_code = ?"); + stmt.bind_text(1, fault_code); + if (stmt.step() != SQLITE_DONE) { + throw std::runtime_error(std::string("Failed to mark fault as owned by the planned stop: ") + sqlite3_errmsg(db_)); + } +} + bool SqliteFaultStorage::report_fault_event_locked(const std::string & fault_code, uint8_t event_type, uint8_t severity, const std::string & description, const std::string & source_id, - const rclcpp::Time & timestamp, const DebounceConfig & config) { + const rclcpp::Time & timestamp, const DebounceConfig & config, + bool planned_stop_active) { int64_t timestamp_ns = timestamp.nanoseconds(); const bool is_failed = (event_type == EventType::EVENT_FAILED); @@ -837,6 +916,16 @@ bool SqliteFaultStorage::report_fault_event_locked(const std::string & fault_cod } } + // The cycle boundary the planned stop marks, decided inside the same transaction as the row it + // belongs to. A reactivation out of CLEARED starts one, and so does a failure out of HEALED - + // the heal published the fault's end, so the confirmation that follows is fresh news. A repeat + // report of a condition that is already up is the same cycle: it neither takes ownership nor + // drops the ownership an earlier report took. + if (planned_stop_active && is_failed && + (is_reactivation || current_status == ros2_medkit_msgs::msg::Fault::STATUS_HEALED)) { + mark_planned_stop_owned_locked(fault_code); + } + return is_reactivation; // Reactivation treated as new occurrence for event publishing } @@ -883,6 +972,11 @@ bool SqliteFaultStorage::report_fault_event_locked(const std::string & fault_cod record_near_miss_locked(fault_code, timestamp_ns, initial_counter, config, severity, source_id, initial_status); } + // A new fault is always the start of a cycle. + if (planned_stop_active) { + mark_planned_stop_owned_locked(fault_code); + } + return true; // New fault created } @@ -1006,7 +1100,9 @@ bool SqliteFaultStorage::clear_fault(const std::string & fault_code) { } } - SqliteStatement stmt(db_, "UPDATE faults SET status = ? WHERE fault_code = ?"); + // An acknowledged cycle is over, so the planned stop no longer owns it: it has + // nothing left to release or to announce for this fault. + SqliteStatement stmt(db_, "UPDATE faults SET status = ?, planned_stop_owned = 0 WHERE fault_code = ?"); stmt.bind_text(1, ros2_medkit_msgs::msg::Fault::STATUS_CLEARED); stmt.bind_text(2, fault_code); @@ -1049,7 +1145,7 @@ std::vector SqliteFaultStorage::reclassify_healed_as_cleared() { } } - SqliteStatement stmt(db_, "UPDATE faults SET status = ? WHERE status = ?"); + SqliteStatement stmt(db_, "UPDATE faults SET status = ?, planned_stop_owned = 0 WHERE status = ?"); stmt.bind_text(1, ros2_medkit_msgs::msg::Fault::STATUS_CLEARED); stmt.bind_text(2, ros2_medkit_msgs::msg::Fault::STATUS_HEALED); if (stmt.step() != SQLITE_DONE) { @@ -1285,6 +1381,80 @@ void SqliteFaultStorage::store_freeze_frame(const FreezeFrameData & frame) { } } +void SqliteFaultStorage::set_planned_stop(const PlannedStopState & state) { + std::lock_guard lock(mutex_); + + SqliteStatement stmt(db_, + "INSERT OR REPLACE INTO planned_stop (id, active, reason, declared_by, since_ns, ended_at_ns) " + "VALUES (1, ?, ?, ?, ?, ?)"); + stmt.bind_int(1, state.active ? 1 : 0); + stmt.bind_text(2, state.reason); + stmt.bind_text(3, state.declared_by); + stmt.bind_int64(4, state.since_ns); + stmt.bind_int64(5, state.ended_at_ns); + + if (stmt.step() != SQLITE_DONE) { + throw std::runtime_error(std::string("Failed to store planned stop: ") + sqlite3_errmsg(db_)); + } +} + +PlannedStopState SqliteFaultStorage::get_planned_stop() const { + std::lock_guard lock(mutex_); + + SqliteStatement stmt(db_, "SELECT active, reason, declared_by, since_ns, ended_at_ns FROM planned_stop WHERE id = 1"); + + PlannedStopState state; + if (stmt.step() != SQLITE_ROW) { + return state; // never declared: the default is "no stop" + } + + state.active = stmt.column_int(0) != 0; + state.reason = stmt.column_text(1); + state.declared_by = stmt.column_text(2); + state.since_ns = stmt.column_int64(3); + state.ended_at_ns = stmt.column_int64(4); + return state; +} + +std::vector SqliteFaultStorage::get_planned_stop_owned() const { + std::lock_guard lock(mutex_); + + SqliteStatement stmt(db_, "SELECT fault_code FROM faults WHERE planned_stop_owned != 0"); + + std::vector owned; + while (stmt.step() == SQLITE_ROW) { + owned.push_back(stmt.column_text(0)); + } + return owned; +} + +size_t SqliteFaultStorage::clear_planned_stop_owned() { + std::lock_guard lock(mutex_); + + SqliteStatement stmt(db_, "UPDATE faults SET planned_stop_owned = 0 WHERE planned_stop_owned != 0"); + if (stmt.step() != SQLITE_DONE) { + throw std::runtime_error(std::string("Failed to release planned-stop ownership: ") + sqlite3_errmsg(db_)); + } + return static_cast(sqlite3_changes(db_)); +} + +size_t SqliteFaultStorage::clear_planned_stop_owned(const std::vector & fault_codes) { + std::lock_guard lock(mutex_); + + size_t cleared = 0; + SqliteStatement stmt(db_, + "UPDATE faults SET planned_stop_owned = 0 WHERE fault_code = ? AND planned_stop_owned != 0"); + for (const auto & fault_code : fault_codes) { + stmt.reset(); + stmt.bind_text(1, fault_code); + if (stmt.step() != SQLITE_DONE) { + throw std::runtime_error(std::string("Failed to release planned-stop ownership: ") + sqlite3_errmsg(db_)); + } + cleared += static_cast(sqlite3_changes(db_)); + } + return cleared; +} + std::optional SqliteFaultStorage::get_freeze_frame(const std::string & fault_code) const { std::lock_guard lock(mutex_); diff --git a/src/ros2_medkit_fault_manager/test/integration/test_planned_stop.test.py b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop.test.py new file mode 100644 index 000000000..e1edcf315 --- /dev/null +++ b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop.test.py @@ -0,0 +1,960 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +End-to-end tests for the planned-stop switch on a live fault manager. + +Drives the real services over the real ROS graph against the SQLite backend +with the audit log on, and reads the audit database directly, because the +manager exposes no service that reads it back. + +The instrument for "the stream stayed quiet" is a subscriber on the manager's +own ``~/events`` topic, counted PER FAULT CODE: a count alone would pass while +the wrong fault was announced. +""" + +import json +import os +import sqlite3 +import tempfile +import threading +import time +import unittest + +from launch import LaunchDescription +import launch.actions +import launch_ros.actions +import launch_testing.actions +import rclpy +from rclpy.node import Node +from rclpy.qos import HistoryPolicy, QoSProfile, ReliabilityPolicy +from ros2_medkit_msgs.msg import Fault, FaultEvent +from ros2_medkit_msgs.srv import ( + ClearFault, + GetFault, + GetPlannedStop, + GetSnapshots, + ListFaults, + ReportFault, + SetPlannedStop, +) +from sensor_msgs.msg import Temperature + +STORAGE_DIR = tempfile.mkdtemp(prefix='planned_stop_') +DATABASE_PATH = os.path.join(STORAGE_DIR, 'faults.db') +AUDIT_PATH = os.path.join(STORAGE_DIR, 'fault_audit.db') +CORRELATION_PATH = os.path.join(STORAGE_DIR, 'correlation.yaml') + +# Two FAILED reports confirm, one leaves the fault PREFAILED, which is what the +# mid-debounce case needs. +CONFIRMATION_THRESHOLD = -2 + +SOURCE_ID = '/test_node' +CAPTURE_TOPIC = '/test/temperature' + +# Three rules. The hierarchical one mutes MOTOR_PS_* while ESTOP_PS_001 stands; the +# auto-cluster one claims VALVE_PS_* faults without ever writing them into the mute +# map, which is the case a planned stop's own entry has to survive. The second cluster +# rule needs four members, which is what makes a burst reachable that has shrunk two +# short of forming again. All windows are long enough that no test can lose the +# correlation to its own pacing. +CORRELATION_RULES = """ +correlation: + enabled: true + patterns: + ps_motor_errors: + codes: ["MOTOR_PS_*"] + ps_valve_errors: + codes: ["VALVE_PS_*"] + ps_pump_errors: + codes: ["PUMP_PS_*"] + ps_storm4_errors: + codes: ["STORM4_PS_*"] + rules: + - id: ps_estop_cascade + name: "E-Stop Cascade" + mode: hierarchical + root_cause: + codes: ["ESTOP_PS_*"] + symptoms: + - pattern: ps_motor_errors + window_ms: 60000 + mute_symptoms: true + auto_clear_with_root: true + - id: ps_estop_noclear + name: "E-Stop Cascade, symptoms left up" + mode: hierarchical + root_cause: + codes: ["ESTOP_NOCLEAR_001"] + symptoms: + - pattern: ps_pump_errors + window_ms: 60000 + mute_symptoms: true + auto_clear_with_root: false + - id: ps_valve_storm + name: "Valve Storm" + mode: auto_cluster + match: + - pattern: ps_valve_errors + min_count: 2 + window_ms: 60000 + show_as_single: true + representative: first + - id: ps_storm4 + name: "Four Valve Storm" + mode: auto_cluster + match: + - pattern: ps_storm4_errors + min_count: 4 + window_ms: 60000 + show_as_single: true + representative: first +""" + + +def get_coverage_env(): + """Get environment variables for gcov coverage data collection.""" + try: + from ament_index_python.packages import get_package_prefix + pkg_prefix = get_package_prefix('ros2_medkit_fault_manager') + workspace = os.path.dirname(os.path.dirname(pkg_prefix)) + build_dir = os.path.join(workspace, 'build', 'ros2_medkit_fault_manager') + + if os.path.exists(build_dir): + return { + 'GCOV_PREFIX': build_dir, + 'GCOV_PREFIX_STRIP': str(build_dir.count(os.sep)), + } + except Exception: + # Coverage environment is optional; on any error, fall back to no extra config + pass + return {} + + +def generate_test_description(): + """Launch a fault manager with SQLite storage, the audit log and one rule.""" + with open(CORRELATION_PATH, 'w') as handle: + handle.write(CORRELATION_RULES) + + fault_manager_env = get_coverage_env() + fault_manager_env['ROS_LOCALHOST_ONLY'] = '1' + + fault_manager_node = launch_ros.actions.Node( + package='ros2_medkit_fault_manager', + executable='fault_manager_node', + name='fault_manager', + output='screen', + additional_env=fault_manager_env, + parameters=[{ + 'storage_type': 'sqlite', + 'database_path': DATABASE_PATH, + 'confirmation_threshold': CONFIRMATION_THRESHOLD, + # A single PASSED heals, which is what makes the HEALED -> re-fail + # cycle boundary reachable from a test. + 'healing_enabled': True, + 'healing_threshold': 0, + 'audit_log.enabled': True, + 'audit_log.database_path': AUDIT_PATH, + 'correlation.config_file': CORRELATION_PATH, + # Capture must run for a muted fault exactly as it does for any + # other, so it is on and pointed at a topic this test publishes. + 'snapshots.enabled': True, + 'snapshots.default_topics': [CAPTURE_TOPIC], + 'snapshots.timeout_sec': 2.0, + 'snapshots.recapture_cooldown_sec': 0.0, + 'snapshots.rosbag.enabled': False, + }], + sigterm_timeout='30', + sigkill_timeout='15', + ) + + return ( + LaunchDescription([ + launch.actions.TimerAction(period=2.0, actions=[fault_manager_node]), + launch_testing.actions.ReadyToTest(), + ]), + {'fault_manager_node': fault_manager_node}, + ) + + +class TestPlannedStop(unittest.TestCase): + """The switch marks, counts, audits and later releases what it muted.""" + + @classmethod + def setUpClass(cls): + os.environ['ROS_LOCALHOST_ONLY'] = '1' + rclpy.init() + cls.node = Node('test_planned_stop_client') + + cls.report_client = cls.node.create_client(ReportFault, '/fault_manager/report_fault') + cls.list_client = cls.node.create_client(ListFaults, '/fault_manager/list_faults') + cls.get_client = cls.node.create_client(GetFault, '/fault_manager/get_fault') + cls.clear_client = cls.node.create_client(ClearFault, '/fault_manager/clear_fault') + cls.snapshots_client = cls.node.create_client(GetSnapshots, '/fault_manager/get_snapshots') + cls.set_stop_client = cls.node.create_client( + SetPlannedStop, '/fault_manager/set_planned_stop') + cls.get_stop_client = cls.node.create_client( + GetPlannedStop, '/fault_manager/get_planned_stop') + + for client, name in ( + (cls.report_client, 'report_fault'), + (cls.list_client, 'list_faults'), + (cls.get_client, 'get_fault'), + (cls.clear_client, 'clear_fault'), + (cls.snapshots_client, 'get_snapshots'), + (cls.set_stop_client, 'set_planned_stop'), + (cls.get_stop_client, 'get_planned_stop'), + ): + assert client.wait_for_service(timeout_sec=20.0), f'{name} service not available' + + # A topic for the capture set to sample, so a muted fault's freeze frame + # is a real reading rather than an empty one. + cls._temperature_pub = cls.node.create_publisher( + Temperature, CAPTURE_TOPIC, + QoSProfile(reliability=ReliabilityPolicy.BEST_EFFORT, + history=HistoryPolicy.KEEP_LAST, depth=10)) + cls._publish_timer = cls.node.create_timer(0.05, cls._publish_temperature) + + # Deep enough that a switch-off releasing hundreds of faults at once + # cannot overrun the reader while the spin thread is between wakeups. + cls._events = [] + cls._events_lock = threading.Lock() + cls._event_sub = cls.node.create_subscription( + FaultEvent, '/fault_manager/events', cls._on_event, + QoSProfile(reliability=ReliabilityPolicy.RELIABLE, + history=HistoryPolicy.KEEP_LAST, depth=2000)) + + cls._spinning = True + cls._spin_thread = threading.Thread(target=cls._spin, daemon=True) + cls._spin_thread.start() + + cls._prime_event_stream() + + @classmethod + def tearDownClass(cls): + cls._spinning = False + cls._spin_thread.join(timeout=5) + cls.node.destroy_node() + rclpy.shutdown() + + def tearDown(self): + # The switch is one declaration per manager, so a test that leaves it on + # would decide the next test's outcome. + self._set_stop(False, reason='teardown', declared_by='test') + + @classmethod + def _publish_temperature(cls): + msg = Temperature() + msg.temperature = 42.0 + msg.variance = 0.1 + cls._temperature_pub.publish(msg) + + @classmethod + def _on_event(cls, msg): + with cls._events_lock: + cls._events.append((msg.event_type, msg.fault.fault_code, msg.fault.status)) + + @classmethod + def _spin(cls): + while cls._spinning: + rclpy.spin_once(cls.node, timeout_sec=0.05) + + @classmethod + def _call(cls, client, request, timeout_sec=15.0): + future = client.call_async(request) + deadline = time.monotonic() + timeout_sec + while not future.done() and time.monotonic() < deadline: + time.sleep(0.005) + assert future.done(), 'service call timed out' + return future.result() + + @classmethod + def _report(cls, fault_code, event_type=ReportFault.Request.EVENT_FAILED): + request = ReportFault.Request() + request.fault_code = fault_code + request.event_type = event_type + request.severity = Fault.SEVERITY_ERROR + request.description = 'planned stop test fault' + request.source_id = SOURCE_ID + return cls._call(cls.report_client, request) + + @classmethod + def _confirm(cls, fault_code): + """Drive a fault to CONFIRMED with the configured debounce threshold.""" + for _ in range(abs(CONFIRMATION_THRESHOLD)): + cls._report(fault_code) + + @classmethod + def _prime_event_stream(cls): + """ + Block until an event demonstrably reaches this subscriber. + + ``~/events`` is reliable but volatile: an event published before the + subscription has matched the publisher is lost outright, so a one-shot + assertion on the first fault of the suite would race the match. + """ + deadline = time.monotonic() + 30.0 + attempt = 0 + while time.monotonic() < deadline: + code = f'PS_PRIME_{attempt}' + cls._confirm(code) + settle = time.monotonic() + 1.0 + while time.monotonic() < settle: + if cls._count_events(code, FaultEvent.EVENT_CONFIRMED) > 0: + return + time.sleep(0.05) + attempt += 1 + raise AssertionError('no event reached the subscriber; the events topic never went live') + + @classmethod + def _count_events(cls, fault_code, event_type): + with cls._events_lock: + return sum(1 for kind, code, _ in cls._events + if code == fault_code and kind == event_type) + + @classmethod + def _event_types(cls, fault_code): + """ + Return every event type published for this code, as a set. + + Counting one type cannot see a leak of another: a stop that announced an + UPDATE it should have withheld passes a CONFIRMED-only assertion. + """ + with cls._events_lock: + return {kind for kind, code, _ in cls._events if code == fault_code} + + def _set_stop(self, active, *, reason='', declared_by=''): + request = SetPlannedStop.Request() + request.active = active + request.reason = reason + request.declared_by = declared_by + return self._call(self.set_stop_client, request) + + def _get_stop(self): + return self._call(self.get_stop_client, GetPlannedStop.Request()) + + def _list(self, *, statuses=None, include_muted=False): + request = ListFaults.Request() + request.filter_by_severity = False + request.severity = 0 + request.statuses = statuses if statuses is not None else [] + request.include_muted = include_muted + request.include_clusters = False + return self._call(self.list_client, request) + + def _codes_in_default_list(self): + return {fault.fault_code for fault in self._list().faults} + + def _muted_entry(self, fault_code): + response = self._list(include_muted=True) + for info in response.muted_faults: + if info.fault_code == fault_code: + return info + return None + + def _clear(self, fault_code): + request = ClearFault.Request() + request.fault_code = fault_code + request.skip_correlation_auto_clear = False + return self._call(self.clear_client, request) + + def _get_fault(self, fault_code): + request = GetFault.Request() + request.fault_code = fault_code + return self._call(self.get_client, request) + + def _captured_topics(self, fault_code): + """Topic names present in the fault's stored capture, empty when none.""" + request = GetSnapshots.Request() + request.fault_code = fault_code + request.topic = '' + response = self._call(self.snapshots_client, request) + if not response.success or not response.data: + return {} + return json.loads(response.data).get('topics', {}) + + @staticmethod + def _audit_rows(transition=None): + """Read the audit database directly - no service serves it back.""" + connection = sqlite3.connect(f'file:{AUDIT_PATH}?mode=ro', uri=True) + try: + if transition is None: + cursor = connection.execute( + 'SELECT fault_code, transition, status, source_id, description FROM audit_log ' + 'ORDER BY seq') + else: + cursor = connection.execute( + 'SELECT fault_code, transition, status, source_id, description FROM audit_log ' + 'WHERE transition = ? ORDER BY seq', (transition,)) + return cursor.fetchall() + finally: + connection.close() + + def _wait_until(self, predicate, message, timeout=20.0): + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + value = predicate() + if value: + return value + time.sleep(0.1) + raise AssertionError(message) + + # ------------------------------------------------------------------ + # Cases + # ------------------------------------------------------------------ + + def test_a_fault_raised_during_a_stop_is_marked_not_dropped(self): + code = 'PS_MARKED' + + response = self._set_stop(True, reason='line 3 maintenance', declared_by='shift_lead') + self.assertTrue(response.success) + self.assertFalse(response.was_active) + + self._confirm(code) + + # In the store, CONFIRMED, and absent from the default list. + self._wait_until(lambda: self._get_fault(code).success, f'{code} never reached the store') + self.assertEqual('CONFIRMED', self._get_fault(code).fault.status) + self.assertNotIn(code, self._codes_in_default_list()) + + muted = self._muted_entry(code) + self.assertIsNotNone(muted, f'{code} is not listed as muted') + self.assertEqual('planned_stop', muted.rule_id) + self.assertEqual('PLANNED_STOP', muted.root_cause_code) + self.assertGreaterEqual(self._list().muted_count, 1) + + # Capture ran anyway: the evidence a muted fault leaves is the same. + captured = self._wait_until( + lambda: self._captured_topics(code), + f'no snapshot was captured for the muted fault {code}') + self.assertIn(CAPTURE_TOPIC, captured) + + # Re-report the marked fault while the stop still stands. This is the path + # that produces EVENT_UPDATED, and it is the only way an assertion about + # "nothing was announced" can see an update leaking out. + self._report(code) + self._wait_until(lambda: self._get_fault(code).fault.occurrence_count >= 1, + f'{code} left the store') + + # Nothing at all was announced while the stop stood - not the confirmation, + # and not the update either. + self.assertEqual(set(), self._event_types(code)) + + withdrawal = self._set_stop(False, reason='line 3 back up', declared_by='shift_lead') + self.assertTrue(withdrawal.success) + self.assertTrue(withdrawal.was_active) + + self._wait_until(lambda: code in self._codes_in_default_list(), + f'{code} did not return to the default list after the stop ended') + self.assertIsNone(self._muted_entry(code)) + self.assertEqual( + 1, self._count_events(code, FaultEvent.EVENT_CONFIRMED), + 'the confirmation held back by the stop must be announced exactly once') + self.assertEqual({FaultEvent.EVENT_CONFIRMED}, self._event_types(code), + 'the switch-off announces the confirmation and nothing else') + + def test_a_fault_cleared_during_a_stop_never_reaches_the_default_list(self): + code = 'PS_CLEARED_INSIDE' + + self._set_stop(True, reason='changeover', declared_by='maintenance') + self._confirm(code) + self._wait_until(lambda: self._get_fault(code).success, f'{code} never reached the store') + self.assertIsNotNone(self._muted_entry(code)) + + cleared = self._clear(code) + self.assertTrue(cleared.success) + + self.assertNotIn(code, self._codes_in_default_list()) + self.assertIsNone(self._muted_entry(code)) + + # It is still in the store, and in the audit trail. + stored = self._get_fault(code) + self.assertTrue(stored.success) + self.assertEqual('CLEARED', stored.fault.status) + cleared_codes = {fault.fault_code for fault in self._list(statuses=['CLEARED']).faults} + self.assertIn(code, cleared_codes) + transitions = {row[1] for row in self._audit_rows() if row[0] == code} + self.assertIn('occurred', transitions) + self.assertIn('confirmed', transitions) + self.assertIn('cleared', transitions) + + # The acknowledgement is published, as it is for any muted fault; the + # confirmation behind the mute is not. Assert the whole set, so an + # announcement of any other kind is a failure rather than an unchecked + # extra frame. + self.assertEqual({FaultEvent.EVENT_CLEARED}, self._event_types(code)) + self.assertEqual(1, self._count_events(code, FaultEvent.EVENT_CLEARED)) + + self._set_stop(False, reason='done', declared_by='maintenance') + + self.assertNotIn(code, self._codes_in_default_list()) + self.assertEqual( + {FaultEvent.EVENT_CLEARED}, self._event_types(code), + 'a fault that was acknowledged inside the stop has nothing left to announce') + + def test_repeating_a_request_changes_nothing_and_records_nothing(self): + started_before = len(self._audit_rows('planned_stop_started')) + ended_before = len(self._audit_rows('planned_stop_ended')) + + first_on = self._set_stop(True, reason='weekend shutdown', declared_by='plant_manager') + self.assertTrue(first_on.success) + self.assertFalse(first_on.was_active) + + second_on = self._set_stop(True, reason='ignored', declared_by='someone_else') + self.assertTrue(second_on.success) + self.assertTrue(second_on.was_active) + + started = self._audit_rows('planned_stop_started') + self.assertEqual(started_before + 1, len(started), + 'declaring a stop that is already in force must write no audit record') + self.assertEqual('__audit__', started[-1][0]) + self.assertEqual('ACTIVE', started[-1][2]) + self.assertEqual('plant_manager', started[-1][3]) + self.assertEqual('weekend shutdown', started[-1][4]) + + # The declaration the manager serves is the first one, not the repeat. + state = self._get_stop() + self.assertTrue(state.active) + self.assertEqual('weekend shutdown', state.reason) + self.assertEqual('plant_manager', state.declared_by) + + first_off = self._set_stop(False, reason='restart', declared_by='plant_manager') + self.assertTrue(first_off.success) + self.assertTrue(first_off.was_active) + + second_off = self._set_stop(False, reason='ignored', declared_by='someone_else') + self.assertTrue(second_off.success) + self.assertFalse(second_off.was_active) + + ended = self._audit_rows('planned_stop_ended') + self.assertEqual(ended_before + 1, len(ended), + 'withdrawing a stop that is not in force must write no audit record') + self.assertEqual('__audit__', ended[-1][0]) + self.assertEqual('INACTIVE', ended[-1][2]) + self.assertEqual('plant_manager', ended[-1][3]) + self.assertEqual('restart', ended[-1][4]) + + state = self._get_stop() + self.assertFalse(state.active) + # The declaration outlives the stop: the repeat did not overwrite it, and + # the withdrawal did not erase it. + self.assertEqual('weekend shutdown', state.reason) + self.assertEqual('plant_manager', state.declared_by) + self.assertGreater(state.ended_at.sec, 0) + + def test_the_reason_and_declarer_may_be_empty_or_long(self): + empty = self._set_stop(True, reason='', declared_by='') + self.assertTrue(empty.success) + state = self._get_stop() + self.assertTrue(state.active) + self.assertEqual('', state.reason) + self.assertEqual('', state.declared_by) + self.assertGreater(state.since.sec, 0, 'a declared stop must carry the time it started') + + self._set_stop(False) + + long_reason = 'r' * 1024 + self.assertTrue(self._set_stop(True, reason=long_reason, declared_by='a' * 256).success) + state = self._get_stop() + self.assertTrue(state.active) + self.assertEqual(long_reason, state.reason) + self.assertEqual('a' * 256, state.declared_by) + self.assertEqual(long_reason, self._audit_rows('planned_stop_started')[-1][4]) + + def test_two_hundred_faults_muted_by_one_stop_are_all_released(self): + codes = [f'PS_SCALE_{index:03d}' for index in range(200)] + + self._set_stop(True, reason='annual shutdown', declared_by='plant_manager') + for code in codes: + self._confirm(code) + + listed = self._list(include_muted=True) + muted_codes = {info.fault_code for info in listed.muted_faults} + missing = [code for code in codes if code not in muted_codes] + self.assertEqual([], missing, f'{len(missing)} of 200 faults were not marked muted') + + visible = self._codes_in_default_list() + self.assertEqual([], [code for code in codes if code in visible], + 'a muted fault must not appear in the default list') + + self._set_stop(False, reason='shutdown over', declared_by='plant_manager') + + self._wait_until( + lambda: not [code for code in codes if code not in self._codes_in_default_list()], + 'not every fault the stop muted came back to the default list') + + def _miscounted(): + return [(code, self._count_events(code, FaultEvent.EVENT_CONFIRMED)) + for code in codes + if self._count_events(code, FaultEvent.EVENT_CONFIRMED) != 1] + + deadline = time.monotonic() + 30.0 + wrong = _miscounted() + while wrong and time.monotonic() < deadline: + time.sleep(0.2) + wrong = _miscounted() + self.assertEqual( + [], wrong, + f'{len(wrong)} of 200 faults were not announced exactly once at the switch-off; ' + f'first offenders: {wrong[:10]}') + + def test_a_fault_mid_debounce_when_the_stop_ends_confirms_afterwards(self): + code = 'PS_MID_DEBOUNCE' + + self._set_stop(True, reason='calibration', declared_by='tech') + # One report short of confirmation: the fault is PREFAILED and muted. + self._report(code) + self._wait_until(lambda: self._get_fault(code).success, f'{code} never reached the store') + self.assertEqual('PREFAILED', self._get_fault(code).fault.status) + self.assertIsNotNone(self._muted_entry(code)) + + self._set_stop(False, reason='calibration over', declared_by='tech') + + self.assertIsNone(self._muted_entry(code)) + self.assertEqual(0, self._count_events(code, FaultEvent.EVENT_CONFIRMED), + 'a fault still short of confirmation has nothing to announce') + + # The debounce carries on across the switch: one more report confirms it, + # and with the stop gone the confirmation is announced. + self._report(code) + self._wait_until(lambda: self._count_events(code, FaultEvent.EVENT_CONFIRMED) == 1, + 'the confirmation after the stop ended was never announced') + self.assertEqual('CONFIRMED', self._get_fault(code).fault.status) + self.assertIn(code, self._codes_in_default_list()) + + def test_a_fault_already_up_when_the_stop_begins_stays_visible(self): + """A cycle that started before the stop is not the stop's to hide.""" + code = 'PS_ALREADY_UP' + + # Confirmed and announced with no stop in force. + self._confirm(code) + self._wait_until(lambda: self._count_events(code, FaultEvent.EVENT_CONFIRMED) == 1, + f'{code} was never announced before the stop') + self.assertIn(code, self._codes_in_default_list()) + + self._set_stop(True, reason='line 3 maintenance', declared_by='shift_lead') + + # A level-triggered reporter keeps sending FAILED while the condition + # holds. None of those reports starts a cycle. + for _ in range(3): + self._report(code) + + self.assertIsNone(self._muted_entry(code), + 'the stop took over a fault whose cycle started before it') + self.assertIn(code, self._codes_in_default_list(), + 'a standing alarm left the fault list when the stop was declared') + + self._set_stop(False, reason='line 3 back up', declared_by='shift_lead') + + self.assertEqual( + 1, self._count_events(code, FaultEvent.EVENT_CONFIRMED), + 'the switch-off announced a confirmation that had already been announced') + + def test_a_fault_raised_again_after_a_clear_during_the_stop_is_marked(self): + """Re-raising a cleared fault starts a new cycle, and the stop takes it.""" + code = 'PS_RERAISED' + + self._confirm(code) + self._wait_until(lambda: self._get_fault(code).success, f'{code} never reached the store') + self.assertTrue(self._clear(code).success) + + self._set_stop(True, reason='changeover', declared_by='maintenance') + self._confirm(code) + + muted = self._wait_until(lambda: self._muted_entry(code), + f'the re-raised {code} was not marked by the stop') + self.assertEqual('planned_stop', muted.rule_id) + self.assertNotIn(code, self._codes_in_default_list()) + + before = self._count_events(code, FaultEvent.EVENT_CONFIRMED) + self._set_stop(False, reason='done', declared_by='maintenance') + self._wait_until( + lambda: self._count_events(code, FaultEvent.EVENT_CONFIRMED) == before + 1, + 'the new cycle the stop held back was never announced') + + def test_a_scoped_acknowledgement_still_releases_the_stops_mute(self): + """A clear that skips the correlation cascade still drops the stop's mute.""" + code = 'PS_SCOPED_ACK' + + self._set_stop(True, reason='cell 7 changeover', declared_by='maintenance') + self._confirm(code) + self._wait_until(lambda: self._muted_entry(code), f'{code} was not marked by the stop') + before_count = self._list().muted_count + + # The shape every per-entity DELETE takes: acknowledge this fault only, + # without walking the correlation graph into other entities. + request = ClearFault.Request() + request.fault_code = code + request.skip_correlation_auto_clear = True + self.assertTrue(self._call(self.clear_client, request).success) + + self.assertIsNone(self._muted_entry(code), + 'an acknowledged fault stayed listed as muted by the stop') + self.assertLess(self._list().muted_count, before_count, + 'muted_count still counts a fault that was acknowledged') + + self._set_stop(False, reason='done', declared_by='maintenance') + self.assertEqual(0, self._count_events(code, FaultEvent.EVENT_CONFIRMED)) + + def test_a_cluster_still_hiding_a_fault_keeps_it_at_the_switch_off(self): + """The cluster shows one line for the burst, and the switch-off honours it.""" + first = 'VALVE_PS_A' + second = 'VALVE_PS_B' + + self._set_stop(True, reason='valve bank swap', declared_by='maintenance') + self._confirm(first) + self._confirm(second) + + entries = self._wait_until( + lambda: [self._muted_entry(first), self._muted_entry(second)] + if self._muted_entry(first) and self._muted_entry(second) else None, + 'the clustered faults were not marked by the stop') + for entry in entries: + self.assertEqual('planned_stop', entry.rule_id, + 'the cluster took an entry it never writes') + + visible = self._codes_in_default_list() + self.assertNotIn(first, visible) + self.assertNotIn(second, visible) + + self._set_stop(False, reason='swap done', declared_by='maintenance') + + # The representative is released and announced once. + self._wait_until( + lambda: first in self._codes_in_default_list(), + 'the cluster representative was orphaned in the mute map') + self.assertIsNone(self._muted_entry(first)) + self.assertEqual(1, self._count_events(first, FaultEvent.EVENT_CONFIRMED)) + + # The symptom is not announced, and the stop leaves nothing of its own behind + # on it either. A cluster hides a member by suppressing the member's events on + # every report, never by an entry in the muted list, so after the switch-off + # this burst looks exactly like one that never met a planned stop. + self.assertIsNone(self._muted_entry(second), + 'the switch-off left an entry the cluster never writes') + self.assertIn(second, self._codes_in_default_list()) + self.assertEqual(0, self._count_events(second, FaultEvent.EVENT_CONFIRMED), + 'both symptoms of one cluster were announced at the switch-off') + + # Still hidden: the cluster swallows the non-representative's next report. + types_before = self._event_types(second) + self._report(second) + self._get_fault(second) # a round trip through the same single-threaded node + time.sleep(0.5) + self.assertEqual(types_before, self._event_types(second), + 'the cluster stopped hiding its member once the stop ended') + + # Acknowledging the representative promotes the member, and a promoted + # representative is the line the cluster shows: it is heard from again. + self.assertTrue(self._clear(first).success) + self._report(second) + self._wait_until( + lambda: FaultEvent.EVENT_UPDATED in self._event_types(second), + 'the promoted representative stayed silent') + + def test_a_fault_joining_a_cluster_below_min_count_is_not_folded_into_it(self): + """A cluster folds the members it formed with; a later joiner is a fault of its own.""" + formed = ['STORM4_PS_A', 'STORM4_PS_B', 'STORM4_PS_C', 'STORM4_PS_D'] + joiner = 'STORM4_PS_E' + representative = formed[0] + + for code in formed: + self._confirm(code) + self._wait_until( + lambda: self._count_events(representative, FaultEvent.EVENT_CONFIRMED) == 1, + 'the cluster representative was never announced') + # Only the member whose confirmation landed once the cluster had formed is + # hidden. The ones that confirmed on the way to min_count were announced + # before there was a cluster to fold them into, stop or no stop. + self.assertEqual(0, self._count_events(formed[3], FaultEvent.EVENT_CONFIRMED), + 'the member that confirmed inside the formed cluster was announced') + + # Two acknowledgements leave the burst two members short of forming again. + for code in formed[2:]: + self.assertTrue(self._clear(code).success) + + # The joiner maps to the formed cluster through the pending twin, but it is not + # one of the members that cluster shows as a single line. + self._confirm(joiner) + self._wait_until( + lambda: self._count_events(joiner, FaultEvent.EVENT_CONFIRMED) == 1, + 'a fault that joined a cluster below min_count was folded into it and never announced') + + # And it keeps being heard from, which is what a fault outside any cluster does. + self._report(joiner) + self._wait_until( + lambda: self._count_events(joiner, FaultEvent.EVENT_UPDATED) >= 1, + 'a fault that joined below min_count went silent after its first announcement') + self.assertIsNone(self._muted_entry(joiner)) + self.assertIn(joiner, self._codes_in_default_list()) + + def test_the_last_declaration_is_readable_after_the_withdrawal(self): + """The reason survives the withdrawal, with the time the stop ended.""" + self._set_stop(True, reason='line 3 quarterly maintenance', declared_by='shift_lead') + during = self._get_stop() + self.assertTrue(during.active) + self.assertEqual(0, during.ended_at.sec) + + self._set_stop(False, reason='plant back up', declared_by='night_shift') + + after = self._get_stop() + self.assertFalse(after.active) + self.assertEqual('line 3 quarterly maintenance', after.reason) + self.assertEqual('shift_lead', after.declared_by) + self.assertEqual(during.since.sec, after.since.sec) + self.assertGreater(after.ended_at.sec, 0) + + # The withdrawal's own reason and declarer are in the audit row, not in the + # declaration. + ended = self._audit_rows('planned_stop_ended')[-1] + self.assertEqual('night_shift', ended[3]) + self.assertEqual('plant back up', ended[4]) + + def test_a_healed_fault_that_fails_again_inside_the_stop_is_owned(self): + """A heal ends the cycle, so the next failure starts one the stop owns.""" + code = 'PS_HEALED_REFAIL' + + self._confirm(code) + self._wait_until(lambda: self._count_events(code, FaultEvent.EVENT_CONFIRMED) == 1, + f'{code} was never announced before the stop') + # The debounce counter sits at the confirmation threshold, so it takes that + # many PASSED reports to climb back to the healing threshold of 0. + for _ in range(abs(CONFIRMATION_THRESHOLD)): + self._report(code, event_type=ReportFault.Request.EVENT_PASSED) + self._wait_until(lambda: self._get_fault(code).fault.status == 'HEALED', + f'{code} never healed') + + self._set_stop(True, reason='bench work', declared_by='tech') + + # The condition returns. That is a new cycle by the manager's own model, + # and the stop owns it. + self._confirm(code) + muted = self._wait_until(lambda: self._muted_entry(code), + f'the re-failed {code} was not marked by the stop') + self.assertEqual('planned_stop', muted.rule_id) + self.assertNotIn(code, self._codes_in_default_list()) + self.assertEqual(1, self._count_events(code, FaultEvent.EVENT_CONFIRMED), + 'the re-failure was announced despite the stop') + + self._set_stop(False, reason='bench work over', declared_by='tech') + self._wait_until(lambda: self._count_events(code, FaultEvent.EVENT_CONFIRMED) == 2, + 'the confirmation the stop held back was never announced') + + def test_a_passed_report_announces_nothing_for_a_marked_fault(self): + """Muting gates the PASSED path too, not only the FAILED one.""" + code = 'PS_PASSED_QUIET' + + self._set_stop(True, reason='changeover', declared_by='maintenance') + self._confirm(code) + self._wait_until(lambda: self._muted_entry(code), f'{code} was not marked by the stop') + + # A PASSED that does not heal leaves the fault CONFIRMED with fresh data - + # the update path. Nothing about it may reach the stream. + self._report(code, event_type=ReportFault.Request.EVENT_PASSED) + self._wait_until(lambda: self._get_fault(code).success, f'{code} left the store') + self.assertEqual(set(), self._event_types(code), + 'a PASSED report announced an update for a marked fault') + + self._set_stop(False, reason='done', declared_by='maintenance') + + def test_a_rule_muted_symptom_announces_nothing_on_a_passed_report(self): + """ + Apply the same gate to a fault a correlation rule is muting. + + Its own root cause code: a root shared with another case would already be + CONFIRMED there, and a cycle that started outside this test's stop is not + the stop's to own. + """ + root = 'ESTOP_PS_PASSED' + symptom = 'MOTOR_PS_PASSED' + + self._confirm(root) + self._confirm(symptom) + muted = self._wait_until(lambda: self._muted_entry(symptom), + f'{symptom} was not muted by the rule') + self.assertEqual('ps_estop_cascade', muted.rule_id) + + self._report(symptom, event_type=ReportFault.Request.EVENT_PASSED) + self._wait_until(lambda: self._get_fault(symptom).success, f'{symptom} left the store') + self.assertEqual(set(), self._event_types(symptom), + 'a PASSED report announced an update for a rule-muted fault') + + def test_a_symptom_returns_to_the_stop_when_its_root_cause_is_acknowledged(self): + """A rule's mute is an overlay; ownership outlives it.""" + root = 'ESTOP_NOCLEAR_001' + symptom = 'PUMP_PS_SYMPTOM' + + self._set_stop(True, reason='cell 7 changeover', declared_by='maintenance') + self._confirm(root) + self._confirm(symptom) + + muted = self._wait_until(lambda: self._muted_entry(symptom), + f'{symptom} was not muted at all') + self.assertEqual('ps_estop_noclear', muted.rule_id, + 'the rule, not the stop, holds the mute while the root cause stands') + + # Acknowledging the root cause ends the rule's mute. This rule does not + # auto-clear its symptoms, so the symptom is still up - and still the + # stop's. + self.assertTrue(self._clear(root).success) + + still_muted = self._wait_until(lambda: self._muted_entry(symptom), + 'the symptom fell out of the stop when the rule let go') + self.assertEqual('planned_stop', still_muted.rule_id) + self.assertNotIn(symptom, self._codes_in_default_list()) + self.assertEqual(set(), self._event_types(symptom)) + + self._set_stop(False, reason='done', declared_by='maintenance') + self._wait_until(lambda: self._count_events(symptom, FaultEvent.EVENT_CONFIRMED) == 1, + 'the symptom the stop had owned all along was never announced') + + def test_the_switch_does_not_release_a_rule_muted_symptom(self): + root = 'ESTOP_PS_001' + symptom = 'MOTOR_PS_LEFT' + + self._set_stop(True, reason='e-stop drill', declared_by='safety') + self._confirm(root) + self._confirm(symptom) + + self._wait_until(lambda: self._get_fault(symptom).success, + f'{symptom} never reached the store') + root_muted = self._muted_entry(root) + self.assertIsNotNone(root_muted) + self.assertEqual('planned_stop', root_muted.rule_id) + symptom_muted = self._muted_entry(symptom) + self.assertIsNotNone(symptom_muted) + self.assertEqual('ps_estop_cascade', symptom_muted.rule_id, + 'the correlation rule, not the switch, must own this mute') + + self._set_stop(False, reason='drill over', declared_by='safety') + + self._wait_until(lambda: root in self._codes_in_default_list(), + 'the root cause did not return to the default list') + self.assertEqual(1, self._count_events(root, FaultEvent.EVENT_CONFIRMED)) + + still_muted = self._muted_entry(symptom) + self.assertIsNotNone(still_muted, 'the switch released a symptom a rule was muting') + self.assertEqual('ps_estop_cascade', still_muted.rule_id) + self.assertNotIn(symptom, self._codes_in_default_list()) + self.assertEqual(0, self._count_events(symptom, FaultEvent.EVENT_CONFIRMED)) + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + for info in proc_info: + self.assertIn(info.returncode, (0, -2, -15), + f'{info.process_name} exited with code {info.returncode}') diff --git a/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_auto_confirm.test.py b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_auto_confirm.test.py new file mode 100644 index 000000000..590de59e4 --- /dev/null +++ b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_auto_confirm.test.py @@ -0,0 +1,264 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +A time-based confirmation obeys the planned stop like any other. + +``auto_confirm_after_sec`` confirms a lingering PREFAILED fault from a timer +rather than from a report, on a code path of its own. A stop that only silenced +the report path would let that timer announce, over the stream, a fault the list +is hiding - so this drives a confirmation THROUGH the timer while a stop stands. +""" + +import os +import tempfile +import threading +import time +import unittest + +from launch import LaunchDescription +import launch.actions +import launch_ros.actions +import launch_testing.actions +import rclpy +from rclpy.node import Node +from rclpy.qos import HistoryPolicy, QoSProfile, ReliabilityPolicy +from ros2_medkit_msgs.msg import Fault, FaultEvent +from ros2_medkit_msgs.srv import GetFault, ListFaults, ReportFault, SetPlannedStop + +STORAGE_DIR = tempfile.mkdtemp(prefix='planned_stop_auto_confirm_') +DATABASE_PATH = os.path.join(STORAGE_DIR, 'faults.db') + +# One FAILED report leaves the fault PREFAILED; the timer is then the only thing +# that can confirm it. +CONFIRMATION_THRESHOLD = -2 +AUTO_CONFIRM_AFTER_SEC = 2.0 + +MUTED_CODE = 'PS_TIMER_MUTED' +CONTROL_CODE = 'PS_TIMER_CONTROL' + + +def get_coverage_env(): + """Get environment variables for gcov coverage data collection.""" + try: + from ament_index_python.packages import get_package_prefix + pkg_prefix = get_package_prefix('ros2_medkit_fault_manager') + workspace = os.path.dirname(os.path.dirname(pkg_prefix)) + build_dir = os.path.join(workspace, 'build', 'ros2_medkit_fault_manager') + + if os.path.exists(build_dir): + return { + 'GCOV_PREFIX': build_dir, + 'GCOV_PREFIX_STRIP': str(build_dir.count(os.sep)), + } + except Exception: + # Coverage environment is optional; on any error, fall back to no extra config + pass + return {} + + +def generate_test_description(): + """Launch a fault manager that auto-confirms lingering PREFAILED faults.""" + fault_manager_env = get_coverage_env() + fault_manager_env['ROS_LOCALHOST_ONLY'] = '1' + + fault_manager_node = launch_ros.actions.Node( + package='ros2_medkit_fault_manager', + executable='fault_manager_node', + name='fault_manager', + output='screen', + additional_env=fault_manager_env, + parameters=[{ + 'storage_type': 'sqlite', + 'database_path': DATABASE_PATH, + 'confirmation_threshold': CONFIRMATION_THRESHOLD, + 'auto_confirm_after_sec': AUTO_CONFIRM_AFTER_SEC, + 'snapshots.enabled': False, + 'snapshots.rosbag.enabled': False, + }], + sigterm_timeout='30', + sigkill_timeout='15', + ) + + return ( + LaunchDescription([ + launch.actions.TimerAction(period=2.0, actions=[fault_manager_node]), + launch_testing.actions.ReadyToTest(), + ]), + {'fault_manager_node': fault_manager_node}, + ) + + +class TestPlannedStopAutoConfirm(unittest.TestCase): + """The timer path honours the stop, and announces once it is withdrawn.""" + + @classmethod + def setUpClass(cls): + os.environ['ROS_LOCALHOST_ONLY'] = '1' + rclpy.init() + cls.node = Node('test_planned_stop_auto_confirm_client') + + cls.report_client = cls.node.create_client(ReportFault, '/fault_manager/report_fault') + cls.list_client = cls.node.create_client(ListFaults, '/fault_manager/list_faults') + cls.get_client = cls.node.create_client(GetFault, '/fault_manager/get_fault') + cls.set_stop_client = cls.node.create_client( + SetPlannedStop, '/fault_manager/set_planned_stop') + for client, name in ( + (cls.report_client, 'report_fault'), + (cls.list_client, 'list_faults'), + (cls.get_client, 'get_fault'), + (cls.set_stop_client, 'set_planned_stop'), + ): + assert client.wait_for_service(timeout_sec=20.0), f'{name} service not available' + + cls._events = [] + cls._events_lock = threading.Lock() + cls._event_sub = cls.node.create_subscription( + FaultEvent, '/fault_manager/events', cls._on_event, + QoSProfile(reliability=ReliabilityPolicy.RELIABLE, + history=HistoryPolicy.KEEP_LAST, depth=200)) + + cls._spinning = True + cls._spin_thread = threading.Thread(target=cls._spin, daemon=True) + cls._spin_thread.start() + cls._prime_event_stream() + + @classmethod + def tearDownClass(cls): + cls._spinning = False + cls._spin_thread.join(timeout=5) + cls.node.destroy_node() + rclpy.shutdown() + + @classmethod + def _on_event(cls, msg): + with cls._events_lock: + cls._events.append((msg.event_type, msg.fault.fault_code)) + + @classmethod + def _spin(cls): + while cls._spinning: + rclpy.spin_once(cls.node, timeout_sec=0.05) + + @classmethod + def _call(cls, client, request, timeout_sec=15.0): + future = client.call_async(request) + deadline = time.monotonic() + timeout_sec + while not future.done() and time.monotonic() < deadline: + time.sleep(0.005) + assert future.done(), 'service call timed out' + return future.result() + + @classmethod + def _report(cls, fault_code): + request = ReportFault.Request() + request.fault_code = fault_code + request.event_type = ReportFault.Request.EVENT_FAILED + request.severity = Fault.SEVERITY_ERROR + request.description = 'auto confirm test fault' + request.source_id = '/test_node' + return cls._call(cls.report_client, request) + + @classmethod + def _count_events(cls, fault_code, event_type): + with cls._events_lock: + return sum(1 for kind, code in cls._events + if code == fault_code and kind == event_type) + + @classmethod + def _prime_event_stream(cls): + """Block until an event demonstrably reaches this subscriber.""" + deadline = time.monotonic() + 40.0 + attempt = 0 + while time.monotonic() < deadline: + code = f'PS_TIMER_PRIME_{attempt}' + cls._report(code) + cls._report(code) + settle = time.monotonic() + 1.0 + while time.monotonic() < settle: + if cls._count_events(code, FaultEvent.EVENT_CONFIRMED) > 0: + return + time.sleep(0.05) + attempt += 1 + raise AssertionError('no event reached the subscriber; the events topic never went live') + + def _set_stop(self, active, *, reason='', declared_by=''): + request = SetPlannedStop.Request() + request.active = active + request.reason = reason + request.declared_by = declared_by + return self._call(self.set_stop_client, request) + + def _status(self, fault_code): + request = GetFault.Request() + request.fault_code = fault_code + response = self._call(self.get_client, request) + return response.fault.status if response.success else None + + def _muted_codes(self): + request = ListFaults.Request() + request.filter_by_severity = False + request.severity = 0 + request.statuses = [] + request.include_muted = True + request.include_clusters = False + return {info.fault_code for info in self._call(self.list_client, request).muted_faults} + + def _wait_until(self, predicate, message, timeout=30.0): + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + value = predicate() + if value: + return value + time.sleep(0.1) + raise AssertionError(message) + + def test_a_timer_confirmation_inside_a_stop_is_held_back_until_it_ends(self): + # The control fault proves the timer really fires in this configuration: + # without it, silence for the muted code would prove nothing. + self._report(CONTROL_CODE) + self._wait_until(lambda: self._status(CONTROL_CODE) == 'CONFIRMED', + 'the auto-confirm timer never confirmed the control fault, so the ' + 'muted case below would prove nothing') + self.assertEqual(1, self._count_events(CONTROL_CODE, FaultEvent.EVENT_CONFIRMED)) + + self.assertTrue(self._set_stop(True, reason='timer case', declared_by='tech').success) + + self._report(MUTED_CODE) + self.assertIn(MUTED_CODE, self._muted_codes()) + + # The timer confirms it in the store while the stop stands. + self._wait_until(lambda: self._status(MUTED_CODE) == 'CONFIRMED', + 'the auto-confirm timer never confirmed the muted fault') + self.assertEqual( + 0, self._count_events(MUTED_CODE, FaultEvent.EVENT_CONFIRMED), + 'the timer announced a confirmation the planned stop was holding back') + + withdrawn = self._set_stop(False, reason='timer case over', declared_by='tech') + self.assertTrue(withdrawn.success) + + self._wait_until( + lambda: self._count_events(MUTED_CODE, FaultEvent.EVENT_CONFIRMED) == 1, + 'the timer-driven confirmation was never announced after the stop ended') + self.assertNotIn(MUTED_CODE, self._muted_codes()) + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + for info in proc_info: + self.assertIn(info.returncode, (0, -2, -15), + f'{info.process_name} exited with code {info.returncode}') diff --git a/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_interrupted_release.test.py b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_interrupted_release.test.py new file mode 100644 index 000000000..d96a9519b --- /dev/null +++ b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_interrupted_release.test.py @@ -0,0 +1,397 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +A switch-off killed halfway is finished by the next startup. + +Withdrawing a planned stop writes the declaration, announces the confirmations it +was holding back, and only then drops the ownership flags. A process that dies +between the write and the flags leaves faults owned by a declaration that is +already over - and their confirmations, suppressed while the stop stood, have +been announced by nobody. + +The kill is real (SIGTERM, launch respawns the node); the crash *point* is +simulated by editing the store while the manager is down, which is exactly the +state such a crash leaves behind. + +The recovery runs in the constructor, before this node has a service or a timer, +so its announcement may reach no subscriber - nothing has matched a publisher that +is milliseconds old. That loss is accepted, so the assertions here are on what +survives it: the manager's own log line, the ownership flags being gone, and the +released faults being back in the default fault list. What the recovery must NEVER +do is announce or release a fault owned by a stop declared after it. +""" + +import os +import signal +import sqlite3 +import tempfile +import threading +import time +import unittest + +from launch import LaunchDescription +import launch.actions +import launch_ros.actions +import launch_testing.actions +import rclpy +from rclpy.node import Node +from rclpy.qos import HistoryPolicy, QoSProfile, ReliabilityPolicy +from ros2_medkit_msgs.msg import Fault, FaultEvent +from ros2_medkit_msgs.srv import ( + ClearFault, + GetPlannedStop, + ListFaults, + ReportFault, + SetPlannedStop, +) + +STORAGE_DIR = tempfile.mkdtemp(prefix='planned_stop_interrupted_') +DATABASE_PATH = os.path.join(STORAGE_DIR, 'faults.db') + +OWNED_CODES = ('PS_INTERRUPTED_ONE', 'PS_INTERRUPTED_TWO') +# Re-raised under a NEW stop declared right after the recovery. +REUSED_CODE = OWNED_CODES[0] +NEW_STOP_CODE = 'PS_AFTER_RECOVERY' + +# Long enough that the test can edit the store between the kill and the +# replacement opening it. +RESPAWN_DELAY_SEC = 6.0 + + +def get_coverage_env(): + """Get environment variables for gcov coverage data collection.""" + try: + from ament_index_python.packages import get_package_prefix + pkg_prefix = get_package_prefix('ros2_medkit_fault_manager') + workspace = os.path.dirname(os.path.dirname(pkg_prefix)) + build_dir = os.path.join(workspace, 'build', 'ros2_medkit_fault_manager') + + if os.path.exists(build_dir): + return { + 'GCOV_PREFIX': build_dir, + 'GCOV_PREFIX_STRIP': str(build_dir.count(os.sep)), + } + except Exception: + # Coverage environment is optional; on any error, fall back to no extra config + pass + return {} + + +def generate_test_description(): + """Launch a fault manager launch brings back after it is killed.""" + fault_manager_env = get_coverage_env() + fault_manager_env['ROS_LOCALHOST_ONLY'] = '1' + + fault_manager_node = launch_ros.actions.Node( + package='ros2_medkit_fault_manager', + executable='fault_manager_node', + name='fault_manager', + output='screen', + additional_env=fault_manager_env, + parameters=[{ + 'storage_type': 'sqlite', + 'database_path': DATABASE_PATH, + 'confirmation_threshold': -1, + 'snapshots.enabled': False, + 'snapshots.rosbag.enabled': False, + }], + respawn=True, + respawn_delay=RESPAWN_DELAY_SEC, + sigterm_timeout='30', + sigkill_timeout='15', + ) + + return ( + LaunchDescription([ + launch.actions.TimerAction(period=2.0, actions=[fault_manager_node]), + launch_testing.actions.ReadyToTest(), + ]), + {'fault_manager_node': fault_manager_node}, + ) + + +class TestInterruptedRelease(unittest.TestCase): + """A release the previous process did not finish is finished at startup.""" + + @classmethod + def setUpClass(cls): + os.environ['ROS_LOCALHOST_ONLY'] = '1' + rclpy.init() + cls.node = Node('test_planned_stop_interrupted_client') + + cls._events = [] + cls._events_lock = threading.Lock() + cls._event_sub = cls.node.create_subscription( + FaultEvent, '/fault_manager/events', cls._on_event, + QoSProfile(reliability=ReliabilityPolicy.RELIABLE, + history=HistoryPolicy.KEEP_LAST, depth=200)) + cls._connect() + + @classmethod + def tearDownClass(cls): + cls.node.destroy_node() + rclpy.shutdown() + + @classmethod + def _on_event(cls, msg): + with cls._events_lock: + cls._events.append((msg.event_type, msg.fault.fault_code)) + + @classmethod + def _connect(cls, timeout_sec=40.0): + cls.report_client = cls.node.create_client(ReportFault, '/fault_manager/report_fault') + cls.clear_client = cls.node.create_client(ClearFault, '/fault_manager/clear_fault') + cls.list_client = cls.node.create_client(ListFaults, '/fault_manager/list_faults') + cls.set_stop_client = cls.node.create_client( + SetPlannedStop, '/fault_manager/set_planned_stop') + cls.get_stop_client = cls.node.create_client( + GetPlannedStop, '/fault_manager/get_planned_stop') + for client, name in ( + (cls.report_client, 'report_fault'), + (cls.clear_client, 'clear_fault'), + (cls.list_client, 'list_faults'), + (cls.set_stop_client, 'set_planned_stop'), + (cls.get_stop_client, 'get_planned_stop'), + ): + assert client.wait_for_service(timeout_sec=timeout_sec), \ + f'{name} service not available' + + def _call(self, client, request, timeout_sec=15.0): + future = client.call_async(request) + rclpy.spin_until_future_complete(self.node, future, timeout_sec=timeout_sec) + self.assertIsNotNone(future.result(), 'service call timed out') + return future.result() + + def _report(self, fault_code): + request = ReportFault.Request() + request.fault_code = fault_code + request.event_type = ReportFault.Request.EVENT_FAILED + request.severity = Fault.SEVERITY_ERROR + request.description = 'interrupted release test fault' + request.source_id = '/test_node' + return self._call(self.report_client, request) + + def _clear(self, fault_code): + request = ClearFault.Request() + request.fault_code = fault_code + request.skip_correlation_auto_clear = False + return self._call(self.clear_client, request) + + def _muted_entries(self): + request = ListFaults.Request() + request.filter_by_severity = False + request.severity = 0 + request.statuses = [] + request.include_muted = True + request.include_clusters = False + response = self._call(self.list_client, request) + return {info.fault_code: info for info in response.muted_faults} + + def _set_stop(self, active, *, reason='', declared_by=''): + request = SetPlannedStop.Request() + request.active = active + request.reason = reason + request.declared_by = declared_by + return self._call(self.set_stop_client, request) + + def _muted_codes(self): + request = ListFaults.Request() + request.filter_by_severity = False + request.severity = 0 + request.statuses = [] + request.include_muted = True + request.include_clusters = False + return {info.fault_code for info in self._call(self.list_client, request).muted_faults} + + def _confirmed_codes(self): + request = ListFaults.Request() + request.filter_by_severity = False + request.severity = 0 + request.statuses = [] + request.include_muted = False + request.include_clusters = False + return {fault.fault_code for fault in self._call(self.list_client, request).faults} + + def _count_events(self, fault_code, event_type): + with self._events_lock: + return sum(1 for kind, code in self._events + if code == fault_code and kind == event_type) + + def _wait_until(self, predicate, message, timeout=60.0): + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if predicate(): + return + rclpy.spin_once(self.node, timeout_sec=0.05) + raise AssertionError(message) + + def _spin_for(self, seconds): + deadline = time.monotonic() + seconds + while time.monotonic() < deadline: + rclpy.spin_once(self.node, timeout_sec=0.05) + + @staticmethod + def _process_is_running(pid): + """ + Whether `pid` is still a LIVE process - a zombie counts as gone. + + `launch`'s own `process_details` is no help here: it keeps naming the dead + process until the respawn replaces it, so waiting on it waits for the + REPLACEMENT rather than for the death, and anything done "while the node is + down" would land after the new one had already read the store. + """ + try: + with open(f'/proc/{pid}/stat', 'rb') as handle: + raw = handle.read() + except (FileNotFoundError, ProcessLookupError): + return False + except OSError: + return True + state = raw[raw.rfind(b')') + 2:raw.rfind(b')') + 3] + return state not in (b'Z', b'X') + + @staticmethod + def _withdraw_declaration_in_the_store(): + """ + Leave the store exactly as a crash mid-release would. + + The declaration is withdrawn - that write happens first - while the + ownership flags, which are cleared last, are still set. + """ + connection = sqlite3.connect(DATABASE_PATH) + try: + connection.execute( + 'UPDATE planned_stop SET active = 0, ended_at_ns = ? WHERE id = 1', + (int(time.time() * 1e9),)) + connection.commit() + owned = connection.execute( + 'SELECT COUNT(*) FROM faults WHERE planned_stop_owned != 0').fetchone()[0] + finally: + connection.close() + return owned + + def test_startup_finishes_a_release_the_previous_process_did_not( + self, fault_manager_node, proc_output): + self.assertTrue(self._set_stop(True, reason='cell 4 rebuild', + declared_by='plant_manager').success) + for code in OWNED_CODES: + self._report(code) + self._wait_until(lambda: set(OWNED_CODES) <= self._muted_codes(), + 'the faults were never marked by the stop') + for code in OWNED_CODES: + self.assertEqual(0, self._count_events(code, FaultEvent.EVENT_CONFIRMED)) + + original_pid = fault_manager_node.process_details['pid'] + os.kill(original_pid, signal.SIGTERM) + + deadline = time.monotonic() + 30.0 + while time.monotonic() < deadline and self._process_is_running(original_pid): + time.sleep(0.1) + self.assertFalse(self._process_is_running(original_pid), + 'the fault manager was still running, so the store edit below would ' + 'not be the state a crash leaves') + + still_owned = self._withdraw_declaration_in_the_store() + self.assertEqual(len(OWNED_CODES), still_owned, + 'the store did not hold the ownership flags a crash would leave') + + self._wait_until( + lambda: fault_manager_node.process_details is not None + and fault_manager_node.process_details['pid'] != original_pid, + 'the fault manager was never restarted') + self._connect() + + # A stop declared right after the restart owns its own cycles. Declared + # FIRST, before this test waits for anything: a recovery that runs later + # than the constructor would be inside this window, which is exactly the + # window in which it can announce and unflag a fault the NEW stop holds. + self.assertTrue(self._set_stop(True, reason='second shutdown', + declared_by='night_shift').success) + + # Whatever the recovery did or did not manage to announce, nothing more may + # be announced for this code while the new stop holds its new cycle. + announced_before = self._count_events(REUSED_CODE, FaultEvent.EVENT_CONFIRMED) + + # A NEW cycle of a code the previous release captured: acknowledge it, then + # raise it again under the new declaration. + self.assertTrue(self._clear(REUSED_CODE).success) + self._report(REUSED_CODE) + self._report(NEW_STOP_CODE) + self._wait_until( + lambda: {REUSED_CODE, NEW_STOP_CODE} <= set(self._muted_entries()), + 'the new stop did not take the cycles that started under it') + + # Startup finishes the interrupted release, and says so itself - the one + # instrument a publication made before anyone has matched the publisher + # cannot take away. + expected = ('Finished an interrupted planned-stop release: ' + f'released {len(OWNED_CODES)} fault(s)') + proc_output.assertWaitFor(expected_output=expected, process='fault_manager_node-1', + timeout=40, stream='stderr') + self._spin_for(2.0) + + # What the release captured is released: back in the default list, not + # marked, announced at most once. + for code in OWNED_CODES: + if code == REUSED_CODE: + continue # it has since started a new cycle under the new stop + self.assertIn(code, self._confirmed_codes(), + f'{code} did not come back to the default fault list') + self.assertNotIn(code, self._muted_codes(), + f'{code} is still marked after the release was finished') + self.assertLessEqual(self._count_events(code, FaultEvent.EVENT_CONFIRMED), 1, + f'{code} was announced more than once') + + # And what it did NOT capture is untouched: the new stop still holds both + # of its cycles, flags and all. + entries = self._muted_entries() + for code in (REUSED_CODE, NEW_STOP_CODE): + self.assertIn(code, entries, + f'{code} was released by a recovery that never owned it') + self.assertEqual('planned_stop', entries[code].rule_id) + self.assertNotIn(code, self._confirmed_codes()) + self.assertEqual(2, self._owned_flag_count(), + 'the recovery wiped ownership flags belonging to the new stop') + self.assertEqual(announced_before, + self._count_events(REUSED_CODE, FaultEvent.EVENT_CONFIRMED), + 'the new cycle was announced while the new stop still stands') + self.assertEqual(0, self._count_events(NEW_STOP_CODE, FaultEvent.EVENT_CONFIRMED)) + + # The new stop still releases its own faults normally. + self.assertTrue(self._set_stop(False, reason='back up', declared_by='night_shift').success) + for code in (REUSED_CODE, NEW_STOP_CODE): + self._wait_until(lambda code=code: code in self._confirmed_codes(), + f'{code} was not released by its own switch-off') + self.assertEqual(0, self._owned_flag_count()) + self.assertFalse(self._call(self.get_stop_client, GetPlannedStop.Request()).active) + + @staticmethod + def _owned_flag_count(): + connection = sqlite3.connect(f'file:{DATABASE_PATH}?mode=ro', uri=True) + try: + return connection.execute( + 'SELECT COUNT(*) FROM faults WHERE planned_stop_owned != 0').fetchone()[0] + finally: + connection.close() + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + for info in proc_info: + self.assertIn(info.returncode, (0, -2, -15), + f'{info.process_name} exited with code {info.returncode}') diff --git a/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_restart.test.py b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_restart.test.py new file mode 100644 index 000000000..b056bb046 --- /dev/null +++ b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_restart.test.py @@ -0,0 +1,434 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +A planned stop declared before a restart is still in force after it. + +The subject IS the restart, so the fault manager runs with ``respawn`` and the +test kills it by PID: a weekend stop must not end because the box rebooted. +Runs on SQLite, the shipped default and the only backend with a file behind it. +""" + +import os +import signal +import tempfile +import threading +import time +import unittest + +from launch import LaunchDescription +import launch.actions +import launch_ros.actions +import launch_testing.actions +import rclpy +from rclpy.node import Node +from rclpy.qos import HistoryPolicy, QoSProfile, ReliabilityPolicy +from ros2_medkit_msgs.msg import Fault, FaultEvent +from ros2_medkit_msgs.srv import GetFault, GetPlannedStop, ListFaults, ReportFault, SetPlannedStop + +STORAGE_DIR = tempfile.mkdtemp(prefix='planned_stop_restart_') +DATABASE_PATH = os.path.join(STORAGE_DIR, 'faults.db') +CORRELATION_PATH = os.path.join(STORAGE_DIR, 'correlation.yaml') +THRESHOLDS_PATH = os.path.join(STORAGE_DIR, 'entity_thresholds.yaml') + +# Two rules whose state is NOT persisted anywhere, so a restart is where the +# difference between "the stop owns this cycle" and "a rule is holding it" shows. The +# cluster is the harder of the two: it hides a member with no entry at all, and the +# replacement process has no cluster to hide anything with. +CORRELATION_RULES = """ +correlation: + enabled: true + patterns: + restart_motor_errors: + codes: ["MOTOR_RESTART_*"] + restart_valve_errors: + codes: ["VALVE_RESTART_*"] + rules: + - id: restart_estop_cascade + mode: hierarchical + root_cause: + codes: ["ESTOP_RESTART_001"] + symptoms: + - pattern: restart_motor_errors + window_ms: 60000 + mute_symptoms: true + auto_clear_with_root: false + - id: restart_valve_storm + mode: auto_cluster + match: + - pattern: restart_valve_errors + min_count: 2 + window_ms: 60000 + show_as_single: true + representative: first +""" + +# One source whose faults take three FAILED reports to confirm, so a single report +# leaves a fault the stop owns and that has nothing to announce yet. +ENTITY_THRESHOLDS = """ +/slow_node: + confirmation_threshold: -3 +""" + +RULE_ROOT_CODE = 'ESTOP_RESTART_001' +RULE_SYMPTOM_CODE = 'MOTOR_RESTART_LEFT' + +REASON = 'weekend shutdown, cell 4' +DECLARED_BY = 'plant_manager' +BEFORE_CODE = 'PS_RESTART_BEFORE' +SECOND_BEFORE_CODE = 'PS_RESTART_BEFORE_TWO' +AFTER_CODE = 'PS_RESTART_AFTER' +CLUSTER_REPRESENTATIVE = 'VALVE_RESTART_A' +CLUSTER_MEMBER = 'VALVE_RESTART_B' +PREFAILED_CODE = 'PS_RESTART_PREFAILED' +SLOW_SOURCE = '/slow_node' + + +def get_coverage_env(): + """Get environment variables for gcov coverage data collection.""" + try: + from ament_index_python.packages import get_package_prefix + pkg_prefix = get_package_prefix('ros2_medkit_fault_manager') + workspace = os.path.dirname(os.path.dirname(pkg_prefix)) + build_dir = os.path.join(workspace, 'build', 'ros2_medkit_fault_manager') + + if os.path.exists(build_dir): + return { + 'GCOV_PREFIX': build_dir, + 'GCOV_PREFIX_STRIP': str(build_dir.count(os.sep)), + } + except Exception: + # Coverage environment is optional; on any error, fall back to no extra config + pass + return {} + + +def generate_test_description(): + """Launch a fault manager that launch brings back after it is killed.""" + with open(CORRELATION_PATH, 'w') as handle: + handle.write(CORRELATION_RULES) + with open(THRESHOLDS_PATH, 'w') as handle: + handle.write(ENTITY_THRESHOLDS) + + fault_manager_env = get_coverage_env() + fault_manager_env['ROS_LOCALHOST_ONLY'] = '1' + + fault_manager_node = launch_ros.actions.Node( + package='ros2_medkit_fault_manager', + executable='fault_manager_node', + name='fault_manager', + output='screen', + additional_env=fault_manager_env, + parameters=[{ + 'storage_type': 'sqlite', + 'database_path': DATABASE_PATH, + 'confirmation_threshold': -1, + 'correlation.config_file': CORRELATION_PATH, + 'entity_thresholds.config_file': THRESHOLDS_PATH, + 'snapshots.enabled': False, + 'snapshots.rosbag.enabled': False, + }], + # The restart is the subject. The delay gives the DDS participant time to + # go before its replacement appears, so the test sees a real gap rather + # than two managers on one domain. + respawn=True, + respawn_delay=1.0, + sigterm_timeout='30', + sigkill_timeout='15', + ) + + return ( + LaunchDescription([ + launch.actions.TimerAction(period=2.0, actions=[fault_manager_node]), + launch_testing.actions.ReadyToTest(), + ]), + {'fault_manager_node': fault_manager_node}, + ) + + +class TestPlannedStopSurvivesRestart(unittest.TestCase): + """The declaration outlives the process that made it.""" + + @classmethod + def setUpClass(cls): + os.environ['ROS_LOCALHOST_ONLY'] = '1' + rclpy.init() + cls.node = Node('test_planned_stop_restart_client') + + # The subscription outlives the manager it listens to and rematches the + # replacement's publisher, which is what lets one test span the restart. + cls._events = [] + cls._events_lock = threading.Lock() + cls._event_sub = cls.node.create_subscription( + FaultEvent, '/fault_manager/events', cls._on_event, + QoSProfile(reliability=ReliabilityPolicy.RELIABLE, + history=HistoryPolicy.KEEP_LAST, depth=200)) + cls._connect() + + @classmethod + def tearDownClass(cls): + cls.node.destroy_node() + rclpy.shutdown() + + @classmethod + def _on_event(cls, msg): + with cls._events_lock: + cls._events.append((msg.event_type, msg.fault.fault_code)) + + def _count_events(self, fault_code, event_type): + with self._events_lock: + return sum(1 for kind, code in self._events + if code == fault_code and kind == event_type) + + @classmethod + def _connect(cls, timeout_sec=30.0): + """Create clients and wait for the manager's services to answer.""" + cls.report_client = cls.node.create_client(ReportFault, '/fault_manager/report_fault') + cls.list_client = cls.node.create_client(ListFaults, '/fault_manager/list_faults') + cls.get_client = cls.node.create_client(GetFault, '/fault_manager/get_fault') + cls.set_stop_client = cls.node.create_client( + SetPlannedStop, '/fault_manager/set_planned_stop') + cls.get_stop_client = cls.node.create_client( + GetPlannedStop, '/fault_manager/get_planned_stop') + for client, name in ( + (cls.report_client, 'report_fault'), + (cls.list_client, 'list_faults'), + (cls.get_client, 'get_fault'), + (cls.set_stop_client, 'set_planned_stop'), + (cls.get_stop_client, 'get_planned_stop'), + ): + assert client.wait_for_service(timeout_sec=timeout_sec), \ + f'{name} service not available' + + def _call(self, client, request, timeout_sec=15.0): + future = client.call_async(request) + rclpy.spin_until_future_complete(self.node, future, timeout_sec=timeout_sec) + self.assertIsNotNone(future.result(), 'service call timed out') + return future.result() + + def _spin_for(self, seconds): + """Pump the executor so published events reach the subscription.""" + deadline = time.monotonic() + seconds + while time.monotonic() < deadline: + rclpy.spin_once(self.node, timeout_sec=0.05) + + def _wait_until(self, predicate, message, timeout=30.0): + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if predicate(): + return + rclpy.spin_once(self.node, timeout_sec=0.05) + raise AssertionError(message) + + def _report(self, fault_code, source_id='/test_node'): + request = ReportFault.Request() + request.fault_code = fault_code + request.event_type = ReportFault.Request.EVENT_FAILED + request.severity = Fault.SEVERITY_ERROR + request.description = 'restart test fault' + request.source_id = source_id + return self._call(self.report_client, request) + + def _clusters(self): + request = ListFaults.Request() + request.filter_by_severity = False + request.severity = 0 + request.statuses = [] + request.include_muted = True + request.include_clusters = True + return self._call(self.list_client, request).clusters + + def _status_of(self, fault_code): + request = GetFault.Request() + request.fault_code = fault_code + response = self._call(self.get_client, request) + return response.fault.status if response.success else '' + + def _set_stop(self, active, *, reason='', declared_by=''): + request = SetPlannedStop.Request() + request.active = active + request.reason = reason + request.declared_by = declared_by + return self._call(self.set_stop_client, request) + + def _get_stop(self): + return self._call(self.get_stop_client, GetPlannedStop.Request()) + + def _muted_codes(self): + request = ListFaults.Request() + request.filter_by_severity = False + request.severity = 0 + request.statuses = [] + request.include_muted = True + request.include_clusters = False + response = self._call(self.list_client, request) + return {info.fault_code: info for info in response.muted_faults} + + def _confirmed_codes(self): + request = ListFaults.Request() + request.filter_by_severity = False + request.severity = 0 + request.statuses = [] + request.include_muted = False + request.include_clusters = False + return {fault.fault_code for fault in self._call(self.list_client, request).faults} + + def test_a_stop_declared_before_a_restart_still_mutes_after_it(self, fault_manager_node): + # A rule-muted fault whose cycle starts BEFORE the stop. Its mute lives only + # in the engine, so the restart is where the stop could wrongly adopt it. + self._report(RULE_ROOT_CODE) + self._report(RULE_SYMPTOM_CODE) + self._wait_until(lambda: RULE_SYMPTOM_CODE in self._muted_codes(), + f'{RULE_SYMPTOM_CODE} was never muted by the rule') + self.assertEqual('restart_estop_cascade', self._muted_codes()[RULE_SYMPTOM_CODE].rule_id) + + declared = self._set_stop(True, reason=REASON, declared_by=DECLARED_BY) + self.assertTrue(declared.success) + self.assertFalse(declared.was_active) + + before = self._get_stop() + self.assertTrue(before.active) + self.assertEqual(REASON, before.reason) + + self._report(BEFORE_CODE) + self._report(SECOND_BEFORE_CODE) + + # A cluster formed inside the stop. Its membership lives in the process, so + # this is what a restart cannot bring back. + self._report(CLUSTER_REPRESENTATIVE) + self._report(CLUSTER_MEMBER) + self._wait_until(lambda: len(self._clusters()) == 1, + 'the cluster rule never formed a cluster') + self.assertEqual(CLUSTER_REPRESENTATIVE, self._clusters()[0].representative_code) + + # And a fault the stop owns that has nothing to announce: one FAILED report + # from a source whose threshold needs three. + self._report(PREFAILED_CODE, source_id=SLOW_SOURCE) + self.assertEqual(Fault.STATUS_PREFAILED, self._status_of(PREFAILED_CODE)) + + owned_before_the_restart = (BEFORE_CODE, SECOND_BEFORE_CODE, CLUSTER_REPRESENTATIVE, + CLUSTER_MEMBER, PREFAILED_CODE) + muted_before = self._muted_codes() + for code in owned_before_the_restart: + self.assertIn(code, muted_before) + self.assertEqual(0, self._count_events(code, FaultEvent.EVENT_CONFIRMED), + 'a fault the stop marked was announced anyway') + + original_pid = fault_manager_node.process_details['pid'] + os.kill(original_pid, signal.SIGTERM) + + deadline = time.monotonic() + 60.0 + while time.monotonic() < deadline: + current = fault_manager_node.process_details + if current is not None and current['pid'] != original_pid: + break + time.sleep(0.2) + self.assertNotEqual( + original_pid, fault_manager_node.process_details['pid'], + 'the fault manager was never restarted, so nothing below tests a restart') + + self._connect(timeout_sec=60.0) + + after = self._get_stop() + self.assertTrue(after.active, 'the planned stop did not survive the restart') + self.assertEqual(REASON, after.reason) + self.assertEqual(DECLARED_BY, after.declared_by) + self.assertEqual(before.since.sec, after.since.sec, + 'the restart moved the time the stop started') + self.assertEqual(before.since.nanosec, after.since.nanosec) + + # Exactly what the stop owned comes back - nothing else. Asserted before + # anything new is reported, so a fault raised after the restart cannot + # stand in for one the restore was supposed to find. + restored = self._muted_codes() + self.assertEqual( + set(owned_before_the_restart), set(restored), + 'the restart restored something other than exactly the faults the stop owned') + + # The cluster is gone with the process that formed it. Nothing persists + # membership, and nothing rebuilds it from the store. + self.assertEqual([], list(self._clusters()), + 'a cluster came back from a restart, which nothing persists') + # The rule's mute is not persisted, so it is gone after the restart - and the + # stop must not have adopted a cycle that started before it was declared. + self.assertNotIn(RULE_SYMPTOM_CODE, restored) + self.assertIn(RULE_SYMPTOM_CODE, self._confirmed_codes()) + + # The declaration is not a museum piece: a fault reported after the + # restart is muted by it. + self._report(AFTER_CODE) + muted = self._muted_codes() + self.assertIn(AFTER_CODE, muted, + 'a fault reported after the restart was not muted by the standing stop') + self.assertEqual('planned_stop', muted[AFTER_CODE].rule_id) + self.assertNotIn(AFTER_CODE, self._confirmed_codes()) + + # The mute set lived in the process that is gone, so the manager rebuilds it + # from the store: the faults marked before the restart are still marked. + for code in (BEFORE_CODE, SECOND_BEFORE_CODE): + self.assertIn(code, restored, + f'{code} was silently released by the restart, so its ' + 'confirmation could never be announced') + self.assertEqual('planned_stop', restored[code].rule_id) + self.assertNotIn(code, self._confirmed_codes()) + + withdrawn = self._set_stop(False, reason='plant back up', declared_by=DECLARED_BY) + self.assertTrue(withdrawn.success) + self.assertTrue(withdrawn.was_active) + final_state = self._get_stop() + self.assertFalse(final_state.active) + self.assertEqual(REASON, final_state.reason) + self.assertGreater(final_state.ended_at.sec, 0) + self.assertIn(AFTER_CODE, self._confirmed_codes()) + + # Every CONFIRMED fault the stop was holding - across the restart - is released + # and announced exactly once. The cluster member included: the cluster that was + # folding it into one line did not survive the restart, so after a reboot the + # switch-off announces the whole burst. + announced_codes = (BEFORE_CODE, SECOND_BEFORE_CODE, AFTER_CODE, + CLUSTER_REPRESENTATIVE, CLUSTER_MEMBER) + for code in announced_codes: + self._wait_until( + lambda code=code: self._count_events(code, FaultEvent.EVENT_CONFIRMED) == 1, + f'{code} was never announced at the switch-off') + self.assertIn(code, self._confirmed_codes()) + + # Released is not announced. A fault the stop owned that never reached + # CONFIRMED has nothing to announce, so it leaves the muted list silently and + # goes on debouncing. + self.assertNotIn(PREFAILED_CODE, self._muted_codes()) + self.assertEqual(0, self._count_events(PREFAILED_CODE, FaultEvent.EVENT_CONFIRMED), + 'a fault still short of confirmation was announced at the switch-off') + self.assertEqual(Fault.STATUS_PREFAILED, self._status_of(PREFAILED_CODE)) + + self.assertEqual( + 0, self._count_events(RULE_SYMPTOM_CODE, FaultEvent.EVENT_CONFIRMED), + 'the switch-off announced a fault the stop never owned') + + # A second frame arriving late would still be a double announcement. + self._spin_for(3.0) + for code in announced_codes: + self.assertEqual(1, self._count_events(code, FaultEvent.EVENT_CONFIRMED), + f'{code} was announced more than once') + self.assertEqual(0, self._count_events(PREFAILED_CODE, FaultEvent.EVENT_CONFIRMED)) + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + for info in proc_info: + self.assertIn(info.returncode, (0, -2, -15), + f'{info.process_name} exited with code {info.returncode}') diff --git a/src/ros2_medkit_fault_manager/test/test_correlation_engine.cpp b/src/ros2_medkit_fault_manager/test/test_correlation_engine.cpp index 040c2df91..adb1ef250 100644 --- a/src/ros2_medkit_fault_manager/test/test_correlation_engine.cpp +++ b/src/ros2_medkit_fault_manager/test/test_correlation_engine.cpp @@ -14,6 +14,7 @@ #include +#include #include #include @@ -75,6 +76,103 @@ class CorrelationEngineTest : public ::testing::Test { return parse_config_string(yaml); } + /// The same hierarchical rule, but acknowledging the root cause does NOT clear + /// the symptoms. That is what makes the symptom outlive the rule's mute and + /// fall back to whatever else was holding it. + CorrelationConfig create_hierarchical_no_autoclear_config() { + const std::string yaml = R"( +correlation: + enabled: true + default_window_ms: 500 + patterns: + motor_errors: + codes: ["MOTOR_COMM_*", "MOTOR_TIMEOUT_*"] + rules: + - id: estop_cascade + name: "E-Stop Cascade" + mode: hierarchical + root_cause: + codes: ["ESTOP_001"] + symptoms: + - pattern: motor_errors + window_ms: 1000 + mute_symptoms: true + auto_clear_with_root: false +)"; + return parse_config_string(yaml); + } + + /// A hierarchical rule and an auto-cluster rule side by side. The cluster path + /// sets should_mute without ever writing muted_faults_, which is the case the + /// planned stop has to survive. + CorrelationConfig create_cluster_config() { + const std::string yaml = R"( +correlation: + enabled: true + patterns: + valve_errors: + codes: ["VALVE_*"] + rules: + - id: valve_storm + name: "Valve Storm" + mode: auto_cluster + match: + - pattern: valve_errors + min_count: 2 + window_ms: 60000 + show_as_single: true + representative: first +)"; + return parse_config_string(yaml); + } + + /// The same cluster rule at a chosen min_count, so a burst can shrink back below + /// the threshold without the cluster dissolving. + CorrelationConfig create_cluster_config_with_min_count(int min_count) { + const std::string yaml = R"( +correlation: + enabled: true + patterns: + valve_errors: + codes: ["VALVE_*"] + rules: + - id: valve_storm + name: "Valve Storm" + mode: auto_cluster + match: + - pattern: valve_errors + min_count: )" + std::to_string(min_count) + + R"( + window_ms: 60000 + show_as_single: true + representative: first +)"; + return parse_config_string(yaml); + } + + /// A cluster rule that picks the loudest member as representative, which is the + /// policy that needs each member's severity to promote a replacement. + CorrelationConfig create_cluster_config_highest_severity() { + const std::string yaml = R"( +correlation: + enabled: true + patterns: + valve_errors: + codes: ["VALVE_*"] + rules: + - id: valve_storm + name: "Valve Storm" + mode: auto_cluster + match: + - pattern: valve_errors + min_count: 2 + window_ms: 60000 + show_as_single: true + representative: highest_severity +)"; + return parse_config_string(yaml); + } + CorrelationConfig create_mixed_config() { const std::string yaml = R"( correlation: @@ -762,6 +860,800 @@ TEST_F(CorrelationEngineTest, CleanupExpiredRemovesFaultToClusterEntries) { EXPECT_EQ(3u, clusters[0].fault_codes.size()); } +// ============================================================================ +// Planned stop: a second, operator-driven mute source +// ============================================================================ + +TEST_F(CorrelationEngineTest, PlannedStopMutesAnUncorrelatedFault) { + CorrelationEngine engine(create_hierarchical_config()); + + EXPECT_FALSE(engine.planned_stop_active()); + auto before = engine.process_fault("PUMP_SEAL_LEAK", "ERROR"); + EXPECT_FALSE(before.should_mute); + EXPECT_FALSE(engine.is_muted("PUMP_SEAL_LEAK")); + + engine.begin_planned_stop(); + EXPECT_TRUE(engine.planned_stop_active()); + + auto during = engine.process_fault("VALVE_STUCK", "ERROR"); + EXPECT_TRUE(during.should_mute); + EXPECT_TRUE(engine.is_muted("VALVE_STUCK")); + EXPECT_EQ(1u, engine.get_muted_count()); + + auto muted = engine.get_muted_faults(); + ASSERT_EQ(1u, muted.size()); + EXPECT_EQ("VALVE_STUCK", muted[0].fault_code); + EXPECT_EQ(CorrelationEngine::kPlannedStopRootCause, muted[0].root_cause_code); + EXPECT_EQ(CorrelationEngine::kPlannedStopRuleId, muted[0].rule_id); +} + +TEST_F(CorrelationEngineTest, PlannedStopIsIdempotentAndSurvivesRepeatedReports) { + CorrelationEngine engine(create_hierarchical_config()); + + engine.begin_planned_stop(); + engine.begin_planned_stop(); + EXPECT_TRUE(engine.planned_stop_active()); + + engine.process_fault("VALVE_STUCK", "ERROR"); + engine.process_fault("VALVE_STUCK", "ERROR"); + EXPECT_EQ(1u, engine.get_muted_count()); +} + +TEST_F(CorrelationEngineTest, EndingPlannedStopUnmutesWhatItMuted) { + CorrelationEngine engine(create_hierarchical_config()); + + engine.begin_planned_stop(); + engine.process_fault("VALVE_STUCK", "ERROR"); + engine.process_fault("PUMP_SEAL_LEAK", "WARNING"); + EXPECT_EQ(2u, engine.get_muted_count()); + + auto unmuted = engine.end_planned_stop().to_announce; + EXPECT_FALSE(engine.planned_stop_active()); + std::sort(unmuted.begin(), unmuted.end()); + ASSERT_EQ(2u, unmuted.size()); + EXPECT_EQ("PUMP_SEAL_LEAK", unmuted[0]); + EXPECT_EQ("VALVE_STUCK", unmuted[1]); + EXPECT_EQ(0u, engine.get_muted_count()); + EXPECT_FALSE(engine.is_muted("VALVE_STUCK")); + EXPECT_FALSE(engine.is_muted("PUMP_SEAL_LEAK")); +} + +TEST_F(CorrelationEngineTest, EndingPlannedStopWithNothingMutedReturnsNothing) { + CorrelationEngine engine(create_hierarchical_config()); + + engine.begin_planned_stop(); + auto unmuted = engine.end_planned_stop().to_announce; + EXPECT_TRUE(unmuted.empty()); + + // Ending a stop that is already off changes nothing either. + EXPECT_TRUE(engine.end_planned_stop().to_announce.empty()); + EXPECT_FALSE(engine.planned_stop_active()); +} + +TEST_F(CorrelationEngineTest, AFaultReportedAfterTheStopEndsIsNotMuted) { + CorrelationEngine engine(create_hierarchical_config()); + + engine.begin_planned_stop(); + engine.end_planned_stop().to_announce; + + auto result = engine.process_fault("VALVE_STUCK", "ERROR"); + EXPECT_FALSE(result.should_mute); + EXPECT_FALSE(engine.is_muted("VALVE_STUCK")); +} + +TEST_F(CorrelationEngineTest, EndingPlannedStopLeavesARuleMutedSymptomMuted) { + CorrelationEngine engine(create_hierarchical_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + + // ESTOP_001 is a root cause: muted by the stop, but not a symptom. + engine.process_fault("ESTOP_001", "CRITICAL", t0); + // MOTOR_COMM_FL matches the rule's symptom pattern inside the window, so the + // RULE mutes it - the stop is not the reason it is muted. + auto symptom = engine.process_fault("MOTOR_COMM_FL", "ERROR", t0 + 10ms); + EXPECT_TRUE(symptom.should_mute); + EXPECT_EQ("estop_cascade", symptom.rule_id); + + auto unmuted = engine.end_planned_stop().to_announce; + EXPECT_EQ(1u, unmuted.size()); + EXPECT_EQ("ESTOP_001", unmuted[0]); + + // The rule still holds the symptom down. + EXPECT_TRUE(engine.is_muted("MOTOR_COMM_FL")); + auto muted = engine.get_muted_faults(); + ASSERT_EQ(1u, muted.size()); + EXPECT_EQ("MOTOR_COMM_FL", muted[0].fault_code); + EXPECT_EQ("estop_cascade", muted[0].rule_id); +} + +TEST_F(CorrelationEngineTest, ARuleTakingOverAStopMutedFaultKeepsItMuted) { + CorrelationEngine engine(create_hierarchical_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + + // Muted by the stop first: no root cause is pending yet. + auto first = engine.process_fault("MOTOR_COMM_FL", "ERROR", t0); + EXPECT_TRUE(first.should_mute); + EXPECT_EQ(CorrelationEngine::kPlannedStopRuleId, engine.get_muted_faults()[0].rule_id); + + // The root cause arrives, then the same code is reported again and the rule + // claims it. + engine.process_fault("ESTOP_001", "CRITICAL", t0 + 10ms); + auto second = engine.process_fault("MOTOR_COMM_FL", "ERROR", t0 + 20ms); + EXPECT_TRUE(second.should_mute); + EXPECT_EQ("estop_cascade", second.rule_id); + + auto unmuted = engine.end_planned_stop().to_announce; + EXPECT_EQ(1u, unmuted.size()); + EXPECT_EQ("ESTOP_001", unmuted[0]); + EXPECT_TRUE(engine.is_muted("MOTOR_COMM_FL")); +} + +TEST_F(CorrelationEngineTest, ClearingDuringAStopTakesTheFaultOutOfTheMute) { + CorrelationEngine engine(create_hierarchical_config()); + + engine.begin_planned_stop(); + engine.process_fault("VALVE_STUCK", "ERROR"); + EXPECT_TRUE(engine.is_muted("VALVE_STUCK")); + + engine.process_clear("VALVE_STUCK"); + EXPECT_FALSE(engine.is_muted("VALVE_STUCK")); + EXPECT_EQ(0u, engine.get_muted_count()); + + // Nothing survives to be unmuted, so the switch-off announces nothing. + EXPECT_TRUE(engine.end_planned_stop().to_announce.empty()); +} + +TEST_F(CorrelationEngineTest, ClearingARootCauseDuringAStopDropsItsStopMutedSymptoms) { + CorrelationEngine engine(create_hierarchical_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + engine.process_fault("ESTOP_001", "CRITICAL", t0); + engine.process_fault("MOTOR_COMM_FL", "ERROR", t0 + 10ms); + + auto cleared = engine.process_clear("ESTOP_001"); + ASSERT_EQ(1u, cleared.auto_cleared_codes.size()); + EXPECT_EQ("MOTOR_COMM_FL", cleared.auto_cleared_codes[0]); + EXPECT_FALSE(engine.is_muted("MOTOR_COMM_FL")); + + // Both codes left the mute with the clear; the switch-off has nothing to say. + EXPECT_TRUE(engine.end_planned_stop().to_announce.empty()); +} + +TEST_F(CorrelationEngineTest, PlannedStopMutesWithoutAnyRulesConfigured) { + CorrelationEngine engine{CorrelationConfig{}}; + + engine.begin_planned_stop(); + auto result = engine.process_fault("VALVE_STUCK", "ERROR"); + EXPECT_TRUE(result.should_mute); + EXPECT_TRUE(engine.is_muted("VALVE_STUCK")); + + auto unmuted = engine.end_planned_stop().to_announce; + ASSERT_EQ(1u, unmuted.size()); + EXPECT_EQ("VALVE_STUCK", unmuted[0]); +} + +// ============================================================================ +// Planned stop: only a cycle that STARTS inside it is marked +// ============================================================================ + +TEST_F(CorrelationEngineTest, AStillActiveReReportIsNotTakenByTheStop) { + CorrelationEngine engine(create_hierarchical_config()); + + // The fault's cycle started before the stop was declared. + engine.process_fault("PUMP_SEAL_LEAK", "ERROR", std::chrono::steady_clock::now(), /*cycle_started=*/true); + EXPECT_FALSE(engine.is_muted("PUMP_SEAL_LEAK")); + + engine.begin_planned_stop(); + + // A level-triggered reporter keeps sending FAILED while the condition holds. + auto result = engine.process_fault("PUMP_SEAL_LEAK", "ERROR", std::chrono::steady_clock::now(), + /*cycle_started=*/false); + EXPECT_FALSE(result.should_mute); + EXPECT_FALSE(engine.is_muted("PUMP_SEAL_LEAK")); + EXPECT_TRUE(engine.end_planned_stop().to_announce.empty()) << "the stop released a fault it never muted"; +} + +TEST_F(CorrelationEngineTest, ACycleStartingInsideTheStopIsTakenByIt) { + CorrelationEngine engine(create_hierarchical_config()); + + engine.begin_planned_stop(); + auto result = engine.process_fault("PUMP_SEAL_LEAK", "ERROR", std::chrono::steady_clock::now(), + /*cycle_started=*/true); + EXPECT_TRUE(result.should_mute); + EXPECT_TRUE(engine.is_muted("PUMP_SEAL_LEAK")); + + auto unmuted = engine.end_planned_stop().to_announce; + ASSERT_EQ(1u, unmuted.size()); + EXPECT_EQ("PUMP_SEAL_LEAK", unmuted[0]); +} + +TEST_F(CorrelationEngineTest, AStillActiveReReportDoesNotReleaseAStopMute) { + CorrelationEngine engine(create_hierarchical_config()); + + engine.begin_planned_stop(); + engine.process_fault("VALVE_STUCK", "ERROR", std::chrono::steady_clock::now(), /*cycle_started=*/true); + ASSERT_TRUE(engine.is_muted("VALVE_STUCK")); + + // The same condition is reported again while it holds. Nothing about the mute + // changes: the cycle is the one the stop already took. + auto again = engine.process_fault("VALVE_STUCK", "ERROR", std::chrono::steady_clock::now(), + /*cycle_started=*/false); + EXPECT_TRUE(again.should_mute); + EXPECT_TRUE(engine.is_muted("VALVE_STUCK")); + + auto unmuted = engine.end_planned_stop().to_announce; + ASSERT_EQ(1u, unmuted.size()); + EXPECT_EQ("VALVE_STUCK", unmuted[0]); +} + +// ============================================================================ +// Planned stop: exactly one mute owner per fault +// ============================================================================ + +TEST_F(CorrelationEngineTest, ARuleMuteOverlaysTheStopsWithoutReplacingOwnership) { + CorrelationEngine engine(create_hierarchical_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.process_fault("ESTOP_001", "CRITICAL", t0); + auto symptom = engine.process_fault("MOTOR_COMM_FL", "ERROR", t0 + 10ms); + ASSERT_TRUE(symptom.should_mute); + ASSERT_EQ("estop_cascade", engine.get_muted_faults()[0].rule_id); + + // The correlation window closes, then a stop is declared and the same code + // starts a new cycle. The rule still owns the mute; the stop must not take it. + engine.begin_planned_stop(); + engine.process_fault("MOTOR_COMM_FL", "ERROR", t0 + 5000ms, /*cycle_started=*/true); + + auto muted = engine.get_muted_faults(); + ASSERT_EQ(1u, muted.size()); + EXPECT_EQ("estop_cascade", muted[0].rule_id) << "the stop overwrote a rule's mute"; + + EXPECT_TRUE(engine.end_planned_stop().to_announce.empty()); + EXPECT_TRUE(engine.is_muted("MOTOR_COMM_FL")); +} + +TEST_F(CorrelationEngineTest, TheClusterKeepsHidingTheSymptomWhenTheStopEnds) { + CorrelationEngine engine(create_cluster_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + + // First of the burst: the cluster is still below min_count, so the stop mutes it. + auto first = engine.process_fault("VALVE_A", "ERROR", t0, /*cycle_started=*/true); + EXPECT_TRUE(first.should_mute); + + // Second reaches min_count. The cluster mutes it as a non-representative, but + // the cluster path writes nothing to the mute map, so the stop stays its owner. + auto second = engine.process_fault("VALVE_B", "ERROR", t0 + 10ms, /*cycle_started=*/true); + EXPECT_TRUE(second.should_mute); + EXPECT_FALSE(second.cluster_id.empty()); + + auto muted = engine.get_muted_faults(); + ASSERT_EQ(2u, muted.size()); + for (const auto & entry : muted) { + EXPECT_EQ(CorrelationEngine::kPlannedStopRuleId, entry.rule_id); + } + + // The switch-off releases the representative and no more: releasing the rest + // would announce, in one wave, the burst the cluster exists to fold into a + // single line. + auto unmuted = engine.end_planned_stop().to_announce; + ASSERT_EQ(1u, unmuted.size()) << "the stop announced a fault the cluster is still hiding"; + EXPECT_EQ("VALVE_A", unmuted[0]) << "the representative is what the cluster shows"; + + // Nothing of the stop's is left behind on the member it did not announce, and + // nothing is written in its place: the cluster hides a member by suppressing + // its events on every report, never by an entry. + EXPECT_EQ(0u, engine.get_muted_count()) << "the switch-off left an entry the cluster never writes"; + EXPECT_FALSE(engine.is_muted("VALVE_B")); + + auto repeat = engine.process_fault("VALVE_B", "ERROR", t0 + 20ms, /*cycle_started=*/false); + EXPECT_TRUE(repeat.should_mute) << "the cluster stopped hiding its member once the stop ended"; +} + +TEST_F(CorrelationEngineTest, EveryNonRepresentativeOfALiveClusterIsHeldAtTheSwitchOff) { + CorrelationEngine engine(create_cluster_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + for (const auto & code : {"VALVE_A", "VALVE_B", "VALVE_C", "VALVE_D"}) { + engine.process_fault(code, "ERROR", t0, /*cycle_started=*/true); + } + + auto unmuted = engine.end_planned_stop().to_announce; + ASSERT_EQ(1u, unmuted.size()) << "a burst of four announced more than the one line the cluster shows"; + EXPECT_EQ("VALVE_A", unmuted[0]); + EXPECT_EQ(0u, engine.get_muted_count()); + + for (const auto & code : {"VALVE_B", "VALVE_C", "VALVE_D"}) { + EXPECT_TRUE(engine.process_fault(code, "ERROR", t0 + 20ms, /*cycle_started=*/false).should_mute) + << code << " left the cluster when the stop ended"; + } +} + +// The point of hiding rather than muting: what the cluster does after a stop has to be +// what it does when there was never one, or muted_count and the default fault list depend +// on plant history rather than on the faults. +TEST_F(CorrelationEngineTest, AClusterBehavesTheSameAfterAStopAsItDoesWithoutOne) { + auto run = [this](bool with_stop) { + CorrelationEngine engine(create_cluster_config()); + auto t0 = std::chrono::steady_clock::now(); + if (with_stop) { + engine.begin_planned_stop(); + } + engine.process_fault("VALVE_A", "ERROR", t0, /*cycle_started=*/true); + engine.process_fault("VALVE_B", "ERROR", t0 + 10ms, /*cycle_started=*/true); + if (with_stop) { + engine.end_planned_stop().to_announce; + } + + std::vector observed; + observed.push_back(static_cast(engine.get_muted_count())); + observed.push_back(engine.process_fault("VALVE_B", "ERROR", t0 + 20ms, /*cycle_started=*/false).should_mute); + // Acknowledging the representative promotes the member it was hiding. + engine.process_clear("VALVE_A"); + observed.push_back(engine.process_fault("VALVE_B", "ERROR", t0 + 30ms, /*cycle_started=*/false).should_mute); + observed.push_back(static_cast(engine.get_muted_count())); + return observed; + }; + + const auto without_stop = run(false); + const auto after_stop = run(true); + EXPECT_EQ(without_stop, after_stop) + << "a cluster that lived through a stop answers differently from one that did not"; + ASSERT_EQ(4u, without_stop.size()); + EXPECT_EQ(0, without_stop[0]); + EXPECT_EQ(1, without_stop[1]) << "a non-representative of a live cluster is hidden"; + EXPECT_EQ(0, without_stop[2]) << "the promoted representative is the line the cluster shows"; + EXPECT_EQ(0, without_stop[3]); +} + +TEST_F(CorrelationEngineTest, AClusterShortOfMinCountHidesNothingAtTheSwitchOff) { + CorrelationEngine engine(create_cluster_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + // min_count is 2, so one member is not a cluster and hides nobody. + engine.process_fault("VALVE_A", "ERROR", t0, /*cycle_started=*/true); + + auto unmuted = engine.end_planned_stop().to_announce; + ASSERT_EQ(1u, unmuted.size()) << "a cluster that never formed withheld a confirmation"; + EXPECT_EQ("VALVE_A", unmuted[0]); + EXPECT_EQ(0u, engine.get_muted_count()); +} + +TEST_F(CorrelationEngineTest, AClusterWithoutShowAsSingleReleasesEveryMember) { + const std::string yaml = R"( +correlation: + enabled: true + patterns: + valve_errors: + codes: ["VALVE_*"] + rules: + - id: valve_storm + name: "Valve Storm" + mode: auto_cluster + match: + - pattern: valve_errors + min_count: 2 + window_ms: 60000 + show_as_single: false + representative: first +)"; + CorrelationEngine engine(parse_config_string(yaml)); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + engine.process_fault("VALVE_A", "ERROR", t0, /*cycle_started=*/true); + engine.process_fault("VALVE_B", "ERROR", t0 + 10ms, /*cycle_started=*/true); + + // A cluster that groups without hiding has no verdict to outrank the stop with. + auto unmuted = engine.end_planned_stop().to_announce; + std::sort(unmuted.begin(), unmuted.end()); + ASSERT_EQ(2u, unmuted.size()) << "a cluster that hides nobody withheld a confirmation"; + EXPECT_EQ("VALVE_A", unmuted[0]); + EXPECT_EQ("VALVE_B", unmuted[1]); + EXPECT_EQ(0u, engine.get_muted_count()); +} + +// min_count decides whether a cluster FORMS. Once formed, the cluster folds its members +// into the representative until the last of them is acknowledged and it dissolves, so a +// burst that shrinks below the threshold does not start announcing its members again. +TEST_F(CorrelationEngineTest, AnActiveClusterKeepsHidingAfterItShrinksBelowMinCount) { + CorrelationEngine engine(create_cluster_config_with_min_count(3)); + + auto t0 = std::chrono::steady_clock::now(); + engine.process_fault("VALVE_A", "ERROR", t0, /*cycle_started=*/true); + engine.process_fault("VALVE_B", "ERROR", t0 + 10ms, /*cycle_started=*/true); + auto third = engine.process_fault("VALVE_C", "ERROR", t0 + 20ms, /*cycle_started=*/true); + ASSERT_TRUE(third.should_mute) << "the cluster did not form at min_count"; + + engine.process_clear("VALVE_C"); + + auto repeat = engine.process_fault("VALVE_B", "ERROR", t0 + 30ms, /*cycle_started=*/false); + EXPECT_TRUE(repeat.should_mute) << "a cluster stopped hiding its member because a sibling was acknowledged"; + + // It dissolves with its last member, not with the threshold. + engine.process_clear("VALVE_B"); + auto alone = engine.process_fault("VALVE_A", "ERROR", t0 + 40ms, /*cycle_started=*/false); + EXPECT_FALSE(alone.should_mute) << "the representative was hidden by its own cluster"; +} + +TEST_F(CorrelationEngineTest, AShrunkenActiveClusterStillHoldsItsMemberAtTheSwitchOff) { + CorrelationEngine engine(create_cluster_config_with_min_count(3)); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + engine.process_fault("VALVE_A", "ERROR", t0, /*cycle_started=*/true); + engine.process_fault("VALVE_B", "ERROR", t0 + 10ms, /*cycle_started=*/true); + engine.process_fault("VALVE_C", "ERROR", t0 + 20ms, /*cycle_started=*/true); + + engine.process_clear("VALVE_C"); + + auto unmuted = engine.end_planned_stop().to_announce; + ASSERT_EQ(1u, unmuted.size()) << "a cluster below min_count announced its whole burst"; + EXPECT_EQ("VALVE_A", unmuted[0]); + EXPECT_TRUE(engine.process_fault("VALVE_B", "ERROR", t0 + 30ms, /*cycle_started=*/false).should_mute); +} + +// cleanup_expired drops the pending twin and keeps the active cluster, so after the window +// closes the active cluster is the only record of who is left. Promoted from the twin that +// is no longer there, the representative keeps naming the acknowledged fault and every +// remaining member is hidden by a cluster whose representative can never be reported again. +TEST_F(CorrelationEngineTest, AcknowledgingTheRepresentativeAfterItsWindowClosedPromotesAMember) { + CorrelationEngine engine(create_cluster_config()); + + // Reported as of two minutes ago, so the rule's 60 s window is already behind them. + auto t0 = std::chrono::steady_clock::now() - 120000ms; + engine.begin_planned_stop(); + engine.process_fault("VALVE_A", "ERROR", t0, /*cycle_started=*/true); + engine.process_fault("VALVE_B", "ERROR", t0 + 10ms, /*cycle_started=*/true); + + engine.cleanup_expired(); + engine.process_clear("VALVE_A"); + + auto unmuted = engine.end_planned_stop().to_announce; + ASSERT_EQ(1u, unmuted.size()) << "the cluster kept naming the acknowledged fault, so nobody was released"; + EXPECT_EQ("VALVE_B", unmuted[0]) << "the promoted representative is the one the switch-off announces"; + EXPECT_EQ(0u, engine.get_muted_count()); +} + +TEST_F(CorrelationEngineTest, PromotionAfterTheWindowClosedFollowsTheRulesPolicy) { + CorrelationEngine engine(create_cluster_config_highest_severity()); + + auto t0 = std::chrono::steady_clock::now() - 120000ms; + engine.process_fault("VALVE_A", "ERROR", t0, /*cycle_started=*/true); + engine.process_fault("VALVE_B", "WARN", t0 + 10ms, /*cycle_started=*/true); + engine.process_fault("VALVE_C", "CRITICAL", t0 + 20ms, /*cycle_started=*/true); + + // CRITICAL is the representative; with the pending twin gone the promotion has to + // read the severities the active cluster carries. + engine.cleanup_expired(); + engine.process_clear("VALVE_C"); + + // Read through get_clusters() rather than by re-reporting: a report after the window + // has closed starts a fresh cluster and would answer about that one instead. + auto clusters = engine.get_clusters(); + ASSERT_EQ(1u, clusters.size()); + EXPECT_EQ("VALVE_A", clusters[0].representative_code) << "the highest-severity survivor was not promoted"; + EXPECT_EQ("ERROR", clusters[0].representative_severity); +} + +// ============================================================================ +// Planned stop: dropping and restoring one fault's mute +// ============================================================================ + +TEST_F(CorrelationEngineTest, ReleasingOneFaultTakesOnlyTheStopsOwnMute) { + CorrelationEngine engine(create_hierarchical_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + engine.process_fault("VALVE_STUCK", "ERROR", t0, /*cycle_started=*/true); + engine.process_fault("ESTOP_001", "CRITICAL", t0 + 10ms, /*cycle_started=*/true); + engine.process_fault("MOTOR_COMM_FL", "ERROR", t0 + 20ms, /*cycle_started=*/true); + // get_muted_faults() is ordered by fault code, so look the entry up rather than + // taking a position in the list. + auto rule_id_of = [&engine](const std::string & code) { + for (const auto & entry : engine.get_muted_faults()) { + if (entry.fault_code == code) { + return entry.rule_id; + } + } + return std::string{""}; + }; + ASSERT_EQ("estop_cascade", rule_id_of("MOTOR_COMM_FL")); + + engine.release_planned_stop_ownership("VALVE_STUCK"); + EXPECT_FALSE(engine.is_muted("VALVE_STUCK")); + + // A rule's mute is not the stop's to drop. + engine.release_planned_stop_ownership("MOTOR_COMM_FL"); + EXPECT_TRUE(engine.is_muted("MOTOR_COMM_FL")); + + auto unmuted = engine.end_planned_stop().to_announce; + ASSERT_EQ(1u, unmuted.size()); + EXPECT_EQ("ESTOP_001", unmuted[0]); +} + +TEST_F(CorrelationEngineTest, RestoredOwnershipBehavesLikeOwnershipTakenLive) { + CorrelationEngine engine{CorrelationConfig{}}; + + // What a restart does: the declaration is read back, then the faults whose + // cycles started inside it are marked again before anything is served. + engine.begin_planned_stop(); + engine.restore_planned_stop_ownership("VALVE_STUCK"); + engine.restore_planned_stop_ownership("PUMP_SEAL_LEAK"); + + EXPECT_TRUE(engine.is_muted("VALVE_STUCK")); + EXPECT_EQ(2u, engine.get_muted_count()); + const auto muted = engine.get_muted_faults(); + EXPECT_EQ(CorrelationEngine::kPlannedStopRootCause, muted[0].root_cause_code); + + auto unmuted = engine.end_planned_stop().to_announce; + std::sort(unmuted.begin(), unmuted.end()); + ASSERT_EQ(2u, unmuted.size()); + EXPECT_EQ("PUMP_SEAL_LEAK", unmuted[0]); + EXPECT_EQ("VALVE_STUCK", unmuted[1]); +} + +// ============================================================================ +// Ownership is the truth; the mute map is derived from it +// ============================================================================ + +TEST_F(CorrelationEngineTest, AnOwnedFaultReturnsToTheStopsMuteWhenARuleReleasesIt) { + // auto_clear_with_root is off here on purpose: the symptom stays up when the + // root cause is acknowledged, so the rule's mute ends while the fault does not. + CorrelationEngine engine(create_hierarchical_no_autoclear_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + + // Both cycles start inside the stop, so the stop owns both. The rule then + // overlays the symptom's mute. + engine.process_fault("ESTOP_001", "CRITICAL", t0, /*cycle_started=*/true); + auto symptom = engine.process_fault("MOTOR_COMM_FL", "ERROR", t0 + 10ms, /*cycle_started=*/true); + EXPECT_TRUE(symptom.should_mute); + auto rule_id_of = [&engine](const std::string & code) { + for (const auto & entry : engine.get_muted_faults()) { + if (entry.fault_code == code) { + return entry.rule_id; + } + } + return std::string{""}; + }; + EXPECT_EQ("estop_cascade", rule_id_of("MOTOR_COMM_FL")); + + // Acknowledging the root cause takes the rule's mute away. The stop still owns + // the symptom's cycle, so it goes back to being the stop's. + engine.process_clear("ESTOP_001"); + + EXPECT_TRUE(engine.is_muted("MOTOR_COMM_FL")) << "the symptom fell out of the stop when the rule let go"; + EXPECT_EQ(CorrelationEngine::kPlannedStopRuleId, rule_id_of("MOTOR_COMM_FL")); + + auto unmuted = engine.end_planned_stop().to_announce; + ASSERT_EQ(1u, unmuted.size()); + EXPECT_EQ("MOTOR_COMM_FL", unmuted[0]); +} + +TEST_F(CorrelationEngineTest, ARuleThatStopsMatchingLeavesTheFaultMuted) { + CorrelationEngine engine(create_hierarchical_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + engine.process_fault("ESTOP_001", "CRITICAL", t0, /*cycle_started=*/true); + engine.process_fault("MOTOR_COMM_FL", "ERROR", t0 + 10ms, /*cycle_started=*/true); + + // The correlation window closes and the pending root cause expires. A repeat + // report of the still-active symptom matches no rule any more - and must still + // report itself as muted, or the manager announces an update for a fault the + // list is hiding. + engine.cleanup_expired(); + auto repeat = engine.process_fault("MOTOR_COMM_FL", "ERROR", t0 + 5000ms, /*cycle_started=*/false); + EXPECT_TRUE(repeat.should_mute) << "a repeat report of a muted fault reported itself unmuted"; + EXPECT_TRUE(engine.is_muted("MOTOR_COMM_FL")); +} + +// The stop keeps its own entry on a member whose cluster window has lapsed, so the +// member is still reported as muted while the stop stands. What it is no longer is +// HIDDEN BY THE CLUSTER: the lapsed report started a burst of its own, so the +// switch-off announces both faults, not just the representative. +TEST_F(CorrelationEngineTest, AClusterMemberWhoseWindowLapsedIsAnnouncedAtTheSwitchOff) { + CorrelationEngine engine(create_cluster_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + engine.process_fault("VALVE_A", "ERROR", t0, /*cycle_started=*/true); + engine.process_fault("VALVE_B", "ERROR", t0 + 10ms, /*cycle_started=*/true); + ASSERT_TRUE(engine.is_muted("VALVE_B")); + + // The periodic sweep drops the lapsed pending cluster; the stop still owns both + // cycles and is still what mutes them. + engine.cleanup_expired(); + auto repeat = engine.process_fault("VALVE_B", "ERROR", t0 + 120000ms, /*cycle_started=*/false); + EXPECT_TRUE(repeat.should_mute) << "the stop's own entry stopped muting the fault"; + EXPECT_TRUE(engine.is_muted("VALVE_B")); + + auto announced = engine.end_planned_stop().to_announce; + std::sort(announced.begin(), announced.end()); + ASSERT_EQ(2u, announced.size()) << "the lapsed member was withheld by a burst it is no longer part of"; + EXPECT_EQ("VALVE_A", announced[0]); + EXPECT_EQ("VALVE_B", announced[1]); +} + +// A cluster hides the reports that fall inside its window. A report after the window +// is a new burst - try_auto_cluster drops the lapsed pending cluster and starts one +// with this fault as its own representative - so the fault is no longer folded into +// the old line, and the switch-off announces it like any owned fault. That is the +// cluster's own contract rather than anything the stop does: with no stop in force +// the same late report publishes an update. +TEST_F(CorrelationEngineTest, AReportAfterTheClusterWindowStartsANewBurstAndIsAnnounced) { + auto announced_when_the_member_repeats_at = [this](std::chrono::milliseconds offset) { + CorrelationEngine engine(create_cluster_config()); // window_ms 60000, min_count 2 + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + engine.process_fault("VALVE_A", "ERROR", t0, /*cycle_started=*/true); + engine.process_fault("VALVE_B", "ERROR", t0 + 10ms, /*cycle_started=*/true); + engine.process_fault("VALVE_B", "ERROR", t0 + offset, /*cycle_started=*/false); + auto announced = engine.end_planned_stop().to_announce; + std::sort(announced.begin(), announced.end()); + return announced; + }; + + // Inside the window the cluster is still folding B into A's line. + const std::vector representative_only{"VALVE_A"}; + EXPECT_EQ(representative_only, announced_when_the_member_repeats_at(20ms)); + + // Past it, B's report is a burst of its own and B is announced too. + const std::vector both{"VALVE_A", "VALVE_B"}; + EXPECT_EQ(both, announced_when_the_member_repeats_at(120000ms)); +} + +TEST_F(CorrelationEngineTest, OwnershipSurvivesARuleOverlayAndIsReadableBack) { + CorrelationEngine engine(create_hierarchical_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + engine.process_fault("ESTOP_001", "CRITICAL", t0, /*cycle_started=*/true); + engine.process_fault("MOTOR_COMM_FL", "ERROR", t0 + 10ms, /*cycle_started=*/true); + + auto owned = engine.planned_stop_owned_codes(); + std::sort(owned.begin(), owned.end()); + ASSERT_EQ(2u, owned.size()) << "a rule overlay took the stop's ownership away"; + EXPECT_EQ("ESTOP_001", owned[0]); + EXPECT_EQ("MOTOR_COMM_FL", owned[1]); +} + +TEST_F(CorrelationEngineTest, AFaultWhoseCycleStartedBeforeTheStopIsNeverOwned) { + CorrelationEngine engine(create_hierarchical_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.process_fault("ESTOP_001", "CRITICAL", t0); + engine.process_fault("MOTOR_COMM_FL", "ERROR", t0 + 10ms); + + engine.begin_planned_stop(); + engine.process_fault("MOTOR_COMM_FL", "ERROR", t0 + 20ms, /*cycle_started=*/false); + + EXPECT_TRUE(engine.planned_stop_owned_codes().empty()); + EXPECT_TRUE(engine.end_planned_stop().to_announce.empty()); + EXPECT_TRUE(engine.is_muted("MOTOR_COMM_FL")) << "the rule's mute was collateral damage"; +} + +TEST_F(CorrelationEngineTest, ClearingAnOwnedFaultEndsTheStopsOwnershipOfIt) { + CorrelationEngine engine(create_hierarchical_config()); + + engine.begin_planned_stop(); + engine.process_fault("VALVE_STUCK", "ERROR", std::chrono::steady_clock::now(), /*cycle_started=*/true); + ASSERT_EQ(1u, engine.planned_stop_owned_codes().size()); + + engine.process_clear("VALVE_STUCK"); + EXPECT_TRUE(engine.planned_stop_owned_codes().empty()); + EXPECT_FALSE(engine.is_muted("VALVE_STUCK")); + EXPECT_TRUE(engine.end_planned_stop().to_announce.empty()); +} + +TEST_F(CorrelationEngineTest, AutoClearedSymptomsLeaveTheStopsOwnership) { + CorrelationEngine engine(create_hierarchical_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + engine.process_fault("ESTOP_001", "CRITICAL", t0, /*cycle_started=*/true); + engine.process_fault("MOTOR_COMM_FL", "ERROR", t0 + 10ms, /*cycle_started=*/true); + + auto cleared = engine.process_clear("ESTOP_001"); + ASSERT_EQ(1u, cleared.auto_cleared_codes.size()); + EXPECT_EQ("MOTOR_COMM_FL", cleared.auto_cleared_codes[0]); + + // Both cycles ended with the acknowledgement, so neither is the stop's any more. + EXPECT_TRUE(engine.planned_stop_owned_codes().empty()); + EXPECT_FALSE(engine.is_muted("MOTOR_COMM_FL")); + EXPECT_TRUE(engine.end_planned_stop().to_announce.empty()); +} + +// `fault_to_cluster_` is written on every join, including a join to a PENDING cluster +// that is below min_count, while `active_clusters_` is refreshed only when the burst +// reaches the threshold. A fault that joined a pending twin two members short therefore +// maps to a formed cluster it is not part of. Folding it into that cluster's line +// announces it once - the join itself is below threshold and not muted - and then +// silences every repeat, so an alarm that is still up stops updating and appears in +// neither the muted list nor the cluster listing. +TEST_F(CorrelationEngineTest, AFaultJoiningAClusterBelowMinCountIsNotFoldedIntoIt) { + CorrelationEngine engine(create_cluster_config_with_min_count(4)); + + auto t0 = std::chrono::steady_clock::now(); + for (const auto & code : {"VALVE_A", "VALVE_B", "VALVE_C", "VALVE_D"}) { + engine.process_fault(code, "ERROR", t0, /*cycle_started=*/true); + } + engine.process_clear("VALVE_C"); + engine.process_clear("VALVE_D"); + + // Two members short of forming again, so this one joins nothing that hides. + auto joined = engine.process_fault("VALVE_E", "ERROR", t0 + 10ms, /*cycle_started=*/true); + EXPECT_FALSE(joined.should_mute); + + auto repeat = engine.process_fault("VALVE_E", "ERROR", t0 + 20ms, /*cycle_started=*/false); + EXPECT_FALSE(repeat.should_mute) << "a fault the cluster never formed with was folded into it and went silent"; + EXPECT_FALSE(engine.is_muted("VALVE_E")); + + // The members it did form with are still hidden. + EXPECT_TRUE(engine.process_fault("VALVE_B", "ERROR", t0 + 30ms, /*cycle_started=*/false).should_mute); +} + +TEST_F(CorrelationEngineTest, AFaultJoiningAClusterBelowMinCountIsReleasedAtTheSwitchOff) { + CorrelationEngine engine(create_cluster_config_with_min_count(4)); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + for (const auto & code : {"VALVE_A", "VALVE_B", "VALVE_C", "VALVE_D"}) { + engine.process_fault(code, "ERROR", t0, /*cycle_started=*/true); + } + engine.process_clear("VALVE_C"); + engine.process_clear("VALVE_D"); + engine.process_fault("VALVE_E", "ERROR", t0 + 10ms, /*cycle_started=*/true); + + auto unmuted = engine.end_planned_stop().to_announce; + std::sort(unmuted.begin(), unmuted.end()); + ASSERT_EQ(2u, unmuted.size()) << "the fault that joined below min_count was withheld by a cluster it is not in"; + EXPECT_EQ("VALVE_A", unmuted[0]) << "the representative"; + EXPECT_EQ("VALVE_E", unmuted[1]) << "a fault of its own, announced like any other"; +} + +// The cluster hold lives in the process. A replacement manager restores ownership from +// the store and has no clusters, so the switch-off releases and announces every owned +// fault - including the members the cluster was folding into one line before the +// restart. Persisting cluster state would change that and is not part of this. +TEST_F(CorrelationEngineTest, ARestartReleasesEveryOwnedFaultIncludingOnesAClusterWasHiding) { + auto t0 = std::chrono::steady_clock::now(); + + CorrelationEngine before_restart(create_cluster_config_with_min_count(2)); + before_restart.begin_planned_stop(); + before_restart.process_fault("VALVE_A", "ERROR", t0, /*cycle_started=*/true); + before_restart.process_fault("VALVE_B", "ERROR", t0 + 10ms, /*cycle_started=*/true); + ASSERT_TRUE(before_restart.process_fault("VALVE_B", "ERROR", t0 + 20ms, /*cycle_started=*/false).should_mute) + << "the cluster was not hiding the member before the restart"; + + // What the store holds, and all the replacement process has to go on. + auto owned = before_restart.planned_stop_owned_codes(); + std::sort(owned.begin(), owned.end()); + ASSERT_EQ(2u, owned.size()); + + CorrelationEngine after_restart(create_cluster_config_with_min_count(2)); + after_restart.begin_planned_stop(); + for (const auto & code : owned) { + after_restart.restore_planned_stop_ownership(code); + } + + auto unmuted = after_restart.end_planned_stop().to_announce; + std::sort(unmuted.begin(), unmuted.end()); + ASSERT_EQ(2u, unmuted.size()) << "a restart is expected to release the whole burst; the cluster is gone with the " + "process that formed it"; + EXPECT_EQ("VALVE_A", unmuted[0]); + EXPECT_EQ("VALVE_B", unmuted[1]); +} + int main(int argc, char ** argv) { ::testing::InitGoogleTest(&argc, argv); return RUN_ALL_TESTS(); diff --git a/src/ros2_medkit_fault_manager/test/test_fault_audit_log.cpp b/src/ros2_medkit_fault_manager/test/test_fault_audit_log.cpp index 2107aab71..bf0f31fc5 100644 --- a/src/ros2_medkit_fault_manager/test/test_fault_audit_log.cpp +++ b/src/ros2_medkit_fault_manager/test/test_fault_audit_log.cpp @@ -527,6 +527,48 @@ TEST_F(FaultAuditLogTest, AnchorTriggersRejectOutOfBandWrites) { EXPECT_TRUE(reopened.verify().ok); } +// A planned-stop transition belongs to the installation, not to a fault: it +// carries no fault code, and the chain has to survive that. +TEST_F(FaultAuditLogTest, PlannedStopTransitionsChainWithNoFaultCode) { + FaultAuditLog log(path_); + + AuditEvent started; + started.fault_code = ""; + started.transition = ros2_medkit_fault_manager::kTransitionPlannedStopStarted; + started.status = "ACTIVE"; + started.source_id = "shift_lead"; + started.description = "line 3 quarterly maintenance"; + started.occurred_at_ns = 1000; + EXPECT_EQ(log.append(started), 1); + + AuditEvent ended; + ended.fault_code = ""; + ended.transition = ros2_medkit_fault_manager::kTransitionPlannedStopEnded; + ended.status = "INACTIVE"; + ended.source_id = "shift_lead"; + ended.description = "line 3 back up"; + ended.occurred_at_ns = 2000; + EXPECT_EQ(log.append(ended), 2); + + EXPECT_TRUE(log.verify().ok); + + const auto records = log.read(); + ASSERT_EQ(records.size(), 2u); + EXPECT_EQ(records[0].event.transition, "planned_stop_started"); + EXPECT_EQ(records[0].event.fault_code, ""); + EXPECT_EQ(records[0].event.source_id, "shift_lead"); + EXPECT_EQ(records[0].event.description, "line 3 quarterly maintenance"); + EXPECT_EQ(records[1].event.transition, "planned_stop_ended"); + EXPECT_EQ(records[1].event.status, "INACTIVE"); + EXPECT_EQ(records[1].prev_hash, records[0].record_hash); + + // The declaration is part of the hashed record, not a side note: rewriting the + // reason without recomputing the chain has to be detectable. + raw_exec(path_, std::string(kDropTriggers) + "UPDATE audit_log SET description = 'unplanned outage' WHERE seq = 1;"); + FaultAuditLog reopened(path_); + EXPECT_FALSE(reopened.verify().ok); +} + int main(int argc, char ** argv) { ::testing::InitGoogleTest(&argc, argv); return RUN_ALL_TESTS(); diff --git a/src/ros2_medkit_fault_manager/test/test_fault_manager.cpp b/src/ros2_medkit_fault_manager/test/test_fault_manager.cpp index 92b0abaaf..8fab2c54d 100644 --- a/src/ros2_medkit_fault_manager/test/test_fault_manager.cpp +++ b/src/ros2_medkit_fault_manager/test/test_fault_manager.cpp @@ -15,6 +15,7 @@ #include #include +#include #include #include #include @@ -38,9 +39,11 @@ #include "ros2_medkit_msgs/msg/snapshot.hpp" #include "ros2_medkit_msgs/srv/clear_fault.hpp" #include "ros2_medkit_msgs/srv/get_fault.hpp" +#include "ros2_medkit_msgs/srv/get_planned_stop.hpp" #include "ros2_medkit_msgs/srv/get_snapshots.hpp" #include "ros2_medkit_msgs/srv/list_faults_for_entity.hpp" #include "ros2_medkit_msgs/srv/report_fault.hpp" +#include "ros2_medkit_msgs/srv/set_planned_stop.hpp" using ros2_medkit_fault_manager::clamp_debounce_counter; using ros2_medkit_fault_manager::compute_debounce_status; @@ -1060,7 +1063,7 @@ class FaultEventPublishingTest : public ::testing::Test { std::string events_topic = ns + "/fault_manager/events"; auto qos = rclcpp::QoS(100).reliable().durability_volatile(); event_subscription_ = - test_node_->create_subscription(events_topic, qos, [this](const FaultEvent::SharedPtr msg) { + test_node_->create_subscription(events_topic, qos, [this](const FaultEvent::ConstSharedPtr & msg) { received_events_.push_back(*msg); }); @@ -1206,7 +1209,7 @@ TEST_F(FaultEventPublishingTest, NewFaultPublishesConfirmedEvent) { // Wait for event to arrive (polling, robust under CPU contention) ASSERT_TRUE(spin_until([this]() { - return received_events_.size() >= 1; + return !received_events_.empty(); })); // Verify EVENT_CONFIRMED was published @@ -1221,7 +1224,7 @@ TEST_F(FaultEventPublishingTest, UpdateExistingFaultPublishesUpdatedEvent) { // Report a new fault first ASSERT_TRUE(call_report_fault("TEST_FAULT_2", Fault::SEVERITY_WARN, "/test_node1")); ASSERT_TRUE(spin_until([this]() { - return received_events_.size() >= 1; + return !received_events_.empty(); })); // Clear received events @@ -1230,7 +1233,7 @@ TEST_F(FaultEventPublishingTest, UpdateExistingFaultPublishesUpdatedEvent) { // Report same fault again - should trigger EVENT_UPDATED ASSERT_TRUE(call_report_fault("TEST_FAULT_2", Fault::SEVERITY_ERROR, "/test_node2")); ASSERT_TRUE(spin_until([this]() { - return received_events_.size() >= 1; + return !received_events_.empty(); })); // Verify EVENT_UPDATED was published (severity/sources changed; still one occurrence) @@ -1244,7 +1247,7 @@ TEST_F(FaultEventPublishingTest, ClearFaultPublishesClearedEvent) { // Report a fault first ASSERT_TRUE(call_report_fault("TEST_FAULT_3", Fault::SEVERITY_ERROR, "/test_node")); ASSERT_TRUE(spin_until([this]() { - return received_events_.size() >= 1; + return !received_events_.empty(); })); // Clear received events @@ -1253,7 +1256,7 @@ TEST_F(FaultEventPublishingTest, ClearFaultPublishesClearedEvent) { // Clear the fault ASSERT_TRUE(call_clear_fault("TEST_FAULT_3")); ASSERT_TRUE(spin_until([this]() { - return received_events_.size() >= 1; + return !received_events_.empty(); })); // Verify EVENT_CLEARED was published @@ -1280,7 +1283,7 @@ class HealingFaultEventPublishingTest : public FaultEventPublishingTest { TEST_F(HealingFaultEventPublishingTest, HealPublishesClearedEventSoStreamConsumersSeeTheEnd) { ASSERT_TRUE(call_report_fault("HEAL_ME", Fault::SEVERITY_ERROR, "/test_node")); ASSERT_TRUE(spin_until([this]() { - return received_events_.size() >= 1; + return !received_events_.empty(); })); received_events_.clear(); @@ -1323,7 +1326,7 @@ TEST_F(FaultEventPublishingTest, EventContainsCorrectTimestamp) { ASSERT_TRUE(call_report_fault("TEST_FAULT_4", Fault::SEVERITY_WARN, "/test_node")); ASSERT_TRUE(spin_until([this]() { - return received_events_.size() >= 1; + return !received_events_.empty(); })); auto after = fault_manager_->now(); @@ -1339,7 +1342,7 @@ TEST_F(FaultEventPublishingTest, EventContainsCorrectTimestamp) { TEST_F(FaultEventPublishingTest, EventContainsFullFaultData) { ASSERT_TRUE(call_report_fault("FULL_DATA_TEST", Fault::SEVERITY_CRITICAL, "/sensor/temperature")); ASSERT_TRUE(spin_until([this]() { - return received_events_.size() >= 1; + return !received_events_.empty(); })); ASSERT_EQ(received_events_.size(), 1u); @@ -1362,7 +1365,7 @@ TEST_F(FaultEventPublishingTest, TimestampUsesWallClockNotSimTime) { ASSERT_TRUE(call_report_fault("WALL_CLOCK_TEST", Fault::SEVERITY_WARN, "/test_node")); ASSERT_TRUE(spin_until([this]() { - return received_events_.size() >= 1; + return !received_events_.empty(); })); auto wall_after = std::chrono::system_clock::now(); @@ -1390,7 +1393,7 @@ TEST_F(FaultEventPublishingTest, GetFaultReturnsExpectedFault) { // Report a fault first ASSERT_TRUE(call_report_fault("GET_FAULT_TEST", Fault::SEVERITY_ERROR, "/test_node")); ASSERT_TRUE(spin_until([this]() { - return received_events_.size() >= 1; + return !received_events_.empty(); })); // Get fault via service @@ -1414,7 +1417,7 @@ TEST_F(FaultEventPublishingTest, GetFaultReturnsEnvironmentData) { // Report a fault ASSERT_TRUE(call_report_fault("ENV_DATA_TEST", Fault::SEVERITY_WARN, "/sensor/temp")); ASSERT_TRUE(spin_until([this]() { - return received_events_.size() >= 1; + return !received_events_.empty(); })); auto response = call_get_fault("ENV_DATA_TEST"); @@ -1434,7 +1437,7 @@ TEST_F(FaultEventPublishingTest, GetFaultReturnsExtendedDataRecords) { // Report fault twice to have first and last occurrence timestamps differ ASSERT_TRUE(call_report_fault("EDR_TEST", Fault::SEVERITY_ERROR, "/node1")); ASSERT_TRUE(spin_until([this]() { - return received_events_.size() >= 1; + return !received_events_.empty(); })); ASSERT_TRUE(call_report_fault("EDR_TEST", Fault::SEVERITY_ERROR, "/node2")); ASSERT_TRUE(spin_until([this]() { @@ -1482,7 +1485,7 @@ TEST_F(FaultEventPublishingTest, ListFaultsForEntityEmptyResult) { // Report faults from a different entity ASSERT_TRUE(call_report_fault("SOME_FAULT", Fault::SEVERITY_ERROR, "/some/other_entity")); ASSERT_TRUE(spin_until([this]() { - return received_events_.size() >= 1; + return !received_events_.empty(); })); // Query faults for non-existent entity @@ -2989,6 +2992,327 @@ TEST_F(SnapshotReadPathTest, FreezeFrameStaysVisibleBehindRetainedSnapshots) { EXPECT_TRUE(frame_served) << "retained snapshots must not hide the most recent freeze-frame"; } +namespace { + +/// A store whose planned-stop write always fails, standing in for a disk that has +/// gone read-only under a running manager. Everything else behaves normally. +class PlannedStopFailingStorage : public ros2_medkit_fault_manager::InMemoryFaultStorage { + public: + void set_planned_stop(const ros2_medkit_fault_manager::PlannedStopState & /*state*/) override { + throw std::runtime_error("planned_stop table is read-only"); + } +}; + +/// Records the order of the calls a switch-off makes, so the ordering that decides +/// what a crash mid-release costs is testable without killing a process inside a +/// sub-millisecond window. +class ReleaseOrderRecordingStorage : public ros2_medkit_fault_manager::InMemoryFaultStorage { + public: + std::optional get_fault(const std::string & fault_code) const override { + calls.push_back("get_fault:" + fault_code); + return InMemoryFaultStorage::get_fault(fault_code); + } + + // The scoped overload is the one the release actually uses; both are recorded so + // neither can be hidden by overriding only one. + using InMemoryFaultStorage::clear_planned_stop_owned; + + size_t clear_planned_stop_owned() override { + calls.push_back("clear_owned"); + return InMemoryFaultStorage::clear_planned_stop_owned(); + } + + size_t clear_planned_stop_owned(const std::vector & fault_codes) override { + calls.push_back("clear_owned_scoped"); + return InMemoryFaultStorage::clear_planned_stop_owned(fault_codes); + } + + mutable std::vector calls; +}; + +/// Reaches the node's protected storage-injecting constructor. The backend is in +/// place before anything borrows it, so nothing is left dangling. +class NodeWithInjectedStorage : public FaultManagerNode { + public: + NodeWithInjectedStorage(const rclcpp::NodeOptions & options, + std::unique_ptr storage) + : FaultManagerNode(options, std::move(storage)) { + } +}; + +} // namespace + +/// The switch driven over its real services, against a node in this process. +class PlannedStopServiceTest : public ::testing::Test { + protected: + static inline std::atomic test_counter_{0}; + + /// Overridden by the failure-path fixture to hand the node a store that refuses + /// to record the declaration. + virtual std::unique_ptr make_storage() { + return nullptr; + } + + void SetUp() override { + const std::string ns = "/test_planned_stop_" + std::to_string(test_counter_.fetch_add(1)); + + rclcpp::NodeOptions fm_options; + fm_options.parameter_overrides({ + rclcpp::Parameter("storage_type", "memory"), + rclcpp::Parameter("confirmation_threshold", -1), + }); + fm_options.arguments({"--ros-args", "-r", "__ns:=" + ns}); + auto storage = make_storage(); + fault_manager_ = storage ? std::make_shared(fm_options, std::move(storage)) + : std::make_shared(fm_options); + + rclcpp::NodeOptions test_options; + test_options.arguments({"--ros-args", "-r", "__ns:=" + ns}); + test_node_ = std::make_shared("planned_stop_test_client", test_options); + + set_client_ = + test_node_->create_client(ns + "/fault_manager/set_planned_stop"); + get_client_ = + test_node_->create_client(ns + "/fault_manager/get_planned_stop"); + report_client_ = test_node_->create_client(ns + "/fault_manager/report_fault"); + ASSERT_TRUE(set_client_->wait_for_service(std::chrono::seconds(5))); + ASSERT_TRUE(get_client_->wait_for_service(std::chrono::seconds(5))); + ASSERT_TRUE(report_client_->wait_for_service(std::chrono::seconds(5))); + } + + void TearDown() override { + set_client_.reset(); + get_client_.reset(); + report_client_.reset(); + test_node_.reset(); + fault_manager_.reset(); + } + + template + bool spin_until_future_ready(FutureT & future, std::chrono::milliseconds timeout = std::chrono::milliseconds(3000)) { + auto start = std::chrono::steady_clock::now(); + while (std::chrono::steady_clock::now() - start < timeout) { + rclcpp::spin_some(fault_manager_); + rclcpp::spin_some(test_node_); + if (future.wait_for(std::chrono::milliseconds(0)) == std::future_status::ready) { + return true; + } + std::this_thread::sleep_for(std::chrono::milliseconds(5)); + } + return future.wait_for(std::chrono::milliseconds(0)) == std::future_status::ready; + } + + ros2_medkit_msgs::srv::SetPlannedStop::Response set_stop(bool active, const std::string & reason, + const std::string & declared_by) { + auto request = std::make_shared(); + request->active = active; + request->reason = reason; + request->declared_by = declared_by; + auto future = set_client_->async_send_request(request); + EXPECT_TRUE(spin_until_future_ready(future)); + return *future.get(); + } + + ros2_medkit_msgs::srv::GetPlannedStop::Response get_stop() { + auto request = std::make_shared(); + auto future = get_client_->async_send_request(request); + EXPECT_TRUE(spin_until_future_ready(future)); + return *future.get(); + } + + /// One FAILED report over the real service. The fixture's confirmation + /// threshold is -1, so a single call confirms the fault. + ros2_medkit_msgs::srv::ReportFault::Response report_failed(const std::string & fault_code) { + auto request = std::make_shared(); + request->fault_code = fault_code; + request->event_type = ros2_medkit_msgs::srv::ReportFault::Request::EVENT_FAILED; + request->severity = Fault::SEVERITY_ERROR; + request->description = "planned stop ownership test"; + request->source_id = "/test_source"; + auto future = report_client_->async_send_request(request); + EXPECT_TRUE(spin_until_future_ready(future)); + return *future.get(); + } + + std::shared_ptr fault_manager_; + std::shared_ptr test_node_; + rclcpp::Client::SharedPtr set_client_; + rclcpp::Client::SharedPtr get_client_; + rclcpp::Client::SharedPtr report_client_; +}; + +/// The same fixture, with a store that records the order of the release. +class PlannedStopReleaseOrderTest : public PlannedStopServiceTest { + protected: + std::unique_ptr make_storage() override { + auto storage = std::make_unique(); + recorder_ = storage.get(); + return storage; + } + + ReleaseOrderRecordingStorage * recorder_{nullptr}; +}; + +// The ownership flags are what a startup uses to finish a release its predecessor +// did not. They must therefore outlive the announcement: cleared first, a crash +// mid-loop would leave nothing for the next process to find, and the confirmations +// suppressed during the stop would be lost for good. +TEST_F(PlannedStopReleaseOrderTest, TheOwnershipFlagsAreClearedAfterTheAnnouncement) { + ASSERT_NE(recorder_, nullptr); + ASSERT_TRUE(set_stop(true, "line 3 maintenance", "shift_lead").success); + + // A confirmed fault the stop owns, put in place through the store directly: this test + // is about the order of the switch-off's own calls, not about the report path. + auto & storage = fault_manager_->get_storage_for_test(); + storage.report_fault_event("ORDER_OWNED", ReportFault::Request::EVENT_FAILED, Fault::SEVERITY_ERROR, "order test", + "/src", fault_manager_->now(), storage.get_debounce_config(), + /*planned_stop_active=*/true); + fault_manager_->restore_planned_stop_ownership_for_test("ORDER_OWNED"); + + recorder_->calls.clear(); + ASSERT_TRUE(set_stop(false, "plant back up", "shift_lead").success); + + const auto & calls = recorder_->calls; + const auto read = std::find(calls.begin(), calls.end(), "get_fault:ORDER_OWNED"); + const auto cleared = std::find(calls.begin(), calls.end(), "clear_owned"); + ASSERT_NE(read, calls.end()) << "the switch-off never looked the released fault up"; + ASSERT_NE(cleared, calls.end()) << "the switch-off never released the ownership flags"; + EXPECT_LT(read - calls.begin(), cleared - calls.begin()) + << "the flags were cleared before the announcement, so a crash in between would lose it"; +} + +// The handler has to hand the store the switch, or a cycle that starts inside the stop +// is confirmed and unowned: no switch-off releases it, no startup recognises it as +// interrupted, and its later reports announce updates mid-stop. The store is the only +// writer of the flag, so the report returning with it set is the whole path. +TEST_F(PlannedStopServiceTest, TheReportItselfRecordsTheStopsOwnership) { + ASSERT_TRUE(set_stop(true, "line 3 maintenance", "shift_lead").success); + + ASSERT_TRUE(report_failed("PS_OWNED_BY_REPORT").accepted); + + const auto owned = fault_manager_->get_storage_for_test().get_planned_stop_owned(); + EXPECT_NE(std::find(owned.begin(), owned.end(), "PS_OWNED_BY_REPORT"), owned.end()) + << "the report returned with the fault unowned, so the handler withheld the switch"; + + // A fault raised after the withdrawal belongs to nobody. + ASSERT_TRUE(set_stop(false, "plant back up", "shift_lead").success); + ASSERT_TRUE(report_failed("PS_AFTER_THE_STOP").accepted); + const auto after = fault_manager_->get_storage_for_test().get_planned_stop_owned(); + EXPECT_EQ(std::find(after.begin(), after.end(), "PS_AFTER_THE_STOP"), after.end()); +} + +/// The same fixture, with a store that refuses to record the declaration. +class PlannedStopFailingStoreTest : public PlannedStopServiceTest { + protected: + std::unique_ptr make_storage() override { + return std::make_unique(); + } +}; + +// A store that cannot record the declaration must not leave the caller believing +// a stop is in force, and must not take the process down with it. +TEST_F(PlannedStopFailingStoreTest, AStoreThatRejectsTheWriteIsReportedNotThrown) { + const auto response = set_stop(true, "line 3 maintenance", "shift_lead"); + EXPECT_FALSE(response.success); + EXPECT_FALSE(response.message.empty()); + EXPECT_FALSE(response.was_active); + + // The switch is where it was, and the node is still answering. + const auto state = get_stop(); + EXPECT_FALSE(state.active); + EXPECT_TRUE(state.reason.empty()); +} + +// "Why was line 3 quiet on Friday?" has to be answerable after the plant is back +// up, without reading a raw audit database. +TEST_F(PlannedStopServiceTest, TheLastDeclarationIsReadableAfterTheWithdrawal) { + const auto before_any = get_stop(); + EXPECT_FALSE(before_any.active); + EXPECT_EQ(0, before_any.since.sec); + EXPECT_EQ(0, before_any.ended_at.sec); + + ASSERT_TRUE(set_stop(true, "line 3 quarterly maintenance", "shift_lead").success); + const auto during = get_stop(); + ASSERT_TRUE(during.active); + EXPECT_EQ("line 3 quarterly maintenance", during.reason); + EXPECT_GT(during.since.sec, 0); + EXPECT_EQ(0, during.ended_at.sec) << "a stop that is still in force has not ended"; + + ASSERT_TRUE(set_stop(false, "plant back up", "shift_lead").success); + + const auto after = get_stop(); + EXPECT_FALSE(after.active); + EXPECT_EQ("line 3 quarterly maintenance", after.reason) << "the declaration that started the stop"; + EXPECT_EQ("shift_lead", after.declared_by); + EXPECT_EQ(during.since.sec, after.since.sec); + EXPECT_GT(after.ended_at.sec, 0) << "the withdrawal has to be readable too"; +} + +// The audit rows for the switch describe the installation, so they carry the same +// sentinel fault code the log's own lifecycle markers use. +TEST(PlannedStopAuditTest, TransitionsUseTheInstallationSentinelFaultCode) { + const std::string audit_path = make_temp_audit_path("test_audit_planned_stop_"); + { + rclcpp::NodeOptions options; + options.parameter_overrides({ + {"storage_type", "memory"}, + {"audit_log.enabled", true}, + {"audit_log.database_path", audit_path}, + }); + options.arguments({"--ros-args", "-r", "__ns:=/test_planned_stop_audit"}); + auto node = std::make_shared(options); + + auto client_node = std::make_shared( + "planned_stop_audit_client", + rclcpp::NodeOptions().arguments({"--ros-args", "-r", "__ns:=/test_planned_stop_audit"})); + auto client = client_node->create_client( + "/test_planned_stop_audit/fault_manager/set_planned_stop"); + ASSERT_TRUE(client->wait_for_service(std::chrono::seconds(5))); + + auto call = [&](bool active, const std::string & reason) { + auto request = std::make_shared(); + request->active = active; + request->reason = reason; + request->declared_by = "shift_lead"; + auto future = client->async_send_request(request); + auto start = std::chrono::steady_clock::now(); + while (std::chrono::steady_clock::now() - start < std::chrono::seconds(3)) { + rclcpp::spin_some(node); + rclcpp::spin_some(client_node); + if (future.wait_for(std::chrono::milliseconds(0)) == std::future_status::ready) { + break; + } + std::this_thread::sleep_for(std::chrono::milliseconds(5)); + } + ASSERT_EQ(future.wait_for(std::chrono::milliseconds(0)), std::future_status::ready); + ASSERT_TRUE(future.get()->success); + }; + call(true, "maintenance"); + call(false, "back up"); + + const auto * log = node->get_audit_log_for_test(); + ASSERT_NE(log, nullptr); + bool saw_started = false; + bool saw_ended = false; + for (const auto & record : log->read()) { + if (record.event.transition == ros2_medkit_fault_manager::kTransitionPlannedStopStarted) { + saw_started = true; + EXPECT_EQ("__audit__", record.event.fault_code); + EXPECT_EQ("maintenance", record.event.description); + EXPECT_EQ("shift_lead", record.event.source_id); + } else if (record.event.transition == ros2_medkit_fault_manager::kTransitionPlannedStopEnded) { + saw_ended = true; + EXPECT_EQ("__audit__", record.event.fault_code); + EXPECT_EQ("back up", record.event.description); + } + } + EXPECT_TRUE(saw_started); + EXPECT_TRUE(saw_ended); + EXPECT_TRUE(log->verify().ok); + } + remove_audit_files(audit_path); +} + int main(int argc, char ** argv) { rclcpp::init(argc, argv); ::testing::InitGoogleTest(&argc, argv); diff --git a/src/ros2_medkit_fault_manager/test/test_sqlite_storage.cpp b/src/ros2_medkit_fault_manager/test/test_sqlite_storage.cpp index 391ddd908..500c034b8 100644 --- a/src/ros2_medkit_fault_manager/test/test_sqlite_storage.cpp +++ b/src/ros2_medkit_fault_manager/test/test_sqlite_storage.cpp @@ -16,6 +16,7 @@ #include +#include #include #include #include @@ -26,6 +27,7 @@ #include #include "rclcpp/rclcpp.hpp" +#include "ros2_medkit_fault_manager/fault_storage.hpp" #include "ros2_medkit_fault_manager/sqlite_fault_storage.hpp" #include "ros2_medkit_msgs/msg/fault.hpp" #include "ros2_medkit_msgs/srv/report_fault.hpp" @@ -2391,6 +2393,442 @@ TEST_F(SqliteFaultStorageTest, SnapshotsDroppedByHealedReclassificationByDefault EXPECT_TRUE(storage_->get_snapshots("SNAPSHOT_DROP_TEST").empty()); } +// ============================================================================ +// Planned-stop declaration (survives a restart) +// ============================================================================ + +TEST_F(SqliteFaultStorageTest, PlannedStopDefaultsToInactive) { + const auto state = storage_->get_planned_stop(); + EXPECT_FALSE(state.active); + EXPECT_TRUE(state.reason.empty()); + EXPECT_TRUE(state.declared_by.empty()); + EXPECT_EQ(0, state.since_ns); +} + +TEST_F(SqliteFaultStorageTest, PlannedStopRoundTripsThroughAReopen) { + ros2_medkit_fault_manager::PlannedStopState declared; + declared.active = true; + declared.reason = "line 3 quarterly maintenance"; + declared.declared_by = "shift_lead"; + declared.since_ns = 1757000000000000000; + storage_->set_planned_stop(declared); + + // Reopening the same file is what a restart of the fault manager does. + storage_.reset(); + storage_ = std::make_unique(temp_db_path_.string()); + + const auto reopened = storage_->get_planned_stop(); + EXPECT_TRUE(reopened.active); + EXPECT_EQ("line 3 quarterly maintenance", reopened.reason); + EXPECT_EQ("shift_lead", reopened.declared_by); + EXPECT_EQ(1757000000000000000, reopened.since_ns); +} + +TEST_F(SqliteFaultStorageTest, PlannedStopKeepsOneRowAcrossRepeatedWrites) { + ros2_medkit_fault_manager::PlannedStopState first; + first.active = true; + first.reason = "first"; + first.declared_by = "a"; + first.since_ns = 111; + storage_->set_planned_stop(first); + + ros2_medkit_fault_manager::PlannedStopState second; + second.active = true; + second.reason = "second"; + second.declared_by = "b"; + second.since_ns = 222; + storage_->set_planned_stop(second); + + storage_.reset(); + storage_ = std::make_unique(temp_db_path_.string()); + + const auto state = storage_->get_planned_stop(); + EXPECT_EQ("second", state.reason); + EXPECT_EQ("b", state.declared_by); + EXPECT_EQ(222, state.since_ns); +} + +TEST_F(SqliteFaultStorageTest, WithdrawnPlannedStopIsWhatAReopenSees) { + ros2_medkit_fault_manager::PlannedStopState declared; + declared.active = true; + declared.reason = "maintenance"; + declared.declared_by = "shift_lead"; + declared.since_ns = 999; + storage_->set_planned_stop(declared); + storage_->set_planned_stop(ros2_medkit_fault_manager::PlannedStopState{}); + + storage_.reset(); + storage_ = std::make_unique(temp_db_path_.string()); + + const auto state = storage_->get_planned_stop(); + EXPECT_FALSE(state.active); + EXPECT_TRUE(state.reason.empty()); + EXPECT_TRUE(state.declared_by.empty()); + EXPECT_EQ(0, state.since_ns); +} + +TEST_F(SqliteFaultStorageTest, AWithdrawnDeclarationKeepsItsReasonAndItsEndTime) { + ros2_medkit_fault_manager::PlannedStopState declared; + declared.active = true; + declared.reason = "line 3 quarterly maintenance"; + declared.declared_by = "shift_lead"; + declared.since_ns = 1757000000000000000; + storage_->set_planned_stop(declared); + + // What a withdrawal writes: the declaration survives it, stamped with its end. + auto withdrawn = declared; + withdrawn.active = false; + withdrawn.ended_at_ns = 1757000009000000000; + storage_->set_planned_stop(withdrawn); + + storage_.reset(); + storage_ = std::make_unique(temp_db_path_.string()); + + const auto state = storage_->get_planned_stop(); + EXPECT_FALSE(state.active); + EXPECT_EQ("line 3 quarterly maintenance", state.reason); + EXPECT_EQ("shift_lead", state.declared_by); + EXPECT_EQ(1757000000000000000, state.since_ns); + EXPECT_EQ(1757000009000000000, state.ended_at_ns); +} + +TEST_F(SqliteFaultStorageTest, ADatabaseWrittenBeforeTheEndTimeExistedStillOpens) { + // A row written by a build that had no ended_at_ns column reads as one that was + // never withdrawn, which is what a missing end time means. + storage_.reset(); + { + sqlite3 * raw = nullptr; + ASSERT_EQ(sqlite3_open(temp_db_path_.string().c_str(), &raw), SQLITE_OK); + ASSERT_EQ(sqlite3_exec(raw, "DROP TABLE IF EXISTS planned_stop", nullptr, nullptr, nullptr), SQLITE_OK); + ASSERT_EQ(sqlite3_exec(raw, + "CREATE TABLE planned_stop (id INTEGER PRIMARY KEY CHECK (id = 1), active INTEGER NOT NULL," + " reason TEXT NOT NULL, declared_by TEXT NOT NULL, since_ns INTEGER NOT NULL);" + "INSERT INTO planned_stop (id, active, reason, declared_by, since_ns)" + " VALUES (1, 1, 'legacy stop', 'shift_lead', 42);", + nullptr, nullptr, nullptr), + SQLITE_OK); + sqlite3_close(raw); + } + + storage_ = std::make_unique(temp_db_path_.string()); + + const auto state = storage_->get_planned_stop(); + EXPECT_TRUE(state.active); + EXPECT_EQ("legacy stop", state.reason); + EXPECT_EQ(42, state.since_ns); + EXPECT_EQ(0, state.ended_at_ns); +} + +// ============================================================================ +// Planned-stop ownership: one persisted flag per fault +// ============================================================================ + +namespace { + +/// Drive a fault to CONFIRMED with the immediate-confirmation default, letting a declared +/// planned stop take the cycle. A report is the only thing that makes a fault the stop's, so +/// it is also how a test sets one up. +void raise(ros2_medkit_fault_manager::FaultStorage & storage, const std::string & code, + bool planned_stop_active = false) { + rclcpp::Clock clock; + storage.report_fault_event(code, ReportFault::Request::EVENT_FAILED, Fault::SEVERITY_ERROR, "owned test", "/src", + clock.now(), default_config(), planned_stop_active); +} + +} // namespace + +TEST_F(SqliteFaultStorageTest, PlannedStopOwnershipRoundTripsThroughAReopen) { + raise(*storage_, "NOT_OWNED"); + EXPECT_TRUE(storage_->get_planned_stop_owned().empty()); + + raise(*storage_, "OWNED_ONE", /*planned_stop_active=*/true); + raise(*storage_, "OWNED_TWO", /*planned_stop_active=*/true); + + storage_.reset(); + storage_ = std::make_unique(temp_db_path_.string()); + + auto owned = storage_->get_planned_stop_owned(); + std::sort(owned.begin(), owned.end()); + ASSERT_EQ(2u, owned.size()); + EXPECT_EQ("OWNED_ONE", owned[0]); + EXPECT_EQ("OWNED_TWO", owned[1]); +} + +TEST_F(SqliteFaultStorageTest, AcknowledgingAFaultEndsTheStopsOwnershipOfIt) { + raise(*storage_, "OWNED_CLEARED", /*planned_stop_active=*/true); + ASSERT_EQ(1u, storage_->get_planned_stop_owned().size()); + + ASSERT_TRUE(storage_->clear_fault("OWNED_CLEARED")); + + EXPECT_TRUE(storage_->get_planned_stop_owned().empty()) + << "an acknowledged fault is over, so the stop has nothing left to own"; +} + +TEST_F(SqliteFaultStorageTest, ClearingOwnershipDropsEveryFlagAndReportsHowMany) { + raise(*storage_, "OWNED_A", /*planned_stop_active=*/true); + raise(*storage_, "OWNED_B", /*planned_stop_active=*/true); + + EXPECT_EQ(2u, storage_->clear_planned_stop_owned()); + EXPECT_TRUE(storage_->get_planned_stop_owned().empty()); + EXPECT_EQ(0u, storage_->clear_planned_stop_owned()); +} + +TEST_F(SqliteFaultStorageTest, ClearingNamedOwnershipLeavesEveryOtherFlagAlone) { + for (const auto * code : {"OWNED_CAPTURED_A", "OWNED_CAPTURED_B", "OWNED_LATER"}) { + raise(*storage_, code, /*planned_stop_active=*/true); + } + + // A release finishes what it captured. Anything that became owned after that + // capture belongs to a different declaration and must survive. + EXPECT_EQ(2u, storage_->clear_planned_stop_owned({"OWNED_CAPTURED_A", "OWNED_CAPTURED_B"})); + + auto owned = storage_->get_planned_stop_owned(); + ASSERT_EQ(1u, owned.size()); + EXPECT_EQ("OWNED_LATER", owned[0]); + + // Idempotent, and a code that is not owned is not an error. + EXPECT_EQ(0u, storage_->clear_planned_stop_owned({"OWNED_CAPTURED_A", "NEVER_OWNED"})); + EXPECT_EQ(1u, storage_->get_planned_stop_owned().size()); + EXPECT_EQ(0u, storage_->clear_planned_stop_owned({})); + EXPECT_EQ(1u, storage_->get_planned_stop_owned().size()); +} + +TEST(InMemoryFaultStorageTest, ClearingNamedOwnershipLeavesEveryOtherFlagAlone) { + ros2_medkit_fault_manager::InMemoryFaultStorage storage; + raise(storage, "OWNED_CAPTURED", /*planned_stop_active=*/true); + raise(storage, "OWNED_LATER", /*planned_stop_active=*/true); + + EXPECT_EQ(1u, storage.clear_planned_stop_owned({"OWNED_CAPTURED"})); + + auto owned = storage.get_planned_stop_owned(); + ASSERT_EQ(1u, owned.size()); + EXPECT_EQ("OWNED_LATER", owned[0]); +} + +TEST_F(SqliteFaultStorageTest, ADatabaseWrittenBeforeOwnershipExistedStillOpens) { + raise(*storage_, "LEGACY_FAULT"); + storage_.reset(); + { + sqlite3 * raw = nullptr; + ASSERT_EQ(sqlite3_open(temp_db_path_.string().c_str(), &raw), SQLITE_OK); + // What a database from before the column looks like. + ASSERT_EQ(sqlite3_exec(raw, "ALTER TABLE faults DROP COLUMN planned_stop_owned", nullptr, nullptr, nullptr), + SQLITE_OK); + sqlite3_close(raw); + } + + storage_ = std::make_unique(temp_db_path_.string()); + + EXPECT_TRUE(storage_->get_planned_stop_owned().empty()); + EXPECT_TRUE(storage_->contains("LEGACY_FAULT")); + + // The re-added column takes writes: acknowledge the migrated fault and raise it again + // inside a stop, which is a cycle starting, and the flag lands on the migrated row. + ASSERT_TRUE(storage_->clear_fault("LEGACY_FAULT")); + raise(*storage_, "LEGACY_FAULT", /*planned_stop_active=*/true); + EXPECT_EQ(1u, storage_->get_planned_stop_owned().size()); +} + +TEST(InMemoryFaultStorageTest, PlannedStopOwnershipIsTrackedPerFault) { + ros2_medkit_fault_manager::InMemoryFaultStorage storage; + raise(storage, "OWNED_ONE", /*planned_stop_active=*/true); + raise(storage, "NOT_OWNED"); + + auto owned = storage.get_planned_stop_owned(); + ASSERT_EQ(1u, owned.size()); + EXPECT_EQ("OWNED_ONE", owned[0]); + + ASSERT_TRUE(storage.clear_fault("OWNED_ONE")); + EXPECT_TRUE(storage.get_planned_stop_owned().empty()); + + raise(storage, "NOT_OWNED", /*planned_stop_active=*/true); + EXPECT_TRUE(storage.get_planned_stop_owned().empty()) + << "a report that repeats a standing condition is the same cycle, so the stop cannot take it"; +} + +TEST_F(SqliteFaultStorageTest, PlannedStopCarriesAKilobyteReason) { + const std::string long_reason(1024, 'r'); + ros2_medkit_fault_manager::PlannedStopState declared; + declared.active = true; + declared.reason = long_reason; + declared.since_ns = 7; + storage_->set_planned_stop(declared); + + storage_.reset(); + storage_ = std::make_unique(temp_db_path_.string()); + + const auto state = storage_->get_planned_stop(); + EXPECT_TRUE(state.active); + EXPECT_EQ(long_reason, state.reason); + EXPECT_TRUE(state.declared_by.empty()); +} + +TEST(InMemoryFaultStorageTest, PlannedStopRoundTripsInMemory) { + ros2_medkit_fault_manager::InMemoryFaultStorage storage; + + EXPECT_FALSE(storage.get_planned_stop().active); + + ros2_medkit_fault_manager::PlannedStopState declared; + declared.active = true; + declared.reason = "cell 7 changeover"; + declared.declared_by = "maintenance"; + declared.since_ns = 42; + storage.set_planned_stop(declared); + + const auto state = storage.get_planned_stop(); + EXPECT_TRUE(state.active); + EXPECT_EQ("cell 7 changeover", state.reason); + EXPECT_EQ("maintenance", state.declared_by); + EXPECT_EQ(42, state.since_ns); + + storage.set_planned_stop(ros2_medkit_fault_manager::PlannedStopState{}); + EXPECT_FALSE(storage.get_planned_stop().active); + EXPECT_TRUE(storage.get_planned_stop().reason.empty()); +} + +namespace { + +/// Where a planned stop takes ownership of a fault cycle, driven against one backend. +/// Both backends have to answer identically: which faults a switch-off releases must +/// not depend on the storage a deployment picked. +void expect_planned_stop_cycle_boundary(ros2_medkit_fault_manager::FaultStorage & storage) { + rclcpp::Clock clock; + DebounceConfig config = default_config(); + config.healing_enabled = true; + config.healing_threshold = 0; + + auto owned_codes = [&storage]() { + auto codes = storage.get_planned_stop_owned(); + std::sort(codes.begin(), codes.end()); + return codes; + }; + const std::vector inside_only{"PS_INSIDE"}; + const std::vector both{"PS_INSIDE", "PS_OUTSIDE"}; + + // A fault raised inside a stop is owned by the report that created it. Read back + // straight after that one call: nothing else has run, so nothing else could have + // written the flag. + storage.report_fault_event("PS_INSIDE", ReportFault::Request::EVENT_FAILED, Fault::SEVERITY_ERROR, "inside", "/src", + clock.now(), config, /*planned_stop_active=*/true); + EXPECT_EQ(inside_only, owned_codes()) << "the report did not carry the stop's ownership"; + + // With no stop in force nothing is owned. + storage.report_fault_event("PS_OUTSIDE", ReportFault::Request::EVENT_FAILED, Fault::SEVERITY_ERROR, "outside", "/src", + clock.now(), config, /*planned_stop_active=*/false); + EXPECT_EQ(inside_only, owned_codes()); + + // A repeat report of a condition that was already up is the same cycle, so a stop + // declared after it started does not take it. + storage.report_fault_event("PS_OUTSIDE", ReportFault::Request::EVENT_FAILED, Fault::SEVERITY_ERROR, "outside", "/src", + clock.now(), config, /*planned_stop_active=*/true); + EXPECT_EQ(inside_only, owned_codes()) << "a standing fault was taken by a stop declared after it"; + + // Nor does a later report hand an owned cycle back. + storage.report_fault_event("PS_INSIDE", ReportFault::Request::EVENT_FAILED, Fault::SEVERITY_ERROR, "inside", "/src", + clock.now(), config, /*planned_stop_active=*/false); + EXPECT_EQ(inside_only, owned_codes()) << "a repeat report dropped the stop ownership"; + + // Acknowledging ends the cycle the stop owned; raising the fault again starts one + // the stop owns afresh. + ASSERT_TRUE(storage.clear_fault("PS_INSIDE")); + EXPECT_TRUE(owned_codes().empty()); + storage.report_fault_event("PS_INSIDE", ReportFault::Request::EVENT_FAILED, Fault::SEVERITY_ERROR, "inside", "/src", + clock.now(), config, /*planned_stop_active=*/true); + EXPECT_EQ(inside_only, owned_codes()) << "a fault raised again inside the stop was not owned"; + + // A PASSED report starts nothing, whatever the switch says. + storage.report_fault_event("PS_OUTSIDE", ReportFault::Request::EVENT_PASSED, Fault::SEVERITY_ERROR, "outside", "/src", + clock.now(), config, /*planned_stop_active=*/true); + EXPECT_EQ(inside_only, owned_codes()) << "a PASSED report took ownership of a cycle"; + auto healed = storage.get_fault("PS_OUTSIDE"); + ASSERT_TRUE(healed.has_value()); + ASSERT_EQ(Fault::STATUS_HEALED, healed->status); + + // The heal published the fault's end, so failing again is fresh news and a new + // cycle for the stop to own. + storage.report_fault_event("PS_OUTSIDE", ReportFault::Request::EVENT_FAILED, Fault::SEVERITY_ERROR, "outside", "/src", + clock.now(), config, /*planned_stop_active=*/true); + EXPECT_EQ(both, owned_codes()) << "a fault that failed again after healing was not owned"; +} + +} // namespace + +TEST_F(SqliteFaultStorageTest, TheReportCarriesThePlannedStopsOwnership) { + expect_planned_stop_cycle_boundary(*storage_); +} + +TEST(InMemoryFaultStorageTest, TheReportCarriesThePlannedStopsOwnership) { + ros2_medkit_fault_manager::InMemoryFaultStorage storage; + expect_planned_stop_cycle_boundary(storage); +} + +namespace { + +/// Run one statement on a second connection to the same database file. Used to install +/// and remove a trigger, which is a schema change every connection to the file honours, +/// including the one the store is holding. +void exec_on_a_second_connection(const std::string & db_path, const char * sql) { + sqlite3 * raw = nullptr; + ASSERT_EQ(sqlite3_open(db_path.c_str(), &raw), SQLITE_OK); + char * error = nullptr; + const int rc = sqlite3_exec(raw, sql, nullptr, nullptr, &error); + const std::string message = error != nullptr ? error : ""; + sqlite3_free(error); + sqlite3_close(raw); + ASSERT_EQ(rc, SQLITE_OK) << message; +} + +constexpr const char * kRefuseOwnershipWrites = + "CREATE TRIGGER deny_ownership BEFORE UPDATE OF planned_stop_owned ON faults " + "BEGIN SELECT RAISE(ABORT, 'ownership write refused'); END;"; + +} // namespace + +// The whole point of carrying ownership with the report is that the two cannot come +// apart. A store that keeps the fault row when the ownership write fails leaves a +// confirmed fault inside a stop that nothing owns: no switch-off releases it, no +// startup recognises it, and its later reports announce updates mid-stop. Driven on +// SQLite because that is the backend the crash window is about, and through a trigger +// because that makes the ownership statement fail where it really runs, inside the +// report's transaction. +TEST_F(SqliteFaultStorageTest, AnOwnershipWriteThatFailsTakesTheReportDownWithIt) { + rclcpp::Clock clock; + ASSERT_NO_FATAL_FAILURE(exec_on_a_second_connection(storage_->db_path(), kRefuseOwnershipWrites)); + + EXPECT_THROW(storage_->report_fault_event("PS_ATOMIC", ReportFault::Request::EVENT_FAILED, Fault::SEVERITY_ERROR, + "atomic test", "/src", clock.now(), default_config(), + /*planned_stop_active=*/true), + std::runtime_error); + + storage_.reset(); + ASSERT_NO_FATAL_FAILURE(exec_on_a_second_connection(temp_db_path_.string(), "DROP TRIGGER deny_ownership")); + + // Reopened, because the question is what survived the commit, not what one + // connection happens to be holding. + SqliteFaultStorage reopened(temp_db_path_.string()); + EXPECT_FALSE(reopened.contains("PS_ATOMIC")) << "the fault row committed without the ownership that belongs to it"; + EXPECT_TRUE(reopened.get_planned_stop_owned().empty()); +} + +// A report with no stop in force writes no ownership at all, so the same refusal +// cannot touch it: the negative control that keeps the test above about the +// ownership statement rather than about the trigger. +TEST_F(SqliteFaultStorageTest, TheSameRefusalDoesNotTouchAReportOutsideAStop) { + rclcpp::Clock clock; + ASSERT_NO_FATAL_FAILURE(exec_on_a_second_connection(storage_->db_path(), kRefuseOwnershipWrites)); + + EXPECT_NO_THROW(storage_->report_fault_event("PS_NO_STOP", ReportFault::Request::EVENT_FAILED, Fault::SEVERITY_ERROR, + "atomic test", "/src", clock.now(), default_config(), + /*planned_stop_active=*/false)); + + storage_.reset(); + ASSERT_NO_FATAL_FAILURE(exec_on_a_second_connection(temp_db_path_.string(), "DROP TRIGGER deny_ownership")); + + SqliteFaultStorage reopened(temp_db_path_.string()); + EXPECT_TRUE(reopened.contains("PS_NO_STOP")); + EXPECT_TRUE(reopened.get_planned_stop_owned().empty()); +} + int main(int argc, char ** argv) { rclcpp::init(argc, argv); ::testing::InitGoogleTest(&argc, argv); diff --git a/src/ros2_medkit_integration_tests/test/features/test_planned_stop_switch.test.py b/src/ros2_medkit_integration_tests/test/features/test_planned_stop_switch.test.py new file mode 100644 index 000000000..b1f019364 --- /dev/null +++ b/src/ros2_medkit_integration_tests/test/features/test_planned_stop_switch.test.py @@ -0,0 +1,381 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""The planned-stop switch driven the way an operator reaches it. + +No route was added for the switch: the fault manager's services are SOVD +operations on its own App entity, so an operator turns the stop on with +``POST /apps/fault_manager/operations/set_planned_stop/executions``. This test +drives it that way, watches ``GET /faults`` and the ``/faults/stream`` SSE feed, +and asserts what an operator sees: nothing about the faults raised during the +stop until it ends, then every one of them at once. + +The stream instrument counts ``fault_confirmed`` frames PER FAULT CODE, before +and after the switch-off. A total alone would pass while the wrong fault was +announced. +""" + +import json +import threading +import time +import unittest + +import launch_testing +import launch_testing.actions +import rclpy +from rclpy.node import Node +import requests +from ros2_medkit_msgs.msg import Fault +from ros2_medkit_msgs.srv import ReportFault + +from ros2_medkit_test_utils.constants import ALLOWED_EXIT_CODES, FAULT_TIMEOUT +from ros2_medkit_test_utils.gateway_test_case import GatewayTestCase +from ros2_medkit_test_utils.launch_helpers import create_test_launch + +SOURCE_ID = '/powertrain/engine/temp_sensor' +MUTED_CODE = 'GW_PS_MUTED' +SECOND_CODE = 'GW_PS_MUTED_TWO' +AFTER_CODE = 'GW_PS_AFTER' +ALREADY_UP_CODE = 'GW_PS_ALREADY_UP' + +# How long to keep watching after the counts are right. The publish path is one +# service call plus a topic hop plus the SSE writer, so a second frame that the +# assertion could otherwise race arrives well inside this. +SETTLE_SEC = 4.0 + +SET_STOP = '/apps/fault_manager/operations/set_planned_stop/executions' +GET_STOP = '/apps/fault_manager/operations/get_planned_stop/executions' + + +def generate_test_description(): + return create_test_launch( + demo_nodes=['temp_sensor'], + fault_manager=True, + fault_manager_params={ + 'storage_type': 'memory', + 'confirmation_threshold': -1, + 'snapshots.rosbag.enabled': False, + }, + ) + + +class TestPlannedStopSwitch(GatewayTestCase): + """An operator declares a stop over SOVD and the fault view obeys it.""" + + MIN_EXPECTED_APPS = 2 + # The fault manager is an App like any other node, which is exactly why the + # switch needs no route of its own. + REQUIRED_APPS = {'temp_sensor', 'fault_manager'} + + @classmethod + def setUpClass(cls): + super().setUpClass() + rclpy.init() + cls._reporter = Node('planned_stop_switch_reporter') + cls._report_client = cls._reporter.create_client( + ReportFault, '/fault_manager/report_fault') + assert cls._report_client.wait_for_service(timeout_sec=20.0), \ + 'report_fault service not available' + + # One stream for the whole class. Frames accumulate across the cases and + # every assertion counts them per fault code, so there is nothing to + # reset between tests - and a stream reopened per test costs far more + # than it proves, because the gateway only notices a closed SSE client + # on its next write. + cls._frames = [] + cls._stop_event = threading.Event() + cls._response = requests.get( + f'{cls.BASE_URL}/faults/stream', stream=True, timeout=(5, 300)) + assert cls._response.status_code == 200, 'the fault stream did not open' + cls._pump = threading.Thread( + target=cls._pump_stream, args=(cls._response, cls._frames, cls._stop_event), + daemon=True) + cls._pump.start() + cls._prime_stream() + + @classmethod + def tearDownClass(cls): + cls._stop_event.set() + cls._response.close() + cls._pump.join(timeout=10) + cls._reporter.destroy_node() + rclpy.shutdown() + + def tearDown(self): + # One declaration per manager: a test that left it on would decide the + # next one's outcome. + self._set_stop(False, reason='teardown', declared_by='test') + + # ------------------------------------------------------------------ + # Helpers + # ------------------------------------------------------------------ + + @classmethod + def _report(cls, fault_code): + request = ReportFault.Request() + request.fault_code = fault_code + request.event_type = ReportFault.Request.EVENT_FAILED + request.severity = Fault.SEVERITY_ERROR + request.description = 'planned stop switch test fault' + request.source_id = SOURCE_ID + future = cls._report_client.call_async(request) + deadline = time.monotonic() + 10.0 + while not future.done() and time.monotonic() < deadline: + rclpy.spin_once(cls._reporter, timeout_sec=0.05) + assert future.done(), 'report_fault call timed out' + assert future.result().accepted, 'report_fault was rejected' + + @staticmethod + def _pump_stream(response, frames, stop_event): + """Collect SSE frames as dicts of field -> value.""" + current = {} + try: + for line in response.iter_lines(decode_unicode=True): + if stop_event.is_set(): + break + if line is None: + continue + if line == '': + if current: + frames.append(current) + current = {} + continue + if line.startswith(':'): + continue # keepalive comment + key, _, value = line.partition(':') + current[key.strip()] = value.strip() + except Exception: # noqa: BLE001 - closed socket on test teardown + pass + + @classmethod + def _prime_stream(cls): + """Block until the fault event pipeline demonstrably reaches the stream. + + ``/fault_manager/events`` is reliable but volatile: an event published + before the gateway's subscription has matched the publisher is lost, and + a one-shot assertion would race that match. + """ + deadline = time.monotonic() + FAULT_TIMEOUT + attempt = 0 + while time.monotonic() < deadline: + # A fresh code per attempt: re-reporting one that is already + # CONFIRMED publishes fault_updated, never fault_confirmed. + code = f'GW_PS_PRIME_{attempt}' + cls._report(code) + settle = min(time.monotonic() + 1.0, deadline) + while time.monotonic() < settle: + if cls._count_confirmed(code) > 0: + return + time.sleep(0.1) + attempt += 1 + raise AssertionError( + f'no priming event on /faults/stream within {FAULT_TIMEOUT}s; ' + 'the events pipeline never went live') + + @classmethod + def _count_confirmed(cls, fault_code): + count = 0 + for frame in list(cls._frames): + if frame.get('event') != 'fault_confirmed' or 'data' not in frame: + continue + payload = json.loads(frame['data']) + if payload.get('fault', {}).get('fault_code') == fault_code: + count += 1 + return count + + def _set_stop(self, active, *, reason='', declared_by=''): + response = requests.post( + f'{self.BASE_URL}{SET_STOP}', + json={'parameters': {'active': active, 'reason': reason, + 'declared_by': declared_by}}, + timeout=20) + self.assertEqual(200, response.status_code, response.text) + return response.json()['parameters'] + + def _get_stop(self): + response = requests.post(f'{self.BASE_URL}{GET_STOP}', json={}, timeout=20) + self.assertEqual(200, response.status_code, response.text) + return response.json()['parameters'] + + def _faults(self, *, include_muted=False): + """Return the fault list an operator sees, and the set of codes in it. + + No ``status`` filter: the default view (pending + confirmed) is the one + the switch is supposed to change. + """ + params = {'include_muted': 'true'} if include_muted else {} + response = requests.get(f'{self.BASE_URL}/faults', params=params, timeout=15) + self.assertEqual(200, response.status_code, response.text) + data = response.json() + return data, {item['fault_code'] for item in data.get('items', [])} + + def _wait_until(self, predicate, message, timeout=None): + deadline = time.monotonic() + (timeout if timeout is not None else FAULT_TIMEOUT) + while time.monotonic() < deadline: + value = predicate() + if value: + return value + time.sleep(0.2) + raise AssertionError(message) + + # ------------------------------------------------------------------ + # Cases + # ------------------------------------------------------------------ + + def test_the_switch_is_an_operation_on_the_fault_manager_entity(self): + """The switch is reachable as an operation on the fault manager entity. + + @verifies REQ_INTEROP_035 + """ + data = self.get_json('/apps/fault_manager/operations') + operation_ids = {item['id'] for item in data.get('items', [])} + self.assertIn('set_planned_stop', operation_ids) + self.assertIn('get_planned_stop', operation_ids) + + state = self._get_stop() + self.assertFalse(state['active']) + + def test_faults_raised_during_a_stop_surface_only_when_it_ends(self): + """Faults raised inside a stop stay hidden until it is withdrawn. + + @verifies REQ_INTEROP_012 + """ + declared = self._set_stop(True, reason='line 3 maintenance', declared_by='shift_lead') + self.assertTrue(declared['success']) + self.assertFalse(declared['was_active']) + + state = self._get_stop() + self.assertTrue(state['active']) + self.assertEqual('line 3 maintenance', state['reason']) + self.assertEqual('shift_lead', state['declared_by']) + + before_muted = self._faults()[0]['x-medkit']['muted_count'] + + self._report(MUTED_CODE) + self._report(SECOND_CODE) + + # The default view hides them; asking for muted entries shows them, with + # the switch named as the reason. + def _both_muted(): + data, codes = self._faults(include_muted=True) + muted = {entry['fault_code']: entry + for entry in data['x-medkit'].get('muted_faults', [])} + if MUTED_CODE not in muted or SECOND_CODE not in muted: + return None + # Asking for muted entries also stops them being stripped from items. + self.assertIn(MUTED_CODE, codes) + self.assertIn(SECOND_CODE, codes) + return muted + + muted = self._wait_until(_both_muted, 'the faults raised during the stop were not marked') + for code in (MUTED_CODE, SECOND_CODE): + self.assertEqual('planned_stop', muted[code]['rule_id']) + self.assertEqual('PLANNED_STOP', muted[code]['root_cause_code']) + + data, visible = self._faults() + self.assertNotIn(MUTED_CODE, visible) + self.assertNotIn(SECOND_CODE, visible) + self.assertGreaterEqual(data['x-medkit']['muted_count'], before_muted + 2) + + # And the stream said nothing about either of them. + self.assertEqual(0, self._count_confirmed(MUTED_CODE)) + self.assertEqual(0, self._count_confirmed(SECOND_CODE)) + + withdrawn = self._set_stop(False, reason='line 3 back up', declared_by='shift_lead') + self.assertTrue(withdrawn['success']) + self.assertTrue(withdrawn['was_active']) + + self._wait_until( + lambda: all(code in self._faults()[1] for code in (MUTED_CODE, SECOND_CODE)), + 'the faults the stop muted never appeared in the default list after it ended') + self._wait_until( + lambda: self._count_confirmed(MUTED_CODE) == 1 + and self._count_confirmed(SECOND_CODE) == 1, + 'the stream did not carry exactly one confirmation per released fault') + + # Reaching one is not the claim: staying at one is. A duplicate frame + # published a moment later would satisfy the poll above and never be seen. + time.sleep(SETTLE_SEC) + self.assertEqual(1, self._count_confirmed(MUTED_CODE), + 'a second confirmation arrived after the first') + self.assertEqual(1, self._count_confirmed(SECOND_CODE), + 'a second confirmation arrived after the first') + + # The declaration stays readable after the plant is back up, with the time + # it ended, so "why was line 3 quiet?" is answerable over the same route. + state = self._get_stop() + self.assertFalse(state['active']) + self.assertEqual('line 3 maintenance', state['reason']) + self.assertEqual('shift_lead', state['declared_by']) + self.assertGreater(state['ended_at']['sec'], 0) + + def test_a_fault_raised_after_the_stop_is_announced_at_once(self): + """A fault raised once the stop is over is announced immediately. + + @verifies REQ_INTEROP_012 + """ + self._set_stop(True, reason='short pause', declared_by='tech') + self._set_stop(False, reason='pause over', declared_by='tech') + + self._report(AFTER_CODE) + + self._wait_until(lambda: self._count_confirmed(AFTER_CODE) == 1, + 'a fault raised after the stop ended was not announced') + time.sleep(SETTLE_SEC) + self.assertEqual(1, self._count_confirmed(AFTER_CODE)) + self.assertIn(AFTER_CODE, self._faults()[1]) + + def test_a_fault_already_up_when_the_stop_begins_stays_visible(self): + """A standing alarm is not hidden, and not announced twice. + + @verifies REQ_INTEROP_012 + """ + self._report(ALREADY_UP_CODE) + self._wait_until(lambda: self._count_confirmed(ALREADY_UP_CODE) == 1, + 'the fault was never announced before the stop') + self.assertIn(ALREADY_UP_CODE, self._faults()[1]) + + self._set_stop(True, reason='line 3 maintenance', declared_by='shift_lead') + + # The reporter keeps sending FAILED while the condition holds. + for _ in range(3): + self._report(ALREADY_UP_CODE) + + data, visible = self._faults(include_muted=True) + muted_codes = {entry['fault_code'] + for entry in data['x-medkit'].get('muted_faults', [])} + self.assertNotIn(ALREADY_UP_CODE, muted_codes, + 'the stop took over a cycle that started before it') + self.assertIn(ALREADY_UP_CODE, self._faults()[1], + 'a standing alarm vanished from GET /faults when the stop began') + + self._set_stop(False, reason='line 3 back up', declared_by='shift_lead') + + time.sleep(SETTLE_SEC) + self.assertEqual( + 1, self._count_confirmed(ALREADY_UP_CODE), + 'the switch-off announced a confirmation the operator had already seen') + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + """All processes exited cleanly (SIGTERM allowed for SSE teardown).""" + for info in proc_info: + self.assertIn( + info.returncode, ALLOWED_EXIT_CODES, + f'{info.process_name} exited with code {info.returncode}' + ) diff --git a/src/ros2_medkit_msgs/CMakeLists.txt b/src/ros2_medkit_msgs/CMakeLists.txt index 71ab7039c..a00535667 100644 --- a/src/ros2_medkit_msgs/CMakeLists.txt +++ b/src/ros2_medkit_msgs/CMakeLists.txt @@ -40,6 +40,8 @@ rosidl_generate_interfaces(${PROJECT_NAME} "srv/GetRosbag.srv" "srv/ListRosbags.srv" "srv/ListFaultsForEntity.srv" + "srv/SetPlannedStop.srv" + "srv/GetPlannedStop.srv" "srv/ListEntities.srv" "srv/GetEntityData.srv" "srv/GetCapabilities.srv" diff --git a/src/ros2_medkit_msgs/README.md b/src/ros2_medkit_msgs/README.md index fe8cf92a7..d78e98f9c 100644 --- a/src/ros2_medkit_msgs/README.md +++ b/src/ros2_medkit_msgs/README.md @@ -138,6 +138,64 @@ Clear/acknowledge a fault. Cleared faults are retained and queryable with `statu > **Note:** `skip_correlation_auto_clear` was added in `ros2_medkit_msgs` post-0.4.0. Adding a request field changes the service type hash, so out-of-tree callers that invoke `/fault_manager/clear_fault` directly (via `ros2 service call` or a generated client) must rebuild against the new `ros2_medkit_msgs` release to keep talking to `fault_manager`. +### SetPlannedStop.srv + +Declare, or withdraw, a planned stop on the FaultManager. + +**Request:** +| Field | Type | Description | +|-------|------|-------------| +| `active` | bool | `true` declares a planned stop, `false` withdraws it | +| `reason` | string | Why the plant is stopped; recorded in the audit log and served by `GetPlannedStop`, including after the withdrawal | +| `declared_by` | string | Who declared the transition; recorded as the audit record's source | + +**Response:** +| Field | Type | Description | +|-------|------|-------------| +| `success` | bool | True if the request was applied | +| `message` | string | Status or error message | +| `was_active` | bool | The state of the switch before this request | + +While a planned stop is on, it owns every fault cycle that *starts* - a new fault, one +raised again after being cleared, or one that fails again after healing. An owned fault is +reported, debounced, confirmed, captured and audited unchanged, and is marked as muted: +absent from the default `ListFaults` response, counted in `muted_count`, and listed under +`muted_faults` with `rule_id = "planned_stop"` when `include_muted` is set. A cycle that +started *before* the stop is untouched - reporters re-send FAILED for as long as a +condition holds, and that fault's confirmation has already been announced. + +Publication matches a rule-muted symptom exactly: `EVENT_CONFIRMED` and `EVENT_UPDATED` +are withheld whichever kind of report produced them, `EVENT_CLEARED` is published as +usual. Withdrawing the stop *unmutes* every fault it owns whose entry is the stop's own - +a fault a hierarchical rule has since claimed stays muted - and *announces* the subset of +those that is CONFIRMED and that no live cluster is hiding, one `EVENT_CONFIRMED` each. An +auto-cluster rule hides a non-representative member with no entry at all, by suppressing +that member's events on each report, so such a member is unmuted and not announced, and +what is announced is the representative. Afterwards the burst matches one that never met a +planned stop in the muted list, the counts, the cluster listing and the audit log, but not +in the event stream: a confirmation that fell inside the stop and behind a cluster is never +announced. A cluster hides only the reports that fall inside its `window_ms`; one after +that starts a new burst, and the fault is announced at the switch-off like any owned fault. +Cluster membership is not persisted, so a stop that spanned a restart releases every fault +the store says it owns and announces the CONFIRMED ones among them. A request asking for the state the switch is already in succeeds, +changes nothing, and writes no audit record; a request the store cannot record answers +`success: false` and changes nothing. Audit records exist only when `audit_log.enabled` is +set, which it is not by default. + +### GetPlannedStop.srv + +Read the planned-stop declaration, or - once it has been withdrawn - the last one there +was. The request carries no fields. + +**Response:** +| Field | Type | Description | +|-------|------|-------------| +| `active` | bool | Whether a planned stop is declared | +| `reason` | string | The reason given when it was declared; retained after the withdrawal | +| `declared_by` | string | Who declared it; retained after the withdrawal | +| `since` | builtin_interfaces/Time | When it was declared, wall clock | +| `ended_at` | builtin_interfaces/Time | When it was withdrawn; zero while one is in force | + ## Usage ### C++ diff --git a/src/ros2_medkit_msgs/srv/GetPlannedStop.srv b/src/ros2_medkit_msgs/srv/GetPlannedStop.srv new file mode 100644 index 000000000..ef8328aeb --- /dev/null +++ b/src/ros2_medkit_msgs/srv/GetPlannedStop.srv @@ -0,0 +1,44 @@ +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# GetPlannedStop.srv - Read the FaultManager's planned-stop declaration. +# +# The request carries no fields: the switch is a single declaration per fault manager. +# (rosidl gives an empty request one `structure_needs_at_least_one_member` byte, which +# is why introspecting clients see a single phantom parameter here.) +# +# The response describes the declaration in force, or - once it has been withdrawn - +# the last one there was, so "why was line 3 quiet on Friday?" is answerable after +# the plant is back up without reading the audit database. + +--- +# Response fields + +# Whether a planned stop is currently declared. +bool active + +# The reason given when the stop was declared, verbatim - nothing validates or +# fills it in, so it is empty when the declaration left it empty, and when no stop +# has ever been declared. Retained after the withdrawal. +string reason + +# Who declared the stop, on the same terms as `reason`. +string declared_by + +# When the stop was declared, wall clock. Zero while no stop has ever been declared. +builtin_interfaces/Time since + +# When the stop was withdrawn, wall clock. Zero while one is in force, and while +# no stop has ever been declared. +builtin_interfaces/Time ended_at diff --git a/src/ros2_medkit_msgs/srv/SetPlannedStop.srv b/src/ros2_medkit_msgs/srv/SetPlannedStop.srv new file mode 100644 index 000000000..d1bc30542 --- /dev/null +++ b/src/ros2_medkit_msgs/srv/SetPlannedStop.srv @@ -0,0 +1,89 @@ +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# SetPlannedStop.srv - Declare, or withdraw, a planned stop on the FaultManager. +# +# While a planned stop is on, it OWNS every fault cycle that STARTS - a new fault, one +# raised again after being cleared, or one that failed again after healing. An owned +# fault is reported, debounced, confirmed, captured and audited exactly as it would be +# otherwise, and is MARKED as muted rather than dropped: absent from the default +# ListFaults response, counted in `muted_count`, and listed under `muted_faults` when +# `include_muted` is set. A cycle that started BEFORE the stop is untouched: it stays in +# the fault list, because its confirmation has already been announced, and reporters +# re-send FAILED for as long as the condition holds. +# +# A muted fault is published exactly as a rule-muted symptom is: its EVENT_CONFIRMED and +# EVENT_UPDATED are withheld, whichever kind of report produced them, while an +# EVENT_CLEARED - the fault healing, or being acknowledged - is published as usual. +# Withdrawing the stop does two things that do not cover the same faults. It UNMUTES +# every fault it owns whose `muted_faults` entry is the stop's own, so a fault a +# hierarchical rule has since claimed stays muted and everything else leaves +# `muted_count`. It ANNOUNCES the subset of those that is CONFIRMED and that no live +# cluster is hiding, one EVENT_CONFIRMED each, because that confirmation was never +# announced. +# +# A hierarchical rule muting an owned fault OVERLAYS the stop rather than taking the +# fault from it: while the rule holds the fault the withdrawal leaves it alone, and when +# its root cause is acknowledged the fault is the stop's again. +# +# An auto-cluster rule hides without an overlay - it suppresses a non-representative +# member's events on each report and writes no `muted_faults` entry - and the withdrawal +# honours that too: a member an ACTIVE `show_as_single` cluster still hides is unmuted +# and NOT announced, with no entry left in the stop's place. Afterwards the burst matches +# one that never met a planned stop in `muted_faults`, in the counts, in the cluster +# listing and in the audit log, but NOT in the event stream: a confirmation that fell +# inside the stop and behind a cluster is never announced. The member announced is the +# representative, when the stop owned its cycle. `min_count` decides whether a cluster +# forms, not how long it hides: a formed cluster holds the members it formed with for as +# long as its `window_ms` covers their reports, until the last is acknowledged, while a +# fault that joins it below the threshold is not one of them and is announced like any +# fault of its own. A report that arrives after the window starts a new burst, so that +# fault is no longer folded into the old line and the switch-off announces it like any +# owned fault - the same thing the cluster does with no stop in force, where that late +# report publishes an update. Cluster membership is not persisted, so a stop that spanned +# a restart releases every fault the store says it owns and announces the CONFIRMED ones +# among them. +# +# With `storage_type: sqlite` the declaration survives a restart of the fault manager, +# and so does its ownership of the cycles that started inside it; the in-memory backend +# keeps neither past the process. +# +# Both transitions are recorded in the fault audit log when `audit_log.enabled` is set +# (it is off by default). + +# Request fields + +# True declares a planned stop, false withdraws it. +bool active + +# Why the plant is stopped ("line 3 quarterly maintenance"). Recorded in the audit log +# with the transition, and served back by GetPlannedStop - including after the stop has +# been withdrawn, so the reason outlives it. +string reason + +# Who declared the transition. Recorded in the audit log as the transition's source. +string declared_by +--- +# Response fields + +# True when the request was applied. A request that asks for the state the switch is +# already in succeeds and changes nothing (no audit record is written for it). +bool success + +# Status message or error description. +string message + +# The state of the switch BEFORE this request, so a caller can tell "I turned it on" +# from "it was already on". +bool was_active