fault_manager: a planned-stop switch, so a weekend is not a wave of faults

docs/api/messages.rst

@@ -294,6 +294,48 @@ Retrieve diagnostic snapshots captured at fault occurrence time.
 
 See :doc:`/tutorials/snapshots` for detailed usage.
 
+SetPlannedStop.srv / GetPlannedStop.srv
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Declare, withdraw and read a planned stop on the fault manager.
+
+.. code-block:: text
+
+   # SetPlannedStop.srv
+   bool active          # true declares a planned stop, false withdraws it
+   string reason        # why the plant is stopped
+   string declared_by   # who declared the transition
+   ---
+   bool success         # true when the request was applied
+   string message       # status or error description
+   bool was_active      # the state of the switch before this request
+
+   # GetPlannedStop.srv
+   ---
+   bool active                       # whether a planned stop is declared
+   string reason                     # the reason given; retained after the withdrawal
+   string declared_by                # who declared it; retained after the withdrawal
+   builtin_interfaces/Time since     # when it was declared
+   builtin_interfaces/Time ended_at  # when it was withdrawn; zero while one is in force
+
+While a planned stop is on, it owns every fault cycle that *starts* - a new fault,
+one raised again after being cleared, or one that fails again after healing. An
+owned fault is reported, debounced, confirmed, captured and audited unchanged, and
+is marked as muted: absent from the default ``ListFaults`` response, counted in
+``muted_count``, and listed under ``muted_faults`` with ``rule_id: planned_stop``
+when ``include_muted`` is set. A cycle that started before the stop is untouched.
+
+Publication matches a rule-muted symptom exactly: ``EVENT_CONFIRMED`` and
+``EVENT_UPDATED`` are withheld whichever kind of report produced them,
+``EVENT_CLEARED`` is published as usual. Withdrawing the stop releases every fault
+it owns and publishes one ``EVENT_CONFIRMED`` per released fault that is CONFIRMED;
+a fault a correlation rule is muting stays muted and is not announced, and returns
+to the stop's mute if the rule lets go while the stop is still on. The audit records
+exist only when ``audit_log.enabled`` is set, which it is not by default.
+
+See :doc:`/config/fault-manager` for the configuration that decides whether the
+declaration survives a restart and whether the transitions are audited.
+
 MedkitDiscoveryHint.msg
 ~~~~~~~~~~~~~~~~~~~~~~~
 

docs/config/fault-manager.rst

@@ -608,12 +608,91 @@ Fault correlation identifies root causes and filters symptom faults.
      - Path to YAML file defining correlation rules.
    * - ``correlation.cleanup_interval_sec``
      - ``5.0``
-     - Interval for running correlation cleanup tasks.
+     - Interval for running correlation cleanup tasks. The correlation engine is
+       constructed on every fault manager, configured rules or not, because the
+       planned-stop switch needs none - so this timer runs everywhere. With no
+       rules loaded a tick takes the engine's mutex, walks two empty containers and
+       re-asserts an empty ownership set: measured at ~190 ns per call, so ~2.3 us
+       of CPU per minute at the 5 s default.
 
 .. seealso::
 
    :doc:`/tutorials/fault-correlation` for correlation rule syntax and examples.
 
+Planned Stop
+------------
+
+A planned stop is declared at runtime, not configured: an operator calls
+``~/set_planned_stop`` (or the matching SOVD operation on the ``fault_manager``
+entity) with a reason and a declarer, and calls it again with ``active: false``
+when the plant is back up. There is no parameter for it - a stop has a reason and
+an audit trail, and a parameter has neither.
+
+.. code-block:: bash
+
+   ros2 service call /fault_manager/set_planned_stop ros2_medkit_msgs/srv/SetPlannedStop \
+     "{active: true, reason: 'line 3 quarterly maintenance', declared_by: 'shift_lead'}"
+
+Two configuration choices decide how much the switch can do:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 35 15 50
+
+   * - Parameter
+     - Default
+     - Effect on the planned stop
+   * - ``storage_type``
+     - ``sqlite``
+     - The declaration, and the per-fault ownership flags behind it, are stored
+       beside the faults, so both survive a restart. With ``memory`` they live only
+       for the process, and a restart inside a stop ends it. A correlation rule's
+       mute is not persisted on either backend, so a rule-muted fault comes back
+       unmuted after a restart unless the stop owns it.
+   * - ``audit_log.enabled``
+     - ``false``
+     - **Off by default, so a stock manager records no audit row for the switch
+       at all.** When on, each transition is recorded as ``planned_stop_started``
+       / ``planned_stop_ended`` under the ``__audit__`` fault code, with that
+       transition's own reason and declarer. Recorded even under
+       ``audit_log.transitions: confirmed_only``.
+
+While the stop stands, it owns every fault cycle that *starts*: a new fault, one
+raised again after being cleared, or one that fails again after healing. An owned
+fault is reported, debounced, confirmed, captured and audited unchanged, and is
+marked as muted rather than dropped: absent from the default fault list, counted in
+``muted_count``, and listed under ``muted_faults`` with ``rule_id: planned_stop``
+when muted entries are requested. A fault that was already up when the stop was
+declared is left alone - its confirmation has already been announced, and reporters
+re-send FAILED for as long as the condition holds.
+
+Ownership is a flag on the fault row, so it survives a restart and is what the
+switch-off releases. The mute is derived from it: a correlation rule muting an owned
+fault overlays the stop rather than taking the fault from it, the withdrawal leaves
+a rule-held fault alone, and when the rule lets go - root cause acknowledged, window
+closed, cluster expired - the fault is muted by the stop again.
+
+A muted fault is published exactly as a rule-muted symptom is: ``EVENT_CONFIRMED``
+and ``EVENT_UPDATED`` are withheld whichever kind of report produced them, while
+``EVENT_CLEARED`` - the fault healing, or being acknowledged - is published as
+usual. Withdrawing the stop releases every fault it owns and publishes one
+confirmation event for each of those that is CONFIRMED.
+
+``~/get_planned_stop`` keeps serving the declaration after the withdrawal, with
+``ended_at`` stamped, so the reason stays readable once the plant is back up. A
+request the store cannot record answers ``success: false`` and changes nothing.
+
+Over HTTP the switch is an operation on the fault manager's App entity
+(``POST /apps/fault_manager/operations/set_planned_stop/executions``), which
+exists automatically in ``runtime_only`` discovery. Under ``hybrid`` or
+``manifest_only`` the fault manager must be declared in the manifest, or the
+entity is not there to address.
+
+.. seealso::
+
+   :doc:`/design/ros2_medkit_fault_manager/index` for how the switch composes
+   with correlation rules.
+
 Complete Example
 ----------------
 

src/ros2_medkit_fault_manager/CMakeLists.txt

@@ -208,6 +208,29 @@ if(BUILD_TESTING)
   medkit_add_launch_test(test_rosbag_entity_scope test/integration/test_rosbag_entity_scope.test.py TIMEOUT 120
     LABELS "integration")
 
+  # The planned-stop switch end to end: marking, counting, auditing, capture, and
+  # the release at switch-off. One case raises 200 faults inside a stop, so the
+  # budget is well above the 60s default.
+  medkit_add_launch_test(test_planned_stop test/integration/test_planned_stop.test.py TIMEOUT 240
+    LABELS "integration")
+
+  # Kills the manager and lets launch bring it back, so the declaration is read
+  # from the store by a process that never saw the request that made it.
+  medkit_add_launch_test(test_planned_stop_restart test/integration/test_planned_stop_restart.test.py
+    TIMEOUT 180 LABELS "integration")
+
+  # A switch-off killed between the declaration write and the flag clearing: the
+  # store is left as that crash leaves it and the replacement process has to
+  # finish the release. Its respawn delay is what the timeout is sized for.
+  medkit_add_launch_test(test_planned_stop_interrupted_release
+    test/integration/test_planned_stop_interrupted_release.test.py TIMEOUT 180 LABELS "integration")
+
+  # auto_confirm_after_sec confirms from a timer rather than a report, so the
+  # mute has to be honoured on that path too. Its own launch, because every other
+  # planned-stop case would then race a timer that confirms behind its back.
+  medkit_add_launch_test(test_planned_stop_auto_confirm
+    test/integration/test_planned_stop_auto_confirm.test.py TIMEOUT 120 LABELS "integration")
+
   # The only suite running above max_bags_per_fault=1: drives confirm / clear /
   # confirm on one code and asserts both recordings survive and stay separately
   # addressable. Six occurrences plus their post-roll windows, hence the timeout.

src/ros2_medkit_fault_manager/README.md

@@ -43,6 +43,115 @@ ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
 | `~/list_faults` | `ros2_medkit_msgs/srv/ListFaults` | Query faults with filtering |
 | `~/clear_fault` | `ros2_medkit_msgs/srv/ClearFault` | Clear/acknowledge a fault |
 | `~/get_snapshots` | `ros2_medkit_msgs/srv/GetSnapshots` | Get topic snapshots for a fault |
+| `~/set_planned_stop` | `ros2_medkit_msgs/srv/SetPlannedStop` | Declare or withdraw a planned stop |
+| `~/get_planned_stop` | `ros2_medkit_msgs/srv/GetPlannedStop` | Read the planned-stop declaration |
+
+## Planned Stop
+
+Maintenance produces faults that nobody wants paged: a robot on a bench raises
+`NODE_UNREACHABLE` for every peer it can no longer see, and the alarm reaches
+whoever is on call. The planned stop is the operator saying "this is us, not the
+plant".
+
+```bash
+ros2 service call /fault_manager/set_planned_stop ros2_medkit_msgs/srv/SetPlannedStop \
+  "{active: true, reason: 'line 3 quarterly maintenance', declared_by: 'shift_lead'}"
+
+ros2 service call /fault_manager/get_planned_stop ros2_medkit_msgs/srv/GetPlannedStop "{}"
+```
+
+**Marked, never dropped.** While the stop stands, it *owns* every fault cycle that
+starts. An owned fault goes through report, debounce, confirmation, snapshot and
+rosbag capture and the audit log exactly as it would otherwise. What changes is
+only how it is *shown*: it is registered as muted with `rule_id: planned_stop` and
+the pseudo root cause `PLANNED_STOP`, so it is absent from the default
+`~/list_faults` response, counted in `muted_count`, and listed under
+`muted_faults` when `include_muted` is set.
+
+**Ownership is the truth, and it is persisted.** The flag lives on the fault row in
+the store, not in the process, so a restart reads back exactly which cycles the
+stop owns - no timestamp comparison, which a clock step would break and which
+cannot tell a rule's mute from the stop's. The mute is derived from ownership: an
+owned fault is muted unless a rule's mute overlays it, and when that overlay ends
+the fault is muted by the stop again.
+
+**Only a cycle that starts inside the stop.** A fault that was already up when the
+stop was declared keeps its place in the fault list. Reporters are level-triggered
+- `FaultReporter::report()` is called for as long as the condition holds, not once
+per transition - so marking on any report would take a standing alarm off the list
+and then announce it a second time at the switch-off. A cycle starts on a new
+fault, on one raised again after being cleared, and on one that fails again after
+healing (the heal published the fault's end, so the next confirmation is fresh
+news). `occurrence_count` keeps its own, narrower definition and counts the first
+two only.
+
+**Published exactly like a rule-muted symptom.** `EVENT_CONFIRMED` and
+`EVENT_UPDATED` are withheld while the fault is muted, whichever kind of report
+produced them - a PASSED that leaves the fault CONFIRMED announces nothing either.
+`EVENT_CLEARED` is not withheld: a fault that heals past the healing threshold, or
+that is acknowledged, publishes it as any muted fault does. Consumers therefore see
+the end of a fault whose start they never heard - that is the existing muting
+contract, not something the switch changes.
+
+**Withdrawing releases the survivors.** Every fault the stop alone was muting and
+that is still active is unmuted, and each one that is CONFIRMED publishes a single
+`EVENT_CONFIRMED` - that confirmation happened behind the mute and was never
+announced, while the condition it reports still stands on the machine. A fault
+still short of confirmation announces nothing (there is nothing yet to announce),
+and one acknowledged during the stop announces nothing either.
+
+**Correlation rules and the stop compose.** A rule's mute is an *overlay* on an
+owned fault, not a transfer: while the rule holds it, the withdrawal leaves it
+alone, and when the rule lets go - its root cause acknowledged, its window closed,
+its cluster expired - the fault goes back to being muted by the stop. A fault whose
+cycle started before the stop is not owned at all, so the stop neither hides it nor
+releases it. This holds for cluster rules as well as hierarchical ones, even though
+the cluster path never writes a mute entry of its own.
+
+**Both transitions are audited** - when the audit log is on. `audit_log.enabled`
+is `false` by default, and with it off the switch writes no audit row at all. With
+it on, each transition appends `planned_stop_started` / `planned_stop_ended`
+carrying that transition's own reason in `description` and its declarer in
+`source_id`, under the `__audit__` fault code the log's own lifecycle markers use,
+because the transition is about the installation rather than one fault. They are
+recorded whatever `audit_log.transitions` says. A request asking for the state the
+switch is already in succeeds, changes nothing and records nothing, so a retried
+call cannot manufacture evidence of a stop that never started. A request the store
+cannot record answers `success: false` with the reason, and changes nothing.
+
+**The declaration outlives the stop.** `~/get_planned_stop` keeps serving the
+reason, the declarer and the start time after the withdrawal, with `ended_at`
+stamped, so "why was line 3 quiet on Friday?" is answerable once the plant is back
+up without reading the audit database.
+
+**The declaration survives a restart** when the SQLite backend is in use: it is a
+row in the fault store, so a stop declared on Friday still mutes on Monday. On
+startup the manager reads the ownership flags back, so the switch-off after a
+reboot releases exactly the cycles the stop owned and announces their
+confirmations. A *rule's* mute is not persisted anywhere, so a fault a correlation
+rule was muting before a restart comes back unmuted unless the stop owns it - a
+pre-existing property of correlation, not of the switch. The in-memory backend has
+no file behind it and starts every process with no declaration and no ownership.
+
+**A switch-off interrupted by a crash is finished at startup.** The withdrawal
+writes the declaration first, announces the confirmations it was holding back, and
+drops the ownership flags last. A process that dies in between leaves faults owned
+by a declaration that is already over; the next startup recognises exactly that,
+announces them once a consumer is listening, and clears the flags.
+
+**Over SOVD.** The gateway maps a node's services to operations on its App entity,
+so an operator reaches the switch without any new route:
+
+```bash
+curl -X POST http://localhost:8080/api/v1/apps/fault_manager/operations/set_planned_stop/executions \
+  -H 'Content-Type: application/json' \
+  -d '{"parameters": {"active": true, "reason": "line 3 maintenance", "declared_by": "shift_lead"}}'
+```
+
+That works out of the box in `runtime_only` discovery, where every node is an
+App. In `hybrid` or `manifest_only` discovery the fault manager has to be
+declared in the manifest like any other entity, or there is no `fault_manager`
+entity to address.
 
 ## Features
 
@@ -55,7 +164,8 @@ ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
 - **Snapshot capture**: Captures topic data when faults are confirmed for debugging (snapshots are deleted when fault is cleared)
 - **Near-miss series**: Appends one entry per FAILED report that moved the debounce counter without confirming, bounded per fault code and retained when the fault is cleared
 - **Freeze-frame retention**: One compact JSON freeze-frame per fault code, retained across `clear_fault` (see below)
-- **Fault correlation** (optional): Root cause analysis with symptom muting and auto-clear
+- **Planned stop**: An operator declares the plant deliberately down, and faults raised while it stands are marked rather than announced (see below)
+- **Fault correlation** (optional): Root cause analysis with symptom muting and auto-clear. The correlation *engine* is always constructed, because the planned stop needs no configuration; without a `correlation.config_file` it carries no rules, correlates nothing, and its cleanup timer (`correlation.cleanup_interval_sec`, default 5 s) walks two empty containers
 - **Tamper-evident audit log** (optional): Append-only, hash-chained record of fault state transitions for verifiable history
 
 ## Parameters