From 09b602239edaf2f937ea506cdfff9c045d6d9610 Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Mon, 7 Sep 2026 16:16:00 +0200 Subject: [PATCH 01/12] msgs: add the planned-stop switch service pair SetPlannedStop declares or withdraws a planned stop and answers with the state the switch was in before the call; GetPlannedStop reads back the declaration, its reason, its declarer and when it started. Refs #656 --- src/ros2_medkit_msgs/CMakeLists.txt | 2 + src/ros2_medkit_msgs/README.md | 38 ++++++++++++++++ src/ros2_medkit_msgs/srv/GetPlannedStop.srv | 32 +++++++++++++ src/ros2_medkit_msgs/srv/SetPlannedStop.srv | 50 +++++++++++++++++++++ 4 files changed, 122 insertions(+) create mode 100644 src/ros2_medkit_msgs/srv/GetPlannedStop.srv create mode 100644 src/ros2_medkit_msgs/srv/SetPlannedStop.srv diff --git a/src/ros2_medkit_msgs/CMakeLists.txt b/src/ros2_medkit_msgs/CMakeLists.txt index 71ab7039c..a00535667 100644 --- a/src/ros2_medkit_msgs/CMakeLists.txt +++ b/src/ros2_medkit_msgs/CMakeLists.txt @@ -40,6 +40,8 @@ rosidl_generate_interfaces(${PROJECT_NAME} "srv/GetRosbag.srv" "srv/ListRosbags.srv" "srv/ListFaultsForEntity.srv" + "srv/SetPlannedStop.srv" + "srv/GetPlannedStop.srv" "srv/ListEntities.srv" "srv/GetEntityData.srv" "srv/GetCapabilities.srv" diff --git a/src/ros2_medkit_msgs/README.md b/src/ros2_medkit_msgs/README.md index fe8cf92a7..01ac6d604 100644 --- a/src/ros2_medkit_msgs/README.md +++ b/src/ros2_medkit_msgs/README.md @@ -138,6 +138,44 @@ Clear/acknowledge a fault. Cleared faults are retained and queryable with `statu > **Note:** `skip_correlation_auto_clear` was added in `ros2_medkit_msgs` post-0.4.0. Adding a request field changes the service type hash, so out-of-tree callers that invoke `/fault_manager/clear_fault` directly (via `ros2 service call` or a generated client) must rebuild against the new `ros2_medkit_msgs` release to keep talking to `fault_manager`. +### SetPlannedStop.srv + +Declare, or withdraw, a planned stop on the FaultManager. + +**Request:** +| Field | Type | Description | +|-------|------|-------------| +| `active` | bool | `true` declares a planned stop, `false` withdraws it | +| `reason` | string | Why the plant is stopped; recorded in the audit log and served by `GetPlannedStop` | +| `declared_by` | string | Who declared the transition; recorded as the audit record's source | + +**Response:** +| Field | Type | Description | +|-------|------|-------------| +| `success` | bool | True if the request was applied | +| `message` | string | Status or error message | +| `was_active` | bool | The state of the switch before this request | + +While a planned stop is on, a fault whose cycle starts is reported, debounced, confirmed, +captured and audited unchanged, and is marked as muted: absent from the default +`ListFaults` response, counted in `muted_count`, listed under `muted_faults` with +`rule_id = "planned_stop"` when `include_muted` is set, and never published as a +`FaultEvent`. Withdrawing the stop unmutes every fault it muted that is still active and +publishes one `EVENT_CONFIRMED` for each of those that is CONFIRMED. A request asking for +the state the switch is already in succeeds, changes nothing, and writes no audit record. + +### GetPlannedStop.srv + +Read the planned-stop declaration. The request carries no fields. + +**Response:** +| Field | Type | Description | +|-------|------|-------------| +| `active` | bool | Whether a planned stop is declared | +| `reason` | string | The reason given when it was declared; empty while none is | +| `declared_by` | string | Who declared it; empty while none is declared | +| `since` | builtin_interfaces/Time | When it was declared, wall clock; zero while none is | + ## Usage ### C++ diff --git a/src/ros2_medkit_msgs/srv/GetPlannedStop.srv b/src/ros2_medkit_msgs/srv/GetPlannedStop.srv new file mode 100644 index 000000000..ad752cf07 --- /dev/null +++ b/src/ros2_medkit_msgs/srv/GetPlannedStop.srv @@ -0,0 +1,32 @@ +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# GetPlannedStop.srv - Read the FaultManager's planned-stop declaration. +# +# The request carries no fields: the switch is a single declaration per fault manager. + +--- +# Response fields + +# Whether a planned stop is currently declared. +bool active + +# The reason given when the stop was declared. Empty while no stop is declared. +string reason + +# Who declared the stop. Empty while no stop is declared. +string declared_by + +# When the stop was declared, wall clock. Zero while no stop is declared. +builtin_interfaces/Time since diff --git a/src/ros2_medkit_msgs/srv/SetPlannedStop.srv b/src/ros2_medkit_msgs/srv/SetPlannedStop.srv new file mode 100644 index 000000000..0eea40cb5 --- /dev/null +++ b/src/ros2_medkit_msgs/srv/SetPlannedStop.srv @@ -0,0 +1,50 @@ +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# SetPlannedStop.srv - Declare, or withdraw, a planned stop on the FaultManager. +# +# While a planned stop is on, every fault whose cycle starts is reported, debounced, +# confirmed, captured and audited exactly as it would be otherwise, and is MARKED as +# muted rather than dropped: it is absent from the default ListFaults response, counted +# in `muted_count`, listed under `muted_faults` when `include_muted` is set, and no +# FaultEvent is published for it. Withdrawing the stop unmutes every fault it muted that +# is still active, and republishes an EVENT_CONFIRMED for each one that is CONFIRMED, +# because that confirmation was never announced. +# +# The declaration survives a restart of the fault manager. + +# Request fields + +# True declares a planned stop, false withdraws it. +bool active + +# Why the plant is stopped ("line 3 quarterly maintenance"). Recorded in the audit log +# with the transition and served back by GetPlannedStop while the stop is on. +string reason + +# Who declared the transition. Recorded in the audit log as the transition's source. +string declared_by +--- +# Response fields + +# True when the request was applied. A request that asks for the state the switch is +# already in succeeds and changes nothing (no audit record is written for it). +bool success + +# Status message or error description. +string message + +# The state of the switch BEFORE this request, so a caller can tell "I turned it on" +# from "it was already on". +bool was_active From c7c9895e50a3d883679ea2cc8d48dffe2dece9df Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Mon, 7 Sep 2026 16:16:14 +0200 Subject: [PATCH 02/12] fault_manager: mark faults raised during a planned stop instead of dropping them The correlation engine gains a second mute source. While a planned stop is declared, a fault whose cycle starts is reported, debounced, confirmed, captured and audited exactly as it would be otherwise, and is registered as muted with rule_id 'planned_stop' and the pseudo root cause 'PLANNED_STOP': absent from the default fault list, counted in muted_count, and never announced on the event stream. A rule that mutes the same code takes the mute over, and withdrawing the stop then leaves it alone. Withdrawing releases every fault the stop alone was holding down and publishes one EVENT_CONFIRMED for each of those that is CONFIRMED, because that confirmation was never announced. Both transitions are recorded in the audit log with their reason and declarer, whatever audit_log.transitions says; a request for the state the switch is already in changes nothing and records nothing. The declaration is stored, so a stop declared before a restart is still in force after it. The correlation engine is now always constructed, with no rules when none are configured, because the switch needs no configuration. Refs #656 --- .../correlation/correlation_engine.hpp | 41 ++++ .../fault_audit_log.hpp | 7 + .../fault_manager_node.hpp | 46 ++++- .../fault_storage.hpp | 28 +++ .../sqlite_fault_storage.hpp | 3 + .../src/correlation/correlation_engine.cpp | 55 ++++++ .../src/fault_manager_node.cpp | 175 ++++++++++++++--- .../src/fault_storage.cpp | 10 + .../src/sqlite_fault_storage.cpp | 52 +++++ .../test/test_correlation_engine.cpp | 177 ++++++++++++++++++ .../test/test_sqlite_storage.cpp | 115 ++++++++++++ 11 files changed, 681 insertions(+), 28 deletions(-) diff --git a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/correlation/correlation_engine.hpp b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/correlation/correlation_engine.hpp index f1092138a..fac91c110 100644 --- a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/correlation/correlation_engine.hpp +++ b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/correlation/correlation_engine.hpp @@ -19,6 +19,7 @@ #include #include #include +#include #include #include @@ -87,10 +88,22 @@ struct ClusterData { /// - Whether they should be muted (symptoms of a root cause) /// - Whether they are root causes that should collect symptoms /// - Whether they form part of an auto-detected cluster +/// - Whether a planned stop is in force, which mutes every fault whose cycle +/// starts while it is on /// /// Thread-safe: all public methods can be called from multiple threads. class CorrelationEngine { public: + /// Pseudo root cause recorded against a fault the planned stop muted. It is + /// not a fault code any reporter can raise; it names the declaration as the + /// reason the fault is not being announced. + static constexpr const char * kPlannedStopRootCause = "PLANNED_STOP"; + + /// Rule id recorded against a fault the planned stop muted. The switch is not + /// a configured rule, so it carries a fixed id of its own and a consumer of + /// `muted_faults` can tell an operator's declaration from a correlation rule. + static constexpr const char * kPlannedStopRuleId = "planned_stop"; + /// Create correlation engine from configuration /// @param config Correlation configuration (must be enabled and valid) explicit CorrelationEngine(const CorrelationConfig & config); @@ -131,7 +144,27 @@ class CorrelationEngine { /// Called periodically to remove old state void cleanup_expired(); + /// Declare a planned stop. From here every fault reported through + /// process_fault is muted unless a rule already mutes it, and stays muted + /// until the stop ends or the fault is cleared. Idempotent. + void begin_planned_stop(); + + /// Withdraw the planned stop and release the faults it alone was muting. + /// A fault a rule has since claimed keeps that rule's mute and is NOT + /// returned; nor is one that was cleared while the stop was on. + /// @return the fault codes this call unmuted, so the caller can announce the + /// confirmations that were never published + std::vector end_planned_stop(); + + /// Whether a planned stop is currently declared. + bool planned_stop_active() const; + private: + /// The correlation half of process_fault: rules, clusters and their muting, + /// with no planned-stop involvement. Caller holds mutex_. + ProcessFaultResult correlate(const std::string & fault_code, const std::string & severity, + std::chrono::steady_clock::time_point timestamp); + /// Check if fault matches a root cause pattern in any hierarchical rule /// @return Rule ID if matched, empty optional otherwise std::optional try_as_root_cause(const std::string & fault_code); @@ -187,6 +220,14 @@ class CorrelationEngine { /// Counter for cluster ID generation uint64_t cluster_counter_{0}; + /// Whether an operator has declared a planned stop. + bool planned_stop_active_{false}; + + /// Fault codes muted BY the planned stop and by nothing else. A code leaves + /// this set the moment a rule claims it or it is cleared, which is what makes + /// the switch-off release exactly the faults it is still holding down. + std::set planned_stop_muted_; + mutable std::mutex mutex_; }; diff --git a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_audit_log.hpp b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_audit_log.hpp index 967090690..cea41c8ad 100644 --- a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_audit_log.hpp +++ b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_audit_log.hpp @@ -67,6 +67,13 @@ constexpr const char * kTransitionHealed = "healed"; /// its own start and stop. constexpr const char * kTransitionLoggingActivated = "logging_activated"; constexpr const char * kTransitionLoggingDeactivated = "logging_deactivated"; +/// Planned-stop lifecycle: an operator declared, or withdrew, a deliberate stop +/// of the plant. Like the logging markers these describe the installation rather +/// than one fault, so they carry an empty fault_code and are appended +/// independently of the per-fault transition filter. The reason travels in +/// `description` and the declarer in `source_id`. +constexpr const char * kTransitionPlannedStopStarted = "planned_stop_started"; +constexpr const char * kTransitionPlannedStopEnded = "planned_stop_ended"; // NOTE: there is deliberately no "ack" kind. The open fault_manager has no // acknowledge action separate from clearing: ~/clear_fault IS the acknowledge, // and it is recorded as kTransitionCleared (clear == ack). A separate "ack" kind diff --git a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_manager_node.hpp b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_manager_node.hpp index 9157102d9..290174059 100644 --- a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_manager_node.hpp +++ b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_manager_node.hpp @@ -18,6 +18,7 @@ #include #include #include +#include #include #include @@ -32,12 +33,14 @@ #include "ros2_medkit_msgs/msg/fault_event.hpp" #include "ros2_medkit_msgs/srv/clear_fault.hpp" #include "ros2_medkit_msgs/srv/get_fault.hpp" +#include "ros2_medkit_msgs/srv/get_planned_stop.hpp" #include "ros2_medkit_msgs/srv/get_rosbag.hpp" #include "ros2_medkit_msgs/srv/get_snapshots.hpp" #include "ros2_medkit_msgs/srv/list_faults.hpp" #include "ros2_medkit_msgs/srv/list_faults_for_entity.hpp" #include "ros2_medkit_msgs/srv/list_rosbags.hpp" #include "ros2_medkit_msgs/srv/report_fault.hpp" +#include "ros2_medkit_msgs/srv/set_planned_stop.hpp" namespace ros2_medkit_fault_manager { @@ -151,6 +154,14 @@ class FaultManagerNode : public rclcpp::Node { void handle_list_rosbags(const std::shared_ptr & request, const std::shared_ptr & response); + /// Handle SetPlannedStop service request + void handle_set_planned_stop(const std::shared_ptr & request, + const std::shared_ptr & response); + + /// Handle GetPlannedStop service request + void handle_get_planned_stop(const std::shared_ptr & request, + const std::shared_ptr & response); + /// Handle ListFaultsForEntity service request void handle_list_faults_for_entity(const std::shared_ptr & request, @@ -164,10 +175,19 @@ class FaultManagerNode : public rclcpp::Node { /// @param config SnapshotConfig to populate with loaded values void load_snapshot_config_from_yaml(const std::string & config_file, SnapshotConfig & config); - /// Initialize correlation engine from configuration file - /// @return CorrelationEngine instance if enabled and config is valid, nullptr otherwise + /// Build the correlation engine. + /// + /// Never null. The engine also owns the planned-stop mute, which an operator can + /// declare on any fault manager, so it exists whether or not correlation rules are + /// configured; without a config file (or with one that fails to load) it simply + /// carries no rules and correlates nothing. std::unique_ptr create_correlation_engine(); + /// Parse the correlation config file named by `correlation.config_file`. + /// @return the parsed config, or nullopt when correlation is not configured, + /// explicitly disabled, or the file is missing or invalid + std::optional load_correlation_config(); + /// Publish a fault event to the events topic /// @param event_type One of FaultEvent::EVENT_CONFIRMED, EVENT_CLEARED, EVENT_UPDATED /// @param fault The fault data associated with this event @@ -203,6 +223,18 @@ class FaultManagerNode : public rclcpp::Node { void audit_transition(const char * transition, const ros2_medkit_msgs::msg::Fault & fault, const std::string & source_id, int64_t occurred_at_ns); + /// Append one already-built record to the audit log, carrying the shared + /// failure handling (health signals, fail-closed rethrow). No-op when the log + /// is off. Bypasses the per-fault transition filter, so a caller that has + /// decided a record must be written gets it written. + void append_audit_event(const AuditEvent & event); + + /// Record a planned-stop transition. Written whatever `audit_log.transitions` + /// says, for the same reason the logging markers are: the switch describes the + /// installation, and the evidence chain has to show when the plant was + /// declared stopped. + void audit_planned_stop(const char * transition, const PlannedStopState & state, int64_t occurred_at_ns); + std::string storage_type_; std::string database_path_; int32_t confirmation_threshold_{-1}; @@ -237,6 +269,8 @@ class FaultManagerNode : public rclcpp::Node { rclcpp::Service::SharedPtr get_rosbag_srv_; rclcpp::Service::SharedPtr list_rosbags_srv_; rclcpp::Service::SharedPtr list_faults_for_entity_srv_; + rclcpp::Service::SharedPtr set_planned_stop_srv_; + rclcpp::Service::SharedPtr get_planned_stop_srv_; rclcpp::TimerBase::SharedPtr auto_confirm_timer_; /// Timer for periodic cleanup of expired correlation data @@ -253,9 +287,15 @@ class FaultManagerNode : public rclcpp::Node { /// shared_ptr to allow safe capture-by-value in the pool's capture jobs. std::shared_ptr rosbag_capture_; - /// Correlation engine for fault correlation/muting (nullptr if disabled) + /// Correlation engine for fault correlation/muting. Always present: it also + /// holds the planned-stop mute, which needs no configuration. std::unique_ptr correlation_engine_; + /// The planned-stop declaration in force, mirrored from the store at startup and + /// written back on every transition. Only the service handlers touch it, and they + /// run on the node's single-threaded executor, so it needs no lock of its own. + PlannedStopState planned_stop_; + /// Bounded pool that runs capture jobs off the service thread (issue #441). /// Declared after snapshot_capture_/rosbag_capture_ so it is destroyed first. std::unique_ptr capture_pool_; diff --git a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_storage.hpp b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_storage.hpp index 3bc60dc35..b8033a2dd 100644 --- a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_storage.hpp +++ b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_storage.hpp @@ -195,6 +195,18 @@ struct RosbagFileInfo { int64_t created_at_ns{0}; ///< Timestamp when bag was created }; +/// The operator's planned-stop declaration, as the store holds it. +/// +/// One declaration per fault manager, not one per fault: it says the plant is +/// deliberately down, which is a fact about the installation. It is persisted so +/// a stop declared on Friday is still in force after a Saturday reboot. +struct PlannedStopState { + bool active{false}; + std::string reason; ///< Why the plant is stopped; empty while none is declared + std::string declared_by; ///< Who declared it; empty while none is declared + int64_t since_ns{0}; ///< Wall-clock time of the declaration; 0 while none is declared +}; + /// Abstract interface for fault storage backends class FaultStorage { public: @@ -454,6 +466,15 @@ class FaultStorage { return {}; } + /// Replace the planned-stop declaration. Writing a default-constructed state + /// withdraws it. A backend keeps exactly one declaration, so this overwrites + /// rather than appends. + virtual void set_planned_stop(const PlannedStopState & state) = 0; + + /// The planned-stop declaration this store holds. A store that has never been + /// given one answers with a default-constructed (inactive) state. + virtual PlannedStopState get_planned_stop() const = 0; + protected: FaultStorage() = default; FaultStorage(const FaultStorage &) = default; @@ -520,6 +541,9 @@ class InMemoryFaultStorage : public FaultStorage { std::vector get_all_faults() const override; std::vector reclassify_healed_as_cleared() override; + void set_planned_stop(const PlannedStopState & state) override; + PlannedStopState get_planned_stop() const override; + private: /// Update fault status based on debounce counter and given config void update_status(FaultState & state, const DebounceConfig & config); @@ -564,6 +588,10 @@ class InMemoryFaultStorage : public FaultStorage { size_t max_snapshots_per_fault_{0}; ///< 0 = unlimited bool retain_snapshots_on_clear_{false}; size_t max_near_misses_per_fault_{0}; ///< 0 = unlimited + /// Held for the life of the process only. This backend has no file behind it, + /// so a restart starts with no declaration - which is why a deployment that + /// needs the stop to outlive a reboot runs on the SQLite backend. + PlannedStopState planned_stop_; }; } // namespace ros2_medkit_fault_manager diff --git a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/sqlite_fault_storage.hpp b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/sqlite_fault_storage.hpp index 3565febf4..377e72fbf 100644 --- a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/sqlite_fault_storage.hpp +++ b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/sqlite_fault_storage.hpp @@ -95,6 +95,9 @@ class SqliteFaultStorage : public FaultStorage { std::vector get_all_faults() const override; std::vector reclassify_healed_as_cleared() override; + void set_planned_stop(const PlannedStopState & state) override; + PlannedStopState get_planned_stop() const override; + /// Get the database path const std::string & db_path() const { return db_path_; diff --git a/src/ros2_medkit_fault_manager/src/correlation/correlation_engine.cpp b/src/ros2_medkit_fault_manager/src/correlation/correlation_engine.cpp index c7cd12728..6c9c01833 100644 --- a/src/ros2_medkit_fault_manager/src/correlation/correlation_engine.cpp +++ b/src/ros2_medkit_fault_manager/src/correlation/correlation_engine.cpp @@ -28,6 +28,30 @@ ProcessFaultResult CorrelationEngine::process_fault(const std::string & fault_co std::chrono::steady_clock::time_point timestamp) { std::lock_guard lock(mutex_); + ProcessFaultResult result = correlate(fault_code, severity, timestamp); + + if (result.should_mute) { + // A rule is now the reason this fault is quiet. Hand it over: ending the + // planned stop must not release a fault a rule is still holding down. + planned_stop_muted_.erase(fault_code); + return result; + } + + if (planned_stop_active_) { + MutedFaultData muted; + muted.fault_code = fault_code; + muted.root_cause_code = kPlannedStopRootCause; + muted.rule_id = kPlannedStopRuleId; + muted_faults_[fault_code] = muted; + planned_stop_muted_.insert(fault_code); + result.should_mute = true; + } + + return result; +} + +ProcessFaultResult CorrelationEngine::correlate(const std::string & fault_code, const std::string & severity, + std::chrono::steady_clock::time_point timestamp) { ProcessFaultResult result; // First, clean up expired entries @@ -121,6 +145,7 @@ ProcessClearResult CorrelationEngine::process_clear(const std::string & fault_co // Clean up muted faults for (const auto & symptom_code : it->second) { muted_faults_.erase(symptom_code); + planned_stop_muted_.erase(symptom_code); } // Remove from root_to_symptoms @@ -229,10 +254,40 @@ ProcessClearResult CorrelationEngine::process_clear(const std::string & fault_co // Remove from muted faults if it was a symptom muted_faults_.erase(fault_code); + // A cleared fault has nothing left to announce, so the planned stop must not + // hand it back at switch-off. + planned_stop_muted_.erase(fault_code); return result; } +void CorrelationEngine::begin_planned_stop() { + std::lock_guard lock(mutex_); + planned_stop_active_ = true; +} + +std::vector CorrelationEngine::end_planned_stop() { + std::lock_guard lock(mutex_); + + planned_stop_active_ = false; + + std::vector unmuted; + unmuted.reserve(planned_stop_muted_.size()); + for (const auto & fault_code : planned_stop_muted_) { + if (muted_faults_.erase(fault_code) > 0) { + unmuted.push_back(fault_code); + } + } + planned_stop_muted_.clear(); + + return unmuted; +} + +bool CorrelationEngine::planned_stop_active() const { + std::lock_guard lock(mutex_); + return planned_stop_active_; +} + std::vector CorrelationEngine::get_muted_faults() const { std::lock_guard lock(mutex_); diff --git a/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp b/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp index 146f0e3b4..172dc2599 100644 --- a/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp +++ b/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp @@ -377,6 +377,18 @@ FaultManagerNode::FaultManagerNode(const rclcpp::NodeOptions & options) : Node(" handle_list_faults_for_entity(request, response); }); + set_planned_stop_srv_ = create_service( + "~/set_planned_stop", [this](const std::shared_ptr & request, + const std::shared_ptr & response) { + handle_set_planned_stop(request, response); + }); + + get_planned_stop_srv_ = create_service( + "~/get_planned_stop", [this](const std::shared_ptr & request, + const std::shared_ptr & response) { + handle_get_planned_stop(request, response); + }); + // Initialize snapshot capture auto snapshot_config = create_snapshot_config(); if (snapshot_config.enabled) { @@ -427,21 +439,29 @@ FaultManagerNode::FaultManagerNode(const rclcpp::NodeOptions & options) : Node(" }); } - // Initialize correlation engine (nullptr if disabled or not configured) + // Correlation engine: rules when configured, and in every case the planned-stop mute correlation_engine_ = create_correlation_engine(); - // Create correlation cleanup timer if correlation is enabled - if (correlation_engine_) { - auto cleanup_interval_sec = declare_parameter("correlation.cleanup_interval_sec", 5.0); - if (!(std::isfinite(cleanup_interval_sec) && cleanup_interval_sec > 0.0)) { - RCLCPP_WARN(get_logger(), "correlation.cleanup_interval_sec must be positive, got %.2f. Using default 5.0s", - cleanup_interval_sec); - cleanup_interval_sec = 5.0; - } - auto cleanup_interval_ms = static_cast(cleanup_interval_sec * 1000); - correlation_cleanup_timer_ = create_wall_timer(std::chrono::milliseconds(cleanup_interval_ms), [this]() { - correlation_engine_->cleanup_expired(); - }); + auto cleanup_interval_sec = declare_parameter("correlation.cleanup_interval_sec", 5.0); + if (!(std::isfinite(cleanup_interval_sec) && cleanup_interval_sec > 0.0)) { + RCLCPP_WARN(get_logger(), "correlation.cleanup_interval_sec must be positive, got %.2f. Using default 5.0s", + cleanup_interval_sec); + cleanup_interval_sec = 5.0; + } + auto cleanup_interval_ms = static_cast(cleanup_interval_sec * 1000); + correlation_cleanup_timer_ = create_wall_timer(std::chrono::milliseconds(cleanup_interval_ms), [this]() { + correlation_engine_->cleanup_expired(); + }); + + // A planned stop outlives the process that declared it: a weekend stop must not + // end because the box rebooted. Faults muted before the restart are not restored - + // muting is per fault cycle and those cycles are over as far as this process is + // concerned - but every fault reported from here is muted again. + planned_stop_ = storage_->get_planned_stop(); + if (planned_stop_.active) { + correlation_engine_->begin_planned_stop(); + RCLCPP_INFO(get_logger(), "Planned stop is in force (reason='%s', declared_by='%s'): new faults are marked muted", + planned_stop_.reason.c_str(), planned_stop_.declared_by.c_str()); } // Create auto-confirmation timer if enabled @@ -657,6 +677,14 @@ void FaultManagerNode::audit_transition(const char * transition, const ros2_medk event.description = fault.description; event.occurred_at_ns = occurred_at_ns; + append_audit_event(event); +} + +void FaultManagerNode::append_audit_event(const AuditEvent & event) { + if (!audit_log_) { + return; + } + try { audit_log_->append(event); } catch (const std::exception & e) { @@ -670,7 +698,7 @@ void FaultManagerNode::audit_transition(const char * transition, const ros2_medk audit_healthy_.store(false, std::memory_order_relaxed); const uint64_t dropped = audit_dropped_writes_.load(std::memory_order_relaxed); RCLCPP_ERROR(get_logger(), "Failed to append audit record for '%s' (%s): %s [audit dropped_writes=%" PRIu64 "]", - fault.fault_code.c_str(), transition, e.what(), dropped); + event.fault_code.c_str(), event.transition.c_str(), e.what(), dropped); if (audit_fail_closed_) { // Explicit fail-FAST: surface the broken audit to the caller so a // compliance-strict deployment learns the chain is now incomplete and an @@ -681,12 +709,26 @@ void FaultManagerNode::audit_transition(const char * transition, const ros2_medk "audit_log.fail_closed is set: audit append failed for '%s' (%s); the audit chain is now " "incomplete and requires operator action. The already-committed fault-state change is NOT " "rolled back (the fault store is a separate database).", - fault.fault_code.c_str(), transition); + event.fault_code.c_str(), event.transition.c_str()); throw; } } } +void FaultManagerNode::audit_planned_stop(const char * transition, const PlannedStopState & state, + int64_t occurred_at_ns) { + AuditEvent event; + // Empty: the transition is about the installation, not about one fault. + event.fault_code = ""; + event.transition = transition; + event.status = state.active ? "ACTIVE" : "INACTIVE"; + event.source_id = state.declared_by; + event.description = state.reason; + event.occurred_at_ns = occurred_at_ns; + + append_audit_event(event); +} + void FaultManagerNode::capture_on_confirm(const std::string & fault_code) { // Capture snapshots/rosbag when a fault confirms, via the bounded pool. // Both callers - the report handler and the auto-confirm timer - run on the @@ -1073,6 +1115,79 @@ void FaultManagerNode::handle_clear_fault( } } +void FaultManagerNode::handle_set_planned_stop( + const std::shared_ptr & request, + const std::shared_ptr & response) { + response->was_active = planned_stop_.active; + response->success = true; + + if (request->active == planned_stop_.active) { + // Asking for the state the switch is already in is not an error and is not a + // transition: nothing changes and the audit chain records nothing, so a client + // that retries a call cannot manufacture evidence of a stop that never started. + response->message = planned_stop_.active ? "Planned stop already in force" : "No planned stop in force"; + return; + } + + const int64_t transition_at_ns = get_wall_clock_time().nanoseconds(); + + if (request->active) { + planned_stop_.active = true; + planned_stop_.reason = request->reason; + planned_stop_.declared_by = request->declared_by; + planned_stop_.since_ns = transition_at_ns; + + correlation_engine_->begin_planned_stop(); + storage_->set_planned_stop(planned_stop_); + audit_planned_stop(kTransitionPlannedStopStarted, planned_stop_, transition_at_ns); + + response->message = "Planned stop declared"; + RCLCPP_INFO(get_logger(), "Planned stop declared by '%s': %s", planned_stop_.declared_by.c_str(), + planned_stop_.reason.c_str()); + return; + } + + const auto unmuted = correlation_engine_->end_planned_stop(); + + // The switch-off is the moment those faults become news. A CONFIRMED one was + // never announced - the confirmation happened behind the mute - so publish it + // now, once, or the alarm standing on the controller is invisible to every + // consumer of the stream. PREFAILED has nothing to announce yet, and a HEALED + // one already published its end. + size_t announced = 0; + for (const auto & fault_code : unmuted) { + auto fault = storage_->get_fault(fault_code); + if (fault && fault->status == ros2_medkit_msgs::msg::Fault::STATUS_CONFIRMED) { + publish_fault_event(ros2_medkit_msgs::msg::FaultEvent::EVENT_CONFIRMED, *fault); + ++announced; + } + } + + // The withdrawal carries its own reason and declarer into the audit record; the + // stored declaration goes back to "no stop", which is what GetPlannedStop then + // reports. + PlannedStopState ended; + ended.reason = request->reason; + ended.declared_by = request->declared_by; + audit_planned_stop(kTransitionPlannedStopEnded, ended, transition_at_ns); + + planned_stop_ = PlannedStopState{}; + storage_->set_planned_stop(planned_stop_); + + response->message = "Planned stop withdrawn"; + RCLCPP_INFO(get_logger(), "Planned stop withdrawn by '%s': unmuted %zu fault(s), announced %zu confirmation(s)", + request->declared_by.c_str(), unmuted.size(), announced); +} + +void FaultManagerNode::handle_get_planned_stop( + const std::shared_ptr & /*request*/, + const std::shared_ptr & response) { + response->active = planned_stop_.active; + response->reason = planned_stop_.reason; + response->declared_by = planned_stop_.declared_by; + response->since = rclcpp::Time(planned_stop_.since_ns, RCL_SYSTEM_TIME); +} + bool FaultManagerNode::is_valid_severity(uint8_t severity) { return severity <= ros2_medkit_msgs::msg::Fault::SEVERITY_CRITICAL; } @@ -1365,17 +1480,30 @@ void FaultManagerNode::load_snapshot_config_from_yaml(const std::string & config } std::unique_ptr FaultManagerNode::create_correlation_engine() { + auto config = load_correlation_config(); + if (config) { + RCLCPP_INFO(get_logger(), "Correlation engine enabled (default_window=%ums, patterns=%zu, rules=%zu)", + config->default_window_ms, config->patterns.size(), config->rules.size()); + return std::make_unique(*config); + } + + // No rules, but still an engine: the planned stop is a mute source an operator + // declares at runtime and it needs no configuration to exist. + return std::make_unique(correlation::CorrelationConfig{}); +} + +std::optional FaultManagerNode::load_correlation_config() { // Get correlation config file path from parameter auto config_file = declare_parameter("correlation.config_file", ""); if (config_file.empty()) { - RCLCPP_DEBUG(get_logger(), "Correlation disabled: no config_file specified"); - return nullptr; + RCLCPP_DEBUG(get_logger(), "Correlation rules disabled: no config_file specified"); + return std::nullopt; } if (!std::filesystem::exists(config_file)) { RCLCPP_ERROR(get_logger(), "Correlation config file not found: %s", config_file.c_str()); - return nullptr; + return std::nullopt; } try { @@ -1384,7 +1512,7 @@ std::unique_ptr FaultManagerNode::create_correla if (!config.enabled) { RCLCPP_INFO(get_logger(), "Correlation explicitly disabled in config"); - return nullptr; + return std::nullopt; } // Validate the config @@ -1396,17 +1524,14 @@ std::unique_ptr FaultManagerNode::create_correla for (const auto & error : validation.errors) { RCLCPP_ERROR(get_logger(), "Correlation config error: %s", error.c_str()); } - return nullptr; + return std::nullopt; } - RCLCPP_INFO(get_logger(), "Correlation engine enabled (default_window=%ums, patterns=%zu, rules=%zu)", - config.default_window_ms, config.patterns.size(), config.rules.size()); - - return std::make_unique(config); + return config; } catch (const std::exception & e) { RCLCPP_ERROR(get_logger(), "Failed to load correlation config: %s", e.what()); - return nullptr; + return std::nullopt; } } diff --git a/src/ros2_medkit_fault_manager/src/fault_storage.cpp b/src/ros2_medkit_fault_manager/src/fault_storage.cpp index 34f9053e8..2611f042b 100644 --- a/src/ros2_medkit_fault_manager/src/fault_storage.cpp +++ b/src/ros2_medkit_fault_manager/src/fault_storage.cpp @@ -875,4 +875,14 @@ std::vector InMemoryFaultStorage::reclassify_healed_as_cleared() { return reclassified; } +void InMemoryFaultStorage::set_planned_stop(const PlannedStopState & state) { + std::lock_guard lock(mutex_); + planned_stop_ = state; +} + +PlannedStopState InMemoryFaultStorage::get_planned_stop() const { + std::lock_guard lock(mutex_); + return planned_stop_; +} + } // namespace ros2_medkit_fault_manager diff --git a/src/ros2_medkit_fault_manager/src/sqlite_fault_storage.cpp b/src/ros2_medkit_fault_manager/src/sqlite_fault_storage.cpp index e466ea06a..7fa8ad042 100644 --- a/src/ros2_medkit_fault_manager/src/sqlite_fault_storage.cpp +++ b/src/ros2_medkit_fault_manager/src/sqlite_fault_storage.cpp @@ -316,6 +316,25 @@ void SqliteFaultStorage::initialize_schema() { } } + // Create planned_stop table: the operator's declaration that the plant is + // deliberately down. A fault manager holds exactly one, so the table is pinned + // to a single row by a constant primary key and written with INSERT OR REPLACE. + const char * create_planned_stop_table_sql = R"( + CREATE TABLE IF NOT EXISTS planned_stop ( + id INTEGER PRIMARY KEY CHECK (id = 1), + active INTEGER NOT NULL, + reason TEXT NOT NULL, + declared_by TEXT NOT NULL, + since_ns INTEGER NOT NULL + ); + )"; + + if (sqlite3_exec(db_, create_planned_stop_table_sql, nullptr, nullptr, &err_msg) != SQLITE_OK) { + std::string error = err_msg ? err_msg : "Unknown error"; + sqlite3_free(err_msg); + throw std::runtime_error("Failed to create planned_stop table: " + error); + } + // Create rosbag_files table. One row = one LINK (a fault claiming a recording): // several faults of a burst link to one bag, and one fault links to several bags // over time. Bytes belong to file_path, not to the row. @@ -1285,6 +1304,39 @@ void SqliteFaultStorage::store_freeze_frame(const FreezeFrameData & frame) { } } +void SqliteFaultStorage::set_planned_stop(const PlannedStopState & state) { + std::lock_guard lock(mutex_); + + SqliteStatement stmt(db_, + "INSERT OR REPLACE INTO planned_stop (id, active, reason, declared_by, since_ns) " + "VALUES (1, ?, ?, ?, ?)"); + stmt.bind_int(1, state.active ? 1 : 0); + stmt.bind_text(2, state.reason); + stmt.bind_text(3, state.declared_by); + stmt.bind_int64(4, state.since_ns); + + if (stmt.step() != SQLITE_DONE) { + throw std::runtime_error(std::string("Failed to store planned stop: ") + sqlite3_errmsg(db_)); + } +} + +PlannedStopState SqliteFaultStorage::get_planned_stop() const { + std::lock_guard lock(mutex_); + + SqliteStatement stmt(db_, "SELECT active, reason, declared_by, since_ns FROM planned_stop WHERE id = 1"); + + PlannedStopState state; + if (stmt.step() != SQLITE_ROW) { + return state; // never declared: the default is "no stop" + } + + state.active = stmt.column_int(0) != 0; + state.reason = stmt.column_text(1); + state.declared_by = stmt.column_text(2); + state.since_ns = stmt.column_int64(3); + return state; +} + std::optional SqliteFaultStorage::get_freeze_frame(const std::string & fault_code) const { std::lock_guard lock(mutex_); diff --git a/src/ros2_medkit_fault_manager/test/test_correlation_engine.cpp b/src/ros2_medkit_fault_manager/test/test_correlation_engine.cpp index 040c2df91..442db831e 100644 --- a/src/ros2_medkit_fault_manager/test/test_correlation_engine.cpp +++ b/src/ros2_medkit_fault_manager/test/test_correlation_engine.cpp @@ -14,6 +14,7 @@ #include +#include #include #include @@ -762,6 +763,182 @@ TEST_F(CorrelationEngineTest, CleanupExpiredRemovesFaultToClusterEntries) { EXPECT_EQ(3u, clusters[0].fault_codes.size()); } +// ============================================================================ +// Planned stop: a second, operator-driven mute source +// ============================================================================ + +TEST_F(CorrelationEngineTest, PlannedStopMutesAnUncorrelatedFault) { + CorrelationEngine engine(create_hierarchical_config()); + + EXPECT_FALSE(engine.planned_stop_active()); + auto before = engine.process_fault("PUMP_SEAL_LEAK", "ERROR"); + EXPECT_FALSE(before.should_mute); + EXPECT_FALSE(engine.is_muted("PUMP_SEAL_LEAK")); + + engine.begin_planned_stop(); + EXPECT_TRUE(engine.planned_stop_active()); + + auto during = engine.process_fault("VALVE_STUCK", "ERROR"); + EXPECT_TRUE(during.should_mute); + EXPECT_TRUE(engine.is_muted("VALVE_STUCK")); + EXPECT_EQ(1u, engine.get_muted_count()); + + auto muted = engine.get_muted_faults(); + ASSERT_EQ(1u, muted.size()); + EXPECT_EQ("VALVE_STUCK", muted[0].fault_code); + EXPECT_EQ(CorrelationEngine::kPlannedStopRootCause, muted[0].root_cause_code); + EXPECT_EQ(CorrelationEngine::kPlannedStopRuleId, muted[0].rule_id); +} + +TEST_F(CorrelationEngineTest, PlannedStopIsIdempotentAndSurvivesRepeatedReports) { + CorrelationEngine engine(create_hierarchical_config()); + + engine.begin_planned_stop(); + engine.begin_planned_stop(); + EXPECT_TRUE(engine.planned_stop_active()); + + engine.process_fault("VALVE_STUCK", "ERROR"); + engine.process_fault("VALVE_STUCK", "ERROR"); + EXPECT_EQ(1u, engine.get_muted_count()); +} + +TEST_F(CorrelationEngineTest, EndingPlannedStopUnmutesWhatItMuted) { + CorrelationEngine engine(create_hierarchical_config()); + + engine.begin_planned_stop(); + engine.process_fault("VALVE_STUCK", "ERROR"); + engine.process_fault("PUMP_SEAL_LEAK", "WARNING"); + EXPECT_EQ(2u, engine.get_muted_count()); + + auto unmuted = engine.end_planned_stop(); + EXPECT_FALSE(engine.planned_stop_active()); + std::sort(unmuted.begin(), unmuted.end()); + ASSERT_EQ(2u, unmuted.size()); + EXPECT_EQ("PUMP_SEAL_LEAK", unmuted[0]); + EXPECT_EQ("VALVE_STUCK", unmuted[1]); + EXPECT_EQ(0u, engine.get_muted_count()); + EXPECT_FALSE(engine.is_muted("VALVE_STUCK")); + EXPECT_FALSE(engine.is_muted("PUMP_SEAL_LEAK")); +} + +TEST_F(CorrelationEngineTest, EndingPlannedStopWithNothingMutedReturnsNothing) { + CorrelationEngine engine(create_hierarchical_config()); + + engine.begin_planned_stop(); + auto unmuted = engine.end_planned_stop(); + EXPECT_TRUE(unmuted.empty()); + + // Ending a stop that is already off changes nothing either. + EXPECT_TRUE(engine.end_planned_stop().empty()); + EXPECT_FALSE(engine.planned_stop_active()); +} + +TEST_F(CorrelationEngineTest, AFaultReportedAfterTheStopEndsIsNotMuted) { + CorrelationEngine engine(create_hierarchical_config()); + + engine.begin_planned_stop(); + engine.end_planned_stop(); + + auto result = engine.process_fault("VALVE_STUCK", "ERROR"); + EXPECT_FALSE(result.should_mute); + EXPECT_FALSE(engine.is_muted("VALVE_STUCK")); +} + +TEST_F(CorrelationEngineTest, EndingPlannedStopLeavesARuleMutedSymptomMuted) { + CorrelationEngine engine(create_hierarchical_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + + // ESTOP_001 is a root cause: muted by the stop, but not a symptom. + engine.process_fault("ESTOP_001", "CRITICAL", t0); + // MOTOR_COMM_FL matches the rule's symptom pattern inside the window, so the + // RULE mutes it - the stop is not the reason it is muted. + auto symptom = engine.process_fault("MOTOR_COMM_FL", "ERROR", t0 + 10ms); + EXPECT_TRUE(symptom.should_mute); + EXPECT_EQ("estop_cascade", symptom.rule_id); + + auto unmuted = engine.end_planned_stop(); + EXPECT_EQ(1u, unmuted.size()); + EXPECT_EQ("ESTOP_001", unmuted[0]); + + // The rule still holds the symptom down. + EXPECT_TRUE(engine.is_muted("MOTOR_COMM_FL")); + auto muted = engine.get_muted_faults(); + ASSERT_EQ(1u, muted.size()); + EXPECT_EQ("MOTOR_COMM_FL", muted[0].fault_code); + EXPECT_EQ("estop_cascade", muted[0].rule_id); +} + +TEST_F(CorrelationEngineTest, ARuleTakingOverAStopMutedFaultKeepsItMuted) { + CorrelationEngine engine(create_hierarchical_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + + // Muted by the stop first: no root cause is pending yet. + auto first = engine.process_fault("MOTOR_COMM_FL", "ERROR", t0); + EXPECT_TRUE(first.should_mute); + EXPECT_EQ(CorrelationEngine::kPlannedStopRuleId, engine.get_muted_faults()[0].rule_id); + + // The root cause arrives, then the same code is reported again and the rule + // claims it. + engine.process_fault("ESTOP_001", "CRITICAL", t0 + 10ms); + auto second = engine.process_fault("MOTOR_COMM_FL", "ERROR", t0 + 20ms); + EXPECT_TRUE(second.should_mute); + EXPECT_EQ("estop_cascade", second.rule_id); + + auto unmuted = engine.end_planned_stop(); + EXPECT_EQ(1u, unmuted.size()); + EXPECT_EQ("ESTOP_001", unmuted[0]); + EXPECT_TRUE(engine.is_muted("MOTOR_COMM_FL")); +} + +TEST_F(CorrelationEngineTest, ClearingDuringAStopTakesTheFaultOutOfTheMute) { + CorrelationEngine engine(create_hierarchical_config()); + + engine.begin_planned_stop(); + engine.process_fault("VALVE_STUCK", "ERROR"); + EXPECT_TRUE(engine.is_muted("VALVE_STUCK")); + + engine.process_clear("VALVE_STUCK"); + EXPECT_FALSE(engine.is_muted("VALVE_STUCK")); + EXPECT_EQ(0u, engine.get_muted_count()); + + // Nothing survives to be unmuted, so the switch-off announces nothing. + EXPECT_TRUE(engine.end_planned_stop().empty()); +} + +TEST_F(CorrelationEngineTest, ClearingARootCauseDuringAStopDropsItsStopMutedSymptoms) { + CorrelationEngine engine(create_hierarchical_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + engine.process_fault("ESTOP_001", "CRITICAL", t0); + engine.process_fault("MOTOR_COMM_FL", "ERROR", t0 + 10ms); + + auto cleared = engine.process_clear("ESTOP_001"); + ASSERT_EQ(1u, cleared.auto_cleared_codes.size()); + EXPECT_EQ("MOTOR_COMM_FL", cleared.auto_cleared_codes[0]); + EXPECT_FALSE(engine.is_muted("MOTOR_COMM_FL")); + + // Both codes left the mute with the clear; the switch-off has nothing to say. + EXPECT_TRUE(engine.end_planned_stop().empty()); +} + +TEST_F(CorrelationEngineTest, PlannedStopMutesWithoutAnyRulesConfigured) { + CorrelationEngine engine{CorrelationConfig{}}; + + engine.begin_planned_stop(); + auto result = engine.process_fault("VALVE_STUCK", "ERROR"); + EXPECT_TRUE(result.should_mute); + EXPECT_TRUE(engine.is_muted("VALVE_STUCK")); + + auto unmuted = engine.end_planned_stop(); + ASSERT_EQ(1u, unmuted.size()); + EXPECT_EQ("VALVE_STUCK", unmuted[0]); +} + int main(int argc, char ** argv) { ::testing::InitGoogleTest(&argc, argv); return RUN_ALL_TESTS(); diff --git a/src/ros2_medkit_fault_manager/test/test_sqlite_storage.cpp b/src/ros2_medkit_fault_manager/test/test_sqlite_storage.cpp index 391ddd908..c75a5a91f 100644 --- a/src/ros2_medkit_fault_manager/test/test_sqlite_storage.cpp +++ b/src/ros2_medkit_fault_manager/test/test_sqlite_storage.cpp @@ -26,6 +26,7 @@ #include #include "rclcpp/rclcpp.hpp" +#include "ros2_medkit_fault_manager/fault_storage.hpp" #include "ros2_medkit_fault_manager/sqlite_fault_storage.hpp" #include "ros2_medkit_msgs/msg/fault.hpp" #include "ros2_medkit_msgs/srv/report_fault.hpp" @@ -2391,6 +2392,120 @@ TEST_F(SqliteFaultStorageTest, SnapshotsDroppedByHealedReclassificationByDefault EXPECT_TRUE(storage_->get_snapshots("SNAPSHOT_DROP_TEST").empty()); } +// ============================================================================ +// Planned-stop declaration (survives a restart) +// ============================================================================ + +TEST_F(SqliteFaultStorageTest, PlannedStopDefaultsToInactive) { + const auto state = storage_->get_planned_stop(); + EXPECT_FALSE(state.active); + EXPECT_TRUE(state.reason.empty()); + EXPECT_TRUE(state.declared_by.empty()); + EXPECT_EQ(0, state.since_ns); +} + +TEST_F(SqliteFaultStorageTest, PlannedStopRoundTripsThroughAReopen) { + ros2_medkit_fault_manager::PlannedStopState declared; + declared.active = true; + declared.reason = "line 3 quarterly maintenance"; + declared.declared_by = "shift_lead"; + declared.since_ns = 1757000000000000000; + storage_->set_planned_stop(declared); + + // Reopening the same file is what a restart of the fault manager does. + storage_.reset(); + storage_ = std::make_unique(temp_db_path_.string()); + + const auto reopened = storage_->get_planned_stop(); + EXPECT_TRUE(reopened.active); + EXPECT_EQ("line 3 quarterly maintenance", reopened.reason); + EXPECT_EQ("shift_lead", reopened.declared_by); + EXPECT_EQ(1757000000000000000, reopened.since_ns); +} + +TEST_F(SqliteFaultStorageTest, PlannedStopKeepsOneRowAcrossRepeatedWrites) { + ros2_medkit_fault_manager::PlannedStopState first; + first.active = true; + first.reason = "first"; + first.declared_by = "a"; + first.since_ns = 111; + storage_->set_planned_stop(first); + + ros2_medkit_fault_manager::PlannedStopState second; + second.active = true; + second.reason = "second"; + second.declared_by = "b"; + second.since_ns = 222; + storage_->set_planned_stop(second); + + storage_.reset(); + storage_ = std::make_unique(temp_db_path_.string()); + + const auto state = storage_->get_planned_stop(); + EXPECT_EQ("second", state.reason); + EXPECT_EQ("b", state.declared_by); + EXPECT_EQ(222, state.since_ns); +} + +TEST_F(SqliteFaultStorageTest, WithdrawnPlannedStopIsWhatAReopenSees) { + ros2_medkit_fault_manager::PlannedStopState declared; + declared.active = true; + declared.reason = "maintenance"; + declared.declared_by = "shift_lead"; + declared.since_ns = 999; + storage_->set_planned_stop(declared); + storage_->set_planned_stop(ros2_medkit_fault_manager::PlannedStopState{}); + + storage_.reset(); + storage_ = std::make_unique(temp_db_path_.string()); + + const auto state = storage_->get_planned_stop(); + EXPECT_FALSE(state.active); + EXPECT_TRUE(state.reason.empty()); + EXPECT_TRUE(state.declared_by.empty()); + EXPECT_EQ(0, state.since_ns); +} + +TEST_F(SqliteFaultStorageTest, PlannedStopCarriesAKilobyteReason) { + const std::string long_reason(1024, 'r'); + ros2_medkit_fault_manager::PlannedStopState declared; + declared.active = true; + declared.reason = long_reason; + declared.since_ns = 7; + storage_->set_planned_stop(declared); + + storage_.reset(); + storage_ = std::make_unique(temp_db_path_.string()); + + const auto state = storage_->get_planned_stop(); + EXPECT_TRUE(state.active); + EXPECT_EQ(long_reason, state.reason); + EXPECT_TRUE(state.declared_by.empty()); +} + +TEST(InMemoryFaultStorageTest, PlannedStopRoundTripsInMemory) { + ros2_medkit_fault_manager::InMemoryFaultStorage storage; + + EXPECT_FALSE(storage.get_planned_stop().active); + + ros2_medkit_fault_manager::PlannedStopState declared; + declared.active = true; + declared.reason = "cell 7 changeover"; + declared.declared_by = "maintenance"; + declared.since_ns = 42; + storage.set_planned_stop(declared); + + const auto state = storage.get_planned_stop(); + EXPECT_TRUE(state.active); + EXPECT_EQ("cell 7 changeover", state.reason); + EXPECT_EQ("maintenance", state.declared_by); + EXPECT_EQ(42, state.since_ns); + + storage.set_planned_stop(ros2_medkit_fault_manager::PlannedStopState{}); + EXPECT_FALSE(storage.get_planned_stop().active); + EXPECT_TRUE(storage.get_planned_stop().reason.empty()); +} + int main(int argc, char ** argv) { rclcpp::init(argc, argv); ::testing::InitGoogleTest(&argc, argv); From 517122c4dfeebd149c964a8a9e3f2ef0ff907516 Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Mon, 7 Sep 2026 16:41:42 +0200 Subject: [PATCH 03/12] fault_manager: prove the planned-stop switch end to end Engine unit cases pin the two mute sources composing: a rule takes a mute over from the switch and keeps it at switch-off, and a fault cleared inside the stop leaves both maps. Storage cases pin the declaration surviving a reopen on SQLite and living for the process on the memory backend, and the audit case pins a transition with no fault code chaining and staying tamper-evident. Three launch tests drive a live manager: the main one covers marking, counting, capture, the audit rows, idempotent requests, an empty and a kilobyte reason, 200 faults released by one switch-off, a fault mid-debounce when the stop ends, and a rule-muted symptom the switch must not release; one kills the manager and lets launch bring it back; one drives a confirmation through the auto-confirm timer with a control fault proving the timer fires at all. The gateway feature test is the acceptance: it turns the stop on and off through POST /apps/fault_manager/operations/set_planned_stop/executions, watches GET /faults and the SSE fault stream, and counts confirmations per fault code either side of the switch-off. Refs #656 --- src/ros2_medkit_fault_manager/CMakeLists.txt | 17 + .../integration/test_planned_stop.test.py | 601 ++++++++++++++++++ .../test_planned_stop_auto_confirm.test.py | 264 ++++++++ .../test_planned_stop_restart.test.py | 239 +++++++ .../test/test_fault_audit_log.cpp | 42 ++ .../features/test_planned_stop_switch.test.py | 330 ++++++++++ 6 files changed, 1493 insertions(+) create mode 100644 src/ros2_medkit_fault_manager/test/integration/test_planned_stop.test.py create mode 100644 src/ros2_medkit_fault_manager/test/integration/test_planned_stop_auto_confirm.test.py create mode 100644 src/ros2_medkit_fault_manager/test/integration/test_planned_stop_restart.test.py create mode 100644 src/ros2_medkit_integration_tests/test/features/test_planned_stop_switch.test.py diff --git a/src/ros2_medkit_fault_manager/CMakeLists.txt b/src/ros2_medkit_fault_manager/CMakeLists.txt index 064cf44f1..4b02cd3d8 100644 --- a/src/ros2_medkit_fault_manager/CMakeLists.txt +++ b/src/ros2_medkit_fault_manager/CMakeLists.txt @@ -208,6 +208,23 @@ if(BUILD_TESTING) medkit_add_launch_test(test_rosbag_entity_scope test/integration/test_rosbag_entity_scope.test.py TIMEOUT 120 LABELS "integration") + # The planned-stop switch end to end: marking, counting, auditing, capture, and + # the release at switch-off. One case raises 200 faults inside a stop, so the + # budget is well above the 60s default. + medkit_add_launch_test(test_planned_stop test/integration/test_planned_stop.test.py TIMEOUT 240 + LABELS "integration") + + # Kills the manager and lets launch bring it back, so the declaration is read + # from the store by a process that never saw the request that made it. + medkit_add_launch_test(test_planned_stop_restart test/integration/test_planned_stop_restart.test.py + TIMEOUT 180 LABELS "integration") + + # auto_confirm_after_sec confirms from a timer rather than a report, so the + # mute has to be honoured on that path too. Its own launch, because every other + # planned-stop case would then race a timer that confirms behind its back. + medkit_add_launch_test(test_planned_stop_auto_confirm + test/integration/test_planned_stop_auto_confirm.test.py TIMEOUT 120 LABELS "integration") + # The only suite running above max_bags_per_fault=1: drives confirm / clear / # confirm on one code and asserts both recordings survive and stay separately # addressable. Six occurrences plus their post-roll windows, hence the timeout. diff --git a/src/ros2_medkit_fault_manager/test/integration/test_planned_stop.test.py b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop.test.py new file mode 100644 index 000000000..c5e59897c --- /dev/null +++ b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop.test.py @@ -0,0 +1,601 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +End-to-end tests for the planned-stop switch on a live fault manager. + +Drives the real services over the real ROS graph against the SQLite backend +with the audit log on, and reads the audit database directly, because the +manager exposes no service that reads it back. + +The instrument for "the stream stayed quiet" is a subscriber on the manager's +own ``~/events`` topic, counted PER FAULT CODE: a count alone would pass while +the wrong fault was announced. +""" + +import json +import os +import sqlite3 +import tempfile +import threading +import time +import unittest + +from launch import LaunchDescription +import launch.actions +import launch_ros.actions +import launch_testing.actions +import rclpy +from rclpy.node import Node +from rclpy.qos import HistoryPolicy, QoSProfile, ReliabilityPolicy +from ros2_medkit_msgs.msg import Fault, FaultEvent +from ros2_medkit_msgs.srv import ( + ClearFault, + GetFault, + GetPlannedStop, + GetSnapshots, + ListFaults, + ReportFault, + SetPlannedStop, +) +from sensor_msgs.msg import Temperature + +STORAGE_DIR = tempfile.mkdtemp(prefix='planned_stop_') +DATABASE_PATH = os.path.join(STORAGE_DIR, 'faults.db') +AUDIT_PATH = os.path.join(STORAGE_DIR, 'fault_audit.db') +CORRELATION_PATH = os.path.join(STORAGE_DIR, 'correlation.yaml') + +# Two FAILED reports confirm, one leaves the fault PREFAILED, which is what the +# mid-debounce case needs. +CONFIRMATION_THRESHOLD = -2 + +SOURCE_ID = '/test_node' +CAPTURE_TOPIC = '/test/temperature' + +# A rule that mutes MOTOR_PS_* while ESTOP_PS_001 stands. Its window is long +# enough that no test can lose the correlation to its own pacing. +CORRELATION_RULES = """ +correlation: + enabled: true + patterns: + ps_motor_errors: + codes: ["MOTOR_PS_*"] + rules: + - id: ps_estop_cascade + name: "E-Stop Cascade" + mode: hierarchical + root_cause: + codes: ["ESTOP_PS_001"] + symptoms: + - pattern: ps_motor_errors + window_ms: 60000 + mute_symptoms: true + auto_clear_with_root: true +""" + + +def get_coverage_env(): + """Get environment variables for gcov coverage data collection.""" + try: + from ament_index_python.packages import get_package_prefix + pkg_prefix = get_package_prefix('ros2_medkit_fault_manager') + workspace = os.path.dirname(os.path.dirname(pkg_prefix)) + build_dir = os.path.join(workspace, 'build', 'ros2_medkit_fault_manager') + + if os.path.exists(build_dir): + return { + 'GCOV_PREFIX': build_dir, + 'GCOV_PREFIX_STRIP': str(build_dir.count(os.sep)), + } + except Exception: + # Coverage environment is optional; on any error, fall back to no extra config + pass + return {} + + +def generate_test_description(): + """Launch a fault manager with SQLite storage, the audit log and one rule.""" + with open(CORRELATION_PATH, 'w') as handle: + handle.write(CORRELATION_RULES) + + fault_manager_env = get_coverage_env() + fault_manager_env['ROS_LOCALHOST_ONLY'] = '1' + + fault_manager_node = launch_ros.actions.Node( + package='ros2_medkit_fault_manager', + executable='fault_manager_node', + name='fault_manager', + output='screen', + additional_env=fault_manager_env, + parameters=[{ + 'storage_type': 'sqlite', + 'database_path': DATABASE_PATH, + 'confirmation_threshold': CONFIRMATION_THRESHOLD, + 'audit_log.enabled': True, + 'audit_log.database_path': AUDIT_PATH, + 'correlation.config_file': CORRELATION_PATH, + # Capture must run for a muted fault exactly as it does for any + # other, so it is on and pointed at a topic this test publishes. + 'snapshots.enabled': True, + 'snapshots.default_topics': [CAPTURE_TOPIC], + 'snapshots.timeout_sec': 2.0, + 'snapshots.recapture_cooldown_sec': 0.0, + 'snapshots.rosbag.enabled': False, + }], + sigterm_timeout='30', + sigkill_timeout='15', + ) + + return ( + LaunchDescription([ + launch.actions.TimerAction(period=2.0, actions=[fault_manager_node]), + launch_testing.actions.ReadyToTest(), + ]), + {'fault_manager_node': fault_manager_node}, + ) + + +class TestPlannedStop(unittest.TestCase): + """The switch marks, counts, audits and later releases what it muted.""" + + @classmethod + def setUpClass(cls): + os.environ['ROS_LOCALHOST_ONLY'] = '1' + rclpy.init() + cls.node = Node('test_planned_stop_client') + + cls.report_client = cls.node.create_client(ReportFault, '/fault_manager/report_fault') + cls.list_client = cls.node.create_client(ListFaults, '/fault_manager/list_faults') + cls.get_client = cls.node.create_client(GetFault, '/fault_manager/get_fault') + cls.clear_client = cls.node.create_client(ClearFault, '/fault_manager/clear_fault') + cls.snapshots_client = cls.node.create_client(GetSnapshots, '/fault_manager/get_snapshots') + cls.set_stop_client = cls.node.create_client( + SetPlannedStop, '/fault_manager/set_planned_stop') + cls.get_stop_client = cls.node.create_client( + GetPlannedStop, '/fault_manager/get_planned_stop') + + for client, name in ( + (cls.report_client, 'report_fault'), + (cls.list_client, 'list_faults'), + (cls.get_client, 'get_fault'), + (cls.clear_client, 'clear_fault'), + (cls.snapshots_client, 'get_snapshots'), + (cls.set_stop_client, 'set_planned_stop'), + (cls.get_stop_client, 'get_planned_stop'), + ): + assert client.wait_for_service(timeout_sec=20.0), f'{name} service not available' + + # A topic for the capture set to sample, so a muted fault's freeze frame + # is a real reading rather than an empty one. + cls._temperature_pub = cls.node.create_publisher( + Temperature, CAPTURE_TOPIC, + QoSProfile(reliability=ReliabilityPolicy.BEST_EFFORT, + history=HistoryPolicy.KEEP_LAST, depth=10)) + cls._publish_timer = cls.node.create_timer(0.05, cls._publish_temperature) + + # Deep enough that a switch-off releasing hundreds of faults at once + # cannot overrun the reader while the spin thread is between wakeups. + cls._events = [] + cls._events_lock = threading.Lock() + cls._event_sub = cls.node.create_subscription( + FaultEvent, '/fault_manager/events', cls._on_event, + QoSProfile(reliability=ReliabilityPolicy.RELIABLE, + history=HistoryPolicy.KEEP_LAST, depth=2000)) + + cls._spinning = True + cls._spin_thread = threading.Thread(target=cls._spin, daemon=True) + cls._spin_thread.start() + + cls._prime_event_stream() + + @classmethod + def tearDownClass(cls): + cls._spinning = False + cls._spin_thread.join(timeout=5) + cls.node.destroy_node() + rclpy.shutdown() + + def tearDown(self): + # The switch is one declaration per manager, so a test that leaves it on + # would decide the next test's outcome. + self._set_stop(False, reason='teardown', declared_by='test') + + @classmethod + def _publish_temperature(cls): + msg = Temperature() + msg.temperature = 42.0 + msg.variance = 0.1 + cls._temperature_pub.publish(msg) + + @classmethod + def _on_event(cls, msg): + with cls._events_lock: + cls._events.append((msg.event_type, msg.fault.fault_code, msg.fault.status)) + + @classmethod + def _spin(cls): + while cls._spinning: + rclpy.spin_once(cls.node, timeout_sec=0.05) + + @classmethod + def _call(cls, client, request, timeout_sec=15.0): + future = client.call_async(request) + deadline = time.monotonic() + timeout_sec + while not future.done() and time.monotonic() < deadline: + time.sleep(0.005) + assert future.done(), 'service call timed out' + return future.result() + + @classmethod + def _report(cls, fault_code, event_type=ReportFault.Request.EVENT_FAILED): + request = ReportFault.Request() + request.fault_code = fault_code + request.event_type = event_type + request.severity = Fault.SEVERITY_ERROR + request.description = 'planned stop test fault' + request.source_id = SOURCE_ID + return cls._call(cls.report_client, request) + + @classmethod + def _confirm(cls, fault_code): + """Drive a fault to CONFIRMED with the configured debounce threshold.""" + for _ in range(abs(CONFIRMATION_THRESHOLD)): + cls._report(fault_code) + + @classmethod + def _prime_event_stream(cls): + """ + Block until an event demonstrably reaches this subscriber. + + ``~/events`` is reliable but volatile: an event published before the + subscription has matched the publisher is lost outright, so a one-shot + assertion on the first fault of the suite would race the match. + """ + deadline = time.monotonic() + 30.0 + attempt = 0 + while time.monotonic() < deadline: + code = f'PS_PRIME_{attempt}' + cls._confirm(code) + settle = time.monotonic() + 1.0 + while time.monotonic() < settle: + if cls._count_events(code, FaultEvent.EVENT_CONFIRMED) > 0: + return + time.sleep(0.05) + attempt += 1 + raise AssertionError('no event reached the subscriber; the events topic never went live') + + @classmethod + def _count_events(cls, fault_code, event_type): + with cls._events_lock: + return sum(1 for kind, code, _ in cls._events + if code == fault_code and kind == event_type) + + def _set_stop(self, active, *, reason='', declared_by=''): + request = SetPlannedStop.Request() + request.active = active + request.reason = reason + request.declared_by = declared_by + return self._call(self.set_stop_client, request) + + def _get_stop(self): + return self._call(self.get_stop_client, GetPlannedStop.Request()) + + def _list(self, *, statuses=None, include_muted=False): + request = ListFaults.Request() + request.filter_by_severity = False + request.severity = 0 + request.statuses = statuses if statuses is not None else [] + request.include_muted = include_muted + request.include_clusters = False + return self._call(self.list_client, request) + + def _codes_in_default_list(self): + return {fault.fault_code for fault in self._list().faults} + + def _muted_entry(self, fault_code): + response = self._list(include_muted=True) + for info in response.muted_faults: + if info.fault_code == fault_code: + return info + return None + + def _clear(self, fault_code): + request = ClearFault.Request() + request.fault_code = fault_code + request.skip_correlation_auto_clear = False + return self._call(self.clear_client, request) + + def _get_fault(self, fault_code): + request = GetFault.Request() + request.fault_code = fault_code + return self._call(self.get_client, request) + + def _captured_topics(self, fault_code): + """Topic names present in the fault's stored capture, empty when none.""" + request = GetSnapshots.Request() + request.fault_code = fault_code + request.topic = '' + response = self._call(self.snapshots_client, request) + if not response.success or not response.data: + return {} + return json.loads(response.data).get('topics', {}) + + @staticmethod + def _audit_rows(transition=None): + """Read the audit database directly - no service serves it back.""" + connection = sqlite3.connect(f'file:{AUDIT_PATH}?mode=ro', uri=True) + try: + if transition is None: + cursor = connection.execute( + 'SELECT fault_code, transition, status, source_id, description FROM audit_log ' + 'ORDER BY seq') + else: + cursor = connection.execute( + 'SELECT fault_code, transition, status, source_id, description FROM audit_log ' + 'WHERE transition = ? ORDER BY seq', (transition,)) + return cursor.fetchall() + finally: + connection.close() + + def _wait_until(self, predicate, message, timeout=20.0): + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + value = predicate() + if value: + return value + time.sleep(0.1) + raise AssertionError(message) + + # ------------------------------------------------------------------ + # Cases + # ------------------------------------------------------------------ + + def test_a_fault_raised_during_a_stop_is_marked_not_dropped(self): + code = 'PS_MARKED' + + response = self._set_stop(True, reason='line 3 maintenance', declared_by='shift_lead') + self.assertTrue(response.success) + self.assertFalse(response.was_active) + + self._confirm(code) + + # In the store, CONFIRMED, and absent from the default list. + self._wait_until(lambda: self._get_fault(code).success, f'{code} never reached the store') + self.assertEqual('CONFIRMED', self._get_fault(code).fault.status) + self.assertNotIn(code, self._codes_in_default_list()) + + muted = self._muted_entry(code) + self.assertIsNotNone(muted, f'{code} is not listed as muted') + self.assertEqual('planned_stop', muted.rule_id) + self.assertEqual('PLANNED_STOP', muted.root_cause_code) + self.assertGreaterEqual(self._list().muted_count, 1) + + # Capture ran anyway: the evidence a muted fault leaves is the same. + captured = self._wait_until( + lambda: self._captured_topics(code), + f'no snapshot was captured for the muted fault {code}') + self.assertIn(CAPTURE_TOPIC, captured) + + # Nothing was announced while the stop stood. + self.assertEqual(0, self._count_events(code, FaultEvent.EVENT_CONFIRMED)) + + withdrawal = self._set_stop(False, reason='line 3 back up', declared_by='shift_lead') + self.assertTrue(withdrawal.success) + self.assertTrue(withdrawal.was_active) + + self._wait_until(lambda: code in self._codes_in_default_list(), + f'{code} did not return to the default list after the stop ended') + self.assertIsNone(self._muted_entry(code)) + self.assertEqual( + 1, self._count_events(code, FaultEvent.EVENT_CONFIRMED), + 'the confirmation held back by the stop must be announced exactly once') + + def test_a_fault_cleared_during_a_stop_never_reaches_the_default_list(self): + code = 'PS_CLEARED_INSIDE' + + self._set_stop(True, reason='changeover', declared_by='maintenance') + self._confirm(code) + self._wait_until(lambda: self._get_fault(code).success, f'{code} never reached the store') + self.assertIsNotNone(self._muted_entry(code)) + + cleared = self._clear(code) + self.assertTrue(cleared.success) + + self.assertNotIn(code, self._codes_in_default_list()) + self.assertIsNone(self._muted_entry(code)) + + # It is still in the store, and in the audit trail. + stored = self._get_fault(code) + self.assertTrue(stored.success) + self.assertEqual('CLEARED', stored.fault.status) + cleared_codes = {fault.fault_code for fault in self._list(statuses=['CLEARED']).faults} + self.assertIn(code, cleared_codes) + transitions = {row[1] for row in self._audit_rows() if row[0] == code} + self.assertIn('occurred', transitions) + self.assertIn('confirmed', transitions) + self.assertIn('cleared', transitions) + + self._set_stop(False, reason='done', declared_by='maintenance') + + self.assertNotIn(code, self._codes_in_default_list()) + self.assertEqual( + 0, self._count_events(code, FaultEvent.EVENT_CONFIRMED), + 'a fault that was acknowledged inside the stop has nothing to announce') + + def test_repeating_a_request_changes_nothing_and_records_nothing(self): + started_before = len(self._audit_rows('planned_stop_started')) + ended_before = len(self._audit_rows('planned_stop_ended')) + + first_on = self._set_stop(True, reason='weekend shutdown', declared_by='plant_manager') + self.assertTrue(first_on.success) + self.assertFalse(first_on.was_active) + + second_on = self._set_stop(True, reason='ignored', declared_by='someone_else') + self.assertTrue(second_on.success) + self.assertTrue(second_on.was_active) + + started = self._audit_rows('planned_stop_started') + self.assertEqual(started_before + 1, len(started), + 'declaring a stop that is already in force must write no audit record') + self.assertEqual('', started[-1][0]) + self.assertEqual('ACTIVE', started[-1][2]) + self.assertEqual('plant_manager', started[-1][3]) + self.assertEqual('weekend shutdown', started[-1][4]) + + # The declaration the manager serves is the first one, not the repeat. + state = self._get_stop() + self.assertTrue(state.active) + self.assertEqual('weekend shutdown', state.reason) + self.assertEqual('plant_manager', state.declared_by) + + first_off = self._set_stop(False, reason='restart', declared_by='plant_manager') + self.assertTrue(first_off.success) + self.assertTrue(first_off.was_active) + + second_off = self._set_stop(False, reason='ignored', declared_by='someone_else') + self.assertTrue(second_off.success) + self.assertFalse(second_off.was_active) + + ended = self._audit_rows('planned_stop_ended') + self.assertEqual(ended_before + 1, len(ended), + 'withdrawing a stop that is not in force must write no audit record') + self.assertEqual('', ended[-1][0]) + self.assertEqual('INACTIVE', ended[-1][2]) + self.assertEqual('plant_manager', ended[-1][3]) + self.assertEqual('restart', ended[-1][4]) + + state = self._get_stop() + self.assertFalse(state.active) + self.assertEqual('', state.reason) + self.assertEqual('', state.declared_by) + + def test_the_reason_and_declarer_may_be_empty_or_long(self): + empty = self._set_stop(True, reason='', declared_by='') + self.assertTrue(empty.success) + state = self._get_stop() + self.assertTrue(state.active) + self.assertEqual('', state.reason) + self.assertEqual('', state.declared_by) + self.assertGreater(state.since.sec, 0, 'a declared stop must carry the time it started') + + self._set_stop(False) + + long_reason = 'r' * 1024 + self.assertTrue(self._set_stop(True, reason=long_reason, declared_by='a' * 256).success) + state = self._get_stop() + self.assertTrue(state.active) + self.assertEqual(long_reason, state.reason) + self.assertEqual('a' * 256, state.declared_by) + self.assertEqual(long_reason, self._audit_rows('planned_stop_started')[-1][4]) + + def test_two_hundred_faults_muted_by_one_stop_are_all_released(self): + codes = [f'PS_SCALE_{index:03d}' for index in range(200)] + + self._set_stop(True, reason='annual shutdown', declared_by='plant_manager') + for code in codes: + self._confirm(code) + + listed = self._list(include_muted=True) + muted_codes = {info.fault_code for info in listed.muted_faults} + missing = [code for code in codes if code not in muted_codes] + self.assertEqual([], missing, f'{len(missing)} of 200 faults were not marked muted') + + visible = self._codes_in_default_list() + self.assertEqual([], [code for code in codes if code in visible], + 'a muted fault must not appear in the default list') + + self._set_stop(False, reason='shutdown over', declared_by='plant_manager') + + self._wait_until( + lambda: not [code for code in codes if code not in self._codes_in_default_list()], + 'not every fault the stop muted came back to the default list') + + def _miscounted(): + return [(code, self._count_events(code, FaultEvent.EVENT_CONFIRMED)) + for code in codes + if self._count_events(code, FaultEvent.EVENT_CONFIRMED) != 1] + + deadline = time.monotonic() + 30.0 + wrong = _miscounted() + while wrong and time.monotonic() < deadline: + time.sleep(0.2) + wrong = _miscounted() + self.assertEqual( + [], wrong, + f'{len(wrong)} of 200 faults were not announced exactly once at the switch-off; ' + f'first offenders: {wrong[:10]}') + + def test_a_fault_mid_debounce_when_the_stop_ends_confirms_afterwards(self): + code = 'PS_MID_DEBOUNCE' + + self._set_stop(True, reason='calibration', declared_by='tech') + # One report short of confirmation: the fault is PREFAILED and muted. + self._report(code) + self._wait_until(lambda: self._get_fault(code).success, f'{code} never reached the store') + self.assertEqual('PREFAILED', self._get_fault(code).fault.status) + self.assertIsNotNone(self._muted_entry(code)) + + self._set_stop(False, reason='calibration over', declared_by='tech') + + self.assertIsNone(self._muted_entry(code)) + self.assertEqual(0, self._count_events(code, FaultEvent.EVENT_CONFIRMED), + 'a fault still short of confirmation has nothing to announce') + + # The debounce carries on across the switch: one more report confirms it, + # and with the stop gone the confirmation is announced. + self._report(code) + self._wait_until(lambda: self._count_events(code, FaultEvent.EVENT_CONFIRMED) == 1, + 'the confirmation after the stop ended was never announced') + self.assertEqual('CONFIRMED', self._get_fault(code).fault.status) + self.assertIn(code, self._codes_in_default_list()) + + def test_the_switch_does_not_release_a_rule_muted_symptom(self): + root = 'ESTOP_PS_001' + symptom = 'MOTOR_PS_LEFT' + + self._set_stop(True, reason='e-stop drill', declared_by='safety') + self._confirm(root) + self._confirm(symptom) + + self._wait_until(lambda: self._get_fault(symptom).success, + f'{symptom} never reached the store') + root_muted = self._muted_entry(root) + self.assertIsNotNone(root_muted) + self.assertEqual('planned_stop', root_muted.rule_id) + symptom_muted = self._muted_entry(symptom) + self.assertIsNotNone(symptom_muted) + self.assertEqual('ps_estop_cascade', symptom_muted.rule_id, + 'the correlation rule, not the switch, must own this mute') + + self._set_stop(False, reason='drill over', declared_by='safety') + + self._wait_until(lambda: root in self._codes_in_default_list(), + 'the root cause did not return to the default list') + self.assertEqual(1, self._count_events(root, FaultEvent.EVENT_CONFIRMED)) + + still_muted = self._muted_entry(symptom) + self.assertIsNotNone(still_muted, 'the switch released a symptom a rule was muting') + self.assertEqual('ps_estop_cascade', still_muted.rule_id) + self.assertNotIn(symptom, self._codes_in_default_list()) + self.assertEqual(0, self._count_events(symptom, FaultEvent.EVENT_CONFIRMED)) + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + for info in proc_info: + self.assertIn(info.returncode, (0, -2, -15), + f'{info.process_name} exited with code {info.returncode}') diff --git a/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_auto_confirm.test.py b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_auto_confirm.test.py new file mode 100644 index 000000000..590de59e4 --- /dev/null +++ b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_auto_confirm.test.py @@ -0,0 +1,264 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +A time-based confirmation obeys the planned stop like any other. + +``auto_confirm_after_sec`` confirms a lingering PREFAILED fault from a timer +rather than from a report, on a code path of its own. A stop that only silenced +the report path would let that timer announce, over the stream, a fault the list +is hiding - so this drives a confirmation THROUGH the timer while a stop stands. +""" + +import os +import tempfile +import threading +import time +import unittest + +from launch import LaunchDescription +import launch.actions +import launch_ros.actions +import launch_testing.actions +import rclpy +from rclpy.node import Node +from rclpy.qos import HistoryPolicy, QoSProfile, ReliabilityPolicy +from ros2_medkit_msgs.msg import Fault, FaultEvent +from ros2_medkit_msgs.srv import GetFault, ListFaults, ReportFault, SetPlannedStop + +STORAGE_DIR = tempfile.mkdtemp(prefix='planned_stop_auto_confirm_') +DATABASE_PATH = os.path.join(STORAGE_DIR, 'faults.db') + +# One FAILED report leaves the fault PREFAILED; the timer is then the only thing +# that can confirm it. +CONFIRMATION_THRESHOLD = -2 +AUTO_CONFIRM_AFTER_SEC = 2.0 + +MUTED_CODE = 'PS_TIMER_MUTED' +CONTROL_CODE = 'PS_TIMER_CONTROL' + + +def get_coverage_env(): + """Get environment variables for gcov coverage data collection.""" + try: + from ament_index_python.packages import get_package_prefix + pkg_prefix = get_package_prefix('ros2_medkit_fault_manager') + workspace = os.path.dirname(os.path.dirname(pkg_prefix)) + build_dir = os.path.join(workspace, 'build', 'ros2_medkit_fault_manager') + + if os.path.exists(build_dir): + return { + 'GCOV_PREFIX': build_dir, + 'GCOV_PREFIX_STRIP': str(build_dir.count(os.sep)), + } + except Exception: + # Coverage environment is optional; on any error, fall back to no extra config + pass + return {} + + +def generate_test_description(): + """Launch a fault manager that auto-confirms lingering PREFAILED faults.""" + fault_manager_env = get_coverage_env() + fault_manager_env['ROS_LOCALHOST_ONLY'] = '1' + + fault_manager_node = launch_ros.actions.Node( + package='ros2_medkit_fault_manager', + executable='fault_manager_node', + name='fault_manager', + output='screen', + additional_env=fault_manager_env, + parameters=[{ + 'storage_type': 'sqlite', + 'database_path': DATABASE_PATH, + 'confirmation_threshold': CONFIRMATION_THRESHOLD, + 'auto_confirm_after_sec': AUTO_CONFIRM_AFTER_SEC, + 'snapshots.enabled': False, + 'snapshots.rosbag.enabled': False, + }], + sigterm_timeout='30', + sigkill_timeout='15', + ) + + return ( + LaunchDescription([ + launch.actions.TimerAction(period=2.0, actions=[fault_manager_node]), + launch_testing.actions.ReadyToTest(), + ]), + {'fault_manager_node': fault_manager_node}, + ) + + +class TestPlannedStopAutoConfirm(unittest.TestCase): + """The timer path honours the stop, and announces once it is withdrawn.""" + + @classmethod + def setUpClass(cls): + os.environ['ROS_LOCALHOST_ONLY'] = '1' + rclpy.init() + cls.node = Node('test_planned_stop_auto_confirm_client') + + cls.report_client = cls.node.create_client(ReportFault, '/fault_manager/report_fault') + cls.list_client = cls.node.create_client(ListFaults, '/fault_manager/list_faults') + cls.get_client = cls.node.create_client(GetFault, '/fault_manager/get_fault') + cls.set_stop_client = cls.node.create_client( + SetPlannedStop, '/fault_manager/set_planned_stop') + for client, name in ( + (cls.report_client, 'report_fault'), + (cls.list_client, 'list_faults'), + (cls.get_client, 'get_fault'), + (cls.set_stop_client, 'set_planned_stop'), + ): + assert client.wait_for_service(timeout_sec=20.0), f'{name} service not available' + + cls._events = [] + cls._events_lock = threading.Lock() + cls._event_sub = cls.node.create_subscription( + FaultEvent, '/fault_manager/events', cls._on_event, + QoSProfile(reliability=ReliabilityPolicy.RELIABLE, + history=HistoryPolicy.KEEP_LAST, depth=200)) + + cls._spinning = True + cls._spin_thread = threading.Thread(target=cls._spin, daemon=True) + cls._spin_thread.start() + cls._prime_event_stream() + + @classmethod + def tearDownClass(cls): + cls._spinning = False + cls._spin_thread.join(timeout=5) + cls.node.destroy_node() + rclpy.shutdown() + + @classmethod + def _on_event(cls, msg): + with cls._events_lock: + cls._events.append((msg.event_type, msg.fault.fault_code)) + + @classmethod + def _spin(cls): + while cls._spinning: + rclpy.spin_once(cls.node, timeout_sec=0.05) + + @classmethod + def _call(cls, client, request, timeout_sec=15.0): + future = client.call_async(request) + deadline = time.monotonic() + timeout_sec + while not future.done() and time.monotonic() < deadline: + time.sleep(0.005) + assert future.done(), 'service call timed out' + return future.result() + + @classmethod + def _report(cls, fault_code): + request = ReportFault.Request() + request.fault_code = fault_code + request.event_type = ReportFault.Request.EVENT_FAILED + request.severity = Fault.SEVERITY_ERROR + request.description = 'auto confirm test fault' + request.source_id = '/test_node' + return cls._call(cls.report_client, request) + + @classmethod + def _count_events(cls, fault_code, event_type): + with cls._events_lock: + return sum(1 for kind, code in cls._events + if code == fault_code and kind == event_type) + + @classmethod + def _prime_event_stream(cls): + """Block until an event demonstrably reaches this subscriber.""" + deadline = time.monotonic() + 40.0 + attempt = 0 + while time.monotonic() < deadline: + code = f'PS_TIMER_PRIME_{attempt}' + cls._report(code) + cls._report(code) + settle = time.monotonic() + 1.0 + while time.monotonic() < settle: + if cls._count_events(code, FaultEvent.EVENT_CONFIRMED) > 0: + return + time.sleep(0.05) + attempt += 1 + raise AssertionError('no event reached the subscriber; the events topic never went live') + + def _set_stop(self, active, *, reason='', declared_by=''): + request = SetPlannedStop.Request() + request.active = active + request.reason = reason + request.declared_by = declared_by + return self._call(self.set_stop_client, request) + + def _status(self, fault_code): + request = GetFault.Request() + request.fault_code = fault_code + response = self._call(self.get_client, request) + return response.fault.status if response.success else None + + def _muted_codes(self): + request = ListFaults.Request() + request.filter_by_severity = False + request.severity = 0 + request.statuses = [] + request.include_muted = True + request.include_clusters = False + return {info.fault_code for info in self._call(self.list_client, request).muted_faults} + + def _wait_until(self, predicate, message, timeout=30.0): + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + value = predicate() + if value: + return value + time.sleep(0.1) + raise AssertionError(message) + + def test_a_timer_confirmation_inside_a_stop_is_held_back_until_it_ends(self): + # The control fault proves the timer really fires in this configuration: + # without it, silence for the muted code would prove nothing. + self._report(CONTROL_CODE) + self._wait_until(lambda: self._status(CONTROL_CODE) == 'CONFIRMED', + 'the auto-confirm timer never confirmed the control fault, so the ' + 'muted case below would prove nothing') + self.assertEqual(1, self._count_events(CONTROL_CODE, FaultEvent.EVENT_CONFIRMED)) + + self.assertTrue(self._set_stop(True, reason='timer case', declared_by='tech').success) + + self._report(MUTED_CODE) + self.assertIn(MUTED_CODE, self._muted_codes()) + + # The timer confirms it in the store while the stop stands. + self._wait_until(lambda: self._status(MUTED_CODE) == 'CONFIRMED', + 'the auto-confirm timer never confirmed the muted fault') + self.assertEqual( + 0, self._count_events(MUTED_CODE, FaultEvent.EVENT_CONFIRMED), + 'the timer announced a confirmation the planned stop was holding back') + + withdrawn = self._set_stop(False, reason='timer case over', declared_by='tech') + self.assertTrue(withdrawn.success) + + self._wait_until( + lambda: self._count_events(MUTED_CODE, FaultEvent.EVENT_CONFIRMED) == 1, + 'the timer-driven confirmation was never announced after the stop ended') + self.assertNotIn(MUTED_CODE, self._muted_codes()) + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + for info in proc_info: + self.assertIn(info.returncode, (0, -2, -15), + f'{info.process_name} exited with code {info.returncode}') diff --git a/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_restart.test.py b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_restart.test.py new file mode 100644 index 000000000..29377d4e3 --- /dev/null +++ b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_restart.test.py @@ -0,0 +1,239 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +A planned stop declared before a restart is still in force after it. + +The subject IS the restart, so the fault manager runs with ``respawn`` and the +test kills it by PID: a weekend stop must not end because the box rebooted. +Runs on SQLite, the shipped default and the only backend with a file behind it. +""" + +import os +import signal +import tempfile +import time +import unittest + +from launch import LaunchDescription +import launch.actions +import launch_ros.actions +import launch_testing.actions +import rclpy +from rclpy.node import Node +from ros2_medkit_msgs.msg import Fault +from ros2_medkit_msgs.srv import GetFault, GetPlannedStop, ListFaults, ReportFault, SetPlannedStop + +STORAGE_DIR = tempfile.mkdtemp(prefix='planned_stop_restart_') +DATABASE_PATH = os.path.join(STORAGE_DIR, 'faults.db') + +REASON = 'weekend shutdown, cell 4' +DECLARED_BY = 'plant_manager' +BEFORE_CODE = 'PS_RESTART_BEFORE' +AFTER_CODE = 'PS_RESTART_AFTER' + + +def get_coverage_env(): + """Get environment variables for gcov coverage data collection.""" + try: + from ament_index_python.packages import get_package_prefix + pkg_prefix = get_package_prefix('ros2_medkit_fault_manager') + workspace = os.path.dirname(os.path.dirname(pkg_prefix)) + build_dir = os.path.join(workspace, 'build', 'ros2_medkit_fault_manager') + + if os.path.exists(build_dir): + return { + 'GCOV_PREFIX': build_dir, + 'GCOV_PREFIX_STRIP': str(build_dir.count(os.sep)), + } + except Exception: + # Coverage environment is optional; on any error, fall back to no extra config + pass + return {} + + +def generate_test_description(): + """Launch a fault manager that launch brings back after it is killed.""" + fault_manager_env = get_coverage_env() + fault_manager_env['ROS_LOCALHOST_ONLY'] = '1' + + fault_manager_node = launch_ros.actions.Node( + package='ros2_medkit_fault_manager', + executable='fault_manager_node', + name='fault_manager', + output='screen', + additional_env=fault_manager_env, + parameters=[{ + 'storage_type': 'sqlite', + 'database_path': DATABASE_PATH, + 'confirmation_threshold': -1, + 'snapshots.enabled': False, + 'snapshots.rosbag.enabled': False, + }], + # The restart is the subject. The delay gives the DDS participant time to + # go before its replacement appears, so the test sees a real gap rather + # than two managers on one domain. + respawn=True, + respawn_delay=1.0, + sigterm_timeout='30', + sigkill_timeout='15', + ) + + return ( + LaunchDescription([ + launch.actions.TimerAction(period=2.0, actions=[fault_manager_node]), + launch_testing.actions.ReadyToTest(), + ]), + {'fault_manager_node': fault_manager_node}, + ) + + +class TestPlannedStopSurvivesRestart(unittest.TestCase): + """The declaration outlives the process that made it.""" + + @classmethod + def setUpClass(cls): + os.environ['ROS_LOCALHOST_ONLY'] = '1' + rclpy.init() + cls.node = Node('test_planned_stop_restart_client') + cls._connect() + + @classmethod + def tearDownClass(cls): + cls.node.destroy_node() + rclpy.shutdown() + + @classmethod + def _connect(cls, timeout_sec=30.0): + """Create clients and wait for the manager's services to answer.""" + cls.report_client = cls.node.create_client(ReportFault, '/fault_manager/report_fault') + cls.list_client = cls.node.create_client(ListFaults, '/fault_manager/list_faults') + cls.get_client = cls.node.create_client(GetFault, '/fault_manager/get_fault') + cls.set_stop_client = cls.node.create_client( + SetPlannedStop, '/fault_manager/set_planned_stop') + cls.get_stop_client = cls.node.create_client( + GetPlannedStop, '/fault_manager/get_planned_stop') + for client, name in ( + (cls.report_client, 'report_fault'), + (cls.list_client, 'list_faults'), + (cls.get_client, 'get_fault'), + (cls.set_stop_client, 'set_planned_stop'), + (cls.get_stop_client, 'get_planned_stop'), + ): + assert client.wait_for_service(timeout_sec=timeout_sec), \ + f'{name} service not available' + + def _call(self, client, request, timeout_sec=15.0): + future = client.call_async(request) + rclpy.spin_until_future_complete(self.node, future, timeout_sec=timeout_sec) + self.assertIsNotNone(future.result(), 'service call timed out') + return future.result() + + def _report(self, fault_code): + request = ReportFault.Request() + request.fault_code = fault_code + request.event_type = ReportFault.Request.EVENT_FAILED + request.severity = Fault.SEVERITY_ERROR + request.description = 'restart test fault' + request.source_id = '/test_node' + return self._call(self.report_client, request) + + def _set_stop(self, active, *, reason='', declared_by=''): + request = SetPlannedStop.Request() + request.active = active + request.reason = reason + request.declared_by = declared_by + return self._call(self.set_stop_client, request) + + def _get_stop(self): + return self._call(self.get_stop_client, GetPlannedStop.Request()) + + def _muted_codes(self): + request = ListFaults.Request() + request.filter_by_severity = False + request.severity = 0 + request.statuses = [] + request.include_muted = True + request.include_clusters = False + response = self._call(self.list_client, request) + return {info.fault_code: info for info in response.muted_faults} + + def _confirmed_codes(self): + request = ListFaults.Request() + request.filter_by_severity = False + request.severity = 0 + request.statuses = [] + request.include_muted = False + request.include_clusters = False + return {fault.fault_code for fault in self._call(self.list_client, request).faults} + + def test_a_stop_declared_before_a_restart_still_mutes_after_it(self, fault_manager_node): + declared = self._set_stop(True, reason=REASON, declared_by=DECLARED_BY) + self.assertTrue(declared.success) + self.assertFalse(declared.was_active) + + before = self._get_stop() + self.assertTrue(before.active) + self.assertEqual(REASON, before.reason) + + self._report(BEFORE_CODE) + self.assertIn(BEFORE_CODE, self._muted_codes()) + + original_pid = fault_manager_node.process_details['pid'] + os.kill(original_pid, signal.SIGTERM) + + deadline = time.monotonic() + 60.0 + while time.monotonic() < deadline: + current = fault_manager_node.process_details + if current is not None and current['pid'] != original_pid: + break + time.sleep(0.2) + self.assertNotEqual( + original_pid, fault_manager_node.process_details['pid'], + 'the fault manager was never restarted, so nothing below tests a restart') + + self._connect(timeout_sec=60.0) + + after = self._get_stop() + self.assertTrue(after.active, 'the planned stop did not survive the restart') + self.assertEqual(REASON, after.reason) + self.assertEqual(DECLARED_BY, after.declared_by) + self.assertEqual(before.since.sec, after.since.sec, + 'the restart moved the time the stop started') + self.assertEqual(before.since.nanosec, after.since.nanosec) + + # The declaration is not a museum piece: a fault reported after the + # restart is muted by it. + self._report(AFTER_CODE) + muted = self._muted_codes() + self.assertIn(AFTER_CODE, muted, + 'a fault reported after the restart was not muted by the standing stop') + self.assertEqual('planned_stop', muted[AFTER_CODE].rule_id) + self.assertNotIn(AFTER_CODE, self._confirmed_codes()) + + withdrawn = self._set_stop(False, reason='plant back up', declared_by=DECLARED_BY) + self.assertTrue(withdrawn.success) + self.assertTrue(withdrawn.was_active) + self.assertFalse(self._get_stop().active) + self.assertIn(AFTER_CODE, self._confirmed_codes()) + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + for info in proc_info: + self.assertIn(info.returncode, (0, -2, -15), + f'{info.process_name} exited with code {info.returncode}') diff --git a/src/ros2_medkit_fault_manager/test/test_fault_audit_log.cpp b/src/ros2_medkit_fault_manager/test/test_fault_audit_log.cpp index 2107aab71..bf0f31fc5 100644 --- a/src/ros2_medkit_fault_manager/test/test_fault_audit_log.cpp +++ b/src/ros2_medkit_fault_manager/test/test_fault_audit_log.cpp @@ -527,6 +527,48 @@ TEST_F(FaultAuditLogTest, AnchorTriggersRejectOutOfBandWrites) { EXPECT_TRUE(reopened.verify().ok); } +// A planned-stop transition belongs to the installation, not to a fault: it +// carries no fault code, and the chain has to survive that. +TEST_F(FaultAuditLogTest, PlannedStopTransitionsChainWithNoFaultCode) { + FaultAuditLog log(path_); + + AuditEvent started; + started.fault_code = ""; + started.transition = ros2_medkit_fault_manager::kTransitionPlannedStopStarted; + started.status = "ACTIVE"; + started.source_id = "shift_lead"; + started.description = "line 3 quarterly maintenance"; + started.occurred_at_ns = 1000; + EXPECT_EQ(log.append(started), 1); + + AuditEvent ended; + ended.fault_code = ""; + ended.transition = ros2_medkit_fault_manager::kTransitionPlannedStopEnded; + ended.status = "INACTIVE"; + ended.source_id = "shift_lead"; + ended.description = "line 3 back up"; + ended.occurred_at_ns = 2000; + EXPECT_EQ(log.append(ended), 2); + + EXPECT_TRUE(log.verify().ok); + + const auto records = log.read(); + ASSERT_EQ(records.size(), 2u); + EXPECT_EQ(records[0].event.transition, "planned_stop_started"); + EXPECT_EQ(records[0].event.fault_code, ""); + EXPECT_EQ(records[0].event.source_id, "shift_lead"); + EXPECT_EQ(records[0].event.description, "line 3 quarterly maintenance"); + EXPECT_EQ(records[1].event.transition, "planned_stop_ended"); + EXPECT_EQ(records[1].event.status, "INACTIVE"); + EXPECT_EQ(records[1].prev_hash, records[0].record_hash); + + // The declaration is part of the hashed record, not a side note: rewriting the + // reason without recomputing the chain has to be detectable. + raw_exec(path_, std::string(kDropTriggers) + "UPDATE audit_log SET description = 'unplanned outage' WHERE seq = 1;"); + FaultAuditLog reopened(path_); + EXPECT_FALSE(reopened.verify().ok); +} + int main(int argc, char ** argv) { ::testing::InitGoogleTest(&argc, argv); return RUN_ALL_TESTS(); diff --git a/src/ros2_medkit_integration_tests/test/features/test_planned_stop_switch.test.py b/src/ros2_medkit_integration_tests/test/features/test_planned_stop_switch.test.py new file mode 100644 index 000000000..eb0f0a6a9 --- /dev/null +++ b/src/ros2_medkit_integration_tests/test/features/test_planned_stop_switch.test.py @@ -0,0 +1,330 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""The planned-stop switch driven the way an operator reaches it. + +No route was added for the switch: the fault manager's services are SOVD +operations on its own App entity, so an operator turns the stop on with +``POST /apps/fault_manager/operations/set_planned_stop/executions``. This test +drives it that way, watches ``GET /faults`` and the ``/faults/stream`` SSE feed, +and asserts what an operator sees: nothing about the faults raised during the +stop until it ends, then every one of them at once. + +The stream instrument counts ``fault_confirmed`` frames PER FAULT CODE, before +and after the switch-off. A total alone would pass while the wrong fault was +announced. +""" + +import json +import threading +import time +import unittest + +import launch_testing +import launch_testing.actions +import rclpy +from rclpy.node import Node +import requests +from ros2_medkit_msgs.msg import Fault +from ros2_medkit_msgs.srv import ReportFault + +from ros2_medkit_test_utils.constants import ALLOWED_EXIT_CODES, FAULT_TIMEOUT +from ros2_medkit_test_utils.gateway_test_case import GatewayTestCase +from ros2_medkit_test_utils.launch_helpers import create_test_launch + +SOURCE_ID = '/powertrain/engine/temp_sensor' +MUTED_CODE = 'GW_PS_MUTED' +SECOND_CODE = 'GW_PS_MUTED_TWO' +AFTER_CODE = 'GW_PS_AFTER' + +SET_STOP = '/apps/fault_manager/operations/set_planned_stop/executions' +GET_STOP = '/apps/fault_manager/operations/get_planned_stop/executions' + + +def generate_test_description(): + return create_test_launch( + demo_nodes=['temp_sensor'], + fault_manager=True, + fault_manager_params={ + 'storage_type': 'memory', + 'confirmation_threshold': -1, + 'snapshots.rosbag.enabled': False, + }, + ) + + +class TestPlannedStopSwitch(GatewayTestCase): + """An operator declares a stop over SOVD and the fault view obeys it.""" + + MIN_EXPECTED_APPS = 2 + # The fault manager is an App like any other node, which is exactly why the + # switch needs no route of its own. + REQUIRED_APPS = {'temp_sensor', 'fault_manager'} + + @classmethod + def setUpClass(cls): + super().setUpClass() + rclpy.init() + cls._reporter = Node('planned_stop_switch_reporter') + cls._report_client = cls._reporter.create_client( + ReportFault, '/fault_manager/report_fault') + assert cls._report_client.wait_for_service(timeout_sec=20.0), \ + 'report_fault service not available' + + # One stream for the whole class. Frames accumulate across the cases and + # every assertion counts them per fault code, so there is nothing to + # reset between tests - and a stream reopened per test costs far more + # than it proves, because the gateway only notices a closed SSE client + # on its next write. + cls._frames = [] + cls._stop_event = threading.Event() + cls._response = requests.get( + f'{cls.BASE_URL}/faults/stream', stream=True, timeout=(5, 300)) + assert cls._response.status_code == 200, 'the fault stream did not open' + cls._pump = threading.Thread( + target=cls._pump_stream, args=(cls._response, cls._frames, cls._stop_event), + daemon=True) + cls._pump.start() + cls._prime_stream() + + @classmethod + def tearDownClass(cls): + cls._stop_event.set() + cls._response.close() + cls._pump.join(timeout=10) + cls._reporter.destroy_node() + rclpy.shutdown() + + def tearDown(self): + # One declaration per manager: a test that left it on would decide the + # next one's outcome. + self._set_stop(False, reason='teardown', declared_by='test') + + # ------------------------------------------------------------------ + # Helpers + # ------------------------------------------------------------------ + + @classmethod + def _report(cls, fault_code): + request = ReportFault.Request() + request.fault_code = fault_code + request.event_type = ReportFault.Request.EVENT_FAILED + request.severity = Fault.SEVERITY_ERROR + request.description = 'planned stop switch test fault' + request.source_id = SOURCE_ID + future = cls._report_client.call_async(request) + deadline = time.monotonic() + 10.0 + while not future.done() and time.monotonic() < deadline: + rclpy.spin_once(cls._reporter, timeout_sec=0.05) + assert future.done(), 'report_fault call timed out' + assert future.result().accepted, 'report_fault was rejected' + + @staticmethod + def _pump_stream(response, frames, stop_event): + """Collect SSE frames as dicts of field -> value.""" + current = {} + try: + for line in response.iter_lines(decode_unicode=True): + if stop_event.is_set(): + break + if line is None: + continue + if line == '': + if current: + frames.append(current) + current = {} + continue + if line.startswith(':'): + continue # keepalive comment + key, _, value = line.partition(':') + current[key.strip()] = value.strip() + except Exception: # noqa: BLE001 - closed socket on test teardown + pass + + @classmethod + def _prime_stream(cls): + """Block until the fault event pipeline demonstrably reaches the stream. + + ``/fault_manager/events`` is reliable but volatile: an event published + before the gateway's subscription has matched the publisher is lost, and + a one-shot assertion would race that match. + """ + deadline = time.monotonic() + FAULT_TIMEOUT + attempt = 0 + while time.monotonic() < deadline: + # A fresh code per attempt: re-reporting one that is already + # CONFIRMED publishes fault_updated, never fault_confirmed. + code = f'GW_PS_PRIME_{attempt}' + cls._report(code) + settle = min(time.monotonic() + 1.0, deadline) + while time.monotonic() < settle: + if cls._count_confirmed(code) > 0: + return + time.sleep(0.1) + attempt += 1 + raise AssertionError( + f'no priming event on /faults/stream within {FAULT_TIMEOUT}s; ' + 'the events pipeline never went live') + + @classmethod + def _count_confirmed(cls, fault_code): + count = 0 + for frame in list(cls._frames): + if frame.get('event') != 'fault_confirmed' or 'data' not in frame: + continue + payload = json.loads(frame['data']) + if payload.get('fault', {}).get('fault_code') == fault_code: + count += 1 + return count + + def _set_stop(self, active, *, reason='', declared_by=''): + response = requests.post( + f'{self.BASE_URL}{SET_STOP}', + json={'parameters': {'active': active, 'reason': reason, + 'declared_by': declared_by}}, + timeout=20) + self.assertEqual(200, response.status_code, response.text) + return response.json()['parameters'] + + def _get_stop(self): + response = requests.post(f'{self.BASE_URL}{GET_STOP}', json={}, timeout=20) + self.assertEqual(200, response.status_code, response.text) + return response.json()['parameters'] + + def _faults(self, *, include_muted=False): + """Return the fault list an operator sees, and the set of codes in it. + + No ``status`` filter: the default view (pending + confirmed) is the one + the switch is supposed to change. + """ + params = {'include_muted': 'true'} if include_muted else {} + response = requests.get(f'{self.BASE_URL}/faults', params=params, timeout=15) + self.assertEqual(200, response.status_code, response.text) + data = response.json() + return data, {item['fault_code'] for item in data.get('items', [])} + + def _wait_until(self, predicate, message, timeout=None): + deadline = time.monotonic() + (timeout if timeout is not None else FAULT_TIMEOUT) + while time.monotonic() < deadline: + value = predicate() + if value: + return value + time.sleep(0.2) + raise AssertionError(message) + + # ------------------------------------------------------------------ + # Cases + # ------------------------------------------------------------------ + + def test_the_switch_is_an_operation_on_the_fault_manager_entity(self): + """The switch is reachable as an operation on the fault manager entity. + + @verifies REQ_INTEROP_035 + """ + data = self.get_json('/apps/fault_manager/operations') + operation_ids = {item['id'] for item in data.get('items', [])} + self.assertIn('set_planned_stop', operation_ids) + self.assertIn('get_planned_stop', operation_ids) + + state = self._get_stop() + self.assertFalse(state['active']) + + def test_faults_raised_during_a_stop_surface_only_when_it_ends(self): + """Faults raised inside a stop stay hidden until it is withdrawn. + + @verifies REQ_INTEROP_012 + """ + declared = self._set_stop(True, reason='line 3 maintenance', declared_by='shift_lead') + self.assertTrue(declared['success']) + self.assertFalse(declared['was_active']) + + state = self._get_stop() + self.assertTrue(state['active']) + self.assertEqual('line 3 maintenance', state['reason']) + self.assertEqual('shift_lead', state['declared_by']) + + before_muted = self._faults()[0]['x-medkit']['muted_count'] + + self._report(MUTED_CODE) + self._report(SECOND_CODE) + + # The default view hides them; asking for muted entries shows them, with + # the switch named as the reason. + def _both_muted(): + data, codes = self._faults(include_muted=True) + muted = {entry['fault_code']: entry + for entry in data['x-medkit'].get('muted_faults', [])} + if MUTED_CODE not in muted or SECOND_CODE not in muted: + return None + # Asking for muted entries also stops them being stripped from items. + self.assertIn(MUTED_CODE, codes) + self.assertIn(SECOND_CODE, codes) + return muted + + muted = self._wait_until(_both_muted, 'the faults raised during the stop were not marked') + for code in (MUTED_CODE, SECOND_CODE): + self.assertEqual('planned_stop', muted[code]['rule_id']) + self.assertEqual('PLANNED_STOP', muted[code]['root_cause_code']) + + data, visible = self._faults() + self.assertNotIn(MUTED_CODE, visible) + self.assertNotIn(SECOND_CODE, visible) + self.assertGreaterEqual(data['x-medkit']['muted_count'], before_muted + 2) + + # And the stream said nothing about either of them. + self.assertEqual(0, self._count_confirmed(MUTED_CODE)) + self.assertEqual(0, self._count_confirmed(SECOND_CODE)) + + withdrawn = self._set_stop(False, reason='line 3 back up', declared_by='shift_lead') + self.assertTrue(withdrawn['success']) + self.assertTrue(withdrawn['was_active']) + + self._wait_until( + lambda: all(code in self._faults()[1] for code in (MUTED_CODE, SECOND_CODE)), + 'the faults the stop muted never appeared in the default list after it ended') + self._wait_until( + lambda: self._count_confirmed(MUTED_CODE) == 1 + and self._count_confirmed(SECOND_CODE) == 1, + 'the stream did not carry exactly one confirmation per released fault') + + state = self._get_stop() + self.assertFalse(state['active']) + self.assertEqual('', state['reason']) + + def test_a_fault_raised_after_the_stop_is_announced_at_once(self): + """A fault raised once the stop is over is announced immediately. + + @verifies REQ_INTEROP_012 + """ + self._set_stop(True, reason='short pause', declared_by='tech') + self._set_stop(False, reason='pause over', declared_by='tech') + + self._report(AFTER_CODE) + + self._wait_until(lambda: self._count_confirmed(AFTER_CODE) == 1, + 'a fault raised after the stop ended was not announced') + self.assertIn(AFTER_CODE, self._faults()[1]) + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + """All processes exited cleanly (SIGTERM allowed for SSE teardown).""" + for info in proc_info: + self.assertIn( + info.returncode, ALLOWED_EXIT_CODES, + f'{info.process_name} exited with code {info.returncode}' + ) From 707b5afdf3df4abde04341ad5d86b9b840f761eb Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Mon, 7 Sep 2026 16:55:25 +0200 Subject: [PATCH 04/12] docs: describe the planned-stop switch Covers what marking means and what it does not change, how the switch and a correlation rule compose, which two configuration choices decide whether the declaration survives a restart and whether transitions are audited, and how an operator reaches the switch over SOVD as an operation on the fault manager entity - including that hybrid and manifest discovery need the entity declared before that address exists. Refs #656 --- docs/api/messages.rst | 34 ++++++++++ docs/config/fault-manager.rst | 54 +++++++++++++++ src/ros2_medkit_fault_manager/README.md | 67 +++++++++++++++++++ .../design/index.rst | 39 +++++++++++ 4 files changed, 194 insertions(+) diff --git a/docs/api/messages.rst b/docs/api/messages.rst index 1b796d66e..92fb16b80 100644 --- a/docs/api/messages.rst +++ b/docs/api/messages.rst @@ -294,6 +294,40 @@ Retrieve diagnostic snapshots captured at fault occurrence time. See :doc:`/tutorials/snapshots` for detailed usage. +SetPlannedStop.srv / GetPlannedStop.srv +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Declare, withdraw and read a planned stop on the fault manager. + +.. code-block:: text + + # SetPlannedStop.srv + bool active # true declares a planned stop, false withdraws it + string reason # why the plant is stopped + string declared_by # who declared the transition + --- + bool success # true when the request was applied + string message # status or error description + bool was_active # the state of the switch before this request + + # GetPlannedStop.srv + --- + bool active # whether a planned stop is declared + string reason # the reason given; empty while none is declared + string declared_by # who declared it; empty while none is declared + builtin_interfaces/Time since # when it was declared; zero while none is + +While a planned stop is on, a fault whose cycle starts is reported, debounced, +confirmed, captured and audited unchanged, and is marked as muted: absent from +the default ``ListFaults`` response, counted in ``muted_count``, listed under +``muted_faults`` with ``rule_id: planned_stop`` when ``include_muted`` is set, +and never published as a ``FaultEvent``. Withdrawing the stop releases every +fault it alone was muting and publishes one ``EVENT_CONFIRMED`` per released +fault that is CONFIRMED. + +See :doc:`/config/fault-manager` for the configuration that decides whether the +declaration survives a restart and whether the transitions are audited. + MedkitDiscoveryHint.msg ~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/config/fault-manager.rst b/docs/config/fault-manager.rst index 400de56c0..1ec74bb0c 100644 --- a/docs/config/fault-manager.rst +++ b/docs/config/fault-manager.rst @@ -614,6 +614,60 @@ Fault correlation identifies root causes and filters symptom faults. :doc:`/tutorials/fault-correlation` for correlation rule syntax and examples. +Planned Stop +------------ + +A planned stop is declared at runtime, not configured: an operator calls +``~/set_planned_stop`` (or the matching SOVD operation on the ``fault_manager`` +entity) with a reason and a declarer, and calls it again with ``active: false`` +when the plant is back up. There is no parameter for it - a stop has a reason and +an audit trail, and a parameter has neither. + +.. code-block:: bash + + ros2 service call /fault_manager/set_planned_stop ros2_medkit_msgs/srv/SetPlannedStop \ + "{active: true, reason: 'line 3 quarterly maintenance', declared_by: 'shift_lead'}" + +Two configuration choices decide how much the switch can do: + +.. list-table:: + :header-rows: 1 + :widths: 35 15 50 + + * - Parameter + - Default + - Effect on the planned stop + * - ``storage_type`` + - ``sqlite`` + - The declaration is stored beside the faults, so it survives a restart. + With ``memory`` it lives only for the process, and a restart inside a stop + ends it. + * - ``audit_log.enabled`` + - ``false`` + - When on, each transition is recorded as ``planned_stop_started`` / + ``planned_stop_ended``, with the reason and the declarer. Recorded even + under ``audit_log.transitions: confirmed_only``. + +While the stop stands, a fault whose cycle starts is reported, debounced, +confirmed, captured and audited unchanged, and is marked as muted rather than +dropped: absent from the default fault list, counted in ``muted_count``, listed +under ``muted_faults`` with ``rule_id: planned_stop`` when muted entries are +requested, and never published on ``~/events``. Withdrawing the stop releases +every fault it alone was muting and publishes one confirmation event for each of +those that is CONFIRMED. Correlation rules are unaffected: a symptom a rule mutes +stays muted when the stop is withdrawn. + +Over HTTP the switch is an operation on the fault manager's App entity +(``POST /apps/fault_manager/operations/set_planned_stop/executions``), which +exists automatically in ``runtime_only`` discovery. Under ``hybrid`` or +``manifest_only`` the fault manager must be declared in the manifest, or the +entity is not there to address. + +.. seealso:: + + :doc:`/design/ros2_medkit_fault_manager/index` for how the switch composes + with correlation rules. + Complete Example ---------------- diff --git a/src/ros2_medkit_fault_manager/README.md b/src/ros2_medkit_fault_manager/README.md index c12e54c8a..e1a275b98 100644 --- a/src/ros2_medkit_fault_manager/README.md +++ b/src/ros2_medkit_fault_manager/README.md @@ -43,6 +43,72 @@ ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \ | `~/list_faults` | `ros2_medkit_msgs/srv/ListFaults` | Query faults with filtering | | `~/clear_fault` | `ros2_medkit_msgs/srv/ClearFault` | Clear/acknowledge a fault | | `~/get_snapshots` | `ros2_medkit_msgs/srv/GetSnapshots` | Get topic snapshots for a fault | +| `~/set_planned_stop` | `ros2_medkit_msgs/srv/SetPlannedStop` | Declare or withdraw a planned stop | +| `~/get_planned_stop` | `ros2_medkit_msgs/srv/GetPlannedStop` | Read the planned-stop declaration | + +## Planned Stop + +Maintenance produces faults that nobody wants paged: a robot on a bench raises +`NODE_UNREACHABLE` for every peer it can no longer see, and the alarm reaches +whoever is on call. The planned stop is the operator saying "this is us, not the +plant". + +```bash +ros2 service call /fault_manager/set_planned_stop ros2_medkit_msgs/srv/SetPlannedStop \ + "{active: true, reason: 'line 3 quarterly maintenance', declared_by: 'shift_lead'}" + +ros2 service call /fault_manager/get_planned_stop ros2_medkit_msgs/srv/GetPlannedStop "{}" +``` + +**Marked, never dropped.** While the stop stands, a fault whose cycle starts goes +through report, debounce, confirmation, snapshot and rosbag capture and the audit +log exactly as it would otherwise. What changes is only how it is *shown*: it is +registered as muted with `rule_id: planned_stop` and the pseudo root cause +`PLANNED_STOP`, so it is absent from the default `~/list_faults` response, +counted in `muted_count`, listed under `muted_faults` when `include_muted` is +set, and never published on `~/events`. + +**Withdrawing releases the survivors.** Every fault the stop alone was muting and +that is still active is unmuted, and each one that is CONFIRMED publishes a single +`EVENT_CONFIRMED` - that confirmation happened behind the mute and was never +announced, while the condition it reports still stands on the machine. A fault +still short of confirmation announces nothing (there is nothing yet to announce), +and one acknowledged during the stop announces nothing either. + +**Correlation rules and the stop compose.** A fault a rule mutes stays muted by +that rule, and withdrawing the stop does not release it. A fault the stop muted +that a rule later claims likewise stays muted after the withdrawal: two +independent reasons to be quiet, and one of them going away is not both. + +**Both transitions are audited** as `planned_stop_started` and +`planned_stop_ended`, carrying the reason in `description` and the declarer in +`source_id`, with an empty `fault_code` because the transition is about the +installation rather than one fault. They are recorded whatever +`audit_log.transitions` says, like the log's own activation markers. A request +asking for the state the switch is already in succeeds, changes nothing and +records nothing, so a retried call cannot manufacture evidence of a stop that +never started. + +**The declaration survives a restart** when the SQLite backend is in use: it is a +row in the fault store, so a stop declared on Friday still mutes on Monday. The +in-memory backend has no file behind it and starts every process with no +declaration. Faults muted *before* a restart are not restored - their mute lived +in the process that is gone - so a restart inside a stop makes those visible +again while every fault reported after it is muted afresh. + +**Over SOVD.** The gateway maps a node's services to operations on its App entity, +so an operator reaches the switch without any new route: + +```bash +curl -X POST http://localhost:8080/api/v1/apps/fault_manager/operations/set_planned_stop/executions \ + -H 'Content-Type: application/json' \ + -d '{"parameters": {"active": true, "reason": "line 3 maintenance", "declared_by": "shift_lead"}}' +``` + +That works out of the box in `runtime_only` discovery, where every node is an +App. In `hybrid` or `manifest_only` discovery the fault manager has to be +declared in the manifest like any other entity, or there is no `fault_manager` +entity to address. ## Features @@ -55,6 +121,7 @@ ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \ - **Snapshot capture**: Captures topic data when faults are confirmed for debugging (snapshots are deleted when fault is cleared) - **Near-miss series**: Appends one entry per FAILED report that moved the debounce counter without confirming, bounded per fault code and retained when the fault is cleared - **Freeze-frame retention**: One compact JSON freeze-frame per fault code, retained across `clear_fault` (see below) +- **Planned stop**: An operator declares the plant deliberately down, and faults raised while it stands are marked rather than announced (see below) - **Fault correlation** (optional): Root cause analysis with symptom muting and auto-clear - **Tamper-evident audit log** (optional): Append-only, hash-chained record of fault state transitions for verifiable history diff --git a/src/ros2_medkit_fault_manager/design/index.rst b/src/ros2_medkit_fault_manager/design/index.rst index e37d62acb..40dd547d8 100644 --- a/src/ros2_medkit_fault_manager/design/index.rst +++ b/src/ros2_medkit_fault_manager/design/index.rst @@ -186,6 +186,17 @@ Clears (acknowledges) a fault by setting its status to CLEARED. - **Idempotent**: Clearing an already-cleared fault succeeds - **Returns**: ``success=true`` if fault existed, ``success=false`` if not found +~/set_planned_stop and ~/get_planned_stop +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Declares, withdraws and reads the planned stop. + +- **One declaration per manager**: the stop is a fact about the installation, not about a fault +- **Idempotent**: a request for the state the switch is already in succeeds, changes nothing and writes no audit record; ``was_active`` tells the caller which of the two happened +- **Audited**: each real transition appends ``planned_stop_started`` / ``planned_stop_ended`` with the reason and the declarer +- **Persisted**: stored in the fault store, so the declaration outlives the process (SQLite backend) +- **Returns**: ``success``, a message, and the state the switch was in before the call + Design Decisions ---------------- @@ -246,6 +257,34 @@ merged per-entity config at startup, logs a warning, and falls back to safe defa healing is disabled, any HEALED row left by a previous (healing-enabled) run is reclassified to CLEARED once at startup so it does not behave inconsistently under the latch. +Planned Stop as a Second Mute Source +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Maintenance produces faults that are real, correctly detected, and not news. The +planned stop marks them instead of dropping them: while it stands, a fault whose +cycle starts is registered in the correlation engine as muted with +``rule_id = "planned_stop"`` and the pseudo root cause ``PLANNED_STOP``, and +everything else about it - debounce, confirmation, snapshot and rosbag capture, +the audit records - happens exactly as it would otherwise. Dropping the report +instead was rejected because the bridges send one FAILED per transition: a fault +that outlived the stop would never be raised again. + +The engine keeps the codes the stop muted in a set of its own, separate from the +mute map, and that set is what makes the two sources compose. A code leaves it the +moment a rule claims the mute, so withdrawing the stop cannot release a symptom a +rule is still holding down; and it leaves on ``process_clear``, so a fault +acknowledged inside the stop is not handed back at the withdrawal. + +Withdrawing releases the rest and publishes ``EVENT_CONFIRMED`` once for each +released fault that is CONFIRMED. That republication is the point of marking +rather than dropping: the confirmation happened behind the mute, no consumer of +the event stream ever heard it, and the condition still stands on the machine. + +The switch is a service rather than a parameter because it carries a reason and a +declarer and produces an audit record, none of which a parameter can do; and it is +not a new HTTP route because the gateway already exposes a node's services as SOVD +operations on its App entity. + Rosbag Black-Box Recording ~~~~~~~~~~~~~~~~~~~~~~~~~~ From 08d6407126c6eeaaf1fecee3225ac82e0c3a9f86 Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Mon, 7 Sep 2026 17:55:25 +0200 Subject: [PATCH 05/12] fault_manager: mark only a cycle that starts inside the planned stop Reporters are level-triggered: FaultReporter and the diagnostic bridge send FAILED for as long as the condition holds, not once per transition. Muting on any report therefore took a fault that was confirmed and announced BEFORE the stop off the fault list within a second of the declaration, and announced it a second time at the switch-off. The mark is now gated on the cycle boundary the report path already computes: a new fault, or one raised again after a clear. A repeat report of a fault the stop already marked changes nothing. One mute now has one owner. The hand-over to a rule happens where the rule registers its mute, not where any path reports should_mute: the auto-cluster path sets that flag without ever writing the mute map, so the old hand-over left the stop's entry owned by nobody - muted for good, released by nothing. In the other direction the stop no longer overwrites an entry a rule holds. An acknowledgement that skips the correlation cascade, which is every scoped per-entity clear, now also releases the stop's mute. The mark survives a restart: with a declaration still in force the manager re-registers it for every stored fault that is not CLEARED and whose cycle started at or after the declaration, so the switch-off after a reboot still releases and announces them. A store that refuses the write ends the request with success=false instead of taking the process down, and the declaration is written before anything is muted, unmuted, announced or audited. GetPlannedStop keeps serving the declaration after the withdrawal, stamped with ended_at, so the reason stays readable once the plant is back up. The audit rows carry the same __audit__ fault code the log's own lifecycle markers use. Refs #656 --- docs/api/messages.rst | 25 ++- docs/config/fault-manager.rst | 37 +++- src/ros2_medkit_fault_manager/README.md | 62 ++++-- .../design/index.rst | 49 ++++- .../correlation/correlation_engine.hpp | 23 ++- .../fault_audit_log.hpp | 6 + .../fault_manager_node.hpp | 18 +- .../fault_storage.hpp | 9 +- .../src/correlation/correlation_engine.cpp | 70 +++++-- .../src/fault_manager_node.cpp | 108 +++++++--- .../src/sqlite_fault_storage.cpp | 33 ++- .../integration/test_planned_stop.test.py | 182 +++++++++++++++- .../test_planned_stop_restart.test.py | 75 ++++++- .../test/test_correlation_engine.cpp | 191 +++++++++++++++++ .../test/test_fault_manager.cpp | 194 ++++++++++++++++++ .../test/test_sqlite_storage.cpp | 52 +++++ .../features/test_planned_stop_switch.test.py | 53 ++++- src/ros2_medkit_msgs/srv/GetPlannedStop.srv | 18 +- src/ros2_medkit_msgs/srv/SetPlannedStop.srv | 22 +- 19 files changed, 1106 insertions(+), 121 deletions(-) diff --git a/docs/api/messages.rst b/docs/api/messages.rst index 92fb16b80..7f923a729 100644 --- a/docs/api/messages.rst +++ b/docs/api/messages.rst @@ -312,18 +312,23 @@ Declare, withdraw and read a planned stop on the fault manager. # GetPlannedStop.srv --- - bool active # whether a planned stop is declared - string reason # the reason given; empty while none is declared - string declared_by # who declared it; empty while none is declared - builtin_interfaces/Time since # when it was declared; zero while none is + bool active # whether a planned stop is declared + string reason # the reason given; retained after the withdrawal + string declared_by # who declared it; retained after the withdrawal + builtin_interfaces/Time since # when it was declared + builtin_interfaces/Time ended_at # when it was withdrawn; zero while one is in force -While a planned stop is on, a fault whose cycle starts is reported, debounced, +While a planned stop is on, a fault whose cycle *starts* is reported, debounced, confirmed, captured and audited unchanged, and is marked as muted: absent from -the default ``ListFaults`` response, counted in ``muted_count``, listed under -``muted_faults`` with ``rule_id: planned_stop`` when ``include_muted`` is set, -and never published as a ``FaultEvent``. Withdrawing the stop releases every -fault it alone was muting and publishes one ``EVENT_CONFIRMED`` per released -fault that is CONFIRMED. +the default ``ListFaults`` response, counted in ``muted_count``, and listed under +``muted_faults`` with ``rule_id: planned_stop`` when ``include_muted`` is set. A +cycle that started before the stop is untouched. + +Publication matches a rule-muted symptom exactly: ``EVENT_CONFIRMED`` and +``EVENT_UPDATED`` are withheld, ``EVENT_CLEARED`` is published as usual. +Withdrawing the stop releases every fault it alone was muting and publishes one +``EVENT_CONFIRMED`` per released fault that is CONFIRMED. The audit records exist +only when ``audit_log.enabled`` is set, which it is not by default. See :doc:`/config/fault-manager` for the configuration that decides whether the declaration survives a restart and whether the transitions are audited. diff --git a/docs/config/fault-manager.rst b/docs/config/fault-manager.rst index 1ec74bb0c..dff5602df 100644 --- a/docs/config/fault-manager.rst +++ b/docs/config/fault-manager.rst @@ -608,7 +608,10 @@ Fault correlation identifies root causes and filters symptom faults. - Path to YAML file defining correlation rules. * - ``correlation.cleanup_interval_sec`` - ``5.0`` - - Interval for running correlation cleanup tasks. + - Interval for running correlation cleanup tasks. The correlation engine is + constructed on every fault manager, configured rules or not, because the + planned-stop switch needs none - so this timer runs everywhere. With no + rules loaded each tick walks two empty containers and returns. .. seealso:: @@ -644,18 +647,30 @@ Two configuration choices decide how much the switch can do: ends it. * - ``audit_log.enabled`` - ``false`` - - When on, each transition is recorded as ``planned_stop_started`` / - ``planned_stop_ended``, with the reason and the declarer. Recorded even - under ``audit_log.transitions: confirmed_only``. + - **Off by default, so a stock manager records no audit row for the switch + at all.** When on, each transition is recorded as ``planned_stop_started`` + / ``planned_stop_ended`` under the ``__audit__`` fault code, with that + transition's own reason and declarer. Recorded even under + ``audit_log.transitions: confirmed_only``. -While the stop stands, a fault whose cycle starts is reported, debounced, +While the stop stands, a fault whose cycle *starts* is reported, debounced, confirmed, captured and audited unchanged, and is marked as muted rather than -dropped: absent from the default fault list, counted in ``muted_count``, listed -under ``muted_faults`` with ``rule_id: planned_stop`` when muted entries are -requested, and never published on ``~/events``. Withdrawing the stop releases -every fault it alone was muting and publishes one confirmation event for each of -those that is CONFIRMED. Correlation rules are unaffected: a symptom a rule mutes -stays muted when the stop is withdrawn. +dropped: absent from the default fault list, counted in ``muted_count``, and +listed under ``muted_faults`` with ``rule_id: planned_stop`` when muted entries +are requested. A fault that was already up when the stop was declared is left +alone - its confirmation has already been announced, and reporters re-send FAILED +for as long as the condition holds. + +A marked fault is published exactly as a rule-muted symptom is: ``EVENT_CONFIRMED`` +and ``EVENT_UPDATED`` are withheld, while ``EVENT_CLEARED`` - the fault healing, or +being acknowledged - is published as usual. Withdrawing the stop releases every +fault it alone was muting and publishes one confirmation event for each of those +that is CONFIRMED. Correlation rules are unaffected: a symptom a rule mutes stays +muted when the stop is withdrawn, and the stop never takes a mute a rule owns. + +``~/get_planned_stop`` keeps serving the declaration after the withdrawal, with +``ended_at`` stamped, so the reason stays readable once the plant is back up. A +request the store cannot record answers ``success: false`` and changes nothing. Over HTTP the switch is an operation on the fault manager's App entity (``POST /apps/fault_manager/operations/set_planned_stop/executions``), which diff --git a/src/ros2_medkit_fault_manager/README.md b/src/ros2_medkit_fault_manager/README.md index e1a275b98..b59cbac3e 100644 --- a/src/ros2_medkit_fault_manager/README.md +++ b/src/ros2_medkit_fault_manager/README.md @@ -60,13 +60,28 @@ ros2 service call /fault_manager/set_planned_stop ros2_medkit_msgs/srv/SetPlanne ros2 service call /fault_manager/get_planned_stop ros2_medkit_msgs/srv/GetPlannedStop "{}" ``` -**Marked, never dropped.** While the stop stands, a fault whose cycle starts goes -through report, debounce, confirmation, snapshot and rosbag capture and the audit -log exactly as it would otherwise. What changes is only how it is *shown*: it is -registered as muted with `rule_id: planned_stop` and the pseudo root cause +**Marked, never dropped.** While the stop stands, a fault whose cycle *starts* +goes through report, debounce, confirmation, snapshot and rosbag capture and the +audit log exactly as it would otherwise. What changes is only how it is *shown*: +it is registered as muted with `rule_id: planned_stop` and the pseudo root cause `PLANNED_STOP`, so it is absent from the default `~/list_faults` response, -counted in `muted_count`, listed under `muted_faults` when `include_muted` is -set, and never published on `~/events`. +counted in `muted_count`, and listed under `muted_faults` when `include_muted` is +set. + +**Only a cycle that starts inside the stop.** A fault that was already up when the +stop was declared keeps its place in the fault list. Reporters are level-triggered +- `FaultReporter::report()` is called for as long as the condition holds, not once +per transition - so marking on any report would take a standing alarm off the list +and then announce it a second time at the switch-off. The cycle boundary is the +same one `~/report_fault` already computes: a new fault, or one raised again after +being cleared. + +**Published exactly like a rule-muted symptom.** `EVENT_CONFIRMED` and +`EVENT_UPDATED` are withheld while the fault is marked. `EVENT_CLEARED` is not: a +fault that heals past the healing threshold, or that is acknowledged, publishes it +as any muted fault does. Consumers therefore see the end of a fault whose start +they never heard - that is the existing muting contract, not something the switch +changes. **Withdrawing releases the survivors.** Every fault the stop alone was muting and that is still active is unmuted, and each one that is CONFIRMED publishes a single @@ -80,21 +95,28 @@ that rule, and withdrawing the stop does not release it. A fault the stop muted that a rule later claims likewise stays muted after the withdrawal: two independent reasons to be quiet, and one of them going away is not both. -**Both transitions are audited** as `planned_stop_started` and -`planned_stop_ended`, carrying the reason in `description` and the declarer in -`source_id`, with an empty `fault_code` because the transition is about the -installation rather than one fault. They are recorded whatever -`audit_log.transitions` says, like the log's own activation markers. A request -asking for the state the switch is already in succeeds, changes nothing and -records nothing, so a retried call cannot manufacture evidence of a stop that -never started. +**Both transitions are audited** - when the audit log is on. `audit_log.enabled` +is `false` by default, and with it off the switch writes no audit row at all. With +it on, each transition appends `planned_stop_started` / `planned_stop_ended` +carrying that transition's own reason in `description` and its declarer in +`source_id`, under the `__audit__` fault code the log's own lifecycle markers use, +because the transition is about the installation rather than one fault. They are +recorded whatever `audit_log.transitions` says. A request asking for the state the +switch is already in succeeds, changes nothing and records nothing, so a retried +call cannot manufacture evidence of a stop that never started. A request the store +cannot record answers `success: false` with the reason, and changes nothing. + +**The declaration outlives the stop.** `~/get_planned_stop` keeps serving the +reason, the declarer and the start time after the withdrawal, with `ended_at` +stamped, so "why was line 3 quiet on Friday?" is answerable once the plant is back +up without reading the audit database. **The declaration survives a restart** when the SQLite backend is in use: it is a -row in the fault store, so a stop declared on Friday still mutes on Monday. The -in-memory backend has no file behind it and starts every process with no -declaration. Faults muted *before* a restart are not restored - their mute lived -in the process that is gone - so a restart inside a stop makes those visible -again while every fault reported after it is muted afresh. +row in the fault store, so a stop declared on Friday still mutes on Monday. On +startup the manager also re-marks every stored fault that is not CLEARED and whose +cycle started at or after the declaration, so the switch-off after a reboot still +releases them and announces their confirmations. The in-memory backend has no file +behind it and starts every process with no declaration. **Over SOVD.** The gateway maps a node's services to operations on its App entity, so an operator reaches the switch without any new route: @@ -122,7 +144,7 @@ entity to address. - **Near-miss series**: Appends one entry per FAILED report that moved the debounce counter without confirming, bounded per fault code and retained when the fault is cleared - **Freeze-frame retention**: One compact JSON freeze-frame per fault code, retained across `clear_fault` (see below) - **Planned stop**: An operator declares the plant deliberately down, and faults raised while it stands are marked rather than announced (see below) -- **Fault correlation** (optional): Root cause analysis with symptom muting and auto-clear +- **Fault correlation** (optional): Root cause analysis with symptom muting and auto-clear. The correlation *engine* is always constructed, because the planned stop needs no configuration; without a `correlation.config_file` it carries no rules, correlates nothing, and its cleanup timer (`correlation.cleanup_interval_sec`, default 5 s) walks two empty containers - **Tamper-evident audit log** (optional): Append-only, hash-chained record of fault state transitions for verifiable history ## Parameters diff --git a/src/ros2_medkit_fault_manager/design/index.rst b/src/ros2_medkit_fault_manager/design/index.rst index 40dd547d8..9250c2a76 100644 --- a/src/ros2_medkit_fault_manager/design/index.rst +++ b/src/ros2_medkit_fault_manager/design/index.rst @@ -193,8 +193,9 @@ Declares, withdraws and reads the planned stop. - **One declaration per manager**: the stop is a fact about the installation, not about a fault - **Idempotent**: a request for the state the switch is already in succeeds, changes nothing and writes no audit record; ``was_active`` tells the caller which of the two happened -- **Audited**: each real transition appends ``planned_stop_started`` / ``planned_stop_ended`` with the reason and the declarer -- **Persisted**: stored in the fault store, so the declaration outlives the process (SQLite backend) +- **Store first**: the declaration is written before anything is muted, unmuted, announced or audited, and a store that refuses the write ends the request with ``success=false`` - a stop the manager could not record must not be one a restart comes back believing in +- **Audited**: when ``audit_log.enabled`` is set (off by default), each real transition appends ``planned_stop_started`` / ``planned_stop_ended`` under the ``__audit__`` fault code, with that transition's own reason and declarer +- **Persisted**: stored in the fault store, so the declaration outlives the process (SQLite backend), and ``~/get_planned_stop`` keeps serving it after the withdrawal with ``ended_at`` stamped - **Returns**: ``success``, a message, and the state the switch was in before the call Design Decisions @@ -262,23 +263,49 @@ Planned Stop as a Second Mute Source Maintenance produces faults that are real, correctly detected, and not news. The planned stop marks them instead of dropping them: while it stands, a fault whose -cycle starts is registered in the correlation engine as muted with +cycle *starts* is registered in the correlation engine as muted with ``rule_id = "planned_stop"`` and the pseudo root cause ``PLANNED_STOP``, and everything else about it - debounce, confirmation, snapshot and rosbag capture, the audit records - happens exactly as it would otherwise. Dropping the report -instead was rejected because the bridges send one FAILED per transition: a fault -that outlived the stop would never be raised again. - -The engine keeps the codes the stop muted in a set of its own, separate from the -mute map, and that set is what makes the two sources compose. A code leaves it the -moment a rule claims the mute, so withdrawing the stop cannot release a symptom a -rule is still holding down; and it leaves on ``process_clear``, so a fault -acknowledged inside the stop is not handed back at the withdrawal. +instead was rejected because a fault that outlived the stop would then never be +raised again: reporters are level-triggered, so nothing re-announces a condition +whose report was thrown away. + +**Only a cycle that starts inside the stop.** That same level-triggered shape is +why the mark is gated on the cycle boundary rather than on the report. +``ros2_medkit_diagnostic_bridge`` sends one FAILED per incoming +``DiagnosticArray`` - typically 1 Hz for as long as the condition holds - and +``FaultReporter::report()`` is documented to be called the same way. Marking on any +report would take a fault that was confirmed and announced *before* the stop off +the fault list within a second of the declaration, and announce it a second time at +the switch-off. The boundary is ``report_fault_event``'s own ``is_new``: a new +fault, or one raised again after being cleared. + +**One mute, one owner.** The engine keeps the codes the stop muted in a set of its +own, separate from the mute map, and that set is what makes the two sources +compose. A code leaves it where a rule *registers* its mute (``try_as_symptom``), +not merely where a rule reports ``should_mute``: the auto-cluster path sets that +flag without ever writing the mute map, so treating the flag as a hand-over would +leave the planned stop's entry in the map owned by nobody - muted for good, +released by nothing. In the other direction the stop never overwrites an entry a +rule already holds. A code also leaves the set when the fault is acknowledged, on +both acknowledgement paths: the correlation clear, and the scoped per-entity clear +that deliberately skips it. + +**The mark survives a restart.** The set itself does not - it lives in the process +- so on startup, with a declaration still in force, the manager re-registers it for +every stored fault that is not CLEARED and whose ``first_occurred`` is at or after +the declaration. Without that, a reboot inside a weekend stop would make the +survivors silently visible and the switch-off would announce none of them: their +confirmations, suppressed before the reboot, would be lost for good. Withdrawing releases the rest and publishes ``EVENT_CONFIRMED`` once for each released fault that is CONFIRMED. That republication is the point of marking rather than dropping: the confirmation happened behind the mute, no consumer of the event stream ever heard it, and the condition still stands on the machine. +What is withheld is exactly what a rule-muted symptom withholds - ``EVENT_CONFIRMED`` +and ``EVENT_UPDATED`` - while ``EVENT_CLEARED`` is published as usual, so a fault +that heals or is acknowledged inside the stop still reports its end. The switch is a service rather than a parameter because it carries a reason and a declarer and produces an audit record, none of which a parameter can do; and it is diff --git a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/correlation/correlation_engine.hpp b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/correlation/correlation_engine.hpp index fac91c110..065575bc3 100644 --- a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/correlation/correlation_engine.hpp +++ b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/correlation/correlation_engine.hpp @@ -112,9 +112,17 @@ class CorrelationEngine { /// @param fault_code The fault code /// @param severity The fault severity (for representative selection) /// @param timestamp When the fault occurred + /// @param cycle_started Whether this report STARTED a fault cycle (a new fault, + /// or one raised again after being cleared) as opposed to a repeat of a + /// condition that is already up. Only a cycle that starts while a planned + /// stop is declared is marked by it: reporters are level-triggered and + /// re-send FAILED for as long as the condition holds, so muting on any + /// report would swallow a fault that was announced before the stop began. + /// Correlation rules are unaffected and match on every report. /// @return Processing result indicating whether to mute, correlations, etc. ProcessFaultResult process_fault(const std::string & fault_code, const std::string & severity, - std::chrono::steady_clock::time_point timestamp = std::chrono::steady_clock::now()); + std::chrono::steady_clock::time_point timestamp = std::chrono::steady_clock::now(), + bool cycle_started = true); /// Process a fault being cleared /// @param fault_code The fault code being cleared @@ -159,6 +167,19 @@ class CorrelationEngine { /// Whether a planned stop is currently declared. bool planned_stop_active() const; + /// Release one fault from the planned stop's mute, if the stop is what is + /// muting it. A rule's mute is left alone. Called when a fault is acknowledged + /// on a path that does not run the correlation clear (a scoped per-entity + /// DELETE), so an acknowledged fault never stays counted as muted. + void drop_planned_stop_mute(const std::string & fault_code); + + /// Register a fault as muted by the planned stop without a report driving it. + /// This is how a restart re-marks the faults whose cycles started inside a stop + /// that is still declared: their mute lived in the process that is gone, and + /// without it the switch-off would neither release nor announce them. A fault a + /// rule is already muting is left to that rule. + void restore_planned_stop_mute(const std::string & fault_code); + private: /// The correlation half of process_fault: rules, clusters and their muting, /// with no planned-stop involvement. Caller holds mutex_. diff --git a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_audit_log.hpp b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_audit_log.hpp index cea41c8ad..9cee32c23 100644 --- a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_audit_log.hpp +++ b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_audit_log.hpp @@ -54,6 +54,12 @@ struct AuditEvent { int64_t occurred_at_ns{0}; ///< wall-clock timestamp of the transition }; +/// Fault code carried by records that describe the INSTALLATION rather than one +/// fault - the log's own activation markers and the planned-stop transitions. One +/// sentinel for all of them, so a reader filtering them out has a single value to +/// filter on. +constexpr const char * kAuditMarkerFaultCode = "__audit__"; + /// Canonical transition kinds. Stored verbatim, so they are part of the hash. constexpr const char * kTransitionOccurred = "occurred"; constexpr const char * kTransitionConfirmed = "confirmed"; diff --git a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_manager_node.hpp b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_manager_node.hpp index 290174059..caa1f5c23 100644 --- a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_manager_node.hpp +++ b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_manager_node.hpp @@ -72,6 +72,12 @@ class FaultManagerNode : public rclcpp::Node { return *storage_; } + /// Replace the storage backend (for testing only). The failure paths of a store + /// that rejects a write are otherwise reachable only on a broken disk. + void set_storage_for_test(std::unique_ptr storage) { + storage_ = std::move(storage); + } + /// Get the tamper-evident audit log (nullptr when disabled), for testing only. const FaultAuditLog * get_audit_log_for_test() const { return audit_log_.get(); @@ -175,6 +181,12 @@ class FaultManagerNode : public rclcpp::Node { /// @param config SnapshotConfig to populate with loaded values void load_snapshot_config_from_yaml(const std::string & config_file, SnapshotConfig & config); + /// Re-register the planned stop's mute for every stored fault whose cycle started + /// inside the declaration currently in force and that is not CLEARED. Runs at + /// startup, before the node serves anything. + /// @return how many faults were re-marked + size_t restore_planned_stop_mutes(); + /// Build the correlation engine. /// /// Never null. The engine also owns the planned-stop mute, which an operator can @@ -233,7 +245,11 @@ class FaultManagerNode : public rclcpp::Node { /// says, for the same reason the logging markers are: the switch describes the /// installation, and the evidence chain has to show when the plant was /// declared stopped. - void audit_planned_stop(const char * transition, const PlannedStopState & state, int64_t occurred_at_ns); + /// @param state The declaration as it stands after the transition (for its status) + /// @param reason Why THIS transition was made + /// @param declared_by Who made it + void audit_planned_stop(const char * transition, const PlannedStopState & state, const std::string & reason, + const std::string & declared_by, int64_t occurred_at_ns); std::string storage_type_; std::string database_path_; diff --git a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_storage.hpp b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_storage.hpp index b8033a2dd..63b0eafe4 100644 --- a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_storage.hpp +++ b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_storage.hpp @@ -202,9 +202,12 @@ struct RosbagFileInfo { /// a stop declared on Friday is still in force after a Saturday reboot. struct PlannedStopState { bool active{false}; - std::string reason; ///< Why the plant is stopped; empty while none is declared - std::string declared_by; ///< Who declared it; empty while none is declared - int64_t since_ns{0}; ///< Wall-clock time of the declaration; 0 while none is declared + std::string reason; ///< Why the plant is stopped; retained after the withdrawal + std::string declared_by; ///< Who declared it; retained after the withdrawal + int64_t since_ns{0}; ///< Wall-clock time of the declaration; 0 when none was ever made + /// Wall-clock time the declaration was withdrawn; 0 while one is in force. The + /// row outlives the stop so the reason stays readable after the plant is back up. + int64_t ended_at_ns{0}; }; /// Abstract interface for fault storage backends diff --git a/src/ros2_medkit_fault_manager/src/correlation/correlation_engine.cpp b/src/ros2_medkit_fault_manager/src/correlation/correlation_engine.cpp index 6c9c01833..a3c0f5af3 100644 --- a/src/ros2_medkit_fault_manager/src/correlation/correlation_engine.cpp +++ b/src/ros2_medkit_fault_manager/src/correlation/correlation_engine.cpp @@ -25,28 +25,48 @@ CorrelationEngine::CorrelationEngine(const CorrelationConfig & config) } ProcessFaultResult CorrelationEngine::process_fault(const std::string & fault_code, const std::string & severity, - std::chrono::steady_clock::time_point timestamp) { + std::chrono::steady_clock::time_point timestamp, + bool cycle_started) { std::lock_guard lock(mutex_); + // Correlation runs on every report. Handing a mute over to a rule happens where + // the rule registers it (try_as_symptom), not here: the auto-cluster path + // reports should_mute without ever writing muted_faults_, so treating + // should_mute as "a rule owns this now" would orphan the planned stop's entry - + // muted for good, released by nothing. ProcessFaultResult result = correlate(fault_code, severity, timestamp); - if (result.should_mute) { - // A rule is now the reason this fault is quiet. Hand it over: ending the - // planned stop must not release a fault a rule is still holding down. - planned_stop_muted_.erase(fault_code); + if (!planned_stop_active_) { return result; } - if (planned_stop_active_) { - MutedFaultData muted; - muted.fault_code = fault_code; - muted.root_cause_code = kPlannedStopRootCause; - muted.rule_id = kPlannedStopRuleId; - muted_faults_[fault_code] = muted; - planned_stop_muted_.insert(fault_code); + if (planned_stop_muted_.count(fault_code) > 0) { + // Already the stop's. A repeat report neither re-registers it nor lets it go. result.should_mute = true; + return result; + } + + if (!cycle_started) { + // The condition was already up when the stop was declared. Its confirmation + // has been announced; hiding it now would take a standing alarm off the list + // and announce it a second time at the switch-off. + return result; + } + + if (muted_faults_.count(fault_code) > 0) { + // A rule is muting this code. One mute, one owner: the stop does not take it + // over and, at the switch-off, does not release it. + return result; } + MutedFaultData muted; + muted.fault_code = fault_code; + muted.root_cause_code = kPlannedStopRootCause; + muted.rule_id = kPlannedStopRuleId; + muted_faults_[fault_code] = muted; + planned_stop_muted_.insert(fault_code); + result.should_mute = true; + return result; } @@ -288,6 +308,29 @@ bool CorrelationEngine::planned_stop_active() const { return planned_stop_active_; } +void CorrelationEngine::drop_planned_stop_mute(const std::string & fault_code) { + std::lock_guard lock(mutex_); + + if (planned_stop_muted_.erase(fault_code) > 0) { + muted_faults_.erase(fault_code); + } +} + +void CorrelationEngine::restore_planned_stop_mute(const std::string & fault_code) { + std::lock_guard lock(mutex_); + + if (muted_faults_.count(fault_code) > 0) { + return; // a rule is already muting it; one mute, one owner + } + + MutedFaultData muted; + muted.fault_code = fault_code; + muted.root_cause_code = kPlannedStopRootCause; + muted.rule_id = kPlannedStopRuleId; + muted_faults_[fault_code] = muted; + planned_stop_muted_.insert(fault_code); +} + std::vector CorrelationEngine::get_muted_faults() const { std::lock_guard lock(mutex_); @@ -439,6 +482,9 @@ std::optional CorrelationEngine::try_as_symptom(const std::s muted.rule_id = rule.id; muted.delay_ms = result.delay_ms; muted_faults_[fault_code] = muted; + // The rule is the owner from here. Ending a planned stop must not release + // a fault a root cause is still holding down. + planned_stop_muted_.erase(fault_code); } return result; diff --git a/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp b/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp index 172dc2599..b8df8c13d 100644 --- a/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp +++ b/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp @@ -460,8 +460,10 @@ FaultManagerNode::FaultManagerNode(const rclcpp::NodeOptions & options) : Node(" planned_stop_ = storage_->get_planned_stop(); if (planned_stop_.active) { correlation_engine_->begin_planned_stop(); - RCLCPP_INFO(get_logger(), "Planned stop is in force (reason='%s', declared_by='%s'): new faults are marked muted", - planned_stop_.reason.c_str(), planned_stop_.declared_by.c_str()); + const size_t restored = restore_planned_stop_mutes(); + RCLCPP_INFO(get_logger(), + "Planned stop is in force (reason='%s', declared_by='%s'): %zu fault(s) re-marked, new faults marked", + planned_stop_.reason.c_str(), planned_stop_.declared_by.c_str(), restored); } // Create auto-confirmation timer if enabled @@ -528,7 +530,7 @@ FaultManagerNode::~FaultManagerNode() { if (audit_log_) { try { AuditEvent marker; - marker.fault_code = "__audit__"; + marker.fault_code = kAuditMarkerFaultCode; marker.transition = kTransitionLoggingDeactivated; marker.status = "INACTIVE"; marker.source_id = "fault_manager"; @@ -641,7 +643,7 @@ std::unique_ptr FaultManagerNode::create_audit_log() { // Appended directly, so it is recorded even in confirmed_only mode. // Best-effort: a failure here does not abort startup. AuditEvent marker; - marker.fault_code = "__audit__"; + marker.fault_code = kAuditMarkerFaultCode; marker.transition = kTransitionLoggingActivated; marker.status = "ACTIVE"; marker.source_id = "fault_manager"; @@ -716,14 +718,18 @@ void FaultManagerNode::append_audit_event(const AuditEvent & event) { } void FaultManagerNode::audit_planned_stop(const char * transition, const PlannedStopState & state, + const std::string & reason, const std::string & declared_by, int64_t occurred_at_ns) { AuditEvent event; - // Empty: the transition is about the installation, not about one fault. - event.fault_code = ""; + // The same sentinel the log's own lifecycle markers use: this transition is + // about the installation, not about one fault. + event.fault_code = kAuditMarkerFaultCode; event.transition = transition; event.status = state.active ? "ACTIVE" : "INACTIVE"; - event.source_id = state.declared_by; - event.description = state.reason; + // The reason and the declarer of THIS transition: who withdrew a stop, and why, + // is not who declared it. + event.source_id = declared_by; + event.description = reason; event.occurred_at_ns = occurred_at_ns; append_audit_event(event); @@ -867,8 +873,13 @@ void FaultManagerNode::handle_report_fault( // Only process FAILED events with correlation bool should_mute = false; if (correlation_engine_ && request->event_type == ros2_medkit_msgs::srv::ReportFault::Request::EVENT_FAILED) { + // is_new is the cycle boundary the planned stop needs: a new fault, or one + // raised again after being cleared. Reporters are level-triggered, so + // without it a condition that was already up - and already announced - + // would be swallowed by a stop declared after it started. auto correlation_result = - correlation_engine_->process_fault(request->fault_code, correlation::severity_to_string(request->severity)); + correlation_engine_->process_fault(request->fault_code, correlation::severity_to_string(request->severity), + std::chrono::steady_clock::now(), is_new); should_mute = correlation_result.should_mute; @@ -1053,9 +1064,15 @@ void FaultManagerNode::handle_clear_fault( // correlated symptom fault codes. Per-entity DELETE routes set it to true // so they cannot reach across entity boundaries via the correlation graph. std::vector auto_cleared_codes; - if (correlation_engine_ && !request->skip_correlation_auto_clear) { + if (!request->skip_correlation_auto_clear) { auto clear_result = correlation_engine_->process_clear(request->fault_code); auto_cleared_codes = clear_result.auto_cleared_codes; + } else { + // Scoped per-entity acknowledgement: the correlation graph is deliberately not + // walked, because it reaches across entity boundaries. The planned stop is not + // a graph - it is this one fault's mute - so it is released either way, or an + // acknowledged fault would stay counted as muted until the stop ended. + correlation_engine_->drop_planned_stop_mute(request->fault_code); } bool cleared = storage_->clear_fault(request->fault_code); @@ -1131,15 +1148,40 @@ void FaultManagerNode::handle_set_planned_stop( const int64_t transition_at_ns = get_wall_clock_time().nanoseconds(); + // What the declaration becomes. A withdrawal keeps the reason, the declarer and + // the start time of the stop it ends, and stamps when it ended, so an operator + // can still read why the plant was quiet once it is back up. + PlannedStopState next = planned_stop_; if (request->active) { - planned_stop_.active = true; - planned_stop_.reason = request->reason; - planned_stop_.declared_by = request->declared_by; - planned_stop_.since_ns = transition_at_ns; + next.active = true; + next.reason = request->reason; + next.declared_by = request->declared_by; + next.since_ns = transition_at_ns; + next.ended_at_ns = 0; + } else { + next.active = false; + next.ended_at_ns = transition_at_ns; + } + + // The store goes first, and a store that refuses the write ends the request + // here: nothing is muted, unmuted, announced or audited on a declaration the + // manager could not record, and a restart cannot come back believing in it. + try { + storage_->set_planned_stop(next); + } catch (const std::exception & e) { + response->success = false; + response->message = std::string("Failed to record the planned stop: ") + e.what(); + RCLCPP_ERROR(get_logger(), "Planned stop %s rejected: %s", request->active ? "declaration" : "withdrawal", + e.what()); + return; + } + + planned_stop_ = next; + if (request->active) { correlation_engine_->begin_planned_stop(); - storage_->set_planned_stop(planned_stop_); - audit_planned_stop(kTransitionPlannedStopStarted, planned_stop_, transition_at_ns); + audit_planned_stop(kTransitionPlannedStopStarted, planned_stop_, request->reason, request->declared_by, + transition_at_ns); response->message = "Planned stop declared"; RCLCPP_INFO(get_logger(), "Planned stop declared by '%s': %s", planned_stop_.declared_by.c_str(), @@ -1164,15 +1206,9 @@ void FaultManagerNode::handle_set_planned_stop( } // The withdrawal carries its own reason and declarer into the audit record; the - // stored declaration goes back to "no stop", which is what GetPlannedStop then - // reports. - PlannedStopState ended; - ended.reason = request->reason; - ended.declared_by = request->declared_by; - audit_planned_stop(kTransitionPlannedStopEnded, ended, transition_at_ns); - - planned_stop_ = PlannedStopState{}; - storage_->set_planned_stop(planned_stop_); + // stored declaration keeps the ones that started the stop. + audit_planned_stop(kTransitionPlannedStopEnded, planned_stop_, request->reason, request->declared_by, + transition_at_ns); response->message = "Planned stop withdrawn"; RCLCPP_INFO(get_logger(), "Planned stop withdrawn by '%s': unmuted %zu fault(s), announced %zu confirmation(s)", @@ -1186,6 +1222,7 @@ void FaultManagerNode::handle_get_planned_stop( response->reason = planned_stop_.reason; response->declared_by = planned_stop_.declared_by; response->since = rclcpp::Time(planned_stop_.since_ns, RCL_SYSTEM_TIME); + response->ended_at = rclcpp::Time(planned_stop_.ended_at_ns, RCL_SYSTEM_TIME); } bool FaultManagerNode::is_valid_severity(uint8_t severity) { @@ -1479,6 +1516,27 @@ void FaultManagerNode::load_snapshot_config_from_yaml(const std::string & config } } +size_t FaultManagerNode::restore_planned_stop_mutes() { + // The mute set lives in the engine, so a restart loses it while the declaration + // itself survives in the store. Without this, a reboot inside a weekend stop + // makes every fault the stop was holding silently visible, and the switch-off + // then announces none of them - the confirmations they never published are lost + // for good. Rebuild it from what the store knows: a fault whose cycle started at + // or after the declaration and that is not over. + size_t restored = 0; + for (const auto & fault : storage_->get_all_faults()) { + if (fault.status == ros2_medkit_msgs::msg::Fault::STATUS_CLEARED) { + continue; // acknowledged: nothing left to hold down or announce + } + if (rclcpp::Time(fault.first_occurred).nanoseconds() < planned_stop_.since_ns) { + continue; // its cycle started before the stop, so the stop never marked it + } + correlation_engine_->restore_planned_stop_mute(fault.fault_code); + ++restored; + } + return restored; +} + std::unique_ptr FaultManagerNode::create_correlation_engine() { auto config = load_correlation_config(); if (config) { diff --git a/src/ros2_medkit_fault_manager/src/sqlite_fault_storage.cpp b/src/ros2_medkit_fault_manager/src/sqlite_fault_storage.cpp index 7fa8ad042..502d745b6 100644 --- a/src/ros2_medkit_fault_manager/src/sqlite_fault_storage.cpp +++ b/src/ros2_medkit_fault_manager/src/sqlite_fault_storage.cpp @@ -325,7 +325,8 @@ void SqliteFaultStorage::initialize_schema() { active INTEGER NOT NULL, reason TEXT NOT NULL, declared_by TEXT NOT NULL, - since_ns INTEGER NOT NULL + since_ns INTEGER NOT NULL, + ended_at_ns INTEGER NOT NULL DEFAULT 0 ); )"; @@ -335,6 +336,28 @@ void SqliteFaultStorage::initialize_schema() { throw std::runtime_error("Failed to create planned_stop table: " + error); } + // Migration: the row outlives the stop now, so it records when the stop ended. + // A database written before that reads as "never withdrawn", which is what a + // row with no end time means. + { + bool has_ended_at = false; + SqliteStatement info(db_, "PRAGMA table_info(planned_stop)"); + while (info.step() == SQLITE_ROW) { + if (info.column_text(1) == "ended_at_ns") { + has_ended_at = true; + break; + } + } + if (!has_ended_at) { + if (sqlite3_exec(db_, "ALTER TABLE planned_stop ADD COLUMN ended_at_ns INTEGER NOT NULL DEFAULT 0", nullptr, + nullptr, &err_msg) != SQLITE_OK) { + std::string error = err_msg ? err_msg : "Unknown error"; + sqlite3_free(err_msg); + throw std::runtime_error("Failed to add ended_at_ns column: " + error); + } + } + } + // Create rosbag_files table. One row = one LINK (a fault claiming a recording): // several faults of a burst link to one bag, and one fault links to several bags // over time. Bytes belong to file_path, not to the row. @@ -1308,12 +1331,13 @@ void SqliteFaultStorage::set_planned_stop(const PlannedStopState & state) { std::lock_guard lock(mutex_); SqliteStatement stmt(db_, - "INSERT OR REPLACE INTO planned_stop (id, active, reason, declared_by, since_ns) " - "VALUES (1, ?, ?, ?, ?)"); + "INSERT OR REPLACE INTO planned_stop (id, active, reason, declared_by, since_ns, ended_at_ns) " + "VALUES (1, ?, ?, ?, ?, ?)"); stmt.bind_int(1, state.active ? 1 : 0); stmt.bind_text(2, state.reason); stmt.bind_text(3, state.declared_by); stmt.bind_int64(4, state.since_ns); + stmt.bind_int64(5, state.ended_at_ns); if (stmt.step() != SQLITE_DONE) { throw std::runtime_error(std::string("Failed to store planned stop: ") + sqlite3_errmsg(db_)); @@ -1323,7 +1347,7 @@ void SqliteFaultStorage::set_planned_stop(const PlannedStopState & state) { PlannedStopState SqliteFaultStorage::get_planned_stop() const { std::lock_guard lock(mutex_); - SqliteStatement stmt(db_, "SELECT active, reason, declared_by, since_ns FROM planned_stop WHERE id = 1"); + SqliteStatement stmt(db_, "SELECT active, reason, declared_by, since_ns, ended_at_ns FROM planned_stop WHERE id = 1"); PlannedStopState state; if (stmt.step() != SQLITE_ROW) { @@ -1334,6 +1358,7 @@ PlannedStopState SqliteFaultStorage::get_planned_stop() const { state.reason = stmt.column_text(1); state.declared_by = stmt.column_text(2); state.since_ns = stmt.column_int64(3); + state.ended_at_ns = stmt.column_int64(4); return state; } diff --git a/src/ros2_medkit_fault_manager/test/integration/test_planned_stop.test.py b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop.test.py index c5e59897c..e27ba058a 100644 --- a/src/ros2_medkit_fault_manager/test/integration/test_planned_stop.test.py +++ b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop.test.py @@ -64,14 +64,18 @@ SOURCE_ID = '/test_node' CAPTURE_TOPIC = '/test/temperature' -# A rule that mutes MOTOR_PS_* while ESTOP_PS_001 stands. Its window is long -# enough that no test can lose the correlation to its own pacing. +# Two rules. The hierarchical one mutes MOTOR_PS_* while ESTOP_PS_001 stands; the +# auto-cluster one claims VALVE_PS_* faults without ever writing them into the mute +# map, which is the case a planned stop's own entry has to survive. Both windows are +# long enough that no test can lose the correlation to its own pacing. CORRELATION_RULES = """ correlation: enabled: true patterns: ps_motor_errors: codes: ["MOTOR_PS_*"] + ps_valve_errors: + codes: ["VALVE_PS_*"] rules: - id: ps_estop_cascade name: "E-Stop Cascade" @@ -83,6 +87,15 @@ window_ms: 60000 mute_symptoms: true auto_clear_with_root: true + - id: ps_valve_storm + name: "Valve Storm" + mode: auto_cluster + match: + - pattern: ps_valve_errors + min_count: 2 + window_ms: 60000 + show_as_single: true + representative: first """ @@ -282,6 +295,16 @@ def _count_events(cls, fault_code, event_type): return sum(1 for kind, code, _ in cls._events if code == fault_code and kind == event_type) + @classmethod + def _event_types(cls, fault_code): + """Every event type published for this code, as a set. + + Counting one type cannot see a leak of another: a stop that announced an + UPDATE it should have withheld passes a CONFIRMED-only assertion. + """ + with cls._events_lock: + return {kind for kind, code, _ in cls._events if code == fault_code} + def _set_stop(self, active, *, reason='', declared_by=''): request = SetPlannedStop.Request() request.active = active @@ -388,8 +411,9 @@ def test_a_fault_raised_during_a_stop_is_marked_not_dropped(self): f'no snapshot was captured for the muted fault {code}') self.assertIn(CAPTURE_TOPIC, captured) - # Nothing was announced while the stop stood. - self.assertEqual(0, self._count_events(code, FaultEvent.EVENT_CONFIRMED)) + # Nothing at all was announced while the stop stood - not the confirmation, + # and not an update either. + self.assertEqual(set(), self._event_types(code)) withdrawal = self._set_stop(False, reason='line 3 back up', declared_by='shift_lead') self.assertTrue(withdrawal.success) @@ -401,6 +425,8 @@ def test_a_fault_raised_during_a_stop_is_marked_not_dropped(self): self.assertEqual( 1, self._count_events(code, FaultEvent.EVENT_CONFIRMED), 'the confirmation held back by the stop must be announced exactly once') + self.assertEqual({FaultEvent.EVENT_CONFIRMED}, self._event_types(code), + 'the switch-off announces the confirmation and nothing else') def test_a_fault_cleared_during_a_stop_never_reaches_the_default_list(self): code = 'PS_CLEARED_INSIDE' @@ -427,12 +453,19 @@ def test_a_fault_cleared_during_a_stop_never_reaches_the_default_list(self): self.assertIn('confirmed', transitions) self.assertIn('cleared', transitions) + # The acknowledgement is published, as it is for any muted fault; the + # confirmation behind the mute is not. Assert the whole set, so an + # announcement of any other kind is a failure rather than an unchecked + # extra frame. + self.assertEqual({FaultEvent.EVENT_CLEARED}, self._event_types(code)) + self.assertEqual(1, self._count_events(code, FaultEvent.EVENT_CLEARED)) + self._set_stop(False, reason='done', declared_by='maintenance') self.assertNotIn(code, self._codes_in_default_list()) self.assertEqual( - 0, self._count_events(code, FaultEvent.EVENT_CONFIRMED), - 'a fault that was acknowledged inside the stop has nothing to announce') + {FaultEvent.EVENT_CLEARED}, self._event_types(code), + 'a fault that was acknowledged inside the stop has nothing left to announce') def test_repeating_a_request_changes_nothing_and_records_nothing(self): started_before = len(self._audit_rows('planned_stop_started')) @@ -449,7 +482,7 @@ def test_repeating_a_request_changes_nothing_and_records_nothing(self): started = self._audit_rows('planned_stop_started') self.assertEqual(started_before + 1, len(started), 'declaring a stop that is already in force must write no audit record') - self.assertEqual('', started[-1][0]) + self.assertEqual('__audit__', started[-1][0]) self.assertEqual('ACTIVE', started[-1][2]) self.assertEqual('plant_manager', started[-1][3]) self.assertEqual('weekend shutdown', started[-1][4]) @@ -471,15 +504,18 @@ def test_repeating_a_request_changes_nothing_and_records_nothing(self): ended = self._audit_rows('planned_stop_ended') self.assertEqual(ended_before + 1, len(ended), 'withdrawing a stop that is not in force must write no audit record') - self.assertEqual('', ended[-1][0]) + self.assertEqual('__audit__', ended[-1][0]) self.assertEqual('INACTIVE', ended[-1][2]) self.assertEqual('plant_manager', ended[-1][3]) self.assertEqual('restart', ended[-1][4]) state = self._get_stop() self.assertFalse(state.active) - self.assertEqual('', state.reason) - self.assertEqual('', state.declared_by) + # The declaration outlives the stop: the repeat did not overwrite it, and + # the withdrawal did not erase it. + self.assertEqual('weekend shutdown', state.reason) + self.assertEqual('plant_manager', state.declared_by) + self.assertGreater(state.ended_at.sec, 0) def test_the_reason_and_declarer_may_be_empty_or_long(self): empty = self._set_stop(True, reason='', declared_by='') @@ -561,6 +597,132 @@ def test_a_fault_mid_debounce_when_the_stop_ends_confirms_afterwards(self): self.assertEqual('CONFIRMED', self._get_fault(code).fault.status) self.assertIn(code, self._codes_in_default_list()) + def test_a_fault_already_up_when_the_stop_begins_stays_visible(self): + """A cycle that started before the stop is not the stop's to hide.""" + code = 'PS_ALREADY_UP' + + # Confirmed and announced with no stop in force. + self._confirm(code) + self._wait_until(lambda: self._count_events(code, FaultEvent.EVENT_CONFIRMED) == 1, + f'{code} was never announced before the stop') + self.assertIn(code, self._codes_in_default_list()) + + self._set_stop(True, reason='line 3 maintenance', declared_by='shift_lead') + + # A level-triggered reporter keeps sending FAILED while the condition + # holds. None of those reports starts a cycle. + for _ in range(3): + self._report(code) + + self.assertIsNone(self._muted_entry(code), + 'the stop took over a fault whose cycle started before it') + self.assertIn(code, self._codes_in_default_list(), + 'a standing alarm left the fault list when the stop was declared') + + self._set_stop(False, reason='line 3 back up', declared_by='shift_lead') + + self.assertEqual( + 1, self._count_events(code, FaultEvent.EVENT_CONFIRMED), + 'the switch-off announced a confirmation that had already been announced') + + def test_a_fault_raised_again_after_a_clear_during_the_stop_is_marked(self): + """Re-raising a cleared fault starts a new cycle, and the stop takes it.""" + code = 'PS_RERAISED' + + self._confirm(code) + self._wait_until(lambda: self._get_fault(code).success, f'{code} never reached the store') + self.assertTrue(self._clear(code).success) + + self._set_stop(True, reason='changeover', declared_by='maintenance') + self._confirm(code) + + muted = self._wait_until(lambda: self._muted_entry(code), + f'the re-raised {code} was not marked by the stop') + self.assertEqual('planned_stop', muted.rule_id) + self.assertNotIn(code, self._codes_in_default_list()) + + before = self._count_events(code, FaultEvent.EVENT_CONFIRMED) + self._set_stop(False, reason='done', declared_by='maintenance') + self._wait_until( + lambda: self._count_events(code, FaultEvent.EVENT_CONFIRMED) == before + 1, + 'the new cycle the stop held back was never announced') + + def test_a_scoped_acknowledgement_still_releases_the_stops_mute(self): + """A clear that skips the correlation cascade still drops the stop's mute.""" + code = 'PS_SCOPED_ACK' + + self._set_stop(True, reason='cell 7 changeover', declared_by='maintenance') + self._confirm(code) + self._wait_until(lambda: self._muted_entry(code), f'{code} was not marked by the stop') + before_count = self._list().muted_count + + # The shape every per-entity DELETE takes: acknowledge this fault only, + # without walking the correlation graph into other entities. + request = ClearFault.Request() + request.fault_code = code + request.skip_correlation_auto_clear = True + self.assertTrue(self._call(self.clear_client, request).success) + + self.assertIsNone(self._muted_entry(code), + 'an acknowledged fault stayed listed as muted by the stop') + self.assertLess(self._list().muted_count, before_count, + 'muted_count still counts a fault that was acknowledged') + + self._set_stop(False, reason='done', declared_by='maintenance') + self.assertEqual(0, self._count_events(code, FaultEvent.EVENT_CONFIRMED)) + + def test_a_cluster_rule_does_not_orphan_the_stops_mute(self): + """A cluster claims the mute without owning it, so the stop still releases.""" + first = 'VALVE_PS_A' + second = 'VALVE_PS_B' + + self._set_stop(True, reason='valve bank swap', declared_by='maintenance') + self._confirm(first) + self._confirm(second) + + entries = self._wait_until( + lambda: [self._muted_entry(first), self._muted_entry(second)] + if self._muted_entry(first) and self._muted_entry(second) else None, + 'the clustered faults were not marked by the stop') + for entry in entries: + self.assertEqual('planned_stop', entry.rule_id, + 'the cluster took an entry it never writes') + + visible = self._codes_in_default_list() + self.assertNotIn(first, visible) + self.assertNotIn(second, visible) + + self._set_stop(False, reason='swap done', declared_by='maintenance') + + self._wait_until( + lambda: all(code in self._codes_in_default_list() for code in (first, second)), + 'a fault a cluster rule claimed was orphaned in the mute map') + for code in (first, second): + self.assertIsNone(self._muted_entry(code)) + self.assertEqual(1, self._count_events(code, FaultEvent.EVENT_CONFIRMED)) + + def test_the_last_declaration_is_readable_after_the_withdrawal(self): + """The reason survives the withdrawal, with the time the stop ended.""" + self._set_stop(True, reason='line 3 quarterly maintenance', declared_by='shift_lead') + during = self._get_stop() + self.assertTrue(during.active) + self.assertEqual(0, during.ended_at.sec) + + self._set_stop(False, reason='plant back up', declared_by='night_shift') + + after = self._get_stop() + self.assertFalse(after.active) + self.assertEqual('line 3 quarterly maintenance', after.reason) + self.assertEqual('shift_lead', after.declared_by) + self.assertEqual(during.since.sec, after.since.sec) + self.assertGreater(after.ended_at.sec, 0) + + # The withdrawal's own reason and declarer are in the audit row, not in the + # declaration. + ended = self._audit_rows('planned_stop_ended')[-1] + self.assertEqual('night_shift', ended[3]) + self.assertEqual('plant back up', ended[4]) + def test_the_switch_does_not_release_a_rule_muted_symptom(self): root = 'ESTOP_PS_001' symptom = 'MOTOR_PS_LEFT' diff --git a/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_restart.test.py b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_restart.test.py index 29377d4e3..f46edd7c5 100644 --- a/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_restart.test.py +++ b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_restart.test.py @@ -24,6 +24,7 @@ import os import signal import tempfile +import threading import time import unittest @@ -33,7 +34,8 @@ import launch_testing.actions import rclpy from rclpy.node import Node -from ros2_medkit_msgs.msg import Fault +from rclpy.qos import HistoryPolicy, QoSProfile, ReliabilityPolicy +from ros2_medkit_msgs.msg import Fault, FaultEvent from ros2_medkit_msgs.srv import GetFault, GetPlannedStop, ListFaults, ReportFault, SetPlannedStop STORAGE_DIR = tempfile.mkdtemp(prefix='planned_stop_restart_') @@ -42,6 +44,7 @@ REASON = 'weekend shutdown, cell 4' DECLARED_BY = 'plant_manager' BEFORE_CODE = 'PS_RESTART_BEFORE' +SECOND_BEFORE_CODE = 'PS_RESTART_BEFORE_TWO' AFTER_CODE = 'PS_RESTART_AFTER' @@ -108,6 +111,15 @@ def setUpClass(cls): os.environ['ROS_LOCALHOST_ONLY'] = '1' rclpy.init() cls.node = Node('test_planned_stop_restart_client') + + # The subscription outlives the manager it listens to and rematches the + # replacement's publisher, which is what lets one test span the restart. + cls._events = [] + cls._events_lock = threading.Lock() + cls._event_sub = cls.node.create_subscription( + FaultEvent, '/fault_manager/events', cls._on_event, + QoSProfile(reliability=ReliabilityPolicy.RELIABLE, + history=HistoryPolicy.KEEP_LAST, depth=200)) cls._connect() @classmethod @@ -115,6 +127,16 @@ def tearDownClass(cls): cls.node.destroy_node() rclpy.shutdown() + @classmethod + def _on_event(cls, msg): + with cls._events_lock: + cls._events.append((msg.event_type, msg.fault.fault_code)) + + def _count_events(self, fault_code, event_type): + with self._events_lock: + return sum(1 for kind, code in self._events + if code == fault_code and kind == event_type) + @classmethod def _connect(cls, timeout_sec=30.0): """Create clients and wait for the manager's services to answer.""" @@ -141,6 +163,20 @@ def _call(self, client, request, timeout_sec=15.0): self.assertIsNotNone(future.result(), 'service call timed out') return future.result() + def _spin_for(self, seconds): + """Pump the executor so published events reach the subscription.""" + deadline = time.monotonic() + seconds + while time.monotonic() < deadline: + rclpy.spin_once(self.node, timeout_sec=0.05) + + def _wait_until(self, predicate, message, timeout=30.0): + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if predicate(): + return + rclpy.spin_once(self.node, timeout_sec=0.05) + raise AssertionError(message) + def _report(self, fault_code): request = ReportFault.Request() request.fault_code = fault_code @@ -189,7 +225,13 @@ def test_a_stop_declared_before_a_restart_still_mutes_after_it(self, fault_manag self.assertEqual(REASON, before.reason) self._report(BEFORE_CODE) - self.assertIn(BEFORE_CODE, self._muted_codes()) + self._report(SECOND_BEFORE_CODE) + muted_before = self._muted_codes() + self.assertIn(BEFORE_CODE, muted_before) + self.assertIn(SECOND_BEFORE_CODE, muted_before) + for code in (BEFORE_CODE, SECOND_BEFORE_CODE): + self.assertEqual(0, self._count_events(code, FaultEvent.EVENT_CONFIRMED), + 'a fault the stop marked was announced anyway') original_pid = fault_manager_node.process_details['pid'] os.kill(original_pid, signal.SIGTERM) @@ -223,12 +265,39 @@ def test_a_stop_declared_before_a_restart_still_mutes_after_it(self, fault_manag self.assertEqual('planned_stop', muted[AFTER_CODE].rule_id) self.assertNotIn(AFTER_CODE, self._confirmed_codes()) + # The mute set lived in the process that is gone, so the manager rebuilds it + # from the store: the faults marked before the restart are still marked. + muted_after_restart = self._muted_codes() + for code in (BEFORE_CODE, SECOND_BEFORE_CODE): + self.assertIn(code, muted_after_restart, + f'{code} was silently released by the restart, so its ' + 'confirmation could never be announced') + self.assertEqual('planned_stop', muted_after_restart[code].rule_id) + self.assertNotIn(code, self._confirmed_codes()) + withdrawn = self._set_stop(False, reason='plant back up', declared_by=DECLARED_BY) self.assertTrue(withdrawn.success) self.assertTrue(withdrawn.was_active) - self.assertFalse(self._get_stop().active) + final_state = self._get_stop() + self.assertFalse(final_state.active) + self.assertEqual(REASON, final_state.reason) + self.assertGreater(final_state.ended_at.sec, 0) self.assertIn(AFTER_CODE, self._confirmed_codes()) + # Every fault the stop was holding - across the restart - is released and + # announced exactly once. + for code in (BEFORE_CODE, SECOND_BEFORE_CODE, AFTER_CODE): + self._wait_until( + lambda code=code: self._count_events(code, FaultEvent.EVENT_CONFIRMED) == 1, + f'{code} was never announced at the switch-off') + self.assertIn(code, self._confirmed_codes()) + + # A second frame arriving late would still be a double announcement. + self._spin_for(3.0) + for code in (BEFORE_CODE, SECOND_BEFORE_CODE, AFTER_CODE): + self.assertEqual(1, self._count_events(code, FaultEvent.EVENT_CONFIRMED), + f'{code} was announced more than once') + @launch_testing.post_shutdown_test() class TestShutdown(unittest.TestCase): diff --git a/src/ros2_medkit_fault_manager/test/test_correlation_engine.cpp b/src/ros2_medkit_fault_manager/test/test_correlation_engine.cpp index 442db831e..730e98618 100644 --- a/src/ros2_medkit_fault_manager/test/test_correlation_engine.cpp +++ b/src/ros2_medkit_fault_manager/test/test_correlation_engine.cpp @@ -76,6 +76,30 @@ class CorrelationEngineTest : public ::testing::Test { return parse_config_string(yaml); } + /// A hierarchical rule and an auto-cluster rule side by side. The cluster path + /// sets should_mute without ever writing muted_faults_, which is the case the + /// planned stop has to survive. + CorrelationConfig create_cluster_config() { + const std::string yaml = R"( +correlation: + enabled: true + patterns: + valve_errors: + codes: ["VALVE_*"] + rules: + - id: valve_storm + name: "Valve Storm" + mode: auto_cluster + match: + - pattern: valve_errors + min_count: 2 + window_ms: 60000 + show_as_single: true + representative: first +)"; + return parse_config_string(yaml); + } + CorrelationConfig create_mixed_config() { const std::string yaml = R"( correlation: @@ -939,6 +963,173 @@ TEST_F(CorrelationEngineTest, PlannedStopMutesWithoutAnyRulesConfigured) { EXPECT_EQ("VALVE_STUCK", unmuted[0]); } +// ============================================================================ +// Planned stop: only a cycle that STARTS inside it is marked +// ============================================================================ + +TEST_F(CorrelationEngineTest, AStillActiveReReportIsNotTakenByTheStop) { + CorrelationEngine engine(create_hierarchical_config()); + + // The fault's cycle started before the stop was declared. + engine.process_fault("PUMP_SEAL_LEAK", "ERROR", std::chrono::steady_clock::now(), /*cycle_started=*/true); + EXPECT_FALSE(engine.is_muted("PUMP_SEAL_LEAK")); + + engine.begin_planned_stop(); + + // A level-triggered reporter keeps sending FAILED while the condition holds. + auto result = engine.process_fault("PUMP_SEAL_LEAK", "ERROR", std::chrono::steady_clock::now(), + /*cycle_started=*/false); + EXPECT_FALSE(result.should_mute); + EXPECT_FALSE(engine.is_muted("PUMP_SEAL_LEAK")); + EXPECT_TRUE(engine.end_planned_stop().empty()) << "the stop released a fault it never muted"; +} + +TEST_F(CorrelationEngineTest, ACycleStartingInsideTheStopIsTakenByIt) { + CorrelationEngine engine(create_hierarchical_config()); + + engine.begin_planned_stop(); + auto result = engine.process_fault("PUMP_SEAL_LEAK", "ERROR", std::chrono::steady_clock::now(), + /*cycle_started=*/true); + EXPECT_TRUE(result.should_mute); + EXPECT_TRUE(engine.is_muted("PUMP_SEAL_LEAK")); + + auto unmuted = engine.end_planned_stop(); + ASSERT_EQ(1u, unmuted.size()); + EXPECT_EQ("PUMP_SEAL_LEAK", unmuted[0]); +} + +TEST_F(CorrelationEngineTest, AStillActiveReReportDoesNotReleaseAStopMute) { + CorrelationEngine engine(create_hierarchical_config()); + + engine.begin_planned_stop(); + engine.process_fault("VALVE_STUCK", "ERROR", std::chrono::steady_clock::now(), /*cycle_started=*/true); + ASSERT_TRUE(engine.is_muted("VALVE_STUCK")); + + // The same condition is reported again while it holds. Nothing about the mute + // changes: the cycle is the one the stop already took. + auto again = engine.process_fault("VALVE_STUCK", "ERROR", std::chrono::steady_clock::now(), + /*cycle_started=*/false); + EXPECT_TRUE(again.should_mute); + EXPECT_TRUE(engine.is_muted("VALVE_STUCK")); + + auto unmuted = engine.end_planned_stop(); + ASSERT_EQ(1u, unmuted.size()); + EXPECT_EQ("VALVE_STUCK", unmuted[0]); +} + +// ============================================================================ +// Planned stop: exactly one mute owner per fault +// ============================================================================ + +TEST_F(CorrelationEngineTest, TheStopNeverOverwritesARuleMute) { + CorrelationEngine engine(create_hierarchical_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.process_fault("ESTOP_001", "CRITICAL", t0); + auto symptom = engine.process_fault("MOTOR_COMM_FL", "ERROR", t0 + 10ms); + ASSERT_TRUE(symptom.should_mute); + ASSERT_EQ("estop_cascade", engine.get_muted_faults()[0].rule_id); + + // The correlation window closes, then a stop is declared and the same code + // starts a new cycle. The rule still owns the mute; the stop must not take it. + engine.begin_planned_stop(); + engine.process_fault("MOTOR_COMM_FL", "ERROR", t0 + 5000ms, /*cycle_started=*/true); + + auto muted = engine.get_muted_faults(); + ASSERT_EQ(1u, muted.size()); + EXPECT_EQ("estop_cascade", muted[0].rule_id) << "the stop overwrote a rule's mute"; + + EXPECT_TRUE(engine.end_planned_stop().empty()); + EXPECT_TRUE(engine.is_muted("MOTOR_COMM_FL")); +} + +TEST_F(CorrelationEngineTest, AClusterMuteLeavesTheStopHoldingItsOwnEntry) { + CorrelationEngine engine(create_cluster_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + + // First of the burst: the cluster is still below min_count, so the stop mutes it. + auto first = engine.process_fault("VALVE_A", "ERROR", t0, /*cycle_started=*/true); + EXPECT_TRUE(first.should_mute); + + // Second reaches min_count. The cluster mutes it as a non-representative, but + // the cluster path writes nothing to the mute map, so the stop stays its owner. + auto second = engine.process_fault("VALVE_B", "ERROR", t0 + 10ms, /*cycle_started=*/true); + EXPECT_TRUE(second.should_mute); + EXPECT_FALSE(second.cluster_id.empty()); + + auto muted = engine.get_muted_faults(); + ASSERT_EQ(2u, muted.size()); + for (const auto & entry : muted) { + EXPECT_EQ(CorrelationEngine::kPlannedStopRuleId, entry.rule_id); + } + + auto unmuted = engine.end_planned_stop(); + std::sort(unmuted.begin(), unmuted.end()); + ASSERT_EQ(2u, unmuted.size()) << "a cluster-claimed fault was orphaned in the mute map"; + EXPECT_EQ("VALVE_A", unmuted[0]); + EXPECT_EQ("VALVE_B", unmuted[1]); + EXPECT_EQ(0u, engine.get_muted_count()); +} + +// ============================================================================ +// Planned stop: dropping and restoring one fault's mute +// ============================================================================ + +TEST_F(CorrelationEngineTest, DroppingAStopMuteReleasesOnlyWhatTheStopOwns) { + CorrelationEngine engine(create_hierarchical_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + engine.process_fault("VALVE_STUCK", "ERROR", t0, /*cycle_started=*/true); + engine.process_fault("ESTOP_001", "CRITICAL", t0 + 10ms, /*cycle_started=*/true); + engine.process_fault("MOTOR_COMM_FL", "ERROR", t0 + 20ms, /*cycle_started=*/true); + // get_muted_faults() is ordered by fault code, so look the entry up rather than + // taking a position in the list. + auto rule_id_of = [&engine](const std::string & code) { + for (const auto & entry : engine.get_muted_faults()) { + if (entry.fault_code == code) { + return entry.rule_id; + } + } + return std::string{""}; + }; + ASSERT_EQ("estop_cascade", rule_id_of("MOTOR_COMM_FL")); + + engine.drop_planned_stop_mute("VALVE_STUCK"); + EXPECT_FALSE(engine.is_muted("VALVE_STUCK")); + + // A rule's mute is not the stop's to drop. + engine.drop_planned_stop_mute("MOTOR_COMM_FL"); + EXPECT_TRUE(engine.is_muted("MOTOR_COMM_FL")); + + auto unmuted = engine.end_planned_stop(); + ASSERT_EQ(1u, unmuted.size()); + EXPECT_EQ("ESTOP_001", unmuted[0]); +} + +TEST_F(CorrelationEngineTest, ARestoredStopMuteBehavesLikeOneTakenLive) { + CorrelationEngine engine{CorrelationConfig{}}; + + // What a restart does: the declaration is read back, then the faults whose + // cycles started inside it are marked again before anything is served. + engine.begin_planned_stop(); + engine.restore_planned_stop_mute("VALVE_STUCK"); + engine.restore_planned_stop_mute("PUMP_SEAL_LEAK"); + + EXPECT_TRUE(engine.is_muted("VALVE_STUCK")); + EXPECT_EQ(2u, engine.get_muted_count()); + const auto muted = engine.get_muted_faults(); + EXPECT_EQ(CorrelationEngine::kPlannedStopRootCause, muted[0].root_cause_code); + + auto unmuted = engine.end_planned_stop(); + std::sort(unmuted.begin(), unmuted.end()); + ASSERT_EQ(2u, unmuted.size()); + EXPECT_EQ("PUMP_SEAL_LEAK", unmuted[0]); + EXPECT_EQ("VALVE_STUCK", unmuted[1]); +} + int main(int argc, char ** argv) { ::testing::InitGoogleTest(&argc, argv); return RUN_ALL_TESTS(); diff --git a/src/ros2_medkit_fault_manager/test/test_fault_manager.cpp b/src/ros2_medkit_fault_manager/test/test_fault_manager.cpp index 92b0abaaf..1c5434fd8 100644 --- a/src/ros2_medkit_fault_manager/test/test_fault_manager.cpp +++ b/src/ros2_medkit_fault_manager/test/test_fault_manager.cpp @@ -38,9 +38,11 @@ #include "ros2_medkit_msgs/msg/snapshot.hpp" #include "ros2_medkit_msgs/srv/clear_fault.hpp" #include "ros2_medkit_msgs/srv/get_fault.hpp" +#include "ros2_medkit_msgs/srv/get_planned_stop.hpp" #include "ros2_medkit_msgs/srv/get_snapshots.hpp" #include "ros2_medkit_msgs/srv/list_faults_for_entity.hpp" #include "ros2_medkit_msgs/srv/report_fault.hpp" +#include "ros2_medkit_msgs/srv/set_planned_stop.hpp" using ros2_medkit_fault_manager::clamp_debounce_counter; using ros2_medkit_fault_manager::compute_debounce_status; @@ -2989,6 +2991,198 @@ TEST_F(SnapshotReadPathTest, FreezeFrameStaysVisibleBehindRetainedSnapshots) { EXPECT_TRUE(frame_served) << "retained snapshots must not hide the most recent freeze-frame"; } +namespace { + +/// A store whose planned-stop write always fails, standing in for a disk that has +/// gone read-only under a running manager. Everything else behaves normally. +class PlannedStopFailingStorage : public ros2_medkit_fault_manager::InMemoryFaultStorage { + public: + void set_planned_stop(const ros2_medkit_fault_manager::PlannedStopState & /*state*/) override { + throw std::runtime_error("planned_stop table is read-only"); + } +}; + +} // namespace + +/// The switch driven over its real services, against a node in this process. +class PlannedStopServiceTest : public ::testing::Test { + protected: + static inline std::atomic test_counter_{0}; + + void SetUp() override { + const std::string ns = "/test_planned_stop_" + std::to_string(test_counter_.fetch_add(1)); + + rclcpp::NodeOptions fm_options; + fm_options.parameter_overrides({ + rclcpp::Parameter("storage_type", "memory"), + rclcpp::Parameter("confirmation_threshold", -1), + }); + fm_options.arguments({"--ros-args", "-r", "__ns:=" + ns}); + fault_manager_ = std::make_shared(fm_options); + + rclcpp::NodeOptions test_options; + test_options.arguments({"--ros-args", "-r", "__ns:=" + ns}); + test_node_ = std::make_shared("planned_stop_test_client", test_options); + + set_client_ = + test_node_->create_client(ns + "/fault_manager/set_planned_stop"); + get_client_ = + test_node_->create_client(ns + "/fault_manager/get_planned_stop"); + ASSERT_TRUE(set_client_->wait_for_service(std::chrono::seconds(5))); + ASSERT_TRUE(get_client_->wait_for_service(std::chrono::seconds(5))); + } + + void TearDown() override { + set_client_.reset(); + get_client_.reset(); + test_node_.reset(); + fault_manager_.reset(); + } + + template + bool spin_until_future_ready(FutureT & future, std::chrono::milliseconds timeout = std::chrono::milliseconds(3000)) { + auto start = std::chrono::steady_clock::now(); + while (std::chrono::steady_clock::now() - start < timeout) { + rclcpp::spin_some(fault_manager_); + rclcpp::spin_some(test_node_); + if (future.wait_for(std::chrono::milliseconds(0)) == std::future_status::ready) { + return true; + } + std::this_thread::sleep_for(std::chrono::milliseconds(5)); + } + return future.wait_for(std::chrono::milliseconds(0)) == std::future_status::ready; + } + + ros2_medkit_msgs::srv::SetPlannedStop::Response set_stop(bool active, const std::string & reason, + const std::string & declared_by) { + auto request = std::make_shared(); + request->active = active; + request->reason = reason; + request->declared_by = declared_by; + auto future = set_client_->async_send_request(request); + EXPECT_TRUE(spin_until_future_ready(future)); + return *future.get(); + } + + ros2_medkit_msgs::srv::GetPlannedStop::Response get_stop() { + auto request = std::make_shared(); + auto future = get_client_->async_send_request(request); + EXPECT_TRUE(spin_until_future_ready(future)); + return *future.get(); + } + + std::shared_ptr fault_manager_; + std::shared_ptr test_node_; + rclcpp::Client::SharedPtr set_client_; + rclcpp::Client::SharedPtr get_client_; +}; + +// A store that cannot record the declaration must not leave the caller believing +// a stop is in force, and must not take the process down with it. +TEST_F(PlannedStopServiceTest, AStoreThatRejectsTheWriteIsReportedNotThrown) { + fault_manager_->set_storage_for_test(std::make_unique()); + + const auto response = set_stop(true, "line 3 maintenance", "shift_lead"); + EXPECT_FALSE(response.success); + EXPECT_FALSE(response.message.empty()); + EXPECT_FALSE(response.was_active); + + // The switch is where it was, and the node is still answering. + const auto state = get_stop(); + EXPECT_FALSE(state.active); + EXPECT_TRUE(state.reason.empty()); +} + +// "Why was line 3 quiet on Friday?" has to be answerable after the plant is back +// up, without reading a raw audit database. +TEST_F(PlannedStopServiceTest, TheLastDeclarationIsReadableAfterTheWithdrawal) { + const auto before_any = get_stop(); + EXPECT_FALSE(before_any.active); + EXPECT_EQ(0, before_any.since.sec); + EXPECT_EQ(0, before_any.ended_at.sec); + + ASSERT_TRUE(set_stop(true, "line 3 quarterly maintenance", "shift_lead").success); + const auto during = get_stop(); + ASSERT_TRUE(during.active); + EXPECT_EQ("line 3 quarterly maintenance", during.reason); + EXPECT_GT(during.since.sec, 0); + EXPECT_EQ(0, during.ended_at.sec) << "a stop that is still in force has not ended"; + + ASSERT_TRUE(set_stop(false, "plant back up", "shift_lead").success); + + const auto after = get_stop(); + EXPECT_FALSE(after.active); + EXPECT_EQ("line 3 quarterly maintenance", after.reason) << "the declaration that started the stop"; + EXPECT_EQ("shift_lead", after.declared_by); + EXPECT_EQ(during.since.sec, after.since.sec); + EXPECT_GT(after.ended_at.sec, 0) << "the withdrawal has to be readable too"; +} + +// The audit rows for the switch describe the installation, so they carry the same +// sentinel fault code the log's own lifecycle markers use. +TEST(PlannedStopAuditTest, TransitionsUseTheInstallationSentinelFaultCode) { + const std::string audit_path = make_temp_audit_path("test_audit_planned_stop_"); + { + rclcpp::NodeOptions options; + options.parameter_overrides({ + {"storage_type", "memory"}, + {"audit_log.enabled", true}, + {"audit_log.database_path", audit_path}, + }); + options.arguments({"--ros-args", "-r", "__ns:=/test_planned_stop_audit"}); + auto node = std::make_shared(options); + + auto client_node = std::make_shared( + "planned_stop_audit_client", + rclcpp::NodeOptions().arguments({"--ros-args", "-r", "__ns:=/test_planned_stop_audit"})); + auto client = client_node->create_client( + "/test_planned_stop_audit/fault_manager/set_planned_stop"); + ASSERT_TRUE(client->wait_for_service(std::chrono::seconds(5))); + + auto call = [&](bool active, const std::string & reason) { + auto request = std::make_shared(); + request->active = active; + request->reason = reason; + request->declared_by = "shift_lead"; + auto future = client->async_send_request(request); + auto start = std::chrono::steady_clock::now(); + while (std::chrono::steady_clock::now() - start < std::chrono::seconds(3)) { + rclcpp::spin_some(node); + rclcpp::spin_some(client_node); + if (future.wait_for(std::chrono::milliseconds(0)) == std::future_status::ready) { + break; + } + std::this_thread::sleep_for(std::chrono::milliseconds(5)); + } + ASSERT_EQ(future.wait_for(std::chrono::milliseconds(0)), std::future_status::ready); + ASSERT_TRUE(future.get()->success); + }; + call(true, "maintenance"); + call(false, "back up"); + + const auto * log = node->get_audit_log_for_test(); + ASSERT_NE(log, nullptr); + bool saw_started = false; + bool saw_ended = false; + for (const auto & record : log->read()) { + if (record.event.transition == ros2_medkit_fault_manager::kTransitionPlannedStopStarted) { + saw_started = true; + EXPECT_EQ("__audit__", record.event.fault_code); + EXPECT_EQ("maintenance", record.event.description); + EXPECT_EQ("shift_lead", record.event.source_id); + } else if (record.event.transition == ros2_medkit_fault_manager::kTransitionPlannedStopEnded) { + saw_ended = true; + EXPECT_EQ("__audit__", record.event.fault_code); + EXPECT_EQ("back up", record.event.description); + } + } + EXPECT_TRUE(saw_started); + EXPECT_TRUE(saw_ended); + EXPECT_TRUE(log->verify().ok); + } + remove_audit_files(audit_path); +} + int main(int argc, char ** argv) { rclcpp::init(argc, argv); ::testing::InitGoogleTest(&argc, argv); diff --git a/src/ros2_medkit_fault_manager/test/test_sqlite_storage.cpp b/src/ros2_medkit_fault_manager/test/test_sqlite_storage.cpp index c75a5a91f..859fe8aa7 100644 --- a/src/ros2_medkit_fault_manager/test/test_sqlite_storage.cpp +++ b/src/ros2_medkit_fault_manager/test/test_sqlite_storage.cpp @@ -2466,6 +2466,58 @@ TEST_F(SqliteFaultStorageTest, WithdrawnPlannedStopIsWhatAReopenSees) { EXPECT_EQ(0, state.since_ns); } +TEST_F(SqliteFaultStorageTest, AWithdrawnDeclarationKeepsItsReasonAndItsEndTime) { + ros2_medkit_fault_manager::PlannedStopState declared; + declared.active = true; + declared.reason = "line 3 quarterly maintenance"; + declared.declared_by = "shift_lead"; + declared.since_ns = 1757000000000000000; + storage_->set_planned_stop(declared); + + // What a withdrawal writes: the declaration survives it, stamped with its end. + auto withdrawn = declared; + withdrawn.active = false; + withdrawn.ended_at_ns = 1757000009000000000; + storage_->set_planned_stop(withdrawn); + + storage_.reset(); + storage_ = std::make_unique(temp_db_path_.string()); + + const auto state = storage_->get_planned_stop(); + EXPECT_FALSE(state.active); + EXPECT_EQ("line 3 quarterly maintenance", state.reason); + EXPECT_EQ("shift_lead", state.declared_by); + EXPECT_EQ(1757000000000000000, state.since_ns); + EXPECT_EQ(1757000009000000000, state.ended_at_ns); +} + +TEST_F(SqliteFaultStorageTest, ADatabaseWrittenBeforeTheEndTimeExistedStillOpens) { + // A row written by a build that had no ended_at_ns column reads as one that was + // never withdrawn, which is what a missing end time means. + storage_.reset(); + { + sqlite3 * raw = nullptr; + ASSERT_EQ(sqlite3_open(temp_db_path_.string().c_str(), &raw), SQLITE_OK); + ASSERT_EQ(sqlite3_exec(raw, "DROP TABLE IF EXISTS planned_stop", nullptr, nullptr, nullptr), SQLITE_OK); + ASSERT_EQ(sqlite3_exec(raw, + "CREATE TABLE planned_stop (id INTEGER PRIMARY KEY CHECK (id = 1), active INTEGER NOT NULL," + " reason TEXT NOT NULL, declared_by TEXT NOT NULL, since_ns INTEGER NOT NULL);" + "INSERT INTO planned_stop (id, active, reason, declared_by, since_ns)" + " VALUES (1, 1, 'legacy stop', 'shift_lead', 42);", + nullptr, nullptr, nullptr), + SQLITE_OK); + sqlite3_close(raw); + } + + storage_ = std::make_unique(temp_db_path_.string()); + + const auto state = storage_->get_planned_stop(); + EXPECT_TRUE(state.active); + EXPECT_EQ("legacy stop", state.reason); + EXPECT_EQ(42, state.since_ns); + EXPECT_EQ(0, state.ended_at_ns); +} + TEST_F(SqliteFaultStorageTest, PlannedStopCarriesAKilobyteReason) { const std::string long_reason(1024, 'r'); ros2_medkit_fault_manager::PlannedStopState declared; diff --git a/src/ros2_medkit_integration_tests/test/features/test_planned_stop_switch.test.py b/src/ros2_medkit_integration_tests/test/features/test_planned_stop_switch.test.py index eb0f0a6a9..b1f019364 100644 --- a/src/ros2_medkit_integration_tests/test/features/test_planned_stop_switch.test.py +++ b/src/ros2_medkit_integration_tests/test/features/test_planned_stop_switch.test.py @@ -48,6 +48,12 @@ MUTED_CODE = 'GW_PS_MUTED' SECOND_CODE = 'GW_PS_MUTED_TWO' AFTER_CODE = 'GW_PS_AFTER' +ALREADY_UP_CODE = 'GW_PS_ALREADY_UP' + +# How long to keep watching after the counts are right. The publish path is one +# service call plus a topic hop plus the SSE writer, so a second frame that the +# assertion could otherwise race arrives well inside this. +SETTLE_SEC = 4.0 SET_STOP = '/apps/fault_manager/operations/set_planned_stop/executions' GET_STOP = '/apps/fault_manager/operations/get_planned_stop/executions' @@ -299,9 +305,21 @@ def _both_muted(): and self._count_confirmed(SECOND_CODE) == 1, 'the stream did not carry exactly one confirmation per released fault') + # Reaching one is not the claim: staying at one is. A duplicate frame + # published a moment later would satisfy the poll above and never be seen. + time.sleep(SETTLE_SEC) + self.assertEqual(1, self._count_confirmed(MUTED_CODE), + 'a second confirmation arrived after the first') + self.assertEqual(1, self._count_confirmed(SECOND_CODE), + 'a second confirmation arrived after the first') + + # The declaration stays readable after the plant is back up, with the time + # it ended, so "why was line 3 quiet?" is answerable over the same route. state = self._get_stop() self.assertFalse(state['active']) - self.assertEqual('', state['reason']) + self.assertEqual('line 3 maintenance', state['reason']) + self.assertEqual('shift_lead', state['declared_by']) + self.assertGreater(state['ended_at']['sec'], 0) def test_a_fault_raised_after_the_stop_is_announced_at_once(self): """A fault raised once the stop is over is announced immediately. @@ -315,8 +333,41 @@ def test_a_fault_raised_after_the_stop_is_announced_at_once(self): self._wait_until(lambda: self._count_confirmed(AFTER_CODE) == 1, 'a fault raised after the stop ended was not announced') + time.sleep(SETTLE_SEC) + self.assertEqual(1, self._count_confirmed(AFTER_CODE)) self.assertIn(AFTER_CODE, self._faults()[1]) + def test_a_fault_already_up_when_the_stop_begins_stays_visible(self): + """A standing alarm is not hidden, and not announced twice. + + @verifies REQ_INTEROP_012 + """ + self._report(ALREADY_UP_CODE) + self._wait_until(lambda: self._count_confirmed(ALREADY_UP_CODE) == 1, + 'the fault was never announced before the stop') + self.assertIn(ALREADY_UP_CODE, self._faults()[1]) + + self._set_stop(True, reason='line 3 maintenance', declared_by='shift_lead') + + # The reporter keeps sending FAILED while the condition holds. + for _ in range(3): + self._report(ALREADY_UP_CODE) + + data, visible = self._faults(include_muted=True) + muted_codes = {entry['fault_code'] + for entry in data['x-medkit'].get('muted_faults', [])} + self.assertNotIn(ALREADY_UP_CODE, muted_codes, + 'the stop took over a cycle that started before it') + self.assertIn(ALREADY_UP_CODE, self._faults()[1], + 'a standing alarm vanished from GET /faults when the stop began') + + self._set_stop(False, reason='line 3 back up', declared_by='shift_lead') + + time.sleep(SETTLE_SEC) + self.assertEqual( + 1, self._count_confirmed(ALREADY_UP_CODE), + 'the switch-off announced a confirmation the operator had already seen') + @launch_testing.post_shutdown_test() class TestShutdown(unittest.TestCase): diff --git a/src/ros2_medkit_msgs/srv/GetPlannedStop.srv b/src/ros2_medkit_msgs/srv/GetPlannedStop.srv index ad752cf07..7e97bfdfd 100644 --- a/src/ros2_medkit_msgs/srv/GetPlannedStop.srv +++ b/src/ros2_medkit_msgs/srv/GetPlannedStop.srv @@ -15,6 +15,12 @@ # GetPlannedStop.srv - Read the FaultManager's planned-stop declaration. # # The request carries no fields: the switch is a single declaration per fault manager. +# (rosidl gives an empty request one `structure_needs_at_least_one_member` byte, which +# is why introspecting clients see a single phantom parameter here.) +# +# The response describes the declaration in force, or - once it has been withdrawn - +# the last one there was, so "why was line 3 quiet on Friday?" is answerable after +# the plant is back up without reading the audit database. --- # Response fields @@ -22,11 +28,17 @@ # Whether a planned stop is currently declared. bool active -# The reason given when the stop was declared. Empty while no stop is declared. +# The reason given when the stop was declared. Retained after the withdrawal; +# empty only while no stop has ever been declared. string reason -# Who declared the stop. Empty while no stop is declared. +# Who declared the stop. Retained after the withdrawal; empty only while no stop +# has ever been declared. string declared_by -# When the stop was declared, wall clock. Zero while no stop is declared. +# When the stop was declared, wall clock. Zero while no stop has ever been declared. builtin_interfaces/Time since + +# When the stop was withdrawn, wall clock. Zero while one is in force, and while +# no stop has ever been declared. +builtin_interfaces/Time ended_at diff --git a/src/ros2_medkit_msgs/srv/SetPlannedStop.srv b/src/ros2_medkit_msgs/srv/SetPlannedStop.srv index 0eea40cb5..a439025fd 100644 --- a/src/ros2_medkit_msgs/srv/SetPlannedStop.srv +++ b/src/ros2_medkit_msgs/srv/SetPlannedStop.srv @@ -14,15 +14,25 @@ # # SetPlannedStop.srv - Declare, or withdraw, a planned stop on the FaultManager. # -# While a planned stop is on, every fault whose cycle starts is reported, debounced, +# While a planned stop is on, a fault whose cycle STARTS is reported, debounced, # confirmed, captured and audited exactly as it would be otherwise, and is MARKED as # muted rather than dropped: it is absent from the default ListFaults response, counted -# in `muted_count`, listed under `muted_faults` when `include_muted` is set, and no -# FaultEvent is published for it. Withdrawing the stop unmutes every fault it muted that -# is still active, and republishes an EVENT_CONFIRMED for each one that is CONFIRMED, -# because that confirmation was never announced. +# in `muted_count`, and listed under `muted_faults` when `include_muted` is set. A cycle +# that started BEFORE the stop is untouched: it stays in the fault list, because its +# confirmation has already been announced, and reporters re-send FAILED for as long as +# the condition holds. # -# The declaration survives a restart of the fault manager. +# A marked fault is published exactly as a rule-muted symptom is: its EVENT_CONFIRMED +# and EVENT_UPDATED are withheld, while an EVENT_CLEARED - the fault healing, or being +# acknowledged - is published as usual. Withdrawing the stop unmutes every fault it +# muted that is still active, and publishes one EVENT_CONFIRMED for each that is +# CONFIRMED, because that confirmation was never announced. +# +# The declaration survives a restart of the fault manager, and so does the marking of +# the faults whose cycles started inside it. +# +# Both transitions are recorded in the fault audit log when `audit_log.enabled` is set +# (it is off by default). # Request fields From ce85f51ee70a69d8e759b007cc8e595b76035f21 Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Mon, 7 Sep 2026 18:08:07 +0200 Subject: [PATCH 06/12] fault_manager: pin the mute boundary and the silence, and say what changed The silence assertion now reads the exact set of event types published per fault code, and the scenario re-reports a marked fault so that an update leaking past the mute is a case the test can actually see - counting confirmations alone could not. The exactly-once assertions wait out a settle window after the counts are right, because a duplicate published a moment later satisfies a poll and is never seen. New cases: a fault already up when the stop begins stays visible and is announced once, over the services and over the gateway; a fault raised again after a clear is marked; a scoped acknowledgement releases the mute; a cluster rule does not orphan it; the declaration is readable after the withdrawal; a store that refuses the write is reported rather than fatal; the audit rows carry the installation sentinel. Docs on every surface that described the muting: what a cycle is, that publication equals a rule-muted symptom exactly (CLEARED included), that the audit rows exist only when the audit log is on, that the declaration outlives the stop, and that the correlation cleanup timer now runs on every manager. Refs #656 --- src/ros2_medkit_fault_manager/design/index.rst | 7 +++++++ .../test/integration/test_planned_stop.test.py | 12 ++++++++++-- 2 files changed, 17 insertions(+), 2 deletions(-) diff --git a/src/ros2_medkit_fault_manager/design/index.rst b/src/ros2_medkit_fault_manager/design/index.rst index 9250c2a76..6814b14a8 100644 --- a/src/ros2_medkit_fault_manager/design/index.rst +++ b/src/ros2_medkit_fault_manager/design/index.rst @@ -312,6 +312,13 @@ declarer and produces an audit record, none of which a parameter can do; and it not a new HTTP route because the gateway already exposes a node's services as SOVD operations on its App entity. +One cost falls on every deployment, correlation or not: the engine is now always +constructed, so the ``correlation.cleanup_interval_sec`` timer (5 s by default) +runs on every fault manager. With no rules loaded a tick walks two empty +containers - ``pending_root_causes_`` and ``pending_clusters_`` - and returns. The +alternative, building the engine lazily when a stop is first declared, would put a +second construction path under the service handler for no measurable saving. + Rosbag Black-Box Recording ~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/src/ros2_medkit_fault_manager/test/integration/test_planned_stop.test.py b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop.test.py index e27ba058a..fad5ae5f2 100644 --- a/src/ros2_medkit_fault_manager/test/integration/test_planned_stop.test.py +++ b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop.test.py @@ -297,7 +297,8 @@ def _count_events(cls, fault_code, event_type): @classmethod def _event_types(cls, fault_code): - """Every event type published for this code, as a set. + """ + Return every event type published for this code, as a set. Counting one type cannot see a leak of another: a stop that announced an UPDATE it should have withheld passes a CONFIRMED-only assertion. @@ -411,8 +412,15 @@ def test_a_fault_raised_during_a_stop_is_marked_not_dropped(self): f'no snapshot was captured for the muted fault {code}') self.assertIn(CAPTURE_TOPIC, captured) + # Re-report the marked fault while the stop still stands. This is the path + # that produces EVENT_UPDATED, and it is the only way an assertion about + # "nothing was announced" can see an update leaking out. + self._report(code) + self._wait_until(lambda: self._get_fault(code).fault.occurrence_count >= 1, + f'{code} left the store') + # Nothing at all was announced while the stop stood - not the confirmation, - # and not an update either. + # and not the update either. self.assertEqual(set(), self._event_types(code)) withdrawal = self._set_stop(False, reason='line 3 back up', declared_by='shift_lead') From fbcf0dc4ba513654040bd8d94a7dcfbdc8a6d1d4 Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Mon, 7 Sep 2026 18:19:22 +0200 Subject: [PATCH 07/12] docs: correct the message package's planned-stop description It repeated the claim that a marked fault is never published as a FaultEvent, and listed a GetPlannedStop response that no longer matches the service. Refs #656 --- src/ros2_medkit_msgs/README.md | 33 +++++++++++++++++++++------------ 1 file changed, 21 insertions(+), 12 deletions(-) diff --git a/src/ros2_medkit_msgs/README.md b/src/ros2_medkit_msgs/README.md index 01ac6d604..9bce3944b 100644 --- a/src/ros2_medkit_msgs/README.md +++ b/src/ros2_medkit_msgs/README.md @@ -146,7 +146,7 @@ Declare, or withdraw, a planned stop on the FaultManager. | Field | Type | Description | |-------|------|-------------| | `active` | bool | `true` declares a planned stop, `false` withdraws it | -| `reason` | string | Why the plant is stopped; recorded in the audit log and served by `GetPlannedStop` | +| `reason` | string | Why the plant is stopped; recorded in the audit log and served by `GetPlannedStop`, including after the withdrawal | | `declared_by` | string | Who declared the transition; recorded as the audit record's source | **Response:** @@ -156,25 +156,34 @@ Declare, or withdraw, a planned stop on the FaultManager. | `message` | string | Status or error message | | `was_active` | bool | The state of the switch before this request | -While a planned stop is on, a fault whose cycle starts is reported, debounced, confirmed, -captured and audited unchanged, and is marked as muted: absent from the default -`ListFaults` response, counted in `muted_count`, listed under `muted_faults` with -`rule_id = "planned_stop"` when `include_muted` is set, and never published as a -`FaultEvent`. Withdrawing the stop unmutes every fault it muted that is still active and -publishes one `EVENT_CONFIRMED` for each of those that is CONFIRMED. A request asking for -the state the switch is already in succeeds, changes nothing, and writes no audit record. +While a planned stop is on, a fault whose cycle *starts* is reported, debounced, +confirmed, captured and audited unchanged, and is marked as muted: absent from the default +`ListFaults` response, counted in `muted_count`, and listed under `muted_faults` with +`rule_id = "planned_stop"` when `include_muted` is set. A cycle that started *before* the +stop is untouched - reporters re-send FAILED for as long as a condition holds, and that +fault's confirmation has already been announced. + +Publication matches a rule-muted symptom exactly: `EVENT_CONFIRMED` and `EVENT_UPDATED` +are withheld, `EVENT_CLEARED` is published as usual. Withdrawing the stop unmutes every +fault it alone was muting and publishes one `EVENT_CONFIRMED` for each of those that is +CONFIRMED. A request asking for the state the switch is already in succeeds, changes +nothing, and writes no audit record; a request the store cannot record answers +`success: false` and changes nothing. Audit records exist only when `audit_log.enabled` +is set, which it is not by default. ### GetPlannedStop.srv -Read the planned-stop declaration. The request carries no fields. +Read the planned-stop declaration, or - once it has been withdrawn - the last one there +was. The request carries no fields. **Response:** | Field | Type | Description | |-------|------|-------------| | `active` | bool | Whether a planned stop is declared | -| `reason` | string | The reason given when it was declared; empty while none is | -| `declared_by` | string | Who declared it; empty while none is declared | -| `since` | builtin_interfaces/Time | When it was declared, wall clock; zero while none is | +| `reason` | string | The reason given when it was declared; retained after the withdrawal | +| `declared_by` | string | Who declared it; retained after the withdrawal | +| `since` | builtin_interfaces/Time | When it was declared, wall clock | +| `ended_at` | builtin_interfaces/Time | When it was withdrawn; zero while one is in force | ## Usage From 720a4da96f3105eb900c4a4c14c64817c84772ae Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Tue, 8 Sep 2026 09:05:19 +0200 Subject: [PATCH 08/12] fault_manager: make the planned stop own fault cycles, and persist that The stop marked faults by writing a mute entry and remembering the codes in a set. That set was the only record, so it could not survive a restart intact, a cluster rule could strand an entry it never wrote, and a rule releasing a fault mid-stop dropped it out of the stop for good - its confirmation announced by nobody, which is the failure marking exists to prevent. Ownership is now a flag on the fault row: the stop owns a CYCLE that starts while it is declared, for the whole cycle. The mute is derived from that. A rule's mute overlays an owned fault instead of taking it, and when the overlay ends - root cause acknowledged, window closed, cluster expired - the engine re-asserts the stop's mute. Ownership ends when the fault is acknowledged or the stop is withdrawn, and a restart reads the flags back rather than comparing timestamps, which a clock step would break and which cannot tell a rule's mute from the stop's. A cycle now also starts when a healed fault fails again: the manager publishes EVENT_CLEARED at the heal, so the confirmation that follows is fresh news. occurrence_count keeps its own, narrower definition. Muting gates the PASSED path too. should_mute was computed for FAILED reports only, so a PASSED that left a muted fault CONFIRMED announced an update for a fault the list was hiding; and it was computed from what the report matched rather than from the mute map, so a repeat report of a fault whose rule had stopped matching did the same. The withdrawal writes the declaration, announces, and clears the flags last, so a process that dies in between leaves faults owned by an ended declaration - which the next startup recognises, announces once a consumer is listening, and clears. Clearing first would have turned a crash into permanent silence. The storage-injection seam is a protected constructor rather than a public runtime swap: the backend is in place before snapshot and rosbag capture borrow a raw pointer to it. Refs #656 --- docs/api/messages.rst | 21 +- docs/config/fault-manager.rst | 46 ++- src/ros2_medkit_fault_manager/CMakeLists.txt | 6 + src/ros2_medkit_fault_manager/README.md | 67 ++-- .../design/index.rst | 72 ++-- .../correlation/correlation_engine.hpp | 49 ++- .../fault_audit_log.hpp | 5 +- .../fault_manager_node.hpp | 46 ++- .../fault_storage.hpp | 25 ++ .../sqlite_fault_storage.hpp | 3 + .../src/correlation/correlation_engine.cpp | 120 ++++--- .../src/fault_manager_node.cpp | 121 +++++-- .../src/fault_storage.cpp | 35 ++ .../src/sqlite_fault_storage.cpp | 66 +++- .../integration/test_planned_stop.test.py | 116 ++++++- ...t_planned_stop_interrupted_release.test.py | 315 ++++++++++++++++++ .../test_planned_stop_restart.test.py | 57 +++- .../test/test_correlation_engine.cpp | 178 +++++++++- .../test/test_fault_manager.cpp | 32 +- .../test/test_sqlite_storage.cpp | 96 ++++++ src/ros2_medkit_msgs/README.md | 20 +- src/ros2_medkit_msgs/srv/GetPlannedStop.srv | 8 +- src/ros2_medkit_msgs/srv/SetPlannedStop.srv | 41 ++- 23 files changed, 1309 insertions(+), 236 deletions(-) create mode 100644 src/ros2_medkit_fault_manager/test/integration/test_planned_stop_interrupted_release.test.py diff --git a/docs/api/messages.rst b/docs/api/messages.rst index 7f923a729..37d841738 100644 --- a/docs/api/messages.rst +++ b/docs/api/messages.rst @@ -318,17 +318,20 @@ Declare, withdraw and read a planned stop on the fault manager. builtin_interfaces/Time since # when it was declared builtin_interfaces/Time ended_at # when it was withdrawn; zero while one is in force -While a planned stop is on, a fault whose cycle *starts* is reported, debounced, -confirmed, captured and audited unchanged, and is marked as muted: absent from -the default ``ListFaults`` response, counted in ``muted_count``, and listed under -``muted_faults`` with ``rule_id: planned_stop`` when ``include_muted`` is set. A -cycle that started before the stop is untouched. +While a planned stop is on, it owns every fault cycle that *starts* - a new fault, +one raised again after being cleared, or one that fails again after healing. An +owned fault is reported, debounced, confirmed, captured and audited unchanged, and +is marked as muted: absent from the default ``ListFaults`` response, counted in +``muted_count``, and listed under ``muted_faults`` with ``rule_id: planned_stop`` +when ``include_muted`` is set. A cycle that started before the stop is untouched. Publication matches a rule-muted symptom exactly: ``EVENT_CONFIRMED`` and -``EVENT_UPDATED`` are withheld, ``EVENT_CLEARED`` is published as usual. -Withdrawing the stop releases every fault it alone was muting and publishes one -``EVENT_CONFIRMED`` per released fault that is CONFIRMED. The audit records exist -only when ``audit_log.enabled`` is set, which it is not by default. +``EVENT_UPDATED`` are withheld whichever kind of report produced them, +``EVENT_CLEARED`` is published as usual. Withdrawing the stop releases every fault +it owns and publishes one ``EVENT_CONFIRMED`` per released fault that is CONFIRMED; +a fault a correlation rule is muting stays muted and is not announced, and returns +to the stop's mute if the rule lets go while the stop is still on. The audit records +exist only when ``audit_log.enabled`` is set, which it is not by default. See :doc:`/config/fault-manager` for the configuration that decides whether the declaration survives a restart and whether the transitions are audited. diff --git a/docs/config/fault-manager.rst b/docs/config/fault-manager.rst index dff5602df..390ed2558 100644 --- a/docs/config/fault-manager.rst +++ b/docs/config/fault-manager.rst @@ -611,7 +611,9 @@ Fault correlation identifies root causes and filters symptom faults. - Interval for running correlation cleanup tasks. The correlation engine is constructed on every fault manager, configured rules or not, because the planned-stop switch needs none - so this timer runs everywhere. With no - rules loaded each tick walks two empty containers and returns. + rules loaded a tick takes the engine's mutex, walks two empty containers and + re-asserts an empty ownership set: measured at ~190 ns per call, so ~2.3 us + of CPU per minute at the 5 s default. .. seealso:: @@ -642,9 +644,11 @@ Two configuration choices decide how much the switch can do: - Effect on the planned stop * - ``storage_type`` - ``sqlite`` - - The declaration is stored beside the faults, so it survives a restart. - With ``memory`` it lives only for the process, and a restart inside a stop - ends it. + - The declaration, and the per-fault ownership flags behind it, are stored + beside the faults, so both survive a restart. With ``memory`` they live only + for the process, and a restart inside a stop ends it. A correlation rule's + mute is not persisted on either backend, so a rule-muted fault comes back + unmuted after a restart unless the stop owns it. * - ``audit_log.enabled`` - ``false`` - **Off by default, so a stock manager records no audit row for the switch @@ -653,20 +657,26 @@ Two configuration choices decide how much the switch can do: transition's own reason and declarer. Recorded even under ``audit_log.transitions: confirmed_only``. -While the stop stands, a fault whose cycle *starts* is reported, debounced, -confirmed, captured and audited unchanged, and is marked as muted rather than -dropped: absent from the default fault list, counted in ``muted_count``, and -listed under ``muted_faults`` with ``rule_id: planned_stop`` when muted entries -are requested. A fault that was already up when the stop was declared is left -alone - its confirmation has already been announced, and reporters re-send FAILED -for as long as the condition holds. - -A marked fault is published exactly as a rule-muted symptom is: ``EVENT_CONFIRMED`` -and ``EVENT_UPDATED`` are withheld, while ``EVENT_CLEARED`` - the fault healing, or -being acknowledged - is published as usual. Withdrawing the stop releases every -fault it alone was muting and publishes one confirmation event for each of those -that is CONFIRMED. Correlation rules are unaffected: a symptom a rule mutes stays -muted when the stop is withdrawn, and the stop never takes a mute a rule owns. +While the stop stands, it owns every fault cycle that *starts*: a new fault, one +raised again after being cleared, or one that fails again after healing. An owned +fault is reported, debounced, confirmed, captured and audited unchanged, and is +marked as muted rather than dropped: absent from the default fault list, counted in +``muted_count``, and listed under ``muted_faults`` with ``rule_id: planned_stop`` +when muted entries are requested. A fault that was already up when the stop was +declared is left alone - its confirmation has already been announced, and reporters +re-send FAILED for as long as the condition holds. + +Ownership is a flag on the fault row, so it survives a restart and is what the +switch-off releases. The mute is derived from it: a correlation rule muting an owned +fault overlays the stop rather than taking the fault from it, the withdrawal leaves +a rule-held fault alone, and when the rule lets go - root cause acknowledged, window +closed, cluster expired - the fault is muted by the stop again. + +A muted fault is published exactly as a rule-muted symptom is: ``EVENT_CONFIRMED`` +and ``EVENT_UPDATED`` are withheld whichever kind of report produced them, while +``EVENT_CLEARED`` - the fault healing, or being acknowledged - is published as +usual. Withdrawing the stop releases every fault it owns and publishes one +confirmation event for each of those that is CONFIRMED. ``~/get_planned_stop`` keeps serving the declaration after the withdrawal, with ``ended_at`` stamped, so the reason stays readable once the plant is back up. A diff --git a/src/ros2_medkit_fault_manager/CMakeLists.txt b/src/ros2_medkit_fault_manager/CMakeLists.txt index 4b02cd3d8..cf0513846 100644 --- a/src/ros2_medkit_fault_manager/CMakeLists.txt +++ b/src/ros2_medkit_fault_manager/CMakeLists.txt @@ -219,6 +219,12 @@ if(BUILD_TESTING) medkit_add_launch_test(test_planned_stop_restart test/integration/test_planned_stop_restart.test.py TIMEOUT 180 LABELS "integration") + # A switch-off killed between the declaration write and the flag clearing: the + # store is left as that crash leaves it and the replacement process has to + # finish the release. Its respawn delay is what the timeout is sized for. + medkit_add_launch_test(test_planned_stop_interrupted_release + test/integration/test_planned_stop_interrupted_release.test.py TIMEOUT 180 LABELS "integration") + # auto_confirm_after_sec confirms from a timer rather than a report, so the # mute has to be honoured on that path too. Its own launch, because every other # planned-stop case would then race a timer that confirms behind its back. diff --git a/src/ros2_medkit_fault_manager/README.md b/src/ros2_medkit_fault_manager/README.md index b59cbac3e..c681eff58 100644 --- a/src/ros2_medkit_fault_manager/README.md +++ b/src/ros2_medkit_fault_manager/README.md @@ -60,28 +60,38 @@ ros2 service call /fault_manager/set_planned_stop ros2_medkit_msgs/srv/SetPlanne ros2 service call /fault_manager/get_planned_stop ros2_medkit_msgs/srv/GetPlannedStop "{}" ``` -**Marked, never dropped.** While the stop stands, a fault whose cycle *starts* -goes through report, debounce, confirmation, snapshot and rosbag capture and the -audit log exactly as it would otherwise. What changes is only how it is *shown*: -it is registered as muted with `rule_id: planned_stop` and the pseudo root cause -`PLANNED_STOP`, so it is absent from the default `~/list_faults` response, -counted in `muted_count`, and listed under `muted_faults` when `include_muted` is -set. +**Marked, never dropped.** While the stop stands, it *owns* every fault cycle that +starts. An owned fault goes through report, debounce, confirmation, snapshot and +rosbag capture and the audit log exactly as it would otherwise. What changes is +only how it is *shown*: it is registered as muted with `rule_id: planned_stop` and +the pseudo root cause `PLANNED_STOP`, so it is absent from the default +`~/list_faults` response, counted in `muted_count`, and listed under +`muted_faults` when `include_muted` is set. + +**Ownership is the truth, and it is persisted.** The flag lives on the fault row in +the store, not in the process, so a restart reads back exactly which cycles the +stop owns - no timestamp comparison, which a clock step would break and which +cannot tell a rule's mute from the stop's. The mute is derived from ownership: an +owned fault is muted unless a rule's mute overlays it, and when that overlay ends +the fault is muted by the stop again. **Only a cycle that starts inside the stop.** A fault that was already up when the stop was declared keeps its place in the fault list. Reporters are level-triggered - `FaultReporter::report()` is called for as long as the condition holds, not once per transition - so marking on any report would take a standing alarm off the list -and then announce it a second time at the switch-off. The cycle boundary is the -same one `~/report_fault` already computes: a new fault, or one raised again after -being cleared. +and then announce it a second time at the switch-off. A cycle starts on a new +fault, on one raised again after being cleared, and on one that fails again after +healing (the heal published the fault's end, so the next confirmation is fresh +news). `occurrence_count` keeps its own, narrower definition and counts the first +two only. **Published exactly like a rule-muted symptom.** `EVENT_CONFIRMED` and -`EVENT_UPDATED` are withheld while the fault is marked. `EVENT_CLEARED` is not: a -fault that heals past the healing threshold, or that is acknowledged, publishes it -as any muted fault does. Consumers therefore see the end of a fault whose start -they never heard - that is the existing muting contract, not something the switch -changes. +`EVENT_UPDATED` are withheld while the fault is muted, whichever kind of report +produced them - a PASSED that leaves the fault CONFIRMED announces nothing either. +`EVENT_CLEARED` is not withheld: a fault that heals past the healing threshold, or +that is acknowledged, publishes it as any muted fault does. Consumers therefore see +the end of a fault whose start they never heard - that is the existing muting +contract, not something the switch changes. **Withdrawing releases the survivors.** Every fault the stop alone was muting and that is still active is unmuted, and each one that is CONFIRMED publishes a single @@ -90,10 +100,13 @@ announced, while the condition it reports still stands on the machine. A fault still short of confirmation announces nothing (there is nothing yet to announce), and one acknowledged during the stop announces nothing either. -**Correlation rules and the stop compose.** A fault a rule mutes stays muted by -that rule, and withdrawing the stop does not release it. A fault the stop muted -that a rule later claims likewise stays muted after the withdrawal: two -independent reasons to be quiet, and one of them going away is not both. +**Correlation rules and the stop compose.** A rule's mute is an *overlay* on an +owned fault, not a transfer: while the rule holds it, the withdrawal leaves it +alone, and when the rule lets go - its root cause acknowledged, its window closed, +its cluster expired - the fault goes back to being muted by the stop. A fault whose +cycle started before the stop is not owned at all, so the stop neither hides it nor +releases it. This holds for cluster rules as well as hierarchical ones, even though +the cluster path never writes a mute entry of its own. **Both transitions are audited** - when the audit log is on. `audit_log.enabled` is `false` by default, and with it off the switch writes no audit row at all. With @@ -113,10 +126,18 @@ up without reading the audit database. **The declaration survives a restart** when the SQLite backend is in use: it is a row in the fault store, so a stop declared on Friday still mutes on Monday. On -startup the manager also re-marks every stored fault that is not CLEARED and whose -cycle started at or after the declaration, so the switch-off after a reboot still -releases them and announces their confirmations. The in-memory backend has no file -behind it and starts every process with no declaration. +startup the manager reads the ownership flags back, so the switch-off after a +reboot releases exactly the cycles the stop owned and announces their +confirmations. A *rule's* mute is not persisted anywhere, so a fault a correlation +rule was muting before a restart comes back unmuted unless the stop owns it - a +pre-existing property of correlation, not of the switch. The in-memory backend has +no file behind it and starts every process with no declaration and no ownership. + +**A switch-off interrupted by a crash is finished at startup.** The withdrawal +writes the declaration first, announces the confirmations it was holding back, and +drops the ownership flags last. A process that dies in between leaves faults owned +by a declaration that is already over; the next startup recognises exactly that, +announces them once a consumer is listening, and clears the flags. **Over SOVD.** The gateway maps a node's services to operations on its App entity, so an operator reaches the switch without any new route: diff --git a/src/ros2_medkit_fault_manager/design/index.rst b/src/ros2_medkit_fault_manager/design/index.rst index 6814b14a8..59c730f1f 100644 --- a/src/ros2_medkit_fault_manager/design/index.rst +++ b/src/ros2_medkit_fault_manager/design/index.rst @@ -278,26 +278,49 @@ why the mark is gated on the cycle boundary rather than on the report. ``FaultReporter::report()`` is documented to be called the same way. Marking on any report would take a fault that was confirmed and announced *before* the stop off the fault list within a second of the declaration, and announce it a second time at -the switch-off. The boundary is ``report_fault_event``'s own ``is_new``: a new -fault, or one raised again after being cleared. - -**One mute, one owner.** The engine keeps the codes the stop muted in a set of its -own, separate from the mute map, and that set is what makes the two sources -compose. A code leaves it where a rule *registers* its mute (``try_as_symptom``), -not merely where a rule reports ``should_mute``: the auto-cluster path sets that -flag without ever writing the mute map, so treating the flag as a hand-over would -leave the planned stop's entry in the map owned by nobody - muted for good, -released by nothing. In the other direction the stop never overwrites an entry a -rule already holds. A code also leaves the set when the fault is acknowledged, on -both acknowledgement paths: the correlation clear, and the scoped per-entity clear -that deliberately skips it. - -**The mark survives a restart.** The set itself does not - it lives in the process -- so on startup, with a declaration still in force, the manager re-registers it for -every stored fault that is not CLEARED and whose ``first_occurred`` is at or after -the declaration. Without that, a reboot inside a weekend stop would make the +the switch-off. The boundary is ``report_fault_event``'s own ``is_new`` - a new fault, or one +raised again after being cleared - plus a re-fail out of HEALED, because the manager +publishes ``EVENT_CLEARED`` at the heal and the confirmation that follows is +therefore fresh news. ``occurrence_count`` keeps its own, narrower definition. + +**Ownership is the fact; the mute is derived from it.** The stop owns a fault +CYCLE, and that ownership is a flag on the fault row in the store - not a set in the +process, and not a timestamp comparison. Everything else follows: an owned fault is +muted unless a rule's mute overlays it, and the moment the overlay ends the engine +re-asserts the stop's mute (``reassert_planned_stop_mutes``, called from +``process_clear`` and from ``cleanup_expired``). A rule therefore borrows a fault +rather than taking it, which is what makes the two features compose in both +directions: the withdrawal leaves a rule-held fault muted, and a rule that lets go +mid-stop hands the fault back instead of dropping it out of the stop for good. + +Deriving the mute also fixes what ``should_mute`` means. A repeat report of a fault +whose rule has stopped matching produces no correlation result at all, so the report +path used to treat it as unmuted and announce an update for a fault the list was +hiding. ``process_fault`` now answers with the state of the mute map, not with what +this particular report matched, and the PASSED path asks the same question rather +than defaulting to "not muted". + +Ownership ends in exactly three places: the fault is acknowledged (both +acknowledgement paths - the correlation clear and the scoped per-entity clear that +skips it - and the store clears the flag with the CLEARED status), the fault is +auto-cleared with its root cause, or the stop is withdrawn. + +**The mark survives a restart** because the flags do. Startup reads them back and +re-registers the mute. Without that, a reboot inside a weekend stop would make the survivors silently visible and the switch-off would announce none of them: their -confirmations, suppressed before the reboot, would be lost for good. +confirmations, suppressed before the reboot, would be lost for good. A correlation +rule's mute is not persisted anywhere, so a rule-muted fault comes back unmuted +after a restart unless the stop owns its cycle - a pre-existing property of +correlation rather than something the switch introduces. + +**A withdrawal is ordered so a crash cannot swallow an announcement.** The +declaration is written first (so a store that refuses the write changes nothing, and +a restart never re-arms a stop the operator withdrew), the confirmations are +announced next, and the ownership flags are cleared last. A process that dies in +between leaves faults owned by a declaration that is already over, which is a state +the next startup recognises: it announces them - once a subscriber has matched the +events publisher, because that topic is volatile - and clears the flags. Clearing +first would have turned a crash into permanent silence. Withdrawing releases the rest and publishes ``EVENT_CONFIRMED`` once for each released fault that is CONFIRMED. That republication is the point of marking @@ -314,10 +337,13 @@ operations on its App entity. One cost falls on every deployment, correlation or not: the engine is now always constructed, so the ``correlation.cleanup_interval_sec`` timer (5 s by default) -runs on every fault manager. With no rules loaded a tick walks two empty -containers - ``pending_root_causes_`` and ``pending_clusters_`` - and returns. The -alternative, building the engine lazily when a stop is first declared, would put a -second construction path under the service handler for no measurable saving. +runs on every fault manager. With no rules loaded a tick takes the engine's mutex, +walks two empty containers - ``pending_root_causes_`` and ``pending_clusters_`` - +and re-asserts an empty ownership set. Measured at ~190 ns per call (1e6 calls on a +rules-free engine, three runs: 191.2, 194.7, 190.3 ns), which at the default +interval is ~2.3 us of CPU per minute. Building the engine lazily when a stop is +first declared would put a second construction path under the service handler to +save that. Rosbag Black-Box Recording ~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/correlation/correlation_engine.hpp b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/correlation/correlation_engine.hpp index 065575bc3..c31eafd1f 100644 --- a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/correlation/correlation_engine.hpp +++ b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/correlation/correlation_engine.hpp @@ -67,6 +67,11 @@ struct MutedFaultData { std::string root_cause_code; std::string rule_id; uint32_t delay_ms{0}; + /// Whether the planned stop wrote this entry, as opposed to a correlation rule. + /// Not part of the ListFaults wire shape: it is how the engine tells its own + /// entry from a rule's overlay without matching on the rule id, which an + /// operator's configuration could collide with. + bool by_planned_stop{false}; }; /// Information about an active cluster (for ListFaults response) @@ -167,18 +172,21 @@ class CorrelationEngine { /// Whether a planned stop is currently declared. bool planned_stop_active() const; - /// Release one fault from the planned stop's mute, if the stop is what is - /// muting it. A rule's mute is left alone. Called when a fault is acknowledged - /// on a path that does not run the correlation clear (a scoped per-entity - /// DELETE), so an acknowledged fault never stays counted as muted. - void drop_planned_stop_mute(const std::string & fault_code); + /// End the planned stop's ownership of one fault cycle, and take its mute with + /// it. A rule's overlay is left alone. Called when a fault is acknowledged on a + /// path that does not run the correlation clear (a scoped per-entity DELETE), + /// so an acknowledged fault never stays counted as muted. + void release_planned_stop_ownership(const std::string & fault_code); - /// Register a fault as muted by the planned stop without a report driving it. - /// This is how a restart re-marks the faults whose cycles started inside a stop - /// that is still declared: their mute lived in the process that is gone, and - /// without it the switch-off would neither release nor announce them. A fault a - /// rule is already muting is left to that rule. - void restore_planned_stop_mute(const std::string & fault_code); + /// Record that the planned stop owns a fault cycle, without a report driving it. + /// This is how a restart rebuilds the mute from the flags the store kept: the + /// engine's own record lived in the process that is gone, and without it the + /// switch-off would neither release nor announce those faults. + void restore_planned_stop_ownership(const std::string & fault_code); + + /// Every fault cycle the planned stop currently owns, including those a rule's + /// mute is overlaying. + std::vector planned_stop_owned_codes() const; private: /// The correlation half of process_fault: rules, clusters and their muting, @@ -186,6 +194,15 @@ class CorrelationEngine { ProcessFaultResult correlate(const std::string & fault_code, const std::string & severity, std::chrono::steady_clock::time_point timestamp); + /// Put back the planned stop's mute for every fault it owns that has no entry + /// left - a rule's overlay ended (its root cause was acknowledged, its window + /// closed) and the fault is the stop's again. Ownership is the truth; this is + /// what keeps the mute map derived from it. Caller holds mutex_. + void reassert_planned_stop_mutes(); + + /// Write the planned stop's mute entry for one owned fault. Caller holds mutex_. + void mute_as_planned_stop(const std::string & fault_code); + /// Check if fault matches a root cause pattern in any hierarchical rule /// @return Rule ID if matched, empty optional otherwise std::optional try_as_root_cause(const std::string & fault_code); @@ -244,10 +261,12 @@ class CorrelationEngine { /// Whether an operator has declared a planned stop. bool planned_stop_active_{false}; - /// Fault codes muted BY the planned stop and by nothing else. A code leaves - /// this set the moment a rule claims it or it is cleared, which is what makes - /// the switch-off release exactly the faults it is still holding down. - std::set planned_stop_muted_; + /// Fault cycles the planned stop OWNS: they started while it was declared. This + /// is the runtime mirror of the flag the fault store keeps, and it is the truth + /// the mute map is derived from - a rule's mute overlays an owned fault without + /// taking it, and when the overlay ends the fault is the stop's again. A code + /// leaves the set when the fault is acknowledged or the stop is withdrawn. + std::set planned_stop_owned_; mutable std::mutex mutex_; }; diff --git a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_audit_log.hpp b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_audit_log.hpp index 9cee32c23..40c9d9457 100644 --- a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_audit_log.hpp +++ b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_audit_log.hpp @@ -75,9 +75,10 @@ constexpr const char * kTransitionLoggingActivated = "logging_activated"; constexpr const char * kTransitionLoggingDeactivated = "logging_deactivated"; /// Planned-stop lifecycle: an operator declared, or withdrew, a deliberate stop /// of the plant. Like the logging markers these describe the installation rather -/// than one fault, so they carry an empty fault_code and are appended +/// than one fault, so they carry kAuditMarkerFaultCode and are appended /// independently of the per-fault transition filter. The reason travels in -/// `description` and the declarer in `source_id`. +/// `description` and the declarer in `source_id` - that transition's own reason +/// and declarer, so a withdrawal records who ended the stop, not who began it. constexpr const char * kTransitionPlannedStopStarted = "planned_stop_started"; constexpr const char * kTransitionPlannedStopEnded = "planned_stop_ended"; // NOTE: there is deliberately no "ack" kind. The open fault_manager has no diff --git a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_manager_node.hpp b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_manager_node.hpp index caa1f5c23..4924e8222 100644 --- a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_manager_node.hpp +++ b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_manager_node.hpp @@ -21,6 +21,7 @@ #include #include #include +#include #include "rclcpp/rclcpp.hpp" #include "ros2_medkit_fault_manager/capture_thread_pool.hpp" @@ -72,12 +73,6 @@ class FaultManagerNode : public rclcpp::Node { return *storage_; } - /// Replace the storage backend (for testing only). The failure paths of a store - /// that rejects a write are otherwise reachable only on a broken disk. - void set_storage_for_test(std::unique_ptr storage) { - storage_ = std::move(storage); - } - /// Get the tamper-evident audit log (nullptr when disabled), for testing only. const FaultAuditLog * get_audit_log_for_test() const { return audit_log_.get(); @@ -128,6 +123,17 @@ class FaultManagerNode : public rclcpp::Node { /// @return true if entity_id matches any source static bool matches_entity(const std::vector & reporting_sources, const std::string & entity_id); + protected: + /// Construct with a storage backend the caller supplies instead of the one the + /// parameters describe. + /// + /// The backend is handed over BEFORE the node builds anything that borrows it - + /// snapshot and rosbag capture hold a raw `FaultStorage *` - so this cannot leave + /// a dangling pointer the way swapping the store at runtime would. Protected + /// rather than public because the only callers are tests driving the failure + /// paths of a store that rejects a write, which a healthy disk never produces. + FaultManagerNode(const rclcpp::NodeOptions & options, std::unique_ptr storage); + private: /// Create storage backend based on configuration std::unique_ptr create_storage(); @@ -181,11 +187,11 @@ class FaultManagerNode : public rclcpp::Node { /// @param config SnapshotConfig to populate with loaded values void load_snapshot_config_from_yaml(const std::string & config_file, SnapshotConfig & config); - /// Re-register the planned stop's mute for every stored fault whose cycle started - /// inside the declaration currently in force and that is not CLEARED. Runs at - /// startup, before the node serves anything. - /// @return how many faults were re-marked - size_t restore_planned_stop_mutes(); + /// Publish EVENT_CONFIRMED for each of these faults that is CONFIRMED, which is + /// what releasing a fault from the planned stop means to a consumer of the + /// stream. + /// @return how many confirmations were announced + size_t announce_released(const std::vector & fault_codes); /// Build the correlation engine. /// @@ -251,6 +257,10 @@ class FaultManagerNode : public rclcpp::Node { void audit_planned_stop(const char * transition, const PlannedStopState & state, const std::string & reason, const std::string & declared_by, int64_t occurred_at_ns); + /// Handed to create_storage() when a caller supplied a backend. Empty in every + /// normal construction, and moved from exactly once. + std::unique_ptr storage_override_; + std::string storage_type_; std::string database_path_; int32_t confirmation_threshold_{-1}; @@ -289,6 +299,20 @@ class FaultManagerNode : public rclcpp::Node { rclcpp::Service::SharedPtr get_planned_stop_srv_; rclcpp::TimerBase::SharedPtr auto_confirm_timer_; + /// How often startup checks whether anyone is listening before finishing an + /// interrupted planned-stop release. + static constexpr std::chrono::milliseconds kInterruptedReleasePollInterval{200}; + + /// How long that check waits for a listener before announcing anyway. The events + /// topic is volatile, so an announcement made before a subscriber has matched is + /// lost for good - but a manager nobody subscribes to still has to finish the + /// release rather than carry the ownership flags for ever. + static constexpr std::chrono::seconds kInterruptedReleaseDeadline{15}; + + /// One-shot: finishes a release a previous process did not complete (nullptr + /// when there was nothing left over). + rclcpp::TimerBase::SharedPtr interrupted_release_timer_; + /// Timer for periodic cleanup of expired correlation data rclcpp::TimerBase::SharedPtr correlation_cleanup_timer_; diff --git a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_storage.hpp b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_storage.hpp index 63b0eafe4..6894c4b89 100644 --- a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_storage.hpp +++ b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_storage.hpp @@ -98,6 +98,12 @@ struct FaultState { std::string status; std::set reporting_sources; + /// Whether the planned stop owns this fault cycle: it started while a stop was + /// declared. Persisted, because the declaration outlives the process and the + /// switch-off has to know which faults it is releasing. Cleared when the fault + /// is acknowledged and when the stop is withdrawn. + bool planned_stop_owned{false}; + // Debounce state (internal, not exposed in Fault.msg) int32_t debounce_counter{0}; ///< FAILED decrements (-1), PASSED increments (+1) rclcpp::Time last_failed_time{}; ///< Timestamp of last FAILED event @@ -478,6 +484,22 @@ class FaultStorage { /// given one answers with a default-constructed (inactive) state. virtual PlannedStopState get_planned_stop() const = 0; + /// Record, or withdraw, the planned stop's ownership of one fault cycle. + /// A code with no stored fault is ignored. + virtual void set_planned_stop_owned(const std::string & fault_code, bool owned) = 0; + + /// Every fault the planned stop currently owns. This is what a restart reads to + /// rebuild the mute, and what a switch-off releases - not a time comparison + /// against the declaration, which cannot survive a clock step and cannot tell a + /// rule's mute from the stop's. + virtual std::vector get_planned_stop_owned() const = 0; + + /// Drop every ownership flag. Called once the switch-off has announced what it + /// released, so a crash before this point leaves the flags for the next startup + /// to finish. + /// @return how many faults were released + virtual size_t clear_planned_stop_owned() = 0; + protected: FaultStorage() = default; FaultStorage(const FaultStorage &) = default; @@ -546,6 +568,9 @@ class InMemoryFaultStorage : public FaultStorage { void set_planned_stop(const PlannedStopState & state) override; PlannedStopState get_planned_stop() const override; + void set_planned_stop_owned(const std::string & fault_code, bool owned) override; + std::vector get_planned_stop_owned() const override; + size_t clear_planned_stop_owned() override; private: /// Update fault status based on debounce counter and given config diff --git a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/sqlite_fault_storage.hpp b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/sqlite_fault_storage.hpp index 377e72fbf..c2f48a8e9 100644 --- a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/sqlite_fault_storage.hpp +++ b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/sqlite_fault_storage.hpp @@ -97,6 +97,9 @@ class SqliteFaultStorage : public FaultStorage { void set_planned_stop(const PlannedStopState & state) override; PlannedStopState get_planned_stop() const override; + void set_planned_stop_owned(const std::string & fault_code, bool owned) override; + std::vector get_planned_stop_owned() const override; + size_t clear_planned_stop_owned() override; /// Get the database path const std::string & db_path() const { diff --git a/src/ros2_medkit_fault_manager/src/correlation/correlation_engine.cpp b/src/ros2_medkit_fault_manager/src/correlation/correlation_engine.cpp index a3c0f5af3..26ba47b33 100644 --- a/src/ros2_medkit_fault_manager/src/correlation/correlation_engine.cpp +++ b/src/ros2_medkit_fault_manager/src/correlation/correlation_engine.cpp @@ -29,43 +29,25 @@ ProcessFaultResult CorrelationEngine::process_fault(const std::string & fault_co bool cycle_started) { std::lock_guard lock(mutex_); - // Correlation runs on every report. Handing a mute over to a rule happens where - // the rule registers it (try_as_symptom), not here: the auto-cluster path - // reports should_mute without ever writing muted_faults_, so treating - // should_mute as "a rule owns this now" would orphan the planned stop's entry - - // muted for good, released by nothing. + // Correlation runs on every report and may write its own mute entry for this + // code. That entry OVERLAYS the planned stop rather than replacing it: ownership + // is a fact about which cycle the fault is in, not about who is currently + // holding it quiet. ProcessFaultResult result = correlate(fault_code, severity, timestamp); - if (!planned_stop_active_) { - return result; + if (planned_stop_active_ && cycle_started) { + planned_stop_owned_.insert(fault_code); } - if (planned_stop_muted_.count(fault_code) > 0) { - // Already the stop's. A repeat report neither re-registers it nor lets it go. - result.should_mute = true; - return result; + if (planned_stop_owned_.count(fault_code) > 0 && muted_faults_.count(fault_code) == 0) { + mute_as_planned_stop(fault_code); } - if (!cycle_started) { - // The condition was already up when the stop was declared. Its confirmation - // has been announced; hiding it now would take a standing alarm off the list - // and announce it a second time at the switch-off. - return result; - } - - if (muted_faults_.count(fault_code) > 0) { - // A rule is muting this code. One mute, one owner: the stop does not take it - // over and, at the switch-off, does not release it. - return result; - } - - MutedFaultData muted; - muted.fault_code = fault_code; - muted.root_cause_code = kPlannedStopRootCause; - muted.rule_id = kPlannedStopRuleId; - muted_faults_[fault_code] = muted; - planned_stop_muted_.insert(fault_code); - result.should_mute = true; + // Derived, not remembered: a fault stays reported as muted for as long as an + // entry exists, whatever this particular report matched. Without this a repeat + // report of a fault whose rule stopped matching announces an update for a fault + // the list is hiding. + result.should_mute = result.should_mute || muted_faults_.count(fault_code) > 0; return result; } @@ -162,10 +144,11 @@ ProcessClearResult CorrelationEngine::process_clear(const std::string & fault_co } } - // Clean up muted faults + // The rule's overlay on each symptom goes with the root cause. A symptom the + // stop owns is re-muted by reassert_planned_stop_mutes() below; one that is + // auto-cleared has its cycle ended, so its ownership goes too. for (const auto & symptom_code : it->second) { muted_faults_.erase(symptom_code); - planned_stop_muted_.erase(symptom_code); } // Remove from root_to_symptoms @@ -276,7 +259,14 @@ ProcessClearResult CorrelationEngine::process_clear(const std::string & fault_co muted_faults_.erase(fault_code); // A cleared fault has nothing left to announce, so the planned stop must not // hand it back at switch-off. - planned_stop_muted_.erase(fault_code); + planned_stop_owned_.erase(fault_code); + for (const auto & auto_cleared : result.auto_cleared_codes) { + muted_faults_.erase(auto_cleared); + planned_stop_owned_.erase(auto_cleared); + } + + // A symptom that was NOT auto-cleared is still up, and the stop may own it. + reassert_planned_stop_mutes(); return result; } @@ -291,16 +281,20 @@ std::vector CorrelationEngine::end_planned_stop() { planned_stop_active_ = false; - std::vector unmuted; - unmuted.reserve(planned_stop_muted_.size()); - for (const auto & fault_code : planned_stop_muted_) { - if (muted_faults_.erase(fault_code) > 0) { - unmuted.push_back(fault_code); + std::vector released; + released.reserve(planned_stop_owned_.size()); + for (const auto & fault_code : planned_stop_owned_) { + auto it = muted_faults_.find(fault_code); + // A rule's overlay is not the stop's to lift: that fault stays muted, and + // stays unannounced, for as long as the rule holds it. + if (it != muted_faults_.end() && it->second.by_planned_stop) { + muted_faults_.erase(it); + released.push_back(fault_code); } } - planned_stop_muted_.clear(); + planned_stop_owned_.clear(); - return unmuted; + return released; } bool CorrelationEngine::planned_stop_active() const { @@ -308,27 +302,47 @@ bool CorrelationEngine::planned_stop_active() const { return planned_stop_active_; } -void CorrelationEngine::drop_planned_stop_mute(const std::string & fault_code) { +void CorrelationEngine::release_planned_stop_ownership(const std::string & fault_code) { std::lock_guard lock(mutex_); - if (planned_stop_muted_.erase(fault_code) > 0) { - muted_faults_.erase(fault_code); + if (planned_stop_owned_.erase(fault_code) == 0) { + return; + } + auto it = muted_faults_.find(fault_code); + if (it != muted_faults_.end() && it->second.by_planned_stop) { + muted_faults_.erase(it); } } -void CorrelationEngine::restore_planned_stop_mute(const std::string & fault_code) { +void CorrelationEngine::restore_planned_stop_ownership(const std::string & fault_code) { std::lock_guard lock(mutex_); - if (muted_faults_.count(fault_code) > 0) { - return; // a rule is already muting it; one mute, one owner + planned_stop_owned_.insert(fault_code); + if (muted_faults_.count(fault_code) == 0) { + mute_as_planned_stop(fault_code); } +} + +std::vector CorrelationEngine::planned_stop_owned_codes() const { + std::lock_guard lock(mutex_); + return {planned_stop_owned_.begin(), planned_stop_owned_.end()}; +} +void CorrelationEngine::mute_as_planned_stop(const std::string & fault_code) { MutedFaultData muted; muted.fault_code = fault_code; muted.root_cause_code = kPlannedStopRootCause; muted.rule_id = kPlannedStopRuleId; + muted.by_planned_stop = true; muted_faults_[fault_code] = muted; - planned_stop_muted_.insert(fault_code); +} + +void CorrelationEngine::reassert_planned_stop_mutes() { + for (const auto & fault_code : planned_stop_owned_) { + if (muted_faults_.count(fault_code) == 0) { + mute_as_planned_stop(fault_code); + } + } } std::vector CorrelationEngine::get_muted_faults() const { @@ -413,8 +427,10 @@ void CorrelationEngine::cleanup_expired() { pending_clusters_.erase(it); } } -} + // Whatever a rule just stopped holding, the stop still owns. + reassert_planned_stop_mutes(); +} std::optional CorrelationEngine::try_as_root_cause(const std::string & fault_code) { for (const auto & rule : config_.rules) { if (rule.mode != CorrelationMode::HIERARCHICAL) { @@ -481,10 +497,10 @@ std::optional CorrelationEngine::try_as_symptom(const std::s muted.root_cause_code = prc.fault_code; muted.rule_id = rule.id; muted.delay_ms = result.delay_ms; + // Overlays whatever was there, the planned stop's entry included. The + // stop keeps its ownership, so when this rule lets go the fault is muted + // by the stop again rather than falling out of it. muted_faults_[fault_code] = muted; - // The rule is the owner from here. Ending a planned stop must not release - // a fault a root cause is still holding down. - planned_stop_muted_.erase(fault_code); } return result; diff --git a/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp b/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp index b8df8c13d..8fc43ae0a 100644 --- a/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp +++ b/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp @@ -115,7 +115,11 @@ std::string validate_recording_id(const std::string & recording_id) { } // namespace -FaultManagerNode::FaultManagerNode(const rclcpp::NodeOptions & options) : Node("fault_manager", options) { +FaultManagerNode::FaultManagerNode(const rclcpp::NodeOptions & options) : FaultManagerNode(options, nullptr) { +} + +FaultManagerNode::FaultManagerNode(const rclcpp::NodeOptions & options, std::unique_ptr storage) + : Node("fault_manager", options), storage_override_(std::move(storage)) { // Declare and get parameters storage_type_ = declare_parameter("storage_type", "sqlite"); database_path_ = declare_parameter("database_path", "/var/lib/ros2_medkit/faults.db"); @@ -458,12 +462,44 @@ FaultManagerNode::FaultManagerNode(const rclcpp::NodeOptions & options) : Node(" // muting is per fault cycle and those cycles are over as far as this process is // concerned - but every fault reported from here is muted again. planned_stop_ = storage_->get_planned_stop(); + const auto owned_at_startup = storage_->get_planned_stop_owned(); if (planned_stop_.active) { correlation_engine_->begin_planned_stop(); - const size_t restored = restore_planned_stop_mutes(); + for (const auto & fault_code : owned_at_startup) { + correlation_engine_->restore_planned_stop_ownership(fault_code); + } RCLCPP_INFO(get_logger(), "Planned stop is in force (reason='%s', declared_by='%s'): %zu fault(s) re-marked, new faults marked", - planned_stop_.reason.c_str(), planned_stop_.declared_by.c_str(), restored); + planned_stop_.reason.c_str(), planned_stop_.declared_by.c_str(), owned_at_startup.size()); + } else if (!owned_at_startup.empty()) { + // Faults owned by a declaration that is already withdrawn: the previous + // process died between writing the withdrawal and finishing the release, so + // these confirmations were suppressed and never announced. Finish the job. + // + // Not here, though: this publisher was created moments ago in this same + // constructor, and the events topic is volatile, so anything published before + // a subscriber has matched it reaches nobody and is gone for good. Wait for a + // listener - and give up waiting after kInterruptedReleaseDeadline, because a + // manager nobody is listening to must still finish the release rather than + // carry the flags forever. + RCLCPP_WARN(get_logger(), + "%zu fault(s) are still owned by a planned stop that was already withdrawn - a previous run did not " + "finish releasing them. Announcing them once a consumer is listening.", + owned_at_startup.size()); + const auto give_up_at = std::chrono::steady_clock::now() + kInterruptedReleaseDeadline; + interrupted_release_timer_ = + create_wall_timer(kInterruptedReleasePollInterval, [this, owned_at_startup, give_up_at]() { + const bool listener = event_publisher_->get_subscription_count() > 0; + if (!listener && std::chrono::steady_clock::now() < give_up_at) { + return; + } + interrupted_release_timer_->cancel(); + const size_t announced = announce_released(owned_at_startup); + storage_->clear_planned_stop_owned(); + RCLCPP_INFO(get_logger(), + "Finished an interrupted planned-stop release: announced %zu confirmation(s) to %zu listener(s)", + announced, event_publisher_->get_subscription_count()); + }); } // Create auto-confirmation timer if enabled @@ -544,6 +580,11 @@ FaultManagerNode::~FaultManagerNode() { } std::unique_ptr FaultManagerNode::create_storage() { + if (storage_override_) { + RCLCPP_INFO(get_logger(), "Using a caller-supplied fault storage backend"); + return std::move(storage_override_); + } + if (storage_type_ == "memory") { RCLCPP_INFO(get_logger(), "Using in-memory fault storage"); return std::make_unique(); @@ -872,17 +913,28 @@ void FaultManagerNode::handle_report_fault( // Process through correlation engine (if enabled) // Only process FAILED events with correlation bool should_mute = false; - if (correlation_engine_ && request->event_type == ros2_medkit_msgs::srv::ReportFault::Request::EVENT_FAILED) { - // is_new is the cycle boundary the planned stop needs: a new fault, or one - // raised again after being cleared. Reporters are level-triggered, so - // without it a condition that was already up - and already announced - - // would be swallowed by a stop declared after it started. + if (request->event_type == ros2_medkit_msgs::srv::ReportFault::Request::EVENT_FAILED) { + // The cycle boundary the planned stop needs. `is_new` covers a new fault and + // one raised again after being cleared; a re-fail out of HEALED is the third + // case, because the manager publishes EVENT_CLEARED at the heal, so the + // confirmation that follows is a fresh announcement. Reporters are + // level-triggered, so without this boundary a condition that was already up + // - and already announced - would be swallowed by a stop declared after it + // started. occurrence_count keeps its own, narrower definition. + const bool cycle_started = is_new || status_before == ros2_medkit_msgs::msg::Fault::STATUS_HEALED; + auto correlation_result = correlation_engine_->process_fault(request->fault_code, correlation::severity_to_string(request->severity), - std::chrono::steady_clock::now(), is_new); + std::chrono::steady_clock::now(), cycle_started); should_mute = correlation_result.should_mute; + // Ownership is persisted, not inferred: the switch-off, and a restart, both + // read it back from the store rather than comparing timestamps. + if (planned_stop_.active && cycle_started) { + storage_->set_planned_stop_owned(request->fault_code, true); + } + if (correlation_result.is_root_cause) { RCLCPP_DEBUG(get_logger(), "Fault %s identified as root cause (rule=%s)", request->fault_code.c_str(), correlation_result.rule_id.c_str()); @@ -899,6 +951,11 @@ void FaultManagerNode::handle_report_fault( correlation_result.cluster_id.c_str(), correlation_result.retroactive_mute_codes.size()); } } + } else { + // A PASSED report can still produce an event - the fault stays CONFIRMED and + // its data is updated - and a muted fault must not announce that either. The + // mute is a property of the fault, not of the report that happened to arrive. + should_mute = correlation_engine_->is_muted(request->fault_code); } // Determine event type based on status transition @@ -1072,7 +1129,7 @@ void FaultManagerNode::handle_clear_fault( // walked, because it reaches across entity boundaries. The planned stop is not // a graph - it is this one fault's mute - so it is released either way, or an // acknowledged fault would stay counted as muted until the stop ended. - correlation_engine_->drop_planned_stop_mute(request->fault_code); + correlation_engine_->release_planned_stop_ownership(request->fault_code); } bool cleared = storage_->clear_fault(request->fault_code); @@ -1196,14 +1253,13 @@ void FaultManagerNode::handle_set_planned_stop( // now, once, or the alarm standing on the controller is invisible to every // consumer of the stream. PREFAILED has nothing to announce yet, and a HEALED // one already published its end. - size_t announced = 0; - for (const auto & fault_code : unmuted) { - auto fault = storage_->get_fault(fault_code); - if (fault && fault->status == ros2_medkit_msgs::msg::Fault::STATUS_CONFIRMED) { - publish_fault_event(ros2_medkit_msgs::msg::FaultEvent::EVENT_CONFIRMED, *fault); - ++announced; - } - } + const size_t announced = announce_released(unmuted); + + // The flags go LAST. A crash between the declaration write and this point + // leaves owned faults behind an ENDED declaration, which the next startup + // recognises and finishes - announcing what this loop did not get to. Clearing + // first would turn a crash into permanent silence for those faults. + storage_->clear_planned_stop_owned(); // The withdrawal carries its own reason and declarer into the audit record; the // stored declaration keeps the ones that started the stop. @@ -1516,25 +1572,20 @@ void FaultManagerNode::load_snapshot_config_from_yaml(const std::string & config } } -size_t FaultManagerNode::restore_planned_stop_mutes() { - // The mute set lives in the engine, so a restart loses it while the declaration - // itself survives in the store. Without this, a reboot inside a weekend stop - // makes every fault the stop was holding silently visible, and the switch-off - // then announces none of them - the confirmations they never published are lost - // for good. Rebuild it from what the store knows: a fault whose cycle started at - // or after the declaration and that is not over. - size_t restored = 0; - for (const auto & fault : storage_->get_all_faults()) { - if (fault.status == ros2_medkit_msgs::msg::Fault::STATUS_CLEARED) { - continue; // acknowledged: nothing left to hold down or announce - } - if (rclcpp::Time(fault.first_occurred).nanoseconds() < planned_stop_.since_ns) { - continue; // its cycle started before the stop, so the stop never marked it +size_t FaultManagerNode::announce_released(const std::vector & fault_codes) { + // A released fault that is CONFIRMED was never announced - the confirmation + // happened behind the mute - so it is announced now, once, or the alarm standing + // on the controller is invisible to every consumer of the stream. PREFAILED has + // nothing to announce yet, and a HEALED one already published its end. + size_t announced = 0; + for (const auto & fault_code : fault_codes) { + auto fault = storage_->get_fault(fault_code); + if (fault && fault->status == ros2_medkit_msgs::msg::Fault::STATUS_CONFIRMED) { + publish_fault_event(ros2_medkit_msgs::msg::FaultEvent::EVENT_CONFIRMED, *fault); + ++announced; } - correlation_engine_->restore_planned_stop_mute(fault.fault_code); - ++restored; } - return restored; + return announced; } std::unique_ptr FaultManagerNode::create_correlation_engine() { diff --git a/src/ros2_medkit_fault_manager/src/fault_storage.cpp b/src/ros2_medkit_fault_manager/src/fault_storage.cpp index 2611f042b..6487d8373 100644 --- a/src/ros2_medkit_fault_manager/src/fault_storage.cpp +++ b/src/ros2_medkit_fault_manager/src/fault_storage.cpp @@ -333,6 +333,9 @@ bool InMemoryFaultStorage::clear_fault(const std::string & fault_code) { } it->second.status = ros2_medkit_msgs::msg::Fault::STATUS_CLEARED; + // An acknowledged cycle is over, so the planned stop no longer owns it: it has + // nothing left to release or to announce for this fault. + it->second.planned_stop_owned = false; return true; } @@ -856,6 +859,7 @@ std::vector InMemoryFaultStorage::reclassify_healed_as_cleared() { for (auto & [code, state] : faults_) { if (state.status == ros2_medkit_msgs::msg::Fault::STATUS_HEALED) { state.status = ros2_medkit_msgs::msg::Fault::STATUS_CLEARED; + state.planned_stop_owned = false; // CLEARED ends the cycle the stop owned reclassified.push_back(code); } } @@ -875,6 +879,37 @@ std::vector InMemoryFaultStorage::reclassify_healed_as_cleared() { return reclassified; } +void InMemoryFaultStorage::set_planned_stop_owned(const std::string & fault_code, bool owned) { + std::lock_guard lock(mutex_); + auto it = faults_.find(fault_code); + if (it != faults_.end()) { + it->second.planned_stop_owned = owned; + } +} + +std::vector InMemoryFaultStorage::get_planned_stop_owned() const { + std::lock_guard lock(mutex_); + std::vector owned; + for (const auto & [code, state] : faults_) { + if (state.planned_stop_owned) { + owned.push_back(code); + } + } + return owned; +} + +size_t InMemoryFaultStorage::clear_planned_stop_owned() { + std::lock_guard lock(mutex_); + size_t cleared = 0; + for (auto & [code, state] : faults_) { + if (state.planned_stop_owned) { + state.planned_stop_owned = false; + ++cleared; + } + } + return cleared; +} + void InMemoryFaultStorage::set_planned_stop(const PlannedStopState & state) { std::lock_guard lock(mutex_); planned_stop_ = state; diff --git a/src/ros2_medkit_fault_manager/src/sqlite_fault_storage.cpp b/src/ros2_medkit_fault_manager/src/sqlite_fault_storage.cpp index 502d745b6..46293a053 100644 --- a/src/ros2_medkit_fault_manager/src/sqlite_fault_storage.cpp +++ b/src/ros2_medkit_fault_manager/src/sqlite_fault_storage.cpp @@ -164,7 +164,8 @@ void SqliteFaultStorage::initialize_schema() { debounce_counter INTEGER NOT NULL DEFAULT 0, last_failed_ns INTEGER NOT NULL DEFAULT 0, last_passed_ns INTEGER NOT NULL DEFAULT 0, - confirmed_at_ns INTEGER NOT NULL DEFAULT 0 + confirmed_at_ns INTEGER NOT NULL DEFAULT 0, + planned_stop_owned INTEGER NOT NULL DEFAULT 0 ); )"; @@ -197,6 +198,29 @@ void SqliteFaultStorage::initialize_schema() { } } + // Migration: the planned stop records which fault CYCLES it owns, so the flag + // lives with the fault rather than being inferred from timestamps. Rows written + // before it arrive unowned, which is what a database that predates the switch + // means. + { + bool has_owned = false; + SqliteStatement info(db_, "PRAGMA table_info(faults)"); + while (info.step() == SQLITE_ROW) { + if (info.column_text(1) == "planned_stop_owned") { + has_owned = true; + break; + } + } + if (!has_owned) { + if (sqlite3_exec(db_, "ALTER TABLE faults ADD COLUMN planned_stop_owned INTEGER NOT NULL DEFAULT 0", nullptr, + nullptr, &err_msg) != SQLITE_OK) { + std::string error = err_msg ? err_msg : "Unknown error"; + sqlite3_free(err_msg); + throw std::runtime_error("Failed to add planned_stop_owned column: " + error); + } + } + } + // Migration: releases that advanced last_occurred_ns on PASSED events left // inflated rows behind, and a latched CONFIRMED fault that only ever heals // would keep the wrong timestamp forever. last_failed_ns holds the true @@ -1048,7 +1072,9 @@ bool SqliteFaultStorage::clear_fault(const std::string & fault_code) { } } - SqliteStatement stmt(db_, "UPDATE faults SET status = ? WHERE fault_code = ?"); + // An acknowledged cycle is over, so the planned stop no longer owns it: it has + // nothing left to release or to announce for this fault. + SqliteStatement stmt(db_, "UPDATE faults SET status = ?, planned_stop_owned = 0 WHERE fault_code = ?"); stmt.bind_text(1, ros2_medkit_msgs::msg::Fault::STATUS_CLEARED); stmt.bind_text(2, fault_code); @@ -1091,7 +1117,7 @@ std::vector SqliteFaultStorage::reclassify_healed_as_cleared() { } } - SqliteStatement stmt(db_, "UPDATE faults SET status = ? WHERE status = ?"); + SqliteStatement stmt(db_, "UPDATE faults SET status = ?, planned_stop_owned = 0 WHERE status = ?"); stmt.bind_text(1, ros2_medkit_msgs::msg::Fault::STATUS_CLEARED); stmt.bind_text(2, ros2_medkit_msgs::msg::Fault::STATUS_HEALED); if (stmt.step() != SQLITE_DONE) { @@ -1362,6 +1388,40 @@ PlannedStopState SqliteFaultStorage::get_planned_stop() const { return state; } +void SqliteFaultStorage::set_planned_stop_owned(const std::string & fault_code, bool owned) { + std::lock_guard lock(mutex_); + + SqliteStatement stmt(db_, "UPDATE faults SET planned_stop_owned = ? WHERE fault_code = ?"); + stmt.bind_int(1, owned ? 1 : 0); + stmt.bind_text(2, fault_code); + + if (stmt.step() != SQLITE_DONE) { + throw std::runtime_error(std::string("Failed to record planned-stop ownership: ") + sqlite3_errmsg(db_)); + } +} + +std::vector SqliteFaultStorage::get_planned_stop_owned() const { + std::lock_guard lock(mutex_); + + SqliteStatement stmt(db_, "SELECT fault_code FROM faults WHERE planned_stop_owned != 0"); + + std::vector owned; + while (stmt.step() == SQLITE_ROW) { + owned.push_back(stmt.column_text(0)); + } + return owned; +} + +size_t SqliteFaultStorage::clear_planned_stop_owned() { + std::lock_guard lock(mutex_); + + SqliteStatement stmt(db_, "UPDATE faults SET planned_stop_owned = 0 WHERE planned_stop_owned != 0"); + if (stmt.step() != SQLITE_DONE) { + throw std::runtime_error(std::string("Failed to release planned-stop ownership: ") + sqlite3_errmsg(db_)); + } + return static_cast(sqlite3_changes(db_)); +} + std::optional SqliteFaultStorage::get_freeze_frame(const std::string & fault_code) const { std::lock_guard lock(mutex_); diff --git a/src/ros2_medkit_fault_manager/test/integration/test_planned_stop.test.py b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop.test.py index fad5ae5f2..4d42b1ab6 100644 --- a/src/ros2_medkit_fault_manager/test/integration/test_planned_stop.test.py +++ b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop.test.py @@ -76,17 +76,29 @@ codes: ["MOTOR_PS_*"] ps_valve_errors: codes: ["VALVE_PS_*"] + ps_pump_errors: + codes: ["PUMP_PS_*"] rules: - id: ps_estop_cascade name: "E-Stop Cascade" mode: hierarchical root_cause: - codes: ["ESTOP_PS_001"] + codes: ["ESTOP_PS_*"] symptoms: - pattern: ps_motor_errors window_ms: 60000 mute_symptoms: true auto_clear_with_root: true + - id: ps_estop_noclear + name: "E-Stop Cascade, symptoms left up" + mode: hierarchical + root_cause: + codes: ["ESTOP_NOCLEAR_001"] + symptoms: + - pattern: ps_pump_errors + window_ms: 60000 + mute_symptoms: true + auto_clear_with_root: false - id: ps_valve_storm name: "Valve Storm" mode: auto_cluster @@ -136,6 +148,10 @@ def generate_test_description(): 'storage_type': 'sqlite', 'database_path': DATABASE_PATH, 'confirmation_threshold': CONFIRMATION_THRESHOLD, + # A single PASSED heals, which is what makes the HEALED -> re-fail + # cycle boundary reachable from a test. + 'healing_enabled': True, + 'healing_threshold': 0, 'audit_log.enabled': True, 'audit_log.database_path': AUDIT_PATH, 'correlation.config_file': CORRELATION_PATH, @@ -731,6 +747,104 @@ def test_the_last_declaration_is_readable_after_the_withdrawal(self): self.assertEqual('night_shift', ended[3]) self.assertEqual('plant back up', ended[4]) + def test_a_healed_fault_that_fails_again_inside_the_stop_is_owned(self): + """A heal ends the cycle, so the next failure starts one the stop owns.""" + code = 'PS_HEALED_REFAIL' + + self._confirm(code) + self._wait_until(lambda: self._count_events(code, FaultEvent.EVENT_CONFIRMED) == 1, + f'{code} was never announced before the stop') + # The debounce counter sits at the confirmation threshold, so it takes that + # many PASSED reports to climb back to the healing threshold of 0. + for _ in range(abs(CONFIRMATION_THRESHOLD)): + self._report(code, event_type=ReportFault.Request.EVENT_PASSED) + self._wait_until(lambda: self._get_fault(code).fault.status == 'HEALED', + f'{code} never healed') + + self._set_stop(True, reason='bench work', declared_by='tech') + + # The condition returns. That is a new cycle by the manager's own model, + # and the stop owns it. + self._confirm(code) + muted = self._wait_until(lambda: self._muted_entry(code), + f'the re-failed {code} was not marked by the stop') + self.assertEqual('planned_stop', muted.rule_id) + self.assertNotIn(code, self._codes_in_default_list()) + self.assertEqual(1, self._count_events(code, FaultEvent.EVENT_CONFIRMED), + 'the re-failure was announced despite the stop') + + self._set_stop(False, reason='bench work over', declared_by='tech') + self._wait_until(lambda: self._count_events(code, FaultEvent.EVENT_CONFIRMED) == 2, + 'the confirmation the stop held back was never announced') + + def test_a_passed_report_announces_nothing_for_a_marked_fault(self): + """Muting gates the PASSED path too, not only the FAILED one.""" + code = 'PS_PASSED_QUIET' + + self._set_stop(True, reason='changeover', declared_by='maintenance') + self._confirm(code) + self._wait_until(lambda: self._muted_entry(code), f'{code} was not marked by the stop') + + # A PASSED that does not heal leaves the fault CONFIRMED with fresh data - + # the update path. Nothing about it may reach the stream. + self._report(code, event_type=ReportFault.Request.EVENT_PASSED) + self._wait_until(lambda: self._get_fault(code).success, f'{code} left the store') + self.assertEqual(set(), self._event_types(code), + 'a PASSED report announced an update for a marked fault') + + self._set_stop(False, reason='done', declared_by='maintenance') + + def test_a_rule_muted_symptom_announces_nothing_on_a_passed_report(self): + """ + Apply the same gate to a fault a correlation rule is muting. + + Its own root cause code: a root shared with another case would already be + CONFIRMED there, and a cycle that started outside this test's stop is not + the stop's to own. + """ + root = 'ESTOP_PS_PASSED' + symptom = 'MOTOR_PS_PASSED' + + self._confirm(root) + self._confirm(symptom) + muted = self._wait_until(lambda: self._muted_entry(symptom), + f'{symptom} was not muted by the rule') + self.assertEqual('ps_estop_cascade', muted.rule_id) + + self._report(symptom, event_type=ReportFault.Request.EVENT_PASSED) + self._wait_until(lambda: self._get_fault(symptom).success, f'{symptom} left the store') + self.assertEqual(set(), self._event_types(symptom), + 'a PASSED report announced an update for a rule-muted fault') + + def test_a_symptom_returns_to_the_stop_when_its_root_cause_is_acknowledged(self): + """A rule's mute is an overlay; ownership outlives it.""" + root = 'ESTOP_NOCLEAR_001' + symptom = 'PUMP_PS_SYMPTOM' + + self._set_stop(True, reason='cell 7 changeover', declared_by='maintenance') + self._confirm(root) + self._confirm(symptom) + + muted = self._wait_until(lambda: self._muted_entry(symptom), + f'{symptom} was not muted at all') + self.assertEqual('ps_estop_noclear', muted.rule_id, + 'the rule, not the stop, holds the mute while the root cause stands') + + # Acknowledging the root cause ends the rule's mute. This rule does not + # auto-clear its symptoms, so the symptom is still up - and still the + # stop's. + self.assertTrue(self._clear(root).success) + + still_muted = self._wait_until(lambda: self._muted_entry(symptom), + 'the symptom fell out of the stop when the rule let go') + self.assertEqual('planned_stop', still_muted.rule_id) + self.assertNotIn(symptom, self._codes_in_default_list()) + self.assertEqual(set(), self._event_types(symptom)) + + self._set_stop(False, reason='done', declared_by='maintenance') + self._wait_until(lambda: self._count_events(symptom, FaultEvent.EVENT_CONFIRMED) == 1, + 'the symptom the stop had owned all along was never announced') + def test_the_switch_does_not_release_a_rule_muted_symptom(self): root = 'ESTOP_PS_001' symptom = 'MOTOR_PS_LEFT' diff --git a/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_interrupted_release.test.py b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_interrupted_release.test.py new file mode 100644 index 000000000..14484ae78 --- /dev/null +++ b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_interrupted_release.test.py @@ -0,0 +1,315 @@ +#!/usr/bin/env python3 +# Copyright 2026 bburda +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +A switch-off killed halfway is finished by the next startup. + +Withdrawing a planned stop writes the declaration, announces the confirmations it +was holding back, and only then drops the ownership flags. A process that dies +between the write and the flags leaves faults owned by a declaration that is +already over - and their confirmations, suppressed while the stop stood, have +been announced by nobody. + +The kill is real (SIGTERM, launch respawns the node); the crash *point* is +simulated by editing the store while the manager is down, which is exactly the +state such a crash leaves behind. +""" + +import os +import signal +import sqlite3 +import tempfile +import threading +import time +import unittest + +from launch import LaunchDescription +import launch.actions +import launch_ros.actions +import launch_testing.actions +import rclpy +from rclpy.node import Node +from rclpy.qos import HistoryPolicy, QoSProfile, ReliabilityPolicy +from ros2_medkit_msgs.msg import Fault, FaultEvent +from ros2_medkit_msgs.srv import GetPlannedStop, ListFaults, ReportFault, SetPlannedStop + +STORAGE_DIR = tempfile.mkdtemp(prefix='planned_stop_interrupted_') +DATABASE_PATH = os.path.join(STORAGE_DIR, 'faults.db') + +OWNED_CODES = ('PS_INTERRUPTED_ONE', 'PS_INTERRUPTED_TWO') + +# Long enough that the test can edit the store between the kill and the +# replacement opening it. +RESPAWN_DELAY_SEC = 6.0 + + +def get_coverage_env(): + """Get environment variables for gcov coverage data collection.""" + try: + from ament_index_python.packages import get_package_prefix + pkg_prefix = get_package_prefix('ros2_medkit_fault_manager') + workspace = os.path.dirname(os.path.dirname(pkg_prefix)) + build_dir = os.path.join(workspace, 'build', 'ros2_medkit_fault_manager') + + if os.path.exists(build_dir): + return { + 'GCOV_PREFIX': build_dir, + 'GCOV_PREFIX_STRIP': str(build_dir.count(os.sep)), + } + except Exception: + # Coverage environment is optional; on any error, fall back to no extra config + pass + return {} + + +def generate_test_description(): + """Launch a fault manager launch brings back after it is killed.""" + fault_manager_env = get_coverage_env() + fault_manager_env['ROS_LOCALHOST_ONLY'] = '1' + + fault_manager_node = launch_ros.actions.Node( + package='ros2_medkit_fault_manager', + executable='fault_manager_node', + name='fault_manager', + output='screen', + additional_env=fault_manager_env, + parameters=[{ + 'storage_type': 'sqlite', + 'database_path': DATABASE_PATH, + 'confirmation_threshold': -1, + 'snapshots.enabled': False, + 'snapshots.rosbag.enabled': False, + }], + respawn=True, + respawn_delay=RESPAWN_DELAY_SEC, + sigterm_timeout='30', + sigkill_timeout='15', + ) + + return ( + LaunchDescription([ + launch.actions.TimerAction(period=2.0, actions=[fault_manager_node]), + launch_testing.actions.ReadyToTest(), + ]), + {'fault_manager_node': fault_manager_node}, + ) + + +class TestInterruptedRelease(unittest.TestCase): + """A release the previous process did not finish is finished at startup.""" + + @classmethod + def setUpClass(cls): + os.environ['ROS_LOCALHOST_ONLY'] = '1' + rclpy.init() + cls.node = Node('test_planned_stop_interrupted_client') + + cls._events = [] + cls._events_lock = threading.Lock() + cls._event_sub = cls.node.create_subscription( + FaultEvent, '/fault_manager/events', cls._on_event, + QoSProfile(reliability=ReliabilityPolicy.RELIABLE, + history=HistoryPolicy.KEEP_LAST, depth=200)) + cls._connect() + + @classmethod + def tearDownClass(cls): + cls.node.destroy_node() + rclpy.shutdown() + + @classmethod + def _on_event(cls, msg): + with cls._events_lock: + cls._events.append((msg.event_type, msg.fault.fault_code)) + + @classmethod + def _connect(cls, timeout_sec=40.0): + cls.report_client = cls.node.create_client(ReportFault, '/fault_manager/report_fault') + cls.list_client = cls.node.create_client(ListFaults, '/fault_manager/list_faults') + cls.set_stop_client = cls.node.create_client( + SetPlannedStop, '/fault_manager/set_planned_stop') + cls.get_stop_client = cls.node.create_client( + GetPlannedStop, '/fault_manager/get_planned_stop') + for client, name in ( + (cls.report_client, 'report_fault'), + (cls.list_client, 'list_faults'), + (cls.set_stop_client, 'set_planned_stop'), + (cls.get_stop_client, 'get_planned_stop'), + ): + assert client.wait_for_service(timeout_sec=timeout_sec), \ + f'{name} service not available' + + def _call(self, client, request, timeout_sec=15.0): + future = client.call_async(request) + rclpy.spin_until_future_complete(self.node, future, timeout_sec=timeout_sec) + self.assertIsNotNone(future.result(), 'service call timed out') + return future.result() + + def _report(self, fault_code): + request = ReportFault.Request() + request.fault_code = fault_code + request.event_type = ReportFault.Request.EVENT_FAILED + request.severity = Fault.SEVERITY_ERROR + request.description = 'interrupted release test fault' + request.source_id = '/test_node' + return self._call(self.report_client, request) + + def _set_stop(self, active, *, reason='', declared_by=''): + request = SetPlannedStop.Request() + request.active = active + request.reason = reason + request.declared_by = declared_by + return self._call(self.set_stop_client, request) + + def _muted_codes(self): + request = ListFaults.Request() + request.filter_by_severity = False + request.severity = 0 + request.statuses = [] + request.include_muted = True + request.include_clusters = False + return {info.fault_code for info in self._call(self.list_client, request).muted_faults} + + def _confirmed_codes(self): + request = ListFaults.Request() + request.filter_by_severity = False + request.severity = 0 + request.statuses = [] + request.include_muted = False + request.include_clusters = False + return {fault.fault_code for fault in self._call(self.list_client, request).faults} + + def _count_events(self, fault_code, event_type): + with self._events_lock: + return sum(1 for kind, code in self._events + if code == fault_code and kind == event_type) + + def _wait_until(self, predicate, message, timeout=60.0): + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if predicate(): + return + rclpy.spin_once(self.node, timeout_sec=0.05) + raise AssertionError(message) + + def _spin_for(self, seconds): + deadline = time.monotonic() + seconds + while time.monotonic() < deadline: + rclpy.spin_once(self.node, timeout_sec=0.05) + + @staticmethod + def _process_is_running(pid): + """ + Whether `pid` is still a LIVE process - a zombie counts as gone. + + `launch`'s own `process_details` is no help here: it keeps naming the dead + process until the respawn replaces it, so waiting on it waits for the + REPLACEMENT rather than for the death, and anything done "while the node is + down" would land after the new one had already read the store. + """ + try: + with open(f'/proc/{pid}/stat', 'rb') as handle: + raw = handle.read() + except (FileNotFoundError, ProcessLookupError): + return False + except OSError: + return True + state = raw[raw.rfind(b')') + 2:raw.rfind(b')') + 3] + return state not in (b'Z', b'X') + + @staticmethod + def _withdraw_declaration_in_the_store(): + """ + Leave the store exactly as a crash mid-release would. + + The declaration is withdrawn - that write happens first - while the + ownership flags, which are cleared last, are still set. + """ + connection = sqlite3.connect(DATABASE_PATH) + try: + connection.execute( + 'UPDATE planned_stop SET active = 0, ended_at_ns = ? WHERE id = 1', + (int(time.time() * 1e9),)) + connection.commit() + owned = connection.execute( + 'SELECT COUNT(*) FROM faults WHERE planned_stop_owned != 0').fetchone()[0] + finally: + connection.close() + return owned + + def test_startup_finishes_a_release_the_previous_process_did_not( + self, fault_manager_node): + self.assertTrue(self._set_stop(True, reason='cell 4 rebuild', + declared_by='plant_manager').success) + for code in OWNED_CODES: + self._report(code) + self._wait_until(lambda: set(OWNED_CODES) <= self._muted_codes(), + 'the faults were never marked by the stop') + for code in OWNED_CODES: + self.assertEqual(0, self._count_events(code, FaultEvent.EVENT_CONFIRMED)) + + original_pid = fault_manager_node.process_details['pid'] + os.kill(original_pid, signal.SIGTERM) + + deadline = time.monotonic() + 30.0 + while time.monotonic() < deadline and self._process_is_running(original_pid): + time.sleep(0.1) + self.assertFalse(self._process_is_running(original_pid), + 'the fault manager was still running, so the store edit below would ' + 'not be the state a crash leaves') + + still_owned = self._withdraw_declaration_in_the_store() + self.assertEqual(len(OWNED_CODES), still_owned, + 'the store did not hold the ownership flags a crash would leave') + + self._wait_until( + lambda: fault_manager_node.process_details is not None + and fault_manager_node.process_details['pid'] != original_pid, + 'the fault manager was never restarted') + self._connect() + + # Startup sees flags behind a withdrawn declaration and finishes the job. + for code in OWNED_CODES: + self._wait_until( + lambda code=code: self._count_events(code, FaultEvent.EVENT_CONFIRMED) == 1, + f'{code} was never announced after the interrupted release') + + self._spin_for(3.0) + for code in OWNED_CODES: + self.assertEqual(1, self._count_events(code, FaultEvent.EVENT_CONFIRMED), + f'{code} was announced more than once') + self.assertIn(code, self._confirmed_codes()) + + self.assertEqual(set(), set(OWNED_CODES) & self._muted_codes(), + 'the faults are still marked after the release was finished') + self.assertFalse(self._call(self.get_stop_client, GetPlannedStop.Request()).active) + + # And the flags are gone, so a second restart announces nothing again. + connection = sqlite3.connect(f'file:{DATABASE_PATH}?mode=ro', uri=True) + try: + remaining = connection.execute( + 'SELECT COUNT(*) FROM faults WHERE planned_stop_owned != 0').fetchone()[0] + finally: + connection.close() + self.assertEqual(0, remaining) + + +@launch_testing.post_shutdown_test() +class TestShutdown(unittest.TestCase): + + def test_exit_codes(self, proc_info): + for info in proc_info: + self.assertIn(info.returncode, (0, -2, -15), + f'{info.process_name} exited with code {info.returncode}') diff --git a/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_restart.test.py b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_restart.test.py index f46edd7c5..3445fa93c 100644 --- a/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_restart.test.py +++ b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_restart.test.py @@ -40,6 +40,30 @@ STORAGE_DIR = tempfile.mkdtemp(prefix='planned_stop_restart_') DATABASE_PATH = os.path.join(STORAGE_DIR, 'faults.db') +CORRELATION_PATH = os.path.join(STORAGE_DIR, 'correlation.yaml') + +# A rule whose mute is NOT persisted anywhere, so a restart is where the +# difference between "the stop owns this cycle" and "a rule is muting it" shows. +CORRELATION_RULES = """ +correlation: + enabled: true + patterns: + restart_motor_errors: + codes: ["MOTOR_RESTART_*"] + rules: + - id: restart_estop_cascade + mode: hierarchical + root_cause: + codes: ["ESTOP_RESTART_001"] + symptoms: + - pattern: restart_motor_errors + window_ms: 60000 + mute_symptoms: true + auto_clear_with_root: false +""" + +RULE_ROOT_CODE = 'ESTOP_RESTART_001' +RULE_SYMPTOM_CODE = 'MOTOR_RESTART_LEFT' REASON = 'weekend shutdown, cell 4' DECLARED_BY = 'plant_manager' @@ -69,6 +93,9 @@ def get_coverage_env(): def generate_test_description(): """Launch a fault manager that launch brings back after it is killed.""" + with open(CORRELATION_PATH, 'w') as handle: + handle.write(CORRELATION_RULES) + fault_manager_env = get_coverage_env() fault_manager_env['ROS_LOCALHOST_ONLY'] = '1' @@ -82,6 +109,7 @@ def generate_test_description(): 'storage_type': 'sqlite', 'database_path': DATABASE_PATH, 'confirmation_threshold': -1, + 'correlation.config_file': CORRELATION_PATH, 'snapshots.enabled': False, 'snapshots.rosbag.enabled': False, }], @@ -216,6 +244,14 @@ def _confirmed_codes(self): return {fault.fault_code for fault in self._call(self.list_client, request).faults} def test_a_stop_declared_before_a_restart_still_mutes_after_it(self, fault_manager_node): + # A rule-muted fault whose cycle starts BEFORE the stop. Its mute lives only + # in the engine, so the restart is where the stop could wrongly adopt it. + self._report(RULE_ROOT_CODE) + self._report(RULE_SYMPTOM_CODE) + self._wait_until(lambda: RULE_SYMPTOM_CODE in self._muted_codes(), + f'{RULE_SYMPTOM_CODE} was never muted by the rule') + self.assertEqual('restart_estop_cascade', self._muted_codes()[RULE_SYMPTOM_CODE].rule_id) + declared = self._set_stop(True, reason=REASON, declared_by=DECLARED_BY) self.assertTrue(declared.success) self.assertFalse(declared.was_active) @@ -256,6 +292,18 @@ def test_a_stop_declared_before_a_restart_still_mutes_after_it(self, fault_manag 'the restart moved the time the stop started') self.assertEqual(before.since.nanosec, after.since.nanosec) + # Exactly what the stop owned comes back - nothing else. Asserted before + # anything new is reported, so a fault raised after the restart cannot + # stand in for one the restore was supposed to find. + restored = self._muted_codes() + self.assertEqual( + {BEFORE_CODE, SECOND_BEFORE_CODE}, set(restored), + 'the restart restored something other than exactly the faults the stop owned') + # The rule's mute is not persisted, so it is gone after the restart - and the + # stop must not have adopted a cycle that started before it was declared. + self.assertNotIn(RULE_SYMPTOM_CODE, restored) + self.assertIn(RULE_SYMPTOM_CODE, self._confirmed_codes()) + # The declaration is not a museum piece: a fault reported after the # restart is muted by it. self._report(AFTER_CODE) @@ -267,12 +315,11 @@ def test_a_stop_declared_before_a_restart_still_mutes_after_it(self, fault_manag # The mute set lived in the process that is gone, so the manager rebuilds it # from the store: the faults marked before the restart are still marked. - muted_after_restart = self._muted_codes() for code in (BEFORE_CODE, SECOND_BEFORE_CODE): - self.assertIn(code, muted_after_restart, + self.assertIn(code, restored, f'{code} was silently released by the restart, so its ' 'confirmation could never be announced') - self.assertEqual('planned_stop', muted_after_restart[code].rule_id) + self.assertEqual('planned_stop', restored[code].rule_id) self.assertNotIn(code, self._confirmed_codes()) withdrawn = self._set_stop(False, reason='plant back up', declared_by=DECLARED_BY) @@ -292,6 +339,10 @@ def test_a_stop_declared_before_a_restart_still_mutes_after_it(self, fault_manag f'{code} was never announced at the switch-off') self.assertIn(code, self._confirmed_codes()) + self.assertEqual( + 0, self._count_events(RULE_SYMPTOM_CODE, FaultEvent.EVENT_CONFIRMED), + 'the switch-off announced a fault the stop never owned') + # A second frame arriving late would still be a double announcement. self._spin_for(3.0) for code in (BEFORE_CODE, SECOND_BEFORE_CODE, AFTER_CODE): diff --git a/src/ros2_medkit_fault_manager/test/test_correlation_engine.cpp b/src/ros2_medkit_fault_manager/test/test_correlation_engine.cpp index 730e98618..5b9977e87 100644 --- a/src/ros2_medkit_fault_manager/test/test_correlation_engine.cpp +++ b/src/ros2_medkit_fault_manager/test/test_correlation_engine.cpp @@ -76,6 +76,32 @@ class CorrelationEngineTest : public ::testing::Test { return parse_config_string(yaml); } + /// The same hierarchical rule, but acknowledging the root cause does NOT clear + /// the symptoms. That is what makes the symptom outlive the rule's mute and + /// fall back to whatever else was holding it. + CorrelationConfig create_hierarchical_no_autoclear_config() { + const std::string yaml = R"( +correlation: + enabled: true + default_window_ms: 500 + patterns: + motor_errors: + codes: ["MOTOR_COMM_*", "MOTOR_TIMEOUT_*"] + rules: + - id: estop_cascade + name: "E-Stop Cascade" + mode: hierarchical + root_cause: + codes: ["ESTOP_001"] + symptoms: + - pattern: motor_errors + window_ms: 1000 + mute_symptoms: true + auto_clear_with_root: false +)"; + return parse_config_string(yaml); + } + /// A hierarchical rule and an auto-cluster rule side by side. The cluster path /// sets should_mute without ever writing muted_faults_, which is the case the /// planned stop has to survive. @@ -1021,7 +1047,7 @@ TEST_F(CorrelationEngineTest, AStillActiveReReportDoesNotReleaseAStopMute) { // Planned stop: exactly one mute owner per fault // ============================================================================ -TEST_F(CorrelationEngineTest, TheStopNeverOverwritesARuleMute) { +TEST_F(CorrelationEngineTest, ARuleMuteOverlaysTheStopsWithoutReplacingOwnership) { CorrelationEngine engine(create_hierarchical_config()); auto t0 = std::chrono::steady_clock::now(); @@ -1077,7 +1103,7 @@ TEST_F(CorrelationEngineTest, AClusterMuteLeavesTheStopHoldingItsOwnEntry) { // Planned stop: dropping and restoring one fault's mute // ============================================================================ -TEST_F(CorrelationEngineTest, DroppingAStopMuteReleasesOnlyWhatTheStopOwns) { +TEST_F(CorrelationEngineTest, ReleasingOneFaultTakesOnlyTheStopsOwnMute) { CorrelationEngine engine(create_hierarchical_config()); auto t0 = std::chrono::steady_clock::now(); @@ -1097,11 +1123,11 @@ TEST_F(CorrelationEngineTest, DroppingAStopMuteReleasesOnlyWhatTheStopOwns) { }; ASSERT_EQ("estop_cascade", rule_id_of("MOTOR_COMM_FL")); - engine.drop_planned_stop_mute("VALVE_STUCK"); + engine.release_planned_stop_ownership("VALVE_STUCK"); EXPECT_FALSE(engine.is_muted("VALVE_STUCK")); // A rule's mute is not the stop's to drop. - engine.drop_planned_stop_mute("MOTOR_COMM_FL"); + engine.release_planned_stop_ownership("MOTOR_COMM_FL"); EXPECT_TRUE(engine.is_muted("MOTOR_COMM_FL")); auto unmuted = engine.end_planned_stop(); @@ -1109,14 +1135,14 @@ TEST_F(CorrelationEngineTest, DroppingAStopMuteReleasesOnlyWhatTheStopOwns) { EXPECT_EQ("ESTOP_001", unmuted[0]); } -TEST_F(CorrelationEngineTest, ARestoredStopMuteBehavesLikeOneTakenLive) { +TEST_F(CorrelationEngineTest, RestoredOwnershipBehavesLikeOwnershipTakenLive) { CorrelationEngine engine{CorrelationConfig{}}; // What a restart does: the declaration is read back, then the faults whose // cycles started inside it are marked again before anything is served. engine.begin_planned_stop(); - engine.restore_planned_stop_mute("VALVE_STUCK"); - engine.restore_planned_stop_mute("PUMP_SEAL_LEAK"); + engine.restore_planned_stop_ownership("VALVE_STUCK"); + engine.restore_planned_stop_ownership("PUMP_SEAL_LEAK"); EXPECT_TRUE(engine.is_muted("VALVE_STUCK")); EXPECT_EQ(2u, engine.get_muted_count()); @@ -1130,6 +1156,144 @@ TEST_F(CorrelationEngineTest, ARestoredStopMuteBehavesLikeOneTakenLive) { EXPECT_EQ("VALVE_STUCK", unmuted[1]); } +// ============================================================================ +// Ownership is the truth; the mute map is derived from it +// ============================================================================ + +TEST_F(CorrelationEngineTest, AnOwnedFaultReturnsToTheStopsMuteWhenARuleReleasesIt) { + // auto_clear_with_root is off here on purpose: the symptom stays up when the + // root cause is acknowledged, so the rule's mute ends while the fault does not. + CorrelationEngine engine(create_hierarchical_no_autoclear_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + + // Both cycles start inside the stop, so the stop owns both. The rule then + // overlays the symptom's mute. + engine.process_fault("ESTOP_001", "CRITICAL", t0, /*cycle_started=*/true); + auto symptom = engine.process_fault("MOTOR_COMM_FL", "ERROR", t0 + 10ms, /*cycle_started=*/true); + EXPECT_TRUE(symptom.should_mute); + auto rule_id_of = [&engine](const std::string & code) { + for (const auto & entry : engine.get_muted_faults()) { + if (entry.fault_code == code) { + return entry.rule_id; + } + } + return std::string{""}; + }; + EXPECT_EQ("estop_cascade", rule_id_of("MOTOR_COMM_FL")); + + // Acknowledging the root cause takes the rule's mute away. The stop still owns + // the symptom's cycle, so it goes back to being the stop's. + engine.process_clear("ESTOP_001"); + + EXPECT_TRUE(engine.is_muted("MOTOR_COMM_FL")) << "the symptom fell out of the stop when the rule let go"; + EXPECT_EQ(CorrelationEngine::kPlannedStopRuleId, rule_id_of("MOTOR_COMM_FL")); + + auto unmuted = engine.end_planned_stop(); + ASSERT_EQ(1u, unmuted.size()); + EXPECT_EQ("MOTOR_COMM_FL", unmuted[0]); +} + +TEST_F(CorrelationEngineTest, ARuleThatStopsMatchingLeavesTheFaultMuted) { + CorrelationEngine engine(create_hierarchical_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + engine.process_fault("ESTOP_001", "CRITICAL", t0, /*cycle_started=*/true); + engine.process_fault("MOTOR_COMM_FL", "ERROR", t0 + 10ms, /*cycle_started=*/true); + + // The correlation window closes and the pending root cause expires. A repeat + // report of the still-active symptom matches no rule any more - and must still + // report itself as muted, or the manager announces an update for a fault the + // list is hiding. + engine.cleanup_expired(); + auto repeat = engine.process_fault("MOTOR_COMM_FL", "ERROR", t0 + 5000ms, /*cycle_started=*/false); + EXPECT_TRUE(repeat.should_mute) << "a repeat report of a muted fault reported itself unmuted"; + EXPECT_TRUE(engine.is_muted("MOTOR_COMM_FL")); +} + +TEST_F(CorrelationEngineTest, AClusterMemberStaysMutedWhenTheClusterWindowExpires) { + CorrelationEngine engine(create_cluster_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + engine.process_fault("VALVE_A", "ERROR", t0, /*cycle_started=*/true); + engine.process_fault("VALVE_B", "ERROR", t0 + 10ms, /*cycle_started=*/true); + ASSERT_TRUE(engine.is_muted("VALVE_B")); + + // The cluster's window closes; the stop still owns both cycles. + engine.cleanup_expired(); + auto repeat = engine.process_fault("VALVE_B", "ERROR", t0 + 120000ms, /*cycle_started=*/false); + EXPECT_TRUE(repeat.should_mute); + EXPECT_TRUE(engine.is_muted("VALVE_B")); + + auto unmuted = engine.end_planned_stop(); + std::sort(unmuted.begin(), unmuted.end()); + ASSERT_EQ(2u, unmuted.size()); +} + +TEST_F(CorrelationEngineTest, OwnershipSurvivesARuleOverlayAndIsReadableBack) { + CorrelationEngine engine(create_hierarchical_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + engine.process_fault("ESTOP_001", "CRITICAL", t0, /*cycle_started=*/true); + engine.process_fault("MOTOR_COMM_FL", "ERROR", t0 + 10ms, /*cycle_started=*/true); + + auto owned = engine.planned_stop_owned_codes(); + std::sort(owned.begin(), owned.end()); + ASSERT_EQ(2u, owned.size()) << "a rule overlay took the stop's ownership away"; + EXPECT_EQ("ESTOP_001", owned[0]); + EXPECT_EQ("MOTOR_COMM_FL", owned[1]); +} + +TEST_F(CorrelationEngineTest, AFaultWhoseCycleStartedBeforeTheStopIsNeverOwned) { + CorrelationEngine engine(create_hierarchical_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.process_fault("ESTOP_001", "CRITICAL", t0); + engine.process_fault("MOTOR_COMM_FL", "ERROR", t0 + 10ms); + + engine.begin_planned_stop(); + engine.process_fault("MOTOR_COMM_FL", "ERROR", t0 + 20ms, /*cycle_started=*/false); + + EXPECT_TRUE(engine.planned_stop_owned_codes().empty()); + EXPECT_TRUE(engine.end_planned_stop().empty()); + EXPECT_TRUE(engine.is_muted("MOTOR_COMM_FL")) << "the rule's mute was collateral damage"; +} + +TEST_F(CorrelationEngineTest, ClearingAnOwnedFaultEndsTheStopsOwnershipOfIt) { + CorrelationEngine engine(create_hierarchical_config()); + + engine.begin_planned_stop(); + engine.process_fault("VALVE_STUCK", "ERROR", std::chrono::steady_clock::now(), /*cycle_started=*/true); + ASSERT_EQ(1u, engine.planned_stop_owned_codes().size()); + + engine.process_clear("VALVE_STUCK"); + EXPECT_TRUE(engine.planned_stop_owned_codes().empty()); + EXPECT_FALSE(engine.is_muted("VALVE_STUCK")); + EXPECT_TRUE(engine.end_planned_stop().empty()); +} + +TEST_F(CorrelationEngineTest, AutoClearedSymptomsLeaveTheStopsOwnership) { + CorrelationEngine engine(create_hierarchical_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + engine.process_fault("ESTOP_001", "CRITICAL", t0, /*cycle_started=*/true); + engine.process_fault("MOTOR_COMM_FL", "ERROR", t0 + 10ms, /*cycle_started=*/true); + + auto cleared = engine.process_clear("ESTOP_001"); + ASSERT_EQ(1u, cleared.auto_cleared_codes.size()); + EXPECT_EQ("MOTOR_COMM_FL", cleared.auto_cleared_codes[0]); + + // Both cycles ended with the acknowledgement, so neither is the stop's any more. + EXPECT_TRUE(engine.planned_stop_owned_codes().empty()); + EXPECT_FALSE(engine.is_muted("MOTOR_COMM_FL")); + EXPECT_TRUE(engine.end_planned_stop().empty()); +} + int main(int argc, char ** argv) { ::testing::InitGoogleTest(&argc, argv); return RUN_ALL_TESTS(); diff --git a/src/ros2_medkit_fault_manager/test/test_fault_manager.cpp b/src/ros2_medkit_fault_manager/test/test_fault_manager.cpp index 1c5434fd8..e63e496b1 100644 --- a/src/ros2_medkit_fault_manager/test/test_fault_manager.cpp +++ b/src/ros2_medkit_fault_manager/test/test_fault_manager.cpp @@ -3002,6 +3002,16 @@ class PlannedStopFailingStorage : public ros2_medkit_fault_manager::InMemoryFaul } }; +/// Reaches the node's protected storage-injecting constructor. The backend is in +/// place before anything borrows it, so nothing is left dangling. +class NodeWithInjectedStorage : public FaultManagerNode { + public: + NodeWithInjectedStorage(const rclcpp::NodeOptions & options, + std::unique_ptr storage) + : FaultManagerNode(options, std::move(storage)) { + } +}; + } // namespace /// The switch driven over its real services, against a node in this process. @@ -3009,6 +3019,12 @@ class PlannedStopServiceTest : public ::testing::Test { protected: static inline std::atomic test_counter_{0}; + /// Overridden by the failure-path fixture to hand the node a store that refuses + /// to record the declaration. + virtual std::unique_ptr make_storage() { + return nullptr; + } + void SetUp() override { const std::string ns = "/test_planned_stop_" + std::to_string(test_counter_.fetch_add(1)); @@ -3018,7 +3034,9 @@ class PlannedStopServiceTest : public ::testing::Test { rclcpp::Parameter("confirmation_threshold", -1), }); fm_options.arguments({"--ros-args", "-r", "__ns:=" + ns}); - fault_manager_ = std::make_shared(fm_options); + auto storage = make_storage(); + fault_manager_ = storage ? std::make_shared(fm_options, std::move(storage)) + : std::make_shared(fm_options); rclcpp::NodeOptions test_options; test_options.arguments({"--ros-args", "-r", "__ns:=" + ns}); @@ -3077,11 +3095,17 @@ class PlannedStopServiceTest : public ::testing::Test { rclcpp::Client::SharedPtr get_client_; }; +/// The same fixture, with a store that refuses to record the declaration. +class PlannedStopFailingStoreTest : public PlannedStopServiceTest { + protected: + std::unique_ptr make_storage() override { + return std::make_unique(); + } +}; + // A store that cannot record the declaration must not leave the caller believing // a stop is in force, and must not take the process down with it. -TEST_F(PlannedStopServiceTest, AStoreThatRejectsTheWriteIsReportedNotThrown) { - fault_manager_->set_storage_for_test(std::make_unique()); - +TEST_F(PlannedStopFailingStoreTest, AStoreThatRejectsTheWriteIsReportedNotThrown) { const auto response = set_stop(true, "line 3 maintenance", "shift_lead"); EXPECT_FALSE(response.success); EXPECT_FALSE(response.message.empty()); diff --git a/src/ros2_medkit_fault_manager/test/test_sqlite_storage.cpp b/src/ros2_medkit_fault_manager/test/test_sqlite_storage.cpp index 859fe8aa7..bdbba38e9 100644 --- a/src/ros2_medkit_fault_manager/test/test_sqlite_storage.cpp +++ b/src/ros2_medkit_fault_manager/test/test_sqlite_storage.cpp @@ -16,6 +16,7 @@ #include +#include #include #include #include @@ -2518,6 +2519,101 @@ TEST_F(SqliteFaultStorageTest, ADatabaseWrittenBeforeTheEndTimeExistedStillOpens EXPECT_EQ(0, state.ended_at_ns); } +// ============================================================================ +// Planned-stop ownership: one persisted flag per fault +// ============================================================================ + +namespace { + +/// Drive a fault to CONFIRMED with the immediate-confirmation default. +void raise(ros2_medkit_fault_manager::FaultStorage & storage, const std::string & code) { + rclcpp::Clock clock; + storage.report_fault_event(code, ReportFault::Request::EVENT_FAILED, Fault::SEVERITY_ERROR, "owned test", "/src", + clock.now(), default_config()); +} + +} // namespace + +TEST_F(SqliteFaultStorageTest, PlannedStopOwnershipRoundTripsThroughAReopen) { + raise(*storage_, "OWNED_ONE"); + raise(*storage_, "OWNED_TWO"); + raise(*storage_, "NOT_OWNED"); + + EXPECT_TRUE(storage_->get_planned_stop_owned().empty()); + + storage_->set_planned_stop_owned("OWNED_ONE", true); + storage_->set_planned_stop_owned("OWNED_TWO", true); + + storage_.reset(); + storage_ = std::make_unique(temp_db_path_.string()); + + auto owned = storage_->get_planned_stop_owned(); + std::sort(owned.begin(), owned.end()); + ASSERT_EQ(2u, owned.size()); + EXPECT_EQ("OWNED_ONE", owned[0]); + EXPECT_EQ("OWNED_TWO", owned[1]); +} + +TEST_F(SqliteFaultStorageTest, AcknowledgingAFaultEndsTheStopsOwnershipOfIt) { + raise(*storage_, "OWNED_CLEARED"); + storage_->set_planned_stop_owned("OWNED_CLEARED", true); + ASSERT_EQ(1u, storage_->get_planned_stop_owned().size()); + + ASSERT_TRUE(storage_->clear_fault("OWNED_CLEARED")); + + EXPECT_TRUE(storage_->get_planned_stop_owned().empty()) + << "an acknowledged fault is over, so the stop has nothing left to own"; +} + +TEST_F(SqliteFaultStorageTest, ClearingOwnershipDropsEveryFlagAndReportsHowMany) { + raise(*storage_, "OWNED_A"); + raise(*storage_, "OWNED_B"); + storage_->set_planned_stop_owned("OWNED_A", true); + storage_->set_planned_stop_owned("OWNED_B", true); + + EXPECT_EQ(2u, storage_->clear_planned_stop_owned()); + EXPECT_TRUE(storage_->get_planned_stop_owned().empty()); + EXPECT_EQ(0u, storage_->clear_planned_stop_owned()); +} + +TEST_F(SqliteFaultStorageTest, ADatabaseWrittenBeforeOwnershipExistedStillOpens) { + raise(*storage_, "LEGACY_FAULT"); + storage_.reset(); + { + sqlite3 * raw = nullptr; + ASSERT_EQ(sqlite3_open(temp_db_path_.string().c_str(), &raw), SQLITE_OK); + // What a database from before the column looks like. + ASSERT_EQ(sqlite3_exec(raw, "ALTER TABLE faults DROP COLUMN planned_stop_owned", nullptr, nullptr, nullptr), + SQLITE_OK); + sqlite3_close(raw); + } + + storage_ = std::make_unique(temp_db_path_.string()); + + EXPECT_TRUE(storage_->get_planned_stop_owned().empty()); + EXPECT_TRUE(storage_->contains("LEGACY_FAULT")); + storage_->set_planned_stop_owned("LEGACY_FAULT", true); + EXPECT_EQ(1u, storage_->get_planned_stop_owned().size()); +} + +TEST(InMemoryFaultStorageTest, PlannedStopOwnershipIsTrackedPerFault) { + ros2_medkit_fault_manager::InMemoryFaultStorage storage; + raise(storage, "OWNED_ONE"); + raise(storage, "NOT_OWNED"); + + storage.set_planned_stop_owned("OWNED_ONE", true); + auto owned = storage.get_planned_stop_owned(); + ASSERT_EQ(1u, owned.size()); + EXPECT_EQ("OWNED_ONE", owned[0]); + + ASSERT_TRUE(storage.clear_fault("OWNED_ONE")); + EXPECT_TRUE(storage.get_planned_stop_owned().empty()); + + storage.set_planned_stop_owned("NOT_OWNED", true); + EXPECT_EQ(1u, storage.clear_planned_stop_owned()); + EXPECT_TRUE(storage.get_planned_stop_owned().empty()); +} + TEST_F(SqliteFaultStorageTest, PlannedStopCarriesAKilobyteReason) { const std::string long_reason(1024, 'r'); ros2_medkit_fault_manager::PlannedStopState declared; diff --git a/src/ros2_medkit_msgs/README.md b/src/ros2_medkit_msgs/README.md index 9bce3944b..232ee5970 100644 --- a/src/ros2_medkit_msgs/README.md +++ b/src/ros2_medkit_msgs/README.md @@ -156,17 +156,19 @@ Declare, or withdraw, a planned stop on the FaultManager. | `message` | string | Status or error message | | `was_active` | bool | The state of the switch before this request | -While a planned stop is on, a fault whose cycle *starts* is reported, debounced, -confirmed, captured and audited unchanged, and is marked as muted: absent from the default -`ListFaults` response, counted in `muted_count`, and listed under `muted_faults` with -`rule_id = "planned_stop"` when `include_muted` is set. A cycle that started *before* the -stop is untouched - reporters re-send FAILED for as long as a condition holds, and that -fault's confirmation has already been announced. +While a planned stop is on, it owns every fault cycle that *starts* - a new fault, one +raised again after being cleared, or one that fails again after healing. An owned fault is +reported, debounced, confirmed, captured and audited unchanged, and is marked as muted: +absent from the default `ListFaults` response, counted in `muted_count`, and listed under +`muted_faults` with `rule_id = "planned_stop"` when `include_muted` is set. A cycle that +started *before* the stop is untouched - reporters re-send FAILED for as long as a +condition holds, and that fault's confirmation has already been announced. Publication matches a rule-muted symptom exactly: `EVENT_CONFIRMED` and `EVENT_UPDATED` -are withheld, `EVENT_CLEARED` is published as usual. Withdrawing the stop unmutes every -fault it alone was muting and publishes one `EVENT_CONFIRMED` for each of those that is -CONFIRMED. A request asking for the state the switch is already in succeeds, changes +are withheld whichever kind of report produced them, `EVENT_CLEARED` is published as +usual. Withdrawing the stop releases every fault it owns and publishes one +`EVENT_CONFIRMED` for each of those that is CONFIRMED. A correlation rule muting an owned +fault overlays the stop rather than taking the fault from it. A request asking for the state the switch is already in succeeds, changes nothing, and writes no audit record; a request the store cannot record answers `success: false` and changes nothing. Audit records exist only when `audit_log.enabled` is set, which it is not by default. diff --git a/src/ros2_medkit_msgs/srv/GetPlannedStop.srv b/src/ros2_medkit_msgs/srv/GetPlannedStop.srv index 7e97bfdfd..ef8328aeb 100644 --- a/src/ros2_medkit_msgs/srv/GetPlannedStop.srv +++ b/src/ros2_medkit_msgs/srv/GetPlannedStop.srv @@ -28,12 +28,12 @@ # Whether a planned stop is currently declared. bool active -# The reason given when the stop was declared. Retained after the withdrawal; -# empty only while no stop has ever been declared. +# The reason given when the stop was declared, verbatim - nothing validates or +# fills it in, so it is empty when the declaration left it empty, and when no stop +# has ever been declared. Retained after the withdrawal. string reason -# Who declared the stop. Retained after the withdrawal; empty only while no stop -# has ever been declared. +# Who declared the stop, on the same terms as `reason`. string declared_by # When the stop was declared, wall clock. Zero while no stop has ever been declared. diff --git a/src/ros2_medkit_msgs/srv/SetPlannedStop.srv b/src/ros2_medkit_msgs/srv/SetPlannedStop.srv index a439025fd..147d2fcd8 100644 --- a/src/ros2_medkit_msgs/srv/SetPlannedStop.srv +++ b/src/ros2_medkit_msgs/srv/SetPlannedStop.srv @@ -14,22 +14,28 @@ # # SetPlannedStop.srv - Declare, or withdraw, a planned stop on the FaultManager. # -# While a planned stop is on, a fault whose cycle STARTS is reported, debounced, -# confirmed, captured and audited exactly as it would be otherwise, and is MARKED as -# muted rather than dropped: it is absent from the default ListFaults response, counted -# in `muted_count`, and listed under `muted_faults` when `include_muted` is set. A cycle -# that started BEFORE the stop is untouched: it stays in the fault list, because its -# confirmation has already been announced, and reporters re-send FAILED for as long as -# the condition holds. -# -# A marked fault is published exactly as a rule-muted symptom is: its EVENT_CONFIRMED -# and EVENT_UPDATED are withheld, while an EVENT_CLEARED - the fault healing, or being -# acknowledged - is published as usual. Withdrawing the stop unmutes every fault it -# muted that is still active, and publishes one EVENT_CONFIRMED for each that is -# CONFIRMED, because that confirmation was never announced. -# -# The declaration survives a restart of the fault manager, and so does the marking of -# the faults whose cycles started inside it. +# While a planned stop is on, it OWNS every fault cycle that STARTS - a new fault, one +# raised again after being cleared, or one that failed again after healing. An owned +# fault is reported, debounced, confirmed, captured and audited exactly as it would be +# otherwise, and is MARKED as muted rather than dropped: absent from the default +# ListFaults response, counted in `muted_count`, and listed under `muted_faults` when +# `include_muted` is set. A cycle that started BEFORE the stop is untouched: it stays in +# the fault list, because its confirmation has already been announced, and reporters +# re-send FAILED for as long as the condition holds. +# +# A muted fault is published exactly as a rule-muted symptom is: its EVENT_CONFIRMED and +# EVENT_UPDATED are withheld, whichever kind of report produced them, while an +# EVENT_CLEARED - the fault healing, or being acknowledged - is published as usual. +# Withdrawing the stop releases every fault it owns, and publishes one EVENT_CONFIRMED +# for each that is CONFIRMED, because that confirmation was never announced. +# +# A correlation rule muting an owned fault OVERLAYS the stop rather than taking the +# fault from it: while the rule holds the fault the withdrawal leaves it alone, and when +# the rule lets go the fault is the stop's again. +# +# With `storage_type: sqlite` the declaration survives a restart of the fault manager, +# and so does its ownership of the cycles that started inside it; the in-memory backend +# keeps neither past the process. # # Both transitions are recorded in the fault audit log when `audit_log.enabled` is set # (it is off by default). @@ -40,7 +46,8 @@ bool active # Why the plant is stopped ("line 3 quarterly maintenance"). Recorded in the audit log -# with the transition and served back by GetPlannedStop while the stop is on. +# with the transition, and served back by GetPlannedStop - including after the stop has +# been withdrawn, so the reason outlives it. string reason # Who declared the transition. Recorded in the audit log as the transition's source. From 124e5c39dbb5738ffff03f7c4b22fc3cfc4cff24 Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Tue, 8 Sep 2026 09:17:08 +0200 Subject: [PATCH 09/12] fault_manager: pin the release ordering, and measure the timer it costs The ordering that decides what a crash mid-release costs was only reasoned about: the interrupted-release test starts from the state such a crash leaves and cannot see how the state came about. A storage double now records the switch-off's own calls, so clearing the ownership flags before the announcement fails a test instead of passing one. The cleanup timer's cost is measured rather than asserted: ~190 ns per call on a rules-free engine, ~2.3 us of CPU per minute at the 5 s default. Docs on every surface follow the ownership model - what a cycle is, that a rule's mute overlays ownership rather than taking it and hands the fault back when it ends, that the flags and the declaration survive a restart while a rule's mute does not, and that an interrupted switch-off is finished at startup. Two stale sentences went with them: the audit header still described an empty fault code, and SetPlannedStop still said the reason is served only while the stop is on. Refs #656 --- .../fault_manager_node.hpp | 6 ++ .../test/test_fault_manager.cpp | 58 +++++++++++++++++++ 2 files changed, 64 insertions(+) diff --git a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_manager_node.hpp b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_manager_node.hpp index 4924e8222..3d1762a93 100644 --- a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_manager_node.hpp +++ b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_manager_node.hpp @@ -102,6 +102,12 @@ class FaultManagerNode : public rclcpp::Node { audit_transition(transition, fault, "test", 0); } + /// Test-only: register the planned stop's ownership of a fault in the engine, + /// the way startup does when it reads the flags back. + void restore_planned_stop_ownership_for_test(const std::string & fault_code) { + correlation_engine_->restore_planned_stop_ownership(fault_code); + } + /// Get the storage type being used const std::string & get_storage_type() const { return storage_type_; diff --git a/src/ros2_medkit_fault_manager/test/test_fault_manager.cpp b/src/ros2_medkit_fault_manager/test/test_fault_manager.cpp index e63e496b1..bfea4acbb 100644 --- a/src/ros2_medkit_fault_manager/test/test_fault_manager.cpp +++ b/src/ros2_medkit_fault_manager/test/test_fault_manager.cpp @@ -3002,6 +3002,24 @@ class PlannedStopFailingStorage : public ros2_medkit_fault_manager::InMemoryFaul } }; +/// Records the order of the calls a switch-off makes, so the ordering that decides +/// what a crash mid-release costs is testable without killing a process inside a +/// sub-millisecond window. +class ReleaseOrderRecordingStorage : public ros2_medkit_fault_manager::InMemoryFaultStorage { + public: + std::optional get_fault(const std::string & fault_code) const override { + calls.push_back("get_fault:" + fault_code); + return InMemoryFaultStorage::get_fault(fault_code); + } + + size_t clear_planned_stop_owned() override { + calls.push_back("clear_owned"); + return InMemoryFaultStorage::clear_planned_stop_owned(); + } + + mutable std::vector calls; +}; + /// Reaches the node's protected storage-injecting constructor. The backend is in /// place before anything borrows it, so nothing is left dangling. class NodeWithInjectedStorage : public FaultManagerNode { @@ -3095,6 +3113,46 @@ class PlannedStopServiceTest : public ::testing::Test { rclcpp::Client::SharedPtr get_client_; }; +/// The same fixture, with a store that records the order of the release. +class PlannedStopReleaseOrderTest : public PlannedStopServiceTest { + protected: + std::unique_ptr make_storage() override { + auto storage = std::make_unique(); + recorder_ = storage.get(); + return storage; + } + + ReleaseOrderRecordingStorage * recorder_{nullptr}; +}; + +// The ownership flags are what a startup uses to finish a release its predecessor +// did not. They must therefore outlive the announcement: cleared first, a crash +// mid-loop would leave nothing for the next process to find, and the confirmations +// suppressed during the stop would be lost for good. +TEST_F(PlannedStopReleaseOrderTest, TheOwnershipFlagsAreClearedAfterTheAnnouncement) { + ASSERT_NE(recorder_, nullptr); + ASSERT_TRUE(set_stop(true, "line 3 maintenance", "shift_lead").success); + + // A confirmed fault the stop owns, put in place directly: this test is about the + // order of the switch-off's own calls. + auto & storage = fault_manager_->get_storage_for_test(); + storage.report_fault_event("ORDER_OWNED", ReportFault::Request::EVENT_FAILED, Fault::SEVERITY_ERROR, "order test", + "/src", fault_manager_->now(), storage.get_debounce_config()); + storage.set_planned_stop_owned("ORDER_OWNED", true); + fault_manager_->restore_planned_stop_ownership_for_test("ORDER_OWNED"); + + recorder_->calls.clear(); + ASSERT_TRUE(set_stop(false, "plant back up", "shift_lead").success); + + const auto & calls = recorder_->calls; + const auto read = std::find(calls.begin(), calls.end(), "get_fault:ORDER_OWNED"); + const auto cleared = std::find(calls.begin(), calls.end(), "clear_owned"); + ASSERT_NE(read, calls.end()) << "the switch-off never looked the released fault up"; + ASSERT_NE(cleared, calls.end()) << "the switch-off never released the ownership flags"; + EXPECT_LT(read - calls.begin(), cleared - calls.begin()) + << "the flags were cleared before the announcement, so a crash in between would lose it"; +} + /// The same fixture, with a store that refuses to record the declaration. class PlannedStopFailingStoreTest : public PlannedStopServiceTest { protected: From be92108f5235c2461830eef96d611e6b70c153ef Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Tue, 8 Sep 2026 10:05:22 +0200 Subject: [PATCH 10/12] fault_manager: finish an interrupted planned-stop release in the constructor Deferring it to a wall timer left a window: the services exist by then, so an operator could declare a new stop and new cycles could become owned by it, and the deferred work would announce a fault the new stop was holding and clear every ownership flag - the new stop's included. The recovery now runs synchronously in the constructor, before this node has a service or a timer that could run, and drops exactly the flags it captured; the storage gains a scoped release for that. The price is an announcement that may reach no subscriber, because nothing has matched a publisher that is milliseconds old and the events topic is volatile. That is the same accepted loss as any other publication made at startup, and the released faults are in the default fault list either way - which is what the launch test asserts, alongside the manager's own log line. Refs #656 --- .../design/index.rst | 16 ++- .../fault_manager_node.hpp | 20 +-- .../fault_storage.hpp | 12 +- .../sqlite_fault_storage.hpp | 1 + .../src/fault_manager_node.cpp | 82 ++++++------ .../src/fault_storage.cpp | 13 ++ .../src/sqlite_fault_storage.cpp | 17 +++ ...t_planned_stop_interrupted_release.test.py | 118 +++++++++++++++--- .../test/test_fault_manager.cpp | 9 ++ .../test/test_sqlite_storage.cpp | 37 ++++++ 10 files changed, 252 insertions(+), 73 deletions(-) diff --git a/src/ros2_medkit_fault_manager/design/index.rst b/src/ros2_medkit_fault_manager/design/index.rst index 59c730f1f..fd2a3d7f9 100644 --- a/src/ros2_medkit_fault_manager/design/index.rst +++ b/src/ros2_medkit_fault_manager/design/index.rst @@ -318,9 +318,19 @@ declaration is written first (so a store that refuses the write changes nothing, a restart never re-arms a stop the operator withdrew), the confirmations are announced next, and the ownership flags are cleared last. A process that dies in between leaves faults owned by a declaration that is already over, which is a state -the next startup recognises: it announces them - once a subscriber has matched the -events publisher, because that topic is volatile - and clears the flags. Clearing -first would have turned a crash into permanent silence. +the next startup recognises: it announces them and clears the flags of exactly +those faults. Clearing first would have turned a crash into permanent silence. + +The recovery runs synchronously in the constructor, before the node has a service +or a timer. Deferring it - waiting for a subscriber to match, say - opens a window +in which an operator declares a NEW stop over the same services and new cycles +become owned by it; the deferred work then announces a fault the new stop is +holding and, if it clears ownership wholesale rather than the set it captured, +drops that fault's flag too. Running early costs an announcement that may reach no +subscriber, because nothing has matched a publisher that is milliseconds old and +the events topic is volatile. That is the same accepted loss as every other +publication made at startup, and the released faults are in the default fault list +regardless. Withdrawing releases the rest and publishes ``EVENT_CONFIRMED`` once for each released fault that is CONFIRMED. That republication is the point of marking diff --git a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_manager_node.hpp b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_manager_node.hpp index 3d1762a93..f46af823d 100644 --- a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_manager_node.hpp +++ b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_manager_node.hpp @@ -193,6 +193,12 @@ class FaultManagerNode : public rclcpp::Node { /// @param config SnapshotConfig to populate with loaded values void load_snapshot_config_from_yaml(const std::string & config_file, SnapshotConfig & config); + /// Finish a switch-off a previous process did not: announce the faults still + /// owned by a declaration that is already withdrawn, and drop exactly their + /// flags. Runs from the constructor, before this node has a service or a timer + /// that could declare a new stop underneath it. + void finish_interrupted_release(); + /// Publish EVENT_CONFIRMED for each of these faults that is CONFIRMED, which is /// what releasing a fault from the planned stop means to a consumer of the /// stream. @@ -305,20 +311,6 @@ class FaultManagerNode : public rclcpp::Node { rclcpp::Service::SharedPtr get_planned_stop_srv_; rclcpp::TimerBase::SharedPtr auto_confirm_timer_; - /// How often startup checks whether anyone is listening before finishing an - /// interrupted planned-stop release. - static constexpr std::chrono::milliseconds kInterruptedReleasePollInterval{200}; - - /// How long that check waits for a listener before announcing anyway. The events - /// topic is volatile, so an announcement made before a subscriber has matched is - /// lost for good - but a manager nobody subscribes to still has to finish the - /// release rather than carry the ownership flags for ever. - static constexpr std::chrono::seconds kInterruptedReleaseDeadline{15}; - - /// One-shot: finishes a release a previous process did not complete (nullptr - /// when there was nothing left over). - rclcpp::TimerBase::SharedPtr interrupted_release_timer_; - /// Timer for periodic cleanup of expired correlation data rclcpp::TimerBase::SharedPtr correlation_cleanup_timer_; diff --git a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_storage.hpp b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_storage.hpp index 6894c4b89..5933cc9f8 100644 --- a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_storage.hpp +++ b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_storage.hpp @@ -496,10 +496,19 @@ class FaultStorage { /// Drop every ownership flag. Called once the switch-off has announced what it /// released, so a crash before this point leaves the flags for the next startup - /// to finish. + /// to finish. Safe as an all-or-nothing sweep there: the withdrawal ends the + /// declaration that owns every flagged fault, and it runs to completion before + /// any other request is served. /// @return how many faults were released virtual size_t clear_planned_stop_owned() = 0; + /// Drop the ownership flags of exactly these faults, leaving every other one + /// alone. What a release captured is what it may clear: a fault flagged after + /// the capture belongs to a later declaration, and wiping it would take a fault + /// out of a stop that is still in force. + /// @return how many of them were owned + virtual size_t clear_planned_stop_owned(const std::vector & fault_codes) = 0; + protected: FaultStorage() = default; FaultStorage(const FaultStorage &) = default; @@ -571,6 +580,7 @@ class InMemoryFaultStorage : public FaultStorage { void set_planned_stop_owned(const std::string & fault_code, bool owned) override; std::vector get_planned_stop_owned() const override; size_t clear_planned_stop_owned() override; + size_t clear_planned_stop_owned(const std::vector & fault_codes) override; private: /// Update fault status based on debounce counter and given config diff --git a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/sqlite_fault_storage.hpp b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/sqlite_fault_storage.hpp index c2f48a8e9..ad892a735 100644 --- a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/sqlite_fault_storage.hpp +++ b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/sqlite_fault_storage.hpp @@ -100,6 +100,7 @@ class SqliteFaultStorage : public FaultStorage { void set_planned_stop_owned(const std::string & fault_code, bool owned) override; std::vector get_planned_stop_owned() const override; size_t clear_planned_stop_owned() override; + size_t clear_planned_stop_owned(const std::vector & fault_codes) override; /// Get the database path const std::string & db_path() const { diff --git a/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp b/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp index 8fc43ae0a..2c9b7596c 100644 --- a/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp +++ b/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp @@ -257,6 +257,11 @@ FaultManagerNode::FaultManagerNode(const rclcpp::NodeOptions & options, std::uni // Create event publisher for SSE streaming event_publisher_ = create_publisher("~/events", rclcpp::QoS(100).reliable()); + // A planned stop outlives the process that declared it, so the declaration and + // the cycles it owns are read back from the store rather than started fresh. + planned_stop_ = storage_->get_planned_stop(); + finish_interrupted_release(); + // Build global debounce config global_config_.confirmation_threshold = confirmation_threshold_; global_config_.healing_enabled = healing_enabled_; @@ -457,49 +462,18 @@ FaultManagerNode::FaultManagerNode(const rclcpp::NodeOptions & options, std::uni correlation_engine_->cleanup_expired(); }); - // A planned stop outlives the process that declared it: a weekend stop must not - // end because the box rebooted. Faults muted before the restart are not restored - - // muting is per fault cycle and those cycles are over as far as this process is - // concerned - but every fault reported from here is muted again. - planned_stop_ = storage_->get_planned_stop(); - const auto owned_at_startup = storage_->get_planned_stop_owned(); + // A weekend stop must not end because the box rebooted. The cycles it owns come + // back from the flags the store kept, so the switch-off still releases and + // announces exactly them. if (planned_stop_.active) { correlation_engine_->begin_planned_stop(); - for (const auto & fault_code : owned_at_startup) { + const auto owned = storage_->get_planned_stop_owned(); + for (const auto & fault_code : owned) { correlation_engine_->restore_planned_stop_ownership(fault_code); } RCLCPP_INFO(get_logger(), "Planned stop is in force (reason='%s', declared_by='%s'): %zu fault(s) re-marked, new faults marked", - planned_stop_.reason.c_str(), planned_stop_.declared_by.c_str(), owned_at_startup.size()); - } else if (!owned_at_startup.empty()) { - // Faults owned by a declaration that is already withdrawn: the previous - // process died between writing the withdrawal and finishing the release, so - // these confirmations were suppressed and never announced. Finish the job. - // - // Not here, though: this publisher was created moments ago in this same - // constructor, and the events topic is volatile, so anything published before - // a subscriber has matched it reaches nobody and is gone for good. Wait for a - // listener - and give up waiting after kInterruptedReleaseDeadline, because a - // manager nobody is listening to must still finish the release rather than - // carry the flags forever. - RCLCPP_WARN(get_logger(), - "%zu fault(s) are still owned by a planned stop that was already withdrawn - a previous run did not " - "finish releasing them. Announcing them once a consumer is listening.", - owned_at_startup.size()); - const auto give_up_at = std::chrono::steady_clock::now() + kInterruptedReleaseDeadline; - interrupted_release_timer_ = - create_wall_timer(kInterruptedReleasePollInterval, [this, owned_at_startup, give_up_at]() { - const bool listener = event_publisher_->get_subscription_count() > 0; - if (!listener && std::chrono::steady_clock::now() < give_up_at) { - return; - } - interrupted_release_timer_->cancel(); - const size_t announced = announce_released(owned_at_startup); - storage_->clear_planned_stop_owned(); - RCLCPP_INFO(get_logger(), - "Finished an interrupted planned-stop release: announced %zu confirmation(s) to %zu listener(s)", - announced, event_publisher_->get_subscription_count()); - }); + planned_stop_.reason.c_str(), planned_stop_.declared_by.c_str(), owned.size()); } // Create auto-confirmation timer if enabled @@ -548,6 +522,40 @@ FaultManagerNode::FaultManagerNode(const rclcpp::NodeOptions & options, std::uni } } +void FaultManagerNode::finish_interrupted_release() { + if (planned_stop_.active) { + return; // the declaration still stands; nothing was left half-released + } + + const auto owned = storage_->get_planned_stop_owned(); + if (owned.empty()) { + return; + } + + // Faults owned by a declaration that is already withdrawn: the previous process + // died between writing the withdrawal and dropping the flags, so these + // confirmations were suppressed and announced by nobody. Finish the job. + // + // Synchronously, here, before this node has a service or a timer that could run: + // deferring it opens a window in which an operator declares a NEW stop and new + // cycles become owned by it, and the deferred work would then announce a fault + // the new stop is holding and drop its flag. A publication this early may reach + // no subscriber - the events topic is volatile and nothing has matched a + // publisher this young - and that is the same accepted loss as any other + // publication made at startup. The released faults are in the default fault list + // either way, which is where an operator looks after a restart. + RCLCPP_WARN(get_logger(), + "%zu fault(s) are still owned by a planned stop that was already withdrawn - a previous run did not " + "finish releasing them. Releasing them now.", + owned.size()); + const size_t announced = announce_released(owned); + // Exactly what was captured above: a flag written after that capture belongs to + // a declaration this release knows nothing about. + const size_t released = storage_->clear_planned_stop_owned(owned); + RCLCPP_INFO(get_logger(), "Finished an interrupted planned-stop release: released %zu fault(s), announced %zu", + released, announced); +} + FaultManagerNode::~FaultManagerNode() { // Join capture workers FIRST so no worker is mid-capture when rosbag tears // down. RosbagCapture::stop() is not a barrier against an in-flight diff --git a/src/ros2_medkit_fault_manager/src/fault_storage.cpp b/src/ros2_medkit_fault_manager/src/fault_storage.cpp index 6487d8373..b29f4830a 100644 --- a/src/ros2_medkit_fault_manager/src/fault_storage.cpp +++ b/src/ros2_medkit_fault_manager/src/fault_storage.cpp @@ -910,6 +910,19 @@ size_t InMemoryFaultStorage::clear_planned_stop_owned() { return cleared; } +size_t InMemoryFaultStorage::clear_planned_stop_owned(const std::vector & fault_codes) { + std::lock_guard lock(mutex_); + size_t cleared = 0; + for (const auto & fault_code : fault_codes) { + auto it = faults_.find(fault_code); + if (it != faults_.end() && it->second.planned_stop_owned) { + it->second.planned_stop_owned = false; + ++cleared; + } + } + return cleared; +} + void InMemoryFaultStorage::set_planned_stop(const PlannedStopState & state) { std::lock_guard lock(mutex_); planned_stop_ = state; diff --git a/src/ros2_medkit_fault_manager/src/sqlite_fault_storage.cpp b/src/ros2_medkit_fault_manager/src/sqlite_fault_storage.cpp index 46293a053..4e926400c 100644 --- a/src/ros2_medkit_fault_manager/src/sqlite_fault_storage.cpp +++ b/src/ros2_medkit_fault_manager/src/sqlite_fault_storage.cpp @@ -1422,6 +1422,23 @@ size_t SqliteFaultStorage::clear_planned_stop_owned() { return static_cast(sqlite3_changes(db_)); } +size_t SqliteFaultStorage::clear_planned_stop_owned(const std::vector & fault_codes) { + std::lock_guard lock(mutex_); + + size_t cleared = 0; + SqliteStatement stmt(db_, + "UPDATE faults SET planned_stop_owned = 0 WHERE fault_code = ? AND planned_stop_owned != 0"); + for (const auto & fault_code : fault_codes) { + stmt.reset(); + stmt.bind_text(1, fault_code); + if (stmt.step() != SQLITE_DONE) { + throw std::runtime_error(std::string("Failed to release planned-stop ownership: ") + sqlite3_errmsg(db_)); + } + cleared += static_cast(sqlite3_changes(db_)); + } + return cleared; +} + std::optional SqliteFaultStorage::get_freeze_frame(const std::string & fault_code) const { std::lock_guard lock(mutex_); diff --git a/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_interrupted_release.test.py b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_interrupted_release.test.py index 14484ae78..d96a9519b 100644 --- a/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_interrupted_release.test.py +++ b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_interrupted_release.test.py @@ -25,6 +25,13 @@ The kill is real (SIGTERM, launch respawns the node); the crash *point* is simulated by editing the store while the manager is down, which is exactly the state such a crash leaves behind. + +The recovery runs in the constructor, before this node has a service or a timer, +so its announcement may reach no subscriber - nothing has matched a publisher that +is milliseconds old. That loss is accepted, so the assertions here are on what +survives it: the manager's own log line, the ownership flags being gone, and the +released faults being back in the default fault list. What the recovery must NEVER +do is announce or release a fault owned by a stop declared after it. """ import os @@ -43,12 +50,21 @@ from rclpy.node import Node from rclpy.qos import HistoryPolicy, QoSProfile, ReliabilityPolicy from ros2_medkit_msgs.msg import Fault, FaultEvent -from ros2_medkit_msgs.srv import GetPlannedStop, ListFaults, ReportFault, SetPlannedStop +from ros2_medkit_msgs.srv import ( + ClearFault, + GetPlannedStop, + ListFaults, + ReportFault, + SetPlannedStop, +) STORAGE_DIR = tempfile.mkdtemp(prefix='planned_stop_interrupted_') DATABASE_PATH = os.path.join(STORAGE_DIR, 'faults.db') OWNED_CODES = ('PS_INTERRUPTED_ONE', 'PS_INTERRUPTED_TWO') +# Re-raised under a NEW stop declared right after the recovery. +REUSED_CODE = OWNED_CODES[0] +NEW_STOP_CODE = 'PS_AFTER_RECOVERY' # Long enough that the test can edit the store between the kill and the # replacement opening it. @@ -137,6 +153,7 @@ def _on_event(cls, msg): @classmethod def _connect(cls, timeout_sec=40.0): cls.report_client = cls.node.create_client(ReportFault, '/fault_manager/report_fault') + cls.clear_client = cls.node.create_client(ClearFault, '/fault_manager/clear_fault') cls.list_client = cls.node.create_client(ListFaults, '/fault_manager/list_faults') cls.set_stop_client = cls.node.create_client( SetPlannedStop, '/fault_manager/set_planned_stop') @@ -144,6 +161,7 @@ def _connect(cls, timeout_sec=40.0): GetPlannedStop, '/fault_manager/get_planned_stop') for client, name in ( (cls.report_client, 'report_fault'), + (cls.clear_client, 'clear_fault'), (cls.list_client, 'list_faults'), (cls.set_stop_client, 'set_planned_stop'), (cls.get_stop_client, 'get_planned_stop'), @@ -166,6 +184,22 @@ def _report(self, fault_code): request.source_id = '/test_node' return self._call(self.report_client, request) + def _clear(self, fault_code): + request = ClearFault.Request() + request.fault_code = fault_code + request.skip_correlation_auto_clear = False + return self._call(self.clear_client, request) + + def _muted_entries(self): + request = ListFaults.Request() + request.filter_by_severity = False + request.severity = 0 + request.statuses = [] + request.include_muted = True + request.include_clusters = False + response = self._call(self.list_client, request) + return {info.fault_code: info for info in response.muted_faults} + def _set_stop(self, active, *, reason='', declared_by=''): request = SetPlannedStop.Request() request.active = active @@ -250,7 +284,7 @@ def _withdraw_declaration_in_the_store(): return owned def test_startup_finishes_a_release_the_previous_process_did_not( - self, fault_manager_node): + self, fault_manager_node, proc_output): self.assertTrue(self._set_stop(True, reason='cell 4 rebuild', declared_by='plant_manager').success) for code in OWNED_CODES: @@ -280,30 +314,78 @@ def test_startup_finishes_a_release_the_previous_process_did_not( 'the fault manager was never restarted') self._connect() - # Startup sees flags behind a withdrawn declaration and finishes the job. - for code in OWNED_CODES: - self._wait_until( - lambda code=code: self._count_events(code, FaultEvent.EVENT_CONFIRMED) == 1, - f'{code} was never announced after the interrupted release') - - self._spin_for(3.0) + # A stop declared right after the restart owns its own cycles. Declared + # FIRST, before this test waits for anything: a recovery that runs later + # than the constructor would be inside this window, which is exactly the + # window in which it can announce and unflag a fault the NEW stop holds. + self.assertTrue(self._set_stop(True, reason='second shutdown', + declared_by='night_shift').success) + + # Whatever the recovery did or did not manage to announce, nothing more may + # be announced for this code while the new stop holds its new cycle. + announced_before = self._count_events(REUSED_CODE, FaultEvent.EVENT_CONFIRMED) + + # A NEW cycle of a code the previous release captured: acknowledge it, then + # raise it again under the new declaration. + self.assertTrue(self._clear(REUSED_CODE).success) + self._report(REUSED_CODE) + self._report(NEW_STOP_CODE) + self._wait_until( + lambda: {REUSED_CODE, NEW_STOP_CODE} <= set(self._muted_entries()), + 'the new stop did not take the cycles that started under it') + + # Startup finishes the interrupted release, and says so itself - the one + # instrument a publication made before anyone has matched the publisher + # cannot take away. + expected = ('Finished an interrupted planned-stop release: ' + f'released {len(OWNED_CODES)} fault(s)') + proc_output.assertWaitFor(expected_output=expected, process='fault_manager_node-1', + timeout=40, stream='stderr') + self._spin_for(2.0) + + # What the release captured is released: back in the default list, not + # marked, announced at most once. for code in OWNED_CODES: - self.assertEqual(1, self._count_events(code, FaultEvent.EVENT_CONFIRMED), - f'{code} was announced more than once') - self.assertIn(code, self._confirmed_codes()) - - self.assertEqual(set(), set(OWNED_CODES) & self._muted_codes(), - 'the faults are still marked after the release was finished') + if code == REUSED_CODE: + continue # it has since started a new cycle under the new stop + self.assertIn(code, self._confirmed_codes(), + f'{code} did not come back to the default fault list') + self.assertNotIn(code, self._muted_codes(), + f'{code} is still marked after the release was finished') + self.assertLessEqual(self._count_events(code, FaultEvent.EVENT_CONFIRMED), 1, + f'{code} was announced more than once') + + # And what it did NOT capture is untouched: the new stop still holds both + # of its cycles, flags and all. + entries = self._muted_entries() + for code in (REUSED_CODE, NEW_STOP_CODE): + self.assertIn(code, entries, + f'{code} was released by a recovery that never owned it') + self.assertEqual('planned_stop', entries[code].rule_id) + self.assertNotIn(code, self._confirmed_codes()) + self.assertEqual(2, self._owned_flag_count(), + 'the recovery wiped ownership flags belonging to the new stop') + self.assertEqual(announced_before, + self._count_events(REUSED_CODE, FaultEvent.EVENT_CONFIRMED), + 'the new cycle was announced while the new stop still stands') + self.assertEqual(0, self._count_events(NEW_STOP_CODE, FaultEvent.EVENT_CONFIRMED)) + + # The new stop still releases its own faults normally. + self.assertTrue(self._set_stop(False, reason='back up', declared_by='night_shift').success) + for code in (REUSED_CODE, NEW_STOP_CODE): + self._wait_until(lambda code=code: code in self._confirmed_codes(), + f'{code} was not released by its own switch-off') + self.assertEqual(0, self._owned_flag_count()) self.assertFalse(self._call(self.get_stop_client, GetPlannedStop.Request()).active) - # And the flags are gone, so a second restart announces nothing again. + @staticmethod + def _owned_flag_count(): connection = sqlite3.connect(f'file:{DATABASE_PATH}?mode=ro', uri=True) try: - remaining = connection.execute( + return connection.execute( 'SELECT COUNT(*) FROM faults WHERE planned_stop_owned != 0').fetchone()[0] finally: connection.close() - self.assertEqual(0, remaining) @launch_testing.post_shutdown_test() diff --git a/src/ros2_medkit_fault_manager/test/test_fault_manager.cpp b/src/ros2_medkit_fault_manager/test/test_fault_manager.cpp index bfea4acbb..8419e0c61 100644 --- a/src/ros2_medkit_fault_manager/test/test_fault_manager.cpp +++ b/src/ros2_medkit_fault_manager/test/test_fault_manager.cpp @@ -3012,11 +3012,20 @@ class ReleaseOrderRecordingStorage : public ros2_medkit_fault_manager::InMemoryF return InMemoryFaultStorage::get_fault(fault_code); } + // The scoped overload is the one the release actually uses; both are recorded so + // neither can be hidden by overriding only one. + using InMemoryFaultStorage::clear_planned_stop_owned; + size_t clear_planned_stop_owned() override { calls.push_back("clear_owned"); return InMemoryFaultStorage::clear_planned_stop_owned(); } + size_t clear_planned_stop_owned(const std::vector & fault_codes) override { + calls.push_back("clear_owned_scoped"); + return InMemoryFaultStorage::clear_planned_stop_owned(fault_codes); + } + mutable std::vector calls; }; diff --git a/src/ros2_medkit_fault_manager/test/test_sqlite_storage.cpp b/src/ros2_medkit_fault_manager/test/test_sqlite_storage.cpp index bdbba38e9..ed02de85f 100644 --- a/src/ros2_medkit_fault_manager/test/test_sqlite_storage.cpp +++ b/src/ros2_medkit_fault_manager/test/test_sqlite_storage.cpp @@ -2576,6 +2576,43 @@ TEST_F(SqliteFaultStorageTest, ClearingOwnershipDropsEveryFlagAndReportsHowMany) EXPECT_EQ(0u, storage_->clear_planned_stop_owned()); } +TEST_F(SqliteFaultStorageTest, ClearingNamedOwnershipLeavesEveryOtherFlagAlone) { + raise(*storage_, "OWNED_CAPTURED_A"); + raise(*storage_, "OWNED_CAPTURED_B"); + raise(*storage_, "OWNED_LATER"); + for (const auto * code : {"OWNED_CAPTURED_A", "OWNED_CAPTURED_B", "OWNED_LATER"}) { + storage_->set_planned_stop_owned(code, true); + } + + // A release finishes what it captured. Anything that became owned after that + // capture belongs to a different declaration and must survive. + EXPECT_EQ(2u, storage_->clear_planned_stop_owned({"OWNED_CAPTURED_A", "OWNED_CAPTURED_B"})); + + auto owned = storage_->get_planned_stop_owned(); + ASSERT_EQ(1u, owned.size()); + EXPECT_EQ("OWNED_LATER", owned[0]); + + // Idempotent, and a code that is not owned is not an error. + EXPECT_EQ(0u, storage_->clear_planned_stop_owned({"OWNED_CAPTURED_A", "NEVER_OWNED"})); + EXPECT_EQ(1u, storage_->get_planned_stop_owned().size()); + EXPECT_EQ(0u, storage_->clear_planned_stop_owned({})); + EXPECT_EQ(1u, storage_->get_planned_stop_owned().size()); +} + +TEST(InMemoryFaultStorageTest, ClearingNamedOwnershipLeavesEveryOtherFlagAlone) { + ros2_medkit_fault_manager::InMemoryFaultStorage storage; + raise(storage, "OWNED_CAPTURED"); + raise(storage, "OWNED_LATER"); + storage.set_planned_stop_owned("OWNED_CAPTURED", true); + storage.set_planned_stop_owned("OWNED_LATER", true); + + EXPECT_EQ(1u, storage.clear_planned_stop_owned({"OWNED_CAPTURED"})); + + auto owned = storage.get_planned_stop_owned(); + ASSERT_EQ(1u, owned.size()); + EXPECT_EQ("OWNED_LATER", owned[0]); +} + TEST_F(SqliteFaultStorageTest, ADatabaseWrittenBeforeOwnershipExistedStillOpens) { raise(*storage_, "LEGACY_FAULT"); storage_.reset(); From 3af0e3c78e8a02db6857ff4f2d50981356a15c01 Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Fri, 11 Sep 2026 13:56:46 +0200 Subject: [PATCH 11/12] fault_manager: write planned-stop ownership with the report, and let a cluster keep hiding through a switch-off Ownership of a fault cycle is recorded by the report that starts it, inside the same storage transaction as the fault row: the store is told whether a stop is in force and decides the cycle boundary itself, so no state exists in which a cycle that started inside a stop is confirmed and unowned. Both backends carry it; on SQLite a test fails the ownership statement through a trigger and shows the fault row rolled back with it. The separate ownership setter had no production caller and is gone. At the switch-off, a fault an active show_as_single cluster still hides has its stop entry erased and is not announced, and nothing is written in its place: the cluster hides through the verdict it gives every report, so a member that lived through a stop is indistinguishable in the muted list, the counts and the audit log from one that did not. A representative acknowledged after its window closed is promoted from the active cluster's members. A fault that joins a cluster below min_count is not folded into it. An active cluster hides its members until it dissolves; min_count gates formation only. Cluster membership is not persisted, so a stop that spans a restart releases and announces every fault the store owns, and the docs say so. The docs, the service definition and the message reference describe the withdrawal as what is unmuted and what is announced, and name the one trigger that lets a rule mute go. --- docs/api/messages.rst | 23 +- docs/config/fault-manager.rst | 57 ++- src/ros2_medkit_fault_manager/README.md | 90 +++-- .../design/index.rst | 70 +++- .../correlation/correlation_engine.hpp | 20 +- .../fault_storage.hpp | 32 +- .../sqlite_fault_storage.hpp | 17 +- .../src/correlation/correlation_engine.cpp | 170 +++++---- .../src/fault_manager_node.cpp | 24 +- .../src/fault_storage.cpp | 31 +- .../src/sqlite_fault_storage.cpp | 64 ++-- .../integration/test_planned_stop.test.py | 95 ++++- .../test/test_correlation_engine.cpp | 328 +++++++++++++++++- .../test/test_fault_manager.cpp | 75 +++- .../test/test_sqlite_storage.cpp | 194 +++++++++-- src/ros2_medkit_msgs/README.md | 19 +- src/ros2_medkit_msgs/srv/SetPlannedStop.srv | 25 +- 17 files changed, 1111 insertions(+), 223 deletions(-) diff --git a/docs/api/messages.rst b/docs/api/messages.rst index 37d841738..09285bf4f 100644 --- a/docs/api/messages.rst +++ b/docs/api/messages.rst @@ -327,11 +327,24 @@ when ``include_muted`` is set. A cycle that started before the stop is untouched Publication matches a rule-muted symptom exactly: ``EVENT_CONFIRMED`` and ``EVENT_UPDATED`` are withheld whichever kind of report produced them, -``EVENT_CLEARED`` is published as usual. Withdrawing the stop releases every fault -it owns and publishes one ``EVENT_CONFIRMED`` per released fault that is CONFIRMED; -a fault a correlation rule is muting stays muted and is not announced, and returns -to the stop's mute if the rule lets go while the stop is still on. The audit records -exist only when ``audit_log.enabled`` is set, which it is not by default. +``EVENT_CLEARED`` is published as usual. + +Withdrawing the stop does two things that do not cover the same faults. It *unmutes* +every fault it owns whose entry is the stop's own - a fault a hierarchical rule has +since claimed stays muted, and returns to the stop's mute if that rule's root cause +is acknowledged while the stop is still on. It *announces* the subset of those that +is CONFIRMED and that no live cluster is hiding, one ``EVENT_CONFIRMED`` each. + +An auto-cluster rule hides a non-representative member without an entry of its own - +it suppresses that member's events on each report - so such a member is unmuted and +NOT announced, with nothing written in the stop's place. Afterwards the burst matches +one that never met a planned stop in ``muted_faults``, in the counts, in the cluster +listing and in the audit log, but not in the event stream: a confirmation that fell +inside the stop and behind a cluster is never announced. What is announced is the +representative, when the stop owned its cycle. Cluster membership is not persisted, +so a stop that spanned a restart announces every fault the store says it owns. The +audit records exist only when ``audit_log.enabled`` is set, which it is not by +default. See :doc:`/config/fault-manager` for the configuration that decides whether the declaration survives a restart and whether the transitions are audited. diff --git a/docs/config/fault-manager.rst b/docs/config/fault-manager.rst index 390ed2558..a18381aac 100644 --- a/docs/config/fault-manager.rst +++ b/docs/config/fault-manager.rst @@ -666,17 +666,60 @@ when muted entries are requested. A fault that was already up when the stop was declared is left alone - its confirmation has already been announced, and reporters re-send FAILED for as long as the condition holds. -Ownership is a flag on the fault row, so it survives a restart and is what the -switch-off releases. The mute is derived from it: a correlation rule muting an owned -fault overlays the stop rather than taking the fault from it, the withdrawal leaves -a rule-held fault alone, and when the rule lets go - root cause acknowledged, window -closed, cluster expired - the fault is muted by the stop again. +Ownership is a flag on the fault row, written by the report that starts the cycle +and in the same transaction, so it survives a restart. While the manager runs, the +engine holds its own copy of that set and the switch-off works from it; the stored +flags are what a startup reads back, and they are dropped once the switch-off has +announced what it released. The mute is derived from ownership: a hierarchical rule +muting an owned fault overlays the stop rather than taking the fault from it, the +withdrawal leaves a rule-held fault alone, and when the rule lets go - which means +its root cause being acknowledged, the one path that drops a symptom's entry - the +fault is muted by the stop again. A rule whose window has closed keeps its entry +until then. + +A rule outranks the stop at the switch-off as well. A fault an active +``show_as_single`` cluster still hides is not announced, and nothing is written in +the stop's place: a cluster hides a member by suppressing that member's events on +every report, never by an entry in the muted list. + +Afterwards the burst is indistinguishable from one that never met a planned stop in +the muted list, in ``muted_count``, in the cluster listing and in the audit log. It +is NOT indistinguishable in the event stream, which is the point of the switch: a +confirmation that fell inside the stop and behind a cluster is never announced, +where the same fault outside a stop would have been announced if it confirmed before +its cluster reached ``min_count``. What the stop withheld, it withholds for good. + +The representative is announced if the stop owned its cycle; one whose cycle +predates the stop was announced when it confirmed and is not announced again; +members that joined the cluster during the stop stay hidden by it afterwards, and +the one promoted when the representative is acknowledged is heard from again from +that point on. A fault that joins a cluster whose membership has since fallen below +``min_count`` is not one of the members that cluster shows as a single line: it is +announced and updated like any fault of its own. + +``min_count`` gates whether a cluster FORMS, not how long it hides: once formed, a +cluster folds its members into the representative until the last of them is +acknowledged and it dissolves, so a burst that shrinks back below the threshold does +not start announcing its members again. A cluster that never reached ``min_count``, +or one configured without ``show_as_single``, hides nobody and its members are +released as usual. + +The cluster hold does not survive a restart. Ownership is persisted and cluster +membership is not, so a cluster only holds faults it formed from reports the running +process saw. If the stop spanned a reboot, the switch-off releases and announces +every fault the store says the stop owns, cluster or no cluster. A muted fault is published exactly as a rule-muted symptom is: ``EVENT_CONFIRMED`` and ``EVENT_UPDATED`` are withheld whichever kind of report produced them, while ``EVENT_CLEARED`` - the fault healing, or being acknowledged - is published as -usual. Withdrawing the stop releases every fault it owns and publishes one -confirmation event for each of those that is CONFIRMED. +usual. + +Withdrawing the stop does two things that do not cover the same faults. It *unmutes* +every fault the stop owns whose muted-list entry is the stop's own, so a fault a +hierarchical rule has since claimed stays muted and everything else leaves +``muted_count``. It *announces* the subset of those that is CONFIRMED and that no +live cluster is hiding, one ``EVENT_CONFIRMED`` each. A cluster-hidden member is +therefore unmuted and not announced. ``~/get_planned_stop`` keeps serving the declaration after the withdrawal, with ``ended_at`` stamped, so the reason stays readable once the plant is back up. A diff --git a/src/ros2_medkit_fault_manager/README.md b/src/ros2_medkit_fault_manager/README.md index c681eff58..bfdc45312 100644 --- a/src/ros2_medkit_fault_manager/README.md +++ b/src/ros2_medkit_fault_manager/README.md @@ -68,12 +68,21 @@ the pseudo root cause `PLANNED_STOP`, so it is absent from the default `~/list_faults` response, counted in `muted_count`, and listed under `muted_faults` when `include_muted` is set. -**Ownership is the truth, and it is persisted.** The flag lives on the fault row in -the store, not in the process, so a restart reads back exactly which cycles the -stop owns - no timestamp comparison, which a clock step would break and which -cannot tell a rule's mute from the stop's. The mute is derived from ownership: an -owned fault is muted unless a rule's mute overlays it, and when that overlay ends -the fault is muted by the stop again. +**Ownership is the truth, and it is persisted.** A flag on the fault row records it, +so a restart reads back exactly which cycles the stop owns - no timestamp +comparison, which a clock step would break and which cannot tell a rule's mute from +the stop's. It is written by the report that starts the cycle, in the same store +call and the same transaction, so there is no moment at which a cycle that STARTED +inside a stop exists unowned - a moment a crash would turn into a confirmed fault +that no switch-off releases and no restart recognises. (The cycle a fault was +already in when the stop was declared is never owned; the fault itself becomes owned +if it heals and fails again inside the stop. See below.) The mute is derived from +ownership: an owned fault is muted unless a rule's mute overlays it, and when that +overlay ends the fault is muted by the stop again. + +While the process runs, the engine keeps the same set in memory and the switch-off +works from it. The stored flags are the durable copy: they are what a startup reads +back, and they are dropped once the switch-off has announced what it released. **Only a cycle that starts inside the stop.** A fault that was already up when the stop was declared keeps its place in the fault list. Reporters are level-triggered @@ -93,20 +102,61 @@ that is acknowledged, publishes it as any muted fault does. Consumers therefore the end of a fault whose start they never heard - that is the existing muting contract, not something the switch changes. -**Withdrawing releases the survivors.** Every fault the stop alone was muting and -that is still active is unmuted, and each one that is CONFIRMED publishes a single -`EVENT_CONFIRMED` - that confirmation happened behind the mute and was never -announced, while the condition it reports still stands on the machine. A fault -still short of confirmation announces nothing (there is nothing yet to announce), -and one acknowledged during the stop announces nothing either. - -**Correlation rules and the stop compose.** A rule's mute is an *overlay* on an -owned fault, not a transfer: while the rule holds it, the withdrawal leaves it -alone, and when the rule lets go - its root cause acknowledged, its window closed, -its cluster expired - the fault goes back to being muted by the stop. A fault whose -cycle started before the stop is not owned at all, so the stop neither hides it nor -releases it. This holds for cluster rules as well as hierarchical ones, even though -the cluster path never writes a mute entry of its own. +**Withdrawing unmutes, and separately announces.** Two different things happen, and +they do not cover the same faults. + +*Unmuted* is every fault the stop owns whose entry in the muted list is the stop's +own. A fault a hierarchical rule has since claimed keeps that rule's entry and stays +muted; everything else the stop owned leaves the muted list, `muted_count` included. + +*Announced* is the subset of those that is CONFIRMED and that no live cluster is +hiding: each publishes a single `EVENT_CONFIRMED`, because that confirmation +happened behind the mute and was never heard, while the condition it reports still +stands on the machine. A fault still short of confirmation announces nothing (there +is nothing yet to announce), and one acknowledged during the stop announces nothing +either. A cluster-hidden member is therefore unmuted and NOT announced - it leaves +the muted list and goes on being folded into its cluster's line. + +**Correlation rules and the stop compose, and the rule outranks it.** A hierarchical +rule's mute is an *overlay* on an owned fault, not a transfer: while the rule holds +it, the withdrawal leaves it alone, and when the rule lets go - which means its root +cause being acknowledged, the one path that drops a symptom's entry - the fault goes +back to being muted by the stop. A rule whose window has closed keeps its entry +until then. A fault whose cycle started before the stop is not owned at all, so the +stop neither hides it nor releases it. + +Cluster rules take the same precedence, and they take it without an entry of their +own: a cluster hides a member by suppressing that member's events on every report, +not by writing it into the muted list. So at the switch-off a member an active +`show_as_single` cluster still hides is simply not announced - the stop's entry +goes and nothing is written in its place. Releasing them all would fire, in one +wave, the storm the rule exists to show as a single line. + +Afterwards the burst is indistinguishable from one that never met a planned stop in +the muted list, in `muted_count`, in the cluster listing and in the audit log. It is +NOT indistinguishable in the event stream, and that is the point of the switch: a +confirmation that happened inside the stop and behind a cluster is never announced, +where the same fault outside a stop would have been announced if it confirmed before +its cluster reached `min_count`. What the stop withheld, it withholds for good. + +Which member is announced at the switch-off: the representative, if the stop owned +its cycle. A representative whose cycle predates the stop was announced when it +confirmed and is not announced again. Members that joined an active cluster during +the stop stay hidden by it afterwards, and the one promoted when the representative +is acknowledged is heard from again from that point on. + +`min_count` gates whether a cluster FORMS, not how long it hides. Once formed, a +cluster folds its members into the representative until the last of them is +acknowledged and the cluster dissolves - a burst that shrinks back below the +threshold does not start announcing its members again. A cluster that never reached +`min_count`, or one configured without `show_as_single`, hides nobody, so every +member of it is released as usual. + +**The cluster hold does not survive a restart.** Ownership is persisted; cluster +membership is not. A cluster only holds faults it formed from reports this process +saw, so if the stop spanned a reboot the switch-off releases and announces every +fault the store says the stop owns, cluster or no cluster. Persisting cluster state +is not part of this. **Both transitions are audited** - when the audit log is on. `audit_log.enabled` is `false` by default, and with it off the switch writes no audit row at all. With diff --git a/src/ros2_medkit_fault_manager/design/index.rst b/src/ros2_medkit_fault_manager/design/index.rst index fd2a3d7f9..fd46615e9 100644 --- a/src/ros2_medkit_fault_manager/design/index.rst +++ b/src/ros2_medkit_fault_manager/design/index.rst @@ -284,11 +284,31 @@ publishes ``EVENT_CLEARED`` at the heal and the confirmation that follows is therefore fresh news. ``occurrence_count`` keeps its own, narrower definition. **Ownership is the fact; the mute is derived from it.** The stop owns a fault -CYCLE, and that ownership is a flag on the fault row in the store - not a set in the -process, and not a timestamp comparison. Everything else follows: an owned fault is +CYCLE, and the durable record of that is a flag on the fault row in the store - +not a timestamp comparison. The report that starts the cycle carries it: the store +is told whether a stop is in force, decides inside the same transaction whether +this report started a cycle, and writes the flag with the row. Recorded by a call +that follows the report instead, there is a window in which a fault has confirmed +inside a stop and is not owned, and a process that dies there comes back to a +confirmed fault that no switch-off releases, that no startup recognises as +interrupted, and whose next report announces an update mid-stop. + +The engine keeps the same set in the process (``planned_stop_owned_``) and that is +what a running switch-off works from; the stored flags are what a STARTUP reads +back, and ``clear_planned_stop_owned`` drops them once the switch-off has announced +what it released. Several paths move each of them - the report takes a cycle, the +constructor restores from the store, an acknowledgement and the switch-off give it +up - and they stay in step because every one of them runs on the node's single +thread: ``rclcpp::spin()`` in ``main.cpp``, no callback group off it, so no two of +them are ever in flight at once. + +Everything else follows: an owned fault is muted unless a rule's mute overlays it, and the moment the overlay ends the engine -re-asserts the stop's mute (``reassert_planned_stop_mutes``, called from -``process_clear`` and from ``cleanup_expired``). A rule therefore borrows a fault +re-asserts the stop's mute (``reassert_planned_stop_mutes``). An overlay ends in one +place only - ``process_clear`` on the root cause, which erases its symptoms' entries. +A window closing does not: ``cleanup_expired`` drops pending root causes and pending +clusters and never touches ``muted_faults_``, so it calls the re-assert to cover the +ownership set rather than to hand anything back. A rule therefore borrows a fault rather than taking it, which is what makes the two features compose in both directions: the withdrawal leaves a rule-held fault muted, and a rule that lets go mid-stop hands the fault back instead of dropping it out of the stop for good. @@ -340,6 +360,48 @@ What is withheld is exactly what a rule-muted symptom withholds - ``EVENT_CONFIR and ``EVENT_UPDATED`` - while ``EVENT_CLEARED`` is published as usual, so a fault that heals or is acknowledged inside the stop still reports its end. +**A rule outranks the stop at the switch-off, clusters included.** A hierarchical +rule holds its symptom through the withdrawal because its mute entry is there to +see. A cluster's ``show_as_single`` is different: it sets ``should_mute`` on the +report and writes no entry, so an owned member carries the stop's entry and would be +released with everything else - announcing, in one wave, exactly the storm the rule +exists to fold into a single alarm. The release therefore asks the cluster the same +question the report path asks (``cluster_hides``: an ACTIVE cluster, a +``show_as_single`` rule, and not the representative) and, when the answer is yes, +erases the stop's entry without announcing the fault and without writing anything in +its place. + +A cluster-attributed entry there would be wrong in two ways. It is a remembered +answer to a question whose inputs keep moving - +membership and the representative both change - so acknowledging the representative +afterwards leaves the promoted member hidden behind an entry naming a fault that is +gone, with nothing but clearing it to remove the entry. And it makes ``muted_count`` +and the default fault list depend on whether a stop happened to be in force earlier, +for a cluster that is otherwise identical, because a cluster writes no entry at any +other time. ``cluster_hides`` is a verdict, asked fresh on every report and once +more at the release; it is never stored. + +Ownership ends either way. What is announced is the representative, if the stop +owned its cycle. ``min_count`` gates whether a cluster FORMS, not how long it hides: +a formed cluster holds its members until the last one is acknowledged and it +dissolves, so a burst that shrinks below the threshold keeps hiding. A cluster that +never reached ``min_count``, or one whose rule groups without hiding, has no verdict +to outrank the stop with, and every member of it is released. + +Promotion has to read the ACTIVE cluster, not the pending twin. ``cleanup_expired`` +drops the pending cluster when its window closes and keeps the active one, so once +the window is behind the burst the active cluster is the only record of who is left. +Promoting from a twin that is no longer there leaves ``representative_code`` naming +the acknowledged fault, and every remaining member is then hidden by a cluster whose +representative can never be reported again - at the switch-off nobody is released. +The member severities therefore live on ``ClusterData`` rather than beside the +pending cluster, so a highest-severity rule can still name a replacement. + +**The cluster hold does not survive a restart.** Ownership is persisted; cluster +membership is not, and persisting it is a separate feature this does not build. A +cluster only holds faults it formed from reports the running process saw, so a stop +that spanned a reboot releases and announces every fault the store says it owns. + The switch is a service rather than a parameter because it carries a reason and a declarer and produces an audit record, none of which a parameter can do; and it is not a new HTTP route because the gateway already exposes a node's services as SOVD diff --git a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/correlation/correlation_engine.hpp b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/correlation/correlation_engine.hpp index c31eafd1f..268979890 100644 --- a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/correlation/correlation_engine.hpp +++ b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/correlation/correlation_engine.hpp @@ -83,6 +83,10 @@ struct ClusterData { std::string representative_code; std::string representative_severity; std::vector fault_codes; + /// Severity of each member, keyed by fault code. Carried on the cluster rather than + /// beside it so a cluster that outlives the window it formed in can still name the + /// member a highest-severity rule would promote. + std::map fault_severities; std::chrono::system_clock::time_point first_at; std::chrono::system_clock::time_point last_at; }; @@ -203,6 +207,21 @@ class CorrelationEngine { /// Write the planned stop's mute entry for one owned fault. Caller holds mutex_. void mute_as_planned_stop(const std::string & fault_code); + /// Whether an ACTIVE cluster is currently folding this fault into its representative: + /// the fault belongs to a cluster that reached min_count, the rule says + /// `show_as_single`, and the fault is not the representative. A verdict, not an entry - + /// the cluster path writes nothing to `muted_faults_`, it suppresses the fault's event + /// on each report. The single place that decides it, so the report path and the + /// planned stop's switch-off cannot answer differently. Caller holds mutex_. + bool cluster_hides(const std::string & fault_code) const; + + /// The configured rule with this id, or nullptr. Caller holds mutex_. + const CorrelationRule * find_rule(const std::string & rule_id) const; + + /// Name the member the cluster's rule would pick as representative, from the members + /// the cluster currently has. A no-op on an empty cluster. Caller holds mutex_. + void promote_representative(ClusterData & cluster); + /// Check if fault matches a root cause pattern in any hierarchical rule /// @return Rule ID if matched, empty optional otherwise std::optional try_as_root_cause(const std::string & fault_code); @@ -248,7 +267,6 @@ class CorrelationEngine { struct PendingCluster { ClusterData data; std::chrono::steady_clock::time_point steady_first_at; - std::map fault_severities; ///< fault_code -> severity }; /// Pending clusters being formed (rule_id -> cluster data) diff --git a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_storage.hpp b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_storage.hpp index 5933cc9f8..4399cb0e4 100644 --- a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_storage.hpp +++ b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_storage.hpp @@ -235,11 +235,21 @@ class FaultStorage { /// @param source_id Reporting source identifier /// @param timestamp Current time for tracking /// @param config Debounce configuration to apply for this event (resolved per-entity by the node) + /// @param planned_stop_active Whether a planned stop is declared. When it is, a report that STARTS + /// a fault cycle - a new fault, one raised again after being cleared, or one that fails + /// again after healing - marks that cycle as the stop's, in the same write as the report. + /// Ownership recorded by a separate call afterwards has a window in which a confirmed + /// fault exists unowned, and a process that dies there comes back to a fault no + /// switch-off releases. A report that does not start a cycle never takes ownership and + /// never drops it. + /// The default is repeated on both overrides so a call on a concrete backend means the + /// same as one through this interface. /// @return true if this is a new occurrence (new fault or reactivated CLEARED fault), /// false if existing active fault was updated virtual bool report_fault_event(const std::string & fault_code, uint8_t event_type, uint8_t severity, const std::string & description, const std::string & source_id, - const rclcpp::Time & timestamp, const DebounceConfig & config) = 0; + const rclcpp::Time & timestamp, const DebounceConfig & config, + bool planned_stop_active = false) = 0; /// Get faults matching filter criteria /// @param filter_by_severity Whether to filter by severity @@ -484,14 +494,14 @@ class FaultStorage { /// given one answers with a default-constructed (inactive) state. virtual PlannedStopState get_planned_stop() const = 0; - /// Record, or withdraw, the planned stop's ownership of one fault cycle. - /// A code with no stored fault is ignored. - virtual void set_planned_stop_owned(const std::string & fault_code, bool owned) = 0; - - /// Every fault the planned stop currently owns. This is what a restart reads to - /// rebuild the mute, and what a switch-off releases - not a time comparison - /// against the declaration, which cannot survive a clock step and cannot tell a - /// rule's mute from the stop's. + /// Every fault the planned stop currently owns, as the store has it. This is the + /// durable record: a STARTUP reads it to rebuild the mute in a process that never + /// saw the reports, and a startup that finds faults owned by a declaration already + /// withdrawn finishes that interrupted release from it. A switch-off in a running + /// manager releases from the engine's own copy of the set and only drops the flags + /// here afterwards. Ownership is recorded rather than derived from a time + /// comparison against the declaration, which cannot survive a clock step and cannot + /// tell a rule's mute from the stop's. virtual std::vector get_planned_stop_owned() const = 0; /// Drop every ownership flag. Called once the switch-off has announced what it @@ -527,7 +537,8 @@ class InMemoryFaultStorage : public FaultStorage { bool report_fault_event(const std::string & fault_code, uint8_t event_type, uint8_t severity, const std::string & description, const std::string & source_id, - const rclcpp::Time & timestamp, const DebounceConfig & config) override; + const rclcpp::Time & timestamp, const DebounceConfig & config, + bool planned_stop_active = false) override; std::vector list_faults(bool filter_by_severity, uint8_t severity, const std::vector & statuses) const override; @@ -577,7 +588,6 @@ class InMemoryFaultStorage : public FaultStorage { void set_planned_stop(const PlannedStopState & state) override; PlannedStopState get_planned_stop() const override; - void set_planned_stop_owned(const std::string & fault_code, bool owned) override; std::vector get_planned_stop_owned() const override; size_t clear_planned_stop_owned() override; size_t clear_planned_stop_owned(const std::vector & fault_codes) override; diff --git a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/sqlite_fault_storage.hpp b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/sqlite_fault_storage.hpp index ad892a735..3015b147e 100644 --- a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/sqlite_fault_storage.hpp +++ b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/sqlite_fault_storage.hpp @@ -48,7 +48,8 @@ class SqliteFaultStorage : public FaultStorage { bool report_fault_event(const std::string & fault_code, uint8_t event_type, uint8_t severity, const std::string & description, const std::string & source_id, - const rclcpp::Time & timestamp, const DebounceConfig & config) override; + const rclcpp::Time & timestamp, const DebounceConfig & config, + bool planned_stop_active = false) override; std::vector list_faults(bool filter_by_severity, uint8_t severity, const std::vector & statuses) const override; @@ -97,7 +98,6 @@ class SqliteFaultStorage : public FaultStorage { void set_planned_stop(const PlannedStopState & state) override; PlannedStopState get_planned_stop() const override; - void set_planned_stop_owned(const std::string & fault_code, bool owned) override; std::vector get_planned_stop_owned() const override; size_t clear_planned_stop_owned() override; size_t clear_planned_stop_owned(const std::vector & fault_codes) override; @@ -141,11 +141,18 @@ class SqliteFaultStorage : public FaultStorage { /// still says nobody holds it. std::vector store_rosbag_file_locked(const RosbagFileInfo & info); - /// report_fault_event body without taking mutex_ or opening a transaction. Caller holds mutex_ - /// and wraps the call, so the fault row and any near-miss row commit together. + /// report_fault_event body without taking mutex_ or opening a transaction. Caller holds mutex_. + /// A FAILED report is the one the caller wraps in a transaction, so the fault row, any near-miss + /// row and the planned stop's ownership of the cycle commit together; a PASSED report writes at + /// most one row, starts no cycle, and runs in autocommit. bool report_fault_event_locked(const std::string & fault_code, uint8_t event_type, uint8_t severity, const std::string & description, const std::string & source_id, - const rclcpp::Time & timestamp, const DebounceConfig & config); + const rclcpp::Time & timestamp, const DebounceConfig & config, + bool planned_stop_active); + + /// Mark the fault's current cycle as the planned stop's. Caller holds mutex_ and has already + /// written the fault row inside the same transaction. + void mark_planned_stop_owned_locked(const std::string & fault_code); /// Append one entry to the near-miss series and evict the oldest entries beyond /// max_near_misses_per_fault_. Caller holds mutex_ and has already written the fault row. diff --git a/src/ros2_medkit_fault_manager/src/correlation/correlation_engine.cpp b/src/ros2_medkit_fault_manager/src/correlation/correlation_engine.cpp index 26ba47b33..9ae01d2eb 100644 --- a/src/ros2_medkit_fault_manager/src/correlation/correlation_engine.cpp +++ b/src/ros2_medkit_fault_manager/src/correlation/correlation_engine.cpp @@ -177,7 +177,7 @@ ProcessClearResult CorrelationEngine::process_clear(const std::string & fault_co auto & codes = pending_cluster.fault_codes; codes.erase(std::remove(codes.begin(), codes.end(), fault_code), codes.end()); - pending_it->second.fault_severities.erase(fault_code); + pending_cluster.fault_severities.erase(fault_code); if (codes.empty()) { pending_it = pending_clusters_.erase(pending_it); @@ -186,46 +186,7 @@ ProcessClearResult CorrelationEngine::process_clear(const std::string & fault_co // Reassign representative if the cleared fault was the representative if (pending_cluster.representative_code == fault_code) { - for (const auto & rule : config_.rules) { - if (rule.id == pending_it->first) { - switch (rule.representative) { - case Representative::FIRST: { - auto & sevs = pending_it->second.fault_severities; - const std::string & first_code = codes.front(); - pending_cluster.representative_code = first_code; - auto sev_it = sevs.find(first_code); - pending_cluster.representative_severity = (sev_it != sevs.end()) ? sev_it->second : ""; - break; - } - case Representative::HIGHEST_SEVERITY: { - auto & sevs = pending_it->second.fault_severities; - std::string best_code = codes.front(); - int best_rank = -1; - for (const auto & code : codes) { - auto sev_it = sevs.find(code); - int rank = (sev_it != sevs.end()) ? severity_rank(sev_it->second) : 0; - if (rank > best_rank) { - best_rank = rank; - best_code = code; - } - } - pending_cluster.representative_code = best_code; - auto best_sev_it = sevs.find(best_code); - pending_cluster.representative_severity = (best_sev_it != sevs.end()) ? best_sev_it->second : ""; - break; - } - case Representative::MOST_RECENT: { - auto & sevs = pending_it->second.fault_severities; - const std::string & most_recent_code = codes.back(); - pending_cluster.representative_code = most_recent_code; - auto sev_it = sevs.find(most_recent_code); - pending_cluster.representative_severity = (sev_it != sevs.end()) ? sev_it->second : ""; - break; - } - } - break; - } - } + promote_representative(pending_cluster); } ++pending_it; @@ -237,18 +198,18 @@ ProcessClearResult CorrelationEngine::process_clear(const std::string & fault_co auto & active_cluster = active_it->second; auto & codes = active_cluster.fault_codes; codes.erase(std::remove(codes.begin(), codes.end(), fault_code), codes.end()); + active_cluster.fault_severities.erase(fault_code); if (codes.empty()) { active_clusters_.erase(active_it); } else if (active_cluster.representative_code == fault_code) { - // Sync representative from pending cluster (already updated above) - for (const auto & [rule_id, pending] : pending_clusters_) { - if (pending.data.cluster_id == cluster_id) { - active_cluster.representative_code = pending.data.representative_code; - active_cluster.representative_severity = pending.data.representative_severity; - break; - } - } + // Promoted from the ACTIVE cluster's own members rather than copied from the + // pending twin, because the twin is gone once its window closes + // (cleanup_expired drops it while the active cluster stays). Copied from a + // twin that is not there, the representative keeps naming the acknowledged + // fault, and every remaining member is then hidden by a cluster whose + // representative can never be reported again. + promote_representative(active_cluster); } } @@ -287,8 +248,23 @@ std::vector CorrelationEngine::end_planned_stop() { auto it = muted_faults_.find(fault_code); // A rule's overlay is not the stop's to lift: that fault stays muted, and // stays unannounced, for as long as the rule holds it. - if (it != muted_faults_.end() && it->second.by_planned_stop) { - muted_faults_.erase(it); + if (it == muted_faults_.end() || !it->second.by_planned_stop) { + continue; + } + + // The stop's entry goes whatever else is true: the cycle it owned is over. + const bool held_by_cluster = cluster_hides(fault_code); + muted_faults_.erase(it); + + // A cluster rule outranks the stop, because it is the narrower promise: the + // stop says "not now", the cluster says "this burst is one line, ever". + // Announcing every owned member would fire, in one wave, exactly the storm + // the rule exists to fold into a single alarm. Nothing is written in the + // stop's place: the cluster suppresses the fault's events through the verdict + // it reaches on each report, which is how it hides a member with no stop in + // force, and a remembered copy of that verdict would outlive the membership + // and the representative it was taken from. + if (!held_by_cluster) { released.push_back(fault_code); } } @@ -297,6 +273,85 @@ std::vector CorrelationEngine::end_planned_stop() { return released; } +bool CorrelationEngine::cluster_hides(const std::string & fault_code) const { + auto cluster_it = fault_to_cluster_.find(fault_code); + if (cluster_it == fault_to_cluster_.end()) { + return false; + } + + // An ACTIVE cluster is one that reached min_count. min_count gates FORMATION, not + // the hiding: once formed, the cluster folds its members into the representative + // until every one of them is acknowledged and the cluster dissolves. A cluster that + // never formed is a handful of separate faults matching the same pattern. + auto active_it = active_clusters_.find(cluster_it->second); + if (active_it == active_clusters_.end()) { + return false; + } + + const ClusterData & cluster = active_it->second; + if (fault_code == cluster.representative_code) { + return false; // the representative is the line the cluster shows + } + + // Membership is read from the ACTIVE cluster, not from fault_to_cluster_, which is + // written on every join including one to a pending twin that is below min_count - + // active_clusters_ is only refreshed when the burst reaches the threshold. A fault + // that joined a twin two members short maps to a formed cluster it is not part of; + // it is a fault of its own, announced and updated like any other. + if (std::find(cluster.fault_codes.begin(), cluster.fault_codes.end(), fault_code) == cluster.fault_codes.end()) { + return false; + } + + const CorrelationRule * rule = find_rule(cluster.rule_id); + return rule != nullptr && rule->show_as_single; +} + +const CorrelationRule * CorrelationEngine::find_rule(const std::string & rule_id) const { + for (const auto & rule : config_.rules) { + if (rule.id == rule_id) { + return &rule; + } + } + return nullptr; +} + +void CorrelationEngine::promote_representative(ClusterData & cluster) { + if (cluster.fault_codes.empty()) { + return; + } + + const CorrelationRule * rule = find_rule(cluster.rule_id); + const Representative policy = rule != nullptr ? rule->representative : Representative::FIRST; + const auto & severities = cluster.fault_severities; + auto severity_of = [&severities](const std::string & code) { + auto it = severities.find(code); + return it != severities.end() ? it->second : std::string{}; + }; + + std::string promoted = cluster.fault_codes.front(); + switch (policy) { + case Representative::FIRST: + break; + case Representative::MOST_RECENT: + promoted = cluster.fault_codes.back(); + break; + case Representative::HIGHEST_SEVERITY: { + int best_rank = -1; + for (const auto & code : cluster.fault_codes) { + const int rank = severity_rank(severity_of(code)); + if (rank > best_rank) { + best_rank = rank; + promoted = code; + } + } + break; + } + } + + cluster.representative_code = promoted; + cluster.representative_severity = severity_of(promoted); +} + bool CorrelationEngine::planned_stop_active() const { std::lock_guard lock(mutex_); return planned_stop_active_; @@ -558,7 +613,7 @@ std::optional CorrelationEngine::try_auto_cluster(const std: pending.data.representative_code = fault_code; pending.data.representative_severity = severity; pending.data.fault_codes.push_back(fault_code); - pending.fault_severities[fault_code] = severity; + pending.data.fault_severities[fault_code] = severity; pending.data.first_at = now_system; pending.data.last_at = now_system; @@ -581,15 +636,12 @@ std::optional CorrelationEngine::try_auto_cluster(const std: // Already in cluster - ensure consistent muting for duplicates ProcessFaultResult result; result.cluster_id = cluster.cluster_id; - if (rule.show_as_single && fault_code != cluster.representative_code && - cluster.fault_codes.size() >= rule.min_count) { - result.should_mute = true; - } + result.should_mute = cluster_hides(fault_code); return result; } cluster.fault_codes.push_back(fault_code); - pending.fault_severities[fault_code] = severity; + cluster.fault_severities[fault_code] = severity; cluster.last_at = now_system; fault_to_cluster_[fault_code] = cluster.cluster_id; @@ -640,9 +692,7 @@ std::optional CorrelationEngine::try_auto_cluster(const std: } // Mute non-representative faults - if (rule.show_as_single && fault_code != cluster.representative_code) { - result.should_mute = true; - } + result.should_mute = cluster_hides(fault_code); } return result; diff --git a/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp b/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp index 2c9b7596c..262e81a98 100644 --- a/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp +++ b/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp @@ -908,10 +908,22 @@ void FaultManagerNode::handle_report_fault( // TODO(#276): warn when different entities resolve different configs for the same fault_code auto resolved_config = resolve_config(request->source_id); - // Report the fault event (use wall clock time, not sim time, for proper timestamps) + // Report the fault event (use wall clock time, not sim time, for proper timestamps). + // + // Ownership is persisted, not inferred: the switch-off, and a restart, both read it back from + // the store rather than comparing timestamps. The switch travels WITH the report, so the store + // writes the flag inside the transaction that writes the fault row. + // + // The cycle boundary is therefore decided twice: in the store for the persisted flag, and again + // below for the engine's in-process set. Both read the same two facts - whether the report was a + // new occurrence, and the status the fault held before it - so they are one formula evaluated in + // two places. They cannot drift apart because this handler is the only writer of either, and the + // node's services run one at a time on the single-threaded executor rclcpp::spin() gives it + // (main.cpp), with no callback group taking any of them off that thread. const rclcpp::Time event_time = get_wall_clock_time(); - bool is_new = storage_->report_fault_event(request->fault_code, request->event_type, request->severity, - request->description, request->source_id, event_time, resolved_config); + bool is_new = + storage_->report_fault_event(request->fault_code, request->event_type, request->severity, request->description, + request->source_id, event_time, resolved_config, planned_stop_.active); response->accepted = true; @@ -937,12 +949,6 @@ void FaultManagerNode::handle_report_fault( should_mute = correlation_result.should_mute; - // Ownership is persisted, not inferred: the switch-off, and a restart, both - // read it back from the store rather than comparing timestamps. - if (planned_stop_.active && cycle_started) { - storage_->set_planned_stop_owned(request->fault_code, true); - } - if (correlation_result.is_root_cause) { RCLCPP_DEBUG(get_logger(), "Fault %s identified as root cause (rule=%s)", request->fault_code.c_str(), correlation_result.rule_id.c_str()); diff --git a/src/ros2_medkit_fault_manager/src/fault_storage.cpp b/src/ros2_medkit_fault_manager/src/fault_storage.cpp index b29f4830a..1382c6175 100644 --- a/src/ros2_medkit_fault_manager/src/fault_storage.cpp +++ b/src/ros2_medkit_fault_manager/src/fault_storage.cpp @@ -121,7 +121,8 @@ void InMemoryFaultStorage::update_status(FaultState & state, const DebounceConfi bool InMemoryFaultStorage::report_fault_event(const std::string & fault_code, uint8_t event_type, uint8_t severity, const std::string & description, const std::string & source_id, - const rclcpp::Time & timestamp, const DebounceConfig & config) { + const rclcpp::Time & timestamp, const DebounceConfig & config, + bool planned_stop_active) { std::lock_guard lock(mutex_); const bool is_failed = (event_type == EventType::EVENT_FAILED); @@ -156,6 +157,13 @@ bool InMemoryFaultStorage::report_fault_event(const std::string & fault_code, ui record_near_miss(state, config, severity, source_id, timestamp); } + // A new fault is always the start of a cycle. The flag is set with the rest of the state, + // under the one lock, because the two backends have to make the same fault owned by the same + // report: which faults a switch-off releases must not depend on which store a deployment + // picked. Nothing here outlives the process, so this is parity with the durable backend's + // transaction rather than a crash window being closed. + state.planned_stop_owned = planned_stop_active; + faults_.emplace(fault_code, std::move(state)); return true; } @@ -194,6 +202,10 @@ bool InMemoryFaultStorage::report_fault_event(const std::string & fault_code, ui if (is_near_miss(true, state.status)) { record_near_miss(state, config, severity, source_id, timestamp); } + // A fault raised again after being cleared starts a new cycle. + if (planned_stop_active) { + state.planned_stop_owned = true; + } return true; // Reactivation treated as new occurrence for event publishing } @@ -205,6 +217,15 @@ bool InMemoryFaultStorage::report_fault_event(const std::string & fault_code, ui state.debounce_counter = clamp_debounce_counter(state.debounce_counter, config); if (is_failed) { + // A failure out of HEALED starts a cycle: the heal published the fault's end, so the + // confirmation that follows is fresh news. A repeat report of a condition that is already up + // is the same cycle and takes nothing, so no report ever lowers the flag. It is lowered + // elsewhere: by acknowledgement (clear_fault), by the healed-to-cleared reclassification, and + // by the switch-off (clear_planned_stop_owned). + if (planned_stop_active && state.status == ros2_medkit_msgs::msg::Fault::STATUS_HEALED) { + state.planned_stop_owned = true; + } + // last_occurred tracks occurrences only. A PASSED event is the fault ENDING, not // occurring; bumping it there makes a long-stale CONFIRMED fault look freshly // active to operators. The PASSED instant is kept in last_passed_time. @@ -879,14 +900,6 @@ std::vector InMemoryFaultStorage::reclassify_healed_as_cleared() { return reclassified; } -void InMemoryFaultStorage::set_planned_stop_owned(const std::string & fault_code, bool owned) { - std::lock_guard lock(mutex_); - auto it = faults_.find(fault_code); - if (it != faults_.end()) { - it->second.planned_stop_owned = owned; - } -} - std::vector InMemoryFaultStorage::get_planned_stop_owned() const { std::lock_guard lock(mutex_); std::vector owned; diff --git a/src/ros2_medkit_fault_manager/src/sqlite_fault_storage.cpp b/src/ros2_medkit_fault_manager/src/sqlite_fault_storage.cpp index 4e926400c..0523e9168 100644 --- a/src/ros2_medkit_fault_manager/src/sqlite_fault_storage.cpp +++ b/src/ros2_medkit_fault_manager/src/sqlite_fault_storage.cpp @@ -731,26 +731,30 @@ std::string SqliteFaultStorage::serialize_json_array(const std::vector lock(mutex_); - // Only a FAILED report can write two rows, and only those two have to land together: written as - // separate autocommit statements, a failure on the second would leave the debounce counter + // Only a FAILED report can write more than one row, and those rows have to land together: written + // as separate autocommit statements, a failure on the second would leave the debounce counter // already advanced, so the caller's retry would advance it a second time and the near miss it - // retried for would still be missing from the series. + // retried for would still be missing from the series. The planned stop's ownership of a cycle is + // in the same transaction for the same reason, one step stronger: a cycle that confirms inside a + // stop and is not owned is a fault the switch-off never releases. // - // A PASSED report writes one row at most and never appends a near miss, so it keeps the plain - // autocommit path. BEGIN IMMEDIATE takes the writer lock up front, which would make a heal - // heartbeat - including one that turns out to write nothing at all - contend for that lock and - // fail with SQLITE_BUSY where before it could not. + // A PASSED report writes one row at most, never appends a near miss and never starts a cycle, so + // it keeps the plain autocommit path. BEGIN IMMEDIATE takes the writer lock up front, which would + // make a heal heartbeat - including one that turns out to write nothing at all - contend for that + // lock and fail with SQLITE_BUSY where before it could not. if (event_type != EventType::EVENT_FAILED) { - return report_fault_event_locked(fault_code, event_type, severity, description, source_id, timestamp, config); + return report_fault_event_locked(fault_code, event_type, severity, description, source_id, timestamp, config, + planned_stop_active); } exec_or_throw("BEGIN IMMEDIATE"); try { - const bool is_new_occurrence = - report_fault_event_locked(fault_code, event_type, severity, description, source_id, timestamp, config); + const bool is_new_occurrence = report_fault_event_locked(fault_code, event_type, severity, description, source_id, + timestamp, config, planned_stop_active); exec_or_throw("COMMIT"); return is_new_occurrence; } catch (...) { @@ -759,9 +763,18 @@ bool SqliteFaultStorage::report_fault_event(const std::string & fault_code, uint } } +void SqliteFaultStorage::mark_planned_stop_owned_locked(const std::string & fault_code) { + SqliteStatement stmt(db_, "UPDATE faults SET planned_stop_owned = 1 WHERE fault_code = ?"); + stmt.bind_text(1, fault_code); + if (stmt.step() != SQLITE_DONE) { + throw std::runtime_error(std::string("Failed to mark fault as owned by the planned stop: ") + sqlite3_errmsg(db_)); + } +} + bool SqliteFaultStorage::report_fault_event_locked(const std::string & fault_code, uint8_t event_type, uint8_t severity, const std::string & description, const std::string & source_id, - const rclcpp::Time & timestamp, const DebounceConfig & config) { + const rclcpp::Time & timestamp, const DebounceConfig & config, + bool planned_stop_active) { int64_t timestamp_ns = timestamp.nanoseconds(); const bool is_failed = (event_type == EventType::EVENT_FAILED); @@ -903,6 +916,16 @@ bool SqliteFaultStorage::report_fault_event_locked(const std::string & fault_cod } } + // The cycle boundary the planned stop marks, decided inside the same transaction as the row it + // belongs to. A reactivation out of CLEARED starts one, and so does a failure out of HEALED - + // the heal published the fault's end, so the confirmation that follows is fresh news. A repeat + // report of a condition that is already up is the same cycle: it neither takes ownership nor + // drops the ownership an earlier report took. + if (planned_stop_active && is_failed && + (is_reactivation || current_status == ros2_medkit_msgs::msg::Fault::STATUS_HEALED)) { + mark_planned_stop_owned_locked(fault_code); + } + return is_reactivation; // Reactivation treated as new occurrence for event publishing } @@ -949,6 +972,11 @@ bool SqliteFaultStorage::report_fault_event_locked(const std::string & fault_cod record_near_miss_locked(fault_code, timestamp_ns, initial_counter, config, severity, source_id, initial_status); } + // A new fault is always the start of a cycle. + if (planned_stop_active) { + mark_planned_stop_owned_locked(fault_code); + } + return true; // New fault created } @@ -1388,18 +1416,6 @@ PlannedStopState SqliteFaultStorage::get_planned_stop() const { return state; } -void SqliteFaultStorage::set_planned_stop_owned(const std::string & fault_code, bool owned) { - std::lock_guard lock(mutex_); - - SqliteStatement stmt(db_, "UPDATE faults SET planned_stop_owned = ? WHERE fault_code = ?"); - stmt.bind_int(1, owned ? 1 : 0); - stmt.bind_text(2, fault_code); - - if (stmt.step() != SQLITE_DONE) { - throw std::runtime_error(std::string("Failed to record planned-stop ownership: ") + sqlite3_errmsg(db_)); - } -} - std::vector SqliteFaultStorage::get_planned_stop_owned() const { std::lock_guard lock(mutex_); diff --git a/src/ros2_medkit_fault_manager/test/integration/test_planned_stop.test.py b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop.test.py index 4d42b1ab6..e1edcf315 100644 --- a/src/ros2_medkit_fault_manager/test/integration/test_planned_stop.test.py +++ b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop.test.py @@ -64,10 +64,12 @@ SOURCE_ID = '/test_node' CAPTURE_TOPIC = '/test/temperature' -# Two rules. The hierarchical one mutes MOTOR_PS_* while ESTOP_PS_001 stands; the +# Three rules. The hierarchical one mutes MOTOR_PS_* while ESTOP_PS_001 stands; the # auto-cluster one claims VALVE_PS_* faults without ever writing them into the mute -# map, which is the case a planned stop's own entry has to survive. Both windows are -# long enough that no test can lose the correlation to its own pacing. +# map, which is the case a planned stop's own entry has to survive. The second cluster +# rule needs four members, which is what makes a burst reachable that has shrunk two +# short of forming again. All windows are long enough that no test can lose the +# correlation to its own pacing. CORRELATION_RULES = """ correlation: enabled: true @@ -78,6 +80,8 @@ codes: ["VALVE_PS_*"] ps_pump_errors: codes: ["PUMP_PS_*"] + ps_storm4_errors: + codes: ["STORM4_PS_*"] rules: - id: ps_estop_cascade name: "E-Stop Cascade" @@ -108,6 +112,15 @@ window_ms: 60000 show_as_single: true representative: first + - id: ps_storm4 + name: "Four Valve Storm" + mode: auto_cluster + match: + - pattern: ps_storm4_errors + min_count: 4 + window_ms: 60000 + show_as_single: true + representative: first """ @@ -695,8 +708,8 @@ def test_a_scoped_acknowledgement_still_releases_the_stops_mute(self): self._set_stop(False, reason='done', declared_by='maintenance') self.assertEqual(0, self._count_events(code, FaultEvent.EVENT_CONFIRMED)) - def test_a_cluster_rule_does_not_orphan_the_stops_mute(self): - """A cluster claims the mute without owning it, so the stop still releases.""" + def test_a_cluster_still_hiding_a_fault_keeps_it_at_the_switch_off(self): + """The cluster shows one line for the burst, and the switch-off honours it.""" first = 'VALVE_PS_A' second = 'VALVE_PS_B' @@ -718,12 +731,74 @@ def test_a_cluster_rule_does_not_orphan_the_stops_mute(self): self._set_stop(False, reason='swap done', declared_by='maintenance') + # The representative is released and announced once. + self._wait_until( + lambda: first in self._codes_in_default_list(), + 'the cluster representative was orphaned in the mute map') + self.assertIsNone(self._muted_entry(first)) + self.assertEqual(1, self._count_events(first, FaultEvent.EVENT_CONFIRMED)) + + # The symptom is not announced, and the stop leaves nothing of its own behind + # on it either. A cluster hides a member by suppressing the member's events on + # every report, never by an entry in the muted list, so after the switch-off + # this burst looks exactly like one that never met a planned stop. + self.assertIsNone(self._muted_entry(second), + 'the switch-off left an entry the cluster never writes') + self.assertIn(second, self._codes_in_default_list()) + self.assertEqual(0, self._count_events(second, FaultEvent.EVENT_CONFIRMED), + 'both symptoms of one cluster were announced at the switch-off') + + # Still hidden: the cluster swallows the non-representative's next report. + types_before = self._event_types(second) + self._report(second) + self._get_fault(second) # a round trip through the same single-threaded node + time.sleep(0.5) + self.assertEqual(types_before, self._event_types(second), + 'the cluster stopped hiding its member once the stop ended') + + # Acknowledging the representative promotes the member, and a promoted + # representative is the line the cluster shows: it is heard from again. + self.assertTrue(self._clear(first).success) + self._report(second) + self._wait_until( + lambda: FaultEvent.EVENT_UPDATED in self._event_types(second), + 'the promoted representative stayed silent') + + def test_a_fault_joining_a_cluster_below_min_count_is_not_folded_into_it(self): + """A cluster folds the members it formed with; a later joiner is a fault of its own.""" + formed = ['STORM4_PS_A', 'STORM4_PS_B', 'STORM4_PS_C', 'STORM4_PS_D'] + joiner = 'STORM4_PS_E' + representative = formed[0] + + for code in formed: + self._confirm(code) + self._wait_until( + lambda: self._count_events(representative, FaultEvent.EVENT_CONFIRMED) == 1, + 'the cluster representative was never announced') + # Only the member whose confirmation landed once the cluster had formed is + # hidden. The ones that confirmed on the way to min_count were announced + # before there was a cluster to fold them into, stop or no stop. + self.assertEqual(0, self._count_events(formed[3], FaultEvent.EVENT_CONFIRMED), + 'the member that confirmed inside the formed cluster was announced') + + # Two acknowledgements leave the burst two members short of forming again. + for code in formed[2:]: + self.assertTrue(self._clear(code).success) + + # The joiner maps to the formed cluster through the pending twin, but it is not + # one of the members that cluster shows as a single line. + self._confirm(joiner) + self._wait_until( + lambda: self._count_events(joiner, FaultEvent.EVENT_CONFIRMED) == 1, + 'a fault that joined a cluster below min_count was folded into it and never announced') + + # And it keeps being heard from, which is what a fault outside any cluster does. + self._report(joiner) self._wait_until( - lambda: all(code in self._codes_in_default_list() for code in (first, second)), - 'a fault a cluster rule claimed was orphaned in the mute map') - for code in (first, second): - self.assertIsNone(self._muted_entry(code)) - self.assertEqual(1, self._count_events(code, FaultEvent.EVENT_CONFIRMED)) + lambda: self._count_events(joiner, FaultEvent.EVENT_UPDATED) >= 1, + 'a fault that joined below min_count went silent after its first announcement') + self.assertIsNone(self._muted_entry(joiner)) + self.assertIn(joiner, self._codes_in_default_list()) def test_the_last_declaration_is_readable_after_the_withdrawal(self): """The reason survives the withdrawal, with the time the stop ended.""" diff --git a/src/ros2_medkit_fault_manager/test/test_correlation_engine.cpp b/src/ros2_medkit_fault_manager/test/test_correlation_engine.cpp index 5b9977e87..aa84686fe 100644 --- a/src/ros2_medkit_fault_manager/test/test_correlation_engine.cpp +++ b/src/ros2_medkit_fault_manager/test/test_correlation_engine.cpp @@ -126,6 +126,53 @@ class CorrelationEngineTest : public ::testing::Test { return parse_config_string(yaml); } + /// The same cluster rule at a chosen min_count, so a burst can shrink back below + /// the threshold without the cluster dissolving. + CorrelationConfig create_cluster_config_with_min_count(int min_count) { + const std::string yaml = R"( +correlation: + enabled: true + patterns: + valve_errors: + codes: ["VALVE_*"] + rules: + - id: valve_storm + name: "Valve Storm" + mode: auto_cluster + match: + - pattern: valve_errors + min_count: )" + std::to_string(min_count) + + R"( + window_ms: 60000 + show_as_single: true + representative: first +)"; + return parse_config_string(yaml); + } + + /// A cluster rule that picks the loudest member as representative, which is the + /// policy that needs each member's severity to promote a replacement. + CorrelationConfig create_cluster_config_highest_severity() { + const std::string yaml = R"( +correlation: + enabled: true + patterns: + valve_errors: + codes: ["VALVE_*"] + rules: + - id: valve_storm + name: "Valve Storm" + mode: auto_cluster + match: + - pattern: valve_errors + min_count: 2 + window_ms: 60000 + show_as_single: true + representative: highest_severity +)"; + return parse_config_string(yaml); + } + CorrelationConfig create_mixed_config() { const std::string yaml = R"( correlation: @@ -1069,7 +1116,7 @@ TEST_F(CorrelationEngineTest, ARuleMuteOverlaysTheStopsWithoutReplacingOwnership EXPECT_TRUE(engine.is_muted("MOTOR_COMM_FL")); } -TEST_F(CorrelationEngineTest, AClusterMuteLeavesTheStopHoldingItsOwnEntry) { +TEST_F(CorrelationEngineTest, TheClusterKeepsHidingTheSymptomWhenTheStopEnds) { CorrelationEngine engine(create_cluster_config()); auto t0 = std::chrono::steady_clock::now(); @@ -1091,14 +1138,211 @@ TEST_F(CorrelationEngineTest, AClusterMuteLeavesTheStopHoldingItsOwnEntry) { EXPECT_EQ(CorrelationEngine::kPlannedStopRuleId, entry.rule_id); } + // The switch-off releases the representative and no more: releasing the rest + // would announce, in one wave, the burst the cluster exists to fold into a + // single line. + auto unmuted = engine.end_planned_stop(); + ASSERT_EQ(1u, unmuted.size()) << "the stop announced a fault the cluster is still hiding"; + EXPECT_EQ("VALVE_A", unmuted[0]) << "the representative is what the cluster shows"; + + // Nothing of the stop's is left behind on the member it did not announce, and + // nothing is written in its place: the cluster hides a member by suppressing + // its events on every report, never by an entry. + EXPECT_EQ(0u, engine.get_muted_count()) << "the switch-off left an entry the cluster never writes"; + EXPECT_FALSE(engine.is_muted("VALVE_B")); + + auto repeat = engine.process_fault("VALVE_B", "ERROR", t0 + 20ms, /*cycle_started=*/false); + EXPECT_TRUE(repeat.should_mute) << "the cluster stopped hiding its member once the stop ended"; +} + +TEST_F(CorrelationEngineTest, EveryNonRepresentativeOfALiveClusterIsHeldAtTheSwitchOff) { + CorrelationEngine engine(create_cluster_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + for (const auto & code : {"VALVE_A", "VALVE_B", "VALVE_C", "VALVE_D"}) { + engine.process_fault(code, "ERROR", t0, /*cycle_started=*/true); + } + + auto unmuted = engine.end_planned_stop(); + ASSERT_EQ(1u, unmuted.size()) << "a burst of four announced more than the one line the cluster shows"; + EXPECT_EQ("VALVE_A", unmuted[0]); + EXPECT_EQ(0u, engine.get_muted_count()); + + for (const auto & code : {"VALVE_B", "VALVE_C", "VALVE_D"}) { + EXPECT_TRUE(engine.process_fault(code, "ERROR", t0 + 20ms, /*cycle_started=*/false).should_mute) + << code << " left the cluster when the stop ended"; + } +} + +// The point of hiding rather than muting: what the cluster does after a stop has to be +// what it does when there was never one, or muted_count and the default fault list depend +// on plant history rather than on the faults. +TEST_F(CorrelationEngineTest, AClusterBehavesTheSameAfterAStopAsItDoesWithoutOne) { + auto run = [this](bool with_stop) { + CorrelationEngine engine(create_cluster_config()); + auto t0 = std::chrono::steady_clock::now(); + if (with_stop) { + engine.begin_planned_stop(); + } + engine.process_fault("VALVE_A", "ERROR", t0, /*cycle_started=*/true); + engine.process_fault("VALVE_B", "ERROR", t0 + 10ms, /*cycle_started=*/true); + if (with_stop) { + engine.end_planned_stop(); + } + + std::vector observed; + observed.push_back(static_cast(engine.get_muted_count())); + observed.push_back(engine.process_fault("VALVE_B", "ERROR", t0 + 20ms, /*cycle_started=*/false).should_mute); + // Acknowledging the representative promotes the member it was hiding. + engine.process_clear("VALVE_A"); + observed.push_back(engine.process_fault("VALVE_B", "ERROR", t0 + 30ms, /*cycle_started=*/false).should_mute); + observed.push_back(static_cast(engine.get_muted_count())); + return observed; + }; + + const auto without_stop = run(false); + const auto after_stop = run(true); + EXPECT_EQ(without_stop, after_stop) + << "a cluster that lived through a stop answers differently from one that did not"; + ASSERT_EQ(4u, without_stop.size()); + EXPECT_EQ(0, without_stop[0]); + EXPECT_EQ(1, without_stop[1]) << "a non-representative of a live cluster is hidden"; + EXPECT_EQ(0, without_stop[2]) << "the promoted representative is the line the cluster shows"; + EXPECT_EQ(0, without_stop[3]); +} + +TEST_F(CorrelationEngineTest, AClusterShortOfMinCountHidesNothingAtTheSwitchOff) { + CorrelationEngine engine(create_cluster_config()); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + // min_count is 2, so one member is not a cluster and hides nobody. + engine.process_fault("VALVE_A", "ERROR", t0, /*cycle_started=*/true); + + auto unmuted = engine.end_planned_stop(); + ASSERT_EQ(1u, unmuted.size()) << "a cluster that never formed withheld a confirmation"; + EXPECT_EQ("VALVE_A", unmuted[0]); + EXPECT_EQ(0u, engine.get_muted_count()); +} + +TEST_F(CorrelationEngineTest, AClusterWithoutShowAsSingleReleasesEveryMember) { + const std::string yaml = R"( +correlation: + enabled: true + patterns: + valve_errors: + codes: ["VALVE_*"] + rules: + - id: valve_storm + name: "Valve Storm" + mode: auto_cluster + match: + - pattern: valve_errors + min_count: 2 + window_ms: 60000 + show_as_single: false + representative: first +)"; + CorrelationEngine engine(parse_config_string(yaml)); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + engine.process_fault("VALVE_A", "ERROR", t0, /*cycle_started=*/true); + engine.process_fault("VALVE_B", "ERROR", t0 + 10ms, /*cycle_started=*/true); + + // A cluster that groups without hiding has no verdict to outrank the stop with. auto unmuted = engine.end_planned_stop(); std::sort(unmuted.begin(), unmuted.end()); - ASSERT_EQ(2u, unmuted.size()) << "a cluster-claimed fault was orphaned in the mute map"; + ASSERT_EQ(2u, unmuted.size()) << "a cluster that hides nobody withheld a confirmation"; EXPECT_EQ("VALVE_A", unmuted[0]); EXPECT_EQ("VALVE_B", unmuted[1]); EXPECT_EQ(0u, engine.get_muted_count()); } +// min_count decides whether a cluster FORMS. Once formed, the cluster folds its members +// into the representative until the last of them is acknowledged and it dissolves, so a +// burst that shrinks below the threshold does not start announcing its members again. +TEST_F(CorrelationEngineTest, AnActiveClusterKeepsHidingAfterItShrinksBelowMinCount) { + CorrelationEngine engine(create_cluster_config_with_min_count(3)); + + auto t0 = std::chrono::steady_clock::now(); + engine.process_fault("VALVE_A", "ERROR", t0, /*cycle_started=*/true); + engine.process_fault("VALVE_B", "ERROR", t0 + 10ms, /*cycle_started=*/true); + auto third = engine.process_fault("VALVE_C", "ERROR", t0 + 20ms, /*cycle_started=*/true); + ASSERT_TRUE(third.should_mute) << "the cluster did not form at min_count"; + + engine.process_clear("VALVE_C"); + + auto repeat = engine.process_fault("VALVE_B", "ERROR", t0 + 30ms, /*cycle_started=*/false); + EXPECT_TRUE(repeat.should_mute) << "a cluster stopped hiding its member because a sibling was acknowledged"; + + // It dissolves with its last member, not with the threshold. + engine.process_clear("VALVE_B"); + auto alone = engine.process_fault("VALVE_A", "ERROR", t0 + 40ms, /*cycle_started=*/false); + EXPECT_FALSE(alone.should_mute) << "the representative was hidden by its own cluster"; +} + +TEST_F(CorrelationEngineTest, AShrunkenActiveClusterStillHoldsItsMemberAtTheSwitchOff) { + CorrelationEngine engine(create_cluster_config_with_min_count(3)); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + engine.process_fault("VALVE_A", "ERROR", t0, /*cycle_started=*/true); + engine.process_fault("VALVE_B", "ERROR", t0 + 10ms, /*cycle_started=*/true); + engine.process_fault("VALVE_C", "ERROR", t0 + 20ms, /*cycle_started=*/true); + + engine.process_clear("VALVE_C"); + + auto unmuted = engine.end_planned_stop(); + ASSERT_EQ(1u, unmuted.size()) << "a cluster below min_count announced its whole burst"; + EXPECT_EQ("VALVE_A", unmuted[0]); + EXPECT_TRUE(engine.process_fault("VALVE_B", "ERROR", t0 + 30ms, /*cycle_started=*/false).should_mute); +} + +// cleanup_expired drops the pending twin and keeps the active cluster, so after the window +// closes the active cluster is the only record of who is left. Promoted from the twin that +// is no longer there, the representative keeps naming the acknowledged fault and every +// remaining member is hidden by a cluster whose representative can never be reported again. +TEST_F(CorrelationEngineTest, AcknowledgingTheRepresentativeAfterItsWindowClosedPromotesAMember) { + CorrelationEngine engine(create_cluster_config()); + + // Reported as of two minutes ago, so the rule's 60 s window is already behind them. + auto t0 = std::chrono::steady_clock::now() - 120000ms; + engine.begin_planned_stop(); + engine.process_fault("VALVE_A", "ERROR", t0, /*cycle_started=*/true); + engine.process_fault("VALVE_B", "ERROR", t0 + 10ms, /*cycle_started=*/true); + + engine.cleanup_expired(); + engine.process_clear("VALVE_A"); + + auto unmuted = engine.end_planned_stop(); + ASSERT_EQ(1u, unmuted.size()) << "the cluster kept naming the acknowledged fault, so nobody was released"; + EXPECT_EQ("VALVE_B", unmuted[0]) << "the promoted representative is the one the switch-off announces"; + EXPECT_EQ(0u, engine.get_muted_count()); +} + +TEST_F(CorrelationEngineTest, PromotionAfterTheWindowClosedFollowsTheRulesPolicy) { + CorrelationEngine engine(create_cluster_config_highest_severity()); + + auto t0 = std::chrono::steady_clock::now() - 120000ms; + engine.process_fault("VALVE_A", "ERROR", t0, /*cycle_started=*/true); + engine.process_fault("VALVE_B", "WARN", t0 + 10ms, /*cycle_started=*/true); + engine.process_fault("VALVE_C", "CRITICAL", t0 + 20ms, /*cycle_started=*/true); + + // CRITICAL is the representative; with the pending twin gone the promotion has to + // read the severities the active cluster carries. + engine.cleanup_expired(); + engine.process_clear("VALVE_C"); + + // Read through get_clusters() rather than by re-reporting: a report after the window + // has closed starts a fresh cluster and would answer about that one instead. + auto clusters = engine.get_clusters(); + ASSERT_EQ(1u, clusters.size()); + EXPECT_EQ("VALVE_A", clusters[0].representative_code) << "the highest-severity survivor was not promoted"; + EXPECT_EQ("ERROR", clusters[0].representative_severity); +} + // ============================================================================ // Planned stop: dropping and restoring one fault's mute // ============================================================================ @@ -1294,6 +1538,86 @@ TEST_F(CorrelationEngineTest, AutoClearedSymptomsLeaveTheStopsOwnership) { EXPECT_TRUE(engine.end_planned_stop().empty()); } +// `fault_to_cluster_` is written on every join, including a join to a PENDING cluster +// that is below min_count, while `active_clusters_` is refreshed only when the burst +// reaches the threshold. A fault that joined a pending twin two members short therefore +// maps to a formed cluster it is not part of. Folding it into that cluster's line +// announces it once - the join itself is below threshold and not muted - and then +// silences every repeat, so an alarm that is still up stops updating and appears in +// neither the muted list nor the cluster listing. +TEST_F(CorrelationEngineTest, AFaultJoiningAClusterBelowMinCountIsNotFoldedIntoIt) { + CorrelationEngine engine(create_cluster_config_with_min_count(4)); + + auto t0 = std::chrono::steady_clock::now(); + for (const auto & code : {"VALVE_A", "VALVE_B", "VALVE_C", "VALVE_D"}) { + engine.process_fault(code, "ERROR", t0, /*cycle_started=*/true); + } + engine.process_clear("VALVE_C"); + engine.process_clear("VALVE_D"); + + // Two members short of forming again, so this one joins nothing that hides. + auto joined = engine.process_fault("VALVE_E", "ERROR", t0 + 10ms, /*cycle_started=*/true); + EXPECT_FALSE(joined.should_mute); + + auto repeat = engine.process_fault("VALVE_E", "ERROR", t0 + 20ms, /*cycle_started=*/false); + EXPECT_FALSE(repeat.should_mute) << "a fault the cluster never formed with was folded into it and went silent"; + EXPECT_FALSE(engine.is_muted("VALVE_E")); + + // The members it did form with are still hidden. + EXPECT_TRUE(engine.process_fault("VALVE_B", "ERROR", t0 + 30ms, /*cycle_started=*/false).should_mute); +} + +TEST_F(CorrelationEngineTest, AFaultJoiningAClusterBelowMinCountIsReleasedAtTheSwitchOff) { + CorrelationEngine engine(create_cluster_config_with_min_count(4)); + + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + for (const auto & code : {"VALVE_A", "VALVE_B", "VALVE_C", "VALVE_D"}) { + engine.process_fault(code, "ERROR", t0, /*cycle_started=*/true); + } + engine.process_clear("VALVE_C"); + engine.process_clear("VALVE_D"); + engine.process_fault("VALVE_E", "ERROR", t0 + 10ms, /*cycle_started=*/true); + + auto unmuted = engine.end_planned_stop(); + std::sort(unmuted.begin(), unmuted.end()); + ASSERT_EQ(2u, unmuted.size()) << "the fault that joined below min_count was withheld by a cluster it is not in"; + EXPECT_EQ("VALVE_A", unmuted[0]) << "the representative"; + EXPECT_EQ("VALVE_E", unmuted[1]) << "a fault of its own, announced like any other"; +} + +// The cluster hold lives in the process. A replacement manager restores ownership from +// the store and has no clusters, so the switch-off releases and announces every owned +// fault - including the members the cluster was folding into one line before the +// restart. Persisting cluster state would change that and is not part of this. +TEST_F(CorrelationEngineTest, ARestartReleasesEveryOwnedFaultIncludingOnesAClusterWasHiding) { + auto t0 = std::chrono::steady_clock::now(); + + CorrelationEngine before_restart(create_cluster_config_with_min_count(2)); + before_restart.begin_planned_stop(); + before_restart.process_fault("VALVE_A", "ERROR", t0, /*cycle_started=*/true); + before_restart.process_fault("VALVE_B", "ERROR", t0 + 10ms, /*cycle_started=*/true); + ASSERT_TRUE(before_restart.process_fault("VALVE_B", "ERROR", t0 + 20ms, /*cycle_started=*/false).should_mute) + << "the cluster was not hiding the member before the restart"; + + // What the store holds, and all the replacement process has to go on. + auto owned = before_restart.planned_stop_owned_codes(); + std::sort(owned.begin(), owned.end()); + ASSERT_EQ(2u, owned.size()); + + CorrelationEngine after_restart(create_cluster_config_with_min_count(2)); + after_restart.begin_planned_stop(); + for (const auto & code : owned) { + after_restart.restore_planned_stop_ownership(code); + } + + auto unmuted = after_restart.end_planned_stop(); + std::sort(unmuted.begin(), unmuted.end()); + ASSERT_EQ(2u, unmuted.size()) << "a restart is expected to release the whole burst; the cluster is gone with the " + "process that formed it"; + EXPECT_EQ("VALVE_A", unmuted[0]); + EXPECT_EQ("VALVE_B", unmuted[1]); +} int main(int argc, char ** argv) { ::testing::InitGoogleTest(&argc, argv); return RUN_ALL_TESTS(); diff --git a/src/ros2_medkit_fault_manager/test/test_fault_manager.cpp b/src/ros2_medkit_fault_manager/test/test_fault_manager.cpp index 8419e0c61..8fab2c54d 100644 --- a/src/ros2_medkit_fault_manager/test/test_fault_manager.cpp +++ b/src/ros2_medkit_fault_manager/test/test_fault_manager.cpp @@ -15,6 +15,7 @@ #include #include +#include #include #include #include @@ -1062,7 +1063,7 @@ class FaultEventPublishingTest : public ::testing::Test { std::string events_topic = ns + "/fault_manager/events"; auto qos = rclcpp::QoS(100).reliable().durability_volatile(); event_subscription_ = - test_node_->create_subscription(events_topic, qos, [this](const FaultEvent::SharedPtr msg) { + test_node_->create_subscription(events_topic, qos, [this](const FaultEvent::ConstSharedPtr & msg) { received_events_.push_back(*msg); }); @@ -1208,7 +1209,7 @@ TEST_F(FaultEventPublishingTest, NewFaultPublishesConfirmedEvent) { // Wait for event to arrive (polling, robust under CPU contention) ASSERT_TRUE(spin_until([this]() { - return received_events_.size() >= 1; + return !received_events_.empty(); })); // Verify EVENT_CONFIRMED was published @@ -1223,7 +1224,7 @@ TEST_F(FaultEventPublishingTest, UpdateExistingFaultPublishesUpdatedEvent) { // Report a new fault first ASSERT_TRUE(call_report_fault("TEST_FAULT_2", Fault::SEVERITY_WARN, "/test_node1")); ASSERT_TRUE(spin_until([this]() { - return received_events_.size() >= 1; + return !received_events_.empty(); })); // Clear received events @@ -1232,7 +1233,7 @@ TEST_F(FaultEventPublishingTest, UpdateExistingFaultPublishesUpdatedEvent) { // Report same fault again - should trigger EVENT_UPDATED ASSERT_TRUE(call_report_fault("TEST_FAULT_2", Fault::SEVERITY_ERROR, "/test_node2")); ASSERT_TRUE(spin_until([this]() { - return received_events_.size() >= 1; + return !received_events_.empty(); })); // Verify EVENT_UPDATED was published (severity/sources changed; still one occurrence) @@ -1246,7 +1247,7 @@ TEST_F(FaultEventPublishingTest, ClearFaultPublishesClearedEvent) { // Report a fault first ASSERT_TRUE(call_report_fault("TEST_FAULT_3", Fault::SEVERITY_ERROR, "/test_node")); ASSERT_TRUE(spin_until([this]() { - return received_events_.size() >= 1; + return !received_events_.empty(); })); // Clear received events @@ -1255,7 +1256,7 @@ TEST_F(FaultEventPublishingTest, ClearFaultPublishesClearedEvent) { // Clear the fault ASSERT_TRUE(call_clear_fault("TEST_FAULT_3")); ASSERT_TRUE(spin_until([this]() { - return received_events_.size() >= 1; + return !received_events_.empty(); })); // Verify EVENT_CLEARED was published @@ -1282,7 +1283,7 @@ class HealingFaultEventPublishingTest : public FaultEventPublishingTest { TEST_F(HealingFaultEventPublishingTest, HealPublishesClearedEventSoStreamConsumersSeeTheEnd) { ASSERT_TRUE(call_report_fault("HEAL_ME", Fault::SEVERITY_ERROR, "/test_node")); ASSERT_TRUE(spin_until([this]() { - return received_events_.size() >= 1; + return !received_events_.empty(); })); received_events_.clear(); @@ -1325,7 +1326,7 @@ TEST_F(FaultEventPublishingTest, EventContainsCorrectTimestamp) { ASSERT_TRUE(call_report_fault("TEST_FAULT_4", Fault::SEVERITY_WARN, "/test_node")); ASSERT_TRUE(spin_until([this]() { - return received_events_.size() >= 1; + return !received_events_.empty(); })); auto after = fault_manager_->now(); @@ -1341,7 +1342,7 @@ TEST_F(FaultEventPublishingTest, EventContainsCorrectTimestamp) { TEST_F(FaultEventPublishingTest, EventContainsFullFaultData) { ASSERT_TRUE(call_report_fault("FULL_DATA_TEST", Fault::SEVERITY_CRITICAL, "/sensor/temperature")); ASSERT_TRUE(spin_until([this]() { - return received_events_.size() >= 1; + return !received_events_.empty(); })); ASSERT_EQ(received_events_.size(), 1u); @@ -1364,7 +1365,7 @@ TEST_F(FaultEventPublishingTest, TimestampUsesWallClockNotSimTime) { ASSERT_TRUE(call_report_fault("WALL_CLOCK_TEST", Fault::SEVERITY_WARN, "/test_node")); ASSERT_TRUE(spin_until([this]() { - return received_events_.size() >= 1; + return !received_events_.empty(); })); auto wall_after = std::chrono::system_clock::now(); @@ -1392,7 +1393,7 @@ TEST_F(FaultEventPublishingTest, GetFaultReturnsExpectedFault) { // Report a fault first ASSERT_TRUE(call_report_fault("GET_FAULT_TEST", Fault::SEVERITY_ERROR, "/test_node")); ASSERT_TRUE(spin_until([this]() { - return received_events_.size() >= 1; + return !received_events_.empty(); })); // Get fault via service @@ -1416,7 +1417,7 @@ TEST_F(FaultEventPublishingTest, GetFaultReturnsEnvironmentData) { // Report a fault ASSERT_TRUE(call_report_fault("ENV_DATA_TEST", Fault::SEVERITY_WARN, "/sensor/temp")); ASSERT_TRUE(spin_until([this]() { - return received_events_.size() >= 1; + return !received_events_.empty(); })); auto response = call_get_fault("ENV_DATA_TEST"); @@ -1436,7 +1437,7 @@ TEST_F(FaultEventPublishingTest, GetFaultReturnsExtendedDataRecords) { // Report fault twice to have first and last occurrence timestamps differ ASSERT_TRUE(call_report_fault("EDR_TEST", Fault::SEVERITY_ERROR, "/node1")); ASSERT_TRUE(spin_until([this]() { - return received_events_.size() >= 1; + return !received_events_.empty(); })); ASSERT_TRUE(call_report_fault("EDR_TEST", Fault::SEVERITY_ERROR, "/node2")); ASSERT_TRUE(spin_until([this]() { @@ -1484,7 +1485,7 @@ TEST_F(FaultEventPublishingTest, ListFaultsForEntityEmptyResult) { // Report faults from a different entity ASSERT_TRUE(call_report_fault("SOME_FAULT", Fault::SEVERITY_ERROR, "/some/other_entity")); ASSERT_TRUE(spin_until([this]() { - return received_events_.size() >= 1; + return !received_events_.empty(); })); // Query faults for non-existent entity @@ -3073,13 +3074,16 @@ class PlannedStopServiceTest : public ::testing::Test { test_node_->create_client(ns + "/fault_manager/set_planned_stop"); get_client_ = test_node_->create_client(ns + "/fault_manager/get_planned_stop"); + report_client_ = test_node_->create_client(ns + "/fault_manager/report_fault"); ASSERT_TRUE(set_client_->wait_for_service(std::chrono::seconds(5))); ASSERT_TRUE(get_client_->wait_for_service(std::chrono::seconds(5))); + ASSERT_TRUE(report_client_->wait_for_service(std::chrono::seconds(5))); } void TearDown() override { set_client_.reset(); get_client_.reset(); + report_client_.reset(); test_node_.reset(); fault_manager_.reset(); } @@ -3116,10 +3120,25 @@ class PlannedStopServiceTest : public ::testing::Test { return *future.get(); } + /// One FAILED report over the real service. The fixture's confirmation + /// threshold is -1, so a single call confirms the fault. + ros2_medkit_msgs::srv::ReportFault::Response report_failed(const std::string & fault_code) { + auto request = std::make_shared(); + request->fault_code = fault_code; + request->event_type = ros2_medkit_msgs::srv::ReportFault::Request::EVENT_FAILED; + request->severity = Fault::SEVERITY_ERROR; + request->description = "planned stop ownership test"; + request->source_id = "/test_source"; + auto future = report_client_->async_send_request(request); + EXPECT_TRUE(spin_until_future_ready(future)); + return *future.get(); + } + std::shared_ptr fault_manager_; std::shared_ptr test_node_; rclcpp::Client::SharedPtr set_client_; rclcpp::Client::SharedPtr get_client_; + rclcpp::Client::SharedPtr report_client_; }; /// The same fixture, with a store that records the order of the release. @@ -3142,12 +3161,12 @@ TEST_F(PlannedStopReleaseOrderTest, TheOwnershipFlagsAreClearedAfterTheAnnouncem ASSERT_NE(recorder_, nullptr); ASSERT_TRUE(set_stop(true, "line 3 maintenance", "shift_lead").success); - // A confirmed fault the stop owns, put in place directly: this test is about the - // order of the switch-off's own calls. + // A confirmed fault the stop owns, put in place through the store directly: this test + // is about the order of the switch-off's own calls, not about the report path. auto & storage = fault_manager_->get_storage_for_test(); storage.report_fault_event("ORDER_OWNED", ReportFault::Request::EVENT_FAILED, Fault::SEVERITY_ERROR, "order test", - "/src", fault_manager_->now(), storage.get_debounce_config()); - storage.set_planned_stop_owned("ORDER_OWNED", true); + "/src", fault_manager_->now(), storage.get_debounce_config(), + /*planned_stop_active=*/true); fault_manager_->restore_planned_stop_ownership_for_test("ORDER_OWNED"); recorder_->calls.clear(); @@ -3162,6 +3181,26 @@ TEST_F(PlannedStopReleaseOrderTest, TheOwnershipFlagsAreClearedAfterTheAnnouncem << "the flags were cleared before the announcement, so a crash in between would lose it"; } +// The handler has to hand the store the switch, or a cycle that starts inside the stop +// is confirmed and unowned: no switch-off releases it, no startup recognises it as +// interrupted, and its later reports announce updates mid-stop. The store is the only +// writer of the flag, so the report returning with it set is the whole path. +TEST_F(PlannedStopServiceTest, TheReportItselfRecordsTheStopsOwnership) { + ASSERT_TRUE(set_stop(true, "line 3 maintenance", "shift_lead").success); + + ASSERT_TRUE(report_failed("PS_OWNED_BY_REPORT").accepted); + + const auto owned = fault_manager_->get_storage_for_test().get_planned_stop_owned(); + EXPECT_NE(std::find(owned.begin(), owned.end(), "PS_OWNED_BY_REPORT"), owned.end()) + << "the report returned with the fault unowned, so the handler withheld the switch"; + + // A fault raised after the withdrawal belongs to nobody. + ASSERT_TRUE(set_stop(false, "plant back up", "shift_lead").success); + ASSERT_TRUE(report_failed("PS_AFTER_THE_STOP").accepted); + const auto after = fault_manager_->get_storage_for_test().get_planned_stop_owned(); + EXPECT_EQ(std::find(after.begin(), after.end(), "PS_AFTER_THE_STOP"), after.end()); +} + /// The same fixture, with a store that refuses to record the declaration. class PlannedStopFailingStoreTest : public PlannedStopServiceTest { protected: diff --git a/src/ros2_medkit_fault_manager/test/test_sqlite_storage.cpp b/src/ros2_medkit_fault_manager/test/test_sqlite_storage.cpp index ed02de85f..500c034b8 100644 --- a/src/ros2_medkit_fault_manager/test/test_sqlite_storage.cpp +++ b/src/ros2_medkit_fault_manager/test/test_sqlite_storage.cpp @@ -2525,24 +2525,24 @@ TEST_F(SqliteFaultStorageTest, ADatabaseWrittenBeforeTheEndTimeExistedStillOpens namespace { -/// Drive a fault to CONFIRMED with the immediate-confirmation default. -void raise(ros2_medkit_fault_manager::FaultStorage & storage, const std::string & code) { +/// Drive a fault to CONFIRMED with the immediate-confirmation default, letting a declared +/// planned stop take the cycle. A report is the only thing that makes a fault the stop's, so +/// it is also how a test sets one up. +void raise(ros2_medkit_fault_manager::FaultStorage & storage, const std::string & code, + bool planned_stop_active = false) { rclcpp::Clock clock; storage.report_fault_event(code, ReportFault::Request::EVENT_FAILED, Fault::SEVERITY_ERROR, "owned test", "/src", - clock.now(), default_config()); + clock.now(), default_config(), planned_stop_active); } } // namespace TEST_F(SqliteFaultStorageTest, PlannedStopOwnershipRoundTripsThroughAReopen) { - raise(*storage_, "OWNED_ONE"); - raise(*storage_, "OWNED_TWO"); raise(*storage_, "NOT_OWNED"); - EXPECT_TRUE(storage_->get_planned_stop_owned().empty()); - storage_->set_planned_stop_owned("OWNED_ONE", true); - storage_->set_planned_stop_owned("OWNED_TWO", true); + raise(*storage_, "OWNED_ONE", /*planned_stop_active=*/true); + raise(*storage_, "OWNED_TWO", /*planned_stop_active=*/true); storage_.reset(); storage_ = std::make_unique(temp_db_path_.string()); @@ -2555,8 +2555,7 @@ TEST_F(SqliteFaultStorageTest, PlannedStopOwnershipRoundTripsThroughAReopen) { } TEST_F(SqliteFaultStorageTest, AcknowledgingAFaultEndsTheStopsOwnershipOfIt) { - raise(*storage_, "OWNED_CLEARED"); - storage_->set_planned_stop_owned("OWNED_CLEARED", true); + raise(*storage_, "OWNED_CLEARED", /*planned_stop_active=*/true); ASSERT_EQ(1u, storage_->get_planned_stop_owned().size()); ASSERT_TRUE(storage_->clear_fault("OWNED_CLEARED")); @@ -2566,10 +2565,8 @@ TEST_F(SqliteFaultStorageTest, AcknowledgingAFaultEndsTheStopsOwnershipOfIt) { } TEST_F(SqliteFaultStorageTest, ClearingOwnershipDropsEveryFlagAndReportsHowMany) { - raise(*storage_, "OWNED_A"); - raise(*storage_, "OWNED_B"); - storage_->set_planned_stop_owned("OWNED_A", true); - storage_->set_planned_stop_owned("OWNED_B", true); + raise(*storage_, "OWNED_A", /*planned_stop_active=*/true); + raise(*storage_, "OWNED_B", /*planned_stop_active=*/true); EXPECT_EQ(2u, storage_->clear_planned_stop_owned()); EXPECT_TRUE(storage_->get_planned_stop_owned().empty()); @@ -2577,11 +2574,8 @@ TEST_F(SqliteFaultStorageTest, ClearingOwnershipDropsEveryFlagAndReportsHowMany) } TEST_F(SqliteFaultStorageTest, ClearingNamedOwnershipLeavesEveryOtherFlagAlone) { - raise(*storage_, "OWNED_CAPTURED_A"); - raise(*storage_, "OWNED_CAPTURED_B"); - raise(*storage_, "OWNED_LATER"); for (const auto * code : {"OWNED_CAPTURED_A", "OWNED_CAPTURED_B", "OWNED_LATER"}) { - storage_->set_planned_stop_owned(code, true); + raise(*storage_, code, /*planned_stop_active=*/true); } // A release finishes what it captured. Anything that became owned after that @@ -2601,10 +2595,8 @@ TEST_F(SqliteFaultStorageTest, ClearingNamedOwnershipLeavesEveryOtherFlagAlone) TEST(InMemoryFaultStorageTest, ClearingNamedOwnershipLeavesEveryOtherFlagAlone) { ros2_medkit_fault_manager::InMemoryFaultStorage storage; - raise(storage, "OWNED_CAPTURED"); - raise(storage, "OWNED_LATER"); - storage.set_planned_stop_owned("OWNED_CAPTURED", true); - storage.set_planned_stop_owned("OWNED_LATER", true); + raise(storage, "OWNED_CAPTURED", /*planned_stop_active=*/true); + raise(storage, "OWNED_LATER", /*planned_stop_active=*/true); EXPECT_EQ(1u, storage.clear_planned_stop_owned({"OWNED_CAPTURED"})); @@ -2629,16 +2621,19 @@ TEST_F(SqliteFaultStorageTest, ADatabaseWrittenBeforeOwnershipExistedStillOpens) EXPECT_TRUE(storage_->get_planned_stop_owned().empty()); EXPECT_TRUE(storage_->contains("LEGACY_FAULT")); - storage_->set_planned_stop_owned("LEGACY_FAULT", true); + + // The re-added column takes writes: acknowledge the migrated fault and raise it again + // inside a stop, which is a cycle starting, and the flag lands on the migrated row. + ASSERT_TRUE(storage_->clear_fault("LEGACY_FAULT")); + raise(*storage_, "LEGACY_FAULT", /*planned_stop_active=*/true); EXPECT_EQ(1u, storage_->get_planned_stop_owned().size()); } TEST(InMemoryFaultStorageTest, PlannedStopOwnershipIsTrackedPerFault) { ros2_medkit_fault_manager::InMemoryFaultStorage storage; - raise(storage, "OWNED_ONE"); + raise(storage, "OWNED_ONE", /*planned_stop_active=*/true); raise(storage, "NOT_OWNED"); - storage.set_planned_stop_owned("OWNED_ONE", true); auto owned = storage.get_planned_stop_owned(); ASSERT_EQ(1u, owned.size()); EXPECT_EQ("OWNED_ONE", owned[0]); @@ -2646,9 +2641,9 @@ TEST(InMemoryFaultStorageTest, PlannedStopOwnershipIsTrackedPerFault) { ASSERT_TRUE(storage.clear_fault("OWNED_ONE")); EXPECT_TRUE(storage.get_planned_stop_owned().empty()); - storage.set_planned_stop_owned("NOT_OWNED", true); - EXPECT_EQ(1u, storage.clear_planned_stop_owned()); - EXPECT_TRUE(storage.get_planned_stop_owned().empty()); + raise(storage, "NOT_OWNED", /*planned_stop_active=*/true); + EXPECT_TRUE(storage.get_planned_stop_owned().empty()) + << "a report that repeats a standing condition is the same cycle, so the stop cannot take it"; } TEST_F(SqliteFaultStorageTest, PlannedStopCarriesAKilobyteReason) { @@ -2691,6 +2686,149 @@ TEST(InMemoryFaultStorageTest, PlannedStopRoundTripsInMemory) { EXPECT_TRUE(storage.get_planned_stop().reason.empty()); } +namespace { + +/// Where a planned stop takes ownership of a fault cycle, driven against one backend. +/// Both backends have to answer identically: which faults a switch-off releases must +/// not depend on the storage a deployment picked. +void expect_planned_stop_cycle_boundary(ros2_medkit_fault_manager::FaultStorage & storage) { + rclcpp::Clock clock; + DebounceConfig config = default_config(); + config.healing_enabled = true; + config.healing_threshold = 0; + + auto owned_codes = [&storage]() { + auto codes = storage.get_planned_stop_owned(); + std::sort(codes.begin(), codes.end()); + return codes; + }; + const std::vector inside_only{"PS_INSIDE"}; + const std::vector both{"PS_INSIDE", "PS_OUTSIDE"}; + + // A fault raised inside a stop is owned by the report that created it. Read back + // straight after that one call: nothing else has run, so nothing else could have + // written the flag. + storage.report_fault_event("PS_INSIDE", ReportFault::Request::EVENT_FAILED, Fault::SEVERITY_ERROR, "inside", "/src", + clock.now(), config, /*planned_stop_active=*/true); + EXPECT_EQ(inside_only, owned_codes()) << "the report did not carry the stop's ownership"; + + // With no stop in force nothing is owned. + storage.report_fault_event("PS_OUTSIDE", ReportFault::Request::EVENT_FAILED, Fault::SEVERITY_ERROR, "outside", "/src", + clock.now(), config, /*planned_stop_active=*/false); + EXPECT_EQ(inside_only, owned_codes()); + + // A repeat report of a condition that was already up is the same cycle, so a stop + // declared after it started does not take it. + storage.report_fault_event("PS_OUTSIDE", ReportFault::Request::EVENT_FAILED, Fault::SEVERITY_ERROR, "outside", "/src", + clock.now(), config, /*planned_stop_active=*/true); + EXPECT_EQ(inside_only, owned_codes()) << "a standing fault was taken by a stop declared after it"; + + // Nor does a later report hand an owned cycle back. + storage.report_fault_event("PS_INSIDE", ReportFault::Request::EVENT_FAILED, Fault::SEVERITY_ERROR, "inside", "/src", + clock.now(), config, /*planned_stop_active=*/false); + EXPECT_EQ(inside_only, owned_codes()) << "a repeat report dropped the stop ownership"; + + // Acknowledging ends the cycle the stop owned; raising the fault again starts one + // the stop owns afresh. + ASSERT_TRUE(storage.clear_fault("PS_INSIDE")); + EXPECT_TRUE(owned_codes().empty()); + storage.report_fault_event("PS_INSIDE", ReportFault::Request::EVENT_FAILED, Fault::SEVERITY_ERROR, "inside", "/src", + clock.now(), config, /*planned_stop_active=*/true); + EXPECT_EQ(inside_only, owned_codes()) << "a fault raised again inside the stop was not owned"; + + // A PASSED report starts nothing, whatever the switch says. + storage.report_fault_event("PS_OUTSIDE", ReportFault::Request::EVENT_PASSED, Fault::SEVERITY_ERROR, "outside", "/src", + clock.now(), config, /*planned_stop_active=*/true); + EXPECT_EQ(inside_only, owned_codes()) << "a PASSED report took ownership of a cycle"; + auto healed = storage.get_fault("PS_OUTSIDE"); + ASSERT_TRUE(healed.has_value()); + ASSERT_EQ(Fault::STATUS_HEALED, healed->status); + + // The heal published the fault's end, so failing again is fresh news and a new + // cycle for the stop to own. + storage.report_fault_event("PS_OUTSIDE", ReportFault::Request::EVENT_FAILED, Fault::SEVERITY_ERROR, "outside", "/src", + clock.now(), config, /*planned_stop_active=*/true); + EXPECT_EQ(both, owned_codes()) << "a fault that failed again after healing was not owned"; +} + +} // namespace + +TEST_F(SqliteFaultStorageTest, TheReportCarriesThePlannedStopsOwnership) { + expect_planned_stop_cycle_boundary(*storage_); +} + +TEST(InMemoryFaultStorageTest, TheReportCarriesThePlannedStopsOwnership) { + ros2_medkit_fault_manager::InMemoryFaultStorage storage; + expect_planned_stop_cycle_boundary(storage); +} + +namespace { + +/// Run one statement on a second connection to the same database file. Used to install +/// and remove a trigger, which is a schema change every connection to the file honours, +/// including the one the store is holding. +void exec_on_a_second_connection(const std::string & db_path, const char * sql) { + sqlite3 * raw = nullptr; + ASSERT_EQ(sqlite3_open(db_path.c_str(), &raw), SQLITE_OK); + char * error = nullptr; + const int rc = sqlite3_exec(raw, sql, nullptr, nullptr, &error); + const std::string message = error != nullptr ? error : ""; + sqlite3_free(error); + sqlite3_close(raw); + ASSERT_EQ(rc, SQLITE_OK) << message; +} + +constexpr const char * kRefuseOwnershipWrites = + "CREATE TRIGGER deny_ownership BEFORE UPDATE OF planned_stop_owned ON faults " + "BEGIN SELECT RAISE(ABORT, 'ownership write refused'); END;"; + +} // namespace + +// The whole point of carrying ownership with the report is that the two cannot come +// apart. A store that keeps the fault row when the ownership write fails leaves a +// confirmed fault inside a stop that nothing owns: no switch-off releases it, no +// startup recognises it, and its later reports announce updates mid-stop. Driven on +// SQLite because that is the backend the crash window is about, and through a trigger +// because that makes the ownership statement fail where it really runs, inside the +// report's transaction. +TEST_F(SqliteFaultStorageTest, AnOwnershipWriteThatFailsTakesTheReportDownWithIt) { + rclcpp::Clock clock; + ASSERT_NO_FATAL_FAILURE(exec_on_a_second_connection(storage_->db_path(), kRefuseOwnershipWrites)); + + EXPECT_THROW(storage_->report_fault_event("PS_ATOMIC", ReportFault::Request::EVENT_FAILED, Fault::SEVERITY_ERROR, + "atomic test", "/src", clock.now(), default_config(), + /*planned_stop_active=*/true), + std::runtime_error); + + storage_.reset(); + ASSERT_NO_FATAL_FAILURE(exec_on_a_second_connection(temp_db_path_.string(), "DROP TRIGGER deny_ownership")); + + // Reopened, because the question is what survived the commit, not what one + // connection happens to be holding. + SqliteFaultStorage reopened(temp_db_path_.string()); + EXPECT_FALSE(reopened.contains("PS_ATOMIC")) << "the fault row committed without the ownership that belongs to it"; + EXPECT_TRUE(reopened.get_planned_stop_owned().empty()); +} + +// A report with no stop in force writes no ownership at all, so the same refusal +// cannot touch it: the negative control that keeps the test above about the +// ownership statement rather than about the trigger. +TEST_F(SqliteFaultStorageTest, TheSameRefusalDoesNotTouchAReportOutsideAStop) { + rclcpp::Clock clock; + ASSERT_NO_FATAL_FAILURE(exec_on_a_second_connection(storage_->db_path(), kRefuseOwnershipWrites)); + + EXPECT_NO_THROW(storage_->report_fault_event("PS_NO_STOP", ReportFault::Request::EVENT_FAILED, Fault::SEVERITY_ERROR, + "atomic test", "/src", clock.now(), default_config(), + /*planned_stop_active=*/false)); + + storage_.reset(); + ASSERT_NO_FATAL_FAILURE(exec_on_a_second_connection(temp_db_path_.string(), "DROP TRIGGER deny_ownership")); + + SqliteFaultStorage reopened(temp_db_path_.string()); + EXPECT_TRUE(reopened.contains("PS_NO_STOP")); + EXPECT_TRUE(reopened.get_planned_stop_owned().empty()); +} + int main(int argc, char ** argv) { rclcpp::init(argc, argv); ::testing::InitGoogleTest(&argc, argv); diff --git a/src/ros2_medkit_msgs/README.md b/src/ros2_medkit_msgs/README.md index 232ee5970..ad78cae98 100644 --- a/src/ros2_medkit_msgs/README.md +++ b/src/ros2_medkit_msgs/README.md @@ -166,12 +166,19 @@ condition holds, and that fault's confirmation has already been announced. Publication matches a rule-muted symptom exactly: `EVENT_CONFIRMED` and `EVENT_UPDATED` are withheld whichever kind of report produced them, `EVENT_CLEARED` is published as -usual. Withdrawing the stop releases every fault it owns and publishes one -`EVENT_CONFIRMED` for each of those that is CONFIRMED. A correlation rule muting an owned -fault overlays the stop rather than taking the fault from it. A request asking for the state the switch is already in succeeds, changes -nothing, and writes no audit record; a request the store cannot record answers -`success: false` and changes nothing. Audit records exist only when `audit_log.enabled` -is set, which it is not by default. +usual. Withdrawing the stop *unmutes* every fault it owns whose entry is the stop's own - +a fault a hierarchical rule has since claimed stays muted - and *announces* the subset of +those that is CONFIRMED and that no live cluster is hiding, one `EVENT_CONFIRMED` each. An +auto-cluster rule hides a non-representative member with no entry at all, by suppressing +that member's events on each report, so such a member is unmuted and not announced, and +what is announced is the representative. Afterwards the burst matches one that never met a +planned stop in the muted list, the counts and the audit log, but not in the event stream: +a confirmation that fell inside the stop and behind a cluster is never announced. Cluster +membership is not persisted, so a stop that spanned a restart announces every fault the +store says it owns. A request asking for the state the switch is already in succeeds, +changes nothing, and writes no audit record; a request the store cannot record answers +`success: false` and changes nothing. Audit records exist only when `audit_log.enabled` is +set, which it is not by default. ### GetPlannedStop.srv diff --git a/src/ros2_medkit_msgs/srv/SetPlannedStop.srv b/src/ros2_medkit_msgs/srv/SetPlannedStop.srv index 147d2fcd8..080037c49 100644 --- a/src/ros2_medkit_msgs/srv/SetPlannedStop.srv +++ b/src/ros2_medkit_msgs/srv/SetPlannedStop.srv @@ -26,12 +26,29 @@ # A muted fault is published exactly as a rule-muted symptom is: its EVENT_CONFIRMED and # EVENT_UPDATED are withheld, whichever kind of report produced them, while an # EVENT_CLEARED - the fault healing, or being acknowledged - is published as usual. -# Withdrawing the stop releases every fault it owns, and publishes one EVENT_CONFIRMED -# for each that is CONFIRMED, because that confirmation was never announced. +# Withdrawing the stop does two things that do not cover the same faults. It UNMUTES +# every fault it owns whose `muted_faults` entry is the stop's own, so a fault a +# hierarchical rule has since claimed stays muted and everything else leaves +# `muted_count`. It ANNOUNCES the subset of those that is CONFIRMED and that no live +# cluster is hiding, one EVENT_CONFIRMED each, because that confirmation was never +# announced. # -# A correlation rule muting an owned fault OVERLAYS the stop rather than taking the +# A hierarchical rule muting an owned fault OVERLAYS the stop rather than taking the # fault from it: while the rule holds the fault the withdrawal leaves it alone, and when -# the rule lets go the fault is the stop's again. +# its root cause is acknowledged the fault is the stop's again. +# +# An auto-cluster rule hides without an overlay - it suppresses a non-representative +# member's events on each report and writes no `muted_faults` entry - and the withdrawal +# honours that too: a member an ACTIVE `show_as_single` cluster still hides is unmuted +# and NOT announced, with no entry left in the stop's place. Afterwards the burst matches +# one that never met a planned stop in `muted_faults`, in the counts, in the cluster +# listing and in the audit log, but NOT in the event stream: a confirmation that fell +# inside the stop and behind a cluster is never announced. The member announced is the +# representative, when the stop owned its cycle. `min_count` decides whether a cluster +# forms, not how long it hides: a formed cluster holds the members it formed with until +# the last is acknowledged, while a fault that joins it below the threshold is not one of +# them and is announced like any fault of its own. Cluster membership is not persisted, +# so a stop that spanned a restart announces every fault the store says it owns. # # With `storage_type: sqlite` the declaration survives a restart of the fault manager, # and so does its ownership of the cycles that started inside it; the in-memory backend From ccfdf56ca10dc19b16abe40ef14b45d79173805c Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Fri, 11 Sep 2026 15:02:32 +0200 Subject: [PATCH 12/12] fault_manager: say what a cluster hold is bounded by, and count unmuting apart from announcing A cluster folds in the members it formed with for as long as its window_ms covers their reports; a report after the window starts a new burst, so that fault is announced at the switch-off like any owned fault. The same late report publishes an update with no stop in force, so nothing about the stop causes it. The switch-off releases every fault the store says the stop owns and announces the CONFIRMED ones among them, and the restart integration test now arranges a formed cluster and a PREFAILED fault before the restart to pin both halves. end_planned_stop() returns what it unmuted and what is to be announced as two lists, and the node logs both counts. The design doc scopes the single-thread argument to the paths that touch the ownership sets and names all three forms of process_clear that end an overlay; the message package README names the cluster listing in the parity set. --- docs/api/messages.rst | 10 +- docs/config/fault-manager.rst | 31 ++++-- src/ros2_medkit_fault_manager/README.md | 31 ++++-- .../design/index.rst | 21 ++-- .../correlation/correlation_engine.hpp | 22 +++- .../src/correlation/correlation_engine.cpp | 11 +- .../src/fault_manager_node.cpp | 10 +- .../test_planned_stop_restart.test.py | 99 +++++++++++++++-- .../test/test_correlation_engine.cpp | 104 ++++++++++++------ src/ros2_medkit_msgs/README.md | 10 +- src/ros2_medkit_msgs/srv/SetPlannedStop.srv | 13 ++- 11 files changed, 261 insertions(+), 101 deletions(-) diff --git a/docs/api/messages.rst b/docs/api/messages.rst index 09285bf4f..e6b405ade 100644 --- a/docs/api/messages.rst +++ b/docs/api/messages.rst @@ -341,10 +341,12 @@ NOT announced, with nothing written in the stop's place. Afterwards the burst ma one that never met a planned stop in ``muted_faults``, in the counts, in the cluster listing and in the audit log, but not in the event stream: a confirmation that fell inside the stop and behind a cluster is never announced. What is announced is the -representative, when the stop owned its cycle. Cluster membership is not persisted, -so a stop that spanned a restart announces every fault the store says it owns. The -audit records exist only when ``audit_log.enabled`` is set, which it is not by -default. +representative, when the stop owned its cycle. A cluster hides only the reports that +fall inside its ``window_ms``; one after that starts a new burst, and the fault is +announced at the switch-off like any owned fault. Cluster membership is not +persisted, so a stop that spanned a restart releases every fault the store says it +owns and announces the CONFIRMED ones among them. The audit records exist only when +``audit_log.enabled`` is set, which it is not by default. See :doc:`/config/fault-manager` for the configuration that decides whether the declaration survives a restart and whether the transitions are audited. diff --git a/docs/config/fault-manager.rst b/docs/config/fault-manager.rst index a18381aac..3a089cbdb 100644 --- a/docs/config/fault-manager.rst +++ b/docs/config/fault-manager.rst @@ -690,24 +690,33 @@ where the same fault outside a stop would have been announced if it confirmed be its cluster reached ``min_count``. What the stop withheld, it withholds for good. The representative is announced if the stop owned its cycle; one whose cycle -predates the stop was announced when it confirmed and is not announced again; -members that joined the cluster during the stop stay hidden by it afterwards, and -the one promoted when the representative is acknowledged is heard from again from -that point on. A fault that joins a cluster whose membership has since fallen below -``min_count`` is not one of the members that cluster shows as a single line: it is -announced and updated like any fault of its own. +predates the stop was announced when it confirmed and is not announced again; a +member the cluster is still hiding stays hidden, and the one promoted when the +representative is acknowledged is heard from again from that point on. A fault that +joins a cluster whose membership has since fallen below ``min_count`` is not one of +the members that cluster shows as a single line: it is announced and updated like +any fault of its own. + +*Still hiding* is bounded by the rule's ``window_ms``. A cluster hides the reports +that fall inside its window; a report after it starts a new burst, with the lapsed +cluster dropped and the reporting fault as the representative of a fresh one, so +that fault is no longer folded into the old line and the switch-off announces it +like any owned fault. This is the cluster's own behaviour, not the stop's: with no +stop in force the same late report publishes an update. ``min_count`` gates whether a cluster FORMS, not how long it hides: once formed, a -cluster folds its members into the representative until the last of them is -acknowledged and it dissolves, so a burst that shrinks back below the threshold does -not start announcing its members again. A cluster that never reached ``min_count``, +cluster folds into the representative the members it formed with, for as long as its +window holds them, until the last of them is acknowledged and it dissolves, so a +burst that shrinks back below the threshold does not start announcing its members +again. A cluster that never reached ``min_count``, or one configured without ``show_as_single``, hides nobody and its members are released as usual. The cluster hold does not survive a restart. Ownership is persisted and cluster membership is not, so a cluster only holds faults it formed from reports the running -process saw. If the stop spanned a reboot, the switch-off releases and announces -every fault the store says the stop owns, cluster or no cluster. +process saw. If the stop spanned a reboot, the switch-off releases every fault the +store says the stop owns and announces the CONFIRMED ones among them, cluster or no +cluster. A muted fault is published exactly as a rule-muted symptom is: ``EVENT_CONFIRMED`` and ``EVENT_UPDATED`` are withheld whichever kind of report produced them, while diff --git a/src/ros2_medkit_fault_manager/README.md b/src/ros2_medkit_fault_manager/README.md index bfdc45312..de31326cb 100644 --- a/src/ros2_medkit_fault_manager/README.md +++ b/src/ros2_medkit_fault_manager/README.md @@ -141,22 +141,31 @@ its cluster reached `min_count`. What the stop withheld, it withholds for good. Which member is announced at the switch-off: the representative, if the stop owned its cycle. A representative whose cycle predates the stop was announced when it -confirmed and is not announced again. Members that joined an active cluster during -the stop stay hidden by it afterwards, and the one promoted when the representative -is acknowledged is heard from again from that point on. +confirmed and is not announced again. A member the cluster is still hiding stays +hidden, and the one promoted when the representative is acknowledged is heard from +again from that point on. + +*Still hiding* is bounded by the rule's `window_ms`. A cluster hides the reports that +fall inside its window; a report after it is a new burst - the lapsed cluster is +dropped and the reporting fault becomes the representative of a fresh one - so that +fault is no longer folded into the old line, and the switch-off announces it like any +owned fault. Nothing about the stop causes this: with no stop in force the same late +report publishes an update. `min_count` gates whether a cluster FORMS, not how long it hides. Once formed, a -cluster folds its members into the representative until the last of them is -acknowledged and the cluster dissolves - a burst that shrinks back below the -threshold does not start announcing its members again. A cluster that never reached -`min_count`, or one configured without `show_as_single`, hides nobody, so every -member of it is released as usual. +cluster folds into the representative the members it formed with, for as long as its +window holds them, until the last of them is acknowledged and it dissolves - a burst +that shrinks back below the threshold does not start announcing its members again, +and a fault that JOINS below the threshold was never one of them and is announced and +updated like any fault of its own. A cluster that never reached `min_count`, or one +configured without `show_as_single`, hides nobody, so every member of it is released +as usual. **The cluster hold does not survive a restart.** Ownership is persisted; cluster membership is not. A cluster only holds faults it formed from reports this process -saw, so if the stop spanned a reboot the switch-off releases and announces every -fault the store says the stop owns, cluster or no cluster. Persisting cluster state -is not part of this. +saw, so if the stop spanned a reboot the switch-off releases every fault the store +says the stop owns and announces the CONFIRMED ones among them, cluster or no +cluster. Persisting cluster state is not part of this. **Both transitions are audited** - when the audit log is on. `audit_log.enabled` is `false` by default, and with it off the switch writes no audit row at all. With diff --git a/src/ros2_medkit_fault_manager/design/index.rst b/src/ros2_medkit_fault_manager/design/index.rst index fd46615e9..acc0ca643 100644 --- a/src/ros2_medkit_fault_manager/design/index.rst +++ b/src/ros2_medkit_fault_manager/design/index.rst @@ -298,17 +298,22 @@ what a running switch-off works from; the stored flags are what a STARTUP reads back, and ``clear_planned_stop_owned`` drops them once the switch-off has announced what it released. Several paths move each of them - the report takes a cycle, the constructor restores from the store, an acknowledgement and the switch-off give it -up - and they stay in step because every one of them runs on the node's single -thread: ``rclcpp::spin()`` in ``main.cpp``, no callback group off it, so no two of -them are ever in flight at once. +up - and they stay in step because every one of THOSE runs on the node's single +thread: the constructor before anything is spinning, and the rest as service +callbacks under the ``rclcpp::spin()`` in ``main.cpp``, none of them on a callback +group of its own, so no two are ever in flight at once. (The node does create a +callback group elsewhere - ``SnapshotCapture`` drains one on a thread of its own - +but nothing on it reads or writes either set.) Everything else follows: an owned fault is muted unless a rule's mute overlays it, and the moment the overlay ends the engine -re-asserts the stop's mute (``reassert_planned_stop_mutes``). An overlay ends in one -place only - ``process_clear`` on the root cause, which erases its symptoms' entries. -A window closing does not: ``cleanup_expired`` drops pending root causes and pending -clusters and never touches ``muted_faults_``, so it calls the re-assert to cover the -ownership set rather than to hand anything back. A rule therefore borrows a fault +re-asserts the stop's mute (``reassert_planned_stop_mutes``). An overlay ends when a +cycle ends, which is ``process_clear`` in all three of its forms: on the root cause, +which erases its symptoms' entries; on the symptom itself; and on a symptom the +auto-clear loop takes with its root cause. A window closing is not one of them: +``cleanup_expired`` drops pending root causes and pending clusters and never touches +``muted_faults_``, so it calls the re-assert to cover the ownership set rather than +to hand anything back. A rule therefore borrows a fault rather than taking it, which is what makes the two features compose in both directions: the withdrawal leaves a rule-held fault muted, and a rule that lets go mid-stop hands the fault back instead of dropping it out of the stop for good. diff --git a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/correlation/correlation_engine.hpp b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/correlation/correlation_engine.hpp index 268979890..47f8a8b38 100644 --- a/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/correlation/correlation_engine.hpp +++ b/src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/correlation/correlation_engine.hpp @@ -74,6 +74,18 @@ struct MutedFaultData { bool by_planned_stop{false}; }; +/// What withdrawing a planned stop did. The two are not the same set: every fault the +/// stop alone was muting loses that entry, while one a live cluster is still hiding is +/// unmuted without being announced, because the cluster goes on folding it into its +/// representative's line. +struct EndPlannedStopResult { + /// Fault codes whose suppressed confirmation the caller should publish now. + std::vector to_announce; + + /// How many faults left the muted list, `to_announce` included. + size_t unmuted{0}; +}; + /// Information about an active cluster (for ListFaults response) struct ClusterData { std::string cluster_id; @@ -167,11 +179,11 @@ class CorrelationEngine { void begin_planned_stop(); /// Withdraw the planned stop and release the faults it alone was muting. - /// A fault a rule has since claimed keeps that rule's mute and is NOT - /// returned; nor is one that was cleared while the stop was on. - /// @return the fault codes this call unmuted, so the caller can announce the - /// confirmations that were never published - std::vector end_planned_stop(); + /// A fault a hierarchical rule has since claimed keeps that rule's mute and is + /// neither unmuted nor announced; nor is one that was cleared while the stop was on. + /// A fault a live cluster is hiding is unmuted but not announced. + /// @return what was unmuted and what to announce; see EndPlannedStopResult + EndPlannedStopResult end_planned_stop(); /// Whether a planned stop is currently declared. bool planned_stop_active() const; diff --git a/src/ros2_medkit_fault_manager/src/correlation/correlation_engine.cpp b/src/ros2_medkit_fault_manager/src/correlation/correlation_engine.cpp index 9ae01d2eb..3b3934f21 100644 --- a/src/ros2_medkit_fault_manager/src/correlation/correlation_engine.cpp +++ b/src/ros2_medkit_fault_manager/src/correlation/correlation_engine.cpp @@ -237,13 +237,13 @@ void CorrelationEngine::begin_planned_stop() { planned_stop_active_ = true; } -std::vector CorrelationEngine::end_planned_stop() { +EndPlannedStopResult CorrelationEngine::end_planned_stop() { std::lock_guard lock(mutex_); planned_stop_active_ = false; - std::vector released; - released.reserve(planned_stop_owned_.size()); + EndPlannedStopResult result; + result.to_announce.reserve(planned_stop_owned_.size()); for (const auto & fault_code : planned_stop_owned_) { auto it = muted_faults_.find(fault_code); // A rule's overlay is not the stop's to lift: that fault stays muted, and @@ -255,6 +255,7 @@ std::vector CorrelationEngine::end_planned_stop() { // The stop's entry goes whatever else is true: the cycle it owned is over. const bool held_by_cluster = cluster_hides(fault_code); muted_faults_.erase(it); + ++result.unmuted; // A cluster rule outranks the stop, because it is the narrower promise: the // stop says "not now", the cluster says "this burst is one line, ever". @@ -265,12 +266,12 @@ std::vector CorrelationEngine::end_planned_stop() { // force, and a remembered copy of that verdict would outlive the membership // and the representative it was taken from. if (!held_by_cluster) { - released.push_back(fault_code); + result.to_announce.push_back(fault_code); } } planned_stop_owned_.clear(); - return released; + return result; } bool CorrelationEngine::cluster_hides(const std::string & fault_code) const { diff --git a/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp b/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp index 262e81a98..c2222ae05 100644 --- a/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp +++ b/src/ros2_medkit_fault_manager/src/fault_manager_node.cpp @@ -1260,14 +1260,18 @@ void FaultManagerNode::handle_set_planned_stop( return; } - const auto unmuted = correlation_engine_->end_planned_stop(); + const auto withdrawal = correlation_engine_->end_planned_stop(); // The switch-off is the moment those faults become news. A CONFIRMED one was // never announced - the confirmation happened behind the mute - so publish it // now, once, or the alarm standing on the controller is invisible to every // consumer of the stream. PREFAILED has nothing to announce yet, and a HEALED // one already published its end. - const size_t announced = announce_released(unmuted); + // + // Announcing covers fewer faults than unmuting: a member a live cluster is still + // hiding leaves the muted list here and goes on being folded into its + // representative's line, so it is counted as unmuted and never announced. + const size_t announced = announce_released(withdrawal.to_announce); // The flags go LAST. A crash between the declaration write and this point // leaves owned faults behind an ENDED declaration, which the next startup @@ -1282,7 +1286,7 @@ void FaultManagerNode::handle_set_planned_stop( response->message = "Planned stop withdrawn"; RCLCPP_INFO(get_logger(), "Planned stop withdrawn by '%s': unmuted %zu fault(s), announced %zu confirmation(s)", - request->declared_by.c_str(), unmuted.size(), announced); + request->declared_by.c_str(), withdrawal.unmuted, announced); } void FaultManagerNode::handle_get_planned_stop( diff --git a/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_restart.test.py b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_restart.test.py index 3445fa93c..b056bb046 100644 --- a/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_restart.test.py +++ b/src/ros2_medkit_fault_manager/test/integration/test_planned_stop_restart.test.py @@ -41,15 +41,20 @@ STORAGE_DIR = tempfile.mkdtemp(prefix='planned_stop_restart_') DATABASE_PATH = os.path.join(STORAGE_DIR, 'faults.db') CORRELATION_PATH = os.path.join(STORAGE_DIR, 'correlation.yaml') +THRESHOLDS_PATH = os.path.join(STORAGE_DIR, 'entity_thresholds.yaml') -# A rule whose mute is NOT persisted anywhere, so a restart is where the -# difference between "the stop owns this cycle" and "a rule is muting it" shows. +# Two rules whose state is NOT persisted anywhere, so a restart is where the +# difference between "the stop owns this cycle" and "a rule is holding it" shows. The +# cluster is the harder of the two: it hides a member with no entry at all, and the +# replacement process has no cluster to hide anything with. CORRELATION_RULES = """ correlation: enabled: true patterns: restart_motor_errors: codes: ["MOTOR_RESTART_*"] + restart_valve_errors: + codes: ["VALVE_RESTART_*"] rules: - id: restart_estop_cascade mode: hierarchical @@ -60,6 +65,21 @@ window_ms: 60000 mute_symptoms: true auto_clear_with_root: false + - id: restart_valve_storm + mode: auto_cluster + match: + - pattern: restart_valve_errors + min_count: 2 + window_ms: 60000 + show_as_single: true + representative: first +""" + +# One source whose faults take three FAILED reports to confirm, so a single report +# leaves a fault the stop owns and that has nothing to announce yet. +ENTITY_THRESHOLDS = """ +/slow_node: + confirmation_threshold: -3 """ RULE_ROOT_CODE = 'ESTOP_RESTART_001' @@ -70,6 +90,10 @@ BEFORE_CODE = 'PS_RESTART_BEFORE' SECOND_BEFORE_CODE = 'PS_RESTART_BEFORE_TWO' AFTER_CODE = 'PS_RESTART_AFTER' +CLUSTER_REPRESENTATIVE = 'VALVE_RESTART_A' +CLUSTER_MEMBER = 'VALVE_RESTART_B' +PREFAILED_CODE = 'PS_RESTART_PREFAILED' +SLOW_SOURCE = '/slow_node' def get_coverage_env(): @@ -95,6 +119,8 @@ def generate_test_description(): """Launch a fault manager that launch brings back after it is killed.""" with open(CORRELATION_PATH, 'w') as handle: handle.write(CORRELATION_RULES) + with open(THRESHOLDS_PATH, 'w') as handle: + handle.write(ENTITY_THRESHOLDS) fault_manager_env = get_coverage_env() fault_manager_env['ROS_LOCALHOST_ONLY'] = '1' @@ -110,6 +136,7 @@ def generate_test_description(): 'database_path': DATABASE_PATH, 'confirmation_threshold': -1, 'correlation.config_file': CORRELATION_PATH, + 'entity_thresholds.config_file': THRESHOLDS_PATH, 'snapshots.enabled': False, 'snapshots.rosbag.enabled': False, }], @@ -205,15 +232,30 @@ def _wait_until(self, predicate, message, timeout=30.0): rclpy.spin_once(self.node, timeout_sec=0.05) raise AssertionError(message) - def _report(self, fault_code): + def _report(self, fault_code, source_id='/test_node'): request = ReportFault.Request() request.fault_code = fault_code request.event_type = ReportFault.Request.EVENT_FAILED request.severity = Fault.SEVERITY_ERROR request.description = 'restart test fault' - request.source_id = '/test_node' + request.source_id = source_id return self._call(self.report_client, request) + def _clusters(self): + request = ListFaults.Request() + request.filter_by_severity = False + request.severity = 0 + request.statuses = [] + request.include_muted = True + request.include_clusters = True + return self._call(self.list_client, request).clusters + + def _status_of(self, fault_code): + request = GetFault.Request() + request.fault_code = fault_code + response = self._call(self.get_client, request) + return response.fault.status if response.success else '' + def _set_stop(self, active, *, reason='', declared_by=''): request = SetPlannedStop.Request() request.active = active @@ -262,10 +304,25 @@ def test_a_stop_declared_before_a_restart_still_mutes_after_it(self, fault_manag self._report(BEFORE_CODE) self._report(SECOND_BEFORE_CODE) + + # A cluster formed inside the stop. Its membership lives in the process, so + # this is what a restart cannot bring back. + self._report(CLUSTER_REPRESENTATIVE) + self._report(CLUSTER_MEMBER) + self._wait_until(lambda: len(self._clusters()) == 1, + 'the cluster rule never formed a cluster') + self.assertEqual(CLUSTER_REPRESENTATIVE, self._clusters()[0].representative_code) + + # And a fault the stop owns that has nothing to announce: one FAILED report + # from a source whose threshold needs three. + self._report(PREFAILED_CODE, source_id=SLOW_SOURCE) + self.assertEqual(Fault.STATUS_PREFAILED, self._status_of(PREFAILED_CODE)) + + owned_before_the_restart = (BEFORE_CODE, SECOND_BEFORE_CODE, CLUSTER_REPRESENTATIVE, + CLUSTER_MEMBER, PREFAILED_CODE) muted_before = self._muted_codes() - self.assertIn(BEFORE_CODE, muted_before) - self.assertIn(SECOND_BEFORE_CODE, muted_before) - for code in (BEFORE_CODE, SECOND_BEFORE_CODE): + for code in owned_before_the_restart: + self.assertIn(code, muted_before) self.assertEqual(0, self._count_events(code, FaultEvent.EVENT_CONFIRMED), 'a fault the stop marked was announced anyway') @@ -297,8 +354,13 @@ def test_a_stop_declared_before_a_restart_still_mutes_after_it(self, fault_manag # stand in for one the restore was supposed to find. restored = self._muted_codes() self.assertEqual( - {BEFORE_CODE, SECOND_BEFORE_CODE}, set(restored), + set(owned_before_the_restart), set(restored), 'the restart restored something other than exactly the faults the stop owned') + + # The cluster is gone with the process that formed it. Nothing persists + # membership, and nothing rebuilds it from the store. + self.assertEqual([], list(self._clusters()), + 'a cluster came back from a restart, which nothing persists') # The rule's mute is not persisted, so it is gone after the restart - and the # stop must not have adopted a cycle that started before it was declared. self.assertNotIn(RULE_SYMPTOM_CODE, restored) @@ -331,23 +393,36 @@ def test_a_stop_declared_before_a_restart_still_mutes_after_it(self, fault_manag self.assertGreater(final_state.ended_at.sec, 0) self.assertIn(AFTER_CODE, self._confirmed_codes()) - # Every fault the stop was holding - across the restart - is released and - # announced exactly once. - for code in (BEFORE_CODE, SECOND_BEFORE_CODE, AFTER_CODE): + # Every CONFIRMED fault the stop was holding - across the restart - is released + # and announced exactly once. The cluster member included: the cluster that was + # folding it into one line did not survive the restart, so after a reboot the + # switch-off announces the whole burst. + announced_codes = (BEFORE_CODE, SECOND_BEFORE_CODE, AFTER_CODE, + CLUSTER_REPRESENTATIVE, CLUSTER_MEMBER) + for code in announced_codes: self._wait_until( lambda code=code: self._count_events(code, FaultEvent.EVENT_CONFIRMED) == 1, f'{code} was never announced at the switch-off') self.assertIn(code, self._confirmed_codes()) + # Released is not announced. A fault the stop owned that never reached + # CONFIRMED has nothing to announce, so it leaves the muted list silently and + # goes on debouncing. + self.assertNotIn(PREFAILED_CODE, self._muted_codes()) + self.assertEqual(0, self._count_events(PREFAILED_CODE, FaultEvent.EVENT_CONFIRMED), + 'a fault still short of confirmation was announced at the switch-off') + self.assertEqual(Fault.STATUS_PREFAILED, self._status_of(PREFAILED_CODE)) + self.assertEqual( 0, self._count_events(RULE_SYMPTOM_CODE, FaultEvent.EVENT_CONFIRMED), 'the switch-off announced a fault the stop never owned') # A second frame arriving late would still be a double announcement. self._spin_for(3.0) - for code in (BEFORE_CODE, SECOND_BEFORE_CODE, AFTER_CODE): + for code in announced_codes: self.assertEqual(1, self._count_events(code, FaultEvent.EVENT_CONFIRMED), f'{code} was announced more than once') + self.assertEqual(0, self._count_events(PREFAILED_CODE, FaultEvent.EVENT_CONFIRMED)) @launch_testing.post_shutdown_test() diff --git a/src/ros2_medkit_fault_manager/test/test_correlation_engine.cpp b/src/ros2_medkit_fault_manager/test/test_correlation_engine.cpp index aa84686fe..adb1ef250 100644 --- a/src/ros2_medkit_fault_manager/test/test_correlation_engine.cpp +++ b/src/ros2_medkit_fault_manager/test/test_correlation_engine.cpp @@ -907,7 +907,7 @@ TEST_F(CorrelationEngineTest, EndingPlannedStopUnmutesWhatItMuted) { engine.process_fault("PUMP_SEAL_LEAK", "WARNING"); EXPECT_EQ(2u, engine.get_muted_count()); - auto unmuted = engine.end_planned_stop(); + auto unmuted = engine.end_planned_stop().to_announce; EXPECT_FALSE(engine.planned_stop_active()); std::sort(unmuted.begin(), unmuted.end()); ASSERT_EQ(2u, unmuted.size()); @@ -922,11 +922,11 @@ TEST_F(CorrelationEngineTest, EndingPlannedStopWithNothingMutedReturnsNothing) { CorrelationEngine engine(create_hierarchical_config()); engine.begin_planned_stop(); - auto unmuted = engine.end_planned_stop(); + auto unmuted = engine.end_planned_stop().to_announce; EXPECT_TRUE(unmuted.empty()); // Ending a stop that is already off changes nothing either. - EXPECT_TRUE(engine.end_planned_stop().empty()); + EXPECT_TRUE(engine.end_planned_stop().to_announce.empty()); EXPECT_FALSE(engine.planned_stop_active()); } @@ -934,7 +934,7 @@ TEST_F(CorrelationEngineTest, AFaultReportedAfterTheStopEndsIsNotMuted) { CorrelationEngine engine(create_hierarchical_config()); engine.begin_planned_stop(); - engine.end_planned_stop(); + engine.end_planned_stop().to_announce; auto result = engine.process_fault("VALVE_STUCK", "ERROR"); EXPECT_FALSE(result.should_mute); @@ -955,7 +955,7 @@ TEST_F(CorrelationEngineTest, EndingPlannedStopLeavesARuleMutedSymptomMuted) { EXPECT_TRUE(symptom.should_mute); EXPECT_EQ("estop_cascade", symptom.rule_id); - auto unmuted = engine.end_planned_stop(); + auto unmuted = engine.end_planned_stop().to_announce; EXPECT_EQ(1u, unmuted.size()); EXPECT_EQ("ESTOP_001", unmuted[0]); @@ -985,7 +985,7 @@ TEST_F(CorrelationEngineTest, ARuleTakingOverAStopMutedFaultKeepsItMuted) { EXPECT_TRUE(second.should_mute); EXPECT_EQ("estop_cascade", second.rule_id); - auto unmuted = engine.end_planned_stop(); + auto unmuted = engine.end_planned_stop().to_announce; EXPECT_EQ(1u, unmuted.size()); EXPECT_EQ("ESTOP_001", unmuted[0]); EXPECT_TRUE(engine.is_muted("MOTOR_COMM_FL")); @@ -1003,7 +1003,7 @@ TEST_F(CorrelationEngineTest, ClearingDuringAStopTakesTheFaultOutOfTheMute) { EXPECT_EQ(0u, engine.get_muted_count()); // Nothing survives to be unmuted, so the switch-off announces nothing. - EXPECT_TRUE(engine.end_planned_stop().empty()); + EXPECT_TRUE(engine.end_planned_stop().to_announce.empty()); } TEST_F(CorrelationEngineTest, ClearingARootCauseDuringAStopDropsItsStopMutedSymptoms) { @@ -1020,7 +1020,7 @@ TEST_F(CorrelationEngineTest, ClearingARootCauseDuringAStopDropsItsStopMutedSymp EXPECT_FALSE(engine.is_muted("MOTOR_COMM_FL")); // Both codes left the mute with the clear; the switch-off has nothing to say. - EXPECT_TRUE(engine.end_planned_stop().empty()); + EXPECT_TRUE(engine.end_planned_stop().to_announce.empty()); } TEST_F(CorrelationEngineTest, PlannedStopMutesWithoutAnyRulesConfigured) { @@ -1031,7 +1031,7 @@ TEST_F(CorrelationEngineTest, PlannedStopMutesWithoutAnyRulesConfigured) { EXPECT_TRUE(result.should_mute); EXPECT_TRUE(engine.is_muted("VALVE_STUCK")); - auto unmuted = engine.end_planned_stop(); + auto unmuted = engine.end_planned_stop().to_announce; ASSERT_EQ(1u, unmuted.size()); EXPECT_EQ("VALVE_STUCK", unmuted[0]); } @@ -1054,7 +1054,7 @@ TEST_F(CorrelationEngineTest, AStillActiveReReportIsNotTakenByTheStop) { /*cycle_started=*/false); EXPECT_FALSE(result.should_mute); EXPECT_FALSE(engine.is_muted("PUMP_SEAL_LEAK")); - EXPECT_TRUE(engine.end_planned_stop().empty()) << "the stop released a fault it never muted"; + EXPECT_TRUE(engine.end_planned_stop().to_announce.empty()) << "the stop released a fault it never muted"; } TEST_F(CorrelationEngineTest, ACycleStartingInsideTheStopIsTakenByIt) { @@ -1066,7 +1066,7 @@ TEST_F(CorrelationEngineTest, ACycleStartingInsideTheStopIsTakenByIt) { EXPECT_TRUE(result.should_mute); EXPECT_TRUE(engine.is_muted("PUMP_SEAL_LEAK")); - auto unmuted = engine.end_planned_stop(); + auto unmuted = engine.end_planned_stop().to_announce; ASSERT_EQ(1u, unmuted.size()); EXPECT_EQ("PUMP_SEAL_LEAK", unmuted[0]); } @@ -1085,7 +1085,7 @@ TEST_F(CorrelationEngineTest, AStillActiveReReportDoesNotReleaseAStopMute) { EXPECT_TRUE(again.should_mute); EXPECT_TRUE(engine.is_muted("VALVE_STUCK")); - auto unmuted = engine.end_planned_stop(); + auto unmuted = engine.end_planned_stop().to_announce; ASSERT_EQ(1u, unmuted.size()); EXPECT_EQ("VALVE_STUCK", unmuted[0]); } @@ -1112,7 +1112,7 @@ TEST_F(CorrelationEngineTest, ARuleMuteOverlaysTheStopsWithoutReplacingOwnership ASSERT_EQ(1u, muted.size()); EXPECT_EQ("estop_cascade", muted[0].rule_id) << "the stop overwrote a rule's mute"; - EXPECT_TRUE(engine.end_planned_stop().empty()); + EXPECT_TRUE(engine.end_planned_stop().to_announce.empty()); EXPECT_TRUE(engine.is_muted("MOTOR_COMM_FL")); } @@ -1141,7 +1141,7 @@ TEST_F(CorrelationEngineTest, TheClusterKeepsHidingTheSymptomWhenTheStopEnds) { // The switch-off releases the representative and no more: releasing the rest // would announce, in one wave, the burst the cluster exists to fold into a // single line. - auto unmuted = engine.end_planned_stop(); + auto unmuted = engine.end_planned_stop().to_announce; ASSERT_EQ(1u, unmuted.size()) << "the stop announced a fault the cluster is still hiding"; EXPECT_EQ("VALVE_A", unmuted[0]) << "the representative is what the cluster shows"; @@ -1164,7 +1164,7 @@ TEST_F(CorrelationEngineTest, EveryNonRepresentativeOfALiveClusterIsHeldAtTheSwi engine.process_fault(code, "ERROR", t0, /*cycle_started=*/true); } - auto unmuted = engine.end_planned_stop(); + auto unmuted = engine.end_planned_stop().to_announce; ASSERT_EQ(1u, unmuted.size()) << "a burst of four announced more than the one line the cluster shows"; EXPECT_EQ("VALVE_A", unmuted[0]); EXPECT_EQ(0u, engine.get_muted_count()); @@ -1188,7 +1188,7 @@ TEST_F(CorrelationEngineTest, AClusterBehavesTheSameAfterAStopAsItDoesWithoutOne engine.process_fault("VALVE_A", "ERROR", t0, /*cycle_started=*/true); engine.process_fault("VALVE_B", "ERROR", t0 + 10ms, /*cycle_started=*/true); if (with_stop) { - engine.end_planned_stop(); + engine.end_planned_stop().to_announce; } std::vector observed; @@ -1220,7 +1220,7 @@ TEST_F(CorrelationEngineTest, AClusterShortOfMinCountHidesNothingAtTheSwitchOff) // min_count is 2, so one member is not a cluster and hides nobody. engine.process_fault("VALVE_A", "ERROR", t0, /*cycle_started=*/true); - auto unmuted = engine.end_planned_stop(); + auto unmuted = engine.end_planned_stop().to_announce; ASSERT_EQ(1u, unmuted.size()) << "a cluster that never formed withheld a confirmation"; EXPECT_EQ("VALVE_A", unmuted[0]); EXPECT_EQ(0u, engine.get_muted_count()); @@ -1252,7 +1252,7 @@ TEST_F(CorrelationEngineTest, AClusterWithoutShowAsSingleReleasesEveryMember) { engine.process_fault("VALVE_B", "ERROR", t0 + 10ms, /*cycle_started=*/true); // A cluster that groups without hiding has no verdict to outrank the stop with. - auto unmuted = engine.end_planned_stop(); + auto unmuted = engine.end_planned_stop().to_announce; std::sort(unmuted.begin(), unmuted.end()); ASSERT_EQ(2u, unmuted.size()) << "a cluster that hides nobody withheld a confirmation"; EXPECT_EQ("VALVE_A", unmuted[0]); @@ -1294,7 +1294,7 @@ TEST_F(CorrelationEngineTest, AShrunkenActiveClusterStillHoldsItsMemberAtTheSwit engine.process_clear("VALVE_C"); - auto unmuted = engine.end_planned_stop(); + auto unmuted = engine.end_planned_stop().to_announce; ASSERT_EQ(1u, unmuted.size()) << "a cluster below min_count announced its whole burst"; EXPECT_EQ("VALVE_A", unmuted[0]); EXPECT_TRUE(engine.process_fault("VALVE_B", "ERROR", t0 + 30ms, /*cycle_started=*/false).should_mute); @@ -1316,7 +1316,7 @@ TEST_F(CorrelationEngineTest, AcknowledgingTheRepresentativeAfterItsWindowClosed engine.cleanup_expired(); engine.process_clear("VALVE_A"); - auto unmuted = engine.end_planned_stop(); + auto unmuted = engine.end_planned_stop().to_announce; ASSERT_EQ(1u, unmuted.size()) << "the cluster kept naming the acknowledged fault, so nobody was released"; EXPECT_EQ("VALVE_B", unmuted[0]) << "the promoted representative is the one the switch-off announces"; EXPECT_EQ(0u, engine.get_muted_count()); @@ -1374,7 +1374,7 @@ TEST_F(CorrelationEngineTest, ReleasingOneFaultTakesOnlyTheStopsOwnMute) { engine.release_planned_stop_ownership("MOTOR_COMM_FL"); EXPECT_TRUE(engine.is_muted("MOTOR_COMM_FL")); - auto unmuted = engine.end_planned_stop(); + auto unmuted = engine.end_planned_stop().to_announce; ASSERT_EQ(1u, unmuted.size()); EXPECT_EQ("ESTOP_001", unmuted[0]); } @@ -1393,7 +1393,7 @@ TEST_F(CorrelationEngineTest, RestoredOwnershipBehavesLikeOwnershipTakenLive) { const auto muted = engine.get_muted_faults(); EXPECT_EQ(CorrelationEngine::kPlannedStopRootCause, muted[0].root_cause_code); - auto unmuted = engine.end_planned_stop(); + auto unmuted = engine.end_planned_stop().to_announce; std::sort(unmuted.begin(), unmuted.end()); ASSERT_EQ(2u, unmuted.size()); EXPECT_EQ("PUMP_SEAL_LEAK", unmuted[0]); @@ -1434,7 +1434,7 @@ TEST_F(CorrelationEngineTest, AnOwnedFaultReturnsToTheStopsMuteWhenARuleReleases EXPECT_TRUE(engine.is_muted("MOTOR_COMM_FL")) << "the symptom fell out of the stop when the rule let go"; EXPECT_EQ(CorrelationEngine::kPlannedStopRuleId, rule_id_of("MOTOR_COMM_FL")); - auto unmuted = engine.end_planned_stop(); + auto unmuted = engine.end_planned_stop().to_announce; ASSERT_EQ(1u, unmuted.size()); EXPECT_EQ("MOTOR_COMM_FL", unmuted[0]); } @@ -1457,7 +1457,11 @@ TEST_F(CorrelationEngineTest, ARuleThatStopsMatchingLeavesTheFaultMuted) { EXPECT_TRUE(engine.is_muted("MOTOR_COMM_FL")); } -TEST_F(CorrelationEngineTest, AClusterMemberStaysMutedWhenTheClusterWindowExpires) { +// The stop keeps its own entry on a member whose cluster window has lapsed, so the +// member is still reported as muted while the stop stands. What it is no longer is +// HIDDEN BY THE CLUSTER: the lapsed report started a burst of its own, so the +// switch-off announces both faults, not just the representative. +TEST_F(CorrelationEngineTest, AClusterMemberWhoseWindowLapsedIsAnnouncedAtTheSwitchOff) { CorrelationEngine engine(create_cluster_config()); auto t0 = std::chrono::steady_clock::now(); @@ -1466,15 +1470,46 @@ TEST_F(CorrelationEngineTest, AClusterMemberStaysMutedWhenTheClusterWindowExpire engine.process_fault("VALVE_B", "ERROR", t0 + 10ms, /*cycle_started=*/true); ASSERT_TRUE(engine.is_muted("VALVE_B")); - // The cluster's window closes; the stop still owns both cycles. + // The periodic sweep drops the lapsed pending cluster; the stop still owns both + // cycles and is still what mutes them. engine.cleanup_expired(); auto repeat = engine.process_fault("VALVE_B", "ERROR", t0 + 120000ms, /*cycle_started=*/false); - EXPECT_TRUE(repeat.should_mute); + EXPECT_TRUE(repeat.should_mute) << "the stop's own entry stopped muting the fault"; EXPECT_TRUE(engine.is_muted("VALVE_B")); - auto unmuted = engine.end_planned_stop(); - std::sort(unmuted.begin(), unmuted.end()); - ASSERT_EQ(2u, unmuted.size()); + auto announced = engine.end_planned_stop().to_announce; + std::sort(announced.begin(), announced.end()); + ASSERT_EQ(2u, announced.size()) << "the lapsed member was withheld by a burst it is no longer part of"; + EXPECT_EQ("VALVE_A", announced[0]); + EXPECT_EQ("VALVE_B", announced[1]); +} + +// A cluster hides the reports that fall inside its window. A report after the window +// is a new burst - try_auto_cluster drops the lapsed pending cluster and starts one +// with this fault as its own representative - so the fault is no longer folded into +// the old line, and the switch-off announces it like any owned fault. That is the +// cluster's own contract rather than anything the stop does: with no stop in force +// the same late report publishes an update. +TEST_F(CorrelationEngineTest, AReportAfterTheClusterWindowStartsANewBurstAndIsAnnounced) { + auto announced_when_the_member_repeats_at = [this](std::chrono::milliseconds offset) { + CorrelationEngine engine(create_cluster_config()); // window_ms 60000, min_count 2 + auto t0 = std::chrono::steady_clock::now(); + engine.begin_planned_stop(); + engine.process_fault("VALVE_A", "ERROR", t0, /*cycle_started=*/true); + engine.process_fault("VALVE_B", "ERROR", t0 + 10ms, /*cycle_started=*/true); + engine.process_fault("VALVE_B", "ERROR", t0 + offset, /*cycle_started=*/false); + auto announced = engine.end_planned_stop().to_announce; + std::sort(announced.begin(), announced.end()); + return announced; + }; + + // Inside the window the cluster is still folding B into A's line. + const std::vector representative_only{"VALVE_A"}; + EXPECT_EQ(representative_only, announced_when_the_member_repeats_at(20ms)); + + // Past it, B's report is a burst of its own and B is announced too. + const std::vector both{"VALVE_A", "VALVE_B"}; + EXPECT_EQ(both, announced_when_the_member_repeats_at(120000ms)); } TEST_F(CorrelationEngineTest, OwnershipSurvivesARuleOverlayAndIsReadableBack) { @@ -1503,7 +1538,7 @@ TEST_F(CorrelationEngineTest, AFaultWhoseCycleStartedBeforeTheStopIsNeverOwned) engine.process_fault("MOTOR_COMM_FL", "ERROR", t0 + 20ms, /*cycle_started=*/false); EXPECT_TRUE(engine.planned_stop_owned_codes().empty()); - EXPECT_TRUE(engine.end_planned_stop().empty()); + EXPECT_TRUE(engine.end_planned_stop().to_announce.empty()); EXPECT_TRUE(engine.is_muted("MOTOR_COMM_FL")) << "the rule's mute was collateral damage"; } @@ -1517,7 +1552,7 @@ TEST_F(CorrelationEngineTest, ClearingAnOwnedFaultEndsTheStopsOwnershipOfIt) { engine.process_clear("VALVE_STUCK"); EXPECT_TRUE(engine.planned_stop_owned_codes().empty()); EXPECT_FALSE(engine.is_muted("VALVE_STUCK")); - EXPECT_TRUE(engine.end_planned_stop().empty()); + EXPECT_TRUE(engine.end_planned_stop().to_announce.empty()); } TEST_F(CorrelationEngineTest, AutoClearedSymptomsLeaveTheStopsOwnership) { @@ -1535,7 +1570,7 @@ TEST_F(CorrelationEngineTest, AutoClearedSymptomsLeaveTheStopsOwnership) { // Both cycles ended with the acknowledgement, so neither is the stop's any more. EXPECT_TRUE(engine.planned_stop_owned_codes().empty()); EXPECT_FALSE(engine.is_muted("MOTOR_COMM_FL")); - EXPECT_TRUE(engine.end_planned_stop().empty()); + EXPECT_TRUE(engine.end_planned_stop().to_announce.empty()); } // `fault_to_cluster_` is written on every join, including a join to a PENDING cluster @@ -1579,7 +1614,7 @@ TEST_F(CorrelationEngineTest, AFaultJoiningAClusterBelowMinCountIsReleasedAtTheS engine.process_clear("VALVE_D"); engine.process_fault("VALVE_E", "ERROR", t0 + 10ms, /*cycle_started=*/true); - auto unmuted = engine.end_planned_stop(); + auto unmuted = engine.end_planned_stop().to_announce; std::sort(unmuted.begin(), unmuted.end()); ASSERT_EQ(2u, unmuted.size()) << "the fault that joined below min_count was withheld by a cluster it is not in"; EXPECT_EQ("VALVE_A", unmuted[0]) << "the representative"; @@ -1611,13 +1646,14 @@ TEST_F(CorrelationEngineTest, ARestartReleasesEveryOwnedFaultIncludingOnesAClust after_restart.restore_planned_stop_ownership(code); } - auto unmuted = after_restart.end_planned_stop(); + auto unmuted = after_restart.end_planned_stop().to_announce; std::sort(unmuted.begin(), unmuted.end()); ASSERT_EQ(2u, unmuted.size()) << "a restart is expected to release the whole burst; the cluster is gone with the " "process that formed it"; EXPECT_EQ("VALVE_A", unmuted[0]); EXPECT_EQ("VALVE_B", unmuted[1]); } + int main(int argc, char ** argv) { ::testing::InitGoogleTest(&argc, argv); return RUN_ALL_TESTS(); diff --git a/src/ros2_medkit_msgs/README.md b/src/ros2_medkit_msgs/README.md index ad78cae98..d78e98f9c 100644 --- a/src/ros2_medkit_msgs/README.md +++ b/src/ros2_medkit_msgs/README.md @@ -172,10 +172,12 @@ those that is CONFIRMED and that no live cluster is hiding, one `EVENT_CONFIRMED auto-cluster rule hides a non-representative member with no entry at all, by suppressing that member's events on each report, so such a member is unmuted and not announced, and what is announced is the representative. Afterwards the burst matches one that never met a -planned stop in the muted list, the counts and the audit log, but not in the event stream: -a confirmation that fell inside the stop and behind a cluster is never announced. Cluster -membership is not persisted, so a stop that spanned a restart announces every fault the -store says it owns. A request asking for the state the switch is already in succeeds, +planned stop in the muted list, the counts, the cluster listing and the audit log, but not +in the event stream: a confirmation that fell inside the stop and behind a cluster is never +announced. A cluster hides only the reports that fall inside its `window_ms`; one after +that starts a new burst, and the fault is announced at the switch-off like any owned fault. +Cluster membership is not persisted, so a stop that spanned a restart releases every fault +the store says it owns and announces the CONFIRMED ones among them. A request asking for the state the switch is already in succeeds, changes nothing, and writes no audit record; a request the store cannot record answers `success: false` and changes nothing. Audit records exist only when `audit_log.enabled` is set, which it is not by default. diff --git a/src/ros2_medkit_msgs/srv/SetPlannedStop.srv b/src/ros2_medkit_msgs/srv/SetPlannedStop.srv index 080037c49..d1bc30542 100644 --- a/src/ros2_medkit_msgs/srv/SetPlannedStop.srv +++ b/src/ros2_medkit_msgs/srv/SetPlannedStop.srv @@ -45,10 +45,15 @@ # listing and in the audit log, but NOT in the event stream: a confirmation that fell # inside the stop and behind a cluster is never announced. The member announced is the # representative, when the stop owned its cycle. `min_count` decides whether a cluster -# forms, not how long it hides: a formed cluster holds the members it formed with until -# the last is acknowledged, while a fault that joins it below the threshold is not one of -# them and is announced like any fault of its own. Cluster membership is not persisted, -# so a stop that spanned a restart announces every fault the store says it owns. +# forms, not how long it hides: a formed cluster holds the members it formed with for as +# long as its `window_ms` covers their reports, until the last is acknowledged, while a +# fault that joins it below the threshold is not one of them and is announced like any +# fault of its own. A report that arrives after the window starts a new burst, so that +# fault is no longer folded into the old line and the switch-off announces it like any +# owned fault - the same thing the cluster does with no stop in force, where that late +# report publishes an update. Cluster membership is not persisted, so a stop that spanned +# a restart releases every fault the store says it owns and announces the CONFIRMED ones +# among them. # # With `storage_type: sqlite` the declaration survives a restart of the fault manager, # and so does its ownership of the cycles that started inside it; the in-memory backend