fault_manager: a planned-stop switch, so a weekend is not a wave of faults - #664
Draft
bburda wants to merge 10 commits into
Draft
fault_manager: a planned-stop switch, so a weekend is not a wave of faults#664bburda wants to merge 10 commits into
bburda wants to merge 10 commits into
Conversation
SetPlannedStop declares or withdraws a planned stop and answers with the state the switch was in before the call; GetPlannedStop reads back the declaration, its reason, its declarer and when it started. Refs #656
…opping them The correlation engine gains a second mute source. While a planned stop is declared, a fault whose cycle starts is reported, debounced, confirmed, captured and audited exactly as it would be otherwise, and is registered as muted with rule_id 'planned_stop' and the pseudo root cause 'PLANNED_STOP': absent from the default fault list, counted in muted_count, and never announced on the event stream. A rule that mutes the same code takes the mute over, and withdrawing the stop then leaves it alone. Withdrawing releases every fault the stop alone was holding down and publishes one EVENT_CONFIRMED for each of those that is CONFIRMED, because that confirmation was never announced. Both transitions are recorded in the audit log with their reason and declarer, whatever audit_log.transitions says; a request for the state the switch is already in changes nothing and records nothing. The declaration is stored, so a stop declared before a restart is still in force after it. The correlation engine is now always constructed, with no rules when none are configured, because the switch needs no configuration. Refs #656
Engine unit cases pin the two mute sources composing: a rule takes a mute over from the switch and keeps it at switch-off, and a fault cleared inside the stop leaves both maps. Storage cases pin the declaration surviving a reopen on SQLite and living for the process on the memory backend, and the audit case pins a transition with no fault code chaining and staying tamper-evident. Three launch tests drive a live manager: the main one covers marking, counting, capture, the audit rows, idempotent requests, an empty and a kilobyte reason, 200 faults released by one switch-off, a fault mid-debounce when the stop ends, and a rule-muted symptom the switch must not release; one kills the manager and lets launch bring it back; one drives a confirmation through the auto-confirm timer with a control fault proving the timer fires at all. The gateway feature test is the acceptance: it turns the stop on and off through POST /apps/fault_manager/operations/set_planned_stop/executions, watches GET /faults and the SSE fault stream, and counts confirmations per fault code either side of the switch-off. Refs #656
Covers what marking means and what it does not change, how the switch and a correlation rule compose, which two configuration choices decide whether the declaration survives a restart and whether transitions are audited, and how an operator reaches the switch over SOVD as an operation on the fault manager entity - including that hybrid and manifest discovery need the entity declared before that address exists. Refs #656
Reporters are level-triggered: FaultReporter and the diagnostic bridge send FAILED for as long as the condition holds, not once per transition. Muting on any report therefore took a fault that was confirmed and announced BEFORE the stop off the fault list within a second of the declaration, and announced it a second time at the switch-off. The mark is now gated on the cycle boundary the report path already computes: a new fault, or one raised again after a clear. A repeat report of a fault the stop already marked changes nothing. One mute now has one owner. The hand-over to a rule happens where the rule registers its mute, not where any path reports should_mute: the auto-cluster path sets that flag without ever writing the mute map, so the old hand-over left the stop's entry owned by nobody - muted for good, released by nothing. In the other direction the stop no longer overwrites an entry a rule holds. An acknowledgement that skips the correlation cascade, which is every scoped per-entity clear, now also releases the stop's mute. The mark survives a restart: with a declaration still in force the manager re-registers it for every stored fault that is not CLEARED and whose cycle started at or after the declaration, so the switch-off after a reboot still releases and announces them. A store that refuses the write ends the request with success=false instead of taking the process down, and the declaration is written before anything is muted, unmuted, announced or audited. GetPlannedStop keeps serving the declaration after the withdrawal, stamped with ended_at, so the reason stays readable once the plant is back up. The audit rows carry the same __audit__ fault code the log's own lifecycle markers use. Refs #656
…anged The silence assertion now reads the exact set of event types published per fault code, and the scenario re-reports a marked fault so that an update leaking past the mute is a case the test can actually see - counting confirmations alone could not. The exactly-once assertions wait out a settle window after the counts are right, because a duplicate published a moment later satisfies a poll and is never seen. New cases: a fault already up when the stop begins stays visible and is announced once, over the services and over the gateway; a fault raised again after a clear is marked; a scoped acknowledgement releases the mute; a cluster rule does not orphan it; the declaration is readable after the withdrawal; a store that refuses the write is reported rather than fatal; the audit rows carry the installation sentinel. Docs on every surface that described the muting: what a cycle is, that publication equals a rule-muted symptom exactly (CLEARED included), that the audit rows exist only when the audit log is on, that the declaration outlives the stop, and that the correlation cleanup timer now runs on every manager. Refs #656
It repeated the claim that a marked fault is never published as a FaultEvent, and listed a GetPlannedStop response that no longer matches the service. Refs #656
The stop marked faults by writing a mute entry and remembering the codes in a set. That set was the only record, so it could not survive a restart intact, a cluster rule could strand an entry it never wrote, and a rule releasing a fault mid-stop dropped it out of the stop for good - its confirmation announced by nobody, which is the failure marking exists to prevent. Ownership is now a flag on the fault row: the stop owns a CYCLE that starts while it is declared, for the whole cycle. The mute is derived from that. A rule's mute overlays an owned fault instead of taking it, and when the overlay ends - root cause acknowledged, window closed, cluster expired - the engine re-asserts the stop's mute. Ownership ends when the fault is acknowledged or the stop is withdrawn, and a restart reads the flags back rather than comparing timestamps, which a clock step would break and which cannot tell a rule's mute from the stop's. A cycle now also starts when a healed fault fails again: the manager publishes EVENT_CLEARED at the heal, so the confirmation that follows is fresh news. occurrence_count keeps its own, narrower definition. Muting gates the PASSED path too. should_mute was computed for FAILED reports only, so a PASSED that left a muted fault CONFIRMED announced an update for a fault the list was hiding; and it was computed from what the report matched rather than from the mute map, so a repeat report of a fault whose rule had stopped matching did the same. The withdrawal writes the declaration, announces, and clears the flags last, so a process that dies in between leaves faults owned by an ended declaration - which the next startup recognises, announces once a consumer is listening, and clears. Clearing first would have turned a crash into permanent silence. The storage-injection seam is a protected constructor rather than a public runtime swap: the backend is in place before snapshot and rosbag capture borrow a raw pointer to it. Refs #656
The ordering that decides what a crash mid-release costs was only reasoned about: the interrupted-release test starts from the state such a crash leaves and cannot see how the state came about. A storage double now records the switch-off's own calls, so clearing the ownership flags before the announcement fails a test instead of passing one. The cleanup timer's cost is measured rather than asserted: ~190 ns per call on a rules-free engine, ~2.3 us of CPU per minute at the 5 s default. Docs on every surface follow the ownership model - what a cycle is, that a rule's mute overlays ownership rather than taking it and hands the fault back when it ends, that the flags and the declaration survive a restart while a rule's mute does not, and that an interrupted switch-off is finished at startup. Two stale sentences went with them: the audit header still described an empty fault code, and SetPlannedStop still said the reason is served only while the stop is on. Refs #656
…tructor Deferring it to a wall timer left a window: the services exist by then, so an operator could declare a new stop and new cycles could become owned by it, and the deferred work would announce a fault the new stop was holding and clear every ownership flag - the new stop's included. The recovery now runs synchronously in the constructor, before this node has a service or a timer that could run, and drops exactly the flags it captured; the storage gains a scoped release for that. The price is an announcement that may reach no subscriber, because nothing has matched a publisher that is milliseconds old and the events topic is volatile. That is the same accepted loss as any other publication made at startup, and the released faults are in the default fault list either way - which is what the launch test asserts, alongside the manager's own log line. Refs #656
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
Summary
A plant stops on purpose: a weekend, a changeover, a maintenance slot. Every controller the fault manager watches goes quiet at once, and there was no way to tell it that this was expected. With a debounce set and healing on, every one of those faults confirms and heals, every transition is an audit row, and the first Monday starts with a fault list nobody can read.
A planned stop is a switch on the fault manager, and the operator's interface owns the calendar.
~/set_planned_stop {active, reason, declared_by}and~/get_planned_stopare ROS services, so they are already SOVD operations on the fault manager entity:POST /apps/fault_manager/operations/set_planned_stop/executions, behind the operations routes' existing role. No new route, no new DTO, no change to any released message or service type, no gateway source file touched; the served OpenAPI document is byte-identical before and after.Marked, not suppressed. While the switch is on, a fault whose cycle starts (a new code, a reactivation from CLEARED, or a re-fail from HEALED) is recorded, debounced, captured and audited exactly as any other fault, and is muted the way a correlated symptom is muted: out of the default fault list, counted in
muted_count, visible withinclude_muted, its confirmation and updates withheld from the event topic (a clear or heal is published as it is for any muted fault). The bridges report one FAILED per transition, so dropping such a fault would lose it for as long as the alarm stands on the controller; marking keeps it.Ownership is a persisted fact. The stop owns the cycles that started inside it, as a flag on the fault row in the store; the correlation engine derives its mute map from that ownership plus any rule that overlays it, and when the rule's mute ends the stop's mute is back. A restart inside a stop keeps every owned fault muted. Switching off writes the declaration, announces one confirmation per CONFIRMED owned fault, then clears the flags, and a start-up that finds flags behind a withdrawn declaration finishes that release before it serves anything. Every switch transition is an audit row with the reason and the declarer (with
audit_log.enabled), andget_planned_stopserves the last declaration after the withdrawal.Issue
Type
One observable change on a fault manager with no correlation rules: the engine and its cleanup timer now always exist, so
muted_countandcluster_countare emitted as 0 and the timer costs about 190 ns per tick at the 5 s default, measured.Testing
include_muted, nothing on the stream for them, a fault confirmed before the stop untouched by it, one confirmation per released fault after switch-off with a settle window proving no second frame.Checklist