Fix bcp 008#504
Merged
Merged
Conversation
…tatus when the current status is reconfirmed
…e from bad to good, then back to bad. Add edge case test to Tests status transitions from unhealthy (3) -> healthy (1) -> unhealthy (3) within the 3 second report delay window.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The following bug has been reported by Niitsu, Keita-san from Sony.
When a transition to a healthier status is pending and a new event reconfirms that the actual status level has not improved and remains the same as the current status, the pending improvement is not cancelled.
As a result, when the pending timer eventually expires, the system incorrectly applies the healthier status, even though the system is still not actually healthy.
New status is worse than the current status, outside the activation period
New status is worse than the current status, inside the activation period
New status is equal to the current status
Reproduction steps:
Configuration:
Sequence:
Bring NIC1 UP.
The synthesised link status becomes healthy (1).
set_monitor_status_with_delay(status=1) is called.
Since 1 < current_status(3), the function enters the delay branch.
The following pending state is stored:
pending_status = 1
pending_received_time = T
Before the 3-second delay expires, bring NIC1 DOWN again.
The synthesised link status returns to unhealthy (3).
set_monitor_status_with_delay(status=3, message="NIC1 is down") is called.
Since 3 == current_status(3), the function takes the early return path.
Only the status message is updated to "NIC1 is down".
pending_received_time is not reset.
After 3 seconds, the behaviour thread fires.
pending_received_time = T > 0
The delay threshold has been reached.
The behaviour thread applies pending_status = 1.
As a result, monitor status is registered with an incorrect status: healthy (1), and the message, Previously: NIC1 is down.
Expected final state:
linkStatus = unhealthy (3)
message = "NIC1 is down"
Actual final state:
linkStatus = healthy (1)
message = "Previously: NIC1 is down"
Proposed fix:
Reset
status_pending_received_time_field_nameto 0 at the beginning of the status == current_status branch, before any early return.Why this fix is safe:
This change should be safe because pending_received_time becomes non-zero only when there is a pending improvement.
When status == current_status, any existing pending status must represent a healthier state, i.e. a smaller status value.
The new event is re-confirming that the system has not actually reached that healthier state, so the pending transition should be discarded.
If there is no pending transition, meaning pending_received_time is already 0, calling set_property with 0 should effectively be a no-op.