fix(notify): recover unfinished queue claims - #813
Open
somethingwithproof wants to merge 16 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens Thold’s notification queue worker ownership model so queue rows aren’t stranded behind dead process registrations, and so concurrent workers cannot drain each other’s claimed work. It adds bounded process registration with Unix liveness probing, introduces explicit claim/release helpers, and expands unit coverage and fixtures to validate the behavior in isolation.
Changes:
- Add bounded, fail-closed notification worker registration with Unix liveness probing and stale-owner recovery.
- Fence queue drains to a worker PID via explicit claim/run/release helpers and ensure cleanup on shutdown paths.
- Expand unit tests/fixtures and patch-coverage tooling; document ownership/recovery behavior.
Reviewed changes
Copilot reviewed 11 out of 13 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| thold_notify.php | Switch child worker startup to bounded registration + scoped run helper; add shutdown cleanup hook. |
| thold_functions.php | Add process liveness/registration helpers, claim/release/run helpers, and enforce fail-closed draining by PID. |
| tests/Unit/NotificationQueueClaimTest.php | Add regression tests covering fail-closed drain, orphan recovery, bounded registration, and cleanup guarantees. |
| tests/Unit/ThresholdTimeBasedCharacterizationTest.php | Assert maintenance fixture isolation stays within the test fixture tree. |
| tests/TestCase.php | Restore modified global base_path in tearDown to avoid cross-test leakage. |
| tests/Helpers/ThresholdScenario.php | Repoint maintenance fixture base_path to a tracked cacti-root fixture. |
| tests/Helpers/CactiStubs.php | Docblock alignment tweaks for recorded-call helper. |
| tests/fixtures/cacti-root/plugins/thold/includes/arrays.php | Provide fixture path shape matching a Cacti install for loading plugin arrays. |
| tests/fixtures/cacti-root/plugins/maint/functions.php | Add empty maint plugin fixture file for include_once() behavior. |
| tests/bootstrap-unit.php | Preserve original base_path and add stubs for process registration helpers used by new code/tests. |
| tests/bin/patch-coverage.php | Exclude tests/ from production patch coverage accounting and require allowlist for unmeasured prod PHP entry points. |
| README.md | Document notification queue ownership and orphan-claim recovery behavior. |
| CHANGELOG.md | Add entry for issue #812 queue claim recovery + scoped drain behavior. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
* Retry failed queued email notifications * test: cover queued delivery retry paths * style: document notification fixture parameters * fix: batch grouped notification delivery updates * fix: align notification retry status output * refactor: centralize notification retry updates * fix: retain retry queue ownership
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A notify run that died mid-drain left its rows claimed, and nothing ever picked them up again.
The claim is now taken after process registration and only over rows nobody holds, the drain is scoped to the claiming process, and a run that cannot determine whether a peer is alive stands down instead of draining alongside it.
Closes #812.