Skip to content

fix(notify): recover unfinished queue claims - #813

Open
somethingwithproof wants to merge 16 commits into
developfrom
fix/notification-claim-recovery
Open

fix(notify): recover unfinished queue claims#813
somethingwithproof wants to merge 16 commits into
developfrom
fix/notification-claim-recovery

Conversation

@somethingwithproof

@somethingwithproof somethingwithproof commented Aug 18, 2026

Copy link
Copy Markdown
Member

A notify run that died mid-drain left its rows claimed, and nothing ever picked them up again.

The claim is now taken after process registration and only over rows nobody holds, the drain is scoped to the claiming process, and a run that cannot determine whether a peer is alive stands down instead of draining alongside it.

Closes #812.

@somethingwithproof somethingwithproof self-assigned this Aug 18, 2026
@somethingwithproof
somethingwithproof marked this pull request as ready for review August 18, 2026 04:53
Copilot AI lite review requested due to automatic review settings August 18, 2026 04:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens Thold’s notification queue worker ownership model so queue rows aren’t stranded behind dead process registrations, and so concurrent workers cannot drain each other’s claimed work. It adds bounded process registration with Unix liveness probing, introduces explicit claim/release helpers, and expands unit coverage and fixtures to validate the behavior in isolation.

Changes:

  • Add bounded, fail-closed notification worker registration with Unix liveness probing and stale-owner recovery.
  • Fence queue drains to a worker PID via explicit claim/run/release helpers and ensure cleanup on shutdown paths.
  • Expand unit tests/fixtures and patch-coverage tooling; document ownership/recovery behavior.

Reviewed changes

Copilot reviewed 11 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
thold_notify.php Switch child worker startup to bounded registration + scoped run helper; add shutdown cleanup hook.
thold_functions.php Add process liveness/registration helpers, claim/release/run helpers, and enforce fail-closed draining by PID.
tests/Unit/NotificationQueueClaimTest.php Add regression tests covering fail-closed drain, orphan recovery, bounded registration, and cleanup guarantees.
tests/Unit/ThresholdTimeBasedCharacterizationTest.php Assert maintenance fixture isolation stays within the test fixture tree.
tests/TestCase.php Restore modified global base_path in tearDown to avoid cross-test leakage.
tests/Helpers/ThresholdScenario.php Repoint maintenance fixture base_path to a tracked cacti-root fixture.
tests/Helpers/CactiStubs.php Docblock alignment tweaks for recorded-call helper.
tests/fixtures/cacti-root/plugins/thold/includes/arrays.php Provide fixture path shape matching a Cacti install for loading plugin arrays.
tests/fixtures/cacti-root/plugins/maint/functions.php Add empty maint plugin fixture file for include_once() behavior.
tests/bootstrap-unit.php Preserve original base_path and add stubs for process registration helpers used by new code/tests.
tests/bin/patch-coverage.php Exclude tests/ from production patch coverage accounting and require allowlist for unmeasured prod PHP entry points.
README.md Document notification queue ownership and orphan-claim recovery behavior.
CHANGELOG.md Add entry for issue #812 queue claim recovery + scoped drain behavior.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread thold_functions.php
Comment thread thold_functions.php Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug daemon Related to thold daemon notification

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Notification claims can strand rows and child drains remain unscoped

2 participants