Skip to content

fix: reconcile recurring Action Scheduler actions safely#2895

Merged
chubes4 merged 9 commits into
mainfrom
investigate/sqlite-database-bloat
Jul 13, 2026
Merged

fix: reconcile recurring Action Scheduler actions safely#2895
chubes4 merged 9 commits into
mainfrom
investigate/sqlite-database-bloat

Conversation

@chubes4

@chubes4 chubes4 commented Jul 13, 2026

Copy link
Copy Markdown
Member

Fixes #2892

Root Cause

Recurring Data Machine actions were scheduled with Action Scheduler uniqueness disabled. Concurrent activation, upgrade, cron, or bootstrap reconciliation could create parallel recurring chains. Later reconciliation cancelled those chains while queue runners held claims, generating repeated action ignored via Async Request rows against canceled datamachine_recurring_* actions.

Behavior

  • Recurring and cron replacement actions now request Action Scheduler uniqueness.
  • A missing/canceled slot is restored directly; cancellation occurs only when an actual pending conflicting chain exists.
  • Reconciliation remains immediate on action_scheduler_init for activation and upgrades, and also subscribes to Action Scheduler's native action_scheduler_ensure_recurring_actions repair hook.
  • The existing bounded retention path is covered for expired logs attached to canceled Data Machine recurrence actions. It preserves recent canceled logs and non-Data-Machine pending-action logs.

Operational Impact

Existing duplicate pending chains self-heal once into one unique chain. The next reconciliation recreates missing enabled recurrences. The retention task can then remove the 108,439,201 expired Action Scheduler logs observed in #2892 in its existing bounded batches.

The affected table and indexes occupy 19,597,930,496 bytes. Based on the expired-row proportion, cleanup makes approximately 19,349,289,931 bytes (18.02 GiB) reusable. SQLite file size is reclaimed only by a separately approved maintenance compaction after cleanup.

After deployment, an operator should first preview the one-time cleanup:

wp datamachine retention run --dry-run

After reviewing that output, schedule it with:

wp datamachine retention run --yes

How To Test

  1. Run composer install --no-interaction --prefer-dist.
  2. Run php tests/recurring-scheduler-idempotency-smoke.php; confirm all assertions pass, including the concurrent reconciliation case that preserves exactly one unique pending chain without cancellation.
  3. Run php tests/retention-action-scheduler-batching-smoke.php; confirm 38 assertions pass, including expired canceled Data Machine recurrence logs being deleted in bounded batches while recent and non-Data-Machine logs remain.
  4. Run php tests/recurring-scheduler-as-readiness-smoke.php; confirm all readiness assertions pass.
  5. Run composer lint; confirm PHPCS passes.
  6. The full top-level smoke loop is currently blocked before this change's tests by Fix image-template load-order smoke bootstrap for AbilityRegistration #2894, which is filed separately.

AI assistance

  • AI assistance: Yes
  • Tool(s): OpenCode general subagent, OpenAI gpt-5.6-terra
  • Used for: Read-only storage diagnosis, recurrence lifecycle analysis, implementation, and focused test coverage.

@homeboy-ci

homeboy-ci Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Homeboy Results — data-machine

Lint

review lint — passed

ℹ️ Full options: homeboy self docs commands/lint
Deep dive: homeboy review lint data-machine --changed-since b089dbb

Artifacts and drill-down
  • CI results artifact: homeboy-ci-results-data-machine-review-lint-quality-Linux-node24 contains immediate command JSON for this action invocation.
  • Observation artifact: homeboy-observations-data-machine-review-lint-quality-Linux-node24 contains exported Homeboy run history for deeper queries.
  • Drill-down: download the observation artifact, then run homeboy runs import <dir>, homeboy runs list, and homeboy runs findings <run-id>.
  • Artifacts are attached to the workflow run: https://github.com/Extra-Chill/data-machine/actions/runs/29285507246

Test

review test — failed

ℹ️ No tests ran — the runner failed before producing results. See raw_output.stderr_tail / raw_output.stdout_tail for the underlying error (bootstrap failure, missing deps, DB connection, etc.).
ℹ️ To run specific tests: homeboy test data-machine -- --filter=TestName
ℹ️ Auto-fix lint issues: homeboy refactor data-machine --from lint --write
ℹ️ Collect coverage: homeboy test data-machine --coverage
ℹ️ Analyze failures: homeboy test data-machine --analyze
ℹ️ Pass args to test runner: homeboy test -- [args]
ℹ️ Full options: homeboy self docs commands/test
Deep dive: homeboy review test data-machine --changed-since b089dbb

Artifacts and drill-down
  • CI results artifact: homeboy-ci-results-data-machine-review-test-quality-Linux-node24 contains immediate command JSON for this action invocation.
  • Observation artifact: homeboy-observations-data-machine-review-test-quality-Linux-node24 contains exported Homeboy run history for deeper queries.
  • Drill-down: download the observation artifact, then run homeboy runs import <dir>, homeboy runs list, and homeboy runs findings <run-id>.
  • Artifacts are attached to the workflow run: https://github.com/Extra-Chill/data-machine/actions/runs/29285507246

Audit

review audit — passed

  • audit — 88 finding(s)
  • Total: 88 finding(s)

Deep dive: homeboy review audit data-machine --changed-since b089dbb

Artifacts and drill-down
  • CI results artifact: homeboy-ci-results-data-machine-review-audit-quality-Linux-node24 contains immediate command JSON for this action invocation.
  • Observation artifact: homeboy-observations-data-machine-review-audit-quality-Linux-node24 contains exported Homeboy run history for deeper queries.
  • Drill-down: download the observation artifact, then run homeboy runs import <dir>, homeboy runs list, and homeboy runs findings <run-id>.
  • Artifacts are attached to the workflow run: https://github.com/Extra-Chill/data-machine/actions/runs/29285507246
Tooling versions
  • Homeboy CLI: homeboy 0.283.1+10b6335515eb+0f9bcae1
  • Extension: wordpress from https://github.com/Extra-Chill/homeboy-extensions
  • Extension revision: 40f77e42
  • Action: unknown@unknown

@chubes4

chubes4 commented Jul 13, 2026

Copy link
Copy Markdown
Member Author

CI status: Audit and Lint passed. Test failed before running component tests because the Homeboy action dirtied its CI checkout and emitted no structured test result; tracked separately in Extra-Chill/homeboy#7948. No merge while this required check is red.

@chubes4

chubes4 commented Jul 13, 2026

Copy link
Copy Markdown
Member Author

Final CI status: Plan, Audit, and Lint passed; Build skipped; Test failed before tests produced results with Invalid argument 'write': Working tree has uncommitted changes. Job: https://github.com/Extra-Chill/data-machine/actions/runs/29265044109/job/86867969882. This repeats the Homeboy infrastructure failure despite the referenced fix, so no merge was performed. The separate local full-smoke send-email bootstrap failure is tracked in #2897.

@chubes4 chubes4 merged commit 1f68afb into main Jul 13, 2026
4 of 5 checks passed
@chubes4 chubes4 deleted the investigate/sqlite-database-bloat branch July 13, 2026 21:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Retention recurrence is not recreated after Action Scheduler cancellation

1 participant