Skip to content

fix(work): seal cancellations that land before running - #3068

Open
devin-ai-integration[bot] wants to merge 1 commit into
masterfrom
devin/1791127746-work-tail-cancel
Open

devin-ai-integration[bot] wants to merge 1 commit into
masterfrom
devin/1791127746-work-tail-cancel

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • A Work attempt cancelled before mark_running now ends Cancelled with sealed evidence. Before this change it stayed parked in cancellation_requested until a recovery sweep ran.
  • The process registry no longer loses a cancellation signal that arrives before the owner task starts waiting.
  • The large-repos bench now asks worktree_cleanup_* for format: "json", so the inspect → confirm prime chain gets the JSON it parses. It also deletes the bench's resume-sweep workaround for parked cancellations.

Refs #3053. Findings 2 and 3 are fixed. Finding 1, the CPU wedge after work_adjudicate_leak, is not fixed here; see below.

Motivation

#3053 lists three defects found by the #3047 bench run.

Parked cancel (finding 2). cancel_attempt persists CancellationRequested and then signals the live owner. The stdio path launches the provider child before mark_running. CancellationRequested → Running is an illegal transition, so mark_running failed. The failure branch then killed the child and returned without settling the row. No later provider exit could seal it either, so only work_resume_attempts moved it to a terminal state. The app-server path and settle_unstarted (provider unavailable) had the same gap.

Separately, WorkAttemptProcessRegistryV1 signalled through Notify::notify_waiters. That call stores no permit, so a signal sent before the execution task reached cancel.notified() was dropped. The existing tests worked around this by re-signalling in a loop.

Markdown inspect (finding 3). Application surfaces, worktree_cleanup_* included, advertise format and return markdown by default (RequestedOutputFormat). The renderer is behaving correctly. The bench routed these tools through eqn, which sends no format, while its prime chain parses inspection_digest out of the inspect response as JSON.

Changes

  • crates/tracedecay-daemon-service/src/invocation/work_attempt_exec.rs
    • Adds seal_cancellation_before_running. When mark_running or mark_provider_unavailable is refused, it tries acknowledge_cancellation. That call succeeds only when a cancellation request is pending. On success it calls settle_with_artifacts with WorkAttemptProviderOutcomeV1::Cancelled evidence: the normal ladder, with no new contract API. Any other refusal (a stale lease or a recovery fence) still takes the existing path where the durable row stays authoritative.
    • Registry cancellation signals now use notify_one, so one permit is stored for an owner that is not waiting yet.
  • crates/tracedecay/benches/coverage/admin.rs: worktree_cleanup_inspect, confirm, reconcile and remove use eq, and the cleanup prime chain sends format: "json".
  • crates/tracedecay/benches/coverage/mod.rs: deletes the settle_attempt resume-sweep workaround for parked cancellations.

Not fixed: CPU wedge after work_adjudicate_leak (finding 1)

I could not reproduce this one. The #3047 notes describe it as intermittent: 3 of the bench runs hit it and run95 passed on the same profile. Each reproduction needs a full scipy large_repos --test run of about 40 minutes. I traced the path after adjudication: record_work_leak_observation, then the producer, then run_one_rollup_maintenance, plus the Task activity publish in work.rs, the workflow fan-out reconcile and the recovery owner loop. Every loop on that path is bounded and has a sleep or retry cadence. I found no deterministic feedback loop, so I did not guess at a change. The issue should stay open for that finding.

Test plan

  • bash scripts/require-exact-test.sh cargo test -p tracedecay-daemon-service --lib invocation::work_attempt_exec::tests::a_cancellation_requested_before_mark_running_seals_cancelled -- --exact: 1 passed. On the old work_attempt_exec.rs it fails with left: CancellationRequested, right: Cancelled.
  • bash scripts/require-exact-test.sh cargo test -p tracedecay-daemon-service --lib invocation::work_attempt_exec::tests::a_cancellation_signalled_before_the_owner_waits_still_reaches_it -- --exact: 1 passed. On the old code it fails with "a cancellation signalled before the owner waited was lost".
  • cargo test -p tracedecay-daemon-service --lib work_attempt_exec: 22 passed.
  • cargo fmt --all -- --check
  • cargo clippy -p tracedecay-daemon-service -p tracedecay --all-targets -- -D warnings
  • node scripts/lint-commit-range.mjs --repository . origin/master HEAD
  • No new test target, so the partition manifest does not change.

Checklist

  • CHANGELOG.md updated: generated by release tooling from commit messages
  • No secrets, credentials, or .env files included
  • Breaking changes documented (if any): none

Link to Devin session: https://app.devin.ai/sessions/8c3b6020316140699db560903cebf1aa
Open in Devin Desktop: https://app.devin.ai/desktop/session/8c3b6020316140699db560903cebf1aa?variant=devin
Requested by: @ScriptedAlchemy


Devin Review

A cancellation persisted before mark_running parked the attempt in
cancellation_requested: the refused transition killed the provider and
returned without settling. Seal it as cancelled through the normal
acknowledge/settle ladder, and store a notify permit so an early
registry signal is not lost.

The bench cleanup chain now requests JSON from worktree_cleanup_*, and
its resume-sweep workaround for parked cancels is removed.

Refs #3053

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@changeset-bot

changeset-bot Bot commented Oct 4, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: e80c430

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant