fix(work): seal cancellations that land before running - #3068
Open
devin-ai-integration[bot] wants to merge 1 commit into
Open
devin-ai-integration[bot] wants to merge 1 commit into
devin-ai-integration[bot] wants to merge 1 commit into
Conversation
A cancellation persisted before mark_running parked the attempt in cancellation_requested: the refused transition killed the provider and returned without settling. Seal it as cancelled through the normal acknowledge/settle ladder, and store a notify permit so an early registry signal is not lost. The bench cleanup chain now requests JSON from worktree_cleanup_*, and its resume-sweep workaround for parked cancels is removed. Refs #3053 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Contributor
Author
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
mark_runningnow endsCancelledwith sealed evidence. Before this change it stayed parked incancellation_requesteduntil a recovery sweep ran.worktree_cleanup_*forformat: "json", so the inspect → confirm prime chain gets the JSON it parses. It also deletes the bench's resume-sweep workaround for parked cancellations.Refs #3053. Findings 2 and 3 are fixed. Finding 1, the CPU wedge after
work_adjudicate_leak, is not fixed here; see below.Motivation
#3053 lists three defects found by the #3047 bench run.
Parked cancel (finding 2).
cancel_attemptpersistsCancellationRequestedand then signals the live owner. The stdio path launches the provider child beforemark_running.CancellationRequested → Runningis an illegal transition, somark_runningfailed. The failure branch then killed the child and returned without settling the row. No later provider exit could seal it either, so onlywork_resume_attemptsmoved it to a terminal state. The app-server path andsettle_unstarted(provider unavailable) had the same gap.Separately,
WorkAttemptProcessRegistryV1signalled throughNotify::notify_waiters. That call stores no permit, so a signal sent before the execution task reachedcancel.notified()was dropped. The existing tests worked around this by re-signalling in a loop.Markdown inspect (finding 3). Application surfaces,
worktree_cleanup_*included, advertiseformatand return markdown by default (RequestedOutputFormat). The renderer is behaving correctly. The bench routed these tools througheqn, which sends noformat, while its prime chain parsesinspection_digestout of the inspect response as JSON.Changes
crates/tracedecay-daemon-service/src/invocation/work_attempt_exec.rsseal_cancellation_before_running. Whenmark_runningormark_provider_unavailableis refused, it triesacknowledge_cancellation. That call succeeds only when a cancellation request is pending. On success it callssettle_with_artifactswithWorkAttemptProviderOutcomeV1::Cancelledevidence: the normal ladder, with no new contract API. Any other refusal (a stale lease or a recovery fence) still takes the existing path where the durable row stays authoritative.notify_one, so one permit is stored for an owner that is not waiting yet.crates/tracedecay/benches/coverage/admin.rs:worktree_cleanup_inspect, confirm, reconcile and remove useeq, and the cleanup prime chain sendsformat: "json".crates/tracedecay/benches/coverage/mod.rs: deletes thesettle_attemptresume-sweep workaround for parked cancellations.Not fixed: CPU wedge after
work_adjudicate_leak(finding 1)I could not reproduce this one. The #3047 notes describe it as intermittent: 3 of the bench runs hit it and run95 passed on the same profile. Each reproduction needs a full scipy
large_repos --testrun of about 40 minutes. I traced the path after adjudication:record_work_leak_observation, then the producer, thenrun_one_rollup_maintenance, plus the Task activity publish inwork.rs, the workflow fan-out reconcile and the recovery owner loop. Every loop on that path is bounded and has a sleep or retry cadence. I found no deterministic feedback loop, so I did not guess at a change. The issue should stay open for that finding.Test plan
bash scripts/require-exact-test.sh cargo test -p tracedecay-daemon-service --lib invocation::work_attempt_exec::tests::a_cancellation_requested_before_mark_running_seals_cancelled -- --exact: 1 passed. On the oldwork_attempt_exec.rsit fails withleft: CancellationRequested, right: Cancelled.bash scripts/require-exact-test.sh cargo test -p tracedecay-daemon-service --lib invocation::work_attempt_exec::tests::a_cancellation_signalled_before_the_owner_waits_still_reaches_it -- --exact: 1 passed. On the old code it fails with "a cancellation signalled before the owner waited was lost".cargo test -p tracedecay-daemon-service --lib work_attempt_exec: 22 passed.cargo fmt --all -- --checkcargo clippy -p tracedecay-daemon-service -p tracedecay --all-targets -- -D warningsnode scripts/lint-commit-range.mjs --repository . origin/master HEADChecklist
CHANGELOG.mdupdated: generated by release tooling from commit messages.envfiles includedLink to Devin session: https://app.devin.ai/sessions/8c3b6020316140699db560903cebf1aa
Open in Devin Desktop: https://app.devin.ai/desktop/session/8c3b6020316140699db560903cebf1aa?variant=devin
Requested by: @ScriptedAlchemy