fix(code-index): stop reading a held projection slot as work - #3077
Open
devin-ai-integration[bot] wants to merge 2 commits into
Open
devin-ai-integration[bot] wants to merge 2 commits into
devin-ai-integration[bot] wants to merge 2 commits into
Conversation
The paused clock auto-advanced past the test's deadlines while a live notify wake was in flight, and the watcher's own copies of the marker writes could add a second drain. Wait on the debounce observing the operation instead, and assert the release rather than an exact count. Refs #3041 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
text_projection_needs_work treated any contended slot lock as work in progress. Another probe or a no-op advance over ready owners holds the lock too, so the published projection stamped a phantom continuation and freshness stayed Verifying at the serving swap. Mirror the slot's idle state in an atomic written on every transition and read that. The graph-tail ready test also holds admission until the complete-generation demand is posted, so the demand joins the first pass instead of queueing a real follow-up. Fixes #3041 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Contributor
Author
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
text_projection_needs_workreported any contended slot lock as work in progress. Another probe, or a no-op advance over ready owners, holds that lock too. When the two collided, the published text projection stamped a phantom worker continuation, and freshness stayedVerifying/Pendingat the serving swap. This is the reconcile flake in test: four intermittent master failures (reachability, reconcile, git_watch, dashboard) #3041.Fixes #3041
Motivation
#3041 lists four intermittent master failures. Status of each:
(Some(Verifying), Pending): failed 59 of 3,000 runs under local stress. I instrumented every pending-wake writer. There were two late wake sources:request_complete_generationdemand sometimes landed after the first pass had claimed its wake. That queues a real follow-up pass, soVerifyingis the correct reading. Fixed in the test by holding admission until the demand is posted.note_worker_continuation, fired because the slottry_lockreturnedWouldBlockwhile the slot was idle. This is a production bug, fixed here.tests.rs:643): the test ran onstart_pausedtime. The paused clock auto-advanced past the test's deadlines while a live notify wake was still in flight. The watcher's own copies of the marker writes could also produce a second drain.ChunkLoadError: that message is the errorRouteChunkBoundary.dom.test.tsxdeliberately rejects with. It appears in the output whenever that suite runs, so it does not identify the failing test. The suite passed 15 of 15 local runs. No change here.Changes
code_index_scheduler/serving.rs:CodeTextProjectionStateV1keeps the slot's idle state in an atomic (slot_idle), written on every slot transition throughreplace_slot. The probe reads that atomic and never takes the lock, so it still never queues behind an advance's slice.tests/serving.rs: addsa_held_idle_projection_slot_is_not_text_projection_work. It holds an idle slot's lock and asserts no work is reported. It fails before the fix (serving.rs:908) and passes after.tests/reconcile.rs:ready_wait_ends_only_after_the_graph_tail_seats_the_generationholds admission until the complete-generation demand is posted.git_watch/tests.rs,state.rs,git_watch.rs: the linked-worktree test now runs on real time. It waits on a test-onlyoperation_heldsignal (debounce observedInFlight) and asserts the release with>= 1drain instead of an exact count.Test plan
bash scripts/require-exact-test.sh cargo test -p tracedecay-code-index-runtime --lib <test> -- --exactpasses for each of the three tests above, after the rebase onorigin/mastercargo test -p tracedecay-code-index-runtime --lib -- code_index_scheduler::tests::serving code_index_scheduler::tests::reconcile git_watch: 280 passedcargo fmt --all -- --checkandcargo clippy -p tracedecay-code-index-runtime --all-targets -- -D warningsare cleanpython3 scripts/linux-test-partitions.py checkandnode scripts/lint-commit-range.mjs --repository . origin/master HEADpasscargo nextest run: left to CIChecklist
CHANGELOG.mdupdated (not updated; release-please generates it from commits).envfiles includedLink to Devin session: https://app.devin.ai/sessions/e8f223a3ec5c476885a5d8903bafa4c2
Open in Devin Desktop: https://app.devin.ai/desktop/session/e8f223a3ec5c476885a5d8903bafa4c2?variant=devin
Requested by: @ScriptedAlchemy