Repository navigation
fix: support concurrent Boat QA lanes within a shared spend cap - #255
Merged
Merged
Conversation
Use each lane TTL for spend reservation and box creation. Avoid retries for listed failed case IDs and allow opted-in ordered lanes to continue after non-pass selections without changing legacy defaults.
…gation, and cap tests
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Fix bin/fm-boat-lanes.py so release QA lanes from one run can start concurrently under one total spend cap when the cap allows. Add optional per-lane ttl_seconds overrides applied to reservation, cap check, Boat box creation, and TTL update. Add optional top-level known_product_failures case IDs so listed failed product cases retain failure status and evidence and are never retried, while every other failure keeps the existing retry behavior. Add optional lane continue_on_failure, default false, so ordered later selections run after an earlier selection reports failed or blocked cases and produce the maintenance selection result and artifacts; preserve default stop-after-non-pass and stopping on harness errors, timeouts, cancellation, or invalid evidence. Existing specs must behave unchanged; document the new fields in the runner header and --help. Extend tests/fm-boat-lane-runner.test.sh and offline fake-provider cases only; create no live Boat boxes and incur no spend. Create the private gitignored ~/Desktop/firstmate/config/boat-lanes/infoconnect-release.v1.json from the read-only v0.5.15 specs and report evidence, with one maintenance pilot, the main shards, one correctly ordered resilience-then-maintenance lane, and qualification. Size per-lane TTLs from measured v0.5.15 lane durations with real margin, list QA-QUAL-RANKING-01 as a known failure, and keep total worst-case reservation at or below $1.00 so all eight post-pilot lanes can start together with --jobs 8. Validate the recipe offline against the runner without creating boxes; keep it out of the commit and PR. The external config write is the single approved exception to the task worktree boundary; touch nothing else outside the worktree. Use report.md Recipe problems and next-run fixes item 3; leave report items 4 and 5 (InfoConnect scripts) unchanged. The diagnostics startup-persistence test fails on clean origin/main commit bb51ff2; record that as pre-existing with evidence and do not fix it in this task. Record receipts for AC1-AC5. Do not merge a PR.
What Changed
known_product_failurescase IDs that retain failure status and evidence without retries, preserving existing retry behavior for other cases.continue_on_failurefor later ordered selections while retaining earlier failures and harness stop conditions; document the fields and extend offline fake-provider coverage.Risk Assessment
✅ Low: The changes are bounded, and source tracing confirms consistent TTL accounting, known-failure preservation, and continued-selection verdict aggregation.
Testing
All 23 targeted runner cases passed through the real CLI with an isolated fake provider. Saved product reports and ledgers; the clean-base diagnostics check also passed. No real Boat calls or spend occurred. Private recipe validation remains unavailable within the permitted paths.
Evidence: Runner ledgers and generated reports
Evidence: Targeted acceptance results
Evidence: Adversarial results
Evidence: Clean-base diagnostics result
Evidence: Runner help
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 3 issues found → auto-fixed (2) ✅
bin/fm-boat-lanes.py:675- The requirement says listed failed product cases are “never retried,” but the new condition requires status == "fail". The public evidence contract also accepts "failed" and other non-pass statuses. A listed case with status="failed" and no kind becomes uncertain, enters retry targets at bin/fm-boat-lanes.py:766, and executes its retry at :798. The same classification boundary processes retry evidence at :799. Apply known-product protection consistently to supported failure statuses while preserving dependency-blocked handling.bin/fm-boat-lanes.py:896- With continue_on_failure=true, a product failure in selection 0 followed by a dependency-blocked failure in selection 1 changes the lane verdict to known_harness_defect, despite the retained product failure. Each failing selection overwrites the previous verdict. Aggregate classifications across selections using the existing selection precedence at bin/fm-boat-lanes.py:821; the resulting verdict must remain consistent in lane_result (:921) and the complete report (:939).tests/boat-lane-runner-cases.py:183- This concurrent test assumes lane1 acquires the creation lock before lane2. ThreadPoolExecutor does not guarantee that order: lane2 can reserve🔧 Fix applied.
3 issues (2 errors, 1 warning) still open:
bin/fm-boat-lanes.py:675- The requirement says listed failed product cases are “never retried,” but the new condition requires status == "fail". The public evidence contract also accepts "failed" and other non-pass statuses. A listed case with status="failed" and no kind becomes uncertain, enters retry targets at bin/fm-boat-lanes.py:766, and executes its retry at :798. The same classification boundary processes retry evidence at :799. Apply known-product protection consistently to supported failure statuses while preserving dependency-blocked handling.bin/fm-boat-lanes.py:896- With continue_on_failure=true, a product failure in selection 0 followed by a dependency-blocked failure in selection 1 changes the lane verdict to known_harness_defect, despite the retained product failure. Each failing selection overwrites the previous verdict. Aggregate classifications across selections using the existing selection precedence at bin/fm-boat-lanes.py:821; the resulting verdict must remain consistent in lane_result (:921) and the complete report (:939).bin/fm-boat-lanes.py:675- Round 1 fixed retry targeting but left the retry-merge sibling unprotected. The requirement says listed failed cases must “retain failure status and evidence.” Start with listed K failing and eligible E failing; E's supported per-case retry can return additional rows, including K=pass. The unconditional merge at bin/fm-boat-lanes.py:811-817 replaces K's status, request IDs, and signature hash, then :818 updates reported totals and :819-824 can return success, allowing pilot fan-out. Additional protected rows in per-case retry evidence are already exercised by tests/boat-lane-runner-cases.py's test_infra_retried_once_product_never_retried. Preserve previously observed listed failures at the shared retry-merge boundary, including their retry report and final verdict representations.🔧 Fix applied.
✅ Re-checked - no issues remain.
python3 ~/.no-mistakes/evidence/01M51ZHGR7Y9C9B4RWVRHTT81E/drive.py— 18 selected executable acceptance casespython3 ~/.no-mistakes/evidence/01M51ZHGR7Y9C9B4RWVRHTT81E/adversarial.py— eight-lane concurrency and four continuation stop boundariespython3 ~/.no-mistakes/evidence/01M51ZHGR7Y9C9B4RWVRHTT81E/baseline.py— clean-base diagnostics startup-persistence casepython3 bin/fm-boat-lanes.py --helpCaptured runner ledgers, reports, specifications, and isolated provider state; removed disposable worktree data and confirmed clean git status.✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.