Skip to content

fix: stabilize scheduled checks and close Linux mutation coverage gaps - #357

Merged
martin-kolinek merged 40 commits into
mainfrom
makolnek-microsoft-scheduled-build-timings
Nov 10, 2026
Merged

martin-kolinek merged 40 commits into
mainfrom
makolnek-microsoft-scheduled-build-timings

Conversation

@martin-kolinek

@martin-kolinek martin-kolinek commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

🤖

Summary

  • Make the escaped-descendant listing regression deterministic, and keep prepared Gamma caches and CLI fixtures private under permissive umasks.
  • Exercise native process-group and systemd relaunch guards through injected callbacks instead of signalling or recursively re-executing the test harness. The recursive relaunch regression caused the earlier Linux process storm.
  • Retain cfg-aware declared-return coverage alongside the upstream collector implementation, removing redundant local-attribute visits.
  • Add behavioral regressions for deterministic policy, including cgroup slot reuse, permission masks, discovery/retry decisions, parsing, and exact diagnostics.
  • Reduce Miri's repeated planner matrices and generated-text scanning while retaining memory-relevant and core parsing coverage. Update the corresponding design contracts.

Mutation scope

Keep the strict 100% mutation score and zero-flaky gate. Add localized, reasoned exclusions for native resource/I/O boundaries requiring the deferred ownership/backend refactor, plus demonstrated equivalent or non-contract mutations. These boundaries are excluded, not claimed as mutation-covered; there are no blanket file/crate exclusions.

Complete exhaustive verification

Integrated main at 08a51545 in merge 7202bbe9, preserving upstream scripted build supervision and macOS process support.

Full-workspace exhaustive run 42812857111 passed on both platforms at 2c010a4a, including mutation testing, feature powersets, and benchmark compilation:

Platform Full job Gamma including preparation/build/baseline Benchmark compilation Mutation result
Windows 2h15m12s approximately 1h37m09s 27m59s 59,070/59,070 killed
Linux 3h04m20s 2h02m43s 49m01s 59,329/59,329 killed

Both platforms had zero survivors, timeouts, out-of-memory verdicts, or uncovered mutants. Jobs run in parallel, so exhaustive workflow wall time is approximately the longer job, not the sum.

Linux's independent containment observer finalized successfully: peak 7.7384 GiB charged memory, zero outer-boundary max/OOM/PID-limit events, no controlled stop, and campaign exit 0. Early reclaim remained enabled with unchanged external hard limits and serial compilation.

The final follow-up removes the two temporary diagnostic workflow/observer files and strengthens only the existing cache-permission acceptance regression: campaign creation is exercised before workspace preclaiming can create its directory, and the explicit existing-directory permission change is gated behind system-tests. No production behavior changed after the passing exhaustive head. Fresh ordinary PR CI applies to that final follow-up; the complete exhaustive evidence above is specifically for 2c010a4a.

The earlier integrated run 42792368403 safely exposed two redundant visitor calls and one native resume-adapter survivor. Commits ca336ae2 and 2c010a4a addressed these before the successful full rerun. Focused Linux regressions and the native Windows diagnostic regression passed; full-workspace Clippy passed on both platforms.

Miri evidence

Earlier profiled Miri run 41806126868 completed in Windows 1h07m03s, Linux 1h44m58s, Linux ARM 1h29m51s, and Windows ARM 1h56m34s. These measurements preceded upstream integration and are not a new current-head profile. Miri belongs to runtime analysis, not the exhaustive group above.

Runtime and diagnostic findings

Gamma is the largest part of exhaustive execution, but not the only long check: optimized benchmark compilation adds approximately 28 minutes on Windows and 49 minutes on Linux in the final measured run. Linux Gamma's actual testing phase took approximately 97m28s, with approximately 25m15s fixed preparation/build/baseline overhead.

The later Linux benchmark stops were compiler allocations plus charged file cache, not a renewed mutant process storm. Earlier cgroup reclaim resolved those stops without raising the existing hard limits. Temporary diagnostic harness settings are not new Anvil defaults, and the harness is removed from the final changes.

These are observed post-adoption runtimes, not a controlled speedup comparison against the previous mutation tool.

martin-kolinek and others added 30 commits November 3, 2026 17:32
…stic

Replace native process startup and subsecond timing assertions with the existing listing-supervision seam and an exhausted scripted output budget. Keep a live reader sender to model an escaped descendant without launching processes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Exclude 19 expensive generated-text contract assertions only under Miri. Keep native/cargo-careful execution, artifact rendering and ownership checks, and parser/planner coverage unchanged; document the safety tradeoff.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Cover xversion as a missing version and a leading NUL as malformed YAML. Both exact prefix-stripping mutants from the scheduled Gamma campaign now fail their focused behavioral regressions; production parsing remains unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep complete native/cargo-careful failure-injection and adoption matrices. Under Miri retain the successful plan, first/middle/last resolver failures, both TOML shapes, Unicode, both newline styles, and both catalog orders without repeating every Cartesian combination.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Start the timeout assertion only after the contained descendant is ready, then exercise the real supervisor with a zero budget. Keep this OS acceptance behind system-tests.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep expensive planner policy matrices, YAML permutations, and generated-text scans native/cargo-careful only. Retain focused parser and lifecycle coverage, add a small BOM/Unicode/CRLF flow round trip, and shrink builder fixtures without skipping their contracts.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Inject signalling into the rejected-ID tests while retaining native termination acceptance and mutation coverage. Linux killpg(1, SIGKILL) broadcasts beyond the test process tree.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Run the unchanged Anvil group on ubuntu-latest and publish bounded host resource, process, kernel, and phase snapshots to a separate live check.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The active run retains its immutable observer and source snapshot. Keep the lasting change limited to safe process-group guard testing and its design documentation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Check the exact child arguments before launching the current harness. Add a construction-only regression for the empty-user-arguments path and require exactly one passing native child.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Capture synced pre-launch identities and spawned PIDs, preserve campaign journals without saved environments, and observe a fresh hosted runner outside a bounded workload scope.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Run the campaign as a dedicated unprivileged user on the disposable hosted VM. Bound its entire root-owned user slice so automatic systemd sibling scopes cannot leave the memory/task ceiling; observe from outside that identity with no job token in the campaign.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The hosted image rejects MemoryOOMGroup as a systemctl assignment. Set and verify memory.oom.group through the root-owned slice directly, and keep diagnostic output writable by the runner and isolated campaign identity.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The runner home prevents the dedicated user from reaching the checkout. Grant only directory traversal on its ancestors and verify checkout/tool/output access before launching the protected campaign.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Run 42267402403 continues at immutable commit ebadc79. Retain the production selection guards without diagnostic hooks or workflow; session artifacts preserve the harness and verified user-slice limits.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A benign Linux lookup probe confirms CARGO_HOME/bin precedes PATH. Install and byte-compare the diagnostic coordinator in the isolated user's Cargo bin before running the unchanged group, retaining the already verified user-slice limits.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The active hosted run retains the instrumented coordinator and independent observer at 493b9f7. Restore the lasting production fixes without leaving one-off workflow or tracing code.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Repeat the bounded hosted investigation with allowlisted baseline failure messages; do not publish saved environment overrides.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Retained baseline errors show the dedicated user's temporary root was group-writable. Set its mode explicitly and use a private campaign umask, preserving cache ownership checks and the root-managed resource boundary.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…umasks

Use the existing private-directory helper before cache claiming, so preparation does not create and then reject its own redirected cache. Reuse the private temporary root for CLI fixtures. Remove ineffective diagnostic-only umask overrides.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep external boundary counter loss fatal and sample that boundary independently of currently observed child processes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Inject the native relaunch attempt into the once-only policy so guard mutations fail in-process instead of recursively creating sibling systemd scopes. Cover unmarked result propagation and document the non-spawning contract.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The protected campaign identified recursive relaunch mutant ddda37216735, ordinal 50167. Preserve the collected evidence as session artifacts and remove the temporary hosted workflow, external observer, and synced launch tracing.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Reuse the independently enforced Linux user-slice limits and fail-closed observer while executing the unchanged full-workspace exhaustive recipe. Build the current workspace coordinator on both platforms, retain mutation evidence, and keep production sources and generated Anvil artifacts unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Run 42598115692 passed the full Windows tier in 2h20m. Linux completed mutation execution without runner OOM or a safety stop, but failed the 100% score gate at 99.0%: 548 survivors, five timeouts, and two individually memory-limited mutants. The recursive guard mutation was killed in 78 ms. Preserve reports in session artifacts and remove the temporary hosted harness.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep the 100 percent retained-mutant gate. Exclude native ownership and I/O paths that need the deferred resource backend, add deterministic policy regressions, and avoid collecting disabled cfg tails.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Retain the existing whole-user resource limits and fail-closed observer. Finalize diagnostic evidence before removing the user slice so successful teardown cannot erase required counters.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Cache preclaiming can leave newly created directories world-writable under umask 000, and cfg-disabled local items can incorrectly block declared-return-type propagation.

2 open issues
What changed in this PR

This PR stabilizes Gamma’s Linux validation and reduces Anvil’s Miri workload, while retaining the strict mutation gate for the documented mutation scope.

Changes:

  • Makes process-supervision tests deterministic, isolates signalling and relaunch guards, and adds private cache creation during preparation.
  • Fixes configuration-aware tail-expression selection and adds focused behavioral regressions with localized mutation exclusions.
  • Reduces repeated Miri test matrices and generated-text scans, with corresponding design documentation.
File Description
crates/​cargo-heather/​src/​bin/​cargo-heather/​run.rs Documents native header-publication mutation exclusions.
crates/​cargo-gamma-unsafe/​src/​support.rs Excludes native capability discovery from mutation testing.
crates/​cargo-gamma-unsafe/​src/​interrupt.rs Adds registry regressions and targeted mutation exclusions.
crates/​cargo-gamma-unsafe/​src/​identity.rs Documents the native credential boundary.
crates/​cargo-gamma-unsafe/​src/​group.rs Tests signal guards through injected callbacks.
crates/​cargo-gamma-unsafe/​src/​cgroup.rs Adds cgroup policy regressions and native-boundary exclusions.
crates/​cargo-gamma-unsafe/​docs/​DESIGN.md Documents process-group isolation and mutation scope.
crates/​cargo-gamma-process/​src/​process_tree.rs Adds localized lifecycle mutation exclusions.
crates/​cargo-gamma-process/​docs/​IMPLEMENTATION.md Clarifies native lifecycle mutation boundaries.
crates/​cargo-gamma-lib/​tests/​cli.rs Uses private temporary CLI workspaces.
crates/​cargo-gamma-lib/​src/​paths.rs Documents a native path-resolution exclusion.
crates/​cargo-gamma-lib/​src/​exec/​workspace.rs Uses private directory creation and platform-specific branches.
crates/​cargo-gamma-lib/​src/​exec/​verdict.rs Guards native libtest launches against recursive selection.
crates/​cargo-gamma-lib/​src/​exec/​testing_state.rs Extracts and tests executable-permission masking.
crates/​cargo-gamma-lib/​src/​exec/​test_listing.rs Replaces timing-dependent listing coverage with scripted supervision.
crates/​cargo-gamma-lib/​src/​exec/​sync.rs Adds synchronization failure regressions and targeted exclusions.
crates/​cargo-gamma-lib/​src/​exec/​rustc_wrapper.rs Documents native capture-directory cleanup exclusions.
crates/​cargo-gamma-lib/​src/​exec/​relaunch.rs Injects relaunch attempts and strengthens command tests.
crates/​cargo-gamma-lib/​src/​exec/​measure.rs Documents native compiler-evidence publication exclusions.
crates/​cargo-gamma-lib/​src/​exec/​manifest.rs Tests portable paths and documents native creation exclusions.
crates/​cargo-gamma-lib/​src/​exec/​copy.rs Separates platform-specific symlink creation.
crates/​cargo-gamma-lib/​src/​exec/​build/​tests.rs Waits for descendant readiness before timeout acceptance.
crates/​cargo-gamma-lib/​src/​exec/​build/​splices.rs Documents native instrumentation-publication exclusions.
crates/​cargo-gamma-lib/​src/​exec/​build/​invoke.rs Extracts supervision of an already-adopted process tree.
crates/​cargo-gamma-lib/​src/​elements/​publication.rs Adds localized filesystem-publication exclusions.
crates/​cargo-gamma-lib/​src/​discover/​plan.rs Runs Windows package-URL coverage on every platform.
crates/​cargo-gamma-lib/​src/​discover/​hints.rs Documents native hint-file read exclusions.
crates/​cargo-gamma-lib/​src/​discover/​glob.rs Uses compile-time platform branches for normalization.
crates/​cargo-gamma-lib/​src/​discover/​diff.rs Tests inventoried absolute paths without filesystem probes.
crates/​cargo-gamma-lib/​src/​commands/​verdict_log.rs Documents native progress-log repair exclusions.
crates/​cargo-gamma-lib/​src/​commands/​run.rs Documents native resume-relaunch exclusions.
crates/​cargo-gamma-lib/​src/​commands/​dispatch.rs Documents native relaunch dispatch exclusions.
crates/​cargo-gamma-lib/​src/​ci/​summary.rs Documents native summary-file opening exclusions.
crates/​cargo-gamma-lib/​docs/​design/​system-tests.md Defines deterministic acceptance and mutation boundaries.
crates/​cargo-gamma-lib/​docs/​DESIGN.md Describes private directory creation during preparation.
crates/​cargo-gamma-engine/​src/​ops/​collect/​collector/​tests.rs Adds configuration-aware typed-tail regression coverage.
crates/​cargo-gamma-engine/​src/​ops/​collect/​collector.rs Selects the last enabled statement for typed tails.
crates/​cargo-gamma-engine/​src/​model/​identity.rs Excludes non-progress normalization mutations.
crates/​cargo-gamma-engine/​docs/​DESIGN.md Specifies configuration-aware return-type propagation.
crates/​cargo-each/​src/​run.rs Documents native PATH lookup exclusions.
crates/​cargo-coverage-gate/​src/​source_filter.rs Tests preserving workspace sections after compiler filtering.
crates/​cargo-coverage-gate/​src/​bin/​cargo-coverage-gate/​report_file.rs Documents native file-durability exclusions.
crates/​cargo-coverage-gate/​src/​bin/​cargo-coverage-gate/​collect.rs Tests fallback classification and separates platform cleanup messages.
crates/​cargo-aprz-lib/​src/​facts/​crates/​tables/​table_mgr.rs Documents platform-specific archive-path mutation equivalence.
crates/​cargo-aprz-lib/​src/​facts/​collector.rs Documents native cache-creation exclusions.
crates/​cargo-aprz-lib/​src/​facts/​advisories/​archive.rs Adds localized download and archive-path exclusions.
crates/​cargo-aprz-lib/​src/​commands/​github_credentials.rs Extracts and tests executable-permission policy.
crates/​cargo-anvil/​src/​run.rs Reduces repeated planner matrices under Miri.
crates/​cargo-anvil/​src/​region.rs Excludes non-progress rendering mutations.
crates/​cargo-anvil/​src/​dependabot.rs Adds focused parsing tests and reduces Miri matrices.
crates/​cargo-anvil/​src/​catalog/​builder.rs Uses a small catalog for builder tests.
crates/​cargo-anvil/​src/​anvil/​artifacts/​region.rs Skips exhaustive embedded-config checks under Miri.
crates/​cargo-anvil/​src/​anvil/​artifacts/​mod.rs Skips exhaustive catalog scans under Miri.
crates/​cargo-anvil/​src/​anvil/​artifacts/​justfile.rs Skips selected generated-recipe scans under Miri.
crates/​cargo-anvil/​src/​anvil/​artifacts/​github.rs Skips selected workflow-text checks under Miri.
crates/​cargo-anvil/​src/​anvil/​artifacts/​container.rs Skips selected container-text checks under Miri.
crates/​cargo-anvil/​src/​anvil/​artifacts/​ado.rs Skips selected pipeline-text checks under Miri.
crates/​cargo-anvil/​docs/​design/​checks.md Documents retained Miri coverage and workload reductions.

Review effort: Balanced

martin-kolinek and others added 2 commits November 10, 2026 10:37
Keep injected process-group guards, use upstream scripted build supervision, and adapt the listing deadline regression to separate output results.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Run both full-workspace exhaustive groups. Preserve independent Linux observation, earlier reclaim, external memory/task limits, and serial compilation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 10, 2026 09:38
@martin-kolinek martin-kolinek changed the title Stabilize scheduled checks and close Linux mutation coverage gaps fix: stabilize scheduled checks and close Linux mutation coverage gaps Nov 10, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Normal cache-claim paths still leave new caches world-writable under umask 000, while temporary privileged diagnostic tooling and overly permissive diagnostic assertions remain unresolved.

5 open issues

Review effort: Balanced

@codecov-commenter

Codecov Comments Bot (codecov-commenter) commented Nov 10, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.5%. Comparing base (95fa400) to head (0027f2e).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##            main    #357     +/-   ##
=======================================
- Coverage   99.5%   99.5%   -0.1%     
=======================================
  Files        351     331     -20     
  Lines      64328   61842   -2486     
=======================================
- Hits       64029   61545   -2484     
+ Misses       299     297      -2     
Flag Coverage Δ
linux 99.4% <100.0%> (-0.1%) ⬇️
linux-arm 99.4% <100.0%> (-0.1%) ⬇️
scheduled ?
windows 99.5% <100.0%> (-0.1%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

martin-kolinek and others added 2 commits November 10, 2026 12:47
Remove redundant local-item attribute visits, keep platform executable-name helpers cfg-gated, and make initial cache claims private. Strengthen cfg-tail and error diagnostic regressions and restore the Linux-only interrupt test gate.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Exclude only the dispatch adapter's native resume error-to-panic mutation. Scripted resume policy tests still cover checkpoint and delegation error propagation without replacing the harness process.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 10, 2026 12:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Campaign-directory creation lacks independent test coverage, and the mode-preservation test introduces native permission dependencies into the default unit suite instead of being gated behind system-tests.

5 open issues
2 resolved since last review

Review effort: Balanced

Remove the temporary hosted diagnostic harness after both exhaustive jobs pass. Exercise fresh campaign creation independently and gate explicit permission mutation behind system-tests.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 10, 2026 15:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@martin-kolinek
martin-kolinek enabled auto-merge (squash) November 10, 2026 16:27
@st-dev-gh

Copy link
Copy Markdown
Contributor

[copilot speaking] [non-blocking] cargo-gamma-lib's floor is still min-lines-percent = 97.6 (Cargo.toml), while the coverage gate at this head reports 99.0% on Linux and Linux ARM and 99.2% on Windows (Windows ARM does not instrument coverage). Raise it to 99.0 to prevent future drops in coverage.

Comment thread crates/cargo-gamma-lib/src/exec/sync.rs Resolved
Keep build-only helpers private and split independent injected directory-sync failures without changing runtime behavior or native test boundaries.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 10, 2026 18:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are recorded for the scheduled-validation changes.

No open issues 🎉

Review effort: Balanced

@martin-kolinek
martin-kolinek merged commit a093f71 into main Nov 10, 2026
38 checks passed
@martin-kolinek
martin-kolinek deleted the makolnek-microsoft-scheduled-build-timings branch November 10, 2026 20:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants