You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Compare changed skills through the reusable Evals Action - #15
Keep the starter focused on marketplace PR submissions and collaborative review, using native Vally features instead of custom local reporting code.
Reusable comparison infrastructure moved upstream in github/evals-action#28, addressing github/evals-action#26. All three comparison calls pin accessible corrected commit 3c8cdb79a1e55b149950627a21e8723817d0e772, whose metadata and bundled runtime were verified. The upstream PR is open/unmerged and ready for review, not a release or live-qualified paid workflow. Exact Action contract.
Native local results: removed .github/scripts/eval-report.mjs (546 lines), its mixed report test file (563 lines), and the two reporting-only JSONL fixtures. The two genuine behavior/native-invocation regressions were preserved in focused pr-description.test.mjs, with catalog-derived cohort checks instead of a fixed six-skill list. No custom dashboard/parser or second runner replaces them.
Thin starter policy stays: catalog/diff/eval-path selection, protected-environment preflight, trigger/fork policy, job permissions, saved models/backend, timeouts/retention and authenticated concurrency. Actual evals, graders, behavior fixtures, plugins, model experiment and dependency lock are unchanged. Docker-default removal is separate in Remove unsupported project Docker default #19; this pin update does not fold in that configuration change.
The first migration commit 01bf547 removed a net 1,037 lines. The distinct native-results cleanup commit 18a2ec922795eb9d4b129d8b8e3536423c3b0672 removes another net 909 lines. At that cleanup checkpoint, the PR was a net reduction of 308 lines against main.
Current consumer head: 780ed812a91162b01bb8400415846c33a5b687bc. The final upstream pin includes authenticated source-head provenance, ignored/untracked drift checks including NUL-safe filenames, incomplete/skipped-result handling, authenticated bot identity, dotted workflow paths and a registry-portable acquisition lock. Vally stays 0.21.0 and the three-command input/output contract is unchanged. The selector fixture now models runHead == head, not the merge revision.
Native command surface
npm alias
Actual command
eval
vally eval --workers 1 --max-retries 1
eval:models
vally experiment run evals/standard-models.experiment.yaml --output-dir evals/results/standard-models --require-pass
eval:serve
vally serve evals/results
eval:compare
vally compare
eval:serve uses Vally's native reports/dashboard at loopback port 3200 and does not call a model. The pinned CLI 0.21.0 includes @microsoft/vally-server as a dependency; no separate server installation is needed after npm ci. If paths are customized, update the thin alias or pass the directory directly.
eval:compare uses the native baseline/treatment or experiment comparison judge and spends AI credits; it is not silently described as a model-free file diff. It is not automatically invoked by PR publication or the existing model experiment. Native serve and compare documentation were checked against installed 0.21.0 help/source.
The remaining 20-line lint adapter validates the catalog and delegates to native vally lint, selecting only existing local directories so remote-only customizations still work. Catalog/schema/remote-pin validation and repository PR selection are marketplace-specific checks, not reimplemented Vally engine features. Useful native npm aliases are retained.
PR comparison behavior and trust
The read-only PR notification leads to trusted default-branch workflow_run selection. The protected evals job runs native Vally 0.21.0, sequential Before/After, saved model, one trial/worker and zero retries. Only the target skill changes from the verified merge's first parent to PR head; evals, graders, fixtures and neighbors stay fixed. Before failure does not prevent After, and failed/incomplete outcomes keep the paid job failed.
Approval, token roles and concurrency remain explicit. Owners configure required environment reviewers and permit CI-token billing. The paid job has read-only repository/PR/Actions access and billing permission, not PR-write access. A separate fresh publisher has PR-write access, no billing permission and no checkout/candidate execution. Approval permits candidate code to execute with the model credential; fresh HOME/TMP and a narrow environment are process hygiene, not credential/network/shared-kernel isolation or an organization-wide permission ceiling. No settings/authentication policy changes.
Acquisition uses the existing Action's script-disabled job-private generated integrity lock, never candidate manifests. Both arms share one graph and record digests; transitive resolution can differ across jobs, unlike the previous committed-lock bootstrap.
Head-scoped paid concurrency uses authenticated outputs. Same-head duplicates can cancel; newer heads do not immediately cancel an old in-flight arm. Current-head checks block its next arm and stale publication, not already-billed requests. Publishers serialize without cancellation and enforce producing-run/attempt order. Partial paid retries are rejected; use Re-run all jobs. Publisher-only retries reuse the original producing attempt.
Only selection metadata and exact Before/After projection paths are uploaded, never private diagnostics. Intake treats artifacts as bounded untrusted JSON matching trusted selection/attempt, not independent attestations of candidate code. Raw trajectories, model output, grader evidence, exceptions and transcripts are not published. Retention remains seven days subject to repository policy.
The approved per-skill case/model tables retain passing/total graders, visible After failures, collapsed Before failures and collapsed marketplace case coverage. Unmeasured/unpaired skills are not counted as passing. No full-cohort/model-roster campaign or statistical claim was added. The approved fictional sample remains illustrative and was not reposted.
Customization, qualification and dependency
Model-free fixtures exercise adding/removing local and remote catalog entries and evals, remote-only catalogs, custom-plugin pairs, removed evals, advanced-main merge parents, non-ASCII paths and candidate-script non-execution. Contracts remain explicit: conventional local skill/eval paths, exact pinned GitHub remote skill trees, and shared native context maintained in .vally.yaml. Remote-pin-only changes do not trigger this local-skill comparison policy; arbitrary layouts, full plugin activation and live model behavior are not claimed.
Focused checks, lint and native alias help checks pass; the final pin's 29 affected checks were rerun. Packaged Action metadata/runtime and native serve/compare capabilities were source-verified. No server was launched, paid comparison/eval run, authentication check, Docker operation, setting change or merge was performed. Live workflow_run, token acceptance, approval, runtime parity and billing remain separately qualified concerns. Upstream's uncached direct-public dependency restore remains unqualified on its TLS-restricted host; portable lock/cached-consumer checks are not relabeled as that network proof.
Review the unmerged upstream dependency before deployment. Both starter workflows and the trusted selector must reach the default branch for automatic activation. Previous Copilot review 9 assessed 4f212e9, not this work. No review 10 or new review campaign was requested on this PR; human review remains outstanding.
Require protected-environment approval before paid same-repository runs; hold eval measurements fixed while switching only the target skill source and reuse the Action's verified same-job tooling.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Missing working-directory inputs prevent both eval invocations from running, and the selection job lacks the actions: read permission needed for private-repository environment checks.
Introduces approval-gated before/after evaluations for changed existing plugin skills, preserving the marketplace’s shared evaluation setup while switching only the selected skill’s source.
Changes:
Adds sequential, pinned evals-action runs against exact base and PR-head skill versions.
Requires protected-environment approval for same-repository PRs and skips skills without a valid comparison pair.
Adds model-free workflow checks and documents setup, billing, and execution boundaries.
File
Description
README.md
Introduces the approval-gated PR evaluation workflow.
docs/checks-and-evals.md
Documents comparison inputs, approval requirements, and execution limits.
.github/workflows/skill-evals.yml
Adds skill selection, source switching, and sequential evaluation jobs.
.github/workflows/evals.yml
Updates comments to distinguish manual smoke evaluations from skill comparisons.
.github/scripts/validation.test.mjs
Adds checks for workflow policy, selection, source restoration, and measurement drift.
Mock-up only: all results are fictional. Example PR changes marketplace-maintainer/release-plugins and pr-description/pr-description.
Changed skills: 📈 2 resolved · 📉 1 new failure. Other skills: 📉 1 new failure. Failure totals count grader/model results.
Sources: Before <base SHA> → After <PR head SHA> · Evidence:<workflow run link> Trials: One per scenario/model/arm, zero retries. Observed differences do not prove causation.
Read cells as Before → After. A scenario is one test case; graders are its individual checks. Scenario labels below omit their repeated <eval>- prefix; evidence retains the full native IDs.
🌐 All marketplace skills: 6 evaluated · 📉 1 new failure in an unchanged skill · 6 not evaluated
Cells: passing scenarios / total scenarios, Before → After. A scenario passes only when it meets its configured scoring rule; these examples require every grader to pass.
Skill (passing / total scenarios)
Claude Sonnet 5.5
GPT-6.1 Terra
marketplace-maintainer/release-plugins
🔄 4/5 → 4/5 (1 resolved, 1 new failure)
✅ 5/5 → 5/5
marketplace-maintainer/update-remote-pins
✅ 3/3 → 3/3
✅ 3/3 → 3/3
plugin-authoring/add-plugin
✅ 2/2 → 2/2
✅ 2/2 → 2/2
plugin-authoring/submit-plugin
✅ 2/2 → 2/2
✅ 2/2 → 2/2
plugin-authoring/try-plugin-locally
📉 2/2 → 1/2
✅ 2/2 → 2/2
pr-description/pr-description
📈 1/2 → 2/2
✅ 2/2 → 2/2
eval/compare
⏭️ Not evaluated — no repo eval
⏭️ Not evaluated — no repo eval
eval/create
⏭️ Not evaluated — no repo eval
⏭️ Not evaluated — no repo eval
eval/improve
⏭️ Not evaluated — no repo eval
⏭️ Not evaluated — no repo eval
eval/review
⏭️ Not evaluated — no repo eval
⏭️ Not evaluated — no repo eval
eval/run
⏭️ Not evaluated — no repo eval
⏭️ Not evaluated — no repo eval
eval/setup
⏭️ Not evaluated — no repo eval
⏭️ Not evaluated — no repo eval
The six remote Evals skills have no repository evals. Loading them is not behavioral coverage.
🔎 Marketplace failure details: 2 After · 2 Before (grader/model results)
The command omitted the selected plugin's absolute local path.
<after failed assertion/evidence link>
Legend: ✅ All pass in both arms · 📈 More pass · 📉 Fewer pass · 🔄 Same count, different failures · ❌ Failed grader. Change labels distinguish new, resolved and persistent failures.
⚠️ Error · ⏭️ Not run · ❔ Unknown never count as passing. Use 3/4 → ⚠️ Error, not an invented After count; report partial coverage separately.
Same evals, graders and fixtures across arms; only the PR's changed skill sources differ. Links and SHAs are placeholders, not actual run evidence.
Preserve the pending main integration, fix Action inputs and environment-read permissions, and add the approved compact comparison presentation without expanding paid execution scope.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Avoid upload collisions and importing prior-attempt verdicts when a maintainer reruns the same workflow.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
With workflow-token billing enabled, PR-editable workflows can bypass maintainer approval while retaining paid Copilot access, creating a critical authorization gap.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The publishing job can execute PR-controlled code with pull-request write permissions without protected-job approval, creating a privilege-escalation risk.
Require explicit passing verdicts before declaring resolution
.github/scripts/skill-eval-comment.mjs:215
A missing check in the other arm is treated as a passing verdict: Before with completed=true, routing=false and After with only completed=true produces an improving 1/2 → 1/1 cell and labels routing as resolved. Require an explicit passing verdict for the matching check before declaring a resolution; otherwise mark that comparison unknown or incomplete. Apply the same presence check to the headline totals, transition markers, and marketplace comparison counts.
Run the publisher from a trusted revision
.github/workflows/skill-evals.yml:237
This job executes the PR's comment helper with pull-requests: write, even when a new, unpaired skill causes the protected eval job to be skipped. A same-repository integration with source-file write access but no workflow-editing or pull-request-write permission can replace that helper and gain the job's broader token without approval. Run the publisher from a trusted revision rather than the PR checkout, bootstrapping this new helper on the trusted branch before enabling the workflow.
Replace native JSON parse errors with a fixed diagnostic
.github/scripts/skill-eval-comment.mjs:119
If native stdout contains a non-JSON line after the first JSON record, JSON.parse can include a fragment of that line in its exception message. The CLI logs that message verbatim at line 339, exposing raw evaluator content even when the Action has already classified the evidence as malformed-native-jsonl. Catch the parsing exception here and replace it with a fixed diagnostic.
Require matching complete grader evidence for transitions, redact parse failures, cancel stale PR runs, and reject partial retries before either paid arm. Run the publisher from the trusted base and document the still-open billing trust decision without changing triggers.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Remove write permissions from PR publication code; validate the authenticated producer workflow, run, attempt, PR and merge revisions, and treat bounded artifact JSON as untrusted display data. Preserve the existing native paid entrypoint and document its separate approval-boundary gap.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
A same-repository PR author can modify the workflow to bypass evaluation approval and obtain a billing-capable token, enabling unauthorized use of the organization’s Copilot credits.
Keep the PR notifier read-only and use default-branch workflow_run code for selection, environment-approved native CLI execution, and separate fresh-runner publication. Pin Vally 0.21 outside the candidate checkout without spoofing GitHub events. Bind selected revisions and producing attempts, preserve partial grading failures and escape labels once. Track supported Action migration in github/evals-action#26; owner billing policy and approved candidate credential exposure remain explicit.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
ronniegeraghty
changed the title
Add approval-gated before/after skill evals with evals-action
Add trusted before/after skill evals with native Vally
Nov 12, 2026
The current native migration is pushed in 4430df5. The PR notifier now has read-only contents access; trusted default-branch workflow_run selection leads to the existing protected paid job, followed by a fresh publisher-only runner. The original Action-directory/selection-read, partial-grading/check-presence, source-attempt and trusted-publication findings remain covered; this update also preserves completed checks beside grading errors, escapes labels once and skips cancelled source runs.
github/evals-action#26 tracks returning to the Action once trusted source-event/revision handling is supported. The approved fictional sample remains unchanged, not actual measurement evidence. No paid evals or credential/settings changes were made. Current-head CI and cycle-four review are pending, not presumed clean. This entrypoint split does not constrain other workflows an authorized editor can create; owner billing policies and approved candidate-process exposure remain explicit.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
An invalid runner reference in job-level env prevents the workflow from running, while missing PR-read permission blocks private-repository selection and shared state compromises comparison isolation.
Grant selection PR-read access, initialize the native CLI path in a step rather than unavailable job runner context, and give Before and After separate fresh homes and temporary directories. Model-free command execution checks both state boundaries without running models.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Upload only the selected skill before/after verdict JSON files, not additional files candidate processes could place in the shared temporary directory. Trusted selection remains a separate read-only-job artifact. Add a focused upload-path regression; candidate execution exposure remains explicit.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ncies
Integrate main b037e36 after the user merged Docker and standard-model foundations, preserving comparison changes. Bootstrap native Vally only from the trusted revision manifest and integrity lock using npm ci with install scripts disabled, outside the candidate checkout. Derive the baseline from the verified current merge first parent rather than stale PR API base metadata. Model-free regressions and actual locked CLI version check pass; no models, settings or final PR merges.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Addressed both concerns in 406535f, integrating the now-user-merged main foundation b037e36 rather than duplicating or cherry-picking the Docker changes:
Native tools now bootstrap with npm ci --ignore-scripts from the exact trusted default-branch checkout's manifest and integrity lock, copied into a separate temporary tool directory. Candidate dependency files and install hooks are not used. The job summary records that trusted lock revision. An exact-shell synthetic test distinguishes trusted/candidate manifests; a real locked dependency restore and CLI version check resolved Vally 0.21.0 without models.
Selection fetches the actual current PR merge commit, verifies both parents and the expected source head, and takes its first parent as Before. It no longer assumes the PR API's base.sha matches the current merge parent. A stale-base-metadata regression covers this.
These are reproducibility/provenance fixes, not a harmless-dependency or complete process-isolation guarantee. The next changed-head review/CI remain to be collected; no paid eval, authentication/settings change or final PR merge was performed.
…paths
Reuse a trusted current-PR check after environment approval and before Before/After, using immutable selected context and minimum PR-read permission. Exclude the selected skill with a literal Git pathspec instead of parsing displayed filenames. Regressions cover closed/moved/fork PRs and non-ASCII source files while retaining outside-skill drift rejection.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Both seventh-review findings are fixed in a519c9b: literal Git pathspec exclusion handles non-ASCII/quoted skill filenames without hiding outside-skill drift; trusted PR-open/current-head checks run after approval immediately before each paid arm, using immutable selected context and PR-read-only access. The relevant model-free checks pass. No model was run or settings changed. Cycle eight is requested on this changed head, not presumed approved.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed because stale or unauthenticated notifications can cancel pending or approved comparisons before the replacement run is rejected or skipped.
Remove workflow-level privileged cancellation driven by unverified notifications. Only successfully selected paid jobs can cancel duplicates for the verified PR/head/skill; publisher jobs serialize by verified PR without cancellation. Document that a newer head does not immediately cancel an in-flight older arm, and that pre-arm checks stop only subsequent work. No custom cancellation controller or model runs.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Fixed in 4f212e9: privileged workflow-level concurrency no longer uses the unauthenticated notification. Paid cancellation is admitted only after successful trusted selection and is keyed by its verified repository/PR/head/skill. Rejected notifications and delayed old-head selections cannot cancel another head's approved work. Publisher jobs serialize by the verified PR without cancelling each other; current-head and producing-run ordering checks still prevent stale replacement.
The native tradeoff is explicit: same-head duplicates can be cancelled, but a newer head does not immediately cancel an already-running old-head arm. The pre-arm check blocks its next paid arm/publication; it cannot roll back existing model charges. This scoped choice was confirmed, and no custom cancellation infrastructure was added. Focused model-free declaration/helper/order checks pass; live workflow execution is not claimed. No paid runs or settings changes.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed because evaluation error handling can hide recorded failures or misclassify incomplete evidence as complete, making failure comparisons unreliable.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The notifier can bind an old run to a newer PR head and cancel an approved comparison, while the publisher queue can discard newer results and leave them unpublished.
Keep repository selection, approval, token permissions and concurrency visible while moving paired native execution, bounded projections and publication to the pinned comparison subaction.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
ronniegeraghty
changed the title
Add trusted before/after skill evals with native Vally
Compare changed skills through the reusable Evals Action
Nov 12, 2026
Remove the custom local reporting parser and its private result fixtures. Preserve behavioral grader regressions, keep direct native eval aliases, and document customization contracts and comparison judge cost.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use published Action commit 323678c48d0f576da8fbcaf3e4bbe8d95e439641 in all three jobs, preserve Vally 0.21.0, and model the authenticated source head in the repository selector fixture.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use published Action commit 3c8cdb79a1e55b149950627a21e8723817d0e772 after its final filename-drift correction, preserving Vally 0.21.0 and the existing command contract.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
What changed
Keep the starter focused on marketplace PR submissions and collaborative review, using native Vally features instead of custom local reporting code.
comparisoncalls pin accessible corrected commit3c8cdb79a1e55b149950627a21e8723817d0e772, whose metadata and bundled runtime were verified. The upstream PR is open/unmerged and ready for review, not a release or live-qualified paid workflow. Exact Action contract..github/scripts/eval-report.mjs(546 lines), its mixed report test file (563 lines), and the two reporting-only JSONL fixtures. The two genuine behavior/native-invocation regressions were preserved in focusedpr-description.test.mjs, with catalog-derived cohort checks instead of a fixed six-skill list. No custom dashboard/parser or second runner replaces them.The first migration commit
01bf547removed a net 1,037 lines. The distinct native-results cleanup commit18a2ec922795eb9d4b129d8b8e3536423c3b0672removes another net 909 lines. At that cleanup checkpoint, the PR was a net reduction of 308 lines against main.Current consumer head:
780ed812a91162b01bb8400415846c33a5b687bc. The final upstream pin includes authenticated source-head provenance, ignored/untracked drift checks including NUL-safe filenames, incomplete/skipped-result handling, authenticated bot identity, dotted workflow paths and a registry-portable acquisition lock. Vally stays 0.21.0 and the three-command input/output contract is unchanged. The selector fixture now modelsrunHead == head, not the merge revision.Native command surface
evalvally eval --workers 1 --max-retries 1eval:modelsvally experiment run evals/standard-models.experiment.yaml --output-dir evals/results/standard-models --require-passeval:servevally serve evals/resultseval:comparevally compareeval:serveuses Vally's native reports/dashboard at loopback port 3200 and does not call a model. The pinned CLI 0.21.0 includes@microsoft/vally-serveras a dependency; no separate server installation is needed afternpm ci. If paths are customized, update the thin alias or pass the directory directly.eval:compareuses the native baseline/treatment or experiment comparison judge and spends AI credits; it is not silently described as a model-free file diff. It is not automatically invoked by PR publication or the existing model experiment. Native serve and compare documentation were checked against installed 0.21.0 help/source.The remaining 20-line lint adapter validates the catalog and delegates to native
vally lint, selecting only existing local directories so remote-only customizations still work. Catalog/schema/remote-pin validation and repository PR selection are marketplace-specific checks, not reimplemented Vally engine features. Useful native npm aliases are retained.PR comparison behavior and trust
The read-only PR notification leads to trusted default-branch
workflow_runselection. The protectedevalsjob runs native Vally 0.21.0, sequential Before/After, saved model, one trial/worker and zero retries. Only the target skill changes from the verified merge's first parent to PR head; evals, graders, fixtures and neighbors stay fixed. Before failure does not prevent After, and failed/incomplete outcomes keep the paid job failed.Approval, token roles and concurrency remain explicit. Owners configure required environment reviewers and permit CI-token billing. The paid job has read-only repository/PR/Actions access and billing permission, not PR-write access. A separate fresh publisher has PR-write access, no billing permission and no checkout/candidate execution. Approval permits candidate code to execute with the model credential; fresh HOME/TMP and a narrow environment are process hygiene, not credential/network/shared-kernel isolation or an organization-wide permission ceiling. No settings/authentication policy changes.
Acquisition uses the existing Action's script-disabled job-private generated integrity lock, never candidate manifests. Both arms share one graph and record digests; transitive resolution can differ across jobs, unlike the previous committed-lock bootstrap.
Head-scoped paid concurrency uses authenticated outputs. Same-head duplicates can cancel; newer heads do not immediately cancel an old in-flight arm. Current-head checks block its next arm and stale publication, not already-billed requests. Publishers serialize without cancellation and enforce producing-run/attempt order. Partial paid retries are rejected; use Re-run all jobs. Publisher-only retries reuse the original producing attempt.
Only selection metadata and exact Before/After projection paths are uploaded, never private diagnostics. Intake treats artifacts as bounded untrusted JSON matching trusted selection/attempt, not independent attestations of candidate code. Raw trajectories, model output, grader evidence, exceptions and transcripts are not published. Retention remains seven days subject to repository policy.
The approved per-skill case/model tables retain passing/total graders, visible After failures, collapsed Before failures and collapsed marketplace case coverage. Unmeasured/unpaired skills are not counted as passing. No full-cohort/model-roster campaign or statistical claim was added. The approved fictional sample remains illustrative and was not reposted.
Customization, qualification and dependency
Model-free fixtures exercise adding/removing local and remote catalog entries and evals, remote-only catalogs, custom-plugin pairs, removed evals, advanced-main merge parents, non-ASCII paths and candidate-script non-execution. Contracts remain explicit: conventional local skill/eval paths, exact pinned GitHub remote skill trees, and shared native context maintained in
.vally.yaml. Remote-pin-only changes do not trigger this local-skill comparison policy; arbitrary layouts, full plugin activation and live model behavior are not claimed.Focused checks, lint and native alias help checks pass; the final pin's 29 affected checks were rerun. Packaged Action metadata/runtime and native serve/compare capabilities were source-verified. No server was launched, paid comparison/eval run, authentication check, Docker operation, setting change or merge was performed. Live
workflow_run, token acceptance, approval, runtime parity and billing remain separately qualified concerns. Upstream's uncached direct-public dependency restore remains unqualified on its TLS-restricted host; portable lock/cached-consumer checks are not relabeled as that network proof.Review the unmerged upstream dependency before deployment. Both starter workflows and the trusted selector must reach the default branch for automatic activation. Previous Copilot review 9 assessed
4f212e9, not this work. No review 10 or new review campaign was requested on this PR; human review remains outstanding.