[jongio]

Summary

Consolidated repo-maintain and update-remote-pins pass across workflow concurrency/timeout hygiene, updating the remote eval plugin pin to copilot-evals-v0.24.6 (5c3d5213f88149fe1e5476ef029277fe56fd42e4), script branch coverage, and strict Vally 0.21.0 skill/eval lint checks.

Phase Breakdown

Phase	Commit	Summary
1. Governance & Workflow Hygiene	71cf8c4	Add pull-request concurrency (cancel-in-progress: ${{ github.event_name == 'pull_request' }}) and timeout-minutes: 15 to .github/workflows/validate.yml while preserving SHA-pinned actions/checkout and actions/setup-node (node-version: 24, persist-credentials: false, cache: npm).
2. Remote Pin & Dependencies	9fc27c1	Update the eval (github/evals, packages/plugin) remote pin in marketplace.json, .vally.yaml, and README.md from copilot-evals-v0.24.0 (34ddb5803aeaee8330ab70c19ad1a3f37fb77ded) to copilot-evals-v0.24.6 (5c3d5213f88149fe1e5476ef029277fe56fd42e4, version: "0.24.6").
3. Hardening & Config Audit	—	Clean pass; zero secret/CodeQL/Dependabot alerts, argument-array process invocations, and clean package-lock.json integrity hashes without registry tarball URLs.
4. Idiomatic Refactoring	—	Clean pass; avoided churn in .github/scripts/eval-report.mjs while #15 is open.
5. Coverage & Script Checks	1b226f1	Add .github/scripts/lint.test.mjs covering plugins-only marketplaces and catalogs without remote entries (95.20% line / 81.30% branch / 100% function coverage across .github/scripts/*.mjs).
6. Docs, Skills & Evals	—	npm run lint (6/6 skills + 7/7 evals clean under vally lint --strict), Agent Plugins schema validation (3/3 manifests valid), check-local-catalog.mjs, check-remote-pins.mjs, and npm run eval:models -- --dry-run (14/14 plans) passing with zero model calls.

Upstream eval Remote Pin Review (update-remote-pins)

• Old pin: copilot-evals-v0.24.0 @ 34ddb5803aeaee8330ab70c19ad1a3f37fb77ded (version: "0.24.0")
• New pin: copilot-evals-v0.24.6 @ 5c3d5213f88149fe1e5476ef029277fe56fd42e4 (version: "0.24.6", packages/plugin/plugin.json)
• Upstream changes reviewed (34ddb58...5c3d5213):
  • Removes persistent workspace .run-lock markers (skills/run/SKILL.md clarifies recovery files are retained and stale .run-lock markers neither block Native Run / standalone Vally nor authorize executor fallback).
  • Refreshes compiled Canvas, extension, and MCP server bundles (extension.mjs, web/app.mjs, workbench.mjs, dist/mcp/server.mjs) and Windows x64/arm64 native helpers.
  • mcp.json, copilot-extension.json, THIRD-PARTY-NOTICES.txt, LICENSE (GitHub Limited Use Source License), hooks (none), network destinations, and public repository access (github/evals is public; no private-repository token required) are unchanged.

Repo Health Score: 94/100 (A) -> 97/100 (A+) (+3)

Dimension	Before	After	Max	Evidence
CI & Workflow Hygiene	17	20	20	Validate & CodeQL green on main; 40-char SHA pins; added PR concurrency + timeout-minutes in validate.yml
Test & Branch Coverage	20	20	20	560/560 script & recipe tests passing; 95.20% line / 81.30% branch / 100% function coverage
Dependency & License Health	15	15	15	@microsoft/vally-cli@0.21.0 (>= 7d, MIT); eval remote pin synced to v0.24.6; @github/copilot (SEE LICENSE IN LICENSE.md) left for maintainer disposition
Hardening & Config Posture	15	15	15	0 Dependabot/CodeQL findings; Private Reporting enabled; package-lock.json free of registry URLs
Documentation Accuracy	10	10	10	0 broken local links; README.md eval links synced to copilot-evals-v0.24.6 / 5c3d5213
Skill & Eval Quality	10	10	10	vally 0.21.0 --strict clean across 6 skills & 7 evals; evals.yml manual-only with environment: evals
Backlog SLA	7	7	10	Open items triaged (#15 active maintainer PR; #6 bot memory PR; #11 open >24h without comments)
Total	94 (A)	97 (A+)	100	+3 points

Verification

• npm ci --no-audit --no-fund
• VALLY_TELEMETRY_OPTOUT=1 npm run lint
• node .github/scripts/check-local-catalog.mjs
• node .github/scripts/check-remote-pins.mjs
• bash .github/scripts/validate-schema.sh
• npm run eval:models -- --dry-run
• node --test .github/scripts/lint.test.mjs .github/scripts/validation.test.mjs .github/scripts/eval-report.test.mjs .github/scripts/docker-defaults.test.mjs

[Copilot]

🔵 Needs a closer look

The runtime upgrades and run-lock recovery changes introduce cross-host compatibility, cancellation, and executor-fallback interactions that require maintainer review beyond mechanical consistency checks.

No open issues 🎉

What changed in this PR

Updates the marketplace’s optional Evals plugin pin and strengthens repository validation without changing local plugin behavior.

Changes:

• Synchronizes Evals v0.24.6 references across the catalog, shared eval environment, and README.
• Adds validation workflow concurrency controls and a 15-minute job timeout.
• Adds regression coverage for plugins-only marketplaces and catalogs without remote entries.

File	Description
README.md	Updates upstream setup and license links to the new Evals pin.
marketplace.json	Pins the remote Evals plugin to v0.24.6 and its full commit SHA.
.vally.yaml	Aligns shared eval environment setup with the catalog pin.
.github/​workflows/​validate.yml	Adds concurrency controls and a validation job timeout.
.github/​scripts/​lint.test.mjs	Tests plugins-only linting and remote-pin checks with no remote entries.

_{Review effort: Balanced (dynamic)}