[ronniegeraghty]

Summary

Closes #26. Extract the reusable CI comparison machinery from github/plugin-marketplace-starter#15 into an opt-in github/evals-action/comparison subaction with context, run and report commands. The root Action's supported events and HEAD == GITHUB_SHA policy are unchanged.

• context authenticates the exact source workflow path, repository, source run/attempt, same-repository open PR/current head and current merge's two parents, separately from the trusted default-branch workflow and producing attempt. Source head_sha must equal the associated/current PR head. Dotted workflow filenames are supported; malformed selectors have an input-specific error. Trusted job outputs—not artifacts—establish selected coverage.
• run reuses native Vally acquisition/arguments/environment/execution and JSONL validation. Before/After restore only the selected directory, holding measurement files at the merge revision, with live pre-arm context checks, separate HOME/TMP, native saved model configuration, one trial/worker and zero retries. NUL-delimited drift checks include ignored untracked files. Failed Before still allows After. Success requires complete, graded rows and a passing native verdict; skipped/ungraded scenarios cannot pass. Native weighted scoring is unchanged. Partial paid retries fail before acquisition.
• report runs on a fresh publisher-only runner, reads bounded untrusted JSON, checks exact coverage/producing attempt and reconstructs allowlisted failure reasons. It preserves the approved compact grader-count tables, visible After failures, collapsed Before failures and inventory scenario counts. Authenticated bot author IDs support GitHub App bots without duplicating comments or bypassing stale-order protection.

Keep in the starter: sample skills/evals/graders/fixtures, project context, local saved-result reporting, catalog/cohort selection, native model experiments and backend policy, required-reviewer preflight/environment, visible job permissions, head-scoped paid concurrency and non-cancelling publisher serialization. The audit of merged github/plugin-marketplace-starter#12, github/plugin-marketplace-starter#13, github/plugin-marketplace-starter#14, github/plugin-marketplace-starter#16 and github/plugin-marketplace-starter#17 found no other CI runner to lift; native capabilities remain in Vally. This is not a new runner framework or organization-wide permission ceiling.

Final replacement integration pin: 3c8cdb79a1e55b149950627a21e8723817d0e772; both subaction metadata and committed bundle are verified accessible. This supersedes intermediate pins 0083ebd and 323678c. The three-command input/output contract is unchanged. The starter can update its dependent PR before upstream merge/release, but automatic workflow_run activation still requires the trusted workflows/selector on its default branch.

Review feedback: all seven original Copilot threads have been answered and resolved. The string-keyed input lookup is explicit; the originally reported TypeScript error did not reproduce with pinned TypeScript 7. No additional review campaign or approval is claimed.

Vally remains 0.21.0. The earlier claim that it was unpublished was incorrect: npm's failed TLS request fell back to stale metadata. The official 0.21.0 release notes confirm the release. Removed all 217 mirror-specific tarball overrides while preserving every exact dependency version, integrity and graph binding; the lock now uses the consumer's configured registry (public npm by default). No silent downgrade, registry/authentication setting change or supply-chain policy bypass was made.

Verification

• Model-free checks and source-to-bundle verification pass.
• Native integration evidence is distinguished from synthetic controls.
• Credential, process and artifact boundaries were reviewed.
• Documentation and packaged dist match source.

Commands run locally with Node 24.22.2 and pinned pnpm 12.8.1, on Windows ARM64:

• pnpm install --frozen-lockfile --ignore-scripts — passed, including all 548 supply-chain policy checks, using this host's configured package metadata mirror.
• pnpm --dir <clean-consumer> install --frozen-lockfile --offline --ignore-scripts — passed from the verified content-addressed store; the fresh consumer's native CLI reported 0.21.0.
• pnpm build — passed; both committed bundles rebuilt, never manually edited.
• pnpm check — passed: strict typecheck, lint, 260 model-free tests, coverage thresholds, deterministic bundle check and 8 packaging controls, including the registry-portability regression.
• git diff --check — passed.

Actual native evidence uses Vally 0.21.0's built-in mock executor and a deterministic source grader: exact Git Before fails, After passes, neighboring measurement files remain fixed, and the Action still fails the overall comparison. Regressions cover ignored skill/fixture drift, mixed skipped/ungraded rows beside a passing native verdict, private arm state and actual per-check native projection. GitHub provenance/comments/reruns/hostile-artifact cases use synthetic API responses. The comparison bundle is also exercised from a clean consumer directory without dependencies. The whitespace-only filename case is Linux-only and was not executed on this Windows host.

PR-triggered CI remains intentionally skipped with [skip ci] on every pushed commit to avoid the existing paid Copilot E2E PR trigger. No paid E2E run appeared on the replacement head; platform CodeQL/code-scanning checks may still run. These are local results, not green hosted CI; required checks may remain pending. The one explicitly requested Copilot Code Review completed, but no model-backed eval execution, credentials/settings change, merge or release was performed.

Limitations

The PR is ready for review at the user's request. Live authenticated workflow_run/token/protected-environment/billing integration and hosted Linux qualification remain unperformed and require separate authorization; ready status does not establish release readiness or bypass required checks.

A genuinely uncached direct-public registry network restore is unqualified because this host's registry TLS fails. The frozen restore and clean offline control establish lock/graph consistency, not a successful fresh public HTTP download. Earlier stale-cache responses must not be interpreted as version absence.

Acquisition uses the existing Action's job-private generated integrity lock, not the starter's committed root lock: one acquired graph is shared across both arms and digests are recorded, but transitive resolution may differ across jobs. Candidate manifests/hooks are never used for installation.

Approval intentionally exposes execution credentials and ordinary network/shared-runner-kernel capabilities to candidate eval/setup/grader code. Fresh HOME/TMP and filtered environments are hygiene, not a sandbox; artifacts are untrusted data, not independent attestations. Repository approval/permission/concurrency policy remains visible in its workflow, not enforceable as an uneditable organization policy through Action inputs.

[Copilot]

🟡 Changes recommended

Unresolved critical issues block typechecking, make frozen installs depend on an Azure feed, and allow comparisons to pass with skipped trials or consume files left behind by the earlier revision.

7 open issues

• Restore public npm registry URLs in the lockfile · New
• Include ignored untracked files in the drift check · New
• Type the name lookup as a string-keyed record · New
• Require every projected row to be graded before passing · New
• Identify the authenticated bot by author ID · New
• Allow dotted workflow filenames and validate selectors separately · New
• Bind the evaluated head to the source run head SHA · New

What changed in this PR

Adds an opt-in comparison subaction for approval-gated workflow_run jobs, reusing native Vally execution without changing the root Action’s event or checkout policy.

Changes:

• Separates source authentication, Before/After execution and publisher-only reporting into context, run and report commands.
• Adds bounded result projections, artifact validation and sticky-comment publication controls.
• Documents the integration and trust boundaries, adds model-free tests, and packages a separate comparison bundle.

File	Description
vitest.config.ts	Excludes the thin comparison entrypoint from coverage.
tooling/​packaged.test.mjs	Checks standalone comparison-bundle packaging and command rejection.
tooling/​build.mjs	Builds both entrypoints and combines license inventories.
tests/​native.test.ts	Updates the native CLI fixture to Vally 0.21.0.
tests/​comparison-report.test.ts	Tests projection, rendering and untrusted artifact intake.
tests/​comparison-context.test.ts	Tests provenance authentication and selection validation.
tests/​comparison-action.test.ts	Tests execution arms, source restoration and publication controls.
src/​native.ts	Exposes native process exit codes for comparison validation.
src/​comparison-report.ts	Adds bounded projections, artifact loading and comparison rendering.
src/​comparison-index.ts	Adds the comparison entrypoint and sanitized failure handling.
src/​comparison-context.ts	Authenticates source identity, PR context and selection metadata.
src/​comparison-action.ts	Implements comparison commands and native execution orchestration.
README.md	Links the subaction and updates the documented fixture version.
package.json	Pins the development Vally fixture to 0.21.0.
dist/​comparison-index.cjs.LEGAL.txt	Includes bundled third-party license comments.
comparison/​README.md	Documents integration, permissions, trust boundaries and qualification limits.
comparison/​action.yml	Declares subaction inputs, outputs and the Node 24 bundle.

_{Review effort: Max (dynamic)}