[ronniegeraghty]

What changed

Default repository evals to native Docker using pinned Vally 0.21.0 and .vally.yaml grade-network: none. Native CLI/npm runs and the manual Actions launcher inherit the setting. Node minimum is 22.15.0, matching Vally; Node 24+ is recommended. No custom runner, repository Dockerfile or automatic paid campaign.

• Document Linux Docker/Compose requirements, backend precedence, role separation, authentication and cleanup/exposure limits. Missing Docker/image preparation fails closed, not into host execution.
• Local examples use hidden input for a dedicated personal-account fine-grained token: Copilot Requests account permission and Public repositories access, following official guidance. Existing synthetic fixtures/public pinned upstream need no private-repository or repository-write permission. CI's repository/job-scoped GITHUB_TOKEN billing is unchanged.
• Preserve ambient-token isolation/restoration, empty-input rejection and real child-process export checks. Tests execute fixed synthetic recipes, never Markdown or real authentication. Optional-plugin removal and remote-only customizations remain supported.
• Keep registry-neutral lockfile URLs and restore 109 unchanged-version SHA-512 hashes weakened during the initial upgrade: all 218 dependency entries have SHA-512 pins. URL omission itself changed no other entry metadata; the documented registry metadata fallback preceded URL removal.
• Integrate the reported main conflict in source-branch commit d210f0c, preserving the merged standard-model experiment without executing that real matrix. Extend the model-free Docker failure check to both native eval and experiment entrypoints.

No plugin source/version changes. Only the failed unpublished pr-description 0.1.1 candidate was removed; published 0.1.0 history and raw failed snapshots remain preserved. Host Copilot 1.0.98 and remote pins are unchanged.

Review outcome: ten-cycle limit reached

Final reviewed head: 7cc9927e71e535ad59adbb1edddbeacc14675f0d. All six current GitHub check runs succeeded; the branch is mergeable and ready for human review, not merged.

The tenth completed Copilot review reports No open issues, but Needs a closer look, not unconditional approval. Its remaining human judgment is whether the documentation's PR-specific validation context belongs in a starter template. The note deliberately distinguishes historical broad-token evidence from untested narrow-token recipes, as requested. Response and explicit deferral.

The authorized review loop stops here at 10/10; no eleventh review, merge or behavioral-failure campaign was started. All published inline findings were addressed. Seven focused model-free tests pass, including actual credential-free mock eval/experiment commands that must fail at Docker preparation when no Docker executable exists. No Docker or model is started by those checks.

Retained full-suite execution evidence

The single approved run native-docker-21-full-default-20261112T020833014Z completed 7 evals / 19 cases: 12 behavioral PASS, 7 behavioral FAIL; zero execution errors, grading errors, retries or exclusions. Every selected case completed and was graded. Behavioral failure is valid infrastructure evidence, not an execution error; the manual workflow's existing --require-pass still enforces behavioral passes.

This is historical evidence from the recorded pre-removal snapshot, including the failed unpublished routing candidate, not a paid rerun of the final head. It used the separately approved broad gh credential, not the new narrow-token recipe. Prompts, graders and thresholds were not weakened. No credentials were minted/stored, real authentication changed, settings modified or paid evals rerun for these fixes.

• Source manifest SHA256: d4c7babd2bb63367c66cd6b5e6df8307ccc46a0463e093fb9499257513704c32; corrected derived handoff: 474875a6c7b150c630fa0dff20b6963eff0cd22f9a7b02042d301e7df5c47b2c.
• Native report: outputs/2026-11-12T02-31-32-422Z/eval-results.md, with raw results.jsonl retained in that run folder. Actual container Copilot 1.0.109, Vally 0.21.0, model claude-haiku-5.5, distinct from the host pin.
• Sixteen skill cases exposed 15 skill names (12 catalog plus three built-ins); three independent reviewer cases exposed the three built-ins and actual custom agent. Availability is not activation or behavioral success.
• Immutable custom image: sha256:654c029a370cb5cb0bce6cea326fc47cdeda3eaa67e89f20ff3f02566f98723d, using the official 0.21 recipe plus a reviewed credential-free registry build argument. Explicit Docker/custom-image selection and repeated offline-grade policy were required. This proves native Docker task/grade execution, not bare-default live image bootstrap.
• Cleanup telemetry balanced and observed owner container/volume inventories were empty. Live grading inspection was partial; complete live-isolation inspection is not claimed.

Remaining limits

Canonical automatic bootstrap remains environment-blocked/unverified: observed public npm TLS fails before certificate exchange; the cause is not established. The working configured feed is not inherited by the native builder, which exposes no registry override. No company feed, TLS weakening, private Dockerfile or canonical-cache substitution is committed.

The pinned Copilot Evals plugin Run reader rejects project defaults.backend before execution. Use native Vally CLI; this is unsupported Plugin Run integration, not a local fallback.

Docker is not complete isolation: the host orchestrates/builds; tasks intentionally receive model credentials and normal network access; root initialization/evidence helpers and shared-kernel risk remain. Workspaces, captured evidence and model inputs are retention/data surfaces; interrupted cleanup may leave resources. Offline grading is configured policy, not proof from complete live inspection.

No merge or auto-merge authorization.

[Copilot]

🟡 Changes recommended

The lockfile weakens dependency integrity by dropping SHA-512 pins, while unresolved Node-version and regression-test issues affect supported installs, plugin customizations, and launcher safeguards.

3 open issues

• Gate reviewer eval assertion on the plugin being retained · New
• Reject backend overrides in launcher guard checks · New
• Raise the Node minimum to Vally’s required version · New

What changed in this PR

Moves repository eval tooling to Vally 0.21.0 and defaults execution to native Docker, while keeping paid runs manual-only. Plugin source and versions remain unchanged.

Changes:

• Configures Docker execution with offline deterministic grading.
• Adds Docker preflight checks and three model-free backend-selection and fail-closed regressions.
• Documents authentication, backend precedence, isolation limits, cleanup, and the pinned Evals plugin’s unsupported Run entrypoint.

File	Description
README.md	Introduces Docker requirements, defaults, and native CLI guidance.
package.json	Pins Vally CLI to 0.21.0.
package-lock.json	Updates Vally and transitive dependencies; removes registry-specific tarball URLs.
docs/​checks-and-evals.md	Details Docker setup, authentication, overrides, execution boundaries, and lockfile maintenance.
.vally.yaml	Selects Docker by default and disables grading network access.
.github/​workflows/​evals.yml	Adds Docker/Compose preflight checks and removes host executor settings.
.github/​scripts/​docker-defaults.test.mjs	Adds model-free checks for Docker inheritance and failure without Docker.

_{Review effort: Max (auto)}

[ronniegeraghty]

Addressed the integrity warning in review PRR_kwDOVRlbx88AAAABXe7pXA in 438fea4, including the overview-only finding. Comparing the PR base, the saved pre-URL-removal lockfile, and the reviewed head showed 109 unchanged-version packages had already changed from SHA-512 to SHA-1 during the dependency upgrade. The native omit-lockfile-registry-resolved step itself changed no other package fields. The configured registry currently omits dist.integrity for @types/debug@4.1.13 and supplies dist.shasum, which exactly matches the reviewed SHA-1 value. I restored all 109 SHA-512 values from the base for those exact unchanged versions. All 218 dependency entries now have SHA-512 integrity; no package version or other dependency-entry metadata changed, and zero registry-specific resolved URLs were reintroduced. The earlier preservation statement was too narrow: URL stripping preserved the initial upgrade, but that upgrade had already weakened these hashes. Documentation now calls out this mirror fallback. The three inline findings were also fixed, replied to, and resolved. Model-free lint, five focused tests, catalog consistency, diff checks, and the checksum audit pass. New-head GitHub CI and the next completed Copilot review are still pending; no paid evals or Docker operations were run.

[Copilot]

🟡 Changes recommended

The documentation’s token examples leave existing GitHub credentials exposed to agent-controlled tasks instead of isolating the intended token.

1 open issue

• Clear existing GitHub tokens before exposing the selected credential · New

3 resolved since last review

• Raise the Node minimum to Vally’s required version
• Reject backend overrides in launcher guard checks
• Gate reviewer eval assertion on the plugin being retained

_{Review effort: Max (auto)}

[Copilot]

🟡 Changes recommended

The credential regression test can pass without an exported GH_TOKEN because its fake npm is a shell function, leaving missing credentials in real child processes undetected.

1 open issue

• Test does not verify GH_TOKEN is exported to child processes · New

1 resolved since last review

• Clear existing GitHub tokens before exposing the selected credential

_{Review effort: Max (auto)}

[Copilot]

🔵 Needs a closer look

Human assessment is needed to evaluate authentication failures and unintended credential exposure from the Docker example appearing before its authentication prerequisites.

No open issues 🎉

1 resolved since last review

• Test does not verify GH_TOKEN is exported to child processes

_{Review effort: Max (auto)}

[ronniegeraghty]

Addressed the fourth review's overview concern in d745d0b. The README and paid-evals section now put Docker and scoped authentication prerequisites first. The earlier standalone try-plugin-locally command moved below the authenticated Bash/PowerShell examples, with an explicit instruction to replace only the npm command inside that same credential scope. The existing model-free recipe test now requires the first paid code block to be the scoped authentication example. Six focused checks pass. No model run, real token lookup, Docker restart/build or settings change was performed.

_{🤖 AI-authored reply via Agency Finish PR.}

[Copilot]

🔵 Needs a closer look

Human assessment is needed because forwarding a broadly scoped GitHub token to network-enabled tasks that can execute candidate plugin code creates a critical risk of unauthorized access to unrelated repositories.

No open issues 🎉

_{Review effort: Max (auto)}

[ronniegeraghty]

Addressed the broad-token review finding in 95fa624: the local Docker examples now request a dedicated personal-account fine-grained token with only Copilot Requests and Public repositories access, following GitHub's guidance. The fixtures and pinned public skill source require no private-repository permissions. Custom private GitHub access needs a separate minimum-permissions decision, not automatic widening.

The hidden-input recipes clear competing tokens, reject missing input, expose only the selected token and restore ambient values on success/failure. Fixed synthetic tests cover these properties without authenticating or running models. CI's repository/job-scoped GITHUB_TOKEN billing is unchanged. The retained 19-case execution used the explicitly approved broad credential; it is not narrow-credential validation. Normal task credential/network/shared-kernel exposure remains documented.

No credentials were created/stored, real authentication or settings changed, or paid evals rerun. A fresh review is requested on the changed head; “No open issues” alone is not being treated as an unconditional signoff.

_{🤖 AI-authored reply via Agency Finish PR.}

[ronniegeraghty]

The earlier review summaries are also addressed on the current head: 438fea4 restored unchanged-version SHA-512 pins and fixed Node minimum/plugin-conditional coverage/launcher guards; 62fd858 isolated competing credentials; 1f18b82 added a real child-process export check and its no-export negative control. I rechecked all six linked inline threads; they remain resolved. The earlier integrity-audit comment describes that completed correction, not an outstanding request.

95fa624 retains those safeguards while narrowing the documented local credential as described in the latest response. Current-head review is still pending, not presumed approved.

_{🤖 AI-authored reply via Agency Finish PR.}

[Copilot]

🔵 Needs a closer look

Human assessment is needed because an unpublished finding flags validation checks that still depend on a removed eval path.

No open issues 🎉

_{Review effort: Max (auto)}

[ronniegeraghty]

Investigated cycle 6 on 95fa624:

• No evals/, plugins/, or catalog files differ from base 3a513a3; the referenced eval remains tracked.
• Native discovery: 7 evals/19 cases, zero errors. Full model-free script suite, lint, catalog, and diff checks pass. All six current-head check runs succeeded; review automation success is not approval.
• Zero published inline comments; no filename/reproducer. The removed-path claim remains unsubstantiated. Human assessment needs that missing evidence; No open issues is not clean signoff. No speculative edits, paid reruns, or same-head review request.

_{🤖 AI-authored reply via Agency Finish PR.}

[Copilot]

🟡 Changes recommended

The credential-example tests hard-code the removed optional pr-description plugin’s path, causing comparisons to fail when examples switch to a retained eval.

1 open issue

• Normalize eval paths when comparing credential examples · New

_{Review effort: Max (auto)}

[ronniegeraghty]

Cycle-seven's eval-path customization finding is fixed in de7cf46 and the inline thread is resolved. Comparisons normalize only validated eval-spec paths and separately require the documented paths to exist. Only fixed test-owned recipes execute with synthetic credentials. The failed Linux job actually timed out starting PowerShell after 15 seconds; the process allowance is now bounded at 60 seconds with every correctness assertion retained. Local focused checks pass; new-head Linux CI and completed cycle-eight review remain pending. No credential/auth/settings changes, Docker operations or paid reruns.

_{🤖 AI-authored reply via Agency Finish PR.}

[Copilot]

🟡 Changes recommended

The unconditional filesystem check rejects supported remote-only configurations when both local plugins/ and evals/ trees are absent.

1 open issue

• Allow remote-only setups without local plugin or eval folders · New

1 resolved since last review

• Normalize eval paths when comparing credential examples

_{Review effort: Max (auto)}

[ronniegeraghty]

Cycle-eight's remote-only customization finding is addressed in cb082b9: filesystem existence is required only when either local tree remains; safe paths and fixed synthetic credential checks always run. The regression covers both trees absent, either retained, existing/missing specs and traversal. The inline thread is resolved. Cycle nine is requested on this changed head, not completed approval. No paid reruns, credentials/settings changes or Docker operations.

_{🤖 AI-authored reply via Agency Finish PR.}

[Copilot]

🔵 Needs a closer look

Human assessment is needed because the Docker checks do not verify that experiments preserve isolation and fail rather than execute on the host when Docker is unavailable.

No open issues 🎉

1 resolved since last review

• Allow remote-only setups without local plugin or eval folders

_{Review effort: Max (auto)}

[ronniegeraghty]

Addressed this overview in 7cc9927: the existing model-free fail-closed regression now invokes both vally eval and vally experiment run, with a mock executor, two synthetic experiment variants, an empty Docker PATH/context and no credentials. Each must fail specifically at Docker preparation, not pass by executing the mock on the host. The standard-model npm launcher is also checked for backend/in-place overrides. Seven focused tests pass; no actual model matrix, Docker daemon operation or paid run was started.

This adds missing experiment-entrypoint coverage after main's model-experiment integration; it does not claim complete network/credential/kernel isolation or bare-default-image-bootstrap validation. Actual current-head CI and final review cycle ten remain to be collected.

_{🤖 AI-authored reply via Agency Finish PR.}

[Copilot]

🔵 Needs a closer look

Human judgment is needed to assess whether the documentation’s PR-specific validation reference provides appropriate context for template users.

No open issues 🎉

_{Review effort: Max (auto)}

[ronniegeraghty]

The authorized ten-cycle review limit has now been reached, so I am stopping automated re-review here rather than treating “No open issues” as unconditional approval or requesting cycle eleven.

The documentation's historical validation note (docs/checks-and-evals.md:230–232) deliberately says that model-free recipe checks do not prove authentication and that the retained Docker run used an approved broad credential, not the new narrow-token recipe. That distinction was explicitly requested; it avoids relabeling historical evidence as least-privilege validation. Whether that PR-specific context should remain in a starter template is left for the maintainer's human judgment. No speculative documentation rewrite, paid rerun or credential change is being made.

On the exact reviewed head 7cc9927, all six current check runs succeeded and there are no open inline findings. The completed review remains Needs a closer look, not approval. Docker image/bootstrap, pinned Plugin Run and credential/network/shared-kernel limits remain documented; no merge is authorized.

_{🤖 AI-authored reply via Agency Finish PR.}