[ronniegeraghty]

Summary

Remove the project Docker-backend default that the GitHub Copilot Evals tools do not support yet. With no override, native Vally 0.21.0 selects its local host backend.

• Remove defaults.backend and its Docker-specific grade-network argument from .vally.yaml.
• Remove mandatory Docker/Compose preparation from the manually dispatched eval workflow; restore its repository-pinned host Copilot launcher.
• Update README/authentication/model guidance for host execution, scoped credentials and the current tool compatibility boundary.
• Replace Docker-default regressions with native local-default and supported-project-root checks. Preserve credential restoration/failure/empty-input and independent-reviewer coverage.

This is separate from #15. Dependency pins/lock, standard models, plugin sources, evals, behavior graders, fixtures and historical saved evidence are unchanged. No unrelated complexity proposals are included.

Compatibility and exposure

The installed Evals plugin 0.24.6 full-engine reader accepts project roots paths, suites, environments, params, metadata, but rejects defaults. A source-grounded before/after check rejects defaults before this patch and no root fields afterward; the full-engine environment supports the retained shared setup commands.

This intentionally restores local execution, not a Docker-failure fallback. Tasks, setup and graders run on the host. Workspaces and narrow credentials are not filesystem, credential, network or shared-kernel isolation. Existing run/permission/cost approval requirements remain; no authentication or organization/repository settings change.

Validation

• 31 focused model-free checks and lint pass.
• Native standard-model --dry-run produces 14 plans across seven retained evals and two unchanged models.
• Native project resolution selects local with no backend arguments for every retained eval; dependency versions and CLI aliases are unchanged.
• No paid eval, authentication execution, Docker operation or host dependency install was performed. These checks do not claim live paid-tool execution or behavioral success.

Published head: 0f51091716f3111389b8c4e38b18e2f4c2c4160f. CI and completed latest-head Copilot review will be collected before handing back; no merge is authorized.

[Copilot]

🟡 Changes recommended

The README still requires Docker for paid repository evaluations despite this change removing that dependency, leaving the setup instructions inconsistent.

1 open issue

• README still requires Docker for paid repository evals · New

What changed in this PR

Removes the unsupported project-level Docker default from the shared eval configuration, allowing native Vally to select local host execution.

Changes:

• Removes Docker backend settings and workflow prerequisites, restoring the repository-pinned host Copilot launcher.
• Updates execution, authentication and model guidance to explain host execution and exposure limits.
• Replaces Docker-specific regressions with local-default and supported-project-field checks while preserving credential coverage.

File	Description
README.md	Updates paid-eval guidance for local execution.
docs/​checks-and-evals.md	Documents compatibility, host execution and scoped authentication.
.vally.yaml	Removes Docker backend and grading-network defaults.
.github/​workflows/​evals.yml	Restores the host Copilot launcher and removes Docker readiness checks.
.github/​scripts/​backend-defaults.test.mjs	Checks local backend inheritance and supported configuration fields.

_{Review effort: Max (dynamic)}

[ronniegeraghty]

The single README prerequisite finding from review 1 is addressed in 5119b6e. The focused correction is pushed; CI and a completed review on the changed head remain to be collected.

_{🤖 AI-authored reply via Agency Finish PR.}