Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/checkout@v7
with:
persist-credentials: false
- uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
Expand All @@ -42,7 +42,7 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/checkout@v7
with:
persist-credentials: false
- name: Run native mock evaluation through the packaged Action
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/copilot-e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ jobs:
group: evals-${{ github.workflow_ref }}
cancel-in-progress: false
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/checkout@v7
with:
persist-credentials: false
- uses: ./
Expand Down
5 changes: 3 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

- Keep this a thin GitHub adapter over the acquired Vally CLI. Translate
supported commands and consume native JSONL; never duplicate selection,
configuration parsing, grading or execution. No Runtime/Core bundle is required.
configuration parsing, grading or execution.
- Use Node 24, the pinned pnpm version, and one lockfile. Install with scripts
disabled. Run focused tests during development and `pnpm check` before review.
- Rebuild and commit `dist` after source changes. Never manually edit bundles.
Expand All @@ -12,6 +12,7 @@
completion or use a report-only override for native failures.
- Treat prompts, source, raw evidence and credentials as private. Default
artifacts are allowlisted structured projections only.
- Use argument-array process calls and SHA-pinned third-party Actions.
- Use argument-array process calls. Use `actions/checkout@v7`; keep other
third-party Actions SHA-pinned.
- Qualify native behavior with the built-in mock executor and deterministic
graders. Do not substitute model-backed runs for model-free acceptance.
29 changes: 22 additions & 7 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,9 @@ source-to-bundle comparison, and a clean consumer-directory packaging control.
The clean consumer-directory packaging control deliberately lacks artifact
credentials and verifies invalid-input failure. Linux CI separately exercises
the packaged Action with real Vally acquisition, native mock-executor pass/fail
and lint, plus hosted artifact upload. No model credentials belong in CI or
test fixtures.
and lint, plus hosted artifact upload. Do not add model credentials to the
model-free workflow or unit-test fixtures. The separate Copilot E2E workflow
uses its automatic job token for the approved live example.

PR check names map directly to these commands:

Expand All @@ -33,14 +34,14 @@ PR check names map directly to these commands:
| Unit tests | `pnpm test:coverage`: input, policy, reporting, mocked GitHub, tool reuse, orchestration and real model-free native CLI tests with coverage thresholds. |
| Bundle verification | `pnpm check:package`: committed bundle freshness and a clean consumer-directory packaged entry-point test. |
| Action smoke test | Executes `uses: ./` on Ubuntu with actual acquisition, native mock-executor pass/fail, lint, installation reuse and artifact upload. No models run. |
| Copilot E2E | For relevant same-repository PRs, lints the example and runs one real Copilot skill eval. This consumes organization Copilot AI credits. |
| evals (Copilot E2E workflow) | For relevant same-repository PRs, directly invokes the default Action on the example. This consumes organization Copilot AI credits. |

Comment networking tests use synthetic GitHub responses, including serialized
workflow guards, pagination, bot ownership, late runs/attempts, permission
failures and readback. Installer tests use synthetic packages and exercise
concurrent install/reuse and tampered files/receipts. `tests/native.test.ts`
separately qualifies the real pinned Vally CLI with the built-in mock executor
and deterministic graders. No test claims live GitHub bot-token qualification.
and deterministic graders. Unit tests do not establish live GitHub token acceptance.
The live `Copilot E2E` workflow directly exercises the default Action with one
checkout and one `uses: ./` step. It deliberately uses combined Copilot and
PR-write permissions in one job, just like the simplest consumer setup.
Expand All @@ -54,11 +55,12 @@ without installing this repository. `pnpm check:dist` compares build bytes with
the checked-in bundle. Never hand-edit generated files. Update the source,
rebuild, and review both source and distribution changes.

Use strict TypeScript, argument-array process invocation, exact dependencies,
and SHA-pinned third-party Actions. Do not log raw evaluator/process output or
Use strict TypeScript, argument-array process invocation, and exact dependencies.
Use `actions/checkout@v7`; other third-party Actions remain SHA-pinned.
Do not log raw evaluator/process output or
copy native runtime behavior into the Action. New native capabilities require
verified CLI arguments/output contracts and model-free integration controls.
Only trusted own-repository code may eventually receive execution credentials;
Only trusted same-repository code may receive execution credentials;
environment filtering is not an OS sandbox.

The repository's `Copilot E2E` workflow is an explicitly approved live control
Expand All @@ -72,6 +74,19 @@ capability through repository write access and branch governance.

## Release preparation

The first prerelease is [`v0.1.0`](https://github.com/github/evals-action/releases/tag/v0.1.0).
Keep published version tags unchanged. For future releases, verify the packaged
Action at the target commit and update consumer examples to the intended tag.
Action release versions are independent of the selected Vally version.

The moving `v0` tag points to the latest approved 0.x release. After publishing
a new 0.x minor or patch release and verifying its checks, advance only `v0`
to that release's exact commit. Verify the tag target afterward. Do not move
version-specific tags, publish a separate GitHub release for the moving tag,
or use a blanket force-push of all tags. Release publication and advancing the
alias are separate steps; no automatic tag-moving workflow is configured.
Keep `v0` on the 0.x line when introducing a future `v1` release.

Before proposing a release, verify supported CLI versions and their native
evidence contracts, run the Linux model-free controls, and qualify the optional
reporting client with the supported hosted token. Do not advertise a plan
Expand Down
Loading
Loading