Skip to content

Support trusted workflow_run evaluation with authenticated source-PR revision context #26

Description

@ronniegeraghty

Problem

We are using the standard GitHub Actions split: low-permission pull_request work, followed by a trusted default-branch workflow_run workflow with an approval-gated paid eval job and a separate report publisher. We want to use the native Evals Action in that trusted job rather than maintain a CLI invocation indefinitely.

The current policy only accepts pull_request, push and workflow_dispatch, and explicitly rejects workflow_run as unsupported. Separately, assertRevision requires checkout HEAD to equal GITHUB_SHA. For a workflow_run, that SHA describes the trusted default-branch workflow, not necessarily the candidate PR merge revision being evaluated.

Sources: pinned v0.2.0 policy, current policy, GitHub's workflow_run event contract, Security Lab's two-workflow pattern.

Use case

github/plugin-marketplace-starter#15 compares a changed existing skill's exact base/head sources while holding evals, graders, fixtures and neighboring skills at the PR merge revision. Paid execution requires the existing protected environment and repository/job-scoped CI-token billing. Candidate eval/setup/skill code is still untrusted code intentionally executed only after that approval. Artifacts remain untrusted data; moving the trigger does not make them trusted or remove credential/network exposure.

We are temporarily using the existing native Vally CLI in the trusted paid stage. github/plugin-marketplace-starter#17 separately establishes Vally 0.21 Docker defaults and their documented limits. Neither PR requests a model run, repository policy change or broader credentials as part of this issue.

Requested supported behavior

Provide a documented, first-class way to run the Action from a trusted workflow_run while explicitly binding evaluation/reporting to the authenticated triggering run, producing attempt, repository, PR and intended candidate revision. Preserve checkout verification against that authenticated evaluation revision rather than requiring users to overwrite GitHub event variables. The trusted workflow revision and the evaluated revision should remain distinct and visible.

This is not a request to blanket-allow arbitrary events, remove revision/fork guards, trust an artifact's claimed PR/SHA, enable pull_request_target execution, or bypass environment/model-billing policy. Same-repository scope is sufficient for our initial use case; unsupported contexts should fail clearly.

Acceptance

  • A trusted default-branch workflow_run can invoke the Action for an approved source PR/attempt and a separately verified candidate checkout, without pretending to be a pull_request event or changing GITHUB_SHA.
  • Invalid/mismatched source workflow, repository, run/attempt, PR/head or candidate revision fails closed. Untrusted artifacts alone cannot establish provenance.
  • Native execution/reporting retains the actual source and trusted-workflow identities, and reports unsupported publication/event combinations explicitly.
  • Documentation states required approval/token permissions and retained untrusted-code/credential exposure, with tests for both the supported path and rejected contexts.

Goal: switch the starter's temporary native CLI entrypoint back to the supported Action once this contract exists. No implementation in this repository is proposed by this issue.

Activity

  1. added theissue type on Nov 12, 2026
  2. jongio commented on Nov 12, 2026

    @jongio
    Collaborator

    Grooming update (github/evals-action@6e4c077e): Confirmed src/policy.ts:9-27 currently rejects workflow_run and requires checkout HEAD to match GITHUB_SHA. Enrolled in Project #25775 under github/evals-internal#597 (P1 - Next) and assigned @wbreza to review and implement the workflow_run provenance contract (cc @ronniegeraghty).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions