Skip to content

evals: gate paid tiers on PRs behind the paid-evals label #137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JRichlen merged 1 commit into from
Sep 23, 2026
Merged

evals: gate paid tiers on PRs behind the paid-evals label #137

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions .github/workflows/evals.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,19 @@ name: evals
# detect → run(matrix, path-filtered, never-required) → aggregate(always(),
# REQUIRED) fan-out, so an untouched or pack-less plugin costs nothing while the
# required status check is always emitted (never left hanging on a skipped leg).
#
# SPEND GATE: on pull requests the paid legs (behavioral-run, routing-eval,
# deep-run) only execute when the PR carries the `paid-evals` label. Without
# it they skip and their aggregates report green, so an agent pushing fix after
# fix to a PR no longer re-buys the full paid suite on every push. Add the label
# when a PR is ready for a paid pass (adding it triggers a run); remove it again
# to stop paying for follow-up pushes. Pushes to main and manual dispatch are
# unaffected.
on:
push:
branches: [main]
pull_request:
types: [opened, synchronize, reopened, labeled]
workflow_dispatch:

concurrency:
Expand Down Expand Up @@ -264,7 +273,8 @@ jobs:
if: >-
needs.behavioral-detect.outputs.plugins != '[]' &&
(github.event_name != 'pull_request' ||
github.event.pull_request.head.repo.full_name == github.repository)
(github.event.pull_request.head.repo.full_name == github.repository &&
contains(github.event.pull_request.labels.*.name, 'paid-evals')))
Comment on lines +276 to +277

@chatgpt-codex-connector chatgpt-codex-connector Bot Sep 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Update the testing inventory for the new label gate

This predicate re-scopes when the behavioral tier runs, and equivalent predicates now re-scope the routing and deep tiers, but docs/testing.md remains unchanged: its “Fires” entries at lines 152-154, 205-207, and 358-360 still describe these jobs as path-gated without mentioning that PRs also require the paid-evals label. This leaves the canonical inventory claiming qualifying PRs are evaluated when their required aggregates can now report green without running the paid legs; update those entries in this change.

AGENTS.md reference: AGENTS.md:L129-L139

Useful? React with 👍 / 👎.

runs-on: ubuntu-latest
strategy:
fail-fast: false
Expand Down Expand Up @@ -446,7 +456,8 @@ jobs:
name: routing tier (roster trigger routing)
if: >-
github.event_name != 'pull_request' ||
github.event.pull_request.head.repo.full_name == github.repository
(github.event.pull_request.head.repo.full_name == github.repository &&
contains(github.event.pull_request.labels.*.name, 'paid-evals'))
Comment on lines 458 to +460
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
Expand Down Expand Up @@ -714,7 +725,8 @@ jobs:
needs.deep-detect.outputs.changed == 'true' &&
needs.deep-detect.outputs.plugins != '[]' &&
(github.event_name != 'pull_request' ||
github.event.pull_request.head.repo.full_name == github.repository)
(github.event.pull_request.head.repo.full_name == github.repository &&
contains(github.event.pull_request.labels.*.name, 'paid-evals')))
runs-on: ubuntu-latest
# No `environment:` — the pier run no longer pauses on the protected
# `deep-evals` environment for a maintainer to click Approve, so an
Expand Down
Loading