evals: gate paid tiers on PRs behind the paid-evals label - #137
Conversation
Every push to a PR re-ran the full paid suite (12 promptfoo packs x repeat:3 with the Anthropic grader, plus routing and deep tiers). PR #131 was pushed 6 times on 2026-09-22 and bought the whole suite each time. Paid legs now run on PRs only when the PR has the paid-evals label; the aggregates already report green on a skipped leg. Pushes to main and workflow_dispatch are unchanged.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The label trigger can rerun paid evaluations for unrelated labels, and the testing documentation is outdated.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 1
Open (2)
What changed in this PR
Gates paid PR evaluation tiers behind the paid-evals label while preserving main-branch and manual runs.
Changes:
- Adds label-based PR triggering.
- Gates paid behavioral, routing, and deep evaluations.
- Preserves aggregate reporting for skipped jobs.
| File | Summary | Findings |
|---|---|---|
.github/workflows/evals.yml |
Implements label-based paid-tier gating. | Restrict label-triggered runs to additions of paid-evals; update docs/testing.md to document the new gate. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| github.event_name != 'pull_request' || | ||
| github.event.pull_request.head.repo.full_name == github.repository | ||
| (github.event.pull_request.head.repo.full_name == github.repository && | ||
| contains(github.event.pull_request.labels.*.name, 'paid-evals')) |
| push: | ||
| branches: [main] | ||
| pull_request: | ||
| types: [opened, synchronize, reopened, labeled] |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 97c92960c3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| (github.event.pull_request.head.repo.full_name == github.repository && | ||
| contains(github.event.pull_request.labels.*.name, 'paid-evals'))) |
There was a problem hiding this comment.
This predicate re-scopes when the behavioral tier runs, and equivalent predicates now re-scope the routing and deep tiers, but docs/testing.md remains unchanged: its “Fires” entries at lines 152-154, 205-207, and 358-360 still describe these jobs as path-gated without mentioning that PRs also require the paid-evals label. This leaves the canonical inventory claiming qualifying PRs are evaluated when their required aggregates can now report green without running the paid legs; update those entries in this change.
AGENTS.md reference: AGENTS.md:L129-L139
Useful? React with 👍 / 👎.
Brings in #137 (paid tiers gated behind the paid-evals label). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DghMFX2SuCASz3fjHoYwjb
Brings in #137 (paid tiers gated behind the paid-evals label). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DghMFX2SuCASz3fjHoYwjb
Every push to a PR re-ran the full paid suite (12 promptfoo packs x repeat:3 with the Anthropic grader, plus routing and deep tiers). PR #131 was pushed 6 times on 2026-09-22 and bought the whole suite each time.
Paid legs now run on PRs only when the PR has the paid-evals label; the aggregates already report green on a skipped leg. Pushes to main and workflow_dispatch are unchanged.
To get a paid pass on a PR: add the
paid-evalslabel (this triggers a run). Remove it to stop paying for later pushes.🤖 Generated with Claude Code
https://claude.ai/code/session_0169LhXkWbFbcHjngKSCLJBV