Repository navigation
feat(eval): OpenRouter CI tier on DeepSeek V4.1 Flash - #2
Merged
Merged
Conversation
PR and weekly skill evals run on deepseek/deepseek-v4.1-flash through OpenRouter when the OPENROUTER_API_KEY secret is set, so they no longer need a maintainer's Claude subscription. Claude runs become a manual dispatch (provider: claude) and a step in the release checklist. Harness: - OpenRouter model ids select the openrouter provider; the runner sets the OpenRouter environment itself and strips Claude credentials. A session that used another model is recorded as incomplete (provider_model). - OpenRouter models are run-time only; task files may not declare them. - OpenRouter runs are priced from list prices (domain/pricing.py) instead of Claude Code's Claude-priced total_cost_usd. - --max-turns-factor scales each task's max_turns; the factor is stored with runs and baselines, and turn-limit stops are reported. - Baselines are kept per model (baselines/openrouter/<model name>/); gate and baseline update find them from the runs' model. CI: preflight picks the provider, each job gets only that provider's credentials, OpenRouter caps, a smoke session, and the key's real spend before and after (scripts/openrouter-ci.py). OpenSpec change: openrouter-ci-eval-tier. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
Skill Evaluation: rf-desktop (narrow, treatment, N=3, deepseek/deepseek-v4.1-flash)Model: OpenRouter spend
Gaterf-skill-eval reportRuns: 3 · pass: 1 · fail: 2 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.05 · turn-limit stops: 1 Pass rate = runs whose gate result is Tasks
Skills
Skipped / errored checksNone. Incomplete runsNone. Process checks (treatment arm, not in deltas)
Adversarial tier (reported only, never gates)None. |
Skill Evaluation: rf-mobile (narrow, treatment, N=3, deepseek/deepseek-v4.1-flash)Model: OpenRouter spend
Gaterf-skill-eval reportRuns: 3 · pass: 3 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.07 Pass rate = runs whose gate result is Tasks
Skills
Skipped / errored checksNone. Incomplete runsNone. Process checks (treatment arm, not in deltas)
Adversarial tier (reported only, never gates)None. |
Skill Evaluation: rf-web (narrow, treatment, N=3, deepseek/deepseek-v4.1-flash)Model: OpenRouter spend
Gaterf-skill-eval reportRuns: 6 · pass: 6 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.12 Pass rate = runs whose gate result is Tasks
Skills
Skipped / errored checksNone. Incomplete runsNone. Process checks (treatment arm, not in deltas)
Adversarial tier (reported only, never gates)None. |
Skill Evaluation: rf-api (narrow, treatment, N=3, deepseek/deepseek-v4.1-flash)Model: OpenRouter spend
Gaterf-skill-eval reportRuns: 6 · pass: 6 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.17 · turn-limit stops: 3 Pass rate = runs whose gate result is Tasks
Skills
Skipped / errored checksNone. Incomplete runsNone. Process checks (treatment arm, not in deltas)
Adversarial tier (reported only, never gates)None. |
…ping The first OpenRouter baseline run (42885585678) failed its Doctor step, skipped the tiers, and still ran the trigger evals, because that step was `if: always()`. Trigger evals now need a successful Doctor. The OpenRouter auth ping waits 120 s, like the smoke step, instead of 30 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
Skill Evaluation: rf-core (narrow, treatment, N=3, deepseek/deepseek-v4.1-flash)Model: OpenRouter spend
Gaterf-skill-eval reportRuns: 36 · pass: 35 · fail: 1 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.82 · turn-limit stops: 11 Pass rate = runs whose gate result is Tasks
Skills
Skipped / errored checksNone. Incomplete runsNone. Process checks (treatment arm, not in deltas)
Adversarial tier (reported only, never gates)None. |
…imit DeepSeek sometimes answers the one-turn "ok" probe with a tool call, so claude stops at --max-turns 1 and exits 1 although the provider answered. That failed the rf-web and rf-desktop Doctor steps in run 42887127109. The ping now reads the final result event: error_max_turns counts as verified auth, and a real failure reports the result's reason instead of stderr, which only held the unrecognized_model notice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
rf-desktop's PlatynUI tasks stopped at the 1.5 turn limit in 3-5 of 6 sessions and their pass rate swung between 0.33 and 0.83 across the two N=6 runs (42887127109, 42993062842). OPENROUTER_TURNS_FACTORS gives a plugin its own factor (rf-desktop 2.0); each matrix leg carries it, so the PR and full jobs and the stored baselines agree. Claude runs keep 1.0. The new `plugins` input re-records one plugin without rerunning the rest. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
Candidate baselines for rf-api, rf-core, rf-mobile and rf-web from the N=6 run on deepseek/deepseek-v4.1-flash (all tiers, both arms, turn-limit factor 1.5, trigger sets on both splits), copied unchanged from the candidate-baseline artifacts. rf-desktop follows from a run at factor 2.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
At 2.0 (run 43167186288) rf-desktop sessions used the extra turns and still stopped at the limit, so the gate pass rate didn't improve. The per-plugin override stays available; the map is now empty. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
From run 42993062842 (N=6), unchanged. The skill arm scores below the baseline arm here; the analysis of why is tracked separately in fix-desktop-eval-turn-overhead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
…-flash chore(eval): promote OpenRouter DeepSeek baselines from run 42993062842
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The live skill evals only run on Claude, billed to a maintainer's subscription. A spike (run 42822700497) ran the rf-core narrow tier on DeepSeek V4.1 Flash through OpenRouter: all 72 sessions completed, the skills loaded in every treatment run, the treatment arm passed 11 of 12 tasks, and the arm delta stayed visible on 8 tasks. OpenRouter charged $0.84. This PR makes that the default CI tier when the
OPENROUTER_API_KEYsecret is set. OpenSpec change:openspec/changes/openrouter-ci-eval-tier/.What changes
--model deepseek/deepseek-v4.1-flash) selects the OpenRouter provider. The runner points Claude Code at OpenRouter, maps every model alias to DeepSeek, and removes Claude credentials from the session. A session that used any other model is recorded as incomplete (provider_model).--max-cost-usdand the reports use the estimate, and the reports label it as one.--max-turns-factor(CI uses 1.5) scales each task'smax_turns; 25 of the 72 spike sessions hit the limit at 1.0. The factor is stored with runs and baselines; reports count turn-limit stops.plugins/<plugin>/evals/baselines/openrouter/deepseek-v4.1-flash/; Claude baselines stay where they are. A model or factor mismatch isrebaseline-needed.providerdispatch input). Each job gets only that provider's credentials and runs with separate OpenRouter caps. OpenRouter jobs run a one-session smoke test and report the key's real spend before and after (scripts/openrouter-ci.py).docs/ci/usage.md(new "OpenRouter tier" section),docs/ci/local-testing.md, an ADR-005 amendment, a Claude check inRELEASING.md, and.env.example.Before this gates anything
rebaseline-needed(exit 3). The next step is a manual dispatch on this branch (provider=openrouter, all tiers, both arms, N=6) and a reviewed PR with its candidate baselines (tasks 6.1–6.2).Testing
pytest testspasses: 1430 passed, 1 skipped (gitleaks not installed locally). The new tests are intests/eval/test_providers.pyandtests/eval/test_ci_preflight.py; the preflight tests run the provider-selection script for each credential combination.ruff,mypy, the docs-voice check andopenspec validate --strictpass.🤖 Generated with Claude Code
https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP