Skip to content

Add native Jev System One judging with selectable presets - #131

Merged
kargibora merged 18 commits into
mainfrom
feat/jev-prompts
Oct 1, 2026
Merged

kargibora merged 18 commits into
mainfrom
feat/jev-prompts

Conversation

@kargibora

Copy link
Copy Markdown
Member

Summary

  • Add Jev as a native OpenRouter System One judge through the existing judge.prompt_preset interface.
  • Include typesafe-choice, typesafe-multilingual-choice, typesafe-comparative-score, and typesafe-fluency-choice presets. Preserve native hard verdicts and soft probability distributions, including graded Score and both-order evaluation.
  • Integrate with main's inference cache and MT-Bench-101 prompt registry without changing official benchmark judging protocols.

Validation

  • uv run pytest -q: 469 passed.
  • uv run ruff check . and uv run ruff format --check ..
  • Built wheel and verified packaged Jev presets and MT-Bench-101 templates.
  • Live Jev System One both-order smoke test completed before the main merge.

@kargibora
kargibora merged commit b2fb420 into main Oct 1, 2026
1 check passed
kargibora added a commit that referenced this pull request Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant