feat: add rubric-based LLM scenario judging

[Copilot]

Adds repeatable LLM evaluation using weighted, criterion-level rubrics. Each criterion defines concrete 1–5 score descriptions and can enforce a minimum passing score.

Changelog

New

• Scenario-level rubric configuration with judge model, reasoning effort, weighted criteria, and thresholds.
• Structured judgeResult output with weighted score, explanations, and threshold status.
• Read-only workspace evaluation restricted to view, grep, and glob.

rubric: {
  model: 'gpt-5.5',
  criteria: [{
    name: 'Correctness',
    weight: 3,
    minimumScore: 4,
    scores: {
      1: 'Incorrect',
      2: 'Major inaccuracies',
      3: 'Mostly correct with significant gaps',
      4: 'Correct with minor omissions',
      5: 'Complete, correct, and handles edge cases',
    },
  }],
}

Changed

• Scenario loading and output parsing now validate rubric definitions and judge results.

Removed

• None.

[joshblack]

@copilot add support for showing good and bad examples for each criteria and make sure this is provided into the prompt for the judge agent. Also make sure no part of the judge prompt is in the environment when running the scenario

[joshblack]

@copilot include support for this in the UI

[joshblack]

@copilot allow for explicit examples to be provided for each criteria (e.g. code samples) of examples of the criteria or good/bad to help it assess

[Copilot]

Pull request overview

Adds rubric-based, repeatable LLM judging for scenarios in @primer/agent-eval, including schema validation, structured judgeResult output, and website UI surfacing rubric configs and per-run rubric results.

Changes:

• Introduces scenario rubric configuration (model + optional reasoning effort + weighted criteria with optional minimum thresholds) and validates it during scenario loading/output parsing.
• Adds a new judge runner that evaluates the workspace + final agent response, producing a structured, weighted judgeResult.
• Exposes rubric configuration/results in the website scenario and experiment run pages, and documents the new scoring dimension.

Show a summary per file

File	Description
website/src/scenarios.ts	Extends website scenario model to include optional rubric config from resolved scenarios.
website/src/app/scenarios/[id]/components/Page.tsx	Renders rubric configuration details on the scenario detail page.
website/src/app/experiments/[id]/runs/[date]/page.tsx	Includes judgeResult in run details derived from AgentEvalOutput.
website/src/app/experiments/[id]/runs/[date]/components/Page.tsx	Displays rubric evaluation results (score + per-criterion judgments) in run UI.
README.md	Documents rubric score as an additional scoring dimension.
packages/agent-eval/src/treatment.ts	Extends TreatmentResult to optionally carry a typed JudgeResult.
packages/agent-eval/src/scenarios.ts	Adds runtime validation for rubric in scenario configs (model/effort/criteria validation).
packages/agent-eval/src/scenarios.test.ts	Adds tests for accepting valid rubrics and rejecting incomplete score descriptions.
packages/agent-eval/src/scenario-config.ts	Re-exports rubric-related types for scenario authoring.
packages/agent-eval/src/run.ts	Runs rubric judging after the agent run (with workspace lock-down) and includes judgeResult in results; factors copy excludes into a constant.
packages/agent-eval/src/run.test.ts	Adds a small check ensuring scenario config is excluded from copied workspace via SCENARIO_COPY_EXCLUDES.
packages/agent-eval/src/output.ts	Extends output schema to include optional rubric config in resolved scenarios and optional judgeResult in results.
packages/agent-eval/src/output.test.ts	Adds fixture coverage for parsing/serializing judgeResult in output.
packages/agent-eval/src/judge.ts	Implements judge prompt/args, response parsing/validation, weighted scoring, and threshold enforcement.
packages/agent-eval/src/judge.test.ts	Adds unit tests for prompt composition, CLI args, parsing, weighting, thresholds, and fenced JSON handling.
packages/agent-eval/src/index.ts	Exports new judge result types from the package entrypoint.
packages/agent-eval/src/experiment-config.ts	Adds Rubric, RubricCriterion, and RubricScore types and wires rubric into ScenarioConfig.
packages/agent-eval/README.md	Documents how to configure rubrics and what judge results contain.
.changeset/tasty-steaks-give.md	Declares a minor release for adding rubric-based judging.

Review details

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 19/19 changed files
• Comments generated: 3
• Review effort level: Lite