feat: support multi-turn scenarios

[Copilot]

Scenarios can now evaluate context retention across multiple conversation turns. Each follow-up prompt runs in the existing Copilot session and is graded before continuing.

Changes

• Scenario configuration

  • Added ordered follow-up turns with dedicated Node and optional browser tests.
  • Preserved existing single-turn scenario behavior.

• Execution

  • Resumes the same Copilot session for each follow-up.
  • Stops when a turn fails verification.
  • Isolates grading files from the agent workspace between turns.

• Results

  • Aggregates logs, usage, tool calls, and test results across executed turns.
  • Includes multi-turn metadata in serialized output.

export default defineScenario({
  prompt: 'Change the button color to blue',
  turns: [
    {
      prompt: 'Actually, make it red instead',
      test: 'red.test.ts',
      browserTest: 'red.browser.test.ts',
    },
  ],
})

[joshblack]

@copilot make sure to add support for viewing turns in the UI

[Copilot]

@copilot make sure to add support for viewing turns in the UI

Added per-turn run details in 6867924. Multi-turn results now show each executed prompt with its corresponding agent transcript, while single-turn results retain the existing transcript view.

[joshblack]

@copilot create a changeset for this change

[Copilot]

@copilot create a changeset for this change

Added a minor changeset for @primer/agent-eval in 6c5f98c.

[Copilot]

Pull request overview

Adds first-class support for multi-turn evaluation scenarios so experiments can measure context retention across sequential prompts within the same Copilot session, and updates reporting/UI/docs to surface per-turn details.

Changes:

• Extend scenario config + loader to support ordered follow-up turns with per-turn test (and optional browser test) files.
• Update the runner to resume the same Copilot session across turns, stop on first failing turn, and aggregate usage/logs/test results across executed turns.
• Update output schemas and the website run view to display multi-turn transcripts.

Show a summary per file

File	Description
website/src/app/experiments/[id]/runs/[date]/page.tsx	Builds per-turn transcript data for the run details payload used by the UI.
website/src/app/experiments/[id]/runs/[date]/components/Page.tsx	Renders multi-turn conversation UI when present, otherwise falls back to single transcript view.
README.md	Documents how to define follow-up turns in scenarios.
packages/agent-eval/src/scenarios.ts	Loads and validates turns, resolves turn test paths, and returns resolved turn metadata.
packages/agent-eval/src/scenarios.test.ts	Adds coverage for loading valid turns and rejecting invalid turn configs.
packages/agent-eval/src/scenario-config.ts	Re-exports ScenarioTurnConfig for scenario authoring convenience.
packages/agent-eval/src/run.ts	Runs Copilot in a loop across turns (resuming session), runs per-turn tests, and aggregates metrics/results.
packages/agent-eval/src/run.test.ts	Tests --resume argument wiring for follow-up turns.
packages/agent-eval/src/output.ts	Extends zod schemas to serialize/parse multi-turn scenario config and resolved turn metadata.
packages/agent-eval/src/experiment-config.ts	Extends ScenarioConfig with optional turns and defines ScenarioTurnConfig.
packages/agent-eval/README.md	Adds package-level docs for multi-turn scenario authoring and behavior on failure.
.changeset/clear-beans-read.md	Publishes the feature as a minor version bump for @primer/agent-eval.

Review details

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 12/12 changed files
• Comments generated: 3
• Review effort level: Lite

[joshblack]

@copilot apply reiew feedback

[Copilot]

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

packages/agent-eval/src/scenarios.ts:98

• resolveScenarioFile rejects paths in subdirectories (e.g. tests/red.test.ts) because it compares path.dirname(resolvedPath) to the scenario root. This prevents organizing turn test files under nested folders even though they’re still within the scenario directory. Consider validating via path.relative to block .. traversal while allowing subpaths.

  const scenarioDirectory = path.resolve(directory)
  const resolvedPath = path.resolve(scenarioDirectory, filepath)
  if (path.dirname(resolvedPath) !== scenarioDirectory) {
    throw new Error(`Scenario "${name}" turn file must be a file in the scenario directory: ${filepath}`)
  }

• Files reviewed: 12/12 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

@copilot apply reiew feedback

Applied the review feedback in d7c1412: empty turns are accepted, turn file path errors are generic, and per-turn transcripts omit duplicated user messages.