Skip to content

feat: add configurable Copilot runner - #160

Draft
joshblack with Copilot wants to merge 4 commits into
mainfrom
copilot/add-copilot-sdk-support
Draft

feat: add configurable Copilot runner#160
joshblack with Copilot wants to merge 4 commits into
mainfrom
copilot/add-copilot-sdk-support

Conversation

Copilot AI commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Adds support for running experiments through the Copilot SDK in addition to the Copilot CLI. Experiments can now include runner choice as a dimension, making CLI vs SDK behavior comparable across the same models, scenarios, and treatments.

  • Runner configuration

    • Added a Copilot runner option for experiment configs.
    • Supports selecting cli, sdk, or multiple runners to expand experiment coverage.
  • Execution path

    • Preserves the existing Copilot CLI behavior as the default.
    • Adds a Copilot SDK execution path that emits compatible run output.
  • Experiment dimensions

    • Includes runner selection in generated treatments/results so comparisons remain attributable.
export const experiment = defineConfig({
  name: 'Example experiment',
  description: 'Compare Copilot CLI and SDK runs',
  models: ['gpt-5.5'],
  scenarios: ['001-agent-uses-button-from-primer'],
  runners: ['cli', 'sdk'],
  treatments: [],
})

Co-authored-by: joshblack <3901764+joshblack@users.noreply.github.com>
Copilot AI and others added 3 commits August 24, 2026 16:01
Co-authored-by: joshblack <3901764+joshblack@users.noreply.github.com>
Co-authored-by: joshblack <3901764+joshblack@users.noreply.github.com>
Co-authored-by: joshblack <3901764+joshblack@users.noreply.github.com>
Copilot AI requested a review from joshblack August 24, 2026 16:31
@joshblack
joshblack removed their request for review August 26, 2026 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants