Skip to content

feat: add capability-based benchmarks - #156

Draft
joshblack with Copilot wants to merge 2 commits into
mainfrom
copilot/add-benchmark-support
Draft

feat: add capability-based benchmarks#156
joshblack with Copilot wants to merge 2 commits into
mainfrom
copilot/add-benchmark-support

Conversation

Copilot AI commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Benchmarks group curated scenarios by capability, exposing actionable performance breakdowns across models instead of a single aggregate score.

Changelog

New

  • defineBenchmark API and benchmark discovery utilities
  • CLI support via --benchmark and --benchmarks
  • Capability-level and per-model summaries
  • Benchmark metadata in JSON output
export const benchmark = defineBenchmark({
  name: 'UI interactions',
  description: 'Evaluate common UI tasks',
  models: [{name: 'gpt-5.5', reasoningEfforts: ['high']}],
  capabilities: [
    {
      name: 'Edits forms',
      description: 'Creates and updates accessible forms',
      scenarios: ['creates-sign-up-form', 'adds-form-validation'],
    },
  ],
})

Changed

  • Result reporting now associates resolved scenario IDs with benchmark capabilities.
  • Package exports and documentation cover benchmark authoring and execution.

Removed

  • None.

Copilot AI and others added 2 commits August 21, 2026 20:40
Co-authored-by: joshblack <3901764+joshblack@users.noreply.github.com>
Co-authored-by: joshblack <3901764+joshblack@users.noreply.github.com>

@joshblack joshblack left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot add support for this in the UI. Let's use our baseline experiment as a "Design System" benchmark

Copilot AI requested a review from joshblack August 22, 2026 03:05
@joshblack
joshblack removed their request for review August 26, 2026 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants