Describe the feature or problem you'd like to solve
When planning is finished, make it clear whether there is anything to run and what needs to happen next.
In a Windows evaluation (CLI 1.0.88, medium effort), a user asked for a plan for a blinded model-comparison feature in an evaluation tool. Copilot found that the workspace contained only a README, researched the tool's documentation, and returned a design plan.
The user then asked "what decisions do you need from me?" Copilot collected four choices through ask_user: post-run review, two models in the same harness, organizer-configured models, and revealing the models after ratings were submitted.
After those answers, Copilot said "That settles the four decisions needed to define the first experience" and summarized the scope. It didn't offer to implement the design, ask for the source checkout, or give a concrete next action.
When the user asked "how do i try this", Copilot explained that "we made design decisions, not an implementation." It described a manual pilot where an organizer would prepare anonymized A/B results, collect ratings, and then reveal the models. No source edits were made.
Proposed solution
Once the design choices are settled, say that the plan is ready and the feature still needs to be built. Offer implementation as a next step and identify any missing prerequisites, such as access to the source checkout.
Keep that step optional. The user should be able to continue planning without being repeatedly asked to start implementation or revisit questions they've already answered.
Example prompts or workflows
After the user answers the design questions, a response could be:
The plan is ready; the feature isn't built yet. Would you like me to implement it? This workspace doesn't contain the project's source, so I'd need the target checkout before starting.
This is a suggested response, not a captured quote.
Additional context
The evaluator felt that Copilot had stopped at a plan instead of helping them reach an implementation. The original request asked for a plan, and no explicit instruction to implement followed. Copilot's responses completed normally, so this doesn't establish a refusal or a runtime failure.
The feedback comes from one participant/run. Related: #5020 covers recognizing when users want to review a plan before implementation.
Describe the feature or problem you'd like to solve
When planning is finished, make it clear whether there is anything to run and what needs to happen next.
In a Windows evaluation (CLI 1.0.88, medium effort), a user asked for a plan for a blinded model-comparison feature in an evaluation tool. Copilot found that the workspace contained only a README, researched the tool's documentation, and returned a design plan.
The user then asked "what decisions do you need from me?" Copilot collected four choices through
ask_user: post-run review, two models in the same harness, organizer-configured models, and revealing the models after ratings were submitted.After those answers, Copilot said "That settles the four decisions needed to define the first experience" and summarized the scope. It didn't offer to implement the design, ask for the source checkout, or give a concrete next action.
When the user asked "how do i try this", Copilot explained that "we made design decisions, not an implementation." It described a manual pilot where an organizer would prepare anonymized A/B results, collect ratings, and then reveal the models. No source edits were made.
Proposed solution
Once the design choices are settled, say that the plan is ready and the feature still needs to be built. Offer implementation as a next step and identify any missing prerequisites, such as access to the source checkout.
Keep that step optional. The user should be able to continue planning without being repeatedly asked to start implementation or revisit questions they've already answered.
Example prompts or workflows
After the user answers the design questions, a response could be:
This is a suggested response, not a captured quote.
Additional context
The evaluator felt that Copilot had stopped at a plan instead of helping them reach an implementation. The original request asked for a plan, and no explicit instruction to implement followed. Copilot's responses completed normally, so this doesn't establish a refusal or a runtime failure.
The feedback comes from one participant/run. Related: #5020 covers recognizing when users want to review a plan before implementation.