Split the Copilot instruction files by activity - #131438
Open
tannergooding wants to merge 11 commits into
Open
Conversation
Removes the repeated baseline build steps and the duplicated disclosure section, and fills in the missing Windows command equivalents. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Review criteria were loading for every C# edit. Conventions stay path-scoped for authoring, PR process moves to the always-on file, and review criteria move into the code-review skill. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
As an instruction file it matched every path, costing 2654 tokens on every request while only applying when building. As a skill it loads on demand, and splitting the workflows by component keeps the seven a session does not need out of context. Also bounds the cost of watching a long build. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A proposal prototype is evidence on a fork branch, not a submission, so it keeps its surface public and can be amended freely until the proposal links a commit URL. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The invariant is not claiming more than you verified. Requiring a build and tests for a comment or doc fix fights a flow a reviewer would have accepted anyway. Also scopes the CI note to pushes that actually trigger a run. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
An agent reaches for a shell, gh, and curl by habit, so the cost shows up here regardless of what the repo is built with. Re-reading a running command re-sends its whole output each time, which dwarfs anything the instruction files cost. The bash sentinel idiom is also silently wrong on Windows: $? is a [bool] there, so it records True rather than an exit code and still looks like it worked. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Nearly half of a model call is spent re-sending the conversation as cached input before it does any work, so batching independent tool calls into one response is worth more than shaving tokens off any individual step. Nothing said to do that. The review skill asked for the entire source file of everything in the diff. That is unbounded in a repo where `morph.cpp` is ~137k tokens on its own, so scope it to the enclosing functions and callers once a file gets large. Multi-model review also re-reads the diff per sub-agent, which is worth stating where it is prescribed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Azure Pipelines: Successfully started running 1 pipeline(s). 15 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR reorganizes dotnet/runtime’s Copilot guidance by splitting previously “all-in-one” instruction content into activity-specific skills and path-scoped instruction files, reducing always-inlined prompt size and preventing rules from firing in the wrong contexts.
Changes:
- Adds a new
build-and-testskill (plus per-component subdocs) and removes the build/test manual from.github/copilot-instructions.md. - Introduces a standalone holistic review criteria document (
pr-assessment.md) and updates thecode-reviewskill to load it. - Renames/reframes the “review-*” instruction set into domain-focused instruction files (
conventions,csharp,native,tests, etc.) and updates cross-skill references accordingly.
Show a summary per file
| File | Description |
|---|---|
| .github/skills/vectorization/SKILL.md | Updates SIMD guidance to reference the new build-and-test workflow source. |
| .github/skills/update-os-coverage/SKILL.md | Updates baseline-build guidance to reference build-and-test skill instead of inline instructions. |
| .github/skills/system-net-review/SKILL.md | Updates general guidance pointers to match the new split between conventions/build workflow/review process. |
| .github/skills/code-review/SKILL.md | Refactors review process instructions to load pr-assessment.md and clarifies large-file reading guidance. |
| .github/skills/code-review/pr-assessment.md | New file containing holistic PR assessment criteria for reviews. |
| .github/skills/build-and-test/SKILL.md | New build/test skill: baseline rules, sentinels, troubleshooting, and component workflow routing. |
| .github/skills/build-and-test/components/runtime.md | New per-component build/test guidance for CoreCLR and Mono. |
| .github/skills/build-and-test/components/runtime-tests.md | New per-component guidance for src/tests build/run workflows. |
| .github/skills/build-and-test/components/libraries.md | New per-component guidance for library build/test (including WASM/WASI). |
| .github/skills/build-and-test/components/host-and-tools.md | New per-component guidance for host/tools/tasks build/test workflows. |
| .github/skills/api-proposal/SKILL.md | Updates prototype validation prereqs to point to build-and-test; clarifies force-push guidance timing. |
| .github/instructions/tests.instructions.md | Renames/reframes test conventions guidance to be authoring+review applicable. |
| .github/instructions/review-all-src.instructions.md | Removes the old “review-all-src” instruction file (superseded by new split). |
| .github/instructions/native.instructions.md | Renames/reframes native conventions guidance to align with the new instruction taxonomy. |
| .github/instructions/csharp.instructions.md | Renames/reframes managed conventions guidance; updates/clarifies some rules. |
| .github/instructions/core-runtime.instructions.md | Renames/reframes core-runtime conventions and trims PR-process material (now elsewhere). |
| .github/instructions/conventions.instructions.md | New general src/** conventions file replacing the previous review-only general guidance. |
| .github/copilot-instructions.md | Removes the inlined build/test manual; adds a smaller Tool Use section and points builds/tests to the new skill. |
Copilot's findings
Comments suppressed due to low confidence (1)
.github/skills/build-and-test/SKILL.md:66
- The log-redirection example hardcodes the Libraries baseline (
./build.sh clr+libs -rc release), but this section just told readers to pick the baseline command based on the component. As written, it’s easy to copy/paste the wrong baseline for CoreCLR/Host/Tests. Use a placeholder so the snippet stays correct for all components.
./build.sh clr+libs -rc release > artifacts/build.log 2>&1; echo "exit=$?" > artifacts/build.status
- Files reviewed: 18/18 changed files
- Comments generated: 2
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Copilot's findings
Comments suppressed due to low confidence (4)
.github/instructions/csharp.instructions.md:9
- The intro references other instruction files using short names like
conventions/tests/native, which aren’t actual paths and can be ambiguous. Use explicit instruction file paths here to match how other docs (e.g., the code-review skill) reference them.
.github/instructions/native.instructions.md:9 - The intro references other instruction files using short names (e.g.,
conventions,tests,core-runtime) rather than explicit paths, which makes it harder to follow and is inconsistent with other guidance. Prefer explicit.github/instructions/*.instructions.mdpaths.
.github/instructions/tests.instructions.md:11 - This intro references other instruction files using short names like
conventions/csharp/native, which aren’t explicit file paths. Using explicit.github/instructions/*.instructions.mdpaths here makes the cross-reference unambiguous and consistent with the code-review skill.
.github/instructions/core-runtime.instructions.md:8 - The intro references other instruction files using short names (e.g.,
conventions,tests,jit) rather than explicit paths. Using explicit.github/instructions/*.instructions.mdpaths makes the cross-reference clear and consistent with other guidance.
- Files reviewed: 18/18 changed files
- Comments generated: 2
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…onventions Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Member
Author
|
Nits should be handled now and added one more instruction that will help with the back and forth on "fix a nit, find a new nit" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The instruction files had grown by accretion, and rules were firing during the wrong
activity -- review criteria applied while authoring, PR process applied while answering a
question, and a full build manual inlined on every single model call. This splits them by
what you're actually doing:
.github/instructions/*.instructions.md, path-scoped viaapplyTo.github/copilot-instructions.mdcode-reviewskill, with the holistic criteria split intopr-assessment.mdbuild-and-testskillThe
review-*prefix on the instruction files was misleading, since the built-in Copilotreviewer and the human authoring path both consume them. They're now named for the domain
they cover (
csharp,native,core-runtime,tests,conventions).The largest single win:
copilot-instructions.mdcarried the entire build-and-test manual,2,654 tokens sent on every model call for the benefit of the minority that actually build.
That's now the
build-and-testskill, loaded on demand, with the per-component workflowssplit into four files under
components/so a session pulls in only the one it needs.Same for the review criteria, which moved to
pr-assessment.md.Net effect on the always-inlined file: 3,934 -> 1,725 tokens (-56%).
.csMeasured with
tiktoken/o200k_base.This isn't paid once and then cached forever, either -- the prompt cache has a ~5 minute TTL,
so the prefix is re-written at full rate at the start of every prompt that follows any real
think-time. Instrumenting a session bears that out: the first call of a turn averages 78.9%
cached against 96.3% for follow-up calls, and the one turn that started after a >5 minute gap
came back 0% cached. On the opening call of that session the old file was 14.1% of all
fresh input, so cutting it saves 7% of that call outright, and the saving then lands again
on every cold prompt start rather than being amortized away.
In AI credits, calibrated against that session's own billing rows (1 credit ~= 2,039 weighted
tokens, where weighted = fresh + 0.1cached + 6output):
.csColumns are 3, 30, 120 and 480 model calls respectively, with 100%, 80%, 70% and 25% of
prompt starts cold -- an agent run going flat out keeps its cache warm, where-as a human
iterating on an issue mostly doesn't.
The new
## Tool Usesection is small and I think it's worth more than everything abovecombined.
The same instrumentation gives the real shape of the cost. 94.6% of input tokens are cache
reads, and average fresh input is only ~7k tokens per call, so instruction context is
1-9% of a call -- conversation history and tool output is all the rest. The instruction
prefix is re-injected on every call and never compacted though, which makes a token there
cost ~16x a token of tool output. Two consequences:
input before it does any work. That makes the number of round trips a first-order cost in
its own right, so batching independent tool calls into one response is worth more than
shaving tokens off any individual step. Nothing said to do that.
think. Conversation history is compacted, so a tool result does not ride along forever --
measured, a token entering context is re-read ~17.6 times before being compacted out. But
the session still took 18 full cold re-writes of a ~200k context, each re-sending
everything at full rate. That works out to ~37 credits per 1k tokens of persistent context
over a session, putting a 50k-token CI log at ~74 credits and a whole-file read of
morph.cppat ~202.Hence the guidance: batch independent calls, redirect long-running commands to a log and poll
a bounded view, prefer
gh run view --log-failedover--log,git diff --statbefore thefull diff,
--json/--jqto project only what's needed. It also documents the shells'differences, notably that the bash sentinel idiom is silently wrong in PowerShell --
$?is a[bool]there, so a status file written with it recordsTrueand always looks like success.MSBuild and
dotnetalready suppress progress rendering when piped -- I checked, 0 ANSIescapes across
-v:q,--tl:off, and the defaults -- so no flags are needed for those.Relatedly, the review skill asked for the entire source file of everything in the diff.
That's unbounded in this repo --
morph.cppis ~137k tokens on its own -- so it's now scopedto the enclosing functions, their callers, and the types involved once a file gets large. The
intent is preserved, since diff-only review really is the main source of false positives.
Multi-model review re-reads the diff per sub-agent, which is now stated where it's prescribed.
Also in here, smaller:
branch keeps its surface
public-- it's evidence for the proposal, not a submission --and the force-push rule is scoped to pre-publication.
but if a contributor would have submitted the change without building it, say so and move
on rather than burning 40 minutes on a comment fix.
with no PR triggers nothing.
Out of scope: the area-specific instruction files (
system-net-*,extensions-*,compression,cdac,jit) are untouched -- they're already narrowly path-scoped andtheir cost only lands when you're in that area.
Every file here is
.md, which the'**.md'exclusion ineng/pipelines/runtime.ymlfilters out, so this shouldn't queue any CI.
Note
This PR description was drafted with GitHub Copilot.