Skip to content

Split the Copilot instruction files by activity - #131438

Open
tannergooding wants to merge 11 commits into
dotnet:mainfrom
tannergooding:tannergooding-cleanup-copilot-instructions
Open

Split the Copilot instruction files by activity#131438
tannergooding wants to merge 11 commits into
dotnet:mainfrom
tannergooding:tannergooding-cleanup-copilot-instructions

Conversation

@tannergooding

@tannergooding tannergooding commented Jul 27, 2026

Copy link
Copy Markdown
Member

The instruction files had grown by accretion, and rules were firing during the wrong
activity -- review criteria applied while authoring, PR process applied while answering a
question, and a full build manual inlined on every single model call. This splits them by
what you're actually doing:

  • author code -- .github/instructions/*.instructions.md, path-scoped via applyTo
  • author a PR -- .github/copilot-instructions.md
  • review -- the code-review skill, with the holistic criteria split into pr-assessment.md
  • build and test -- a new build-and-test skill

The review-* prefix on the instruction files was misleading, since the built-in Copilot
reviewer and the human authoring path both consume them. They're now named for the domain
they cover (csharp, native, core-runtime, tests, conventions).


The largest single win: copilot-instructions.md carried the entire build-and-test manual,
2,654 tokens sent on every model call for the benefit of the minority that actually build.
That's now the build-and-test skill, loaded on demand, with the per-component workflows
split into four files under components/ so a session pulls in only the one it needs.
Same for the review criteria, which moved to pr-assessment.md.

Net effect on the always-inlined file: 3,934 -> 1,725 tokens (-56%).

context per model call before after
Ask a question, no files read 3,934 1,725 -56%
Ask a question, reads a .cs 9,857 6,255 -37%
Libraries-only C# change 9,857 8,572 -13%
CoreCLR JIT change 8,228 6,976 -15%
Both libraries and coreclr 11,609 10,346 -11%
Review a C# PR 14,682 12,385 -16%

Measured with tiktoken/o200k_base.

This isn't paid once and then cached forever, either -- the prompt cache has a ~5 minute TTL,
so the prefix is re-written at full rate at the start of every prompt that follows any real
think-time. Instrumenting a session bears that out: the first call of a turn averages 78.9%
cached against 96.3% for follow-up calls, and the one turn that started after a >5 minute gap
came back 0% cached. On the opening call of that session the old file was 14.1% of all
fresh input, so cutting it saves 7% of that call outright, and the saving then lands again
on every cold prompt start rather than being amortized away.

In AI credits, calibrated against that session's own billing rows (1 credit ~= 2,039 weighted
tokens, where weighted = fresh + 0.1cached + 6output):

credits saved one-shot 5 prompts iterating, 10 prompts long agent run
Ask a question, no files read +1.6 +8.2 +21.7 +64.5
Ask a question, reads a .cs +2.6 +13.4 +35.4 +105.1
Libraries-only C# change +0.9 +4.8 +12.6 +37.5
CoreCLR JIT change +0.9 +4.7 +12.3 +36.5
Both libraries and coreclr +0.9 +4.7 +12.4 +36.9
Review a C# PR +1.6 +8.6 +22.6 +67.0

Columns are 3, 30, 120 and 480 model calls respectively, with 100%, 80%, 70% and 25% of
prompt starts cold -- an agent run going flat out keeps its cache warm, where-as a human
iterating on an issue mostly doesn't.


The new ## Tool Use section is small and I think it's worth more than everything above
combined.

The same instrumentation gives the real shape of the cost. 94.6% of input tokens are cache
reads
, and average fresh input is only ~7k tokens per call, so instruction context is
1-9% of a call -- conversation history and tool output is all the rest. The instruction
prefix is re-injected on every call and never compacted though, which makes a token there
cost ~16x a token of tool output. Two consequences:

  • Nearly half of every call (47%, measured) is spent re-sending the conversation as cached
    input before it does any work.
    That makes the number of round trips a first-order cost in
    its own right, so batching independent tool calls into one response is worth more than
    shaving tokens off any individual step. Nothing said to do that.
  • Context size is the other driver, because it gets re-sent in full more often than you'd
    think.
    Conversation history is compacted, so a tool result does not ride along forever --
    measured, a token entering context is re-read ~17.6 times before being compacted out. But
    the session still took 18 full cold re-writes of a ~200k context, each re-sending
    everything at full rate. That works out to ~37 credits per 1k tokens of persistent context
    over a session, putting a 50k-token CI log at ~74 credits and a whole-file read of
    morph.cpp at ~202.

Hence the guidance: batch independent calls, redirect long-running commands to a log and poll
a bounded view, prefer gh run view --log-failed over --log, git diff --stat before the
full diff, --json/--jq to project only what's needed. It also documents the shells'
differences, notably that the bash sentinel idiom is silently wrong in PowerShell -- $? is a
[bool] there, so a status file written with it records True and always looks like success.

MSBuild and dotnet already suppress progress rendering when piped -- I checked, 0 ANSI
escapes across -v:q, --tl:off, and the defaults -- so no flags are needed for those.

Relatedly, the review skill asked for the entire source file of everything in the diff.
That's unbounded in this repo -- morph.cpp is ~137k tokens on its own -- so it's now scoped
to the enclosing functions, their callers, and the types involved once a file gets large. The
intent is preserved, since diff-only review really is the main source of false positives.
Multi-model review re-reads the diff per sub-agent, which is now stated where it's prescribed.


Also in here, smaller:

  • The API-approval rule now only applies to what gets submitted. A proposal's prototype
    branch keeps its surface public -- it's evidence for the proposal, not a submission --
    and the force-push rule is scoped to pre-publication.
  • Verification guidance is scaled to risk. Never claim a build or test passed unless it did,
    but if a contributor would have submitted the change without building it, say so and move
    on rather than burning 40 minutes on a comment fix.
  • The CI-cost note is scoped to open PRs and states the trigger paths, since a fork branch
    with no PR triggers nothing.

Out of scope: the area-specific instruction files (system-net-*, extensions-*,
compression, cdac, jit) are untouched -- they're already narrowly path-scoped and
their cost only lands when you're in that area.

Every file here is .md, which the '**.md' exclusion in eng/pipelines/runtime.yml
filters out, so this shouldn't queue any CI.

Note

This PR description was drafted with GitHub Copilot.

tannergooding and others added 7 commits July 27, 2026 13:40
Removes the repeated baseline build steps and the duplicated disclosure section, and fills in the missing Windows command equivalents.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Review criteria were loading for every C# edit. Conventions stay path-scoped for authoring, PR process moves to the always-on file, and review criteria move into the code-review skill.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
As an instruction file it matched every path, costing 2654 tokens on every request while only applying when building. As a skill it loads on demand, and splitting the workflows by component keeps the seven a session does not need out of context. Also bounds the cost of watching a long build.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A proposal prototype is evidence on a fork branch, not a submission, so it keeps its surface public and can be amended freely until the proposal links a commit URL.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The invariant is not claiming more than you verified. Requiring a build and tests for a comment or doc fix fights a flow a reviewer would have accepted anyway. Also scopes the CI note to pushes that actually trigger a run.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
An agent reaches for a shell, gh, and curl by habit, so the cost shows up here
regardless of what the repo is built with. Re-reading a running command re-sends
its whole output each time, which dwarfs anything the instruction files cost.

The bash sentinel idiom is also silently wrong on Windows: $? is a [bool] there,
so it records True rather than an exit code and still looks like it worked.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Nearly half of a model call is spent re-sending the conversation as cached input
before it does any work, so batching independent tool calls into one response is
worth more than shaving tokens off any individual step. Nothing said to do that.

The review skill asked for the entire source file of everything in the diff. That
is unbounded in a repo where `morph.cpp` is ~137k tokens on its own, so scope it to
the enclosing functions and callers once a file gets large. Multi-model review also
re-reads the diff per sub-agent, which is worth stating where it is prescribed.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 27, 2026 20:42
@github-actions github-actions Bot added the area-skills Agent Skills label Jul 27, 2026
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
15 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reorganizes dotnet/runtime’s Copilot guidance by splitting previously “all-in-one” instruction content into activity-specific skills and path-scoped instruction files, reducing always-inlined prompt size and preventing rules from firing in the wrong contexts.

Changes:

  • Adds a new build-and-test skill (plus per-component subdocs) and removes the build/test manual from .github/copilot-instructions.md.
  • Introduces a standalone holistic review criteria document (pr-assessment.md) and updates the code-review skill to load it.
  • Renames/reframes the “review-*” instruction set into domain-focused instruction files (conventions, csharp, native, tests, etc.) and updates cross-skill references accordingly.
Show a summary per file
File Description
.github/skills/vectorization/SKILL.md Updates SIMD guidance to reference the new build-and-test workflow source.
.github/skills/update-os-coverage/SKILL.md Updates baseline-build guidance to reference build-and-test skill instead of inline instructions.
.github/skills/system-net-review/SKILL.md Updates general guidance pointers to match the new split between conventions/build workflow/review process.
.github/skills/code-review/SKILL.md Refactors review process instructions to load pr-assessment.md and clarifies large-file reading guidance.
.github/skills/code-review/pr-assessment.md New file containing holistic PR assessment criteria for reviews.
.github/skills/build-and-test/SKILL.md New build/test skill: baseline rules, sentinels, troubleshooting, and component workflow routing.
.github/skills/build-and-test/components/runtime.md New per-component build/test guidance for CoreCLR and Mono.
.github/skills/build-and-test/components/runtime-tests.md New per-component guidance for src/tests build/run workflows.
.github/skills/build-and-test/components/libraries.md New per-component guidance for library build/test (including WASM/WASI).
.github/skills/build-and-test/components/host-and-tools.md New per-component guidance for host/tools/tasks build/test workflows.
.github/skills/api-proposal/SKILL.md Updates prototype validation prereqs to point to build-and-test; clarifies force-push guidance timing.
.github/instructions/tests.instructions.md Renames/reframes test conventions guidance to be authoring+review applicable.
.github/instructions/review-all-src.instructions.md Removes the old “review-all-src” instruction file (superseded by new split).
.github/instructions/native.instructions.md Renames/reframes native conventions guidance to align with the new instruction taxonomy.
.github/instructions/csharp.instructions.md Renames/reframes managed conventions guidance; updates/clarifies some rules.
.github/instructions/core-runtime.instructions.md Renames/reframes core-runtime conventions and trims PR-process material (now elsewhere).
.github/instructions/conventions.instructions.md New general src/** conventions file replacing the previous review-only general guidance.
.github/copilot-instructions.md Removes the inlined build/test manual; adds a smaller Tool Use section and points builds/tests to the new skill.

Copilot's findings

Comments suppressed due to low confidence (1)

.github/skills/build-and-test/SKILL.md:66

  • The log-redirection example hardcodes the Libraries baseline (./build.sh clr+libs -rc release), but this section just told readers to pick the baseline command based on the component. As written, it’s easy to copy/paste the wrong baseline for CoreCLR/Host/Tests. Use a placeholder so the snippet stays correct for all components.
./build.sh clr+libs -rc release > artifacts/build.log 2>&1; echo "exit=$?" > artifacts/build.status
  • Files reviewed: 18/18 changed files
  • Comments generated: 2

Comment thread .github/skills/build-and-test/SKILL.md Outdated
Comment thread .github/skills/system-net-review/SKILL.md Outdated
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 27, 2026 21:00

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

Comments suppressed due to low confidence (4)

.github/instructions/csharp.instructions.md:9

  • The intro references other instruction files using short names like conventions / tests / native, which aren’t actual paths and can be ambiguous. Use explicit instruction file paths here to match how other docs (e.g., the code-review skill) reference them.
    .github/instructions/native.instructions.md:9
  • The intro references other instruction files using short names (e.g., conventions, tests, core-runtime) rather than explicit paths, which makes it harder to follow and is inconsistent with other guidance. Prefer explicit .github/instructions/*.instructions.md paths.
    .github/instructions/tests.instructions.md:11
  • This intro references other instruction files using short names like conventions / csharp / native, which aren’t explicit file paths. Using explicit .github/instructions/*.instructions.md paths here makes the cross-reference unambiguous and consistent with the code-review skill.
    .github/instructions/core-runtime.instructions.md:8
  • The intro references other instruction files using short names (e.g., conventions, tests, jit) rather than explicit paths. Using explicit .github/instructions/*.instructions.md paths makes the cross-reference clear and consistent with other guidance.
  • Files reviewed: 18/18 changed files
  • Comments generated: 2

Comment thread .github/skills/system-net-review/SKILL.md Outdated
Comment thread .github/copilot-instructions.md
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 27, 2026 21:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

  • Files reviewed: 18/18 changed files
  • Comments generated: 3

Comment thread .github/skills/build-and-test/SKILL.md Outdated
Comment thread .github/skills/build-and-test/components/runtime-tests.md Outdated
Comment thread .github/instructions/conventions.instructions.md Outdated
tannergooding and others added 2 commits July 27, 2026 14:18
…onventions

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 27, 2026 21:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

  • Files reviewed: 18/18 changed files
  • Comments generated: 0 new

@tannergooding

Copy link
Copy Markdown
Member Author

Nits should be handled now and added one more instruction that will help with the back and forth on "fix a nit, find a new nit"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-skills Agent Skills

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants