Skip to content

Fail a routing run that lost a skill before it started - #22

Open
johnl-amd wants to merge 1 commit into
mainfrom
missing-skill-fails-the-run
Open

johnl-amd wants to merge 1 commit into
mainfrom
missing-skill-fails-the-run

Conversation

@johnl-amd

@johnl-amd johnl-amd commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • routing_gate now fails when skills_missing_from_session is non-empty, alongside the two infrastructure checks already there. Placed before the accuracy bar, so neither a clean sweep nor --min-accuracy 0 can pass a room that was not the one asked for.
  • Why: a staged skill the CLI never reports at session init is not in the room the run claims to measure. Its cases still run, every one reads as missed_trigger, and the report tells the author a description does not trigger when that description was never offered to the agent. The two outcomes are indistinguishable in the verdict.
  • The condition was already detected and already rendered as a warning above the numbers. It just never reached the gate, so the run went green apart from a line that is easy to read past.
  • Risk: low. One branch in one function, an optional third parameter so existing callers are unaffected, no change to what is measured or reported.

Why it is not hypothetical

Every enabled skill contributes one line to a listing the agent reads at session start, costing len(name) + 4 + min(len(description), 1536), plus a separator between entries. That listing has a budget:

context window in tokens x bytes per token x skillListingBudgetFraction
200,000 x 4 x 0.01 = 8,000 characters

Measured on one developer machine: 34 enabled skills needing 18,050 characters against that 8,000 budget, 2.3x over. At least 12 of the 34 are reduced to a bare name right now, in any session at the default context size.

On overflow nothing fails. Names are kept, descriptions are dropped, and the order is:

usageCount x max(0.5 ^ (daysSinceLastUse / 7), 0.1)

A skill staged for an eval has been invoked zero times, so its score is zero and it is first to lose its description by construction. Not merely likely to be near the front of the queue: last in line for restoration, every time, until it has been used, which it cannot be while it has no description to match on.

It also competes with whatever the runner already has. A real run against a catalog of 8 skills reported 17 further skills registered from the runner's own config, which this code already warns about separately.

Test plan

  • 3 new cases in TestRoutingGate: a missing skill fails the run and names it; it outranks the accuracy bar at both 1.0 and 0.0; None and [] leave the gate alone.
  • Full suite 229, green via python -m unittest discover -s tests -t ..
  • Mutation-tested: reverting the new branch fails exactly those three and nothing else, so they are pinned to the behaviour rather than to the shape of the code.

Note

The failure message is longer than its neighbours on purpose. This is the quietest of the three infrastructure conditions, because the run still produces a plausible number, so the message says what the number would otherwise have been read as.

Related: #23 reports what a repo's own skills cost against the same budget, so a catalog can see the pressure it adds before it ships.

A staged skill the CLI never reports at session init is not in the room
the run claims to measure. Its cases still run, every one reads as a
missed trigger, and the report tells the author a description does not
trigger when that description was never offered to the agent. The two
outcomes are indistinguishable in the verdict.

The condition was already detected and already rendered as a warning
above the numbers. It just did not reach the gate, so the run went green
apart from a line that is easy to read past.

This is the same class as the two checks beside it: infrastructure
rather than score. It is placed before the accuracy bar so that neither
a clean sweep nor `--min-accuracy 0` can license a room that was not the
one asked for.

Worth stating why it is not hypothetical. Descriptions share one listing
budget, and on overflow the text is dropped starting with the skills
invoked least. A skill staged for an eval has been invoked zero times,
so it is first in that queue, while the runner's own user-level skills
compete for the same space.

Signed-off-by: John Lybeck <john.lybeck@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant