Conversation
A staged skill the CLI never reports at session init is not in the room the run claims to measure. Its cases still run, every one reads as a missed trigger, and the report tells the author a description does not trigger when that description was never offered to the agent. The two outcomes are indistinguishable in the verdict. The condition was already detected and already rendered as a warning above the numbers. It just did not reach the gate, so the run went green apart from a line that is easy to read past. This is the same class as the two checks beside it: infrastructure rather than score. It is placed before the accuracy bar so that neither a clean sweep nor `--min-accuracy 0` can license a room that was not the one asked for. Worth stating why it is not hypothetical. Descriptions share one listing budget, and on overflow the text is dropped starting with the skills invoked least. A skill staged for an eval has been invoked zero times, so it is first in that queue, while the runner's own user-level skills compete for the same space. Signed-off-by: John Lybeck <john.lybeck@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
routing_gatenow fails whenskills_missing_from_sessionis non-empty, alongside the two infrastructure checks already there. Placed before the accuracy bar, so neither a clean sweep nor--min-accuracy 0can pass a room that was not the one asked for.missed_trigger, and the report tells the author a description does not trigger when that description was never offered to the agent. The two outcomes are indistinguishable in the verdict.Why it is not hypothetical
Every enabled skill contributes one line to a listing the agent reads at session start, costing
len(name) + 4 + min(len(description), 1536), plus a separator between entries. That listing has a budget:Measured on one developer machine: 34 enabled skills needing 18,050 characters against that 8,000 budget, 2.3x over. At least 12 of the 34 are reduced to a bare name right now, in any session at the default context size.
On overflow nothing fails. Names are kept, descriptions are dropped, and the order is:
A skill staged for an eval has been invoked zero times, so its score is zero and it is first to lose its description by construction. Not merely likely to be near the front of the queue: last in line for restoration, every time, until it has been used, which it cannot be while it has no description to match on.
It also competes with whatever the runner already has. A real run against a catalog of 8 skills reported 17 further skills registered from the runner's own config, which this code already warns about separately.
Test plan
TestRoutingGate: a missing skill fails the run and names it; it outranks the accuracy bar at both1.0and0.0;Noneand[]leave the gate alone.python -m unittest discover -s tests -t ..Note
The failure message is longer than its neighbours on purpose. This is the quietest of the three infrastructure conditions, because the run still produces a plausible number, so the message says what the number would otherwise have been read as.
Related: #23 reports what a repo's own skills cost against the same budget, so a catalog can see the pressure it adds before it ships.