Skip to content

Report paths a skill writes that are not in the skill - #20

Open
johnl-amd wants to merge 1 commit into
mainfrom
unresolvable-skill-paths
Open

johnl-amd wants to merge 1 commit into
mainfrom
unresolvable-skill-paths

Conversation

@johnl-amd

Copy link
Copy Markdown
Collaborator

Summary

  • Structural now reports a path a skill's markdown writes about its own files when that path is not in the skill. Two new pieces: references.path_mentions, which reads path-shaped tokens from raw markdown, and structure._path_errors, wired into skill_errors beside _required_errors.
  • Why: a skill names its own scripts and data in prose and code spans, which are not markdown links, so the reference checks never see them. A path written against the source repo's layout resolves only for someone standing in that repo. It is dead in every way a skill is actually installed: vendored into a catalog, copied into an agent's skills directory, or fetched with the CLI. The agent follows it, finds nothing, and improvises, which is why a skill with one keeps passing its evals.
  • Risk: low. No agent calls, no dataset or config change, nothing added to the schema. Deterministic, so it cannot flake.

What it does not flag

Every one of these started as a false positive against real skills and became a rule:

  • Relative paths resolved against the file they are written in, the way a markdown link is, as well as against the skill root. Either reaches the file.
  • ${SKILL_DIR}/scripts/launch.sh and similar, where the prefix is stripped and the tail is tested.
  • Paths inside shell scripts, which resolve at runtime against the script rather than against the agent.
  • Absolute paths, which are a container mount or a host layout.
  • Files at the skill root, since a bare SKILL.md appears in install instructions for unrelated trees.
  • evals/, which is test data and never the agent's to read.
  • A path whose filename matches but whose tail does not, which is a different file that happens to share a name.

Reported only when the mention ends with exactly where the file ships, so what was meant is not in doubt.

Test plan

  • 9 new tests in TestPathsASkillWritesAboutItself, one per rule above plus the failing case. Full suite 235 tests, green.
  • Run against a real catalog of 10 skills: 2 findings, both genuine, 0 false positives.
  • Verified in both directions. With the affected skill's paths left alone the check reports them; with the same paths made skill-relative, structural passes and the existing internal reference check still resolves 125 references across 43 markdown files.

Before merging

One published skill currently trips this. Its paths need fixing in the product repo it is federated from rather than in the vendored copy, which a re-import would overwrite. Since consumers pin the harness by tag, merging here does not reach them until a tag is cut and the ref is bumped, so this can land first, but a tag should wait for that fix.

A skill names its own scripts and data in prose and in code spans, which
are not markdown links, so the reference checks never see them. A path
written against the source repo's layout resolves only for someone
standing in that repo, and is dead in every way a skill is actually
installed: vendored into a catalog, copied into an agent's skills
directory, or fetched with the CLI. The agent follows it, finds nothing,
and improvises, which is why the skill keeps passing its evals.

A mention is resolved against the file it is written in, the way a
markdown link is, and against the skill root, the way an agent handed a
skill folder reads it. Either reaches the file, so only a path that
answers to neither is reported, and only when it ends with exactly where
the file ships, so what was meant is not in doubt.

Signed-off-by: John Lybeck <john.lybeck@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant