fix(sandbox): install skills natively per harness (AI-1034)

[seanoliver]

What's inside

• ~215 lines of skills install changes (skills.ts, local-stack-runtime.ts)
• ~5 lines of plumbing that does nothing yet (index.ts), for the PR stacked on this one
• ~195 lines of tests, skimmable (unit.test.ts, docker.test.ts)

Bottom of a 2-PR stack under #241, which removes the system prompt we write for the CLI agents.

Claude Tag wrote the first pass at this (previously #180, now closed).

Problem

Installing a skill means copying its folder somewhere an agent will look, and each agent reads a different directory.

• skills add takes a flag naming which agents to install for, but we weren't not passing it.
• Without that flag, the CLI tries to detect an installed agent, and when it cannot, it installs for all 71 it knows about.
• So every sandbox got ~52 folders it had no use for: .aider-desk, .factory, .kilocode, .windsurf, .zencoder, and non-dotted data/ and skills/ among them. Each holds a copy of the same two Supabase skills.
• It also didn't install Codex skills into the native .agents/skills folder, which required us to include an addendum to the Codex prompt to get it to look in the Claude skills folder (.claude/skills).

Changes

Install

Name the three agents we run in this repo:

skills add '<dir>' --agent claude-code codex opencode --skill '*' --copy --yes

That writes two directories, .claude/skills and .agents/skills, instead of 52.

--copy is required so that .claude/skills gets created in an empty eval workspace. Without it only .agents/skills shows up.

After installing, installSkills confirms every skill is actually present in both directories (.claude/skills and .agents/skills). It used to check .claude/skills only.

Plumbing for the next PR

The prompt builders now take the agent as an argument and ignore it. This is used by #241 to skip the prompt text for the CLI agents.

How to review

Confirm the new install works:

pnpm --filter @supabase-evals/sandbox test:docker

If interested, run both install strategies side by side in a temp folder to see the difference:

mkdir -p /tmp/probe/stage/skills/demo /tmp/probe/old /tmp/probe/new
printf -- '---\nname: demo\ndescription: Demo.\n---\n\nBody.\n' > /tmp/probe/stage/skills/demo/SKILL.md

# old, no --agent
cd /tmp/probe/old
env -u CLAUDECODE -u CLAUDE_CODE_ENTRYPOINT npx -y skills@1.5.11 add /tmp/probe/stage --skill '*' --copy --yes
ls -A

# new, three agents named
cd /tmp/probe/new
env -u CLAUDECODE -u CLAUDE_CODE_ENTRYPOINT npx -y skills@1.5.11 add /tmp/probe/stage --agent claude-code codex opencode --skill '*' --copy --yes
ls -A

You should see 53 entries (old) vs. 3 (new).

Follow up tasks

• Remove skills-lock.json from the workspace after installing. The CLI drops it in the root and it ends up in the exported workspace, same as before this change.
• Assert the full set of workspace directories in the docker test. It currently samples five of the ~52 the old fallback created, so it catches that fallback firing but does not prove nothing else was written.
• Add a check that each agent actually surfaces the skills it finds. Both this PR and fix(framework): remove the harness system prompt for CLI agents (AI-1034) #241 assume it, and nothing tests it.

Ref AI-1034

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
evals	Ignored	Preview	Aug 31, 2026 8:41pm

[mattrossman]

When someone tries to add a new harness elsewhere in the repo, can we make this fail loudly if they forget to update it, instead of silently running evals w/o skills?

An exhaustive Record<AgentHarnessId, string | null> could do it, since the runner relies on the same id. SKILLS_INSTALL_AGENTS and SKILLS_INSTALL_DIRS could then be derived from it.

E.g.

const SKILLS_PATH_BY_AGENT: Record<AgentHarnessId, string | null> = {
  'ai-sdk': null,
  'claude-code': '.claude/skills',
  codex: '.agents/skills',
  opencode: '.agents/skills',
};

export const SKILLS_INSTALL_AGENTS: AgentHarnessId[] = [];
export const SKILLS_INSTALL_DIRS: string[] = [];
for (const id of agentHarnessIdSchema.options) {
  const path = SKILLS_PATH_BY_AGENT[id];
  if (path === null) continue;
  SKILLS_INSTALL_AGENTS.push(id);
  if (!SKILLS_INSTALL_DIRS.includes(path)) SKILLS_INSTALL_DIRS.push(path);
}

[seanoliver]

Done: SKILLS_PATH_BY_AGENT is the single source and both lists derive from it. A new harness id won't compile until its scope is declared, and the error names the missing harness.

SKILLS_INSTALL_DIR reads from the same map, so the prompt listing can't drift from the install.