Skip to content

fix(framework): remove the harness system prompt for CLI agents (AI-1034) - #241

Open
seanoliver wants to merge 7 commits into
sean/ai-1034-native-skills-installfrom
sean/ai-1034-harness-native-prompts
Open

fix(framework): remove the harness system prompt for CLI agents (AI-1034)#241
seanoliver wants to merge 7 commits into
sean/ai-1034-native-skills-installfrom
sean/ai-1034-harness-native-prompts

Conversation

@seanoliver

@seanoliver seanoliver commented Aug 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What's inside

Stacked on #240, which makes the skills install predictable. Review that one first; this diff is against it, not main. No results are touched here.

Claude Tag wrote the first pass at this (previously #180, now closed).

Problem

We wrote our own system prompt into every eval run, for all four agents, no matter which one was running:

You are an agent solving a Supabase eval task in a Linux workspace. Use the provided tools to inspect and modify the workspace and run commands. When you are done, end your turn with a short summary of what you did.

Four problems with that:

  • Three of the four already have one. Claude Code, Codex and OpenCode are shipped coding agents with their own system prompt. Ours got layered on top. Codex and OpenCode have no flag for setting one at all, so what we passed them landed on the user prompt instead.
  • It described tools they do not have. bash and files_read belong to the in-process ai-sdk agent. The CLI agents have their own, under different names.
  • It coached them on when to stop. "End your turn with a short summary" shapes agentReport, which the LLM judge reads. That makes it a scoring change, not a formatting one.
  • It says the word Supabase. An agent that would otherwise pick a database has been handed the answer. We cannot build an eval that scores whether an agent reaches for Supabase on its own while that string is in the prompt, and skill activation numbers have the same pull on them.

Changes

Agent Prompt before Prompt after Finds skills via
ai-sdk framing + tool list + skills list same, minus the summary line the skills list, unchanged
claude-code same, appended to its own prompt nothing, flag omitted .claude/skills, on its own
codex same, glued onto the user prompt nothing .agents/skills, on its own
opencode same, glued onto the user prompt nothing both directories, on its own

"Nothing" is literal: no empty file, no empty flag, no blank line at the top of the user prompt.

Building the prompt

  • Prompt building moves out of run-eval.ts into a new system-prompt.ts, so it can be unit tested.
  • Fixes a bug where run-eval.ts started an eval run whenever it was imported, not only when you ran it. It now checks it was launched directly first. (Needed because this PR's new tests import it.)
  • Every piece we used to inject now checks which agent is running and returns nothing unless it is ai-sdk.
  • The "end your turn with a short summary" line is deleted for every agent, including ai-sdk.
  • If prompt text somehow reaches a CLI agent anyway, we now throw instead of quietly dropping it.

Runners

An empty prompt now means no prompt rather than an empty one:

  • Claude Code: we leave off the --append-system-prompt-file flag entirely.
  • Codex and OpenCode: no flag exists, so we stop gluing an empty block onto the front of the user prompt.
  • The shared engine writes no prompt file at all.

Artifacts

Each run now records what the agent was actually sent. That is the only way to check after the fact, since a CLI agent receives its prompt as a file outside the workspace we export. It lands in the raw results/*.json and not on the site, because export-results.ts drops the field.

How to review

Open any of these three runs and read systemPrompt in the pair's result.json. Each recorded an empty prompt and still found the Supabase skills it needed:

Run Experiment Skills the agent loaded
32785480269 claude-code-sonnet-5 supabase, supabase-postgres-best-practices
32880822811 codex-gpt-5.6 supabase
32880839623 opencode-kimi-k3 supabase, supabase-postgres-best-practices

Locally:

pnpm --filter @supabase-evals/framework test
pnpm --filter @supabase-evals/core test

Score movement

Ran the full benchmark on both branches, artifact only, nothing published. Both at runs=2, matching how the results on main were produced. Results:

#240, no prompt change #241, prompt removed
all pairs 86.0% 84.7% 21 flips, 12 down 9 up
claude-code 92.3% 92.3% no change
codex 81.5% 75.0% down 6.5 points
opencode 82.6% 89.1% up 6.5 points

This looks like normal run-to-run variability. Once #229 lands and scores average across runs, I'd expect to see the scores get much more stable.

Follow up tasks

  • Re-run the benchmark comparison once feat: average eval scores across independent runs #229 lands and the numbers can carry attribution.
  • Drop the "end your turn with a short summary" line from evals/resolve-performance-001-slow-query-cpu-spike/PROMPT.md:15, the last one left. It sits in the task text so it hits every agent equally, and removing it invalidates that eval's recorded results, so it wants its own change.

Ref AI-1034

@seanoliver
fix(core): pass no system prompt at all to CLI agents (AI-1034)
e8aee79 
The eval harness handed every agent a synthetic system prompt. For the
three CLI harnesses that prompt described a tool surface they do not have
(`bash`, `files_read`) and coached them on how to end a turn — both of
which bias exactly what an eval is supposed to measure: out-of-the-box
behaviour.

The CLI engine now treats the system prompt as optional. When it is
empty, nothing is staged and nothing is passed:

- `engine` writes `$HOME/.eval/system-prompt.txt` only for a non-empty
  prompt, and leaves `RunnerExecArgs.systemPromptPath` undefined otherwise.
- claude-code omits `--append-system-prompt-file` entirely.
- codex and opencode, which have no system-prompt flag, stop prepending a
  block (and its blank-line separator) to the *user* prompt.

Refs AI-1034, #164
@seanoliver
fix(sandbox): render the injected prompt blocks for ai-sdk only (AI-1…
feca65a 
…034)

With each CLI discovering, advertising and loading skills itself,
`buildSkillsPrompt` becomes ai-sdk-only, like `buildToolSurfaceAddendum`.
The block it rendered told agents to read `.claude/skills/<name>/SKILL.md`
with `files_read` — a path Codex cannot see and a tool no CLI harness has,
duplicating and contradicting what the agent's own harness already tells it.

`buildToolSurfaceAddendum` gets the same gate: `createCliAgent` ignores
`args.tools`, so a CLI agent works the workspace through its own built-in
tools and that text names tools it does not have.

Refs AI-1034, #164
@seanoliver
fix(framework): assemble prompts per harness and record what was sent…
f493147 
… (AI-1034)

Prompt assembly moves out of `run-eval.ts` (an entry script that runs
`main()` on import, so it cannot be unit-tested) into
`harness/system-prompt.ts`, keyed on `exp.agent.id`. Every block is now
ai-sdk-only — the task framing, the tool-surface addendum, the skills
listing — so a CLI harness assembles to `''` and the engine stages no
system prompt file. The two "end your turn with a short summary" sentences
are gone from both modes: stopping behaviour is part of what is measured.

`runOne` now returns the exact assembled `systemPrompt`, so it lands in
`results/<experiment>/<eval>.json` and what an agent was told is
verifiable from the artifacts. It was previously unrecorded for every CLI
harness. `export-results.ts` builds an explicit whitelist, so it does not
reach the published web data.

`apps/framework` gains a `test` script (`vitest run harness`), wired into
`check`, so the prompt-assembly tests have a runner.

Refs AI-1034, #164
@seanoliver
fix(framework): gate injected prompt blocks on the harness, not the c…
701144f 
…aller

buildSystemPrompt gated its base framing on the agent but passed the
addendum and skills blocks straight through. Both are ai-sdk-only today,
but that's enforced by their producers across three files rather than by
the assembler, and a block reaching a CLI harness fails silently: no
error, no red test, just an eval measuring our prompt instead of the
agent's own behaviour.

An MCP server carrying a promptAddendum is the live path in. Only
executorMcpServer has one, and only ai-sdk experiments use it, so nothing
changes today — a new CLI-harness experiment paired with it would.
@seanoliver
fix(framework): refuse a harness prompt block instead of dropping it
368c3bf 
buildSystemPrompt discarded a non-empty addendum for a CLI harness. That
is as silent as injecting it, and the block can be load-bearing:
executorMcpServer's addendum is the pause/resume protocol its tools
require, not a tool description. A CLI harness paired with it would get
the tools and none of the protocol, then stall on the first paused
execution with a recorded prompt of '' explaining nothing.

Throw instead. The producers already gate their output, so anything
arriving here means an experiment is misconfigured.

Also drop BareSandboxHandle.promptAddendum. Its only caller is guarded by
agentRunsInSandbox, true only for CLI harnesses, so buildSkillsPrompt
could only ever return '' for it.
@seanoliver
fix(framework): only run the eval harness when it is the entry point
38734a1 
run-eval.ts called main() at module scope, so importing it dispatched a
run and then called process.exit. Nothing exercised that until this
branch added `vitest run harness`, which sweeps in run-eval.test.ts, and
that file imports assertLocalMatchesInterface from run-eval.js. The suite
passes only because main() loses the race with vitest teardown; with
credentials in the environment it would start real sandbox work inside
the test worker and leak containers past process.exit.

Guard the invocation on argv[1]. Importing the module is now inert, and
running it directly is unchanged.

Also name the offending argument in the buildSystemPrompt error and point
at the runtime or MCP server that produced it, rather than calling it a
"harness addendum" when local-stack hands over a pre-joined blob.
@seanoliver
fix(sandbox): drop the bare sandbox's unused agent option
5bed078 
Dropping promptAddendum left createBareSandbox with an agent option it
never reads and a comment claiming it decides whether skills are
advertised in the prompt. Nothing in that function advertises anything.
The local-stack session still needs its agent; this one does not.
@vercel

vercel Bot commented Aug 25, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
evals Ignored Ignored Aug 25, 2026 10:17pm

Request Review

This was referenced Aug 25, 2026
Open
Closed
@seanoliver
seanoliver marked this pull request as ready for review August 25, 2026 23:38
@seanoliver
seanoliver requested review from a team and mattrossman and removed request for mattrossman August 25, 2026 23:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@seanoliver