Conversation
The legacy engine shells out to the claude CLI with permissions bypassed on the runner, and hand-rolls workspace staging, LLM judging and routing termination around it. This adds a second engine that hands that work to inspect_ai, selected with `--engine inspect` and off by default until a benchmark shows what it changes. Its agent is harness-independent rather than the claude CLI, so a skill is graded on whether its instructions work rather than on one harness's reading of them. That also removes what blocked Windows: with no CLI in the sandbox there is no agent bridge, and the bridge is Linux-only. Both engines produce the same outcome objects, so `summarize`, `render_markdown` and the report writers are untouched. Three pieces are not thin wrappers. The tools are ours because inspect's assume a POSIX guest, which the Windows legs do not have. The judge sees artifacts rather than the agent's prose, attaches images so "did it draw a cat" is answerable, and reports whether a requirement is satisfied so a "must not" is never scored by negating a verdict. Routing is a single model call with no agent loop: the decision is a tool call, visible in the first turn, so there is no turn to pay for and kill. machine.yml gains a `sandbox` key naming a compose file, so a skill that must reach the network to pull a model can say so.
Neither engine reported its own cost. The legacy engine discards the `total_cost_usd` the CLI returns with every result, and inspect keeps usage in its own log where skillscope's report never looks. That was fine with one engine and nothing to compare it against. Both now record into `usage`, which the report meta carries, and `tools/benchmark_engines.py` runs the same dataset through both and reports per-case agreement plus what each run spent. Agreement is compared case by case rather than in aggregate: two runs can post identical accuracy while individual cases flip in both directions and cancel out. `--noise` runs the legacy engine twice so flips can be read against how much it already disagrees with itself. The spend columns are not equally trustworthy and the report says so rather than tabulating them as though they were. Legacy token counts are a floor -- assistant events omit the system prompt and cached input, and a routing case is killed before its totals arrive -- so cost is its reliable figure. inspect has a cost only when the provider supplies pricing, which a gateway does not, so wall time is often the only column comparable on both sides.
The inspect engine grades a skill with a harness-independent agent, which tests whether the skill's instructions work rather than how one product reads them. That choice raises an obvious question, so `--engine claude-code` answers it: run the real harness in the sandbox and see whether it agrees. It slots into the existing flag rather than adding a command, so the benchmark tool can diff its report against either other engine with no new machinery. Reporting only -- harness runs are nondeterministic and the harness is not what we grade, so a divergence is a question about the skill rather than a build failure. Linux only, because inspect_swe shells `bash -c` to find the CLI and the proxy it installs in the guest is a Linux binary. Also fixes how the inspect engines count model calls. They recorded one per sample while the legacy engine records one per assistant response: identical for routing, where a case is a single turn, but a large undercount for behavioral, where the agent loops. The benchmark puts the two columns side by side and names that one comparable, so it had to mean the same thing on both sides. Counted from assistant messages now.
The two legacy commands read the same `claude` stream but recorded from different events: routing counted assistant replies, behavioral counted only the result event, which is one per case. A behavioral run of three cases reported three model calls while costing $0.72 -- far more than three small calls -- and the benchmark puts that column beside the other engine's and calls it comparable. Both now go through one recorder in `usage`. The same three cases report fourteen calls against the new engine's twelve, which is the like-for- like reading: two agents looping to similar depth.
`--engine` changes what drives the agent while leaving the dataset, the CLI and the reports identical, so it belongs in usage rather than being discoverable only from --help. Says what each engine needs, why the inspect one is cheaper, why the claude-code leg never gates, and where the sandbox does and does not exist.
…config Which provider to use is a property of the runner -- docker, podman on a host that has that instead, local where there is no container -- and whoever runs the job knows it while a skill does not. What the sandbox has to provide is the opposite: the skill declares a compose file when it needs network egress or a device. `SKILLSCOPE_SANDBOX` already selected the provider but returned it bare, discarding any compose file the skill had declared. A skill that needs the network would then run without it and fail for a reason nothing in the report explains. The two are now resolved independently. podman works with no registration: `inspect-podman` publishes an `inspect_ai` entry point and inspect resolves the bare name, so installing the extra is the whole setup.
A behavioral report showed the same numbers whether the agent had been contained or had worked directly in the harness's filesystem with permissions bypassed, which invites the reader to assume the former. Both engines now record what held the run, and the markdown says it outright -- the legacy engine reports `host`, and the Windows legs, which have no sandbox available at all, no longer look like the Linux ones. Routing under the inspect engine reports `none` rather than a provider: it offers the skill tool and never calls it, so nothing is executed and there is nothing to isolate. That is not the same claim as unprotected. Also stops the gateway-versus-federation guard firing on runs that reach no provider. `--model mockllm/model` is the wiring check that costs nothing, and refusing it because the shell holds both Anthropic variables broke it on exactly the machines most likely to have an OAuth token.
A graded run on the inspect engine starts a sandbox and installs a skill before it first reaches a provider, so a bad key surfaced as a task that failed after all that work -- a 401 buried in a sample error, eighteen seconds and a container in. One tiny call up front turns that into a message on the first line. The legacy engine already had this; the inspect one skipped it on the grounds that the CLI-based probe tested the wrong thing, which was true and left nothing in its place. mockllm reaches no provider, so it is skipped rather than charged for a round trip that proves nothing.
`list_paths` returned an empty list when the listing command failed, so a sandbox that could not be listed was indistinguishable from one the agent had left empty. That is not a cosmetic difference: `files_exist` fails, and the judge builds its evidence from the same listing, so every judged expectation fails too with "no files were produced". Nine checks blame the skill for what the harness did. It now raises. The scorer reports that it could not list the sandbox, and the judge declines to rule rather than being handed an empty workspace as fact -- a judge told "no files" will confidently conclude the agent did nothing.
A container sandbox starts at `/`. So `write_file("router.json")` landed
at `/router.json`, `find . -type f` walked the whole image and died on
`/proc`, and the judge -- had the listing survived -- would have read
twenty arbitrary system files as the case's artifacts. An agent left to
guess reasonably tried `/app`, then `~`, and scattered its output.
Everything a case does now happens in `/workspace`: fixtures are seeded
there, the file tools resolve against it, shell commands run in it, and
it is what gets listed. The agent is told so, because where its output
lands should not be something the dataset has to predict.
`local` keeps the harness's own working directory -- it already has a
sensible one, and creating `/workspace` on someone's machine would not be.
inspect_swe solves the same problem the same way; its agent cwd falls
back to the home directory when the sandbox default is `/`.
Two reasons a case could not pass in a container. The judge read artifacts by the path `list_paths` reported, which is relative to the case's working directory, while `read_file` resolves against the sandbox's own -- `/`. Every artifact came back unreadable and the judge concluded, reasonably, that nothing had been produced. It now resolves them the same way the tools do. And it never saw the agent's final message, so an expectation about what the agent *told* the user -- "output the curl commands they need" -- was unanswerable however well the agent had done. The legacy judge was given both halves and this one was not. They stay labelled apart, because the distinction is what stops an agent writing "I won't call the cloud API" from settling an expectation that it avoided doing so: actions are judged from tool calls and artifacts, and only what the agent told the user is judged from its message.
The judge's transcript was cut at the end, so the agent's most recent tool calls were the first to go -- and those are usually the ones an expectation is about. An agent writes a file, validates it, and reports the result; losing the validation makes the run look like it claimed something it never did, which is precisely how the last failing check read: "only the agent's unsupported claim". Tool calls are now kept whole, since each is short and the list of them is the record of what happened. Only results are capped, individually, because they are what grows without bound. If the whole still overflows, the middle goes rather than the end.
This repository is public and the fixture named a real internal CI service account alongside the gateway's header, which together describe how model access is fronted. The test is about parsing `Key: value` lines; the values were never the point.
Every bug in this branch was found by running against a real container, and none of them would have been caught by the unit suite. `mockllm` reaches no provider, so the machinery around the model can be driven for free, with no key, on a pull request from a fork -- on Ubuntu, where Docker is the default sandbox, and on Windows, where there is none and the cross-platform tools are the only thing that works. The assertions are about the harness rather than the skill, because the mock satisfies nothing. It checks that no case errored, that the sandbox that ran is the one the platform should have chosen, and that no check failed because the sandbox could not be listed -- which otherwise reports identically to an agent that did nothing. The load-bearing one is the seeded fixture. The file exists because the case staged it, not because the agent acted, so `files_exist` passing proves the whole path: staged into the working directory, listed there, and matched against what the case asked for. That is exactly the chain that was broken in a container an hour ago.
Two failures on the first run of the new job, both worth having. The sandbox-selection tests asserted `docker` on any host, which is only true on a POSIX one -- on the Windows runners the answer is `local`, because inspect's sandbox layer assumes a POSIX guest. They pin the platform now and assert both answers, which is what they should have said in the first place. The engine job pointed at its fixture with `working-directory`, but a `--skills-dir` glob that was passed is resolved against the repo root, and `find_root` takes that from the nearest `.git`. Running from a subdirectory of a checkout still globbed the checkout, so the fixture was never found. `SKILLSCOPE_REPO` is the documented way to say which repo is under test, and is what the action already uses. The local reproduction missed this because the throwaway fixture was not inside a git repository, so the root happened to be the fixture.
The CI pin coerces any non-opus model to opus so paid runs stay comparable between runs. `mockllm` reaches no provider, grades nothing and costs nothing, so pinning it turned the free wiring check into a run that needed a key -- in the one environment where not needing a key is the entire point. The new engine job failed on exactly that.
`mockllm` never calls the submit tool, so the agent looped to the full 120-message budget and every turn was a real sandbox round trip. The CI grade step took 43 seconds to learn what it knows in three turns, and the same thing made a mock run over a real catalogue take 157 seconds for three cases. A model that reaches no provider now gets six turns. The job's grade step drops to about two seconds and the assertion it rests on -- the seeded fixture being found -- is unchanged. Also caches the pip download. inspect-ai pulls in the order of eighty packages, which was the other twenty seconds.
A trial on a self-hosted runner installed the podman binary, selected `SKILLSCOPE_SANDBOX=podman`, and failed with a ValueError from inspect's registry naming neither the variable that chose the provider nor the package that supplies it. The binary being present proves nothing: inspect resolves a third-party provider through an entry point, so the Python package has to be installed too, and that is the `[podman]` extra rather than `[inspect]`. Checked once before any container starts, with the install line in the message. The resolver is injectable so this is testable without the inspect extra, which the unit suite runs without on purpose.
A `react` agent delivers its answer through the submit tool, which lands in `output.completion`. The judge read the last assistant message instead, and that is often the preamble -- so an expectation about what the agent told the user could be graded against "the commands are below" rather than the commands. On a real skill this showed up as the judge reporting that the final message "only describes having printed registration/test commands but contains no actual curl commands", which is exactly what it was shown. Falls back to the last assistant message for agents with no submit tool.
Three things, each found by the previous one failing on a real runner: the provider package, podman's own compose rather than the shim that delegates to Docker's, and a search registry because podman will not guess one for an unqualified image name. Worth the setup where the runner's user cannot reach the Docker socket, which is the case this came from.
rominf
added a commit
to amd/skills
that referenced
this pull request
Sep 14, 2026
The default sandbox has no network, which I had been treating as a reason those skills cannot be containerised. It is not: it is a default, and `sandbox:` in machine.yml exists so a skill can say otherwise. That key has been unit-tested and never used for real, so this is its first outing. `local-ai-app-integration` installs a runtime and fetches models. Its compose file is inspect's own default minus the line that removes the network, with the image fully qualified because podman will not guess a registry. Note this cannot merge ahead of amd/skillscope#19: releases before it reject `sandbox:` as an unknown key outright.
It resolved against the skill root, which is what gets published -- so eval infrastructure would ship with the skill. A path written in a file is also most usefully relative to that file. Both point at evals/, beside the machine.yml that names it. Found by writing the first real one.
A malformed sandbox declaration raised out of the whole behavioral command, so one skill's broken setup discarded results for skills already graded and paid for. On a real run that threw away three passing cases to report that a fourth skill named a compose file that was not there. It reports failed outcomes for that skill and carries on, which is the same rule the structural gate already follows: a neighbour's mistake says nothing about whether this run can proceed.
In an agent loop the user sees every assistant turn, so an expectation about what the agent told them is satisfied by any of those. Reading only the final turn credited the agent with its closing summary and called the work missing -- on a real skill the judge reported that the agent "only claims curl commands were delivered as text" when it had printed them a turn earlier. The submitted answer still comes last and is marked, because it is the answer where the rest is working. The split the prompt relies on is unchanged: actions are judged from tool calls and artifacts, and only what the agent told the user is judged from what it said.
`inspect_swe` runs Claude Code inside the sandbox and reaches the model through a bridge whose proxy is a Linux binary, so it cannot run on Windows at all. This drives the same CLI on the host, the way the legacy engine does, and maps what it did into inspect's messages -- so the scorers, the artifact-reading judge and the .eval transcript all work unchanged. The point is fidelity. A skill is written for this harness, and grading a different agent measures something customers will not experience. With this the real harness runs on both platforms and only the isolation differs: a container on Linux via --engine claude-code, the host on Windows via --engine claude-cli, with the report saying which. Unsandboxed by construction, which is what the legacy engine already does, so it is not a regression. Refuses a container provider outright rather than running the CLI in one filesystem and scoring another. `build_task` gains a solver seam: what drives the agent swaps while staging, scoring, judging and reporting stay put.
Legacy against inspect asks whether a different agent reaches the same verdicts. Legacy against claude-cli asks something narrower and sharper: both drive the same CLI, so agreement means the framework around the agent is faithful, and a flip is a defect in the crossing rather than a property of a different agent. That is the measurement that decides whether the legacy path is redundant or merely superseded. The engine list now lives in the CLI and is imported, so a new engine is offered in both places at once rather than in whichever was remembered. On the fixture the two agree on every case at 9 model calls and ~27s each, which is what the same binary run twice should look like.
Every files_exist check on the claude-code leg of the first trial run failed with "sandbox holds: nothing" -- nine of nine, across every skill that asserts files. The agent was not doing nothing: a container starts at /, so it worked there while the scorers read /workspace. The other engines say this in a prompt, because their agent takes instructions. This one takes a cwd. The directory is created before the solver runs, since this leg's agent brings its own tools and so never touches ours, which is what would otherwise have made it. Read as a comparison, the old numbers said real Claude Code performs worse than a harness-independent agent on these skills. They said nothing of the kind.
Every engine but `legacy` runs on inspect_ai, so any of them can hit the missing-extra path -- but the hint said `--engine inspect` regardless. A Windows CI job that had passed `--engine claude-cli` was told to go look at a flag it had never used. The remedy is the same extra either way, so this only costs a reader their bearings. That is enough.
The guard around the inspect dispatch was a literal tuple listing only inspect and claude-code. `claude-cli` was added to the chain inside it but never to the tuple, so it fell through to the legacy engine: every run that asked for it silently got something else, while the report recorded `engine: claude-cli` throughout. The preflight, which does key off INSPECT_ENGINES, meanwhile demanded an extra the run never used -- which is how a Windows job failed on a flag it had not passed. The absent .eval transcripts were the symptom that gave it away: legacy does not write them. This invalidates every claude-cli measurement taken so far, including the benchmark that reported legacy and claude-cli agreeing 4/4 with cost within 3%. They agreed because they were the same engine; the figure matched the noise floor exactly, which should have been the tell. Tests now assert which runner a flag reaches, and that the guard and the choices list cannot drift apart again.
npm installs `claude` as a .cmd shim. The legacy engine starts it with the synchronous subprocess.run, which copes, so this never surfaced while --engine claude-cli was silently running legacy. Under inspect's asyncio subprocess the path is exec'd directly and Windows answers WinError 2 -- which reads as "the CLI is not installed" when it plainly is, and which the report surfaced as the task failing rather than the launch failing.
The react agent installs the skill through inspect's skill() tool, which arrives as part of its toolset. A solver that replaces the react agent replaces that too -- so claude-cli ran the real CLI against an empty workspace and the agent answered from the prompt alone. It scored 4/21 on a skill every other engine passed 21/21, and finished faster doing it, which was the only visible sign. The seam is now a factory taking the skill's directory, because staging is the driver's job: legacy copies the tree into .claude/skills, and inspect_swe does the same through its own skills= argument. This does what both already do. Third invalid claude-cli measurement in a row. The first two were the dispatch falling through to legacy and the CLI failing to launch on Windows; this one produced plausible-looking numbers rather than an error, which is why it needed the comparison to catch.
The probe ran `bash -lc exit 0` and read `success` to decide between bash and PowerShell. But a guest without bash does not answer "that failed" -- there is nothing to exec, so it raises before any result exists. On a Windows host under the `local` sandbox that surfaced as WinError 2 and took the whole task down, reporting 0/0 expectations, which reads as the harness being broken rather than the probe learning exactly what it asked. Also drops the cmd.exe wrapper added last commit. It was reasoned from npm shipping `claude` as a .cmd shim that Windows cannot exec directly. The runner says otherwise: `which` resolves to claude.CMD and async exec starts it with rc=0, as do sync Popen and cmd.exe. The wrapper fixed nothing and added a shell layer between us and the agent's arguments.
inspect keeps a sqlite buffer so `inspect view` can watch a run in progress. It lives under the user data directory, named after the task, and on this runner that directory is the service account's profile -- long enough that a task named after a longer skill crosses MAX_PATH. sqlite then answers "unable to open database file" and the whole task dies. Measured rather than inferred this time: 241 characters writes, 260 does not, on the same machine in the same run. Nothing watches a CI run live and the .eval log is written either way, so the buffer is cost without benefit exactly where it breaks. LOCALAPPDATA does not help -- platformdirs asks Windows, not the environment, so the short path I set last commit was ignored. That commit's reasoning was right and its remedy did nothing.
The `--timeout` deadline ends the process with os._exit, which takes the report and the transcript with it, so an overrun says only that it overran. inspect has its own per-sample limit and we were handing it the whole budget -- so the two fired together and the hard kill won the race. Keep two minutes back. inspect stops the sample, scores what exists and writes the log; the deadline stays as the backstop for a genuinely hung process, which is what it is for. Eight Instinct runs have now overrun. Not one of them said whether the agent was working, waiting, or stuck, and two of the three diagnoses that followed were wrong because there was nothing to read.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A second eval engine, selected with
--engine inspectand off by default. The dataset format, the CLI, and the reports are unchanged; only the thing driving the agent differs.The legacy engine shells out to the
claudeCLI with permissions bypassed on the runner, and hand-rolls the machinery around it: workspace staging, an LLM judge with bespoke verdict parsing, and ~850 lines of streaming and SIGKILL logic to end routing runs early. This hands that work toinspect_ai.Its agent is harness-independent rather than the
claudeCLI, so a skill is graded on whether its instructions work rather than on how one product reads them. That is the stronger claim, and the one a product repo can adopt. It also removes what blocked Windows: with no CLI in the sandbox there is no agent bridge, and that bridge is Linux-only.Both engines produce the same outcome objects, so
summarize,render_markdownand the report writers are untouched.Risk: low while it stays opt-in. The default is
legacyand that path is unchanged.Evidence
Routing over a whole catalogue (8 skills installed together, 68 cases):
All 12 disagreements favour this engine and none go the other way — 8 legacy misses, 3 legacy over-triggers, 1 wrong skill, 1 error. Noise would be symmetric.
These are single runs. Repeating the routing leg on a five-skill room gave 46/46, then 45/46, then 46/46 — so this engine has roughly 2% run-to-run variance of its own. Well inside legacy's, but not zero, and worth knowing before anyone treats a single number as a baseline.
The more interesting number is the noise floor: the legacy engine disagrees with itself on 7 of 68 cases (~10%) across two identical runs. That is a property of the status quo rather than of this change, and it is larger than most real regressions would be.
One caveat stated plainly: "legacy misses 8 activations" is probably not "the product harness routes worse". Legacy kills each case after 4 non-bookkeeping tool calls and infers activation from the stream, so an agent that plans before activating is recorded as a miss. Some of that gap is the legacy measurement being lossy. Still an argument for this engine, but a different claim.
tools/benchmark_engines.pyproduces all of the above, and refuses to present the token columns as comparable, because they are not.Non-obvious decisions
bash(),text_editor()and friends), which the Windows legs do not have.machine.ymlgainssandbox:, naming a compose file for a skill that needs network egress or a device.SKILLSCOPE_SANDBOXselects the provider (docker,podman,local) without discarding what the skill declared.Podman needs three things beyond the binary, each found by the previous one failing on a real runner: the
[podman]extra (the provider is registered by a package, not the binary),podman-composeplusINSPECT_PODMAN_COMPOSE(barepodman composedelegates to Docker's plugin, which then needs the daemon podman was chosen to avoid), and a search registry (podman will not guess Docker Hub for an unqualified image name). All three are indocs/usage.md. Worth it where a runner's user cannot reach the Docker socket, which is the case this came from.Test plan
268 unit tests, Linux and Windows
On self-hosted runners, against a live catalogue, through an enterprise gateway: routing 46/46, and behavioral 3/3 cases with 21/21 expectations met, isolated in a podman sandbox. That is the whole stack — tools, scorers and the LLM judge — on the infrastructure that would host it.
New CI job drives the engine on
mockllm— free, no key, works from a fork — and is green on both: Ubuntu reportssandbox: docker, Windows reportssandbox: local. Its load-bearing assertion is a seeded fixture file: it exists because the case staged it, not because the agent acted, sofiles_existpassing proves staging, working directory, listing and matching all hold. That is the exact chain that was broken in a container earlier in this branch.Two consequences worth naming. The Docker sandbox is now exercised, not just podman. And the Windows leg runs the cross-platform tools on Windows, including the PowerShell listing path, which nothing else covers.
Not verified — why this is a draft
mockllmsatisfies nothing, so the LLM judge is only ever exercised by hand.--engine claude-codehas never executed. Wired, guarded, unrun.Bugs this found in itself
Every one came from running against a real container or a real runner, and none would have been caught by the unit suite: a failed sandbox listing reading as an idle agent; the container working directory defaulting to
/; artifact paths resolved against the wrong root; the judge never seeing the agent's final message; transcript truncation dropping the very action a check turns on; and a credentials guard firing on runs that reach no provider.The new CI job then found two more on its first run: tests that asserted a POSIX answer on any host, and the CI model pin coercing
mockllmto a real model — which turned the one check that needs no key into one that does.