[kyle-sexton]

Closes #12191

Summary

The playgrounds plugin's cross-marketplace dependency kept it from loading in claude plugin eval, so both arms of its suite ran without it and measured nothing. Each case now loads an eval-only playground stand-in, and the plugin loads.

Fix

• Each of the five cases gets a playground-stub subdirectory holding a plugin manifest with no skill, and lists it in plugins: beside ../... The shipped plugin.json dependency is unchanged.
• The stand-in is one copy per case because the CLI refuses a plugins entry outside the case: a shared ../stand-ins/playground failed with "a plugins entry names the case directory, its graders or mocks, or a directory covering them".
• No skill-bearing stub is added. The decision on test(playgrounds): the eval suite runs without the plugin (dependency-unsatisfied) #12191 names one for present-path cases, and none of the five cases is present-path. The two cases the issue names, use-absent-emits-install-commands and use-cloud-session-feature-detects-delivery, are not in this repository at origin/main (git grep finds neither), so the five cases here are the whole suite. A present-path stand-in goes in with the first present-path case.
• The routing case's description no longer says the run copy drops the manifest's dependencies key.

Verification

• One --runs 1 smoke of use-routes-natural-explorer (Claude Code 2.1.317, Sonnet, a copy outside git because this checkout trips the linked-worktree refusal): the startup record lists playgrounds and playground with plugin_errors empty. This is the proof of load the decision on test(playgrounds): the eval suite runs without the plugin (dependency-unsatisfied) #12191 asks for.
• That smoke's plugins: line was in case.yaml, so a 1.1 case.yaml honours the field the way songwriting's prompt.md front matter does (refinement question 3).
• That run failed no-write, the first real signal from this suite: it invoked playgrounds:use, then wrote retry-playground.html itself, though the skill says it never builds a playground. One run, and whether the loaded stand-in contributes is UNVERIFIED.
• validate-cases.py on plugins/playgrounds/evals: no FAIL.
• Left out: a full pass of the suite, which test(playgrounds): the eval suite runs without the plugin (dependency-unsatisfied) #12191 does not ask for.

Related

• feat(evals): convert other plugins' evals.json files to claude plugin eval suites #8635

🤖 Generated with Claude Code

https://claude.ai/code/session_01VQb9Q296cDKeBR2kaZuHaV

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-03T04:02:16.960392Z	f30b68d	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[github-actions]

claude-security-review: 2 inline finding(s) on f30b68da451518d6644f08e4783dbcf7f7f62d01; the summary is in this lane's pull request comment.

claude-lane: claude-security-review

[github-actions]

claude-security-review reviewed f30b68da451518d6644f08e4783dbcf7f7f62d01: read 12 of 12 changed file(s); 2 finding(s), 2 as inline comments.

Security review: PR #12653

I read every hunk of the 12 changed files. Eight are distinct changes. The other four are the identical playground-stub/.claude-plugin/plugin.json copies, which I read once through the first path. None of the changed files is on a trusted path.

What the change does. Five eval cases in plugins/playgrounds/evals/ now list plugins: ["../..", "playground-stub"]. Each case gets a stand-in manifest that holds only name, description and version. The PR also adds a changeset with bump: none and rewords one eval prompt's description.

No exploitable security issues found. The changes don't touch any token, secret, permission, workflow, trust control or merge hold. The diff removes or weakens no standing instruction. The eval plugins: entries point at directories inside each case, so they load no code from outside the repository.

Plugin manifests. Each stand-in is a new .claude-plugin/plugin.json, so the default-deny manifest rule applies. The comparison base is "no file".

• manifest description: pass (inert field)
• manifest name: flag, a stand-in plugin named playground was added under the shipped playgrounds plugin's tree
• manifest version: flag, the field was added ("0.0.0"), not changed

Where the stand-ins could matter. The marketplace entry for playgrounds (.claude-plugin/marketplace.json:561, "source": "./plugins/playgrounds") ships the whole directory, evals/ included. The new manifests therefore reach installed users' copies. A plugin loader reads only the plugin root's .claude-plugin/plugin.json, so these nested files only take effect when claude plugin eval loads them through a case's plugins: list. The manifests declare no hooks, mcpServers, lspServers, monitors, dependencies, skills or commands, so they run nothing. The entry also sets "defaultEnabled": false (.claude-plugin/marketplace.json:563). Both flags are IMPORTANT, short of CRITICAL. The PR's shipped manifest plugins/playgrounds/.claude-plugin/plugin.json is unchanged, including its dependencies entry on playground@claude-plugins-official.

Copies. The four other copies sit at the same relative path under sibling case directories. None of them is a hook, workflow, settings or root plugin path, so each one's content does the same thing at its path. The two inline flags on the first copy cover all five.

Spend. Prepaid usage only: the eval suite now loads the plugin, which may make eval runs longer. This is not a hold.

PR context. The description contains no embedded instructions aimed at this lane.

critical-findings: 0

[github-actions]

claude-review reviewed f30b68da451518d6644f08e4783dbcf7f7f62d01: read 12 of 12 changed file(s); 0 finding(s), 0 as inline comments.

Code review: no findings

The diff is eval-only. Each of the five playgrounds eval cases now lists ["../..", "playground-stub"] under plugins:, and each case gets a copy of the same playground stand-in manifest. That manifest has no skill and version 0.0.0. The routing case's description is updated, and a bump: none changelog fragment is added. The shipped plugins/playgrounds/.claude-plugin/plugin.json is not changed.

What I checked

• The absent path still holds. None of the five cases grants Bash. Every case's allowed_tools is [Skill, Read, Glob, Grep], and the routing case adds Write. So SKILL.md's Step 1 skips claude plugin list and uses the skill-listing fallback (SKILL.md#L28-L32). The stand-in has no skill, so no playground skill appears in the listing and the cases still go to Step 3 (install/uplift). The stand-in only meets the manifest dependency. It does not fake the upstream being present, and the cases' own descriptions say the same ("upstream plugin absent, as in every eval run").
• The description edit. In use-routes-natural-explorer/prompt.md, the old claim that the run copy drops the dependencies key is replaced with a correct one: a stand-in with no skill meets the dependency. The other four prompts never made that claim.
• Identical copies. I read the first stand-in manifest and skipped its four copies, per the identical-files list. All five sit at the same relative path, evals/<case>/playground-stub/.claude-plugin/plugin.json, so none behaves differently because of where it is. The PR body says there is one copy per case because the CLI refused a shared directory outside the case. The author reports that refusal; I did not reproduce it.
• Manifests. The only manifests added are the eval stand-ins. No shipped plugin.json or marketplace.json changes, so there is no userConfig default or options change.
  • manifest version: pass (eval-only stand-in; the changelog fragment's bump: none fits a change that ships no plugin behaviour)
  • manifest description/keywords: pass (the shipped manifest is unchanged)

Closing issue. The PR only says Refs: #12191 and closes no issue. The context file has no closing-issue section, so there are no acceptance criteria to check against.

Verification. All the verification claims come from the author and I could not confirm them: the --runs 1 smoke run showing plugin_errors empty, the failed no-write run, and the clean validate-cases.py result. This lane has no shell, so I ran nothing.

Spend. Once the plugin loads, the eval suite's runs start measuring real behaviour. Those runs spend prepaid model usage, the same as before this change. No billable resource is involved.

[github-actions]

The pr-verify verifier failed head 32fc1f9be62ea3b52be5b7b6043737ec8322322d. What it wrote, as data:

What it wrote (23 lines)

Summary: Fails on a procedural rule only. The PR links #12191 with `Refs:`, so it should deliver
  part of the issue and name the rest. Its body instead says it delivers the whole issue: "Nothing
  else is left on the issue: #12191 closes by hand after this merges". The verifier rules fail a
  `refs` PR that claims a whole delivery. The diff itself looks correct and safe. It adds a
  skill-less `playground` stand-in to all five cases, each listed in `plugins:`, and leaves the
  shipped manifest unchanged, as the decision recorded on the issue asks. The check jobs that ran
  passed. test-python and test-node were skipped by test selection, which is expected since the diff
  contains only manifests, YAML and markdown.
- PR body: it says `Refs: #12191` but also says "Nothing else is left on the issue". That is a claim
  to deliver the whole issue, and a whole delivery should close it. Fix: change the line to `Closes:
  #12191` and keep the smoke-run evidence in the body. Or, if part of the issue really is left, name
  that part as remaining work instead of saying nothing is left.
- plugins/playgrounds/evals/*/playground-stub/.claude-plugin/plugin.json: the five stand-ins are
  identical, skill-less manifests named `playground`. This matches the absent-path stub in the
  decision recorded on #12191. The PR says the CLI refuses a `plugins` entry outside the case
  directory, which is why each case gets its own copy.
- plugins/playgrounds/evals/use-routes-natural-explorer/case.yaml: the triage refinement named this
  case as present-path. The decision recorded on the issue does not, and the case's own description
  treats the upstream plugin as absent. So the PR adds no skill-bearing stub here. That choice is
  consistent with the case's graders, but a human may want to confirm it.
- PR body: the smoke run is recorded only in the body (startup record lists `playgrounds` and
  `playground`, and `plugin_errors` is empty). The run failed the `no-write` grader, so whether
  loading the stand-in changes the result is still unverified.

This pull request leaves the merge queue, and ci-status holds it out while it carries pr-verify-failed. A pr-verify pass at a new commit removes the label.

[github-actions]

claude-review reviewed e8b9a63ac288a081f37f032e39792076cc75ec36: read 1 of 1 changed file(s); 0 finding(s), 0 as inline comments.

Code review (incremental since e1a05e05): no findings

I read the one file in this incremental scope, plugins/playgrounds/evals/use-routes-natural-explorer/prompt.md, and its whole change.

What changed. One sentence was added to the case's description. It says this is an absent-path case: the case's allowed_tools has no Bash, so Step 1 cannot run claude plugin list and uses the skill-listing fallback instead. The stand-in lists no playground skill there.

What I checked

• The claim matches the case. The case's allowed_tools is [Skill, Read, Glob, Grep, Write] (prompt.md#L6), with no Bash.
• The line citation is right. "lines 28-32" points at the fallback bullet in SKILL.md#L28-L32: "Where the CLI is unavailable to the session, fall back to the session's skill listing". The Step 3 range (lines 65-75) also still matches the head file.
• Nothing else moved. Only frontmatter text changed. The prompt body, expected_outcome, allowed_tools and the graders are the same as before, so the case behaves as it did.

Manifests. No manifest changed in this increment, so there are no manifest lines this round.

Settled threads. Every settled thread is on the stand-in plugin.json lines, and no file in this increment touches them, so nothing was raised again.

Closing issue. The PR says only Refs: #12191 and closes no issue, so there are no acceptance criteria to check.

Verification. This lane has no shell, so I ran nothing. Every run result the PR describes is the author's claim and I have not confirmed it.

Spend. This is a text-only change to an eval description. It spends nothing beyond the suite's existing prepaid eval runs.

[github-actions]

claude-security-review: 2 inline finding(s) on 32fc1f9be62ea3b52be5b7b6043737ec8322322d; the summary is in this lane's pull request comment.

claude-lane: claude-security-review

[github-actions]

claude-security-review reviewed e8b9a63ac288a081f37f032e39792076cc75ec36: read 1 of 1 changed file(s); 0 finding(s), 0 as inline comments.

Security review (incremental): PR #12653

Scope. This review covers the changes since e1a05e05. One file changed, plugins/playgrounds/evals/use-routes-natural-explorer/prompt.md, and I read its one hunk in full. The file matches no trusted path.

What changed. Only the eval case's description front matter. The new wording explains that this is an absent-path case: allowed_tools has no Bash, so Step 1 falls back to the skill listing. Nothing else in the front matter changed. The grants (max_turns, timeout_seconds, tags) are the same, and so is the --allow-tools Write note.

No security issues found. The change is a prose description. It touches no token, secret, permission, workflow, trust control, merge hold or plugin manifest. It does not delete or soften any standing instruction.

Earlier rounds. I did not re-check the playground-stub/.claude-plugin/plugin.json manifest flags, manifest name and manifest version. Both are IMPORTANT and were posted in earlier rounds. Those files have not changed since e1a05e05 and fall outside this incremental scope.

PR context. The pull request description and the settled threads contain no text that tries to instruct this review.

critical-findings: 0

[kyle-sexton]

Answer to the pr-verify fail note on 32fc1f9b. New head: e8b9a63ac288a081f37f032e39792076cc75ec36.

1. Refs: while claiming the whole issue. Already fixed before this push: the body now opens with Closes #12191 and the "Nothing else is left on the issue" line is gone. The whole-delivery claim still holds on current main: git grep at origin/main finds neither use-absent-emits-install-commands nor use-cloud-session-feature-detects-delivery, so the five cases here are the whole suite.
2. Five identical skill-less stand-ins. No change. The CLI refuses a plugins entry outside the case directory (the shared ../stand-ins/playground attempt failed), so each case carries its own copy.
3. use-routes-natural-explorer as present-path. It is an absent-path case, and its own graders say so: install-command requires the /plugin install playground@claude-plugins-official line that only Step 3 (plugins/playgrounds/skills/use/SKILL.md:65-75) prints. The mechanism: the case's allowed_tools has no Bash, so Step 1 cannot run claude plugin list and falls back to the skill listing (SKILL.md:28-32), where the skill-less stand-in lists no playground skill. The new commit states this chain in the case description (plugins/playgrounds/evals/use-routes-natural-explorer/prompt.md:2). The triage "present-path" label was a recommendation in the refinement, not an owner decision.
4. Smoke run failed no-write; stand-in effect unverified. test(playgrounds): the eval suite runs without the plugin (dependency-unsatisfied) #12191 asks only for proof the plugin loaded, which the smoke shows (playgrounds and playground loaded, plugin_errors empty). Whether the stand-in changes scores stays UNVERIFIED and out of scope here; no paid run was made.

Checks: validate-cases.py on plugins/playgrounds/evals reports no FAIL (one existing WARN: the routing case lists Write, which needs --allow-tools Write, as its description says). scripts/check-affected.sh --base origin/main --no-fetch: 12 passed, 3 skipped.

Note for Lane 3: PR Lane Automation pushed one commit on top of e1a05e05 (no rebase, no force). Fetch before your next push.