Skip to content

feat: evals for the packaged install's two ways in; /upgrade stamps the release's commit - #30

Merged
mauricios merged 1 commit into
mainfrom
feat/eval-packaged-paths
Oct 5, 2026
Merged

mauricios merged 1 commit into
mainfrom
feat/eval-packaged-paths

Conversation

@mauricios

Copy link
Copy Markdown
Contributor

What changed and why

Two paths into the packaged install had never run in a real session. This PR adds session evals for both, and fixes the defect they found.

The evals:

  • adopt suite, new packaged case. /aplyca-adf:adopt on a new project, where the follow-up chooses the packaged install.

  • New upgrade suite, switch-to-packaged case. project.sh builds a committed adoption at v1.0.0 from that tag: the skeleton without the Antigravity and Cursor layers, the plugin pinned to v1.0.0, PDR-0001, and the stamp. /aplyca-adf:upgrade then moves it to the newest release, and the follow-up accepts the switch.

  • evals/dynamic/check-packaged.sh checks the packaged layout for both cases, marking each check ✓ or ✘:

    • the stamp names the newest release, its commit, and install: packaged;
    • no framework machinery is committed;
    • the settings have no hooks block, pin the newest release, and turn on aplyca-adf;
    • the names note is in CLAUDE.md, and DEV-SETUP.md uses the full command names;
    • everything is committed.

    Each suite's inspect.sh adds its own checks: for the switch, the PDR, PDR-0001 marked amended, and main untouched.

  • Runner: passes this checkout's path to a suite's project.sh; the upgrade suite runs like adopt.

What the runs found, and the fixes:

  • /upgrade stamped an annotated tag's ID instead of the release's commit. The stamp read v1.0.6 · 130753a, from git rev-parse v1.0.6, and the branch and commit message used the same ID. The real commit is ab56cb6. /upgrade and /adopt now name git rev-parse --short '<tag>^{commit}'. check-packaged.sh now checks the stamp's commit, and it marks the first run ✘. The rerun stamped ab56cb6.
  • The packaged smoke test in docs/SETUP.md lacked CLAUDE_PLUGIN_ROOT, which the plugin's hooks need since v1.0.2. Both sessions noticed and set it themselves; the doc now does.
  • Open, not fixed here: /adopt copies the skeleton from the framework source as it is, which can be past the release it pins. In this run the two were identical. Adopting from the release tag would close it, and that's a separate change.

Upgrade impact

The evals are framework-internal. The fixes change /upgrade's and /adopt's instructions and one line of docs/SETUP.md. They reach projects with the next release; the plugin's version moves then.

How to verify

./evals/dynamic/run-session-evals.sh --suite adopt --cases packaged --models sonnet
./evals/dynamic/run-session-evals.sh --suite upgrade --models sonnet

Verified / not verified

  • Verified, on Sonnet:
    • packaged: 8 of 8, $0.83.
    • switch-to-packaged: 10 of 10, $0.81, though its stamp named the tag's ID, which the checker didn't yet check.
    • switch-to-packaged after the fix: 10 of 10, stamped ab56cb6, $1.20.
    • Each transcript graded against its .expected.md (report: evals/dynamic/reports/2026-10-04-packaged-paths.md).
    • The checker marks ✘ on the committed v1.0.0 project and on run 1's stamp.
    • Static suites pass: 154, 81, 45, and 11.
  • Not verified:
    • Opus.
    • A real install, where the framework comes from the marketplace or a clone. These sessions found it through the runner's copy of the checkout.

Merge danger

  • The two skill paragraphs and a doc line are the only things outside evals. A revert is safe.

🤖 Generated with Claude Code

…he release's commit

Two session evals for paths that had never run: the adopt suite's
`packaged` case (/aplyca-adf:adopt on a new project, choosing the
packaged install) and a new upgrade suite whose `switch-to-packaged`
case has /aplyca-adf:upgrade move a committed adoption at v1.0.0 —
built from that tag — to the newest release and switch it to packaged.
Both check the end state with evals/dynamic/check-packaged.sh; the
runner passes this checkout's path to a suite's project.sh.

The upgrade run stamped `v1.0.6 · 130753a`, an annotated tag's own ID
from `git rev-parse v1.0.6`, not the release's commit ab56cb6.
/upgrade and /adopt now name `git rev-parse --short '<tag>^{commit}'`,
and the checker checks the stamp's commit (it marks that run ✘). The
packaged smoke test in docs/SETUP.md sets CLAUDE_PLUGIN_ROOT, which the
plugin's hooks need since v1.0.2.

Sonnet: packaged 8/8 ($0.83); switch-to-packaged 10/10 ($0.81), and
after the fix 10/10 with the commit stamped ($1.20). Report in
evals/dynamic/reports/2026-10-04-packaged-paths.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mauricios
mauricios marked this pull request as ready for review October 5, 2026 03:35
@mauricios
mauricios merged commit a8479dd into main Oct 5, 2026
1 check passed
@mauricios
mauricios deleted the feat/eval-packaged-paths branch October 5, 2026 03:35
mauricios added a commit that referenced this pull request Oct 5, 2026
…al sessions (#32)

A minor release: the packaged install becomes the default for new
projects (decision 0018), with evals for its paths. Unreleased becomes
v1.1.0, covering #29, #30, #31, and one more fix:

/aplyca-adf:adopt took the skeleton from wherever the framework source
was while pinning the newest release tag, so a project's committed files
could be newer than its pinned plugin. It now finds the newest release
tag, takes the framework at that tag (a shallow clone of it, or a
worktree of a local checkout), and stamps that tag's commit; SETUP.md's
manual copy says the same. The packaged adopt eval reran on Sonnet:
8/8, stamped ab56cb6 from a scratch worktree at v1.0.6.

plugin.json 1.1.0; the README names the release.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant