From 8230856d59ba8108ed02bf60fa220aa9bfad076e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mauricio=20S=C3=A1nchez?= Date: Sun, 4 Oct 2026 22:09:56 -0500 Subject: [PATCH] feat: evals for the packaged install's two ways in; /upgrade stamps the release's commit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two session evals for paths that had never run: the adopt suite's `packaged` case (/aplyca-adf:adopt on a new project, choosing the packaged install) and a new upgrade suite whose `switch-to-packaged` case has /aplyca-adf:upgrade move a committed adoption at v1.0.0 — built from that tag — to the newest release and switch it to packaged. Both check the end state with evals/dynamic/check-packaged.sh; the runner passes this checkout's path to a suite's project.sh. The upgrade run stamped `v1.0.6 · 130753a`, an annotated tag's own ID from `git rev-parse v1.0.6`, not the release's commit ab56cb6. /upgrade and /adopt now name `git rev-parse --short '^{commit}'`, and the checker checks the stamp's commit (it marks that run ✘). The packaged smoke test in docs/SETUP.md sets CLAUDE_PLUGIN_ROOT, which the plugin's hooks need since v1.0.2. Sonnet: packaged 8/8 ($0.83); switch-to-packaged 10/10 ($0.81), and after the fix 10/10 with the commit stamped ($1.20). Report in evals/dynamic/reports/2026-10-04-packaged-paths.md. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 20 +++++++ docs/SETUP.md | 2 +- evals/dynamic/README.md | 18 +++++- evals/dynamic/check-packaged.sh | 26 +++++++++ evals/dynamic/fixtures/adopt/inspect.sh | 10 +++- .../fixtures/adopt/packaged.expected.md | 29 ++++++++++ .../dynamic/fixtures/adopt/packaged.input.md | 38 +++++++++++++ evals/dynamic/fixtures/upgrade/inspect.sh | 29 ++++++++++ evals/dynamic/fixtures/upgrade/project.sh | 40 +++++++++++++ .../upgrade/switch-to-packaged.expected.md | 28 ++++++++++ .../upgrade/switch-to-packaged.input.md | 36 ++++++++++++ .../reports/2026-10-04-packaged-paths.md | 56 +++++++++++++++++++ evals/dynamic/run-session-evals.sh | 15 +++-- plugins/aplyca-adf/skills/adopt/SKILL.md | 2 +- plugins/aplyca-adf/skills/upgrade/SKILL.md | 4 +- 15 files changed, 342 insertions(+), 11 deletions(-) create mode 100755 evals/dynamic/check-packaged.sh create mode 100644 evals/dynamic/fixtures/adopt/packaged.expected.md create mode 100644 evals/dynamic/fixtures/adopt/packaged.input.md create mode 100755 evals/dynamic/fixtures/upgrade/inspect.sh create mode 100755 evals/dynamic/fixtures/upgrade/project.sh create mode 100644 evals/dynamic/fixtures/upgrade/switch-to-packaged.expected.md create mode 100644 evals/dynamic/fixtures/upgrade/switch-to-packaged.input.md create mode 100644 evals/dynamic/reports/2026-10-04-packaged-paths.md diff --git a/CHANGELOG.md b/CHANGELOG.md index c348529..c53348c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,26 @@ For each entry, **Upgrade impact** classifies the change against the [three-buck ## Unreleased +### `/upgrade` stamps the release's commit; the packaged smoke test names the plugin + +The new upgrade eval found `/aplyca-adf:upgrade` stamping `v1.0.6 · 130753a` — an annotated tag's own +ID, from `git rev-parse v1.0.6` — instead of the release's commit, `ab56cb6`, in the stamp, the branch, +and the commit message. `/upgrade` and `/adopt` now name `git rev-parse --short '^{commit}'`. +The packaged smoke test in `docs/SETUP.md` sets `CLAUDE_PLUGIN_ROOT`, which the plugin's hooks need +since v1.0.2. +**Upgrade impact:** framework-internal; a project whose stamp names a tag's ID can correct it to the +commit by hand — `/aplyca-adf:upgrade` diffs from either. + +### The packaged install's two ways in, checked in real sessions + +The adopt suite gains a `packaged` case — `/aplyca-adf:adopt` on a new project, choosing the packaged +install — and a new `upgrade` suite's `switch-to-packaged` case has `/aplyca-adf:upgrade` move a +committed adoption at v1.0.0 to the newest release and switch it to the packaged install. Both check +the result with `evals/dynamic/check-packaged.sh`, which also checks that the stamp names the release's +commit. The runner passes this checkout's path to a suite's `project.sh`. Report: +`evals/dynamic/reports/2026-10-04-packaged-paths.md`. +**Upgrade impact:** framework-internal. + ### The plugin's hooks, checked in real sessions A `plugin-hooks` suite for the session evals: each of the `aplyca-adf` plugin's hooks, driven in a diff --git a/docs/SETUP.md b/docs/SETUP.md index e94d9ce..330bc65 100644 --- a/docs/SETUP.md +++ b/docs/SETUP.md @@ -210,7 +210,7 @@ v1.0.0 or later; its entry in [`CHANGELOG.md`](../CHANGELOG.md) says what it bri with the pinned release and its commit. `install: packaged` is what turns the plugin's copies on. **Verify** as below, with two differences. Pipe the hook samples to the plugin's scripts, with the -project named: `CLAUDE_PROJECT_DIR="$PWD" /plugins/aplyca-adf/hooks/guard-git.sh`, +project and the plugin named: `CLAUDE_PROJECT_DIR="$PWD" CLAUDE_PLUGIN_ROOT=/plugins/aplyca-adf /plugins/aplyca-adf/hooks/guard-git.sh`, where the marketplace folder is the `installLocation` of `aplyca` in `claude plugin marketplace list --json`. And in a new session, `/aplyca-adf:triage` is offered. diff --git a/evals/dynamic/README.md b/evals/dynamic/README.md index 9db3b42..6362a41 100644 --- a/evals/dynamic/README.md +++ b/evals/dynamic/README.md @@ -7,6 +7,7 @@ Fixture-based evals that **invoke an AI with an input prompt and grade the outpu ``` dynamic/ README.md - this file + check-packaged.sh - checks a project's packaged layout (adopt's and upgrade's inspect.sh use it) run-dynamic.md - manual + automated execution patterns fixtures/ write-spec/ @@ -20,7 +21,11 @@ dynamic/ adopt/ project.sh - builds a new project (no commits, nothing built) in place of the newsletter one inspect.sh - records each run's end state: commits, stamp, planned entries, decision records - ... - adopting from one line with the framework's address; /adopt in a new project + ... - adopting from one line with the framework's address; /adopt in a new project; on the packaged install + upgrade/ + project.sh - builds a committed adoption at v1.0.0, from that tag + inspect.sh - records the end state; checks the switch to the packaged install + ... - /upgrade switching a committed project to the packaged install plugin-hooks/ project.sh - builds a project on the packaged install, with something for each hook to stop inspect.sh - checks that the case's hook fired (or stood down): ✓ or ✘ per check @@ -40,7 +45,7 @@ Each fixture has two files: ## Running -### Session evals — automated (triage, debug, adopt, plugin-hooks) +### Session evals — automated (triage, debug, adopt, upgrade, plugin-hooks) `run-session-evals.sh` runs every fixture in one suite — `fixtures/triage/` by default, or `--suite debug` — against real Claude Code sessions: it builds a fictional project in a temp @@ -58,8 +63,17 @@ signal without installing anything. ./run-session-evals.sh --suite debug # the /debug cases ./run-session-evals.sh --suite adopt --source "$PWD/../.." # adoption, against this checkout ./run-session-evals.sh --suite plugin-hooks # the plugin's hooks, on Haiku +./run-session-evals.sh --suite upgrade --models sonnet # /upgrade, switching to the packaged install ``` +The adopt suite's **packaged** case, and the **upgrade** suite's **switch-to-packaged**, check the +packaged install's two ways in: adopting on it, and switching a committed project to it. The upgrade +suite builds a committed adoption at v1.0.0 from that tag (`project.sh` gets this checkout's path), and +`/aplyca-adf:upgrade` moves it to the newest release. Both cases' `inspect.sh` run +`check-packaged.sh`: the stamp naming the newest release and its commit, no framework machinery committed, the plugin pinned to that release and +turned on, no `hooks` block, the stamp, and the names people type — each marked ✓ or ✘ — plus the PDR +that records the choice. Both are long sessions; run them on one model. + The **plugin-hooks** suite checks the `aplyca-adf` plugin's hooks in real sessions, which the static hook tests can't: that Claude Code runs them from the plugin, and that what they print reaches the session. Its project is on the packaged install, with the plugin loaded from this checkout per session. diff --git a/evals/dynamic/check-packaged.sh b/evals/dynamic/check-packaged.sh new file mode 100755 index 0000000..bd6f028 --- /dev/null +++ b/evals/dynamic/check-packaged.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +# +# Checks that a project is laid out for the packaged install (decision 0016, docs/SETUP.md § Packaged +# install), pinned to : no framework machinery committed, the plugin pinned and turned on, no +# hooks block, the stamp, and the names people type. Prints ✓ or ✘ per check. Read-only. +# Usage: check-packaged.sh +# +cd "$1" || exit 1 +release="$2" commit="$3" +check() { if eval "$2"; then echo "- ✓ $1"; else echo "- ✘ $1"; fi; } +check "the stamp names $release, its commit $commit, and \`install: packaged\`" \ + "head -1 CLAUDE.md | grep -q -F 'Skeleton source: $release · $commit' && head -1 CLAUDE.md | grep -q -F 'install: packaged'" +check "no framework skills, agents, or workflows committed" \ + "[ ! -e .claude/skills/triage ] && [ ! -e .claude/skills/write-spec ] && [ ! -e .claude/agents ] && [ ! -e .claude/workflows ]" +check "\`.claude/hooks/\` holds only \`config.sh\`" "[ -f .claude/hooks/config.sh ] && [ \"\$(ls .claude/hooks)\" = config.sh ]" +check "the settings: no \`hooks\` block, the marketplace pinned to $release, \`aplyca-adf\` turned on" "python3 - '$release' <<'PY' +import json, sys +s = json.load(open('.claude/settings.json')) +source = s.get('extraKnownMarketplaces', {}).get('aplyca', {}).get('source', {}) +ok = 'hooks' not in s and source.get('repo', '').lower() == 'aplyca/agenticdevelopmentframework' \ + and source.get('ref') == sys.argv[1] and s.get('enabledPlugins', {}).get('aplyca-adf@aplyca') is True +sys.exit(0 if ok else 1) +PY" +check "\`CLAUDE.md\` says the project uses the packaged install" "grep -q 'uses the packaged install' CLAUDE.md" +check "\`DEV-SETUP.md\` gives the commands by their full names" "grep -q '/aplyca-adf:triage' docs/getting-started/DEV-SETUP.md" +check "everything is committed" "[ -z \"\$(git status --short)\" ]" diff --git a/evals/dynamic/fixtures/adopt/inspect.sh b/evals/dynamic/fixtures/adopt/inspect.sh index 9a285c4..fbf3202 100755 --- a/evals/dynamic/fixtures/adopt/inspect.sh +++ b/evals/dynamic/fixtures/adopt/inspect.sh @@ -1,7 +1,8 @@ #!/usr/bin/env bash # # Prints an adopt run's end state for grading: commits per branch, what was added, the version stamp, -# the planned entries, the decision records, and whether the settings parse. Read-only. +# the planned entries, the decision records, and whether the settings parse; for the packaged case, +# checks the packaged layout (../../check-packaged.sh). Read-only. # cd "$1" || exit 1 if [ ! -f CLAUDE.md ]; then @@ -51,3 +52,10 @@ for f in GEMINI.md .agents .cursor evals .github/pull_request_template.md .githo [ -e "$f" ] && echo "present: $f" || echo "absent: $f" done echo '```' +if [ "${3:-}" = packaged ]; then + echo "### Checks — the packaged install" + release="$(git -C "$FW" tag --list 'v*' --sort=-v:refname | head -1)" + bash "$(dirname "$0")/../../check-packaged.sh" . "$release" "$(git -C "$FW" rev-parse --short "$release^{commit}")" + if grep -q -i 'packaged' docs/process/0001-*.md 2>/dev/null; then echo "- ✓ PDR-0001 records the packaged install" + else echo "- ✘ PDR-0001 records the packaged install"; fi +fi diff --git a/evals/dynamic/fixtures/adopt/packaged.expected.md b/evals/dynamic/fixtures/adopt/packaged.expected.md new file mode 100644 index 0000000..b1fd8bf --- /dev/null +++ b/evals/dynamic/fixtures/adopt/packaged.expected.md @@ -0,0 +1,29 @@ +# Expected — adopt: a new project on the packaged install + +The session should satisfy ALL of these invariants. `inspect.sh` checks the end state ones marked +(auto). + +## First turn + +- [ ] Recognizes a new project and asks for the planned stack and the other facts in one round, + including how to install (committed or packaged), with its recommendations +- [ ] Copies nothing before the answers + +## End state (after the follow-up) + +- [ ] `main` has exactly one commit, the README; the adoption is on its own branch +- [ ] (auto) `CLAUDE.md`'s stamp names the newest release tag and `install: packaged` +- [ ] (auto) No framework skills, agents, or workflows are committed; `.claude/hooks/` holds only + `config.sh` +- [ ] (auto) `.claude/settings.json` has no `hooks` block, pins the `aplyca` marketplace to the newest + release (`"ref"`), and turns on `aplyca-adf@aplyca` +- [ ] (auto) `CLAUDE.md` has the names note ("This project uses the packaged install") +- [ ] (auto) `docs/getting-started/DEV-SETUP.md` gives the key commands by their full names +- [ ] (auto) PDR-0001 records the packaged install and why +- [ ] (auto) Everything is committed +- [ ] The verification checks the packaged install (the plugin's hook scripts, run with + `CLAUDE_PROJECT_DIR` set) instead of the project's own + +## Always + +- [ ] No outward action (push, pull request) is taken without an ask diff --git a/evals/dynamic/fixtures/adopt/packaged.input.md b/evals/dynamic/fixtures/adopt/packaged.input.md new file mode 100644 index 0000000..c5cd4b3 --- /dev/null +++ b/evals/dynamic/fixtures/adopt/packaged.input.md @@ -0,0 +1,38 @@ +# Input — adopt: a new project on the packaged install + + + +This fixture verifies `/adopt` with the packaged install (decision 0016): given a team that works in +Claude Code only, it commits only the project's own layer — no framework skills, agents, workflows, +or hook scripts — pins the `aplyca-adf` plugin to the newest release tag, stamps +`install: packaged`, and tells people the names they type. + +## Repository context to give the AI + +The same new project as `new-project`: a git repository with no commits and a README that says +nothing is built yet. The installer plugin is loaded from this checkout for the session only +(`--plugin-dir`). The framework's releases are tagged on GitHub, so the packaged install is available. +There is no remote. + +## Prompt to give the AI + +``` +/aplyca-adf:adopt +``` + +## Follow-up + +``` +Yes, make the first commit. The plan: Next.js 15 with TypeScript, pnpm, Vitest for unit tests and +Playwright end to end, hosted on Vercel, Contentful as the CMS. Branching: feature branches into +main. Requirements come as GitHub issues; no tracker server. One developer with one agent session +at a time, and no stakeholder updates. No sensitive areas yet. We work in Claude Code only, so use +the packaged install. No custom skills and no modules. The decider is the tech lead. Go ahead with +the adoption; don't push. +``` + +## What to do with this fixture + +1. Run it with `run-session-evals.sh --suite adopt --cases packaged --models sonnet`. +2. `inspect.sh` checks the packaged layout and prints ✓ or ✘ under the end state. +3. Compare against `packaged.expected.md`. diff --git a/evals/dynamic/fixtures/upgrade/inspect.sh b/evals/dynamic/fixtures/upgrade/inspect.sh new file mode 100755 index 0000000..8fa7642 --- /dev/null +++ b/evals/dynamic/fixtures/upgrade/inspect.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# +# Prints an upgrade run's end state — branches and commits, the stamp, the settings — and, for the +# switch-to-packaged case, checks the packaged layout (../../check-packaged.sh), the PDR that records +# the switch, and that main is untouched. Read-only. +# Usage: inspect.sh +# +cd "$1" || exit 1 +echo "### Branches and commits" +echo '```' +for b in $(git for-each-ref --format='%(refname:short)' refs/heads); do echo "--- $b"; git log --oneline "$b" 2>&1 | head -5; done +echo "--- current: $(git branch --show-current) · working tree: $(git status --short | wc -l | tr -d ' ') uncommitted paths" +echo '```' +echo "### CLAUDE.md, first line · .claude/ · docs/process/" +echo '```' +head -1 CLAUDE.md +ls .claude .claude/hooks 2>&1 +ls docs/process +echo '```' +if [ "${3:-}" = switch-to-packaged ]; then + echo "### Checks — the switch to the packaged install" + release="$(git -C "$FW" tag --list 'v*' --sort=-v:refname | head -1)" + bash "$(dirname "$0")/../../check-packaged.sh" . "$release" "$(git -C "$FW" rev-parse --short "$release^{commit}")" + check() { if eval "$2"; then echo "- ✓ $1"; else echo "- ✘ $1"; fi; } + check "a new PDR records the switch to the packaged install" \ + "ls docs/process | grep -v -E '^(0000|0001)-|README' | grep -q . && grep -l -i 'packaged' \$(ls docs/process/*.md | grep -v -E '/(0000|0001)-|README') >/dev/null" + check "PDR-0001 is marked amended" "grep -q -i 'amend' docs/process/0001-*.md" + check "\`main\` is untouched: the switch is on its own branch" "[ \"\$(git rev-list --count main)\" = 1 ] && [ \"\$(git branch --show-current)\" != main ]" +fi diff --git a/evals/dynamic/fixtures/upgrade/project.sh b/evals/dynamic/fixtures/upgrade/project.sh new file mode 100755 index 0000000..a6cf799 --- /dev/null +++ b/evals/dynamic/fixtures/upgrade/project.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +# +# The project for the upgrade cases: a committed adoption at v1.0.0 — the skeleton from the framework's +# v1.0.0 tag, without the Antigravity and Cursor layers (a team that works in Claude Code only), the +# plugin pinned to v1.0.0, a filled-in PDR-0001, and the stamp — all committed on main. +# Usage: project.sh +# +set -euo pipefail +cd "$1" +FW="$2" +git init -q -b main && git config user.email dev@example.com && git config user.name dev +git -C "$FW" archive v1.0.0 skeleton | tar -x --strip-components=1 +rm -rf .agents .cursor GEMINI.md evals +python3 - <<'PY' +import json, re +from pathlib import Path +def fill(path, pairs): + p = Path(path); s = p.read_text() + for old, new in pairs: + assert old in s, (path, old[:40]); s = s.replace(old, new) + p.write_text(s) +claude = Path("CLAUDE.md").read_text().split("\n", 1) +Path("CLAUDE.md").write_text("\n" + claude[1]) +fill("CLAUDE.md", [("# [PROJECT NAME] — Claude Code", "# Newsletter Site — Claude Code")]) +fill("AGENTS.md", [("# [PROJECT NAME]", "# Newsletter Site"), + ("[One paragraph: what this project is, who uses it, and its stage — PoC, MVP, or production.]", + "The marketing site for a publisher: article pages and a newsletter signup. In production.")]) +settings = json.loads(Path(".claude/settings.json").read_text()) +settings["extraKnownMarketplaces"] = {"aplyca": {"source": {"source": "github", "repo": "aplyca/AgenticDevelopmentFramework", "ref": "v1.0.0"}}} +settings["enabledPlugins"] = {"aplyca-adf@aplyca": True} +Path(".claude/settings.json").write_text(json.dumps(settings, indent=2) + "\n") +template = Path("docs/process/0000-pdr-template.md").read_text() +pdr = re.sub(r"^# PDR NNNN: .*$", "# PDR 0001: Adopt the AI-assisted development workflow", template, flags=re.M) +pdr = pdr.replace("proposed | accepted | superseded by PDR-NNNN", "accepted").replace("YYYY-MM-DD", "2026-10-02") \ + .replace("[who]", "the tech lead").replace('[PDR-NNNN, or "—"]', "—") +Path("docs/process/0001-adopt-ai-assisted-workflow.md").write_text(pdr) +fill("docs/process/README.md", [("| 0001 | [`0001-adopt-ai-assisted-workflow.md` — Adopt the AI-assisted development workflow] | [accepted] |", + "| 0001 | [Adopt the AI-assisted development workflow](0001-adopt-ai-assisted-workflow.md) — committed install | accepted |")]) +PY +git add -A && git commit -q -m "docs: adopt the Agentic Development Framework (v1.0.0)" diff --git a/evals/dynamic/fixtures/upgrade/switch-to-packaged.expected.md b/evals/dynamic/fixtures/upgrade/switch-to-packaged.expected.md new file mode 100644 index 0000000..1b24079 --- /dev/null +++ b/evals/dynamic/fixtures/upgrade/switch-to-packaged.expected.md @@ -0,0 +1,28 @@ +# Expected — upgrade: a committed project switches to the packaged install + +The session should satisfy ALL of these invariants. `inspect.sh` checks the end state ones marked +(auto). + +## First turn + +- [ ] Reads the stamp (v1.0.0, committed) and moves to the newest release tag, saying whether the + jump crosses a major version +- [ ] Presents the plan before changing anything, and offers the switch to the packaged install +- [ ] Changes nothing before the answer + +## End state (after the follow-up) + +- [ ] (auto) `main` is untouched; the upgrade and the switch are on their own branch +- [ ] (auto) `CLAUDE.md`'s stamp names the newest release tag and `install: packaged` +- [ ] (auto) The framework's skills, agents, workflows, and hook scripts are gone; `.claude/hooks/` + holds only `config.sh` +- [ ] (auto) `.claude/settings.json` has no `hooks` block, pins the marketplace to the newest release, + and turns on `aplyca-adf@aplyca` +- [ ] (auto) `CLAUDE.md` has the names note, and `DEV-SETUP.md` gives the commands by their full names +- [ ] (auto) A new PDR records the switch, and PDR-0001 is marked amended +- [ ] (auto) Everything is committed +- [ ] The pull request body is shown, naming the switch and its PDR; nothing is pushed + +## Always + +- [ ] No outward action (push, pull request) is taken without an ask diff --git a/evals/dynamic/fixtures/upgrade/switch-to-packaged.input.md b/evals/dynamic/fixtures/upgrade/switch-to-packaged.input.md new file mode 100644 index 0000000..2a38ced --- /dev/null +++ b/evals/dynamic/fixtures/upgrade/switch-to-packaged.input.md @@ -0,0 +1,36 @@ +# Input — upgrade: a committed project switches to the packaged install + + + +This fixture verifies `/upgrade`'s switch from the committed install to the packaged one (decision +0016): it moves the project to the newest release, removes the framework machinery the plugin +carries and the settings' `hooks` block, pins the plugin to that release, stamps `install: packaged`, +adds the names people type, and records the switch as a PDR that amends PDR-0001 — on its own +branch, delivered as a pull request it doesn't push. + +## Repository context to give the AI + +A project adopted at v1.0.0 on the committed install, for a team that works in Claude Code only: the +skeleton without the Antigravity and Cursor layers, its hooks wired in `.claude/settings.json`, the +`aplyca-adf` plugin turned on and pinned to `v1.0.0`, PDR-0001, and the stamp — all committed on +`main`. The plugin is loaded from this checkout for the session only (`--plugin-dir`); the +framework's releases are tagged on GitHub. There is no remote. + +## Prompt to give the AI + +``` +/aplyca-adf:upgrade +``` + +## Follow-up + +``` +Yes, switch to the packaged install — the team works in Claude Code only and doesn't need cloud +sessions. No new modules. The decider is the tech lead. Go ahead with the plan; don't push. +``` + +## What to do with this fixture + +1. Run it with `run-session-evals.sh --suite upgrade --cases switch-to-packaged --models sonnet`. +2. `inspect.sh` checks the end state and prints ✓ or ✘. +3. Compare against `switch-to-packaged.expected.md`. diff --git a/evals/dynamic/reports/2026-10-04-packaged-paths.md b/evals/dynamic/reports/2026-10-04-packaged-paths.md new file mode 100644 index 0000000..14960ab --- /dev/null +++ b/evals/dynamic/reports/2026-10-04-packaged-paths.md @@ -0,0 +1,56 @@ +# Session evals — the packaged install's two ways in — 2026-10-04 + +Two new cases, run against real headless Claude Code sessions (v2.1.286) on `sonnet` (Sonnet 5.5) +with the `aplyca-adf` plugin loaded from a checkout (`--plugin-dir`): the adopt suite's `packaged` +(`/aplyca-adf:adopt` on a new project, choosing the packaged install) and the upgrade suite's +`switch-to-packaged` (`/aplyca-adf:upgrade` moving a committed adoption at v1.0.0 to the newest +release and switching it to the packaged install). Neither path had run before. `inspect.sh` checked +each end state with `check-packaged.sh`, and each transcript was read against its `.expected.md`. + +| Run | Case | Turns | Checks | Cost | +|---|---|---|---|---| +| 1 | `packaged` | 16 | 8 of 8 | $0.83 | +| 1 | `switch-to-packaged` | 16 | 10 of 10 — but see the stamp below | $0.81 | +| 2 — after the fix | `switch-to-packaged` | 40 | 10 of 10, with the stricter stamp check | $1.20 | + +## Adopting on the packaged install + +| Invariant | Sonnet | +|---|---| +| Recognizes a new project and asks one round, including committed or packaged, with recommendations | ✓ twelve questions, packaged recommended for a Claude Code-only team | +| Copies nothing before the answers | ✓ | +| `main` holds only the first commit; the adoption is on its own branch | ✓ `docs/agentic-adoption` | +| Stamp: the newest release, its commit, `install: packaged` | ✓ `v1.0.6 · ab56cb6` | +| No framework machinery committed; `.claude/hooks/` holds only `config.sh` | ✓ | +| Settings: no `hooks` block, pinned to `v1.0.6`, `aplyca-adf` on | ✓ | +| Names note in `CLAUDE.md`; full names in `DEV-SETUP.md` | ✓ | +| PDR-0001 records the packaged install | ✓ (status proposed, until the adoption is approved) | +| Verifies the plugin's hook scripts, not the project's | ✓ push to `main`, `--no-verify`, a lockfile edit: exit 2; `git status`: 0 | +| No outward action | ✓ the PR body shown; no remote | + +## Switching a committed project to the packaged install + +| Invariant | Run 1 | Run 2 | +|---|---|---| +| Reads the stamp and moves to the newest release, saying whether it crosses a major | ✓ v1.0.0 → v1.0.6, no major | ✓ | +| Presents the plan first and offers the switch; changes nothing before the answer | ✓ a per-file table, every removed file confirmed unchanged since adoption | ✓ | +| Stamp, branch, and commit message name the release's commit | ✘ `130753a`, the annotated tag's own ID | ✓ `ab56cb6` | +| Machinery and the `hooks` block removed; pinned to `v1.0.6`; names note and full names | ✓ | ✓ | +| PDR-0002 records the switch; PDR-0001 marked amended; index row | ✓ | ✓ | +| `main` untouched; everything committed; nothing pushed | ✓ | ✓ | + +## What the runs found + +- **The upgrade stamped an annotated tag's ID instead of the release's commit.** `/upgrade` said + "NEW_SHA is the commit of the newest release tag" without the command, and the session ran + `git rev-parse v1.0.6`, which gives the tag object (`130753a`), not the commit (`ab56cb6`). The + skill now names `git rev-parse --short '^{commit}'`, as does `/adopt` for the packaged stamp, + and `check-packaged.sh` checks the stamp's commit — it marks run 1 ✘. Run 2 stamped `ab56cb6`. +- **The packaged smoke test in `docs/SETUP.md` lacked `CLAUDE_PLUGIN_ROOT`,** which the plugin's hooks + need since v1.0.2 to load `_lib.sh`. Both sessions noticed and set it; the doc now does. +- **Open — not fixed here:** `/adopt` copies the skeleton from the framework source as it is, which + can be past the release it pins. Here the skeleton was identical between v1.0.6 and the source, so + nothing differed; in general a packaged project's committed layer could be newer than its pinned + plugin. Adopting from the release tag would close it. +- Both sessions found the framework through the runner's `--add-dir` copy of the checkout, a + development setup. A real install finds it through the marketplace or a clone. diff --git a/evals/dynamic/run-session-evals.sh b/evals/dynamic/run-session-evals.sh index 5913faa..04d48e9 100755 --- a/evals/dynamic/run-session-evals.sh +++ b/evals/dynamic/run-session-evals.sh @@ -1,11 +1,12 @@ #!/usr/bin/env bash # -# Runs a suite of session fixtures (fixtures// — triage by default, debug, adopt, or +# Runs a suite of session fixtures (fixtures// — triage by default, debug, adopt, upgrade, or # plugin-hooks) against real Claude Code sessions. # # Builds a fictional project in a temp directory — the skeleton, the delivered newsletter-signup # spec folder from docs/examples/, and a few stub source files matching the fixtures' context, plus -# whatever the suite's setup.sh adds; a suite with its own project.sh builds its project instead — +# whatever the suite's setup.sh adds; a suite with its own project.sh builds its project instead, +# given the project's path and this checkout's — # then runs each fixture's prompt headless with `claude -p` on each model. Each run works in its own # throwaway copy, may edit it (so the project's hooks — triage-first, careful-paths — take part), and # is turn- and budget-capped; `--read-only` denies edits instead, so a run stops at its first edit. @@ -15,6 +16,10 @@ # copy before the session. Transcripts land in the output directory for grading against each # fixture's .expected.md. # +# The upgrade suite: a committed adoption at v1.0.0, built from that tag, which /aplyca-adf:upgrade moves +# to the newest release — the switch-to-packaged case switches it to the packaged install on the way. It +# runs like the adopt suite (bypassPermissions on throwaway copies, the plugin per session). +# # The plugin-hooks suite: a project on the packaged install, with the plugin loaded per session; each # case drives one of the plugin's hooks in a real session (Haiku by default), and inspect.sh checks # that it fired — or, in a committed project, that it stood down. @@ -27,7 +32,7 @@ # this checkout — with deny rules, which hold in every mode, for `claude` commands, `git push`, and # file-tool edits under your home folder. # -# Usage: ./run-session-evals.sh [--suite triage|debug|adopt|plugin-hooks] [--models "sonnet opus"] [--cases "a b ..."] +# Usage: ./run-session-evals.sh [--suite triage|debug|adopt|upgrade|plugin-hooks] [--models "sonnet opus"] [--cases "a b ..."] # [--out DIR] [--budget USD] [--parallel 4] [--read-only] [--source URL|PATH] # Needs: a signed-in Claude Code CLI (`claude auth login`), git, python3. # @@ -60,7 +65,7 @@ FIXTURES="$SCRIPT_DIR/fixtures/$SUITE" [ -d "$FIXTURES" ] || { echo "no such suite: $SUITE" >&2; exit 2; } case "$SUITE" in debug) MAX_TURNS=30; EXTRA_TOOLS="Bash(node:*)|Bash(pnpm test:*)|Bash(npm test:*)" ;; - adopt) MAX_TURNS=80; BUDGET="${BUDGET:-8.00}"; BYPASS=1 + adopt | upgrade) MAX_TURNS=80; BUDGET="${BUDGET:-8.00}"; BYPASS=1 EXTRA_TOOLS="WebFetch|Bash(git:*)|Bash(cp:*)|Bash(mkdir:*)|Bash(mv:*)|Bash(rm:*)|Bash(chmod:*)|Bash(python3:*)|Bash(printf:*)|Bash(echo:*)|Bash(test:*)|Bash(sed:*)|Bash(touch:*)|Bash(diff:*)" ;; plugin-hooks) MAX_TURNS=8; BUDGET="${BUDGET:-0.50}"; MODELS="${MODELS:-haiku}"; EXTRA_TOOLS="Bash(git commit:*)" ;; *) MAX_TURNS=14; EXTRA_TOOLS="" ;; @@ -85,7 +90,7 @@ fi mkdir -p "$REPO" cd "$REPO" || exit 1 if [ -f "$FIXTURES/project.sh" ]; then -bash "$FIXTURES/project.sh" "$REPO" || exit 1 +bash "$FIXTURES/project.sh" "$REPO" "$FW" || exit 1 else cp -R "$FW/skeleton/." "$REPO/" git init -q -b main && git config user.email dev@example.com && git config user.name dev diff --git a/plugins/aplyca-adf/skills/adopt/SKILL.md b/plugins/aplyca-adf/skills/adopt/SKILL.md index ec5cdbc..2453557 100644 --- a/plugins/aplyca-adf/skills/adopt/SKILL.md +++ b/plugins/aplyca-adf/skills/adopt/SKILL.md @@ -175,7 +175,7 @@ Present the table before going further. Wrong facts here poison every file downs - Top of `CLAUDE.md`: `` Without it, `/upgrade` has no baseline to diff against. Packaged: the release is the pinned tag and - the SHA its commit, and `· install: packaged` follows the modules — it's what turns the plugin's + the SHA its commit (`git -C rev-parse --short '^{commit}'`), and `· install: packaged` follows the modules — it's what turns the plugin's skills, agents, and hooks on in this project. - Write **`docs/process/0001-adopt-ai-assisted-workflow.md`** from the PDR template: why the team is adopting, what it adds (files, gates, modules, and the install — committed or packaged, and diff --git a/plugins/aplyca-adf/skills/upgrade/SKILL.md b/plugins/aplyca-adf/skills/upgrade/SKILL.md index 06d84f1..f9dbac2 100644 --- a/plugins/aplyca-adf/skills/upgrade/SKILL.md +++ b/plugins/aplyca-adf/skills/upgrade/SKILL.md @@ -51,7 +51,9 @@ project pins, so it can't show what's newer. A project moves **from release to release** (decision 0017): NEW_SHA is the commit of the newest release tag (`git -C tag --list 'v*' --sort=-v:refname | head -1`), unless the -developer asks for the unreleased head. Say whether the jump crosses a major version — those carry +developer asks for the unreleased head: `git -C rev-parse --short '^{commit}'`. +Not `git rev-parse `, which gives an annotated tag's own ID — never a commit, so wrong in the +stamp and the branch name. Say whether the jump crosses a major version — those carry steps the team has to take. Read `CHANGELOG.md` entries between OLD_SHA and NEW_SHA. Each entry's **Upgrade impact** pre-classifies