chore: remove the one-off arm-B bench harness from the installed package

[mattmillerai]

ELI-5

pip install comfy-cli was shipping a benchmark script that nobody can run from the CLI and that nothing in the project uses. It also made the published package advertise an optional "install the Anthropic SDK too" mode that existed only for that script. This deletes both. Git history still has the harness if anyone wants it back.

Closes #639

What this removes

• comfy_cli/bench/ — a 1,023-line run_arm_b.py with its own argparse main, 40 KB of JSON fixtures, a 3.2 KB tasks.json, and an empty out/ directory carrying both a .gitignore and a .gitkeep.
• tests/comfy_cli/bench/ (378 lines) — the harness's only consumer.
• pyproject.toml: the "comfy_cli.bench" = [ "tasks.json", "fixtures/*.json" ] package-data entry (which shipped those fixtures to every install) and the optional-dependencies.bench = [ "anthropic>=0.40" ] extra.
• uv.lock: regenerated so uv sync --locked stays green.

Why

The harness was always a one-off measurement aid, not a product feature — its own README opened by calling itself a sanctioned proxy rather than the real thing, and the arm it was meant to be compared against lives outside this repo. Meanwhile every pip install comfy-cli paid for it in package size, and the bench extra put an unused optional dependency on a published CLI's dependency surface for no user-facing benefit.

Verification that it was genuinely unreachable

I checked this empirically rather than assuming it, since the whole change rests on that premise:

• No CLI registration. No bench reference in comfy_cli/cmdline.py or comfy_cli/__main__.py, and the only console scripts are the three comfy* aliases pointing at comfy_cli.__main__:main.
• No import path. A repo-wide grep for bench across *.py, *.toml, *.yaml/*.yml, *.md, *.cfg, *.txt, *.json returns hits only inside comfy_cli/bench/ itself, inside its own tests, and on the three pyproject.toml lines this PR deletes. Nothing in docs/, README.md, DEV_README.md, AGENTS.md, .gitignore, .pre-commit-config.yaml, or codecov.yml.
• No CI consumer. None of the ten workflows under .github/workflows/ reference it.
• Suite still green after the deletion — 3705 passed, 37 skipped. Nothing else was importing it.
• Built the wheel and looked inside: 166 entries, zero containing bench, while the other package-data payloads (skills/*/SKILL.md, schemas/, cql/data/, command/generate/spec/openapi.yml) all still ship.

The one capability this genuinely removes is python -m comfy_cli.bench.run_arm_b. That is the point of the change, and the harness is one git checkout away — it was added in #490.

Checks

check	result
ruff check (0.15.15, the version CI pins)	pass
ruff format --diff (0.15.15)	268 files already formatted
pytest	3705 passed, 37 skipped
uv sync --extra dev --locked	pass

Heads-up on one local-toolchain artifact: running the repo's own dev-extra ruff (which resolves to 0.12.7) reports 17 pre-existing UP038 findings in files this PR does not touch. CI installs ruff==0.15.15, where UP038 is gone and everything passes; the table above is the CI-pinned version.

Judgment calls

• Deleted rather than relocated. The issue offered either deleting outright or moving the harness to a top-level bench/ outside the package. I deleted: the harness self-describes as a throwaway proxy, its counterpart arm is in another repo, and keeping a top-level copy while deleting its only tests (which the issue also asks for) would leave unverified dead code in the tree. Git history is the archive.
• uv.lock is regenerated, and that is required, not incidental. CI runs uv sync --extra dev --locked, which fails on any drift from pyproject.toml, so removing an extra forces a relock. The only packages dropped are anthropic and its transitive-only dependencies (annotated-types, docstring-parser, jiter, pydantic, pydantic-core, typing-inspection) — none of which anything else in the project depends on. One incidental line comes with it: exceptiongroup's dependency on typing-extensions loses its python_full_version < '3.11' marker, because collapsing the resolution fork that the SDK created makes the marker unnecessary. That is a widening, and typing-extensions>=4.7 is already an unconditional direct runtime dependency, so it changes nothing installed. I confirmed uv lock is a no-op against an unmodified main with this uv version, so none of the lock delta is toolchain drift.
• pylock.toml left alone. It never contained the SDK (the bench extra was not exported into it) and no workflow reads it. Regenerating it would have pulled in a large, unrelated dependency bump — it is already stale against uv.lock on main (e.g. gitpython 3.1.45 vs 3.1.54) — so that belongs in its own PR.

All the acceptance criteria in #639 are met; nothing was left out.

[coderabbitai]

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 50 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 3cde8e13-2c9b-4b5d-9dd0-9605a770ce94

📥 Commits

Reviewing files that changed from the base of the PR and between 5704b55 and a366bc2.

⛔ Files ignored due to path filters (1)

• uv.lock is excluded by !**/*.lock

📒 Files selected for processing (12)

• comfy_cli/bench/README.md
• comfy_cli/bench/__init__.py
• comfy_cli/bench/fixtures/edit_session_seed.json
• comfy_cli/bench/fixtures/object_info.json
• comfy_cli/bench/fixtures/txt2img_seed.json
• comfy_cli/bench/out/.gitignore
• comfy_cli/bench/out/.gitkeep
• comfy_cli/bench/run_arm_b.py
• comfy_cli/bench/tasks.json
• pyproject.toml
• tests/comfy_cli/bench/__init__.py
• tests/comfy_cli/bench/test_run_arm_b.py

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🔍 Cursor Review — Consolidated panel

Triggered by @mattmillerai.

✅ No high-signal findings.

Panel: 8/8 reviewers contributed findings.

[bigcat88]

Approving. Built the wheel both ways rather than taking the "unreachable, not shipped" premise on trust.

Before / after, from inside the artifact

	main	this branch
wheel entries	172	166
entries containing bench	6	none
Provides-Extra	bench, dev, preview	dev, preview
anthropic in METADATA	yes	no

The six that were shipping to every pip install: run_arm_b.py (44 KB) plus tasks.json and three fixtures (~36 KB of JSON). And the package data that should still ship all does — 5 SKILL.md, 37 schema files, 5 cql/data files, openapi.yml.

Unreachability, re-checked

A repo-wide case-insensitive search for bench across every tracked file (excluding .git/.venv) returns zero files on this branch. uv lock --check resolves cleanly against the regenerated lockfile. Suite green after the deletion: 3747 passed, 31 skipped; ruff check / ruff format --diff clean at the CI-pinned 0.15.15 (269 files, down from 273 — the deleted modules).

One thing you should act on: this supersedes #534

#534 (build(deps): re-sync uv.lock with the pyproject bench extra, BE-3291) exists for the sole purpose of adding anthropic>=0.40 to uv.lock so the bench extra resolves. This PR deletes that extra outright, which makes #534 not just redundant but actively wrong to land afterwards — and it'll conflict on uv.lock either way.

I'd close #534 with a pointer here once this merges. It's currently sitting in the queue as CHANGES_REQUESTED, so it isn't going anywhere on its own.

Good call being explicit that python -m comfy_cli.bench.run_arm_b is the one capability this removes, and pointing at #490 for recovery — that's exactly the framing a deletion PR needs.