Skip to content

feat(installer): public Python API for the bundled catalogue and skill folders - #6

Merged
manykarim merged 3 commits into
mainfrom
change/public-bundle-api
Nov 12, 2026
Merged

feat(installer): public Python API for the bundled catalogue and skill folders#6
manykarim merged 3 commits into
mainfrom
change/public-bundle-api

Conversation

@manykarim

@manykarim manykarim commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator

Closes #5.

Summary

Adds a public Python API to robotframework-plugins, so tools such as rf-mcp can depend on the package and serve the bundled skills without importing private modules.

  • robotframework_plugins.bundle:
    • CATALOG_FORMAT (1);
    • catalog(): the bundled plugins, active and deprecated, as a new list on each call;
    • skills_dirs(): {plugin: Path} of each active, non-bundle plugin's skills/ folder, rf-core first.
  • Real paths in every install: skills_dirs() paths are real folders for the life of the process. When the package is imported from a zip, the bundle is copied once to a temporary folder, removed at exit.
  • bundle.fetch(ref="stable"), opt-in: the same catalog() / skills_dirs() for a branch, tag or commit of this repository.
    • The ref is resolved to a commit, downloaded once from GitHub and cached by commit; a cached commit needs no network.
    • Before use it's checked: only plain files and folders, no paths outside the snapshot, valid catalog.yaml and plugin.json.
    • Any failure raises FetchError, and nothing in a snapshot is run.
    • It's the only part of the package that uses the network, and the installer's commands never import it (a test checks that).
  • One catalogue builder (_catalog.py) for the wheel's build hook and for fetch(). catalog.json gains "format": 1, and each entry gains its robotframework range, which the hook used to drop.
  • Package and docs:
    • py.typed;
    • a "Python API" section in the installer README with the compatibility promise and what choosing main means;
    • a changelog entry under 2026.11.0, which is unreleased.
  • CI: the Test job gets astral-sh/setup-uv. Without uv, the wheel tests, including the existing test_wheel_contains_bundled_plugins, were skipped in CI.

The OpenSpec change public-bundle-api (proposal, spec installer-python-api, design, tasks) is in the first commit.

Issue #5 acceptance criteria

  • robotframework_plugins.bundle with CATALOG_FORMAT, catalog() and skills_dirs() in __all__
  • _assets/catalog.json carries "format": 1
  • Tests on a built wheel: test_api_from_an_installed_wheel builds the wheel, installs it into a fresh venv and checks:
    • skills_dirs() returns existing folders under site-packages, each with */SKILL.md;
    • the names match the active content plugins of catalog();
    • bundles are excluded.
  • Installer README "Python API" section with the compatibility promise
  • _assets.py unchanged for the adapters; bundle wraps it
  • Released to PyPI: after merge, by pushing robotframework-plugins-v2026.11.0 in November 2026 and approving the pypi environment

Test plan

  • pytest tests --ignore=tests/eval: 819 passed
  • pytest tests/installer: 274 passed, including the slow wheel and zip-import tests; the new tests also pass on Python 3.10
  • pytest tests/installer/test_bundle_fetch.py -m network against GitHub: fetch("main") returned commit 3756236 with rf-core, rf-api, rf-desktop, rf-mobile and rf-web; fetch() (stable) raised "ref 'stable' not found"
  • check-drift, build-catalog.py --check, docs voice
  • CI on this PR, including the wheel tests on Linux and Windows

🤖 Generated with Claude Code

https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP

manykarim and others added 2 commits November 12, 2026 19:11
@manykarim @claude
docs(openspec): public-bundle-api change (issue #5)
883d746 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
@manykarim @claude
feat(installer): public Python API robotframework_plugins.bundle (#5)
073f5dd 
- bundle.catalog(), bundle.skills_dirs(), bundle.CATALOG_FORMAT over the
  bundled plugins; real folders also from a zip import
- bundle.fetch(ref="stable"): opt-in snapshot of a repository ref, pinned to
  a commit, cached, checked (safe extraction, valid manifests), FetchError on
  any failure; the only network use, never from the installer's commands
- one catalogue builder (_catalog.py) for the wheel's build hook and fetch();
  catalog.json gains "format": 1 and each entry its robotframework range
- py.typed; README "Python API" with the compatibility promise; changelog
- CI test job gets uv so the wheel tests run

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
@manykarim manykarim mentioned this pull request Nov 12, 2026
Closed
6 tasks
@github-actions

github-actions Bot commented Nov 12, 2026 •
edited
Loading

Copy link
Copy Markdown

Skill Evaluation: rf-mobile (narrow, treatment, N=3, deepseek/deepseek-v4.1-flash)

Model: deepseek/deepseek-v4.1-flash (provider openrouter, turn-limit factor 1.5)

OpenRouter spend

USD
Key spend during this job (includes concurrent jobs) 0.22
Harness estimate (OpenRouter list prices) 0.09
Credit remaining on the key 24.493371752

Gate

gate: PASS (1 task(s) checked)

rf-skill-eval report

Runs: 3 · pass: 3 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.09

Pass rate = runs whose gate result is pass / runs attempted. Deltas are treatment − baseline over outcome checks only.

Tasks

Task Skill Tier Arm Runs Pass rate Outcome pass Incomplete Input tok Output tok Turns Duration s Cost $ Flaky
narrow-appium-login-01 rf-appium narrow treatment 3 100% 100% 0 487235 8805 23.0 64.0 0.0290 (est.) no
narrow-appium-login-01 rf-appium narrow baseline (stored) 6 100% 100% 0 282607 6601 18.3 69.0 0.0206 (est.) –
narrow-appium-login-01 rf-appium narrow Δ T−B +0 pp +204628 +2204 +4.7 -5.1 +0.0084

Skills

Skill Tasks Treatment outcome pass Baseline outcome pass Δ outcome pass Δ input tok Δ turns Δ cost $
rf-appium 1 100% 100% +0 pp +204628 +4.7 +0.0084

Skipped / errored checks

None.

Incomplete runs

None.

Process checks (treatment arm, not in deltas)

  • narrow-appium-login-01 loaded_rf_appium_skill: 3/3

Adversarial tier (reported only, never gates)

None.

@github-actions

github-actions Bot commented Nov 12, 2026 •
edited
Loading

Copy link
Copy Markdown

Skill Evaluation: rf-web (narrow, treatment, N=3, deepseek/deepseek-v4.1-flash)

Model: deepseek/deepseek-v4.1-flash (provider openrouter, turn-limit factor 1.5)

OpenRouter spend

USD
Key spend during this job (includes concurrent jobs) 0.62
Harness estimate (OpenRouter list prices) 0.13
Credit remaining on the key 24.089635867

Gate

gate: PASS (2 task(s) checked)

rf-skill-eval report

Runs: 6 · pass: 6 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.13

Pass rate = runs whose gate result is pass / runs attempted. Deltas are treatment − baseline over outcome checks only.

Tasks

Task Skill Tier Arm Runs Pass rate Outcome pass Incomplete Input tok Output tok Turns Duration s Cost $ Flaky
narrow-browser-login-01 rf-browser narrow treatment 3 100% 100% 0 358113 4595 21.0 116.7 0.0240 (est.) no
narrow-browser-login-01 rf-browser narrow baseline (stored) 6 100% 100% 0 188544 3649 15.2 45.4 0.0141 (est.) –
narrow-browser-login-01 rf-browser narrow Δ T−B +0 pp +169569 +946 +5.8 +71.3 +0.0099
narrow-selenium-login-01 rf-selenium narrow treatment 3 100% 100% 0 383762 7426 23.3 124.2 0.0209 (est.) no
narrow-selenium-login-01 rf-selenium narrow baseline (stored) 6 100% 100% 0 109915 2152 11.7 24.2 0.0087 (est.) –
narrow-selenium-login-01 rf-selenium narrow Δ T−B +0 pp +273847 +5274 +11.7 +100.0 +0.0122

Skills

Skill Tasks Treatment outcome pass Baseline outcome pass Δ outcome pass Δ input tok Δ turns Δ cost $
rf-browser 1 100% 100% +0 pp +169569 +5.8 +0.0099
rf-selenium 1 100% 100% +0 pp +273847 +11.7 +0.0122

Skipped / errored checks

None.

Incomplete runs

None.

Process checks (treatment arm, not in deltas)

  • narrow-browser-login-01 loaded_rf_browser_skill: 3/3
  • narrow-selenium-login-01 loaded_rf_selenium_skill: 3/3

Adversarial tier (reported only, never gates)

None.

@github-actions

github-actions Bot commented Nov 12, 2026 •
edited
Loading

Copy link
Copy Markdown

Skill Evaluation: rf-api (narrow, treatment, N=3, deepseek/deepseek-v4.1-flash)

Model: deepseek/deepseek-v4.1-flash (provider openrouter, turn-limit factor 1.5)

OpenRouter spend

USD
Key spend during this job (includes concurrent jobs) 0.55
Harness estimate (OpenRouter list prices) 0.22
Credit remaining on the key 24.167802251

Gate

gate: PASS (2 task(s) checked)

rf-skill-eval report

Runs: 6 · pass: 6 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.22 · turn-limit stops: 3

Pass rate = runs whose gate result is pass / runs attempted. Deltas are treatment − baseline over outcome checks only.

Tasks

Task Skill Tier Arm Runs Pass rate Outcome pass Incomplete Input tok Output tok Turns Duration s Cost $ Flaky
narrow-requests-users-01 rf-requests narrow treatment 3 100% 100% 0 559901 12952 22.0 (1 at limit) 123.9 0.0430 (est.) no
narrow-requests-users-01 rf-requests narrow baseline (stored) 6 100% 100% 0 132557 2867 11.0 28.3 0.0099 (est.) –
narrow-requests-users-01 rf-requests narrow Δ T−B +0 pp +427344 +10085 +11.0 +95.6 +0.0331
narrow-restinstance-schema-01 rf-restinstance narrow treatment 3 100% 100% 0 473664 7225 21.0 (2 at limit) 66.0 0.0302 (est.) no
narrow-restinstance-schema-01 rf-restinstance narrow baseline (stored) 6 100% 100% 0 232009 3515 17.0 38.7 0.0117 (est.) –
narrow-restinstance-schema-01 rf-restinstance narrow Δ T−B +0 pp +241655 +3710 +4.0 +27.3 +0.0185

Skills

Skill Tasks Treatment outcome pass Baseline outcome pass Δ outcome pass Δ input tok Δ turns Δ cost $
rf-requests 1 100% 100% +0 pp +427344 +11.0 +0.0331
rf-restinstance 1 100% 100% +0 pp +241655 +4.0 +0.0185

Skipped / errored checks

None.

Incomplete runs

None.

Process checks (treatment arm, not in deltas)

  • narrow-requests-users-01 loaded_rf_requests_skill: 3/3
  • narrow-restinstance-schema-01 loaded_rf_restinstance_skill: 3/3

Adversarial tier (reported only, never gates)

None.

@github-actions

github-actions Bot commented Nov 12, 2026 •
edited
Loading

Copy link
Copy Markdown

Skill Evaluation: rf-desktop (narrow, treatment, N=3, deepseek/deepseek-v4.1-flash)

Model: deepseek/deepseek-v4.1-flash (provider openrouter, turn-limit factor 1.5)

OpenRouter spend

USD
Key spend during this job (includes concurrent jobs) 0.38
Harness estimate (OpenRouter list prices) 0.13
Credit remaining on the key 24.328731714

Gate

gate: PASS (1 task(s) checked)

rf-skill-eval report

Runs: 3 · pass: 3 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.13 · turn-limit stops: 3

Pass rate = runs whose gate result is pass / runs attempted. Deltas are treatment − baseline over outcome checks only.

Tasks

Task Skill Tier Arm Runs Pass rate Outcome pass Incomplete Input tok Output tok Turns Duration s Cost $ Flaky
narrow-platynui-calculator-01 rf-platynui narrow treatment 3 100% 100% 0 714091 14613 22.0 (3 at limit) 122.6 0.0449 (est.) no
narrow-platynui-calculator-01 rf-platynui narrow baseline (stored) 6 50% 50% 0 475578 8534 20.7 (4 at limit) 76.5 0.0226 (est.) –
narrow-platynui-calculator-01 rf-platynui narrow Δ T−B +50 pp +238513 +6079 +1.3 +46.1 +0.0223

Skills

Skill Tasks Treatment outcome pass Baseline outcome pass Δ outcome pass Δ input tok Δ turns Δ cost $
rf-platynui 1 100% 50% +50 pp +238513 +1.3 +0.0223

Skipped / errored checks

None.

Incomplete runs

None.

Process checks (treatment arm, not in deltas)

  • narrow-platynui-calculator-01 loaded_rf_platynui_skill: 3/3

Adversarial tier (reported only, never gates)

None.

@github-actions

github-actions Bot commented Nov 12, 2026 •
edited
Loading

Copy link
Copy Markdown

Skill Evaluation: rf-core (narrow, treatment, N=3, deepseek/deepseek-v4.1-flash)

Model: deepseek/deepseek-v4.1-flash (provider openrouter, turn-limit factor 1.5)

OpenRouter spend

USD
Key spend during this job (includes concurrent jobs) 1.38
Harness estimate (OpenRouter list prices) 0.88
Credit remaining on the key 23.337097118

Gate

gate: PASS (12 task(s) checked)

rf-skill-eval report

Runs: 36 · pass: 36 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.88 · turn-limit stops: 11

Pass rate = runs whose gate result is pass / runs attempted. Deltas are treatment − baseline over outcome checks only.

Tasks

Task Skill Tier Arm Runs Pass rate Outcome pass Incomplete Input tok Output tok Turns Duration s Cost $ Flaky
narrow-language-bdd-01 rf-language narrow treatment 3 100% 100% 0 272154 3703 18.3 40.7 0.0217 (est.) no
narrow-language-bdd-01 rf-language narrow baseline (stored) 6 100% 100% 0 134237 2629 12.3 18.5 0.0090 (est.) –
narrow-language-bdd-01 rf-language narrow Δ T−B +0 pp +137917 +1075 +6.0 +22.2 +0.0127
narrow-language-data-driven-01 rf-language narrow treatment 3 100% 100% 0 533337 8259 25.3 (1 at limit) 77.5 0.0220 (est.) no
narrow-language-data-driven-01 rf-language narrow baseline (stored) 6 83% 83% 0 206359 5541 19.2 40.3 0.0134 (est.) –
narrow-language-data-driven-01 rf-language narrow Δ T−B +17 pp +326979 +2719 +6.2 +37.2 +0.0086
narrow-language-embedded-01 rf-language narrow treatment 3 100% 100% 0 322057 4667 17.3 38.8 0.0156 (est.) no
narrow-language-embedded-01 rf-language narrow baseline (stored) 6 100% 100% 0 127580 3524 12.2 21.9 0.0130 (est.) –
narrow-language-embedded-01 rf-language narrow Δ T−B +0 pp +194477 +1143 +5.2 +16.9 +0.0026
narrow-language-env-varfiles-01 rf-language narrow treatment 3 100% 100% 0 761985 17431 22.0 (3 at limit) 129.4 0.0423 (est.) no
narrow-language-env-varfiles-01 rf-language narrow baseline (stored) 6 100% 100% 0 274618 6895 21.7 42.2 0.0193 (est.) –
narrow-language-env-varfiles-01 rf-language narrow Δ T−B +0 pp +487367 +10535 +0.3 +87.2 +0.0230
narrow-language-suite-setup-tags-01 rf-language narrow treatment 3 100% 100% 0 357425 4048 20.3 53.9 0.0169 (est.) no
narrow-language-suite-setup-tags-01 rf-language narrow baseline (stored) 6 83% 83% 0 221498 4474 18.3 (1 at limit) 33.1 0.0146 (est.) –
narrow-language-suite-setup-tags-01 rf-language narrow Δ T−B +17 pp +135927 -426 +2.0 +20.8 +0.0023
narrow-non-rf-control-01 plugin narrow treatment 3 100% 100% 0 32462 336 2.3 3.9 0.0046 (est.) no
narrow-non-rf-control-01 plugin narrow baseline (stored) 6 100% 100% 0 28024 195 2.0 2.7 0.0046 (est.) –
narrow-non-rf-control-01 plugin narrow Δ T−B +0 pp +4437 +141 +0.3 +1.2 -0.0000
narrow-python-library-flaky-listener-01 rf-python-library narrow treatment 3 100% 100% 0 577514 14748 21.7 (2 at limit) 97.0 0.0499 (est.) no
narrow-python-library-flaky-listener-01 rf-python-library narrow baseline (stored) 6 83% 83% 0 433974 10412 20.8 (3 at limit) 76.3 0.0264 (est.) –
narrow-python-library-flaky-listener-01 rf-python-library narrow Δ T−B +17 pp +143540 +4336 +0.8 +20.7 +0.0235
narrow-python-library-inventory-scope-01 rf-python-library narrow treatment 3 100% 100% 0 405813 7274 19.7 (1 at limit) 115.8 0.0224 (est.) no
narrow-python-library-inventory-scope-01 rf-python-library narrow baseline (stored) 6 100% 100% 0 334542 9598 19.5 (3 at limit) 62.5 0.0215 (est.) –
narrow-python-library-inventory-scope-01 rf-python-library narrow Δ T−B +0 pp +71272 -2324 +0.2 +53.3 +0.0009
narrow-python-library-mode-literal-01 rf-python-library narrow treatment 3 100% 100% 0 490166 12292 19.7 (2 at limit) 102.8 0.0393 (est.) no
narrow-python-library-mode-literal-01 rf-python-library narrow baseline (stored) 6 100% 100% 0 330956 10259 19.0 (1 at limit) 74.8 0.0247 (est.) –
narrow-python-library-mode-literal-01 rf-python-library narrow Δ T−B +0 pp +159210 +2032 +0.7 +28.0 +0.0146
narrow-rf-injection-positive-01 plugin narrow treatment 3 100% 100% 0 264275 4329 16.7 32.9 0.0243 (est.) no
narrow-rf-injection-positive-01 plugin narrow baseline (stored) 6 100% 100% 0 358063 6274 18.7 (1 at limit) 47.0 0.0260 (est.) –
narrow-rf-injection-positive-01 plugin narrow Δ T−B +0 pp -93788 -1945 -2.0 -14.1 -0.0017
narrow-rf-mcp-execute-01 plugin narrow treatment 3 100% 100% 0 457875 3623 18.7 (2 at limit) 56.2 0.0204 (est.) no
narrow-rf-mcp-execute-01 plugin narrow baseline (stored) 6 100% 100% 0 151802 1465 8.5 14.1 0.0168 (est.) –
narrow-rf-mcp-execute-01 plugin narrow Δ T−B +0 pp +306073 +2158 +10.2 +42.2 +0.0036
narrow-setup-requests-01 rf-setup narrow treatment 3 100% 100% 0 207210 4427 16.0 40.5 0.0131 (est.) no
narrow-setup-requests-01 rf-setup narrow baseline (stored) 6 100% 100% 0 140285 2194 10.3 20.2 0.0097 (est.) –
narrow-setup-requests-01 rf-setup narrow Δ T−B +0 pp +66925 +2233 +5.7 +20.3 +0.0034

Skills

Skill Tasks Treatment outcome pass Baseline outcome pass Δ outcome pass Δ input tok Δ turns Δ cost $
rf-language 5 100% 93% +7 pp +256533 +3.9 +0.0098
rf-python-library 3 100% 94% +6 pp +124674 +0.6 +0.0130
rf-setup 1 100% 100% +0 pp +66925 +5.7 +0.0034

Skipped / errored checks

None.

Incomplete runs

None.

Process checks (treatment arm, not in deltas)

  • narrow-language-bdd-01 loaded_rf_language_skill: 3/3
  • narrow-language-data-driven-01 loaded_rf_language_skill: 3/3
  • narrow-language-embedded-01 loaded_rf_language_skill: 3/3
  • narrow-language-env-varfiles-01 loaded_rf_language_skill: 3/3
  • narrow-language-suite-setup-tags-01 loaded_rf_language_skill: 3/3
  • narrow-non-rf-control-01 bounded_tool_calls: 3/3
  • narrow-non-rf-control-01 no_libdoc_invocation: 3/3
  • narrow-non-rf-control-01 no_rf_mcp_calls: 3/3
  • narrow-python-library-flaky-listener-01 loaded_rf_python_library_skill: 3/3
  • narrow-python-library-inventory-scope-01 loaded_rf_python_library_skill: 3/3
  • narrow-python-library-mode-literal-01 loaded_rf_python_library_skill: 3/3
  • narrow-rf-injection-positive-01 leveraged_rf_lookup: 3/3
  • narrow-rf-mcp-execute-01 rf_mcp_before_write: 3/3
  • narrow-rf-mcp-execute-01 rf_mcp_called_at_least_twice: 3/3
  • narrow-rf-mcp-execute-01 rf_mcp_had_successful_result: 3/3
  • narrow-setup-requests-01 used_uv_add: 3/3

Adversarial tier (reported only, never gates)

None.

@manykarim @claude
docs(openspec): public-bundle-api 5.1 verified in CI
1b26073 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
@manykarim
manykarim merged commit aa7c579 into main Nov 12, 2026
35 of 36 checks passed
@manykarim
manykarim deleted the change/public-bundle-api branch November 12, 2026 21:54
@manykarim manykarim mentioned this pull request Nov 13, 2026
Merged
4 of 5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Public Python API for the bundled catalogue and skill folders (for rf-mcp)

1 participant

@manykarim