Skip to content

feat(ctl): download marketplace collections per-schema into a named subdirectory#1057

Merged
lancamat1 merged 3 commits into
stablefrom
unleashed-nutmeg
Jun 5, 2026
Merged

feat(ctl): download marketplace collections per-schema into a named subdirectory#1057
lancamat1 merged 3 commits into
stablefrom
unleashed-nutmeg

Conversation

@minitriga
Copy link
Copy Markdown
Contributor

@minitriga minitriga commented Jun 3, 2026

Why

infrahubctl marketplace get <collection> --collection dumped every schema flat into the output directory with the version baked into each filename:

schemas/
├── infrahub-dcim-1.0.0.yml
├── infrahub-ipam-1.0.0.yml
├── infrahub-location-1.0.0.yml
└── infrahub-organization-1.0.0.yml

Two problems:

  • Stale versions accumulate. Re-running after a member bumps to 1.1.0 leaves the old …-1.0.0.yml sitting next to the new file — globbing schemas/*.yml then loads conflicting versions.
  • Inconsistent with single-schema downloads, which already write <name>.yml with no version.

What changed

  • Collections are now resolved via the collection metadata endpoint (/api/v1/collections/{ns}/{name}), which lists each member schema and its latest published version — instead of downloading a ZIP archive.
  • Each member is downloaded individually through the existing _download_schema path, so naming, versioning, and error handling are identical to single-schema downloads.
  • Members are written to <output_dir>/<collection name>/<schema name>.yml:
schemas/
└── base-schemas/
    ├── dcim.yml
    ├── ipam.yml
    ├── location.yml
    └── organization.yml
  • Per-file output is now Downloaded schema infrahub/dcim v1.0.0 -> …, consistent with single-schema downloads.
  • A member entry missing namespace/name is skipped rather than aborting the whole download.

Things worth knowing

  • No version pinning was lost. The metadata exposes latest_version (not a frozen pin), so the old ZIP wasn't a frozen snapshot either — downloading each member at its latest_version reproduces exactly what the ZIP contained.
  • Partial failure is now possible. One request became N+1; if a member fails mid-run, earlier members are already on disk and the command exits non-zero. (Happy to add temp-dir-then-move atomicity if reviewers want it.)

How to review

  1. infrahub_sdk/ctl/marketplace.py_collection_url now points at the metadata endpoint; _download_collection lists members and delegates to _download_schema; _download_schema gained a needs_separator flag for multi-doc stdout streaming.
  2. tests/unit/ctl/test_marketplace_app.py — collection tests rewritten to mock the metadata endpoint + per-member version downloads.

Tests: 25 marketplace unit tests pass; format, lint-code (ruff/ty/mypy) clean; generated docs unchanged.

🤖 Generated with Claude Code


Summary by cubic

infrahubctl marketplace get --collection now downloads each member schema individually and saves them in <output_dir>/<collection>/<schema>.yml (using <collection>/<namespace>/<name>.yml only when names collide). This removes versioned filenames, prevents stale files, and matches single-schema naming.

  • New Features

    • Use collection metadata and download each member via _download_schema for consistent handling and logs.
    • Save under a collection-named subdirectory; disambiguate duplicate names across namespaces with <collection>/<namespace>/<name>.yml.
    • In --stdout mode, insert --- between members when needed; per-file messages show Downloaded schema <ns>/<name> v<semver>.
    • Skip members missing namespace/name with a warning and ignore malformed items; the final count reflects schemas actually downloaded.
  • Migration

    • Update scripts/globs from <output_dir>/*.yml to <output_dir>/<collection>/*.yml; when collisions exist, also handle <output_dir>/<collection>/*/*.yml. Re-downloads now overwrite cleanly.

Written for commit 5ec7a95. Summary will update on new commits.

Review in cubic

…ubdirectory

`marketplace get --collection` previously fetched a ZIP and dumped every
schema flat into the output directory with the version baked into each
filename (e.g. `infrahub-dcim-1.0.0.yml`). Re-downloading accumulated stale
versions side by side, and the naming was inconsistent with single-schema
downloads (`dcim.yml`).

Collections are now resolved via the collection metadata endpoint, which
lists each member schema and its latest published version. Each member is
downloaded individually through the existing `_download_schema` path, so
naming, versioning, and error handling match single-schema downloads.
Members land in `<output_dir>/<collection name>/<schema name>.yml`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@minitriga minitriga requested a review from a team as a code owner June 3, 2026 15:10
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@cloudflare-workers-and-pages
Copy link
Copy Markdown

cloudflare-workers-and-pages Bot commented Jun 3, 2026

Deploying infrahub-sdk-python with  Cloudflare Pages  Cloudflare Pages

Latest commit: 5ec7a95
Status: ✅  Deploy successful!
Preview URL: https://f3696379.infrahub-sdk-python.pages.dev
Branch Preview URL: https://unleashed-nutmeg.infrahub-sdk-python.pages.dev

View logs

@codecov
Copy link
Copy Markdown

codecov Bot commented Jun 3, 2026

Codecov Report

❌ Patch coverage is 95.65217% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
infrahub_sdk/ctl/marketplace.py 95.65% 1 Missing ⚠️
@@            Coverage Diff             @@
##           stable    #1057      +/-   ##
==========================================
+ Coverage   81.71%   81.73%   +0.01%     
==========================================
  Files         135      135              
  Lines       11625    11626       +1     
  Branches     1759     1757       -2     
==========================================
+ Hits         9499     9502       +3     
+ Misses       1575     1574       -1     
+ Partials      551      550       -1     
Flag Coverage Δ
integration-tests 41.55% <0.00%> (-0.03%) ⬇️
python-3.10 55.12% <95.65%> (+0.03%) ⬆️
python-3.11 55.12% <95.65%> (+0.03%) ⬆️
python-3.12 55.10% <95.65%> (+<0.01%) ⬆️
python-3.13 55.10% <95.65%> (+<0.01%) ⬆️
python-3.14 55.12% <95.65%> (+0.02%) ⬆️
python-filler-3.12 22.38% <0.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
infrahub_sdk/ctl/marketplace.py 92.25% <95.65%> (+1.34%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Copy Markdown

@cubic-dev-ai cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 3 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread infrahub_sdk/ctl/marketplace.py Outdated
Comment thread infrahub_sdk/ctl/marketplace.py Outdated
…e collisions

Address review findings on the per-schema collection download:

- The summary now counts schemas actually downloaded instead of every
  member entry, so skipped members (missing namespace/name) no longer
  inflate the count; skipped members also emit a warning instead of
  disappearing silently.
- Members sharing a schema name across namespaces are disambiguated
  into <collection>/<namespace>/<name>.yml instead of silently
  overwriting each other.
- The stdout `---` separator is tied to emitted documents rather than
  the raw member index.
- Collection metadata items with unexpected shapes are filtered out
  instead of raising AttributeError.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown
Contributor

@lancamat1 lancamat1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM ✅

Clean change — collection members now download per-schema into <output_dir>/<collection>/<schema>.yml, dropping the version suffix so re-downloads overwrite instead of accumulating stale files. Nice reuse of the existing _download_schema path so versioning, 404 handling, and stdout streaming stay consistent. The needs_separator logic for multi-doc stdout is correct (leading-only ---, guarded against doubling), and malformed/identity-less members degrade gracefully. Lint clean, 26 marketplace unit tests pass.

No blockers. A few non-critical follow-ups worth keeping in mind (not gating this):

  • namespace/name come from the server and are used as path components — worth a guard against ..// at some point (single-schema path has the same pre-existing exposure).
  • On a member fetch failure mid-run, the "run without --version" hint is misleading since the version came from collection metadata, not the user.
  • Partial failure leaves earlier members on disk + exits non-zero; fine as-is, but a changelog note would help.

🤖 Generated with Claude Code

@lancamat1 lancamat1 merged commit 97b518b into stable Jun 5, 2026
21 checks passed
@lancamat1 lancamat1 deleted the unleashed-nutmeg branch June 5, 2026 14:12
@minitriga minitriga mentioned this pull request Jun 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants