Skip to content

Support nested task dataset layouts when pushing to Hub - #97

Merged
adithya-s-k merged 1 commit into
huggingface:mainfrom
karatarassul4-max:fix/nested-task-dataset-layout
Sep 15, 2026
Merged

adithya-s-k merged 1 commit into
huggingface:mainfrom
karatarassul4-max:fix/nested-task-dataset-layout

Conversation

@karatarassul4-max

Copy link
Copy Markdown
Contributor

Fixes #96

Summary

  • reuse _list_task_dirs() in push_to_hub() so Hub staging and image preparation share the same discovery logic
  • support both flat task layouts and the canonical tasks/<id>/task.toml layout
  • keep hidden directories excluded and preserve deterministic ordering
  • reject duplicate task IDs across flat and nested layouts instead of silently choosing one copy
  • add regression coverage for nested discovery, mixed layouts, duplicate IDs, and nested push_to_hub() metadata discovery

Validation

  • uv run pytest -q tests/registry/test_integration.py tests/test_hub_metadata_read.py → 34 passed
  • uv run ruff check src/repo2rlenv/registry/integration.py src/repo2rlenv/hub.py tests/registry/test_integration.py tests/test_hub_metadata_read.py → passed
  • uv run ruff format --check src/repo2rlenv/registry/integration.py src/repo2rlenv/hub.py tests/registry/test_integration.py tests/test_hub_metadata_read.py → 4 files already formatted
  • git diff --check → clean
  • full uv run pytest -q on Windows: 712 passed, 6 skipped, 7 platform-specific failures related to executable bits, path separators, and Windows local-path parsing; none are in files changed by this PR

@KNambiarDJsc KNambiarDJsc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed and tested 92b5138:

  • CI-equivalent run on Linux: ruff is clean, and the full suite passes on py3.12 and py3.14 (719 passed).
  • End-to-end on a real nested dataset. I copied three tasks from the published AdithyaSK/repo2rlenv-pr-diff dataset in its tasks/<id>/ layout. I also added a root README.md and registry.json, as huggingface-cli download leaves them. Then I ran push_to_hub with HfApi mocked.
    • On main it raises RuntimeError: no Harbor tasks found.
    • With this PR, all 3 tasks are staged under tasks/, and the generated manifest.json has the same 3 rows.
    • Nothing ends up double-nested as tasks/tasks/.
    • The push runs through image preparation without errors; these tasks ship environment/Dockerfiles.

LGTM.

One non-blocking nit: hub.py now imports the private _list_task_dirs from registry.integration. If both modules are meant to share discovery, dropping the underscore would make that dependency explicit.

@adithya-s-k
adithya-s-k merged commit 79426ba into huggingface:main Sep 15, 2026
5 checks passed
@adithya-s-k

Copy link
Copy Markdown
Collaborator

thanks @karatarassul4-max, merged! appreciate the nested task layout support and the duplicate ID checks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Accept nested tasks/ dataset layouts when pushing to Hugging Face Hub

3 participants