Skip to content

feat(cli): add deterministic task preflight checks - #590

Open
ChinmayK0607 wants to merge 1 commit into
mainfrom
chinmay/hud-2527-embed-deterministic-checks
Open

feat(cli): add deterministic task preflight checks#590
ChinmayK0607 wants to merge 1 commit into
mainfrom
chinmay/hud-2527-embed-deterministic-checks

Conversation

@ChinmayK0607

Copy link
Copy Markdown
Contributor

What changed

  • Added live task manifest validation to hud task list, start, and grade, including deployed environment support.
  • Added hud task grade --dry-run with separate environment, task, grader, and reward-shape results plus per-phase timeouts and cleanup.
  • Added pre-upload environment/task manifest and source-fixture validation to hud sync tasks, including verifier task IDs.
  • Reused the task source/runtime resolver from hud eval, updated CLI docs, and added behavioral regression coverage. There are no UI changes.
  • Supersedes the broader standalone-check design in Add production Task lifecycle check command #531.

Why

Deterministic lifecycle failures were discovered only after task upload or rollout. The previous standalone checker duplicated execution lifecycle code and included nondeterministic agent behavior beyond this requirement. These checks now live in the existing commands that own each lifecycle stage.

Impact

Environment and task authors can catch invalid sources, missing deployed tasks, startup failures, broken graders, and malformed rewards before syncing or running agent evaluations. CI can use the existing command exit status without adopting a separate check protocol.

Validation

  • uv run pytest -q (1042 passed, 9 deselected)
  • uv run ruff check .
  • uv run ruff format --check .
  • uv run --extra dev --extra train ty check

Made with Cursor

Embed environment, task, grader, and reward validation in existing task and sync commands so broken definitions fail before rollout or upload.

Co-authored-by: Cursor <cursoragent@cursor.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 49dfac2. Configure here.

Comment thread hud/cli/task.py
selected = matches[0]
placement = SubprocessRuntime(_spawn_target(source or "."))(selected)
return selected.id, args or selected.args, placement
selected = _resolution_or_exit(lambda: select_local_task(task, source or ".", args))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Task prefix ignored for local source

Medium Severity

_task_id strips a leading task/ for --env and --url placements, but the local --source path passes the raw argument into select_local_task. The same task/… form documented for deployed checks therefore fails to match local slugs or ids and exits as unresolved.

Additional Locations (2)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 49dfac2. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant