Skip to content

feat(ci): allow-unsupported input for the repo-eval workflow - #156

Merged
napetrov merged 9 commits into
intel:mainfrom
napetrov:repo-eval-allow-unsupported
Oct 23, 2026
Merged

napetrov merged 9 commits into
intel:mainfrom
napetrov:repo-eval-allow-unsupported

Conversation

@napetrov

Copy link
Copy Markdown
Contributor

Summary

 tasks-check job: repo tasks check
   flags=()
   if warm: flags+=(--warm)
+  if allow-unsupported: flags+=(--allow-unsupported)

New boolean input allow-unsupported (default false). A task whose task.json host block (D3, #112) the runner misses passes its job; broken tasks and failed host probes still fail (CLI semantics unchanged). Caller: oneDAL #3818 fix_cel_thread_race (needs 4+ Intel CPUs; hosted runners may be AMD).

Part of #41.

Evidence

  • Before: no way to pass --allow-unsupported from a caller; an unsupported task fails its job, so oneDAL excludes the task.
    After: allow-unsupported: true adds the flag; pytest tests/test_repo_eval_task_selection.py 37 passed.

Merge Danger

Door: two-way

Blast Radius: none (opt-in input)

🤖 Generated with Claude Code

Passes --allow-unsupported to repo tasks check, so a task whose task.json
host requirements the runner misses passes its job. Off by default.

Part of intel#41.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@codecov-commenter

codecov-commenter commented Oct 22, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.77%. Comparing base (92d92d3) to head (6861cee).
⚠️ Report is 1 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #156      +/-   ##
==========================================
- Coverage   91.78%   91.77%   -0.02%     
==========================================
  Files         223      223              
  Lines       27623    27623              
  Branches     4549     4549              
==========================================
- Hits        25354    25350       -4     
- Misses       1606     1609       +3     
- Partials      663      664       +1     
Files with missing lines Coverage Δ
dokimi/dokimi/commands/repo_run.py 92.44% <ø> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 7d5e00a. The workflow change itself is right and small. The PR also adds dokimi/uv.lock (6,890 lines, 1.4 MB), which breaks CI. One thing to fix before merge.

What I ran, on a Linux review host at 7d5e00a:

  • make ci-local: 23 passed, 3 failed, 6 skipped. One failure is the PR's (the large-file hook). The other two do not come from this diff: pip-audit reports virtualenv 21.14.7 (CVE-2026-113722) from the existing locks (fixed on main by #152), and test_unbalanced_quote_in_hook_completes_quickly took 1.002 s against a 1 s limit (6,714 other tests passed).
  • zizmor on repo-eval.yml: no findings.

dokimi/uv.lock should not be in this PR. It is unrelated to the allow-unsupported input, and it fails CI as it stands.

  • check-added-large-files --enforce-all (.pre-commit-config.yaml:41) fails: dokimi/uv.lock (1377 KB) exceeds 500 KB.
  • The license-header check covers ** and ignores only the three requirements*.lock.txt files (.github/.licenserc.yaml:45-47). uv.lock starts with version = 1 and has no SPDX header: GitHub's Static checks job fails on it ("the following files don't have a valid license header: dokimi/uv.lock").
  • The repo's locks are the hashed requirements*.lock.txt files, which tests/test_dependency_locks.py and Renovate (renovate.json5:54) cover. Nothing generates, checks or installs from a uv.lock, so it would drift unnoticed.

Suggestion: drop the file from the PR (git rm dokimi/uv.lock). If a uv lock is wanted, do it in its own PR with the header, the ignore entries and a check.

Checked, no issues

  • Step script: flags=() with "${flags[@]}" expands correctly. The input arrives through env, as the string true or false, so [ "$ALLOW_UNSUPPORTED" = true ] works. Nothing is interpolated into run.
  • Behaviour claims: repo_run.py:152-159 lets only unsupported_by == "host" pass. Broken tasks and host-probe results still exit 1, as the changelog and docs say.
  • Default: the input is under workflow_call only, defaults to false, so existing callers are unchanged.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Review at 7d5e00a: the one item (dokimi/uv.lock in the PR) is removed in f127e78; the PR diff against main is now the workflow input, its docs line and the changelog fragment only. The other two make ci-local failures were not from this diff (pip-audit is #152, the 1 s timing test is flaky).

Lift the step's script, resolve its env from the inputs per case and run it
with a stub dokimi; each warm/allow-unsupported pair must reach the CLI as given.
Dropping the --allow-unsupported line, or wiring it to inputs.warm, now fails.

Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
@xaleryb

xaleryb commented Oct 23, 2026

Copy link
Copy Markdown
Contributor
  • Ran: the repo tasks check step script from the head, against a toy repo in real docker, with each warm/allow-unsupported pair.
  • warm: true + allow-unsupported: true, on a task whose host the runner misses (null passes) -> rc 1, BROKEN - null: the untouched workspace passes the gate, the same as with allow-unsupported false -> expected: the job passes, as the input description, the contract doc and the changelog say. Under --warm the host is not judged, so the only verdict the flag lets through (unsupported/host) never happens. The caller named in the PR sets warm: true.
  • Commit we pushed: 863109f adds a test that runs the step with a stub dokimi for all four warm/allow pairs. Before it, no test failed when the flag was dropped or wired to inputs.warm.
  • Left for the author: either judge CPU count and vendor under --warm when --allow-unsupported is given, or reject that pair in the step and document it.

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. uv.lock is gone, the diff is now the workflow input, its docs line, the changelog fragment and a test, and the test pins what the step forwards. Two nits inline.

@xaleryb's question on the warm: true + allow-unsupported: true pair waits for the author. New evidence: with a 2-CPU AMD runner and a task needing 4 Intel CPUs, --allow-unsupported alone exits 0 with unsupported_by host, and --warm --allow-unsupported exits 1 with null: the untouched workspace passes the gate (selfcheck.py:81 skips the host judgement under warm).

What I ran, on a Linux review host at 863109f:

  • make ci-local: 25 passed, 1 failed, 6 skipped. pip-audit fails on the lock files, which this PR does not touch. 6,719 tests passed, changed-line coverage passes.
  • 16 mutants of the repo tasks check step against the new test. It catches dropping either flag, wiring the flag to inputs.warm, a renamed env key, a misspelled flag and an inverted comparison.

Checked, no issues

  • Step script: flags=() with "${flags[@]}" expands correctly, including when empty. Nothing is interpolated into run.
  • Test: offline. The stub dokimi is a shell script in tmp_path, and the step env is rebuilt from the step's own env mapping, so a host WARM cannot leak in.
  • Behaviour: without warm, a missed host passes and host-probe or broken results still exit 1.
  • Changelog: fragment name and section match the neighbours.

Comment thread dokimi/docs/repo-evaluation-contract.md Outdated
Comment thread dokimi/tests/test_repo_eval_task_selection.py
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I pushed a commit per nit, up to 5fac907, which fix the two nits from my review:

  • f79830b: dokimi/docs/repo-evaluation-contract.md:902: re-wrapped the inputs bullet so runs-on, timeout-minutes and dokimi-repository no longer sit on a 114-character line.
  • 5fac907: dokimi/tests/test_repo_eval_task_selection.py:385: added a test that allow-unsupported is a boolean workflow_call input with default is False.

make ci-local at 5fac907: 25 passed, 1 failed, 6 skipped.

Not merging yet: @xaleryb on warm: true + allow-unsupported: true never lets an unsupported task pass; judge the host under --warm with the flag, or reject/document the pair waits for your answer, @napetrov.

@napetrov

Copy link
Copy Markdown
Contributor Author

@xaleryb @antonwolfy: went with the first option, judge the host under --warm when --allow-unsupported is given. Rejecting the pair would leave the oneDAL caller (warm: true) unable to use the input.

  • 6180048e: check_task/run_check take judge_host (default not warm); repo tasks check passes True with --allow-unsupported. --warm alone is unchanged (test_warm_does_not_judge_the_host still holds).
  • New test test_warm_with_allow_unsupported_judges_the_host: AMD, 2-CPU host, --warm --allow-unsupported -> unsupported/host, null skipped, isolation probed, exit 0.
  • Contract doc §4.3 and a changelog fragment describe the pair.
  • pytest tests/test_repo_host.py tests/test_repo_eval_task_selection.py: 106 passed.

--warm skipped the host check, so --allow-unsupported never passed a
task under warm: the null run passed where the bug cannot show and the
task failed as broken. The repo-eval workflow caller sets warm: true.

Reported by xaleryb in the review of intel#156.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Correction to the reply above: 6180048e re-added dokimi/uv.lock by mistake (the large-file and license-header checks fail on it). f7dc26bf removes it again; the PR diff is back to the workflow input, docs, changelog fragments, the --warm --allow-unsupported host judgement (option 1) and tests. tests/test_repo_host.py and tests/test_repo_eval_task_selection.py: 106 passed; test_warm_with_allow_unsupported_judges_the_host fails with the pre-6180048e selfcheck.py/repo_run.py.

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The --warm --allow-unsupported fix is right: the host is judged only when the flag is given, and --warm alone is unchanged. One nit inline.

@xaleryb's question on the warm: true + allow-unsupported: true pair is answered by 6180048: a 2-CPU AMD host now reports unsupported/host and exits 0 under both flags.

What I ran, on a Linux review host at f7dc26b:

  • make ci-local: 23 passed, 3 failed, 6 skipped. 6,721 tests passed, changed-line coverage passes (4 of 4 lines). The failures are the gitleaks install and scan jobs, which run no PR code, and pip-audit on the lock files, which this PR does not touch.
  • 9 mutants of the judge_host wiring against test_repo_host.py -k "warm or allow or unsupported" (baseline 10 passed). Each one fails the new test or an existing one: judge_host ignored, inverted default, always judging, run_check dropping it or passing it as warm, and cmd_repo_tasks_check passing None, False, always True, or the warm value.

Checked, no issues

  • Warm-only path: the command passes judge_host=None unless --allow-unsupported is given, and None means not warm. test_warm_does_not_judge_the_host still passes.
  • Callers: check_task is called only from run_check (selfcheck.py:182), and run_check only from cmd_repo_tasks_check (repo_run.py:148).
  • Docs: the contract paragraph in §4.3, the check_task docstring and the new changelog fragment match the code and the test. ruff check passes on the edited files.

Comment thread dokimi/dokimi/commands/repo_run.py Outdated
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed a commit per nit, up to 20d97d3, which fix the nit from my review:

  • 20d97d3: dokimi/dokimi/commands/repo_run.py:333: re-wrapped the --allow-unsupported help string as suggested, so the --warm clause starts its own line.

make ci-local at 20d97d3: 25 passed, 1 failed, 6 skipped.

GitHub CI has not run on these commits: the branch conflicts with main, and Actions runs no workflow on a conflicting PR. CI will check them once main is merged in.

Not approving until CI passes at this head.

main's intel#182 judges the host under --warm in every case, so drop this PR's
judge_host wiring and its changelog fragment; the test stays.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov
napetrov merged commit 13abb4d into intel:main Oct 23, 2026
28 checks passed
napetrov added a commit to napetrov/oneDAL that referenced this pull request Oct 24, 2026
Moves the reusable workflow and the quick-start install from ad1cbd0 to
intel/dokimi main 09be35b, which adds the workflow's allow-unsupported
input (intel/dokimi#156), judges task host requirements under --warm
(intel/dokimi#182) and ships the built-in review graders
(intel/dokimi#120). repo validate passes unchanged on 27 tasks.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
napetrov added a commit to napetrov/oneDAL that referenced this pull request Oct 24, 2026
…host

dokimi now judges task host requirements under --warm (intel/dokimi#182)
and the reusable workflow can pass --allow-unsupported (intel/dokimi#156).
The task leaves exclude-tasks: on a runner with 4+ GenuineIntel CPUs it is
self-checked like every other task; on a smaller or AMD runner dokimi
reports it unsupported with the reason instead of a null run that passes,
and the job passes.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
napetrov added a commit to napetrov/oneDAL that referenced this pull request Oct 26, 2026
Moves the reusable workflow and the quick-start install from ad1cbd0 to
intel/dokimi main 09be35b, which adds the workflow's allow-unsupported
input (intel/dokimi#156), judges task host requirements under --warm
(intel/dokimi#182) and ships the built-in review graders
(intel/dokimi#120). repo validate passes unchanged on 27 tasks.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
napetrov added a commit to napetrov/oneDAL that referenced this pull request Oct 26, 2026
…host

dokimi now judges task host requirements under --warm (intel/dokimi#182)
and the reusable workflow can pass --allow-unsupported (intel/dokimi#156).
The task leaves exclude-tasks: on a runner with 4+ GenuineIntel CPUs it is
self-checked like every other task; on a smaller or AMD runner dokimi
reports it unsupported with the reason instead of a null run that passes,
and the job passes.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants