Repository navigation
feat(ci): allow-unsupported input for the repo-eval workflow - #156
Conversation
Passes --allow-unsupported to repo tasks check, so a task whose task.json host requirements the runner misses passes its job. Off by default. Part of intel#41. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #156 +/- ##
==========================================
- Coverage 91.78% 91.77% -0.02%
==========================================
Files 223 223
Lines 27623 27623
Branches 4549 4549
==========================================
- Hits 25354 25350 -4
- Misses 1606 1609 +3
- Partials 663 664 +1
🚀 New features to boost your workflow:
|
antonwolfy
left a comment
There was a problem hiding this comment.
Review at 7d5e00a. The workflow change itself is right and small. The PR also adds dokimi/uv.lock (6,890 lines, 1.4 MB), which breaks CI. One thing to fix before merge.
What I ran, on a Linux review host at 7d5e00a:
make ci-local: 23 passed, 3 failed, 6 skipped. One failure is the PR's (the large-file hook). The other two do not come from this diff:pip-auditreports virtualenv 21.14.7 (CVE-2026-113722) from the existing locks (fixed onmainby #152), andtest_unbalanced_quote_in_hook_completes_quicklytook 1.002 s against a 1 s limit (6,714 other tests passed).- zizmor on
repo-eval.yml: no findings.
dokimi/uv.lock should not be in this PR. It is unrelated to the allow-unsupported input, and it fails CI as it stands.
check-added-large-files --enforce-all(.pre-commit-config.yaml:41) fails:dokimi/uv.lock (1377 KB) exceeds 500 KB.- The license-header check covers
**and ignores only the threerequirements*.lock.txtfiles (.github/.licenserc.yaml:45-47).uv.lockstarts withversion = 1and has no SPDX header: GitHub's Static checks job fails on it ("the following files don't have a valid license header: dokimi/uv.lock"). - The repo's locks are the hashed
requirements*.lock.txtfiles, whichtests/test_dependency_locks.pyand Renovate (renovate.json5:54) cover. Nothing generates, checks or installs from auv.lock, so it would drift unnoticed.
Suggestion: drop the file from the PR (git rm dokimi/uv.lock). If a uv lock is wanted, do it in its own PR with the header, the ignore entries and a check.
Checked, no issues
- Step script:
flags=()with"${flags[@]}"expands correctly. The input arrives throughenv, as the stringtrueorfalse, so[ "$ALLOW_UNSUPPORTED" = true ]works. Nothing is interpolated intorun. - Behaviour claims:
repo_run.py:152-159lets onlyunsupported_by == "host"pass. Broken tasks andhost-proberesults still exit 1, as the changelog and docs say. - Default: the input is under
workflow_callonly, defaults tofalse, so existing callers are unchanged.
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Lift the step's script, resolve its env from the inputs per case and run it with a stub dokimi; each warm/allow-unsupported pair must reach the CLI as given. Dropping the --allow-unsupported line, or wiring it to inputs.warm, now fails. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
|
antonwolfy
left a comment
There was a problem hiding this comment.
LGTM. uv.lock is gone, the diff is now the workflow input, its docs line, the changelog fragment and a test, and the test pins what the step forwards. Two nits inline.
@xaleryb's question on the warm: true + allow-unsupported: true pair waits for the author. New evidence: with a 2-CPU AMD runner and a task needing 4 Intel CPUs, --allow-unsupported alone exits 0 with unsupported_by host, and --warm --allow-unsupported exits 1 with null: the untouched workspace passes the gate (selfcheck.py:81 skips the host judgement under warm).
What I ran, on a Linux review host at 863109f:
make ci-local: 25 passed, 1 failed, 6 skipped.pip-auditfails on the lock files, which this PR does not touch. 6,719 tests passed, changed-line coverage passes.- 16 mutants of the
repo tasks checkstep against the new test. It catches dropping either flag, wiring the flag toinputs.warm, a renamed env key, a misspelled flag and an inverted comparison.
Checked, no issues
- Step script:
flags=()with"${flags[@]}"expands correctly, including when empty. Nothing is interpolated intorun. - Test: offline. The stub
dokimiis a shell script intmp_path, and the step env is rebuilt from the step's ownenvmapping, so a hostWARMcannot leak in. - Behaviour: without
warm, a missedhostpasses andhost-probeor broken results still exit 1. - Changelog: fragment name and section match the neighbours.
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
antonwolfy
left a comment
There was a problem hiding this comment.
LGTM. I pushed a commit per nit, up to 5fac907, which fix the two nits from my review:
f79830b:dokimi/docs/repo-evaluation-contract.md:902: re-wrapped the inputs bullet soruns-on,timeout-minutesanddokimi-repositoryno longer sit on a 114-character line.5fac907:dokimi/tests/test_repo_eval_task_selection.py:385: added a test thatallow-unsupportedis a booleanworkflow_callinput withdefault is False.
make ci-local at 5fac907: 25 passed, 1 failed, 6 skipped.
Not merging yet: @xaleryb on warm: true + allow-unsupported: true never lets an unsupported task pass; judge the host under --warm with the flag, or reject/document the pair waits for your answer, @napetrov.
|
@xaleryb @antonwolfy: went with the first option, judge the host under
|
--warm skipped the host check, so --allow-unsupported never passed a task under warm: the null run passed where the bug cannot show and the task failed as broken. The repo-eval workflow caller sets warm: true. Reported by xaleryb in the review of intel#156. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Correction to the reply above: |
antonwolfy
left a comment
There was a problem hiding this comment.
LGTM. The --warm --allow-unsupported fix is right: the host is judged only when the flag is given, and --warm alone is unchanged. One nit inline.
@xaleryb's question on the warm: true + allow-unsupported: true pair is answered by 6180048: a 2-CPU AMD host now reports unsupported/host and exits 0 under both flags.
What I ran, on a Linux review host at f7dc26b:
make ci-local: 23 passed, 3 failed, 6 skipped. 6,721 tests passed, changed-line coverage passes (4 of 4 lines). The failures are the gitleaks install and scan jobs, which run no PR code, andpip-auditon the lock files, which this PR does not touch.- 9 mutants of the
judge_hostwiring againsttest_repo_host.py -k "warm or allow or unsupported"(baseline 10 passed). Each one fails the new test or an existing one:judge_hostignored, inverted default, always judging,run_checkdropping it or passing it aswarm, andcmd_repo_tasks_checkpassingNone,False, alwaysTrue, or the warm value.
Checked, no issues
- Warm-only path: the command passes
judge_host=Noneunless--allow-unsupportedis given, andNonemeansnot warm.test_warm_does_not_judge_the_hoststill passes. - Callers:
check_taskis called only fromrun_check(selfcheck.py:182), andrun_checkonly fromcmd_repo_tasks_check(repo_run.py:148). - Docs: the contract paragraph in §4.3, the
check_taskdocstring and the new changelog fragment match the code and the test.ruff checkpasses on the edited files.
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
antonwolfy
left a comment
There was a problem hiding this comment.
I pushed a commit per nit, up to 20d97d3, which fix the nit from my review:
20d97d3:dokimi/dokimi/commands/repo_run.py:333: re-wrapped the --allow-unsupported help string as suggested, so the --warm clause starts its own line.
make ci-local at 20d97d3: 25 passed, 1 failed, 6 skipped.
GitHub CI has not run on these commits: the branch conflicts with main, and Actions runs no workflow on a conflicting PR. CI will check them once main is merged in.
Not approving until CI passes at this head.
main's intel#182 judges the host under --warm in every case, so drop this PR's judge_host wiring and its changelog fragment; the test stays. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Moves the reusable workflow and the quick-start install from ad1cbd0 to intel/dokimi main 09be35b, which adds the workflow's allow-unsupported input (intel/dokimi#156), judges task host requirements under --warm (intel/dokimi#182) and ships the built-in review graders (intel/dokimi#120). repo validate passes unchanged on 27 tasks. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…host dokimi now judges task host requirements under --warm (intel/dokimi#182) and the reusable workflow can pass --allow-unsupported (intel/dokimi#156). The task leaves exclude-tasks: on a runner with 4+ GenuineIntel CPUs it is self-checked like every other task; on a smaller or AMD runner dokimi reports it unsupported with the reason instead of a null run that passes, and the job passes. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Moves the reusable workflow and the quick-start install from ad1cbd0 to intel/dokimi main 09be35b, which adds the workflow's allow-unsupported input (intel/dokimi#156), judges task host requirements under --warm (intel/dokimi#182) and ships the built-in review graders (intel/dokimi#120). repo validate passes unchanged on 27 tasks. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…host dokimi now judges task host requirements under --warm (intel/dokimi#182) and the reusable workflow can pass --allow-unsupported (intel/dokimi#156). The task leaves exclude-tasks: on a runner with 4+ GenuineIntel CPUs it is self-checked like every other task; on a smaller or AMD runner dokimi reports it unsupported with the reason instead of a null run that passes, and the job passes. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Summary
tasks-check job: repo tasks check flags=() if warm: flags+=(--warm) + if allow-unsupported: flags+=(--allow-unsupported)New boolean input
allow-unsupported(default false). A task whose task.jsonhostblock (D3, #112) the runner misses passes its job; broken tasks and failed host probes still fail (CLI semantics unchanged). Caller: oneDAL #3818fix_cel_thread_race(needs 4+ Intel CPUs; hosted runners may be AMD).Part of #41.
Evidence
--allow-unsupportedfrom a caller; an unsupported task fails its job, so oneDAL excludes the task.After:
allow-unsupported: trueadds the flag;pytest tests/test_repo_eval_task_selection.py37 passed.Merge Danger
Door: two-way
Blast Radius: none (opt-in input)
🤖 Generated with Claude Code