Skip to content

fix(repo): PATH@REV takes a revision with a slash; flush progress per task - #159

Merged
antonwolfy merged 7 commits into
intel:mainfrom
napetrov:fix/scan-rev-unbuffered
Oct 23, 2026
Merged

antonwolfy merged 7 commits into
intel:mainfrom
napetrov:fix/scan-rev-unbuffered

Conversation

@napetrov

Copy link
Copy Markdown
Contributor

Part of #41 (found while porting oneDAL).

Summary

 RepoSource.parse("PATH@REV")          # rpartition on the last '@'
-  _REV_RE = [A-Za-z0-9._^~-]+          # origin/main -> no rev -> "not a directory"
+  _is_rev: [A-Za-z0-9._/^~-]+, no '..', '//', trailing '/'
 cmd_repo_tasks_check -> run_check(..., log=print)
+                       run_check(..., log=_progress)   # print(line, flush=True)
 cmd_repo_run         -> run_batch(..., log=_progress)

Every repo command that takes a source goes through RepoSource.parse; resolve_rev uses the same check. PYTHONUNBUFFERED in repo-eval.yml is left as is.

Evidence

  • Before: dokimi repo scan .@origin/main -> local source is not a directory: .@origin/main; per-task check lines appear only at exit under a pipe.
    After: test_source_parse_takes_a_rev_with_a_slash[origin/main|refs/heads/x|HEAD~1|...] pass (local and remote); test_source_parse_leaves_a_non_rev_suffix_in_the_path keeps a..b etc. in the path; test_progress_line_is_flushed_under_a_pipe reads the line from a piped subprocess before it exits.

Merge Danger

Door: two-way

Blast Radius: narrow

A local directory literally named x@origin/main would now be read as x at origin/main; same trade-off that already applied to x@main.

Checklist

  • Tests for both fixes
  • ruff check . clean
  • scriv fragment in dokimi/changelog.d/
  • DCO sign-off

🤖 Generated with Claude Code

…task

repo scan .@origin/main read the spec as a path: the revision pattern had
no '/'. Allow it (no '..', '//', trailing '/'). repo tasks check and repo
run hand a flushing logger to the runners so CI shows lines as they come.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@codecov-commenter

codecov-commenter commented Oct 22, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.78%. Comparing base (5064dc2) to head (c969408).
⚠️ Report is 12 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #159      +/-   ##
==========================================
+ Coverage   91.64%   91.78%   +0.14%     
==========================================
  Files         222      223       +1     
  Lines       27409    27637     +228     
  Branches     4508     4552      +44     
==========================================
+ Hits        25118    25368     +250     
+ Misses       1620     1607      -13     
+ Partials      671      662       -9     
Files with missing lines Coverage Δ
dokimi/dokimi/commands/repo_run.py 92.50% <100.00%> (+0.05%) ⬆️
dokimi/dokimi/repos/source.py 94.94% <100.00%> (+0.80%) ⬆️

... and 19 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 1dec01a. The PATH@REV fix works for origin/main and refs/heads/x, but it breaks the split for unpinned remote URLs with a user. Three points, inline.

What I ran, on a Linux review host at 1dec01a:

  • make ci-local: 24 passed, 2 failed, 6 skipped. The failures are test_unbalanced_quote_in_hook_completes_quickly (1.04 s against a 1 s limit, a timing flake under load; the rest of the suite is 6724 passed) and pip-audit on virtualenv 21.14.7 (CVE-2026-113722) from the existing locks, fixed on main by #152. Changed-line coverage passes.
  • Reproductions of RepoSource.parse on the PR head and on the base.
  • A mutation run over the new tests: 4 of the mutants survive the new tests and the existing test_repo*.py tests.

Checked, no issues

  • Non-rev suffixes: dropping the .., // or trailing-/ check is each caught by the new parametrized test.
  • Wiring: removing log=_progress or passing log=print in either command is caught.
  • Changelog: the fragment is present.

Comment thread dokimi/dokimi/repos/source.py Resolved
Comment thread dokimi/tests/test_repo_progress.py Resolved
Comment thread dokimi/dokimi/repos/source.py Resolved
napetrov and others added 3 commits October 22, 2026 11:41
…st resolve_rev

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
With "/" allowed in a revision, a local path such as ws/job@2/repo or
/home/alice@corp.com/proj was split at the "@" and failed as "not a
directory". Split a local spec only when the whole spec is no directory.

Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
@xaleryb

xaleryb commented Oct 23, 2026

Copy link
Copy Markdown
Contributor
  • Ran: full suite, ruff, mypy, bandit at the head, plus repo scan on local repos whose parent directory contains an @.
  • repo scan ws/job@2/repo (a Jenkins concurrent workspace) or /home/alice@corp.com/proj -> local source is not a directory: ws/job -> scans the existing directory, as on main. With / allowed in a revision, every local path with an @ in a parent directory is split at that @ (node_modules/@types/node and r@v1.2.3/skills too).
  • 6841ae8 fix(repo): read an existing directory with an @ in its path whole: a local spec is split only when the whole spec is not an existing directory. Includes a regression test and a line in the changelog fragment. This also removes the x@origin/main trade-off named in the PR body.
  • Left for the author: resolve the three inline threads. Each is fixed at the head.

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 6841ae8. The three earlier points are fixed, and the existing-directory rule from the last commit is right. One thing is worth fixing before merge: an unpinned remote URL with an @ in its path is still split at that @.

What I ran, on a Linux review host at 6841ae8:

  • make ci-local: 28 passed, 1 failed, 7 skipped. The failure is pip-audit on the existing locks, unrelated to the PR. 7,232 tests passed; changed-line coverage 100%.
  • Reproductions of RepoSource.parse on the PR head and on the base.
  • A mutation run over the new tests: dropping the spec.startswith(_REMOTE_PREFIXES) exemption, dropping file:// from _has_location_path, and allowing a leading / in _REV_RE all pass the full test_repo*.py run. The other mutants (_is_rev clauses, resolve_rev guard, remote prefixes, log=_progress, flush) are caught.

Checked, no issues

  • Pinned specs: PATH@origin/main, ssh://git@host/o/r@origin/main, git@host:o/r@sha and ws/job@2/repo give the same location and rev as before, or the intended new one.
  • Callers: nothing uses .rev as a file name or cache key; cache_path hashes the URL only.
  • Flush test: with print(line) instead of print(line, flush=True) it now fails on the 5 s wait.

Comment thread dokimi/dokimi/repos/source.py Resolved

@xaleryb xaleryb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at 6841ae8. Every earlier point re-run at this head passes and no must-fix or mid item is open; full suite, ruff and mypy clean on Linux; CI green.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The last commit fixes the remaining unpinned-URL splits, and the four specs from my earlier thread get the pin error again. One nit inline.

What I ran, on a Linux review host at 79bca0e:

  • make ci-local: 26 passed, 3 failed, 7 skipped. 7,246 tests passed; changed-line coverage passes. The failures are pip-audit, which fails on the lock files this PR does not touch, and the two gitleaks steps, which could not download the tool; gitleaks detect over the PR's 5 commits finds no leaks.
  • RepoSource.parse on file:///home/a@corp.com/proj, file:///tmp/a@b/c, https://github.com/o@x/r, git@github.com/o/r: each raises "a remote source must be pinned to a revision". ssh://git@host/o/r@origin/main, git@host:o/r@sha, https://github.com/o/r.git@v1.0, https://github.com/o@x/r@abc1234 keep their location and revision.
  • A mutation run over the new tests: dropping file:// from _has_location_path, the _remote_split_ok ref-root check, the origin and refs roots, and the .. and // clauses of _is_rev are each caught.

Checked, no issues

  • Limit the author named: a remote branch like feature/x needs a SHA or a ref-prefixed name (https://github.com/o/r.git@feature/x gives the pin error). The base gave the same error, so nothing regresses.
  • Wiring: repo tasks check and repo run both pass log=_progress.
  • Changelog: the fragment covers both fixes.

Comment thread dokimi/dokimi/repos/source.py Resolved
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I pushed a commit per nit, up to c969408, which fix the nit from my review:

  • c969408: dokimi/tests/test_repo_manifest.py:682: the pinned-spec test now has one case each for the upstream, remotes, heads and tags roots, asserting the rev.

make ci-local at c969408: 28 passed, 1 failed (pip-audit), 7 skipped.

@antonwolfy
antonwolfy merged commit 0d533a2 into intel:main Oct 23, 2026
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants