ci: reusable repo-eval workflow (validate, scan, tasks check) with dokimi pinned

.github/workflows/repo-eval-self.yml

@@ -0,0 +1,58 @@
+# Copyright (C) 2026 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+name: repo-eval-self
+
+# dokimi's own evals (.agent-eval/) through the reusable repo-eval.yml, as a
+# repository calls it, with two differences: the workflow is this commit's
+# (`./`), and so is the dokimi it installs, so a change to the repository
+# mode is checked against these tasks before it merges.
+#
+# A change to a task checks that task. A change to the harness (run.py, the
+# Dockerfile, the manifest), to the repository mode or to either workflow
+# checks the smoke set: one fix task (hidden tests plus the whole suite) and
+# the score-only review task. A dispatch checks all ten by default.
+on:
+  pull_request:
+    paths:
+      - '.agent-eval/**'
+      - 'dokimi/dokimi/repos/**'
+      - 'dokimi/dokimi/commands/repo*.py'
+      - 'dokimi/dokimi/schemas/repo_*.json'
+      - 'dokimi/requirements.lock.txt'
+      - 'dokimi/pyproject.toml'
+      - '.github/workflows/repo-eval.yml'
+      - '.github/workflows/repo-eval-self.yml'
+  workflow_dispatch:
+    inputs:
+      tasks:
+        description: "`all`, or task ids separated by spaces or commas"
+        required: false
+        default: all
+
+permissions: {}
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+jobs:
+  self:
+    permissions:
+      contents: read
+    uses: ./.github/workflows/repo-eval.yml
+    with:
+      # The commit under test: the merge commit on a pull request.
+      dokimi-ref: ${{ github.sha }}
+      dokimi-repository: ${{ github.repository }}
+      manifest: .agent-eval/repo-eval.yaml
+      tasks: ${{ inputs.tasks || 'changed' }}
+      smoke-tasks: fix-degrade-pre review-pr-conventions
+      # Every `paths:` entry above outside .agent-eval/, as a prefix.
+      smoke-paths: >-
+        dokimi/dokimi/repos/ dokimi/dokimi/commands/repo dokimi/dokimi/schemas/repo_
+        dokimi/requirements.lock.txt dokimi/pyproject.toml .github/workflows/repo-eval.yml
+      max-parallel: 4
+      # intel-ubuntu-latest: see ci.yml.
+      runs-on: ${{ github.repository_owner == 'intel' && 'intel-ubuntu-latest' || 'ubuntu-latest' }}
+      timeout-minutes: 60

dokimi/changelog.d/20261019_144048_napetrov_d8-repo-eval-workflow.md

@@ -0,0 +1,3 @@
+### Changed
+
+- **`repo validate` refuses task ids that contain whitespace or a comma** — a task directory whose name has a space, tab or comma is now a `task.json` error instead of passing validation. The repo-eval workflow's `tasks`, `smoke-tasks` and `exclude-tasks` inputs are split on whitespace and commas, so such a task could never be named there. Rename the task directory to a name without whitespace or commas.

dokimi/changelog.d/20261021_100000_napetrov_repo-eval-timeout-verdict.md

@@ -0,0 +1,7 @@
+### Fixed
+
+- **repo-eval: a hung oracle leaves a verdict** — the Summary and Upload steps of
+  `repo-eval.yml` run with `always()`, and a job cancelled by `timeout-minutes` before
+  `check.json` exists records a `timeout` verdict for the task. The self-check
+  (`repo-eval-self.yml`) now also runs on changes to `commands/repo*.py`,
+  `schemas/repo_*.json`, `requirements.lock.txt` and `pyproject.toml`.

dokimi/docs/repo-evaluation-contract.md

@@ -238,7 +238,7 @@ belongs to the repository's grader and is passed through unread.
 
 | Key | Required | Read by | Meaning |
 |---|---|---|---|
-| `id` | yes | framework | equals the directory name |
+| `id` | yes | framework | equals the directory name, and contains no whitespace or comma: the workflow's `tasks`, `smoke-tasks` and `exclude-tasks` inputs are lists of ids split on whitespace and commas, so `repo validate` refuses an id containing either |
 | `family` | yes | framework (reporting) | free string; common values: `build`, `fix`, `feature`, `test`, `review` |
 | `base` | yes | framework | commit to snapshot: a hex SHA, optionally followed by `^` (the parent, the usual form for a mined fix). Branch names, tags and `HEAD` are refused, because repetitions of one task must get one snapshot. Use the full 40-character SHA; a shorter one is accepted with a warning. It must resolve in the source. `meta.json` records it as `base_rev` and the resolved full SHA as `base_sha` |
 | `prompt` | yes | framework | what the agent is told |
@@ -870,3 +870,59 @@ python cli.py repo compare repo-runs/pilot --baseline guidance:none --metric roo
   `@20250929`) or its provider-prefixed form is the same model, `-v2` against
   `-v3` is not. `repo compare` warns the same way when the two arms of a
   comparison were answered by different models.
+
+CI for a repository: `.github/workflows/repo-eval.yml` is a reusable workflow
+that runs the no-LLM checks on the caller's checkout and needs no secret.
+
+- `validate` runs `repo validate` and gates the other two jobs; `scan` runs `repo scan`
+  (report only: the claim counts and failing claims go to the job summary, the
+  `repo_readiness.v1` report to an artifact); `tasks-check` runs `repo tasks check`
+  for each selected task in a job of its own (docker; the image is built from the
+  manifest's Dockerfile), and uploads `check.json` and the grade logs. If the job is
+  cancelled by its `timeout-minutes` before the check finishes, the Summary step
+  writes a `check.json` with the verdict `timeout` for the task.
+- `tasks: changed` (the default, also for an empty value) checks, on a pull request, the tasks whose
+  directories it touches, plus `smoke-tasks` (empty: every task) when it changes
+  the eval directory outside its tasks (Markdown there is excepted, unless it is the manifest,
+  the calling workflow file or under a `smoke-paths` entry), the manifest, the calling workflow
+  file or a path under `smoke-paths` (entries are space- or comma-separated prefixes,
+  normalised like the manifest path; keep the trailing slash for a directory). A file in
+  the tasks directory that is not under an existing task (a shared helper, or the files of
+  a deleted task) does too. On a push it diffs against the previous tip, on any other event
+  it checks every task. `all` or a list of
+  task ids selects directly; `exclude-tasks` always drops tasks.
+- Pin dokimi by a commit SHA or a release tag, the same one in `uses:` and in
+  `dokimi-ref` (a branch is refused). It installs
+  `dokimi @ git+https://github.com/intel/dokimi@<sha>#subdirectory=dokimi` with
+  that commit's `requirements.lock.txt` as constraints, so a rerun checks with the
+  same framework and dependencies. Moving the pin is a change to the calling
+  workflow, so it runs the smoke set.
+- Other inputs: `manifest`, `max-parallel` (4), `warm` (`--warm`, for a cache that
+  starts empty on every runner), `runs-on` (`ubuntu-latest`), `timeout-minutes`
+  (180, per task) and `dokimi-repository` (`intel/dokimi`).
+
+```yaml
+# .github/workflows/agent-evals.yml in the repository
+name: Agent evals
+on:
+  pull_request:
+    paths: ['dev/agent_evals/**', '**/AGENTS.md', '.github/workflows/agent-evals.yml']
+  schedule:
+    - cron: '0 3 * * 0'
+  workflow_dispatch:
+permissions: {}
+jobs:
+  evals:
+    permissions:
+      contents: read
+    uses: intel/dokimi/.github/workflows/repo-eval.yml@<sha>  # vX.Y.Z
+    with:
+      dokimi-ref: <sha>
+      manifest: dev/agent_evals/repo-eval.yaml
+      smoke-tasks: build_kmeans_host fix_finiteness_mut review_seeded_fc
+      max-parallel: 4
+      warm: true
+```
+
+dokimi runs the workflow on its own `.agent-eval/` (`repo-eval-self.yml`), with
+the commit under test as `dokimi-ref`.

dokimi/dokimi/repos/manifest.py

@@ -417,6 +417,12 @@ def _glob_regex(pattern: str) -> re.Pattern[str]:
 def _check_task(tree: RepoTree, task_dir: str, report: RepoCheckReport) -> None:
     name = PurePosixPath(task_dir).name
     where = f"{task_dir}/task.json"
+    if re.search(r"[\s,]", name):
+        # tasks, smoke-tasks and exclude-tasks are lists of ids split on whitespace and commas, so an id
+        # containing either could never be named there
+        report.errors.append(f"{where}: task id {name!r} contains whitespace or a comma; task ids are "
+                             "split on whitespace and commas in the workflow's tasks, smoke-tasks and exclude-tasks inputs")
+        return
     try:
         raw = json.loads(tree.read_text(where))
     except (json.JSONDecodeError, UnicodeDecodeError) as exc: