[napetrov]

Summary

Harbor-backed cells (codex, claude-code, terminal-bench:<agent>) are now graded from the trial result.json Harbor writes. harbor's exit code and stdout no longer decide the verdict. Harbor 0.24.0 exits 0 when a trial raised, so an agent-install failure was recorded as PASS 1/1.

 CommandHarness.run (dokimi/dokimi/harnesses/command.py)
   if timed_out:                      reward 0, failed           (unchanged)
+  elif kind == "harbor":
+    read_harbor_job(<jobs-dir>)      <jobs-dir>/<job>/<trial>/result.json
+      exception_info set          -> errored  (metrics.error, metrics.harbor_exception)
+      no numeric rewards.reward   -> errored
+      no trial file / >1 trial    -> errored
+      else                        -> verifier_result.rewards.reward
+    harbor_usage(agent_result)       prompt/completion/cache_read/total_tokens, cost_usd
   else:  # command:
     _find_reward: reward.txt|json, then "reward: x" on stdout, then rc == 0   (unchanged)

The rules are the same as intel/skills' tools/check_harbor_job.py. An errored cell is excluded from pass_rate and from McNemar, the same way docker-* and BenchFlow cells are (metrics.error → is_errored). The token keys are the ones BenchFlow uses. As in BenchFlow, Harbor's n_input_tokens includes the cached prompt. A null count, such as from the oracle or a trial that never ran its agent, is left out rather than recorded as 0.

Also in this PR:

• docs/coding-harnesses.md: the Harbor section now gives the grading table and the usage mapping. The old text said to read result.json yourself; that paragraph is replaced.
• Changelog fragment (Fixed + Added).
• tests/fixtures/harbor/{pass,fail,exception}/: trimmed real Harbor 0.24.0 job and trial files with paths and hosts removed.
• tests/test_task_formats.py: the fake harbor now writes a passing trial file. Those tests check that a task directory is not refused. Without the file the cell now comes out errored, which is the intended behavior.

command: is unchanged. It has the same weakness: when no reward file is found, rc 0 counts as a pass, and the stdout regex reward\D+([01]…) matches Harbor's ┃ Reward ┃ Count ┃ … │ 1.0 │ table. So a --command-template that wraps harbor can still read an errored trial as a pass. I left it alone because custom templates depend on that fallback. The docs now say this.

Evidence

• Before (intel/main 6e695ce2). A fake harbor exits 0 and leaves each fixture under --jobs-dir, as Harbor 0.24.0 did:

  stdout = Harbor's reward table
  pass      rc=0 passed=True  reward=1.0  cost=None error=None
  fail      rc=0 passed=True  reward=1.0  cost=None error=None
  exception rc=0 passed=True  reward=1.0  cost=None error=None
  None      rc=0 passed=True  reward=1.0  cost=None error=None
  empty stdout
  pass      rc=0 passed=True  reward=None cost=None error=None
  fail      rc=0 passed=True  reward=None cost=None error=None
  exception rc=0 passed=True  reward=None cost=None error=None
  None      rc=0 passed=True  reward=None cost=None error=None
  

  (The fixtures contain only the result.json files. A real failing trial also writes verifier/reward.txt, which main reads, so main's fail row is right in practice. The exception and no-file rows are the bug as measured.)

  After:

  pass      rc=0 passed=True  reward=1.0  cost=0.038918  error=None
  fail      rc=0 passed=False reward=0.0  cost=0.030927  error=None
  exception rc=0 passed=False reward=None cost=None     error=harbor: NonZeroAgentExitCodeError: Command failed (exit 100): apt-get update && apt-get install -y curl nodejs npm
  None      rc=0 passed=False reward=None cost=None     error=harbor: no Harbor result.json (exit 0)
  

• New tests, tests/test_harbor_results.py (21): pass with tokens and cost, scored fail, exception (codex / claude-code / terminal-bench), a reward next to an exception is kept aside rather than scored, missing or non-numeric reward ×5, no result.json (rc 0 and rc 1, with a stray reward.txt and Harbor's table on stdout), job summary without a trial file, two trials, unreadable file, usage mapping, a TaskSuiteRunner sweep where the errored cell is out of pass_rate and the cost is summed, and command: keeping its rc fallback.

  tests/test_harbor_results.py tests/test_task_harnesses.py tests/test_task_formats.py tests/test_tasks_command.py
  226 passed in 29.41s
  

• make ci-local: 0 failed, 19 passed, 4 skipped. pytest 5199 passed, 2 skipped; ruff, mypy, pre-commit, bandit, zizmor, mkdocs, schema_compat and package-smoke all pass. Diff coverage against intel/main: command.py (100%), 67 lines, 0 missing.

• Live: dokimi tasks run --harness claude-code on intel/skills onetbb-transpose with Harbor 0.24.0 and Claude Haiku 4.5. The agent install failed inside the task container, and Harbor exited 0. The cell is now 0/0 pass (n/a), 1 errored (harbor: NonZeroAgentExitCodeError …). The same setup recorded 1/1 pass (100.00%) before this change.

Merge Danger

Door: two-way

The change is limited to the kind == "harbor" branch of CommandHarness.run plus two helpers. Reverting is one commit, and no schema or artifact format changes; the new metrics.* keys are additive, and task_runs.v1 metrics is an open object.

Blast Radius: Harbor-arms

• Harbor-backed pass rates can go down: cells that used to read as passes now read as errored, and the "Reward" stdout scrape no longer applies to them. Earlier Harbor runs with errored trials over-reported and are not comparable; the changelog says to rerun.
• A Harbor cell that harbor writes no trial file for is now errored. That covers a version that changes the jobs-dir layout, or a -k/-n override inside --command-template. The --command-template case is still graded as command:, which is unaffected.
• Harbor's agent-timeout handling changes. AgentTimeoutError with a verifier reward is now errored, with the reward stored as metrics.reward_after_exception, which follows check_harbor_job.py. docker-* and BenchFlow score an agent timeout as 0. If Harbor arms should match that, it is a follow-up, not part of this PR.
• docker-*, BenchFlow and command: are untouched.

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.38%. Comparing base (23b95e8) to head (6732835).
⚠️ Report is 1 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #111      +/-   ##
==========================================
+ Coverage   91.28%   91.38%   +0.10%     
==========================================
  Files         216      216              
  Lines       26565    26722     +157     
  Branches     4337     4373      +36     
==========================================
+ Hits        24250    24421     +171     
+ Misses       1646     1636      -10     
+ Partials      669      665       -4     

Files with missing lines	Coverage Δ
dokimi/dokimi/harnesses/benchflow.py	95.41% <100.00%> (+0.48%)	⬆️
dokimi/dokimi/harnesses/command.py	97.27% <100.00%> (+2.11%)	⬆️

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

🟢 Approval recommended

The isolated grading changes have focused regression coverage and no identified blocking defects.

No open issues 🎉

What changed in this PR

Corrects Harbor-backed task evaluation by using trial results instead of process exit codes or stdout, preventing failed installations from counting as passes.

Changes:

• Grades Harbor cells from result.json and records token usage and cost.
• Adds regression tests and sanitized Harbor fixtures; preserves command: behavior.
• Documents grading rules and the need to rerun affected historical evaluations.

File	Description
dokimi/​tests/​test_task_formats.py	Makes fake Harbor write a passing trial.
dokimi/​tests/​test_harbor_results.py	Tests grading, errors, usage, and aggregation.
dokimi/​tests/​fixtures/​harbor/​pass/​2026-10-06__14-39-30/​result.json	Passing job summary fixture.
dokimi/​tests/​fixtures/​harbor/​pass/​2026-10-06__14-39-30/​onetbb-transpose__4LraMVa/​result.json	Passing trial with usage metrics.
dokimi/​tests/​fixtures/​harbor/​fail/​2026-10-06__14-39-30/​result.json	Failing job summary fixture.
dokimi/​tests/​fixtures/​harbor/​fail/​2026-10-06__14-39-30/​onetbb-transpose__kaHJy7X/​result.json	Zero-reward trial fixture.
dokimi/​tests/​fixtures/​harbor/​exception/​2026-10-06__14-39-30/​result.json	Errored job summary fixture.
dokimi/​tests/​fixtures/​harbor/​exception/​2026-10-06__14-39-30/​onetbb-transpose__SizQxqz/​result.json	Agent-install exception fixture.
dokimi/​dokimi/​harnesses/​command.py	Reads Harbor verdicts and usage metrics.
dokimi/​docs/​coding-harnesses.md	Explains grading and telemetry contracts.
dokimi/​changelog.d/​20261018_120000_napetrov_harbor-result-json.md	Records corrected grading and added metrics.

_{Review effort: Balanced}

[antonwolfy]

Review at d7b8cc7. Grading from the trial result.json is the right fix, and the core rules hold: an exception, a missing reward, no trial file or several trials all come out errored, and a stray reward.txt or Harbor's stdout table no longer decides anything. Three things are worth fixing before merge, all inline.

What I ran, on a Linux review host at d7b8cc7:

• make ci-local with docker: 23 passed, 0 failed, 3 skipped (test-latest, docs deploy, semgrep).
• Reproductions for the code findings with the transport mocked; a 31-mutant run over read_harbor_job, harbor_usage, _harbor_reward and the Harbor branch of run, 7 of which pass the four affected test files.

One more, outside this diff:

1. The artifact-fields text still describes the old fallback. docs/coding-harnesses.md:840 says "If no reward is found, the adapter's exit code is used as the fallback", and :841 lists the token and cost metrics for docker-claude* only. Neither is true for Harbor-backed cells any more: with no reward they are errored, and they now carry prompt_tokens, completion_tokens, cache_read_tokens, total_tokens and cost_usd. Suggestion: add "except Harbor-backed cells, which never fall back to the exit code" and name Harbor on :841.

Checked, no issues

• Exit-code isolation: a stray reward: 1.0 on stdout plus reward.txt, with rc 0 and rc 1, still gives an errored cell. Swapping in _find_reward is caught by the tests.
• Errored cells in the stats: metrics.error with reward=None takes the cell out of pass_rate, avg_reward and McNemar through is_errored.
• Bool and string rewards, a non-object top level, a BOM: all errored, none a pass.
• Stale results: clear_run_dir runs before the command, so a previous run's file is not read back.
• Docs and changelog: metric keys match runner.py and BenchFlow; the fragment name and categories follow the repo's pattern; coding-harnesses.md is in the nav; fixtures hold no hosts or paths.

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

4 open issues

• The error message \"trial result.json is not a JSON object\" is also used when the file is… · New
• Casting token metrics with int(v) will silently truncate non-integer numeric values (e.g., `0.5… · New
• This test's docstring is inconsistent with the implemented validation: _harbor_reward rejects… · New
• PASS_TRIAL is defined but not used anywhere in this test module (based on the full file in this… · New

_{Review effort: Lite}

[antonwolfy]

Review at 340b1a2. The reward and count validation fixes the NaN, Infinity and negative cases from my earlier review. The same validation still lets a huge integer crash the cell, and the relative-path test is still missing. Two inline comments and one doc fix I'd do before merge.

What I ran, on a Linux review host at 340b1a2:

• make ci-local with docker: 23 passed, 0 failed, 3 skipped. pytest: 5209 passed, 1 skipped.
• Reproductions for the code findings, with the LLM transport mocked, driven through CommandHarness.run with a fake harbor that leaves an edited pass fixture.
• A mutation run over read_harbor_job, _harbor_reward, _number and the Harbor branch of run.

One more, outside this diff:

1. The artifact-fields text and the grading table still do not match the code. This is the point from my earlier review, not fixed at this head. docs/coding-harnesses.md:840 still says "If no reward is found, the adapter's exit code is used as the fallback", and :841 still lists the token and cost metrics for docker-claude* only. Harbor-backed cells never fall back to the exit code, and they carry prompt_tokens, completion_tokens, cache_read_tokens, total_tokens and cost_usd. The grading table (:419-422) also does not say that a reward must be finite and in [0, 1]: a verifier that writes 2.0 gets the error "has no verifier_result.rewards.reward", and nothing in the docs explains it. Suggestion: add "except Harbor-backed cells, which never fall back to the exit code" on :840, name Harbor on :841, and add the [0, 1] rule to the table.

I agree with @copilot-pull-request-reviewer's four points (int(v) truncation, the "not a JSON object" message, the no-op assertion, unused PASS_TRIAL). New evidence: n_input_tokens: 0.5, n_output_tokens: 1.7 gives prompt_tokens=0, completion_tokens=1, total_tokens=2, so the parts do not sum to the total.

Still open from my earlier review:

• command.py:151-153: the result_json rewrite back to the caller's relative dir is still not tested. test_result_json_path_is_properly_recorded uses an absolute tmp_path, so given_dir == output_dir and the rewrite does nothing; it asserts only "result.json" in path and that the path exists. Replacing given_dir with output_dir, and forcing the condition to if False:, both still pass all 235 tests in the four affected files. The smaller gaps I named also still survive: or "unknown", the [:200] cut and the isinstance(exception, dict) guard at :269-273.

Checked, no issues

• Reward range: inf, NaN, 2.0 and -1 now come out errored with reward=None; <= 1.0 changed to <= 2.0 and value < 0 changed to value < -1 are both caught by the suite.
• Hostile exception_info: a list, a string, and a dict with non-string type or message do not crash; the cell is errored.
• Errored cells in the stats: metrics.error with reward=None still takes the cell out of pass_rate through is_errored.
• Fixtures and changelog: no hosts or paths in the fixtures; the fragment follows the repo's pattern; task_runs.v1 metrics is open, so no schema bump.

[xaleryb]

• Ran: real Harbor 0.20.0 through dokimi tasks run (single- and multi-step toy tasks, oracle agent), plus the full suite, ruff, mypy and bandit on the head merged into main.
• Multi-step task whose step-2 setup.sh exits 3 -> 1/1 pass (100.00%), because the exception is only on step_results[1] and the top-level reward is step-1's 1.0 -> expected errored. Fixed in dd6d670, which also sums usage over step_results[].agent_result.
• Verifier timeout_sec=5 with a test that sleeps 40 -> errored, out of pass_rate -> expected reward 0, as BenchFlow and docker-* do and as the report footnote says ("a verifier timeout is a failure, not an error"). Left for the author.
• a75c817: removes command.py _load_json, which has no caller. benchflow.py's _load_json still raises on a 5000-digit int, so the reply on that thread does not hold.
• 56efddf: the artifact-field notes and the [0, 1] rule from the review bodies.

Merge order with #104: each passes alone, but on main + #104 + #111 tests/test_task_formats.py::test_migrated_dir_with_task_toml_and_task_md_runs_everywhere fails with harbor: no Harbor result.json (exit 0), because #104's DockerCLI fake answers harbor with exit 0 and writes no result.json. Whichever lands second makes the fake write a passing <--jobs-dir>/job/trial/result.json, as this PR's _no_harbor does.

[napetrov]

Thanks, Alexander. Point by point:

1. Multi-step step-2 setup.sh exit 3 -> errored: verified your dd6d670 and kept it as is. test_a_step_that_raised_errors_a_multi_step_trial passes, and usage is summed over step_results[].agent_result.
2. Verifier timeout_sec=5 -> fixed in 09c9a96. Harbor raises VerifierTimeoutError with no verifier_result; read_harbor_job now scores that cell 0 with metrics.verifier_timeout, not errored, so it stays in the pass-rate denominator. Docs table row added. Regression test added and fails on 56efddf.
3. a75c817 (drop _load_json from command.py): verified, no callers left in that file, kept.
4. 56efddf (artifact-field notes, [0, 1] rule): verified, kept.
5. The benchflow.py _load_json claim in my earlier reply was wrong: it caught only JSONDecodeError/OSError. Fixed in 04e17d2, which catches ValueError as the Harbor reader does. Regression test added.
6. Merge order with refactor(harnesses): docker-* solver, sandbox and probe on the process runner #104: no change in this PR. The harbor: no Harbor result.json failure comes from refactor(harnesses): docker-* solver, sandbox and probe on the process runner #104's DockerCLI fake, so that is where the fix belongs. I have not touched refactor(harnesses): docker-* solver, sandbox and probe on the process runner #104.

Tests: test_harbor_results.py and test_benchflow_harness.py pass (194). Full suite: the remaining failures are in judge/LLM modules that fail on the missing litellm module in this environment, not in Harbor or BenchFlow paths.

[antonwolfy]

Review at 04e17d2. The new multi-step, verifier-timeout and ValueError changes hold up on the cases I ran, and the grading still never reads the exit code or stdout. Four things are worth fixing before merge: three inline and one outside the diff. One nit inline.

What I ran, on a Linux review host at 04e17d2:

• make ci-local with docker: 23 passed, 0 failed, 3 skipped (test-latest, docs deploy, semgrep). Changed-line coverage gate passed.
• Reproductions through read_harbor_job with hand-written trial files, no model calls.
• A 61-mutant run over read_harbor_job, harbor_usage, _step_usage, _harbor_reward, _number, _count and the Harbor branch of run, 16 of which pass the five affected test files. Most survivors are equivalent or cosmetic; the ones worth a test are in the inline comment on the tests.

One more, outside this diff:

1. docs/coding-harnesses.md:912-914 contradicts the new Harbor rule. It says "An agent that exceeds its timeout, or a verifier that does, is a failure (reward 0, returncode 124), not an error: otherwise an arm that hangs would score above one that fails fast." For a Harbor cell, AgentTimeoutError is now errored and out of pass_rate (the PR description names this as a follow-up), and the list of metrics.error causes at :908-911 does not mention Harbor. A user comparing a Harbor arm with docker-* reads a hanging Harbor arm as missing data, and it ranks above an arm that fails fast, which is the effect that paragraph warns against. Suggestion: say there that a Harbor trial that raised, an agent timeout included, is errored, while a lone verifier timeout is a 0.

Checked, no issues

• Verifier timeout: a lone VerifierTimeoutError at the top or on every step gives reward 0.0 with verifier_timeout, and stays in the denominator.
• Odd shapes: non-dict step_results entries, a dict-valued step_results, a list or string exception_info, a bool reward and a 10**400 reward or count do not crash and never come out as a pass.
• Schema: task_runs.v1 metrics is an open object, so the new keys need no bump. No fixture holds a host, path or secret (gitleaks clean).
• Prior threads: the relative output dir, the [0, 1] rule, the artifact-field text and the overlong-integer reads are fixed at this head.

[antonwolfy]

Review at 6fb20af. The three fixes from my last review hold: the label now skips a verifier timeout, the edge cases are mostly pinned, and the fragment states the verifier-timeout and [0, 1] rules. Two things are worth fixing before merge: one doc paragraph outside the diff, and one test gap inline. Two earlier threads stay open.

What I ran, on a Linux review host at 6fb20af:

• make ci-local with docker: 23 passed, 0 failed, 3 skipped. pytest: 5230 passed, 1 skipped. Changed-line coverage 100%.
• Reproductions through read_harbor_job with hand-written trial files, no model calls.
• A 39-mutant run over read_harbor_job, _is_verifier_timeout, _step_usage, harbor_usage and _harbor_reward against test_harbor_results.py: the survivors that matter are the two named under the open threads and the label order inline.

One more, outside this diff:

1. docs/coding-harnesses.md:912-914 still contradicts the Harbor rule. This is the point from my earlier review, not fixed at this head. It says "An agent that exceeds its timeout, or a verifier that does, is a failure (reward 0, returncode 124), not an error". For a Harbor cell, an AgentTimeoutError in exception_info is errored and out of pass_rate (read_harbor_job exempts only a lone VerifierTimeoutError). A trial with exception_info of type AgentTimeoutError and reward 0.0 returns None with error='harbor: AgentTimeoutError: x'. The errored list at :908-911 does not name Harbor either. Suggestion: say there that a Harbor trial that raised, an agent timeout included, is errored, a lone verifier timeout is a 0, and only dokimi's own kill of harbor gives reward 0 and returncode 124.

Still open from my earlier review:

• tests/test_harbor_results.py:450: two of the listed mutants still pass the file: str(...) removed around exception_type (command.py:288; the exception_type: 42 case checks only the f-string in metrics.error, so add assert metrics["harbor_exception"] == "42"), and the isinstance(s.get("agent_result"), dict) filter removed (command.py:334; add steps [{"agent_result": {...tokens...}}, {"agent_result": None}] and expect the first step's numbers).
• changelog.d/20261018_120000_napetrov_harbor-result-json.md:15: this line is still 96 columns, the others wrap at 80.

Checked, no issues

• Label: a verifier timeout beside a real exception, at the top or on a step, in either order, gives the real exception's type and message. all( to any( and dropping the step scan from raised are caught.
• Lone verifier timeout: top-level or on steps only gives 0.0 with verifier_timeout, no error.
• Edge cases now pinned: the if not usages guard, the step isinstance(s, dict) filter, the top-level agent_result beside steps, OSError on a directory, str(...) on exception_message and the reward is not None guard each fail a test when removed.
• Fixed threads: the extra blank line in test_benchflow_harness.py is gone.

[napetrov]

Thanks, Alexander. Point by point, at 2f75c52:

1. Doc contradiction in docs/coding-harnesses.md: fixed. The timeout rule is now scoped to the command harness, and a Harbor trial that raised (an agent timeout included) is errored via metrics.error, with a lone verifier timeout scored 0.
2. test_harbor_results.py mutants: str(...) around exception_type is now pinned by test_a_non_string_exception_type_is_kept_as_text_in_the_label (harbor_exception == "42"). The isinstance(s.get("agent_result"), dict) filter is pinned by test_a_step_without_an_agent_result_does_not_erase_the_other_steps_usage. Each of the three mutants (label order, str, filter) now fails exactly one new test.
3. Changelog line over 80 columns: rewrapped.

Tests: tests/test_harbor_results.py, test_task_harnesses.py, test_task_formats.py, test_tasks_command.py pass (258). ruff check passes.

[xaleryb]

Re-checked at 3515190 with real Harbor 0.20.0 through dokimi tasks run (oracle agent: single-step pass, multi-step pass, multi-step with step-2 setup.sh exit 3, verifier timeout, agent timeout, relative --output-dir) and the earlier reviewers' inputs re-run against read_harbor_job. Every earlier point passes at this head, and no new findings came up. The red Static checks job is the license-header check on dokimi/repos/errors.py, which this PR does not touch. Main fails the same check.

[antonwolfy]

LGTM. The three open threads are fixed at this head, and the grading still never reads the exit code or stdout. One nit inline.

What I ran, on a Linux review host at fb60a08:

• make ci-local with docker: 28 passed, 3 skipped, 2 failed. Both failures are the gitleaks job failing to download its binary; gitleaks detect over the PR's commits finds no leaks. pytest: 6405 passed, 1 skipped. Changed-line coverage gate passed.
• A mutation run over the three newest tests plus three more mutants (reward upper bound, all( to any(, the reward_after_exception assignment): all six are killed.
• Hand-written trial files through read_harbor_job, no model calls: mixed exceptions, steps without agent_result, a second directory holding a result.json, a symlinked trial directory. None scores an errored trial as a pass or raises out of run().

Checked, no issues

• Label order: swapping the trial and step exceptions in raised (command.py:277) fails test_a_trial_exception_is_labelled_before_a_step_exception.
• Text and filter mutants: removing str(...) around exception_type and the isinstance(s.get("agent_result"), dict) filter each fail one new test.
• Docs and changelog: the fragment wraps at 80 columns, and the errored-cell paragraph now scopes the timeout rule and names the Harbor case.

[antonwolfy]

LGTM. I pushed three commits on top of fb60a08:

• 4edeb4b: dokimi/docs/coding-harnesses.md:438: "their sum" now reads "input + output", matching prompt + completion in the code.
• 82c64ed: merges main. #104 landed, and the errored-cell timeout paragraph in docs/coding-harnesses.md conflicted. The resolution keeps main's paragraph, which now covers docker-* too, and puts this branch's Harbor sentence right after the general rule ("A Harbor trial that raised, an agent timeout included, is errored (metrics.error); only a lone verifier timeout is scored 0."), dropping "Under the command harness".
• 6732835: the DockerCLI fake in tests/docker_cli_fake.py answers harbor ... --jobs-dir <d> with a passing trial result.json, as _no_harbor does. Without it, test_task_formats.py::test_migrated_dir_with_task_toml_and_task_md_runs_everywhere failed on the merge with no Harbor result.json, the order issue @xaleryb noted.

make ci-local at 6732835 with docker: 28 passed, 0 failed by the change (the 2 gitleaks rows could not download the tool), 4 skipped. CI here is green.