From f4de6de9346cfa6f2f3e0ff6c7fb1e01a1dea660 Mon Sep 17 00:00:00 2001 From: Phil Leggetter Date: Wed, 2 Sep 2026 14:35:15 +0100 Subject: [PATCH] Let the publish job push when the branch has moved, and commit the counter MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two defects, both found because the 1 September run is red on a page that links to it as evidence. 115 of its 116 jobs succeeded; the one that failed was `publish-results`, after every eval had been measured. The publish job checks out shallow, so when its push is rejected — a matrix takes six hours and `main` moves — the retry loop's `git rebase origin/main` has no merge base to work from and all three attempts fail. Full history fixes it. This cannot repair the run that found it: re-running a job checks out the original commit and the original workflow file, so the fix only protects the runs after it. `results/totals.json` was never staged. `publish-snapshot` writes it and the page reads it for the "runs recorded" counter, so every CI publish left it at whatever the last hand-publish had written. It moved from 222 to 348 today, by hand, and nothing in CI would ever have moved it. Refs https://github.com/hookdeck/evals/issues/74 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MQzUoMAwEBJWpEGVvVzSjK --- .github/workflows/eval-refresh.yml | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/.github/workflows/eval-refresh.yml b/.github/workflows/eval-refresh.yml index 929d14d..4f922f4 100644 --- a/.github/workflows/eval-refresh.yml +++ b/.github/workflows/eval-refresh.yml @@ -476,6 +476,16 @@ jobs: # skills/ symlinks into the agent-skills submodule; without it a # skills experiment installs nothing and silently scores as baseline. submodules: recursive + # Full history, because the commit step below may have to rebase onto + # a branch that moved while the matrix ran — and did, on 1 September. + # A shallow clone has no merge base with `origin/main`, so + # `git rebase` fails and the retry loop exhausts its attempts with + # nothing it can do. That run published nothing and, worse, is red on + # a page that links to it as evidence: 115 of its 116 jobs succeeded + # and the failure was this step. Re-running it cannot fix that, + # because a re-run checks out the original commit and the original + # workflow file, so the fix only protects the runs after it. + fetch-depth: 0 - name: Install pnpm uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0 @@ -585,7 +595,12 @@ jobs: # The published contract and its history. Staged explicitly rather # than with `git add -A`, so a stray file a run happens to leave in # the working tree cannot ride along into a published commit. - git add results/latest.json results/index.json results/runs + # `totals.json` too: `publish-snapshot` writes it — it is the "runs + # recorded" counter the page shows — and it was never staged here, so + # every CI publish left it at whatever the last hand-publish had + # written. Found on 2 September, when a hand-publish moved it from 222 + # to 348 and nothing in CI ever would have. + git add results/latest.json results/index.json results/totals.json results/runs if git diff --cached --quiet; then echo "No eval result changes to commit"