Repository navigation
runner time: drop rerun copies and jobs that failed while they waited (follow-up to #35) - #36
Merged
Merged
Conversation
Re-running failed jobs copies a run's passed jobs into the new attempt as new check runs with the old start and end (2026-10-21: one db2 shard three times); 1,156 such copies added 117 runner hours. Jobs can also fail while they wait and still get an 8-minute span with runner_id 0 and no steps (2026-10-07: 28 compile jobs). summarize counts a job once per run, name, start and end, and the step check now covers every job that did not succeed; measure_run skips any unsuccessful job without a runner. With both, every day 2026-09-15..10-30 peaks at exactly the 20-runner pool, so the 'more runners than the pool' reading in the previous commit was wrong: comments and the alert say so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A heavy run made the steps query return 502/504 six times running and stopped the pass. A failing batch now retries one run at a time; a run that still fails is logged and its jobs stay unchecked (so they are counted). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#35 merged before these two commits landed on its branch. One of its commits (a4a2e1e) said the account ran up to 65 jobs at once; that was wrong. This PR fixes the cause and the wording.
Rerun copies. Re-running failed jobs copies a run's passed jobs into the new attempt as new check runs, with the old start and end. On 2026-10-21 one
db2 2/4job showed up three times, identical to the second. 1,156 such copies added about 117 runner hours.summarizenow counts a job once per run, name, start and end.Jobs that failed while waiting. A job can fail without ever getting a runner and still get a span. On 2026-10-07, 28
compilejobs failed for 8 minutes withrunner_id0 and no steps. The step check now covers every job that didn't succeed, not just cancelled ones, andmeasure_runskips any unsuccessful job without a runner.Result. Every day from 2026-09-15 to 2026-10-30 peaks at exactly the 20-runner pool. The comments and the over-pool alert now say that a measured day above the pool points at the counting, not at a bigger pool.
Also: a step batch that GitHub times out on (502/504) drops to one run per query, and a run that still fails is logged and skipped. Before this, one heavy run stopped the whole pass.
runner_exact.jsonwith these numbers is on thedashboardbranch (bc0f923). Rendered locally with this code against it: 35 days draw as "measured (job times)". On busy days the pool is at its limit for 13–22 h (10-21, 10-24, 10-25 and 10-30 over 21 h).Tests: 165 passed, 7 skipped (the summarize test now covers a rerun copy and a job that failed while waiting).
🤖 Generated with Claude Code