Skip to content

Record the evaluator's best; drop dead stats columns; count patience in checks - #100

Merged
rasros merged 3 commits into
mainfrom
stats-columns
Aug 18, 2026
Merged

Record the evaluator's best; drop dead stats columns; count patience in checks#100
rasros merged 3 commits into
mainfrom
stats-columns

Conversation

@rasros

@rasros rasros commented Aug 18, 2026

Copy link
Copy Markdown
Owner

best_score was empty on every row

Rewiring best_node to come from the evaluator left _note_accepted always passing is_new_best=False, so nothing told the collector a best had been found. The panel's verdict reached best_node and stopped there: best_score came out empty on all 2112 rows of a 45-minute run, and the dashboard and plot script both read it.

on_evaluator_best now records it, from both places the evaluator speaks — the periodic check and the epoch boundary. It is the only event that can move the best, because the evaluator's score is the only score.

Four columns removed

Measured on that run, these carried one value across all 2112 rows:

column why it goes
epoch_patience configuration
epoch_max_tasks configuration
seeds_target configuration
llm_in_flight a live gauge; in a finished file it samples a quantity that was oscillating
seeds_completed llm_call_count within an epoch, counted a second way

A setting repeated on every row cannot be told apart from a measurement that happened not to move, which is the confusion worth removing. All of them remain on SearchStats, where the dashboard reads them to draw progress against — they were never measurements, only ever context.

Fifteen columns remain, each genuinely time-varying. plot_run.py loses its two config readouts and the seeds-per-epoch line.

Two tests pin the invariant so config cannot drift back in, and one covers the best_score fix.


Second commit: patience in checks, and the round-score wording

--epoch-eval-patience counts evaluator checks

It counted generations, and a generation is a poor unit: it is 100 accepted candidates, so its size in tasks moves with the acceptance rate, which itself drifts as the pool converges — and with --pool-size. Measured on one run it came to 18.8 generations per check.

Worse, it had a floor nothing announced. The counter only resets when a check finds an improvement, so any threshold below one check interval's worth of generations fires before a check can ever intervene: the epoch would end at exactly that many generations regardless of quality.

A check is the evaluator's own observation, which is the thing whose failures are being counted. 2 now means two consecutive checks without a better candidate, whatever the cadence, pool size or acceptance rate — and no value of it can trip before a check happens.

On what to set: measured on a 45-minute run, the evaluator's best arrived at the first check and 73 further checks over 145,000 tasks never beat it, while the front it was shown degraded 40%. On that evidence two or three is where to start; it stays unset until a run confirms it.

The round score is gone from the prose too

round_score was deleted, but eleven comments and a log line still described "the round score" as though it existed, including the startup line every run prints. They now name what actually happens: measures traded off by dominance, with the evaluator as the only score.

MOMENT_WEIGHT went with them — defined, documented, and referenced nowhere since the weights were deleted.

@rasros rasros changed the title Record the evaluator's best, and drop the stats columns that never move Record the evaluator's best; drop dead stats columns; count patience in checks Aug 18, 2026
@rasros
rasros merged commit fa9ffac into main Aug 18, 2026
1 check passed
@rasros
rasros deleted the stats-columns branch August 18, 2026 08:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant