Record the evaluator's best; drop dead stats columns; count patience in checks - #100
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
best_score was empty on every row
Rewiring
best_nodeto come from the evaluator left_note_acceptedalways passingis_new_best=False, so nothing told the collector a best had been found. The panel's verdict reachedbest_nodeand stopped there:best_scorecame out empty on all 2112 rows of a 45-minute run, and the dashboard and plot script both read it.on_evaluator_bestnow records it, from both places the evaluator speaks — the periodic check and the epoch boundary. It is the only event that can move the best, because the evaluator's score is the only score.Four columns removed
Measured on that run, these carried one value across all 2112 rows:
epoch_patienceepoch_max_tasksseeds_targetllm_in_flightseeds_completedllm_call_countwithin an epoch, counted a second wayA setting repeated on every row cannot be told apart from a measurement that happened not to move, which is the confusion worth removing. All of them remain on
SearchStats, where the dashboard reads them to draw progress against — they were never measurements, only ever context.Fifteen columns remain, each genuinely time-varying.
plot_run.pyloses its two config readouts and the seeds-per-epoch line.Two tests pin the invariant so config cannot drift back in, and one covers the best_score fix.
Second commit: patience in checks, and the round-score wording
--epoch-eval-patiencecounts evaluator checksIt counted generations, and a generation is a poor unit: it is 100 accepted candidates, so its size in tasks moves with the acceptance rate, which itself drifts as the pool converges — and with
--pool-size. Measured on one run it came to 18.8 generations per check.Worse, it had a floor nothing announced. The counter only resets when a check finds an improvement, so any threshold below one check interval's worth of generations fires before a check can ever intervene: the epoch would end at exactly that many generations regardless of quality.
A check is the evaluator's own observation, which is the thing whose failures are being counted.
2now means two consecutive checks without a better candidate, whatever the cadence, pool size or acceptance rate — and no value of it can trip before a check happens.On what to set: measured on a 45-minute run, the evaluator's best arrived at the first check and 73 further checks over 145,000 tasks never beat it, while the front it was shown degraded 40%. On that evidence two or three is where to start; it stays unset until a run confirms it.
The round score is gone from the prose too
round_scorewas deleted, but eleven comments and a log line still described "the round score" as though it existed, including the startup line every run prints. They now name what actually happens: measures traded off by dominance, with the evaluator as the only score.MOMENT_WEIGHTwent with them — defined, documented, and referenced nowhere since the weights were deleted.