Remove the round score; cap epochs by task budget; delete the diversity criterion - #96
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The blended round score was removed from ranking and from staleness earlier, but it still named every file, picked the run's "best", decided which LLM seeds the archive kept, and drove
--resume-top. It kept producing wrong answers — three separate times today — so it is gone entirely.The tell: in one run the best-named file on disk was
0.033325_139621.svgwhile the artifact the evaluator actually chose read 0.052259 by that same number. Anyone opening the directory would pick the wrong file.What replaces it
scoreis a validity marker.VALID_SCORE = 0.0for a measured candidate,INVALID_SCOREfor one that failed. It carries no magnitude, because with the measures traded off by dominance there is no single number that orders candidates mid-run — inventing one is exactly what the round score was.The evaluator is the only score. It lives in
metrics[FRONT_SCORE], on the nodes the panel has actually seen, andbest_nodeis now set only where the panel speaks: at each epoch boundary and at the end. There is deliberately no "best" in between.No measure is privileged.
COLOUR_WEIGHT,EDGE_WEIGHT,SHAPE_WEIGHTandround_scoreare deleted. Dominance compares component by component, so a weight between the measures could not change a single verdict anyway.Everything else that leaned on the magnitude now uses the relation:
{score}_{id}{id}, oreval{panel}_{id}for evaluated nodesbest_nodeby lowest score--resume-topby leading numberFewer files
A run wrote 106,640 node files. The drawing is now written only for candidates that reached the top tier; the lineage row is written for everything, so the record stays complete.
Behaviour changes worth knowing
The final fallback changed. If the evaluator fails and no boundary ever ran it, there is no scored candidate to fall back on, so a top-tier member is written instead — any unbeaten candidate is as defensible as another, and it beats losing the artifact to a scorer error at shutdown.
--resume-topon a directory with no evaluated nodes now keeps the newest ids rather than the best proxy scores. That is a real change in which nodes resume, and it is honest: nothing in such a directory has been scored by anything.Second commit: epoch budget in, diversity criterion out
--epoch-max-tasksA ceiling on how long one epoch may run before the evaluator ranks the front again. Off by default.
It exists because staleness cannot do this job. Over 1,333 top-tier entries in one run the gap between them had a median of 68 tasks, a 99th percentile of 285, and a maximum of 500 — so patience at 500 fires at the genuine tail and almost never early, which is correct behaviour for a staleness detector. But reaching that tail took 150,800 tasks, all of it with no evaluator in the loop. Lowering patience does not buy supervision either: firing it ends the epoch, so with
--epochs 4a tighter patience just ends the run sooner.This is admittedly a blunt instrument — a number that has to be guessed. The automated form is to run the panel periodically inside an epoch and end the epoch when the panel's best stops improving, which detects proxy drift directly instead of proxying it with a task count. That is a larger change and is not here.
--epoch-diversitydeletedIts threshold was a ratio to the epoch's opening reading, and that reading is not comparable between epochs:
Epoch 0 opens at 1.000, which is not a measurement at all — it is the
len(sigs) < 2sentinel meaning "too little data". Every later epoch opens in a trough and ends above its own baseline, so the rule fires readily in epoch 0 against a fake reference and can never fire afterwards. Same defect that removed--epoch-variance, same cause.It was also measuring the wrong thing: SimHash over the SVG source text, so two candidates that render identically but differ in attribute order or numeric precision read as diverse.
pool_diversitystays as a reported statistic in stats.csv — it is informative, it just cannot be a stopping rule.should_diversifyis gone from the strategy and the protocol.