Skip to content

Remove the round score; cap epochs by task budget; delete the diversity criterion - #96

Merged
rasros merged 2 commits into
mainfrom
remove-round-score
Aug 17, 2026
Merged

Remove the round score; cap epochs by task budget; delete the diversity criterion#96
rasros merged 2 commits into
mainfrom
remove-round-score

Conversation

@rasros

@rasros rasros commented Aug 17, 2026

Copy link
Copy Markdown
Owner

The blended round score was removed from ranking and from staleness earlier, but it still named every file, picked the run's "best", decided which LLM seeds the archive kept, and drove --resume-top. It kept producing wrong answers — three separate times today — so it is gone entirely.

The tell: in one run the best-named file on disk was 0.033325_139621.svg while the artifact the evaluator actually chose read 0.052259 by that same number. Anyone opening the directory would pick the wrong file.

What replaces it

score is a validity marker. VALID_SCORE = 0.0 for a measured candidate, INVALID_SCORE for one that failed. It carries no magnitude, because with the measures traded off by dominance there is no single number that orders candidates mid-run — inventing one is exactly what the round score was.

The evaluator is the only score. It lives in metrics[FRONT_SCORE], on the nodes the panel has actually seen, and best_node is now set only where the panel speaks: at each epoch boundary and at the end. There is deliberately no "best" in between.

No measure is privileged. COLOUR_WEIGHT, EDGE_WEIGHT, SHAPE_WEIGHT and round_score are deleted. Dominance compares component by component, so a weight between the measures could not change a single verdict anyway.

Everything else that leaned on the magnitude now uses the relation:

was now
filenames {score}_{id} {id}, or eval{panel}_{id} for evaluated nodes
best_node by lowest score the evaluator's pick at the last boundary
seed archive evicts the worst score evicts the entry the rest of the archive beats most often
initial pool sorted by score unsorted — a pool is a set
--resume-top by leading number by evaluator score where present, else newest ids

Fewer files

A run wrote 106,640 node files. The drawing is now written only for candidates that reached the top tier; the lineage row is written for everything, so the record stays complete.

Behaviour changes worth knowing

The final fallback changed. If the evaluator fails and no boundary ever ran it, there is no scored candidate to fall back on, so a top-tier member is written instead — any unbeaten candidate is as defensible as another, and it beats losing the artifact to a scorer error at shutdown.

--resume-top on a directory with no evaluated nodes now keeps the newest ids rather than the best proxy scores. That is a real change in which nodes resume, and it is honest: nothing in such a directory has been scored by anything.


Second commit: epoch budget in, diversity criterion out

--epoch-max-tasks

A ceiling on how long one epoch may run before the evaluator ranks the front again. Off by default.

It exists because staleness cannot do this job. Over 1,333 top-tier entries in one run the gap between them had a median of 68 tasks, a 99th percentile of 285, and a maximum of 500 — so patience at 500 fires at the genuine tail and almost never early, which is correct behaviour for a staleness detector. But reaching that tail took 150,800 tasks, all of it with no evaluator in the loop. Lowering patience does not buy supervision either: firing it ends the epoch, so with --epochs 4 a tighter patience just ends the run sooner.

This is admittedly a blunt instrument — a number that has to be guessed. The automated form is to run the panel periodically inside an epoch and end the epoch when the panel's best stops improving, which detects proxy drift directly instead of proxying it with a task count. That is a larger change and is not here.

--epoch-diversity deleted

Its threshold was a ratio to the epoch's opening reading, and that reading is not comparable between epochs:

run epoch first reading final / first
v6 0 1.000 0.03
v6 1 0.027 3.34
v6 2 0.087 1.58
v7 1 0.083 1.20
v7 2 0.086 1.30

Epoch 0 opens at 1.000, which is not a measurement at all — it is the len(sigs) < 2 sentinel meaning "too little data". Every later epoch opens in a trough and ends above its own baseline, so the rule fires readily in epoch 0 against a fake reference and can never fire afterwards. Same defect that removed --epoch-variance, same cause.

It was also measuring the wrong thing: SimHash over the SVG source text, so two candidates that render identically but differ in attribute order or numeric precision read as diverse.

pool_diversity stays as a reported statistic in stats.csv — it is informative, it just cannot be a stopping rule. should_diversify is gone from the strategy and the protocol.

@rasros rasros changed the title Remove the round score so the evaluator is the only score Remove the round score; cap epochs by task budget; delete the diversity criterion Aug 17, 2026
@rasros
rasros merged commit 28d8655 into main Aug 17, 2026
1 check passed
@rasros
rasros deleted the remove-round-score branch August 17, 2026 20:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant