Context
Six nights of SkillOpt-Sleep against real task sets on two of my own skills (an essay skill; a router with 24 candidate targets). The engine never crashed and the gate never behaved contrary to its own spec — but the headline verdict was never self-sufficient. Every night the real story only emerged from the per-task diff in evidence.jsonl. Two structural properties explain that, and both look addressable.
Problem 1 — the gate has no "no task regressed" condition
consolidate.py accepts on cand_score > base_score, where both are slice means. Nothing checks whether an individual val task got worse.
Night 1, real numbers (gate_metric="mixed", val = 4 tasks):
| task |
baseline |
candidate |
essay_w3 |
soft 0.55 |
1.00 |
essay_wl2 |
soft 0.91 |
0.82 ← regressed |
| two others |
1.00 |
1.00 |
Aggregate 0.682 → 0.852, verdict accept_new_best. The accepted rule had made the model drop a required scriptural citation from essay_wl2 — a genuine regression, invisible in the mean.
Worse, the +0.17 was not a quality gain at all: it came from the rule satisfying a check of mine that was too strict (a heading-format artifact). So the gate accepted an edit that damaged one task and improved none, and its report said accept_new_best.
This is the gate-side sibling of #154, and a counterexample to the assumption in #67 that "the held-out gate rejected that candidate, which is exactly the right outcome" — here it accepted one.
Suggestion: an opt-in gate_no_regression: true that rejects a candidate if any val task's score drops, regardless of the mean. For small val slices this is the difference between a usable and a misleading signal.
Problem 2 — improvements are never attributed to the rule that caused them
The gate checks that the mean moved, never which task moved or whether it belongs to the class the rule addresses.
Night 4: the optimizer proposed a genuinely correct rule (prashna vs muhurta disambiguation). Verdict accept_new_best, 0.900 → 1.000. But the entire delta came from tajaka_sixty_hour, an unrelated task the rule says nothing about — and re-running it under the accepted skill gave 1/5, i.e. that night's pass was luck. Meanwhile the task the rule actually fixes was in train, so the gate could not see the real effect at all (independently: 0/1 → 5/5 with the rule).
Night 6 repeated the pattern with a different rule.
So in 3 of 3 accepts across six nights, the reported reason for acceptance was not the actual reason. The rule was right twice and wrong once — but the report gave no way to tell those cases apart.
Suggestion: record, per accepted candidate, which val tasks changed state, and surface that in report.md next to the delta. Even without changing the accept criterion, accepted because: essay_w3 0→1; regressed: essay_wl2 0.91→0.82 would have made all three nights immediately readable. Tasks already carry tags, so a coarse "did any task sharing the rule's class improve?" signal is cheap.
Why this matters beyond my setup
The safety story of the project rests on the gate ("only changes that help are kept"). That holds for the mean, which is a weaker guarantee than it reads as — and the two failure modes above are exactly the ones a user cannot detect without reading raw evidence. I ended up writing an external harness that re-runs targeted tasks with and without a candidate rule, plus guards for neighbouring routes, before trusting any accept.
Happy to prototype either piece if there is interest — the no-regression flag looks like a small, well-contained change to _gate_apply.
Environment
main @ fdeebaf, Python 3.11, azure_openai backend in openai_compatible mode against a local proxy; targets gemini-3.1-pro and gemini-3.5-flash. Task sets were real (mined from my own consultation artifacts), 12–72 tasks, val 4–30.
Context
Six nights of SkillOpt-Sleep against real task sets on two of my own skills (an essay skill; a router with 24 candidate targets). The engine never crashed and the gate never behaved contrary to its own spec — but the headline verdict was never self-sufficient. Every night the real story only emerged from the per-task diff in
evidence.jsonl. Two structural properties explain that, and both look addressable.Problem 1 — the gate has no "no task regressed" condition
consolidate.pyaccepts oncand_score > base_score, where both are slice means. Nothing checks whether an individual val task got worse.Night 1, real numbers (
gate_metric="mixed", val = 4 tasks):essay_w3essay_wl2Aggregate
0.682 → 0.852, verdictaccept_new_best. The accepted rule had made the model drop a required scriptural citation fromessay_wl2— a genuine regression, invisible in the mean.Worse, the +0.17 was not a quality gain at all: it came from the rule satisfying a check of mine that was too strict (a heading-format artifact). So the gate accepted an edit that damaged one task and improved none, and its report said
accept_new_best.This is the gate-side sibling of #154, and a counterexample to the assumption in #67 that "the held-out gate rejected that candidate, which is exactly the right outcome" — here it accepted one.
Suggestion: an opt-in
gate_no_regression: truethat rejects a candidate if any val task's score drops, regardless of the mean. For small val slices this is the difference between a usable and a misleading signal.Problem 2 — improvements are never attributed to the rule that caused them
The gate checks that the mean moved, never which task moved or whether it belongs to the class the rule addresses.
Night 4: the optimizer proposed a genuinely correct rule (prashna vs muhurta disambiguation). Verdict
accept_new_best,0.900 → 1.000. But the entire delta came fromtajaka_sixty_hour, an unrelated task the rule says nothing about — and re-running it under the accepted skill gave 1/5, i.e. that night's pass was luck. Meanwhile the task the rule actually fixes was intrain, so the gate could not see the real effect at all (independently: 0/1 → 5/5 with the rule).Night 6 repeated the pattern with a different rule.
So in 3 of 3 accepts across six nights, the reported reason for acceptance was not the actual reason. The rule was right twice and wrong once — but the report gave no way to tell those cases apart.
Suggestion: record, per accepted candidate, which val tasks changed state, and surface that in
report.mdnext to the delta. Even without changing the accept criterion,accepted because: essay_w3 0→1; regressed: essay_wl2 0.91→0.82would have made all three nights immediately readable. Tasks already carrytags, so a coarse "did any task sharing the rule's class improve?" signal is cheap.Why this matters beyond my setup
The safety story of the project rests on the gate ("only changes that help are kept"). That holds for the mean, which is a weaker guarantee than it reads as — and the two failure modes above are exactly the ones a user cannot detect without reading raw evidence. I ended up writing an external harness that re-runs targeted tasks with and without a candidate rule, plus guards for neighbouring routes, before trusting any accept.
Happy to prototype either piece if there is interest — the no-regression flag looks like a small, well-contained change to
_gate_apply.Environment
main@ fdeebaf, Python 3.11,azure_openaibackend inopenai_compatiblemode against a local proxy; targetsgemini-3.1-proandgemini-3.5-flash. Task sets were real (mined from my own consultation artifacts), 12–72 tasks, val 4–30.