feat(refiner): the refine loop now gates on blast radius too (G3) - #5
Merged
Conversation
`refiner.py` retried until the band-aid blocked the exploit. A refinement that widened the policy
far enough to catch the exploit had nothing telling it it went too far, so the loop would happily
accept a rule that denies production. With a traffic sample, "works" now means two things: the
policy blocks the exploit AND stays under the blast-radius threshold.
An over-broad candidate is fed back as `over_block` — a diagnosis the refine agent already
understands ("blocked the LEGIT request too. Narrow the rule") — with the measured rate and the top
blocked paths attached, so it knows what it over-blocked rather than guessing.
Measured where the policy is already attached. The refiner has just confirmed the candidate is
enforcing, so `simulate.measure_attached` replays through that LB instead of paying a second attach
and propagation wait on another one: slower-but-correct per the decision, but one round trip per
attempt rather than two. `_score` is now shared by `simulate_policies` and `measure_attached`, so
the standalone run and the refine gate cannot drift on what "too broad" means.
The sample comes from `VPCOPILOT_SIM_LOGS` or an explicit `records=`. With neither, the gate is
skipped and the loop behaves exactly as it did before — pinned by a test that fails if anything is
replayed without a sample. An unreadable sample logs a warning and refines without the gate rather
than failing a working loop.
Tests cover both directions: an over-broad candidate is rejected and refined (and the agent is
handed the blast radius), a surgical one passes on the first attempt, an unfixable over-block gives
up honestly with a reason, the threshold is configurable, and the no-sample path replays nothing.
250 pass, coverage 73%.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Roadmap G3. The refiner retried until the band-aid blocked the exploit — and nothing told it
when a refinement had gone too far. A pass that widens the rule enough to catch the exploit was
accepted even if it would deny production. G2 gives it the missing half.
What changed
With a traffic sample, "works" now means two things: blocks the exploit and stays under the
blast-radius threshold.
An over-broad candidate is fed back as
over_block— a diagnosis the refine agent alreadyunderstands ("blocked the LEGIT request too. Narrow the rule so it blocks only the exploit") —
with the measured rate and top blocked paths attached, so it knows what it over-blocked rather
than guessing.
Slower-but-correct, but only one round trip
Per the decision, the check runs per attempt rather than once at the end. It does not cost a
second LB: the refiner has just confirmed the candidate is enforcing, so
measure_attachedreplaysthrough that LB — no second attach, no second propagation wait.
_scoreis now shared bysimulate_policiesandmeasure_attached, so the standalonevpcopilot simulateand the refine gate cannot drift on what "too broad" means.No-regression
The sample comes from
VPCOPILOT_SIM_LOGSor an explicitrecords=. With neither, the gate isskipped and the loop behaves exactly as before — pinned by a test that raises if anything is
replayed without a sample. An unreadable sample logs a warning and refines without the gate rather
than turning a working loop into a failure.
Tests
Both directions, written before the change:
blast_radius250 pass · coverage 73% ·
ruffclean.Surfaces
No new command — the gate rides the existing refine path.
VPCOPILOT_SIM_LOGSdocumented in.env.exampleanddocs/USAGE.md; the console's apply result shows the blast radius when present.🤖 Generated with Claude Code