Skip to content

feat(refiner): the refine loop now gates on blast radius too (G3) - #5

Merged
henleda merged 1 commit into
mainfrom
feat-g3-simulation-feeds-refiner
Jul 27, 2026
Merged

feat(refiner): the refine loop now gates on blast radius too (G3)#5
henleda merged 1 commit into
mainfrom
feat-g3-simulation-feeds-refiner

Conversation

@henleda

@henleda henleda commented Jul 27, 2026

Copy link
Copy Markdown
Owner

Roadmap G3. The refiner retried until the band-aid blocked the exploit — and nothing told it
when a refinement had gone too far. A pass that widens the rule enough to catch the exploit was
accepted even if it would deny production. G2 gives it the missing half.

What changed

With a traffic sample, "works" now means two things: blocks the exploit and stays under the
blast-radius threshold.

An over-broad candidate is fed back as over_block — a diagnosis the refine agent already
understands ("blocked the LEGIT request too. Narrow the rule so it blocks only the exploit") —
with the measured rate and top blocked paths attached, so it knows what it over-blocked rather
than guessing.

Slower-but-correct, but only one round trip

Per the decision, the check runs per attempt rather than once at the end. It does not cost a
second LB: the refiner has just confirmed the candidate is enforcing, so measure_attached replays
through that LB — no second attach, no second propagation wait.

_score is now shared by simulate_policies and measure_attached, so the standalone vpcopilot simulate and the refine gate cannot drift on what "too broad" means.

No-regression

The sample comes from VPCOPILOT_SIM_LOGS or an explicit records=. With neither, the gate is
skipped and the loop behaves exactly as before
— pinned by a test that raises if anything is
replayed without a sample. An unreadable sample logs a warning and refines without the gate rather
than turning a working loop into a failure.

Tests

Both directions, written before the change:

  • an over-broad candidate is rejected, refined, and the agent is handed the blast radius
  • a surgical policy (1 block in 200 records, 0.5%) passes on the first attempt
  • an unfixable over-block gives up honestly with a reason, matching existing behaviour
  • the threshold is configurable
  • with no sample, nothing is replayed and the result carries no blast_radius

250 pass · coverage 73% · ruff clean.

Surfaces

No new command — the gate rides the existing refine path. VPCOPILOT_SIM_LOGS documented in
.env.example and docs/USAGE.md; the console's apply result shows the blast radius when present.

🤖 Generated with Claude Code

`refiner.py` retried until the band-aid blocked the exploit. A refinement that widened the policy
far enough to catch the exploit had nothing telling it it went too far, so the loop would happily
accept a rule that denies production. With a traffic sample, "works" now means two things: the
policy blocks the exploit AND stays under the blast-radius threshold.

An over-broad candidate is fed back as `over_block` — a diagnosis the refine agent already
understands ("blocked the LEGIT request too. Narrow the rule") — with the measured rate and the top
blocked paths attached, so it knows what it over-blocked rather than guessing.

Measured where the policy is already attached. The refiner has just confirmed the candidate is
enforcing, so `simulate.measure_attached` replays through that LB instead of paying a second attach
and propagation wait on another one: slower-but-correct per the decision, but one round trip per
attempt rather than two. `_score` is now shared by `simulate_policies` and `measure_attached`, so
the standalone run and the refine gate cannot drift on what "too broad" means.

The sample comes from `VPCOPILOT_SIM_LOGS` or an explicit `records=`. With neither, the gate is
skipped and the loop behaves exactly as it did before — pinned by a test that fails if anything is
replayed without a sample. An unreadable sample logs a warning and refines without the gate rather
than failing a working loop.

Tests cover both directions: an over-broad candidate is rejected and refined (and the agent is
handed the blast radius), a surgical one passes on the first attempt, an unfixable over-block gives
up honestly with a reason, the threshold is configurable, and the no-sample path replays nothing.
250 pass, coverage 73%.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@henleda
henleda merged commit 8562c23 into main Jul 27, 2026
4 checks passed
@henleda
henleda deleted the feat-g3-simulation-feeds-refiner branch July 27, 2026 16:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant