feat(simulate): shadow simulation — what a band-aid would block before it is applied (G2) - #4
Merged
Merged
Conversation
…e it is applied (G2)
The safety spine proves a policy blocks the finding's exploit and passes one legit request. It says
nothing about the other million requests a day, and that gap is why security teams leave controls in
monitor mode. `vpcopilot simulate` replays a recorded sample against each candidate on a SPARE LB and
reports the would-block set before anything reaches the human gate.
Replay runs on a real LB rather than an offline evaluator (roadmap G1, deferred by decision): XC's
own enforcement is the only oracle that cannot silently disagree with XC.
- traffic.py — one RequestRecord shape from a HAR, a JSONL sample, or XC access logs (new read-only
`XC.access_logs`, the only /data endpoint this client touches). Redaction runs at INGEST because
the redacted sample ships in the evidence bundle: Authorization/Cookie/api-key headers are
dropped, secret-looking body fields are replaced (keeping the document shape so matchers still
see it), src_ip is never carried, and the counts reach simulation.json.
- simulate.py — snapshot -> create a throwaway policy -> attach -> confirm enforcement -> replay ->
restore -> delete. One policy at a time: two candidates share FIRST_MATCH ordering, so a batched
result would describe neither.
- Gate: over-threshold WARNS and requires an explicit override, which writes a `simulate_override`
audit record. A machine veto would invert "the model proposes, code disposes, a human approves".
Silent when nothing was simulated — this adds a check, never a prerequisite.
- Surfaces: `vpcopilot simulate [--logs | --from-tenant]`, GET+POST /api/simulate, a ③ Simulate
console step, a Blast radius panel in the HTML report, simulation.json in the evidence bundle.
Live on vpcopilot-lab: the negative-amount policy blocked 1 of 200 recorded requests — the exploit —
and none of the 60 legitimate transfers (0.5%, under threshold); a deliberately overbroad /api DENY
blocked 154 of 200 (77%) and tripped it. Both acceptance criteria, measured rather than asserted.
Three defects only the live run could find, each now pinned by a test:
1. Attaching and replaying immediately counted an UNENFORCED policy as harmless — 0/200 blocked,
including the exploit. A canary built from the finding's own probe now confirms enforcement
before counting, and an unconfirmed run says so rather than reading as safe.
2. One transient TLS error aborted a whole policy's simulation. Failures are now per-request and
excluded from the rate rather than guessed as blocked or allowed.
3. Attaching by name assumed the policy object already existed. A fresh candidate has none, and
referencing a missing policy broke the LB config outright — every replayed request failed in
transit, not just the matched ones. Simulation now creates a throwaway `<name>-vpcsim` object
and deletes it, so it never rides on, or mutates, the operator's own policies.
Honesty carried in the artifact, not just the docs: XC access logs capture no request body, so
`bodies_available` says a body_matcher cannot be judged from them; zero evaluated reports as "zero
evidence, not a clean result"; caveats state the sample is one window.
No-regression pins: apply is unchanged when nothing was simulated, a dry run is never gated, and
FakeXC gained `delete_service_policy` to match the real client. Console renumbered ④ Mitigate
through ⑦ Benchmark across the UI and eight docs. 34 new tests; 245 pass, coverage 73%.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…shots The Simulate step rendered its empty state in the offline demo, and every screenshot still showed the six-step nav. demo/build_demo_out.py writes a curated `simulation.json`: the SQLi service policy replayed against 500 recorded requests, blocking 3 — all attacks on the login endpoint — for 0.6%, under the threshold. That is the honest picture for this dataset: only `service_policy` is replayable (the other six controls are parameterized by the engine rather than matched per request), so one policy is the whole story, and the result shown is the surgical outcome the demo is about. Redaction counts and the caveats ride along so the demo shows the honesty, not just the number. Screenshots regenerated at 1200px/2x against a credential-free .env, and renamed to match the new step numbers: 3-mitigate.png -> 4-mitigate.png, 5-retire.png -> 6-retire.png, plus a new 3-simulate.png. README and DEMO.md references updated; no stale names remain. report.png grew by the Blast radius panel. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Roadmap G2. The safety spine proves a policy blocks the finding's exploit and passes one legit
request. It says nothing about the other million requests a day — and that gap is why security teams
leave controls in monitor mode.
vpcopilot simulatereplays a recorded sample against each candidateon a spare LB and reports the would-block set before anything reaches the human gate.
Replay runs on a real LB rather than an offline evaluator (G1, deferred by decision): XC's own
enforcement is the only oracle that cannot silently disagree with XC.
Proven live on
vpcopilot-lab, not asserteddeny-negative-pay-amountdeny-too-broad(/apiprefix DENY)Both acceptance criteria: the negative-amount policy blocked the exploit and none of the 60
legitimate transfers; a deliberately overbroad policy tripped. Enforcement was confirmed by canary
before counting; the LB was restored and the throwaway policy deleted after each run.
Three defects only the live run could find
Each is now pinned by a test. Unit tests with a stubbed transport could not have caught any of them.
harmless — 0/200 blocked, including the exploit. A canary built from the finding's own probe now
confirms enforcement first, and an unconfirmed run says so rather than reading as safe.
rate rather than guessed as blocked or allowed.
missing policy broke the LB config outright — every replayed request failed in transit, not
just the matched ones. Simulation now creates a throwaway
<name>-vpcsimobject and deletes it,so it never rides on, or mutates, the operator's own policies.
The gate: warn, don't veto
Over-threshold warns and requires an explicit override, which writes a
simulate_overrideauditrecord with the rate, threshold and actor. A hard block would be code refusing a human, inverting
DESIGN.md's "the model proposes, code disposes, a human approves". Silent when nothing wassimulated — this adds a check, never a prerequisite.
Redaction is a correctness requirement here
The redacted sample ships inside the evidence bundle, so redaction runs at ingest, not export:
Authorization/Cookie/api-keyheaders dropped, secret-looking body fields replaced (keepingthe document shape so matchers still see it),
src_ipnever carried into a record, and the countssurfaced in
simulation.json.Honesty carried in the artifact, not only the docs
bodies_available: falsesays abody_matchercan'tbe judged from them.
failed in transit.
Surfaces
vpcopilot simulate [--logs <path> | --from-tenant --source-lb <lb> --since 6h]·GET+POST /api/simulate· a ③ Simulate console step · a Blast radius panel in the HTMLreport ·
simulation.jsonin the evidence bundle ·bench/fixtures/traffic/nimbus-sample.jsonl(200 records, one negative-amount transfer among 60 legitimate ones).
No-regression
Per the invariant added last change: apply is unchanged when nothing was simulated, a dry run is
never gated,
FakeXCgaineddelete_service_policyto match the real client, and the consolerenumber (④ Mitigate … ⑦ Benchmark) was swept across the UI and eight docs.
34 new tests · 245 pass · coverage 73% ·
ruffclean.Deliberately not in scope
Simulation is not wired into
run_pipeline: replay attaches a policy to an LB, andscanisdocumented as performing no XC writes and being safe to run anywhere. Folding it in would break that.
G3 (feeding the refiner) stays its own item.
🤖 Generated with Claude Code