fix(ai-red-teaming): recover cleanly from transient tool errors + right endpoint per target (ENG-8427) - #141
Merged
Merged
Conversation
…ht endpoint per target (ENG-8427)
A learner's TUI evasion run surfaced two alarming-but-non-fatal errors while the
attack actually completed (assessment 77ab88c7, 100% ASR, 1 finding). Confirmed
from prod Logfire + ClickHouse session_events: a transient TLS handshake timeout
on the first list_environments call, and a 404 from the agent probing /attack on
an ml-extraction classifier that only serves /predict. The agent recovered on its
own but the TUI gave no sign of it.
Fixes:
- safe_tool: retry transient network faults (TLS handshake, timeouts, conn
reset, 5xx) up to 2x with backoff, then surface an explicitly non-fatal Note
('does not affect any attack already running') instead of a raw Error. Works
for sync + async tools.
- provision_environment: return the endpoint that matches the target type -
classifier targets get /predict (+ /pool,/members,/nonmembers) and evasion/
extraction/membership/inversion guidance; only meshes get /attack + ATLAS.
Stops the agent probing /attack on a classifier (the 404).
- fmt_asr: consistent ASR formatting robust to 0-1 fractions vs 0-100 percents,
fixing the final message showing '1.0%' instead of '100%'. Applied in
results/assessment/session.
- agent prompt: require narrating recovery when a tool errors mid-run; never
leave a raw non-fatal error as the last thing the user sees.
Bumps capability 1.13.0 -> 1.14.0. Adds 29 tests (fmt_asr, safe_tool retry/
classification sync+async, _target_kind).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem (ENG-8427)
A learner (
aisf-learner-aug-2026) ran the cookbook evasion prompt in the TUI and saw two alarming errors, with no sign the agent recovered:Error: 'list_environments' could not complete: _ssl.c:983: The handshake operation timed out.Client error '404 Not Found' for url '.../attack'Confirmed from prod Logfire (
api) + ClickHousesession_events(sessionf270066a-1b94-45fc-88fd-d2947e1f645b): both were non-fatal and the agent recovered on its own - the HopSkipJump evasion completed (assessment77ab88c7, 100% ASR, 1 HIGH finding, L2 2.2222, 500 queries). Root causes:list_environmentscall.provision_environment's returned/attackURL andfetched it, but theml-extraction-*classifier serves/predict, not/attack→ 404. It then probed withpython, found/predict, and rangenerate_evasion_attacksuccessfully.It's a UX bug, not a broken run.
Fixes
safe_tool: retries transient network faults (TLS handshake, timeouts, connection reset/refused, 5xx) up to 2x with backoff, then returns an explicitly non-fatalNote:("does not affect any attack already running or already-recorded results") instead of a rawError:. Sync + async.provision_environment: returns the endpoint matching the target type - classifier →/predict(+/pool,/members,/nonmembers) with evasion/extraction/membership/inversion guidance; mesh →/attack+ ATLAS. Stops the agent probing/attackon a classifier.fmt_asr: consistent ASR formatting robust to 0-1 fractions vs 0-100 percentages - fixes the final message showing1.0%instead of100%. Applied inresults.py,assessment.py,session.py.Bumps capability 1.13.0 → 1.14.0.
Tests
tests/test_errors_safe_tool.py(new):fmt_asrcases (incl.1.0 → 100%), transient classification,safe_toolretry-then-note / retry-then-success / non-transient-no-retry, no em dashes, async path.tests/test_environments_teardown.py:_target_kindclassifier/mesh/unknown.Validation
pytest tests/test_errors_safe_tool.py tests/test_environments_teardown.py→ 46 passed.test_attack_runner.py::TestGenerateMultimodalAttack::test_generates_compiling_script_with_image) that fails onorigin/main, unrelated to this change.Root-cause analysis posted to ENG-8427.