Skip to content

fix(ai-red-teaming): recover cleanly from transient tool errors + right endpoint per target (ENG-8427) - #141

Merged
rdheekonda merged 1 commit into
mainfrom
fix/airt-tui-error-recovery
Sep 9, 2026
Merged

fix(ai-red-teaming): recover cleanly from transient tool errors + right endpoint per target (ENG-8427)#141
rdheekonda merged 1 commit into
mainfrom
fix/airt-tui-error-recovery

Conversation

@rdheekonda

Copy link
Copy Markdown
Contributor

Problem (ENG-8427)

A learner (aisf-learner-aug-2026) ran the cookbook evasion prompt in the TUI and saw two alarming errors, with no sign the agent recovered:

  • Error: 'list_environments' could not complete: _ssl.c:983: The handshake operation timed out.
  • Client error '404 Not Found' for url '.../attack'

Confirmed from prod Logfire (api) + ClickHouse session_events (session f270066a-1b94-45fc-88fd-d2947e1f645b): both were non-fatal and the agent recovered on its own - the HopSkipJump evasion completed (assessment 77ab88c7, 100% ASR, 1 HIGH finding, L2 2.2222, 500 queries). Root causes:

  1. Transient TLS handshake timeout on the agent's first list_environments call.
  2. The agent followed provision_environment's returned /attack URL and fetched it, but the ml-extraction-* classifier serves /predict, not /attack → 404. It then probed with python, found /predict, and ran generate_evasion_attack successfully.

It's a UX bug, not a broken run.

Fixes

  • safe_tool: retries transient network faults (TLS handshake, timeouts, connection reset/refused, 5xx) up to 2x with backoff, then returns an explicitly non-fatal Note: ("does not affect any attack already running or already-recorded results") instead of a raw Error:. Sync + async.
  • provision_environment: returns the endpoint matching the target type - classifier → /predict (+ /pool, /members, /nonmembers) with evasion/extraction/membership/inversion guidance; mesh → /attack + ATLAS. Stops the agent probing /attack on a classifier.
  • fmt_asr: consistent ASR formatting robust to 0-1 fractions vs 0-100 percentages - fixes the final message showing 1.0% instead of 100%. Applied in results.py, assessment.py, session.py.
  • Agent prompt: require narrating recovery when a tool errors mid-run; never leave a raw non-fatal error as the last thing the user sees.

Bumps capability 1.13.0 → 1.14.0.

Tests

  • tests/test_errors_safe_tool.py (new): fmt_asr cases (incl. 1.0 → 100%), transient classification, safe_tool retry-then-note / retry-then-success / non-transient-no-retry, no em dashes, async path.
  • tests/test_environments_teardown.py: _target_kind classifier/mesh/unknown.

Validation

  • pytest tests/test_errors_safe_tool.py tests/test_environments_teardown.py → 46 passed.
  • Full suite: 232 passed, 1 skipped, 1 pre-existing failure (test_attack_runner.py::TestGenerateMultimodalAttack::test_generates_compiling_script_with_image) that fails on origin/main, unrelated to this change.

Root-cause analysis posted to ENG-8427.

…ht endpoint per target (ENG-8427)

A learner's TUI evasion run surfaced two alarming-but-non-fatal errors while the
attack actually completed (assessment 77ab88c7, 100% ASR, 1 finding). Confirmed
from prod Logfire + ClickHouse session_events: a transient TLS handshake timeout
on the first list_environments call, and a 404 from the agent probing /attack on
an ml-extraction classifier that only serves /predict. The agent recovered on its
own but the TUI gave no sign of it.

Fixes:
- safe_tool: retry transient network faults (TLS handshake, timeouts, conn
  reset, 5xx) up to 2x with backoff, then surface an explicitly non-fatal Note
  ('does not affect any attack already running') instead of a raw Error. Works
  for sync + async tools.
- provision_environment: return the endpoint that matches the target type -
  classifier targets get /predict (+ /pool,/members,/nonmembers) and evasion/
  extraction/membership/inversion guidance; only meshes get /attack + ATLAS.
  Stops the agent probing /attack on a classifier (the 404).
- fmt_asr: consistent ASR formatting robust to 0-1 fractions vs 0-100 percents,
  fixing the final message showing '1.0%' instead of '100%'. Applied in
  results/assessment/session.
- agent prompt: require narrating recovery when a tool errors mid-run; never
  leave a raw non-fatal error as the last thing the user sees.

Bumps capability 1.13.0 -> 1.14.0. Adds 29 tests (fmt_asr, safe_tool retry/
classification sync+async, _target_kind).
@rdheekonda
rdheekonda merged commit a0b356b into main Sep 9, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant