Skip to content

loopguard: an edit that applied is not an error output (fixes #151) - #158

Closed
Vu Linh Truong (vulinh1704) wants to merge 1 commit into
microsoft:mainfrom
vulinh1704:fix/loopguard-edit-content-151
Closed

Vu Linh Truong (vulinh1704) wants to merge 1 commit into
microsoft:mainfrom
vulinh1704:fix/loopguard-edit-content-151

Conversation

@vulinh1704

Copy link
Copy Markdown
Contributor

Fixes #151.

Claude Code's Edit result names no outcome. It is the file's text (filePath, oldString, newString, originalFile, structuredPatch, ...), and a Write result carries the whole file as content. A success has no exit status, so the guard read that text with ERR_RE. Edits to a document that mentions assert, failed or an ...Error counted as error outputs and were tagged [err#...] in the judge's digest (#151).

An edit that does not apply (text not found, file not read first) comes as PostToolUseFailure. So an Edit, MultiEdit or Write that came back as PostToolUse succeeded, and claude_success_code now gives it 0, the same way it gives 0 to a covered shell call since #128. The history import's envelope replay uses the same function, so it reads edits the same way. ERR_RE, SpiralState and response_text are unchanged.

I did not take the fix suggested in the issue (return "" from response_text for a dict with no known key). It covers Edit, but not Write, whose content is a known key and is the whole file. It would also change what every host folds for an unknown dict.

The tests are in a new tests/test_loopguard_edit_results.py, next to test_loopguard_productive_turns.py. They send Edit, MultiEdit and Write results whose file text contains assert, failed and AssertionError, shaped as Claude Code 2.1 logs Edit and Write results. A failed Edit and a failing test run still count as error outputs.

Verified on Ubuntu 24.04 (WSL), Python 3.12, on main at a71684c:

  • Before the fix, the new file reports 2 failed and 1 passed. Four successful edits gave error_outputs == 4.
  • After the fix, it reports 3 passed.
  • python -m pytest tests -rfEs --tb=line reports 739 passed and 16 skipped. python -m pytest tests experiments/tests reports 1132 passed and 20 skipped. Each skip needs a tool the machine does not have (a real Codex, Hermes or opencode CLI, aider) or Windows.

Risk: only Claude Code edits reported as successes change, and they could not have failed. A failed edit still arrives as PostToolUseFailure and is read by its error text, as before.

Claude Code's Edit result names no outcome: it is the file's text
(filePath, oldString, newString, originalFile, structuredPatch, ...), and a
Write's carries the whole file as `content`. With no exit status, the guard
read that text with ERR_RE, so edits to a document that mentions `assert`,
`failed` or an `...Error` counted as error outputs and were tagged [err#...]
in the judge's digest.

An edit that does not apply (text not found, file not read first) comes as
PostToolUseFailure, so an Edit, MultiEdit or Write that came back as
PostToolUse succeeded: claude_success_code now gives it 0, as it gives a
covered shell call (microsoft#128). The history import's envelope replay does the
same. A failed edit and a failing run still count; ERR_RE and SpiralState
are unchanged.
@SakshamUboweja

Copy link
Copy Markdown
Collaborator

Thanks for this! #152 fixes the same issue (#151) and is now merged. Edits to a file that mentions failures no longer count as errors. Closing this as a duplicate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Loop guard counts a successful Edit as an error when the edited text contains 'assert' or 'failed'

2 participants