Repository navigation
Keep binary tool output from failing a session's memory write #61
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
josephschorr
wants to merge
2
commits into
main
Choose a base branch
from
fix/binary-tool-output
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,134 @@ | ||
| package runner | ||
|
|
||
| import ( | ||
| "fmt" | ||
| "log/slog" | ||
| "strings" | ||
| "unicode" | ||
| "unicode/utf8" | ||
|
|
||
| "github.com/dustin/go-humanize" | ||
|
|
||
| "github.com/authzed/openagentprimitives/pkg/agent/tool" | ||
| ) | ||
|
|
||
| // binarySniffLen is how much of a tool's output is examined to decide whether | ||
| // it is binary. It is the window git's own buffer_is_binary uses (the first | ||
| // 8000 bytes). | ||
| const binarySniffLen = 8000 | ||
|
|
||
| // nulReplacement stands in for a NUL that is removed from text. It is the | ||
| // substitution JSON encoding already makes for invalid UTF-8. | ||
| const nulReplacement = "�" | ||
|
|
||
| // outputCleanup says what sanitizeToolOutput did to a tool's output. | ||
| type outputCleanup string | ||
|
|
||
| const ( | ||
| outputUnchanged outputCleanup = "" | ||
| outputNULReplaced outputCleanup = "nul-replaced" | ||
| outputBinaryOmitted outputCleanup = "binary-omitted" | ||
| ) | ||
|
|
||
| // sanitizeToolOutput makes a tool's raw output safe to carry into a | ||
| // tool_result. It exists because NUL survives every other step: it is valid | ||
| // UTF-8, so JSON encoding keeps it (as \u0000) rather than replacing it, and | ||
| // Postgres jsonb then refuses the whole turn (SQLSTATE 22P05). | ||
| // | ||
| // Output with no NUL is returned untouched. Otherwise the rule is: | ||
| // | ||
| // - Binary — a NUL in the first binarySniffLen bytes AND those bytes do not | ||
| // read as text (see looksBinary) — is replaced whole with a notice naming | ||
| // its size. The model cannot use the bytes. | ||
| // - Anything else keeps its text, with each NUL replaced by U+FFFD. That | ||
| // covers a stray NUL anywhere, and output that uses NUL as a separator | ||
| // (`git ls-files -z`, `find -print0`), where the NULs are the least | ||
| // interesting part of a listing the model asked for. | ||
| // | ||
| // A NUL alone is deliberately not enough to call output binary. git treats it | ||
| // as enough because it is deciding whether to DIFF a file; here the cost of a | ||
| // wrong "binary" is the model losing output it could have read. | ||
| func sanitizeToolOutput(content string) (string, outputCleanup) { | ||
| if strings.IndexByte(content, 0) < 0 { | ||
| return content, outputUnchanged | ||
| } | ||
| if looksBinary(content) { | ||
| return fmt.Sprintf("[binary output omitted: %s. The tool returned raw binary data, which cannot be shown. "+ | ||
| "To inspect binary content, ask for something that describes it instead of its bytes — "+ | ||
| "its size, its type, a checksum.]", | ||
| humanize.Bytes(uint64(len(content)))), outputBinaryOmitted | ||
| } | ||
| return replaceNUL(content), outputNULReplaced | ||
| } | ||
|
|
||
| // replaceNUL swaps each NUL in content for U+FFFD and keeps everything else. | ||
| func replaceNUL(content string) string { | ||
| return strings.ReplaceAll(content, "\x00", nulReplacement) | ||
| } | ||
|
|
||
| // looksBinary reports whether the sniff window holds a NUL and does not read | ||
| // as text. "Not text" is either of: bytes that are not valid UTF-8, or a | ||
| // control character other than NUL itself, whitespace, backspace and ESC (the | ||
| // last two so terminal-styled output still counts as text). An executable or | ||
| // an image trips one of those within a few bytes of its header; a NUL-separated | ||
| // listing trips neither. | ||
| func looksBinary(content string) bool { | ||
| window := content | ||
| if len(window) > binarySniffLen { | ||
| // Cut on a rune boundary, so a multi-byte character straddling the | ||
| // window's edge is not mistaken for invalid UTF-8. | ||
| end := binarySniffLen | ||
| for end > binarySniffLen-utf8.UTFMax && !utf8.RuneStart(window[end]) { | ||
| end-- | ||
| } | ||
| window = window[:end] | ||
| } | ||
| if strings.IndexByte(window, 0) < 0 { | ||
| return false | ||
| } | ||
| if !utf8.ValidString(window) { | ||
| return true | ||
| } | ||
| for _, r := range window { | ||
| switch { | ||
| case r == 0, r == '\b', r == '\x1b', unicode.IsSpace(r): | ||
| case unicode.IsControl(r): | ||
| return true | ||
| } | ||
| } | ||
| return false | ||
| } | ||
|
|
||
| // cleanToolOutput applies sanitizeToolOutput to a tool's result, with the two | ||
| // exceptions that make it safe to run on every result: | ||
| // | ||
| // - A secret-output result is left alone. Its Content is the credential | ||
| // itself, which applySecretOutput diverts into the session's Secret before | ||
| // the LLM or memory sees it; replacing it would store the notice, or a | ||
| // corrupted value, as the credential and report success. | ||
| // - An error result never becomes the binary notice. Its text is what tells | ||
| // the model what went wrong, so only the NULs are replaced. | ||
| // | ||
| // Altered output is logged: the model and the transcript no longer hold what | ||
| // the tool returned, and nothing else would say so. | ||
| func (l *Loop) cleanToolOutput(name string, res tool.Result) tool.Result { | ||
| if res.SecretOutput != nil { | ||
| return res | ||
| } | ||
| raw := res.Content | ||
| cleanup := outputUnchanged | ||
| if res.IsError { | ||
| if strings.IndexByte(raw, 0) >= 0 { | ||
| res.Content, cleanup = replaceNUL(raw), outputNULReplaced | ||
| } | ||
| } else { | ||
| res.Content, cleanup = sanitizeToolOutput(raw) | ||
| } | ||
| if cleanup != outputUnchanged { | ||
| slog.Default().Info("tool output carried NUL, which a tool_result cannot hold; altered before use", | ||
| "session", l.SessionKey.Namespace+"/"+l.SessionKey.Name, | ||
| "tool", name, "action", string(cleanup), | ||
| "bytes", len(raw), "nulCount", strings.Count(raw, "\x00"), "isError", res.IsError) | ||
| } | ||
| return res | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,182 @@ | ||
| package runner | ||
|
|
||
| import ( | ||
| "context" | ||
| "encoding/json" | ||
| "strings" | ||
| "testing" | ||
|
|
||
| "github.com/stretchr/testify/assert" | ||
| "github.com/stretchr/testify/require" | ||
|
|
||
| "github.com/authzed/openagentprimitives/pkg/agent/llm" | ||
| "github.com/authzed/openagentprimitives/pkg/agent/secretout" | ||
| "github.com/authzed/openagentprimitives/pkg/agent/tool" | ||
| "github.com/authzed/openagentprimitives/pkg/authz" | ||
| "github.com/authzed/openagentprimitives/pkg/memory" | ||
| "github.com/authzed/openagentprimitives/pkg/platform/pipeline" | ||
| ) | ||
|
|
||
| func TestSanitizeToolOutput(t *testing.T) { | ||
| cases := []struct { | ||
| name string | ||
| in string | ||
| cleanup outputCleanup | ||
| check func(t *testing.T, out string) | ||
| }{ | ||
| { | ||
| name: "text with no NUL: returned unchanged", | ||
| in: "diff --git a/x b/x\n+hello\n", | ||
| cleanup: outputUnchanged, | ||
| check: func(t *testing.T, out string) { | ||
| assert.Equal(t, "diff --git a/x b/x\n+hello\n", out) | ||
| }, | ||
| }, | ||
| { | ||
| name: "an ELF header (NUL plus control bytes): replaced with a sized binary notice", | ||
| in: "\x7fELF\x02\x01\x01\x00" + strings.Repeat("x", 2040), | ||
| cleanup: outputBinaryOmitted, | ||
| check: func(t *testing.T, out string) { | ||
| assert.Contains(t, out, "[binary output omitted: 2.0 kB.") | ||
| assert.NotContains(t, out, "ELF", "none of the binary bytes may survive") | ||
| }, | ||
| }, | ||
| { | ||
| name: "a PNG header (NUL plus invalid UTF-8): replaced with a binary notice", | ||
| in: "\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR", | ||
| cleanup: outputBinaryOmitted, | ||
| check: func(t *testing.T, out string) { | ||
| assert.True(t, strings.HasPrefix(out, "[binary output omitted:"), "got %q", out) | ||
| }, | ||
| }, | ||
| { | ||
| name: "NUL-separated listing (git ls-files -z): every name kept, NULs replaced with U+FFFD", | ||
| in: "a.go\x00b.go\x00c.go\x00", | ||
| cleanup: outputNULReplaced, | ||
| check: func(t *testing.T, out string) { | ||
| assert.Equal(t, "a.go\ufffdb.go\ufffdc.go\ufffd", out) | ||
| }, | ||
| }, | ||
| { | ||
| name: "NUL-separated listing with non-ASCII names and terminal colour: still text", | ||
| in: "\x1b[32mnaïve.go\x1b[0m\x00日本語.txt\x00", | ||
| cleanup: outputNULReplaced, | ||
| check: func(t *testing.T, out string) { | ||
| assert.Contains(t, out, "naïve.go") | ||
| assert.Contains(t, out, "日本語.txt") | ||
| }, | ||
| }, | ||
| { | ||
| name: "a multi-byte character straddling the sniff window's edge: not mistaken for invalid UTF-8", | ||
| in: "a\x00" + strings.Repeat("b", binarySniffLen-3) + "日tail", | ||
| cleanup: outputNULReplaced, | ||
| check: func(t *testing.T, out string) { | ||
| assert.True(t, strings.HasSuffix(out, "日tail"), "the text must be kept") | ||
| }, | ||
| }, | ||
| { | ||
| name: "binary bytes only past the sniff window, a stray NUL before them: text kept, NULs replaced", | ||
| in: strings.Repeat("a", binarySniffLen) + "\x00\x01tail", | ||
| cleanup: outputNULReplaced, | ||
| check: func(t *testing.T, out string) { | ||
| assert.Equal(t, strings.Repeat("a", binarySniffLen)+"\ufffd\x01tail", out) | ||
| }, | ||
| }, | ||
| } | ||
| for _, tc := range cases { | ||
| t.Run(tc.name, func(t *testing.T) { | ||
| out, cleanup := sanitizeToolOutput(tc.in) | ||
| assert.Equal(t, tc.cleanup, cleanup) | ||
| assert.NotContains(t, out, "\x00", "sanitized output must never carry NUL") | ||
| tc.check(t, out) | ||
| }) | ||
| } | ||
| } | ||
|
|
||
| // TestDispatch_BinaryToolOutputIsSanitizedOnBothPaths pins the regression where | ||
| // a `git show <sha>:<binary>` result carried NUL into the tool_result turn, | ||
| // Postgres jsonb refused it (SQLSTATE 22P05), and the session failed as | ||
| // MemoryUnavailable. executeToolRaw is the one place both dispatch paths | ||
| // execute through, so both must come back clean. | ||
| func TestDispatch_BinaryToolOutputIsSanitizedOnBothPaths(t *testing.T) { | ||
| binary := "\x7fELF\x02\x01\x01\x00\x00\x00binary-body" | ||
| cases := []struct { | ||
| name string | ||
| kind tool.Kind | ||
| perm authz.Permission | ||
| }{ | ||
| {name: "gated MCP tool: result sanitized", kind: tool.KindMCP}, | ||
| {name: "ungated meta tool: result sanitized", kind: tool.KindMeta, perm: authz.Permission{StateImpact: authz.Stateless}}, | ||
| } | ||
| for _, tc := range cases { | ||
| t.Run(tc.name, func(t *testing.T) { | ||
| results := dispatchOnce(t, tc.kind, tc.perm, tool.Result{Content: binary}) | ||
|
|
||
| require.Len(t, results, 1) | ||
| assert.NotContains(t, results[0].Content, "\x00") | ||
| assert.Contains(t, results[0].Content, "[binary output omitted:") | ||
| }) | ||
| } | ||
| } | ||
|
|
||
| // TestDispatch_OutputSanitizerExceptions pins the two results the sanitizer | ||
| // must not treat like ordinary output, on the same dispatch path as above. | ||
| func TestDispatch_OutputSanitizerExceptions(t *testing.T) { | ||
| cases := []struct { | ||
| name string | ||
| result tool.Result | ||
| check func(t *testing.T, got tool.Result) | ||
| }{ | ||
| { | ||
| // applySecretOutput stores Content as the credential. A sanitized | ||
| // value would be captured, reported as success, and be wrong. | ||
| name: "secret-output result that looks binary: Content reaches the capture byte-for-byte", | ||
| result: tool.Result{ | ||
| Content: "\x00\x01\x02key\xff", | ||
| SecretOutput: &secretout.Spec{Name: "token"}, | ||
| }, | ||
| check: func(t *testing.T, got tool.Result) { | ||
| require.NotNil(t, got.SecretOutput, "the result must still be marked for capture") | ||
| assert.Equal(t, "\x00\x01\x02key\xff", got.Content) | ||
| }, | ||
| }, | ||
| { | ||
| name: "error result with a NUL after binary-looking bytes: text kept, only the NUL replaced", | ||
| result: tool.Result{Content: "\x01stderr: boom\n\x00", IsError: true}, | ||
| check: func(t *testing.T, got tool.Result) { | ||
| assert.True(t, got.IsError) | ||
| assert.Contains(t, got.Content, "stderr: boom") | ||
| assert.NotContains(t, got.Content, "\x00") | ||
| assert.NotContains(t, got.Content, "[binary output omitted:") | ||
| }, | ||
| }, | ||
| } | ||
| for _, tc := range cases { | ||
| t.Run(tc.name, func(t *testing.T) { | ||
| // Meta + Trusted keeps the content guard out of the way, so what | ||
| // comes back is what executeToolRaw produced. | ||
| tc.result.Trusted = true | ||
| results := dispatchOnce(t, tool.KindMeta, authz.Permission{StateImpact: authz.Stateless}, tc.result) | ||
|
|
||
| require.Len(t, results, 1) | ||
| tc.check(t, results[0]) | ||
| }) | ||
| } | ||
| } | ||
|
|
||
| // dispatchOnce runs one call of a fake tool returning result through | ||
| // dispatchToolUses and returns what the loop would go on to record. | ||
| func dispatchOnce(t *testing.T, kind tool.Kind, perm authz.Permission, result tool.Result) []tool.Result { | ||
| t.Helper() | ||
| ft := &fakeDispatchTool{name: "git", kind: kind, perm: perm, result: result} | ||
| l := &Loop{ | ||
| Tools: []tool.Tool{ft}, | ||
| SessionKey: memory.NamespacedName{Namespace: "default", Name: "disp"}, | ||
| } | ||
| loopWithInjectedExecutor(t, l, pipeline.NewRegistry()) | ||
|
|
||
| uses := []llm.ToolUseBlock{{ID: "tu-1", Name: "git", Input: json.RawMessage(`{"args":{}}`)}} | ||
| results, _ := l.dispatchToolUses(memory.WithSystemApproval(context.Background(), "test"), uses, dispatchTestSession(), 0, 0, nil, nil) | ||
| require.Equal(t, 1, ft.executed()) | ||
| return results | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[nit] ASCII separator controls (0x1C-0x1F) make NUL-separated text look binary
Problem: The
case unicode.IsControl(r)branch returns true for U+001C to U+001F becauseunicode.IsSpaceis false for them (probed at head). Output such asgit log -z --format=%H%x1f%an%x1f%sis valid UTF-8 text with NUL record ends and unit separators, butlooksBinaryreturns true and the model gets the binary notice instead of the text it asked for.Fix: Add
r >= 0x1c && r <= 0x1fto the allowed case on line 94 (next to\b, ESC andunicode.IsSpace), and mention the separators in thelooksBinarydoc comment above.Test: In
pkg/agent/runner/loop_binaryoutput_test.goadd a table case inTestSanitizeToolOutputwith input"abc\x1fdef\x00ghi\x1f\x00"expecting the text kept with each NUL replaced by U+FFFD. Rungo test -run TestSanitizeToolOutput ./pkg/agent/runner/.