Repository navigation
Keep binary tool output from failing a session's memory write - #61
josephschorr wants to merge 2 commits into
Conversation
A reviewbot session ran `git show <sha>:<binary>`. The raw bytes carried NUL into the tool_result turn, Postgres jsonb refused it (SQLSTATE 22P05), the runner retried the doomed write eight times, and the session failed as MemoryUnavailable. - The runner replaces binary tool output with a sized notice before it reaches the model or memory, and replaces a stray NUL in text with U+FFFD. - The memory facade refuses, on every backend, entries carrying text Postgres cannot store (NUL, unpaired surrogate escapes, invalid UTF-8 in text fields) as ErrInvalidEntry, a permanent 400 rather than a retried 500. - The Postgres backend tags any data-exception SQLSTATE (class 22) as ErrInvalidEntry as a backstop for shapes the facade does not name.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
samkim
left a comment
There was a problem hiding this comment.
This PR stops binary tool output (NUL bytes) from failing a session's memory write: the runner replaces binary output with a notice, the memory facade refuses unstorable entries as a 400 on every backend, and Postgres data exceptions map to the same permanent error. It fixes the reported git show case, but the fix is narrower than the new facade check, and the sanitizer has two defects.
Verdict: needs changes. The three major inline issues (secret-output corruption, NUL-separated text discarded, unsanitized non-tool text now failing local sessions) should be fixed before merge, and question 1 answered.
Issues, highest first:
- Secret-output values are corrupted. A tool result carrying a secret and a NUL has its value replaced by the binary notice before it is stored in the session Secret, and the capture still reports success. See inline on
loop_dispatch.go. - Any NUL in the first 8000 bytes discards the whole output, so valid NUL-separated text (
git ls-files -z,find -print0) is lost. See inline onloop_binaryoutput.go. - The facade now refuses NUL on inmem and sqlite too, but only tool output is cleaned. Assistant text, tool-call JSON and inbound user text can still carry NUL and now fail a local session as well. See inline on
facade.go. - Error results are replaced wholesale, so the model loses stderr. Covered in the
loop_dispatch.gocomment. - Raw invalid UTF-8 in Content, and Provenance strings, pass the facade check but fail on Postgres. See inline on
entry_storable.go. - Nit: per-write
Sprintfallocations incheckStorable. See inline.
Questions for the author:
- When a turn Append is refused with a permanent
ErrInvalidEntry, should the session still end asMemoryUnavailable("the memory service was unavailable")?pkg/agent/runner/loop.go:274,:769and:1066use that reason for any Append error, andpkg/channels/sessionnotice/sessionnotice.go:89shows that text. The Motivation calls the mislabel a problem, but the PR does not change it. - What rule decides "binary" (NUL count, ratio, UTF-8 validity), and why is a single NUL enough? Should output that uses NUL as a separator be kept?
- Is it intended that a byte-identical re-put of an older inmem/sqlite entry containing NUL is now refused (the check runs before the append-only idempotency check)? Does sqlite need a migration or warning?
- When a stray NUL is replaced with U+FFFD, how will operators know? The repo rule is to log or surface altered data; the sanitizer is silent in that case.
Not anchored to a diff line:
- Plausible: toolkit stream events recorded at
internal/cmd/runner/main.go:3291(toolsession.Record) are unsanitized and will now be refused on inmem/sqlite if they contain NUL. The write is best-effort and logged, so only that event is lost. Folded into thefacade.gocomment. pkg/agent/runner/loop.go:274,:769,:1066: session fails withReasonAgentSessionMemoryDownfor any Append error (question 1).
Not verified: the unit, integration and e2e suites were not run in this review. The PR says one e2e test failed under load and passed 4/4 alone but does not name it.
This review was drafted by an automated pipeline and checked before submission.
| // Both dispatch paths (gated and ungated meta) execute through here, so | ||
| // cleaning once covers every consumer downstream: the Post hooks and their | ||
| // audit records, the LLM tool_result, and the session's turn record. | ||
| res.Content = sanitizeToolOutput(res.Content) |
There was a problem hiding this comment.
[major] Sanitizer corrupts secret-output values and replaces error results wholesale
Problem: res.Content = sanitizeToolOutput(res.Content) runs for every result, including ones with res.SecretOutput set. applySecretOutput (pkg/agent/runner/loop.go:1031, pkg/agent/runner/loop_secretout.go:46 value := r.Content) later stores and publishes that Content as the credential. A secretOutput tool that emits binary data with a NUL in its first 8000 bytes gets the notice text written into the session's secret-outputs Secret and reported as success; a later NUL is stored as U+FFFD. The real credential is lost with no error. Separately, error results (including the dispatch-failed wrapper above and tool errors that inline stderr) are replaced whole by the notice when an early NUL appears, so the model loses the error text.
Fix: In executeToolRaw, skip sanitizing when res.SecretOutput != nil (the secret value is diverted before the LLM or memory sees it, loop.go:1027-1031). Alternatively, leave Content raw for secret outputs and let applySecretOutput handle it. For res.IsError results, keep the text and replace only the NUL bytes (U+FFFD substitution) instead of returning the binary notice, so stderr and guidance survive. Update the comment at lines 195-197 to state the exception.
Test: In pkg/agent/runner/loop_binaryoutput_test.go add cases to TestDispatch_BinaryToolOutputIsSanitizedOnBothPaths (or a new table test): (a) a tool returning tool.Result{Content: "\x00key", SecretOutput: &...} reaches applySecretOutput with the exact original bytes; (b) an IsError result containing "stderr: boom\n\x00" keeps stderr: boom and has no NUL. Run go test -run 'TestDispatch_BinaryToolOutput|TestSanitizeToolOutput' ./pkg/agent/runner/.
There was a problem hiding this comment.
Fixed in 34937b6. A secret-output result now skips the sanitizer, so the value reaches the capture byte-for-byte, and an error result keeps its text with only the NULs replaced. Both are pinned in TestDispatch_OutputSanitizerExceptions.
There was a problem hiding this comment.
Verified at 34937b6c: cleanToolOutput returns secret-output results untouched and only replaces NULs in error results (pkg/agent/runner/loop_binaryoutput.go). TestDispatch_OutputSanitizerExceptions covers both.
| if i < 0 { | ||
| return content | ||
| } | ||
| if i < binarySniffLen { |
There was a problem hiding this comment.
[major] One NUL in the first 8000 bytes discards valid NUL-separated text
Problem: i < binarySniffLen classifies the whole output as binary on a single NUL. Output of git ls-files -z, git status --porcelain -z, git grep -z, find -print0 and similar is valid UTF-8 text separated by NUL. The model receives "[binary output omitted ...]" plus advice about git cat-file and loses the listing it asked for. The notice text is also git-specific regardless of which tool produced the output.
Fix: Design decision for the author, so pick one and say why: (a) treat output as binary only when the NUL count is a meaningful fraction of the sniffed window, or when it also has invalid UTF-8 (utf8.Valid), and otherwise replace NULs with U+FFFD or a newline-like separator and keep the text; or (b) keep the git heuristic but make the notice tool-neutral. Whatever the rule, document the threshold in the doc comment above sanitizeToolOutput.
Test: Add table cases in TestSanitizeToolOutput (pkg/agent/runner/loop_binaryoutput_test.go) for "a.go\x00b.go\x00c.go\x00" (expect the three names preserved) and for a real binary header (still a notice). Run go test -run TestSanitizeToolOutput ./pkg/agent/runner/.
There was a problem hiding this comment.
Went with (a), in 34937b6: output is binary only when a NUL in the sniff window comes with invalid UTF-8 or a non-text control character; otherwise the text is kept and each NUL becomes U+FFFD. The rule is documented on sanitizeToolOutput and looksBinary, the notice is tool-neutral, and the -z listing case is in the table.
There was a problem hiding this comment.
Verified at 34937b6c: looksBinary needs a NUL plus invalid UTF-8 or a non-text control character, otherwise replaceNUL keeps the text, and the notice is tool-neutral. The -z listing, ANSI, straddle, ELF and PNG cases pass. One related nit (ASCII separators 0x1C-0x1F) is raised inline as a new comment.
| ErrInvalidEntry, i, lnk.ID, lnk.Kind, target.IDPrefix()) | ||
| } | ||
| } | ||
| if err := checkStorable(e); err != nil { |
There was a problem hiding this comment.
[major] The facade now refuses NUL on every backend, but only tool output is sanitized before the write
Problem: checkStorable rejects any entry with NUL, but the runner only cleans executeToolRaw Content. Other text still reaches memory raw: assistant text and tool_use Input JSON (pkg/agent/runner/loop.go:768, a model-emitted \u0000), inbound user/channel text (loop.go:273, :1065), and toolkit stream events recorded by toolsession.Record (internal/cmd/runner/main.go:3291, best-effort so only that event is lost; I could not confirm the stream carries NUL). Each of these fails the turn Append exactly as the incident did. On Postgres that is unchanged; on inmem and sqlite (local and desktop installs) it is a new failure introduced by this check.
Verify: Check which of those producers can actually carry NUL (an LLM provider decode of "\u0000" into a Go string, a webhook body with \u0000).
Fix: Sanitize at the point where memory.Turn is built for Append (one helper in pkg/agent/runner that walks text blocks, tool_use input and tool_result content), so every block type is covered, and reuse the NUL handling from sanitizeToolOutput. Do not put this in the Postgres backend (signed append-only records must not be rewritten at the store, per the PR description). Apply the same helper to the toolsession.Event fields at internal/cmd/runner/main.go:3291 if the verify step shows they can carry NUL.
Test: In pkg/agent/runner add a loop test where the scripted LLM returns assistant text with NUL and a tool_use whose input contains \u0000, and assert the turn Append succeeds and the persisted text has no NUL. Run go test -run 'Sanitize|Dispatch' ./pkg/agent/runner/.
There was a problem hiding this comment.
Fixed in 34937b6, but in turn.Appender.Append rather than a runner helper: channelsd writes inbound messages through the same appender before the runner sees them, so a runner-only helper would have missed that writer. It replaces before signing, covers every field, and is logged; toolsession.Record does the same. One correction: tool_use Input is []byte and stored base64-encoded, so an escape in it never reaches the column (test added).
There was a problem hiding this comment.
Verified at 34937b6c: turn.Appender.Append replaces unstorable characters before signing and logs it, and every live turn writer (runner, channelsd, inboxwake, the restart inbox append) uses it. You are right that tool_use Input is base64. TestAppender_UnstorableCharactersAreReplaced passes. One gap is raised inline: memcopy.CopyPrefix re-puts legacy turns without going through Append.
| return fmt.Errorf("field %s contains a NUL character", t.field) | ||
| } | ||
| } | ||
| return checkStorableJSON(e.Content) |
There was a problem hiding this comment.
[minor] Raw Content is not checked for invalid UTF-8, and Provenance strings are not checked at all
Problem: checkStorableJSON only looks for NUL and \u escapes. A probe confirmed Content = {"a":"\xff"} passes checkStorable, as does a NUL in Provenance.Publisher. Postgres jsonb refuses invalid UTF-8 (22021), so a client posting raw invalid-UTF-8 JSON gets accepted on inmem/sqlite and refused (via the backstop, as 400) only on Postgres. That is the backend asymmetry this change says it removes. Turn content built by json.Marshal is not affected, because Marshal replaces invalid UTF-8.
Fix: In checkStorable, add if !utf8.Valid(e.Content) { return fmt.Errorf("content is not valid UTF-8") } before checkStorableJSON. Also check the Provenance string fields (Publisher, KeyID and any other string fields in memory.Provenance, pkg/memory/entry.go:38) through the same texts loop when e.Provenance != nil.
Test: Extend the table in pkg/memory/entry_storable_internal_test.go with invalid-UTF-8 Content and NUL in Provenance.Publisher. Run go test -run 'Storable' ./pkg/memory/.
There was a problem hiding this comment.
Fixed in 34937b6. checkStorable now refuses invalid UTF-8 in Content and checks Provenance Publisher, KeyID and PrevHash; both cases are in the facade table.
There was a problem hiding this comment.
Verified at 34937b6c: checkStorable refuses invalid UTF-8 in Content and checks Provenance Publisher, KeyID and PrevHash (pkg/memory/entry_storable.go), with facade table cases.
| {"ID", e.ID}, | ||
| } | ||
| for i, t := range e.Tags { | ||
| texts = append(texts, text{fmt.Sprintf("Tags[%d]", i), t}) |
There was a problem hiding this comment.
[nit] Labels are formatted with fmt.Sprintf on every Put, even when nothing is wrong
Problem: Lines 27 and 30 allocate a label string per tag and five per link on the success path of every write, a hot path for transcript turns.
Fix: Store the field index/name lazily (for example a func() string or a {prefix string; i int}), or run the checks inline and format the label only when a check fails.
Test: Existing entry_storable_internal_test.go cases must still report the same field names. Run go test -run Storable ./pkg/memory/.
There was a problem hiding this comment.
Fixed in 34937b6. Labels are built only when a field fails; the field names are unchanged.
There was a problem hiding this comment.
Verified at 34937b6c: labels are built only when a field fails.
… replace unstorable characters when a turn is recorded Tool output is now treated as binary only when a NUL comes with bytes that do not read as text, so NUL-separated listings keep their content. A secret-output value is passed through untouched, and an error result keeps its text. Altered output is logged. A turn or tool-session event carrying a character the store refuses is recorded with a placeholder instead of failing the write, whichever writer or block it came from. The store's own check now also refuses invalid UTF-8 in content and bad provenance strings, and no longer builds field labels on the success path.
|
Answers to the review's questions, as of 34937b6:
The requested loop test is covered at the appender instead, since that is where the fix lives. All three suites pass locally; the earlier e2e failure did not recur. |
samkim
left a comment
There was a problem hiding this comment.
Round 2 (commit 34937b6, since 6919661). All 5 earlier threads are verified fixed (5 of 5). I re-read the code and ran the targeted tests for the sanitizer, the turn appender, toolsession and the facade checks, and they pass.
Verdict: ready after the inline fixes. Both new findings are small. The first (N1) needs a decision from you before it is a fix.
Open issues, highest impact first:
- Minor: a session whose turn was stored with a NUL before this PR (possible on sqlite installs; inmem does not survive the upgrade) cannot be restarted or forked after upgrade.
memcopy.CopyPrefixre-puts the signed entry unchanged and the new storable check refuses it. See inline onpkg/memory/kinds/turn/accessor.go. - Nit: output separated by NUL and the ASCII unit or record separators (
git log -z --format=%H%x1f%an) is still classed as binary and replaced with the notice. See inline onpkg/agent/runner/loop_binaryoutput.go.
Questions for the author:
- Should a legacy turn that contains NUL stay forkable? If yes, which fix do you prefer: exempt the copy path from the storable check, or have the copier rewrite the entry? The second is hard because the entry is signed. If you would rather accept the limitation, say so in the Appender comment and the PR description.
The earlier questions are answered: the MemoryUnavailable label is deferred to a follow-up, which is reasonable because turns are cleaned before the write now. The binary rule is documented, and altered data is logged.
Not anchored to a diff line: none.
This review was drafted by an automated pipeline and checked before submission.
| // record, so refusing it fails the session, over one character in a model's | ||
| // reply or a user's message. A tool call's arguments are not at risk: they are | ||
| // bytes, stored base64-encoded. Every writer of a turn — the runner, and | ||
| // channelsd for inbound messages — comes through this method, so replacing |
There was a problem hiding this comment.
[minor] Comment claims every turn writer goes through Append, but memcopy bypasses it
Problem: This comment (and the legacy-turn leniency in Append) assumes all turn writers use turn.Appender.Append. memcopy.CopyPrefix (pkg/memory/memcopy/memcopy.go:61-65, called from CopyMemoryPrefix at pkg/controllers/agentsession/restart.go:139) copies existing turn entries verbatim with m.Put. The facade's checkStorable (pkg/memory/facade.go:291) runs on every Put. A turn stored with a NUL (the JSON escape backslash-u-0000) before this PR (sqlite accepted it, and its data survives the upgrade) is refused with ErrInvalidEntry, so restarting or forking that session fails with "memcopy.CopyPrefix: put". The entry is signed, so the copier cannot rewrite it.
Verify: This is a design decision, so do not pick a fix without the author's answer (see the question in the review body). Options: (a) in CopyPrefix, skip or relax checkStorable for entries already persisted, for example through a copy-only context flag the facade honours for entries that carry a valid Provenance; (b) accept the limitation and say so. Whichever is chosen, correct this comment so it names memcopy as the exception or says why the copy is safe.
Test: In pkg/memory/memcopy/memcopy_test.go add TestCopyPrefix_LegacyNULTurn that seeds a source scope with a turn entry whose content holds a NUL escape (backslash-u-0000) through a backend that bypasses the facade check, then asserts the expected behaviour (copy succeeds, or a clear error if the limitation is accepted). Run go test -run TestCopyPrefix ./pkg/memory/memcopy/.
| for _, r := range window { | ||
| switch { | ||
| case r == 0, r == '\b', r == '\x1b', unicode.IsSpace(r): | ||
| case unicode.IsControl(r): |
There was a problem hiding this comment.
[nit] ASCII separator controls (0x1C-0x1F) make NUL-separated text look binary
Problem: The case unicode.IsControl(r) branch returns true for U+001C to U+001F because unicode.IsSpace is false for them (probed at head). Output such as git log -z --format=%H%x1f%an%x1f%s is valid UTF-8 text with NUL record ends and unit separators, but looksBinary returns true and the model gets the binary notice instead of the text it asked for.
Fix: Add r >= 0x1c && r <= 0x1f to the allowed case on line 94 (next to \b, ESC and unicode.IsSpace), and mention the separators in the looksBinary doc comment above.
Test: In pkg/agent/runner/loop_binaryoutput_test.go add a table case in TestSanitizeToolOutput with input "abc\x1fdef\x00ghi\x1f\x00" expecting the text kept with each NUL replaced by U+FFFD. Run go test -run TestSanitizeToolOutput ./pkg/agent/runner/.
| // Both dispatch paths (gated and ungated meta) execute through here, so | ||
| // cleaning once covers every consumer downstream: the Post hooks and their | ||
| // audit records, the LLM tool_result, and the session's turn record. | ||
| res.Content = sanitizeToolOutput(res.Content) |
There was a problem hiding this comment.
Verified at 34937b6c: cleanToolOutput returns secret-output results untouched and only replaces NULs in error results (pkg/agent/runner/loop_binaryoutput.go). TestDispatch_OutputSanitizerExceptions covers both.
| if i < 0 { | ||
| return content | ||
| } | ||
| if i < binarySniffLen { |
There was a problem hiding this comment.
Verified at 34937b6c: looksBinary needs a NUL plus invalid UTF-8 or a non-text control character, otherwise replaceNUL keeps the text, and the notice is tool-neutral. The -z listing, ANSI, straddle, ELF and PNG cases pass. One related nit (ASCII separators 0x1C-0x1F) is raised inline as a new comment.
| ErrInvalidEntry, i, lnk.ID, lnk.Kind, target.IDPrefix()) | ||
| } | ||
| } | ||
| if err := checkStorable(e); err != nil { |
There was a problem hiding this comment.
Verified at 34937b6c: turn.Appender.Append replaces unstorable characters before signing and logs it, and every live turn writer (runner, channelsd, inboxwake, the restart inbox append) uses it. You are right that tool_use Input is base64. TestAppender_UnstorableCharactersAreReplaced passes. One gap is raised inline: memcopy.CopyPrefix re-puts legacy turns without going through Append.
| return fmt.Errorf("field %s contains a NUL character", t.field) | ||
| } | ||
| } | ||
| return checkStorableJSON(e.Content) |
There was a problem hiding this comment.
Verified at 34937b6c: checkStorable refuses invalid UTF-8 in Content and checks Provenance Publisher, KeyID and PrevHash (pkg/memory/entry_storable.go), with facade table cases.
| {"ID", e.ID}, | ||
| } | ||
| for i, t := range e.Tags { | ||
| texts = append(texts, text{fmt.Sprintf("Tags[%d]", i), t}) |
There was a problem hiding this comment.
Verified at 34937b6c: labels are built only when a field fails.
Motivation
A pull-request review session failed partway through with "MemoryUnavailable". The agent had asked git to show a binary file that had just been committed. Git printed the file's raw bytes, which included NUL characters. The session's durable store (Postgres) cannot hold NUL in a JSON column, so every attempt to save that step of the conversation was rejected.
Two things made it worse than it had to be. The rejection was treated like a temporary outage, so the write was retried eight times before giving up. And the session was then labelled "memory unavailable", although the store was healthy and only that one piece of data was the problem.
Summary of changes
git ls-files -z,find -print0) keep their content, with each NUL replaced by a placeholder character. Secret values a tool hands back for capture are never altered, and an error result always keeps its text.Behaviour change: on installs using the in-memory or SQLite store, an entry containing NUL used to be accepted and is now refused. Postgres installs already failed on these. Giving the same answer on every backend seemed better than leaving it depending on configuration.
Not changed here: a session whose write is permanently rejected is still labelled "memory unavailable". Telling the two apart needs a new failure reason and is left for a follow-up. With conversation steps now cleaned before the write, that path is only reachable for rejections the up-front check doesn't name.
Alternatives considered
Provenance
Ship gate
mage test:unitmage test:integrationmage test:e2eResults:
Regeneration
config/**Coverage
Before requesting review