Skip to content

Keep binary tool output from failing a session's memory write - #61

Open
josephschorr wants to merge 2 commits into
mainfrom
fix/binary-tool-output
Open

josephschorr wants to merge 2 commits into
mainfrom
fix/binary-tool-output

Conversation

@josephschorr

@josephschorr josephschorr commented Nov 7, 2026 •

Copy link
Copy Markdown
Member

Motivation

A pull-request review session failed partway through with "MemoryUnavailable". The agent had asked git to show a binary file that had just been committed. Git printed the file's raw bytes, which included NUL characters. The session's durable store (Postgres) cannot hold NUL in a JSON column, so every attempt to save that step of the conversation was rejected.

Two things made it worse than it had to be. The rejection was treated like a temporary outage, so the write was retried eight times before giving up. And the session was then labelled "memory unavailable", although the store was healthy and only that one piece of data was the problem.

Summary of changes

  • Binary tool output is replaced before anything sees it. When a tool prints binary data, the agent now gets a short notice giving the output's size and suggesting it ask for a description of the content instead. The model can't use raw binary anyway. Output counts as binary only when a NUL comes with bytes that don't read as text, so NUL-separated listings (git ls-files -z, find -print0) keep their content, with each NUL replaced by a placeholder character. Secret values a tool hands back for capture are never altered, and an error result always keeps its text.
  • A conversation step is recorded even if it contains a character the store can't hold. The character is replaced with a placeholder when the step is written, whether it came from the model's reply, a user's message, or tool output. The same applies to recorded tool-stream events. Each replacement is logged.
  • Memory refuses data it cannot store, on every backend. A write containing NUL, invalid UTF-8, or a malformed Unicode escape that was not cleaned as above is rejected up front as a bad request, with a message that says why. It is not retried. Before, the same write succeeded on some backends and failed only once it reached Postgres.
  • Postgres data errors are treated as permanent. Any value Postgres refuses on content grounds now comes back as a bad request rather than a server error. This catches cases the check above doesn't name.

Behaviour change: on installs using the in-memory or SQLite store, an entry containing NUL used to be accepted and is now refused. Postgres installs already failed on these. Giving the same answer on every backend seemed better than leaving it depending on configuration.

Not changed here: a session whose write is permanently rejected is still labelled "memory unavailable". Telling the two apart needs a new failure reason and is left for a follow-up. With conversation steps now cleaned before the write, that path is only reachable for rejections the up-front check doesn't name.

Alternatives considered

  • Cleaning the data inside the Postgres store. Ruled out because conversation records are signed and append-only. Changing their bytes on the way in would break the signature and make the durable copy differ from the in-memory one.
  • Changing the column type so it accepts NUL. Ruled out because full-text search and field filters depend on the current JSON column type, and binary output still wouldn't be useful to the model.

Provenance

  • Author: Joseph Schorr
  • Harness or tooling, with version: n/a
  • Person who read the diff: Joseph Schorr

Ship gate

  • mage test:unit
  • mage test:integration
  • mage test:e2e

Results:

unit:        pass
integration: pass
e2e:         pass

Regeneration

  • n/a — no RBAC markers or CRD fields changed
  • n/a — nothing under config/**
  • n/a — no command or CRD schema changes
  • n/a — no Markdown or TypeScript changes

Coverage

  • This does not add user-visible behavior; it fixes a failure. New tests cover the binary-output path and its exceptions (NUL-separated text, secret values, error results), the store's refusal of each kind of bad data (including against a real Postgres), and the conversation-turn write.
  • This touches none of authorization, the SpiceDB schema, or the approver model

Before requesting review

  • The PR holds a single change. The three parts are layers of one fix: removing the bad data at its source, rejecting it clearly at the store, and treating anything that still slips through as permanent.
  • A person has read every line of the diff

A reviewbot session ran `git show <sha>:<binary>`. The raw bytes carried NUL
into the tool_result turn, Postgres jsonb refused it (SQLSTATE 22P05), the
runner retried the doomed write eight times, and the session failed as
MemoryUnavailable.

- The runner replaces binary tool output with a sized notice before it
  reaches the model or memory, and replaces a stray NUL in text with U+FFFD.
- The memory facade refuses, on every backend, entries carrying text Postgres
  cannot store (NUL, unpaired surrogate escapes, invalid UTF-8 in text
  fields) as ErrInvalidEntry, a permanent 400 rather than a retried 500.
- The Postgres backend tags any data-exception SQLSTATE (class 22) as
  ErrInvalidEntry as a backstop for shapes the facade does not name.
@vercel

vercel Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
openagentprimitives Ignored Ignored Preview Nov 11, 2026 3:53am UTC

Request Review

@samkim samkim left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR stops binary tool output (NUL bytes) from failing a session's memory write: the runner replaces binary output with a notice, the memory facade refuses unstorable entries as a 400 on every backend, and Postgres data exceptions map to the same permanent error. It fixes the reported git show case, but the fix is narrower than the new facade check, and the sanitizer has two defects.

Verdict: needs changes. The three major inline issues (secret-output corruption, NUL-separated text discarded, unsanitized non-tool text now failing local sessions) should be fixed before merge, and question 1 answered.

Issues, highest first:

  • Secret-output values are corrupted. A tool result carrying a secret and a NUL has its value replaced by the binary notice before it is stored in the session Secret, and the capture still reports success. See inline on loop_dispatch.go.
  • Any NUL in the first 8000 bytes discards the whole output, so valid NUL-separated text (git ls-files -z, find -print0) is lost. See inline on loop_binaryoutput.go.
  • The facade now refuses NUL on inmem and sqlite too, but only tool output is cleaned. Assistant text, tool-call JSON and inbound user text can still carry NUL and now fail a local session as well. See inline on facade.go.
  • Error results are replaced wholesale, so the model loses stderr. Covered in the loop_dispatch.go comment.
  • Raw invalid UTF-8 in Content, and Provenance strings, pass the facade check but fail on Postgres. See inline on entry_storable.go.
  • Nit: per-write Sprintf allocations in checkStorable. See inline.

Questions for the author:

  1. When a turn Append is refused with a permanent ErrInvalidEntry, should the session still end as MemoryUnavailable ("the memory service was unavailable")? pkg/agent/runner/loop.go:274, :769 and :1066 use that reason for any Append error, and pkg/channels/sessionnotice/sessionnotice.go:89 shows that text. The Motivation calls the mislabel a problem, but the PR does not change it.
  2. What rule decides "binary" (NUL count, ratio, UTF-8 validity), and why is a single NUL enough? Should output that uses NUL as a separator be kept?
  3. Is it intended that a byte-identical re-put of an older inmem/sqlite entry containing NUL is now refused (the check runs before the append-only idempotency check)? Does sqlite need a migration or warning?
  4. When a stray NUL is replaced with U+FFFD, how will operators know? The repo rule is to log or surface altered data; the sanitizer is silent in that case.

Not anchored to a diff line:

  • Plausible: toolkit stream events recorded at internal/cmd/runner/main.go:3291 (toolsession.Record) are unsanitized and will now be refused on inmem/sqlite if they contain NUL. The write is best-effort and logged, so only that event is lost. Folded into the facade.go comment.
  • pkg/agent/runner/loop.go:274, :769, :1066: session fails with ReasonAgentSessionMemoryDown for any Append error (question 1).

Not verified: the unit, integration and e2e suites were not run in this review. The PR says one e2e test failed under load and passed 4/4 alone but does not name it.

This review was drafted by an automated pipeline and checked before submission.

Comment thread pkg/agent/runner/loop_dispatch.go Outdated
// Both dispatch paths (gated and ungated meta) execute through here, so
// cleaning once covers every consumer downstream: the Post hooks and their
// audit records, the LLM tool_result, and the session's turn record.
res.Content = sanitizeToolOutput(res.Content)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[major] Sanitizer corrupts secret-output values and replaces error results wholesale

Problem: res.Content = sanitizeToolOutput(res.Content) runs for every result, including ones with res.SecretOutput set. applySecretOutput (pkg/agent/runner/loop.go:1031, pkg/agent/runner/loop_secretout.go:46 value := r.Content) later stores and publishes that Content as the credential. A secretOutput tool that emits binary data with a NUL in its first 8000 bytes gets the notice text written into the session's secret-outputs Secret and reported as success; a later NUL is stored as U+FFFD. The real credential is lost with no error. Separately, error results (including the dispatch-failed wrapper above and tool errors that inline stderr) are replaced whole by the notice when an early NUL appears, so the model loses the error text.
Fix: In executeToolRaw, skip sanitizing when res.SecretOutput != nil (the secret value is diverted before the LLM or memory sees it, loop.go:1027-1031). Alternatively, leave Content raw for secret outputs and let applySecretOutput handle it. For res.IsError results, keep the text and replace only the NUL bytes (U+FFFD substitution) instead of returning the binary notice, so stderr and guidance survive. Update the comment at lines 195-197 to state the exception.
Test: In pkg/agent/runner/loop_binaryoutput_test.go add cases to TestDispatch_BinaryToolOutputIsSanitizedOnBothPaths (or a new table test): (a) a tool returning tool.Result{Content: "\x00key", SecretOutput: &...} reaches applySecretOutput with the exact original bytes; (b) an IsError result containing "stderr: boom\n\x00" keeps stderr: boom and has no NUL. Run go test -run 'TestDispatch_BinaryToolOutput|TestSanitizeToolOutput' ./pkg/agent/runner/.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 34937b6. A secret-output result now skips the sanitizer, so the value reaches the capture byte-for-byte, and an error result keeps its text with only the NULs replaced. Both are pinned in TestDispatch_OutputSanitizerExceptions.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at 34937b6c: cleanToolOutput returns secret-output results untouched and only replaces NULs in error results (pkg/agent/runner/loop_binaryoutput.go). TestDispatch_OutputSanitizerExceptions covers both.

Comment thread pkg/agent/runner/loop_binaryoutput.go Outdated
if i < 0 {
return content
}
if i < binarySniffLen {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[major] One NUL in the first 8000 bytes discards valid NUL-separated text

Problem: i < binarySniffLen classifies the whole output as binary on a single NUL. Output of git ls-files -z, git status --porcelain -z, git grep -z, find -print0 and similar is valid UTF-8 text separated by NUL. The model receives "[binary output omitted ...]" plus advice about git cat-file and loses the listing it asked for. The notice text is also git-specific regardless of which tool produced the output.
Fix: Design decision for the author, so pick one and say why: (a) treat output as binary only when the NUL count is a meaningful fraction of the sniffed window, or when it also has invalid UTF-8 (utf8.Valid), and otherwise replace NULs with U+FFFD or a newline-like separator and keep the text; or (b) keep the git heuristic but make the notice tool-neutral. Whatever the rule, document the threshold in the doc comment above sanitizeToolOutput.
Test: Add table cases in TestSanitizeToolOutput (pkg/agent/runner/loop_binaryoutput_test.go) for "a.go\x00b.go\x00c.go\x00" (expect the three names preserved) and for a real binary header (still a notice). Run go test -run TestSanitizeToolOutput ./pkg/agent/runner/.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Went with (a), in 34937b6: output is binary only when a NUL in the sniff window comes with invalid UTF-8 or a non-text control character; otherwise the text is kept and each NUL becomes U+FFFD. The rule is documented on sanitizeToolOutput and looksBinary, the notice is tool-neutral, and the -z listing case is in the table.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at 34937b6c: looksBinary needs a NUL plus invalid UTF-8 or a non-text control character, otherwise replaceNUL keeps the text, and the notice is tool-neutral. The -z listing, ANSI, straddle, ELF and PNG cases pass. One related nit (ASCII separators 0x1C-0x1F) is raised inline as a new comment.

Comment thread pkg/memory/facade.go
ErrInvalidEntry, i, lnk.ID, lnk.Kind, target.IDPrefix())
}
}
if err := checkStorable(e); err != nil {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[major] The facade now refuses NUL on every backend, but only tool output is sanitized before the write

Problem: checkStorable rejects any entry with NUL, but the runner only cleans executeToolRaw Content. Other text still reaches memory raw: assistant text and tool_use Input JSON (pkg/agent/runner/loop.go:768, a model-emitted \u0000), inbound user/channel text (loop.go:273, :1065), and toolkit stream events recorded by toolsession.Record (internal/cmd/runner/main.go:3291, best-effort so only that event is lost; I could not confirm the stream carries NUL). Each of these fails the turn Append exactly as the incident did. On Postgres that is unchanged; on inmem and sqlite (local and desktop installs) it is a new failure introduced by this check.
Verify: Check which of those producers can actually carry NUL (an LLM provider decode of "\u0000" into a Go string, a webhook body with \u0000).
Fix: Sanitize at the point where memory.Turn is built for Append (one helper in pkg/agent/runner that walks text blocks, tool_use input and tool_result content), so every block type is covered, and reuse the NUL handling from sanitizeToolOutput. Do not put this in the Postgres backend (signed append-only records must not be rewritten at the store, per the PR description). Apply the same helper to the toolsession.Event fields at internal/cmd/runner/main.go:3291 if the verify step shows they can carry NUL.
Test: In pkg/agent/runner add a loop test where the scripted LLM returns assistant text with NUL and a tool_use whose input contains \u0000, and assert the turn Append succeeds and the persisted text has no NUL. Run go test -run 'Sanitize|Dispatch' ./pkg/agent/runner/.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 34937b6, but in turn.Appender.Append rather than a runner helper: channelsd writes inbound messages through the same appender before the runner sees them, so a runner-only helper would have missed that writer. It replaces before signing, covers every field, and is logged; toolsession.Record does the same. One correction: tool_use Input is []byte and stored base64-encoded, so an escape in it never reaches the column (test added).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at 34937b6c: turn.Appender.Append replaces unstorable characters before signing and logs it, and every live turn writer (runner, channelsd, inboxwake, the restart inbox append) uses it. You are right that tool_use Input is base64. TestAppender_UnstorableCharactersAreReplaced passes. One gap is raised inline: memcopy.CopyPrefix re-puts legacy turns without going through Append.

return fmt.Errorf("field %s contains a NUL character", t.field)
}
}
return checkStorableJSON(e.Content)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] Raw Content is not checked for invalid UTF-8, and Provenance strings are not checked at all

Problem: checkStorableJSON only looks for NUL and \u escapes. A probe confirmed Content = {"a":"\xff"} passes checkStorable, as does a NUL in Provenance.Publisher. Postgres jsonb refuses invalid UTF-8 (22021), so a client posting raw invalid-UTF-8 JSON gets accepted on inmem/sqlite and refused (via the backstop, as 400) only on Postgres. That is the backend asymmetry this change says it removes. Turn content built by json.Marshal is not affected, because Marshal replaces invalid UTF-8.
Fix: In checkStorable, add if !utf8.Valid(e.Content) { return fmt.Errorf("content is not valid UTF-8") } before checkStorableJSON. Also check the Provenance string fields (Publisher, KeyID and any other string fields in memory.Provenance, pkg/memory/entry.go:38) through the same texts loop when e.Provenance != nil.
Test: Extend the table in pkg/memory/entry_storable_internal_test.go with invalid-UTF-8 Content and NUL in Provenance.Publisher. Run go test -run 'Storable' ./pkg/memory/.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 34937b6. checkStorable now refuses invalid UTF-8 in Content and checks Provenance Publisher, KeyID and PrevHash; both cases are in the facade table.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at 34937b6c: checkStorable refuses invalid UTF-8 in Content and checks Provenance Publisher, KeyID and PrevHash (pkg/memory/entry_storable.go), with facade table cases.

Comment thread pkg/memory/entry_storable.go Outdated
{"ID", e.ID},
}
for i, t := range e.Tags {
texts = append(texts, text{fmt.Sprintf("Tags[%d]", i), t})

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] Labels are formatted with fmt.Sprintf on every Put, even when nothing is wrong

Problem: Lines 27 and 30 allocate a label string per tag and five per link on the success path of every write, a hot path for transcript turns.
Fix: Store the field index/name lazily (for example a func() string or a {prefix string; i int}), or run the checks inline and format the label only when a check fails.
Test: Existing entry_storable_internal_test.go cases must still report the same field names. Run go test -run Storable ./pkg/memory/.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 34937b6. Labels are built only when a field fails; the field names are unchanged.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at 34937b6c: labels are built only when a field fails.

… replace unstorable characters when a turn is recorded

Tool output is now treated as binary only when a NUL comes with bytes that
do not read as text, so NUL-separated listings keep their content. A
secret-output value is passed through untouched, and an error result keeps
its text. Altered output is logged.

A turn or tool-session event carrying a character the store refuses is
recorded with a placeholder instead of failing the write, whichever writer
or block it came from. The store's own check now also refuses invalid UTF-8
in content and bad provenance strings, and no longer builds field labels on
the success path.
@josephschorr

Copy link
Copy Markdown
Member Author

Answers to the review's questions, as of 34937b6:

  1. MemoryUnavailable label. Not changed here. The runner can't tell a permanent rejection from an outage over HTTP today, so this needs a new failure reason plus error plumbing; I'd rather do that as a follow-up. Turns are now cleaned before the write, so the path is only reachable for Postgres rejections the up-front check doesn't name. The description now says this.
  2. Binary rule. A NUL in the first 8000 bytes plus invalid UTF-8 or a non-text control character. A NUL alone keeps the text, so separator-style output survives.
  3. Re-put of an older entry containing NUL. Still refused at the store, intentionally; reads are unaffected, so sqlite needs no migration. For turns, re-appending one stored with NUL stays a no-op (TestAppender_ReappendOfTurnStoredWithNULIsNoOp).
  4. Visibility. Every alteration is logged with the session, the tool or entry, and a count.

The requested loop test is covered at the appender instead, since that is where the fix lives. All three suites pass locally; the earlier e2e failure did not recur.

@samkim samkim left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 2 (commit 34937b6, since 6919661). All 5 earlier threads are verified fixed (5 of 5). I re-read the code and ran the targeted tests for the sanitizer, the turn appender, toolsession and the facade checks, and they pass.

Verdict: ready after the inline fixes. Both new findings are small. The first (N1) needs a decision from you before it is a fix.

Open issues, highest impact first:

  • Minor: a session whose turn was stored with a NUL before this PR (possible on sqlite installs; inmem does not survive the upgrade) cannot be restarted or forked after upgrade. memcopy.CopyPrefix re-puts the signed entry unchanged and the new storable check refuses it. See inline on pkg/memory/kinds/turn/accessor.go.
  • Nit: output separated by NUL and the ASCII unit or record separators (git log -z --format=%H%x1f%an) is still classed as binary and replaced with the notice. See inline on pkg/agent/runner/loop_binaryoutput.go.

Questions for the author:

  1. Should a legacy turn that contains NUL stay forkable? If yes, which fix do you prefer: exempt the copy path from the storable check, or have the copier rewrite the entry? The second is hard because the entry is signed. If you would rather accept the limitation, say so in the Appender comment and the PR description.

The earlier questions are answered: the MemoryUnavailable label is deferred to a follow-up, which is reasonable because turns are cleaned before the write now. The binary rule is documented, and altered data is logged.

Not anchored to a diff line: none.

This review was drafted by an automated pipeline and checked before submission.

// record, so refusing it fails the session, over one character in a model's
// reply or a user's message. A tool call's arguments are not at risk: they are
// bytes, stored base64-encoded. Every writer of a turn — the runner, and
// channelsd for inbound messages — comes through this method, so replacing

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] Comment claims every turn writer goes through Append, but memcopy bypasses it

Problem: This comment (and the legacy-turn leniency in Append) assumes all turn writers use turn.Appender.Append. memcopy.CopyPrefix (pkg/memory/memcopy/memcopy.go:61-65, called from CopyMemoryPrefix at pkg/controllers/agentsession/restart.go:139) copies existing turn entries verbatim with m.Put. The facade's checkStorable (pkg/memory/facade.go:291) runs on every Put. A turn stored with a NUL (the JSON escape backslash-u-0000) before this PR (sqlite accepted it, and its data survives the upgrade) is refused with ErrInvalidEntry, so restarting or forking that session fails with "memcopy.CopyPrefix: put". The entry is signed, so the copier cannot rewrite it.
Verify: This is a design decision, so do not pick a fix without the author's answer (see the question in the review body). Options: (a) in CopyPrefix, skip or relax checkStorable for entries already persisted, for example through a copy-only context flag the facade honours for entries that carry a valid Provenance; (b) accept the limitation and say so. Whichever is chosen, correct this comment so it names memcopy as the exception or says why the copy is safe.
Test: In pkg/memory/memcopy/memcopy_test.go add TestCopyPrefix_LegacyNULTurn that seeds a source scope with a turn entry whose content holds a NUL escape (backslash-u-0000) through a backend that bypasses the facade check, then asserts the expected behaviour (copy succeeds, or a clear error if the limitation is accepted). Run go test -run TestCopyPrefix ./pkg/memory/memcopy/.

for _, r := range window {
switch {
case r == 0, r == '\b', r == '\x1b', unicode.IsSpace(r):
case unicode.IsControl(r):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] ASCII separator controls (0x1C-0x1F) make NUL-separated text look binary

Problem: The case unicode.IsControl(r) branch returns true for U+001C to U+001F because unicode.IsSpace is false for them (probed at head). Output such as git log -z --format=%H%x1f%an%x1f%s is valid UTF-8 text with NUL record ends and unit separators, but looksBinary returns true and the model gets the binary notice instead of the text it asked for.
Fix: Add r >= 0x1c && r <= 0x1f to the allowed case on line 94 (next to \b, ESC and unicode.IsSpace), and mention the separators in the looksBinary doc comment above.
Test: In pkg/agent/runner/loop_binaryoutput_test.go add a table case in TestSanitizeToolOutput with input "abc\x1fdef\x00ghi\x1f\x00" expecting the text kept with each NUL replaced by U+FFFD. Run go test -run TestSanitizeToolOutput ./pkg/agent/runner/.

Comment thread pkg/agent/runner/loop_dispatch.go Outdated
// Both dispatch paths (gated and ungated meta) execute through here, so
// cleaning once covers every consumer downstream: the Post hooks and their
// audit records, the LLM tool_result, and the session's turn record.
res.Content = sanitizeToolOutput(res.Content)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at 34937b6c: cleanToolOutput returns secret-output results untouched and only replaces NULs in error results (pkg/agent/runner/loop_binaryoutput.go). TestDispatch_OutputSanitizerExceptions covers both.

Comment thread pkg/agent/runner/loop_binaryoutput.go Outdated
if i < 0 {
return content
}
if i < binarySniffLen {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at 34937b6c: looksBinary needs a NUL plus invalid UTF-8 or a non-text control character, otherwise replaceNUL keeps the text, and the notice is tool-neutral. The -z listing, ANSI, straddle, ELF and PNG cases pass. One related nit (ASCII separators 0x1C-0x1F) is raised inline as a new comment.

Comment thread pkg/memory/facade.go
ErrInvalidEntry, i, lnk.ID, lnk.Kind, target.IDPrefix())
}
}
if err := checkStorable(e); err != nil {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at 34937b6c: turn.Appender.Append replaces unstorable characters before signing and logs it, and every live turn writer (runner, channelsd, inboxwake, the restart inbox append) uses it. You are right that tool_use Input is base64. TestAppender_UnstorableCharactersAreReplaced passes. One gap is raised inline: memcopy.CopyPrefix re-puts legacy turns without going through Append.

return fmt.Errorf("field %s contains a NUL character", t.field)
}
}
return checkStorableJSON(e.Content)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at 34937b6c: checkStorable refuses invalid UTF-8 in Content and checks Provenance Publisher, KeyID and PrevHash (pkg/memory/entry_storable.go), with facade table cases.

Comment thread pkg/memory/entry_storable.go Outdated
{"ID", e.ID},
}
for i, t := range e.Tags {
texts = append(texts, text{fmt.Sprintf("Tags[%d]", i), t})

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at 34937b6c: labels are built only when a field fails.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants