Skip to content

fix(acp): keep the connection open when an incoming line is not valid UTF-8 - #504

Merged
benbrandt merged 1 commit into
mainfrom
heal-non-utf8-lines
Nov 16, 2026
Merged

benbrandt merged 1 commit into
mainfrom
heal-non-utf8-lines

Conversation

@benbrandt

Copy link
Copy Markdown
Member

Problem

ByteStreams, Stdio and AcpAgent read incoming lines with BufReader::lines(), which reports a line that is not valid UTF-8 as an InvalidData read error. The incoming transport actor treats every read error as a transport failure. One such line therefore ended the whole connection with Internal error: stream did not contain valid UTF-8, along with every session on it.

A line that is only not JSON is already handled differently. It becomes a TransportFrame::Malformed, is answered with a parse error, and the connection stays usable.

Change

One shared reader, lossy_lines, splits the byte stream into lines and replaces bytes that are not valid UTF-8 with U+FFFD, the way the agent's stderr is already decoded. All three transports use it.

  • A bad byte inside a JSON string: the message is delivered with U+FFFD in that text.
  • A bad byte anywhere else: the line is still not JSON, so it is a malformed frame like any other.

Replacing the byte instead of dropping the line keeps a response or notification from going missing, which would leave the other side waiting.

Lines is unchanged. It takes strings that are already decoded, and an Err from its incoming stream still ends the connection. Its docs now say so.

Testing

  • New end-to-end test over ByteStreams, test_invalid_utf8_is_replaced_and_connection_stays_alive. It fails without the fix.
  • New unit test of lossy_lines for line endings and the replacement.
  • just test, clippy for native and wasm targets, cargo fmt --check, and rustdoc with -D warnings pass locally.

Stdio and AcpAgent have no test of their own for this. They call the same function as ByteStreams.

… UTF-8

`ByteStreams`, `Stdio` and `AcpAgent` read incoming lines with
`BufReader::lines()`, which reports a line that is not valid UTF-8 as a
read error. The incoming transport actor treats every read error as a
transport failure, so one such line ended the whole connection, and every
session on it. A line that is only not JSON was already a malformed frame
that is answered while the connection stays usable.

One shared reader now replaces bytes that are not valid UTF-8 with U+FFFD,
as the agent's stderr is already decoded. A line with such a byte inside a
JSON string is delivered with the replacement. Any other line is still not
JSON, so it is a malformed frame like any other. Replacing the byte instead
of dropping the line keeps a response or notification from going missing.

`Lines` takes decoded strings and is unchanged. Its docs now say that an
error from its incoming stream ends the connection.
@benbrandt
benbrandt enabled auto-merge (squash) November 16, 2026 17:21
@benbrandt
benbrandt merged commit 6a4a95c into main Nov 16, 2026
13 checks passed
@benbrandt
benbrandt deleted the heal-non-utf8-lines branch November 16, 2026 17:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant