Conversation
The streaming path decoded every token id on its own (`decode_tokens(engine, [tid])`) and shipped the result as the SSE delta. A byte-level BPE token is a byte fragment, not a character, so any multi-byte UTF-8 character split across ids decoded to U+FFFD and the streamed reply was corrupted, while the non-streaming path — which decodes the whole id list — returned correct text. Keep the ids seen so far, decode the cumulative list, and emit only the newly completed suffix; a trailing replacement character is held back until the next id resolves it, so a partial multi-byte character is never emitted, and the tail is flushed when generation ends so the concatenated deltas equal the non-streaming text exactly. Add a regression test that streams an id sequence whose UTF-8 bytes are split across ids through the handler and asserts the deltas match the non-streaming text with no U+FFFD.
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Streamed SSE responses now decode the full id list cumulatively and emit only the newly completed suffix, holding back partial multi-byte characters until they resolve.
Root cause
The streaming path decoded each token id in isolation (src/edge0/server/app.py:83). A byte-level BPE token carrying part of a multi-byte character streamed as U+FFFD while the non-streaming path returned correct text.
Changes
Validation
Live edge0-8b server run (stdlib transport), prompt asking the model to echo 🌊🏖️🦀🍣:
The non-streaming path decodes the whole id list and was already correct.
Test: tests/test_server.py::test_chat_stream_incremental_decode_of_split_characters
Limits
Common CJK text in the shipped vocabulary is not affected. Corruption needs characters that split across byte tokens (emoji, rare CJK, some Korean).
O(n²) cumulative re-decode per request. This is the minimal fix. Throughput was not measured.