Repository navigation
Conversation
…ndaries in head/tail
read_text_file with head or tail corrupts multi-byte UTF-8 characters
that straddle the 1024-byte chunk read boundary. Each chunk was decoded
independently via Buffer.toString('utf-8'), so partial byte sequences
at chunk edges became U+FFFD.
- headFile: use node:string_decoder StringDecoder to hold trailing
partial bytes across reads
- tailFile: accumulate raw Buffer chunks and decode the concatenated
buffer once; count newlines at byte level (0x0A is unambiguous in
UTF-8). The discarded partial first line absorbs any U+FFFD at the
lowest read position.
Adds encoding.test.ts with real-I/O tests placing 3-byte and 4-byte
UTF-8 characters at exact chunk boundaries for both head and tail.
This comment was marked as abuse.
This comment was marked as abuse.
cliffhall
added
the
closed-superseded-v2
|
Thank you for this pull request, and apologies for the long wait for a response. Since it was opened, the reference servers have been rebuilt for the 2026-07-28 MCP specification (#4857) and released as v2 (2.0.0 on npm for the TypeScript servers, 2026.10.19 on PyPI for the Python servers). The code this PR changes has been replaced, and the behavior it addresses has either been fixed in v2 or no longer exists in the same form. This is not a judgment on the change or on you; it's a consequence of the code it targets having been replaced. If the problem still happens on v2, please open a fresh issue with the version you're running and steps to reproduce. Note that this repository now accepts contributions as issues rather than pull requests from anyone but its maintainers; see This is part of the backlog triage in #4875. |
Description
read_text_filewithheadortailcorrupts UTF-8 multibyte characters that straddle the 1024-byte chunk read boundary. The corrupted bytes becomeU+FFFD(replacement character). Reading the same file in full mode (nohead/tail) is unaffected.Server Details
lib.ts(headFile,tailFile)Motivation and Context
Both
headFileandtailFileread the file as raw byte chunks viafileHandle.read(chunk, 0, size, position)and then call.toString('utf-8')on each chunk independently before concatenating the resulting strings. UTF-8 multi-byte sequences (e.g. CJK characters at 3 bytes each, most non-Latin scripts at 2–4 bytes) that span a chunk boundary are split mid-sequence, so each chunk decodes its partial bytes asU+FFFDand the original character is lost.Reproduction (default 1024-byte chunk):
あ(UTF-8:E3 81 82), then a newline.head: 1returns ASCII followed by���instead ofあ.The same mechanism breaks
tailsymmetrically: any multi-byte character spanning a 1024-byte boundary measured from EOF is replaced withU+FFFD.This impacts every script outside Basic Latin in practice — Japanese/Chinese/Korean, Cyrillic, Greek, Arabic, accented Latin (German umlauts, French/Spanish accents, etc.), and emoji.
Changes
headFile: usenode:string_decoderStringDecoderso trailing partial bytes from one read are held internally and combined with the leading bytes of the next read.decoder.end()is called on EOF to flush any genuinely truncated sequence.tailFile: accumulate raw bytes into aBuffer[]in iteration order and decode the concatenated buffer once at the end. Newline counting is performed at the byte level (safe because0x0Anever appears inside a UTF-8 multi-byte sequence by design). The discarded partial first line absorbs anyU+FFFDintroduced at the lowest read position.Behavior on ASCII input is unchanged. Memory characteristics of
tailFileare unchanged (chunks accumulated correspond to the same range of bytes the original read).How Has This Been Tested?
src/filesystem/__tests__/encoding.test.tsusing real temp file I/O (nofsmocking) that place 3-byte (あ) and 4-byte (😀) UTF-8 characters at exact 1024-byte chunk boundaries for both head and tail directions. The new tests fail onmainand pass with this change.lib.test.tsand other suites continue to pass — full local run shows 156 passed across 8 test files.read_text_filewithheadortailreturnedU+FFFDat multiples of 1024 bytes from the relevant end.Breaking Changes
None. The fix only affects previously incorrect output for non-ASCII input.
Types of changes
Checklist
read_text_filealready documents UTF-8 support; this fix brings behavior in line with documentation)