refactor(epub): Japanese CJK line-breaking PR E — WordJoin setup#5
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Japanese / CJK Line-Breaking — PR E (Phase 2: WordJoin setup)
Second PR of the Japanese CJK line-breaking effort (follows PR D, Phase 1 scaffolding). This is a
SETUP PR: it adds the UTF-8 predicates, the 3-state
WordJoinmodel, theaddWord(WordJoin)API,the parser bridge rename, and layout-level Japanese tests. The cluster assembler is NOT wired into
the parser yet (that is PR F), so the parser still chunks Japanese exactly as today.
Behavior scope (read this)
WordJoin::CjkBreakstate exists but is never produced at runtime in this PR).assembler-wiring diff), each affecting only rare/edge inputs:
utf8IsCjkBreakableis extended to supplementary CJK ranges. Its pre-existing consumerHyphenatortherefore suppresses inserted hyphens for those supplementary-plane CJKcharacters too — a consistent extension of the existing "CJK gets no hyphen" rule, observable
only for text containing supplementary-plane CJK.
boundary-isolated CJK char that master would have bolded now renders regular). This is the
correct end-state for per-character CJK tokens (PR F) and is unit-tested here.
Why
The layout join flag was a single
bool(wordContinues) with only two states — "space +breakable" (word gap) and "no-space + non-breakable" (no-break-space glue) — but CJK needs a third:
"no-space + breakable" (break between ideographs, no space). This PR generalises the flag to a
3-state enum and makes CJK codepoints first-class breakable units at the predicate level. Every
read site is routed through
needsSpaceBefore(space-vs-kerning gap) /canBreakBefore(breakallowed). The two legacy states map identically (
false⇔Space,true⇔Glue); the predicatesdiffer only on
CjkBreak, so Latin/legacy output is bit-identical.Commits
feat(utf8): extend utf8IsCjkBreakable and add utf8IsGraphemeExtenderutf8IsCjkBreakableextended to supplementary CJK ranges (Kana Supplement/Extended-A/B, SmallKana Extension, CJK Extensions D/E/F/G/H/I/J, Compatibility Ideographs Supplement).
utf8IsGraphemeExtender(combining marks, kana voicing U+3099-309A / U+FF9E-FF9F,ideographic tone marks U+302A-302F, variation selectors U+FE00-FE0F / U+E0100-E01EF).
test/utf8/(links onlyUtf8.cpp).refactor: model word joins as 3-state WordJoin (space/glue/cjk-break)lib/Epub/Epub/WordJoin.h(enum class WordJoin {Space,Glue,CjkBreak}+needsSpaceBefore/
canBreakBeforeconstexpr predicates).ParsedText.{h,cpp}:wordContinuesbool →WordJoin; every read site routed through thepredicates;
isSingleCjkGraphemeClusterfocus-bypass helper (NFC + NFD).nextWordContinues→nextJoin(every=true→Glue,=false→Space),in the same commit as the
addWordsignature change so the firmware compiles mid-bisect.test: Japanese text wraps at viewport width, not at fixed chunkSpace/Glue/CjkBreak join transitions, focus-bypass for NFC + NFD CJK clusters.
fix(epub): preserve underline on CJK focus-bypass and guard embedded NUL(was dropping underline on a single underlined CJK cluster), and
isSingleCjkGraphemeClusteris length-bounded (rejects an embedded NUL instead of truncating validation). Adds a regression
test pinning underline survival through the bypass.
Out of scope (deferred to PR F)
Utf8ClusterAssemblerand its wiring intoChapterHtmlSlimParser(parser still chunks as today).SECTION_FILE_VERSIONbump (no cache-format change).Known pre-existing limitation (follow-up, NOT introduced here)
The focus-reading tokenizer in
ParsedText::addWord(thewhile (ptr < end)loops atParsedText.cpp:220 and :270) can infinite-loop on a word containing an embedded NUL byte, because
utf8NextCodepointreturns 0 without advancing the pointer at a NUL. This hazard exists on masterindependently of this PR (a Codex review surfaced it). This PR does not introduce or regress it:
isSingleCjkGraphemeCluster's NUL guard (commit 4) returns the CJK-base+NUL case to master'sbehavior. The input is practically unreachable from EPUB content (XML 1.0 forbids U+0000; expat
rejects it), so it is deferred to a dedicated focus-tokenizer robustness PR rather than expanding
this behavior-unchanged setup PR.
Verification
UNCHANGED (behavior-preservation oracle) + the new Japanese cases +
test/utf8predicates +the underline regression. Focus-bypass non-vacuity confirmed by revert-check.
no-runtime-CjkBreak all confirmed; the two findings it raised (underline drop, embedded NUL) are
fixed in commit 4.