Skip to content

refactor(epub): Japanese CJK line-breaking PR E — WordJoin setup#5

Merged
mtskf merged 5 commits into
masterfrom
feat/cjk-wordjoin-setup
Jun 18, 2026
Merged

refactor(epub): Japanese CJK line-breaking PR E — WordJoin setup#5
mtskf merged 5 commits into
masterfrom
feat/cjk-wordjoin-setup

Conversation

@mtskf

@mtskf mtskf commented Jun 18, 2026

Copy link
Copy Markdown
Owner

Japanese / CJK Line-Breaking — PR E (Phase 2: WordJoin setup)

Second PR of the Japanese CJK line-breaking effort (follows PR D, Phase 1 scaffolding). This is a
SETUP PR: it adds the UTF-8 predicates, the 3-state WordJoin model, the addWord(WordJoin) API,
the parser bridge rename, and layout-level Japanese tests. The cluster assembler is NOT wired into
the parser yet (that is PR F), so the parser still chunks Japanese exactly as today.

Behavior scope (read this)

  • English / Latin layout: UNCHANGED (all Phase 1 characterization tests stay green).
  • Common Japanese text wrapping: UNCHANGED (the parser produces the same chunks as master; the new
    WordJoin::CjkBreak state exists but is never produced at runtime in this PR).
  • Two intentional, narrowly-scoped CJK setup changes land here (front-loaded so PR F is a clean
    assembler-wiring diff), each affecting only rare/edge inputs:
    1. utf8IsCjkBreakable is extended to supplementary CJK ranges. Its pre-existing consumer
      Hyphenator therefore suppresses inserted hyphens for those supplementary-plane CJK
      characters too — a consistent extension of the existing "CJK gets no hyphen" rule, observable
      only for text containing supplementary-plane CJK.
    2. With focus-reading enabled, a single CJK grapheme cluster is no longer focus-bolded (a
      boundary-isolated CJK char that master would have bolded now renders regular). This is the
      correct end-state for per-character CJK tokens (PR F) and is unit-tested here.

Why

The layout join flag was a single bool (wordContinues) with only two states — "space +
breakable" (word gap) and "no-space + non-breakable" (no-break-space glue) — but CJK needs a third:
"no-space + breakable" (break between ideographs, no space). This PR generalises the flag to a
3-state enum and makes CJK codepoints first-class breakable units at the predicate level. Every
read site is routed through needsSpaceBefore (space-vs-kerning gap) / canBreakBefore (break
allowed). The two legacy states map identically (falseSpace, trueGlue); the predicates
differ only on CjkBreak, so Latin/legacy output is bit-identical.

Commits

  1. feat(utf8): extend utf8IsCjkBreakable and add utf8IsGraphemeExtender
    • utf8IsCjkBreakable extended to supplementary CJK ranges (Kana Supplement/Extended-A/B, Small
      Kana Extension, CJK Extensions D/E/F/G/H/I/J, Compatibility Ideographs Supplement).
    • New utf8IsGraphemeExtender (combining marks, kana voicing U+3099-309A / U+FF9E-FF9F,
      ideographic tone marks U+302A-302F, variation selectors U+FE00-FE0F / U+E0100-E01EF).
    • Isolated host gtest target test/utf8/ (links only Utf8.cpp).
  2. refactor: model word joins as 3-state WordJoin (space/glue/cjk-break)
    • New lib/Epub/Epub/WordJoin.h (enum class WordJoin {Space,Glue,CjkBreak} + needsSpaceBefore
      / canBreakBefore constexpr predicates).
    • ParsedText.{h,cpp}: wordContinues bool → WordJoin; every read site routed through the
      predicates; isSingleCjkGraphemeCluster focus-bypass helper (NFC + NFD).
    • Parser bridge: nextWordContinuesnextJoin (every =trueGlue, =falseSpace),
      in the same commit as the addWord signature change so the firmware compiles mid-bisect.
  3. test: Japanese text wraps at viewport width, not at fixed chunk
    • Layout-level Japanese tests: viewport-wrap (not fixed-chunk), Latin-survival in a CJK run,
      Space/Glue/CjkBreak join transitions, focus-bypass for NFC + NFD CJK clusters.
  4. fix(epub): preserve underline on CJK focus-bypass and guard embedded NUL
    • Cross-model (Codex) review follow-up: the focus-bypass now pushes the underline-applied style
      (was dropping underline on a single underlined CJK cluster), and isSingleCjkGraphemeCluster
      is length-bounded (rejects an embedded NUL instead of truncating validation). Adds a regression
      test pinning underline survival through the bypass.

Out of scope (deferred to PR F)

  • Utf8ClusterAssembler and its wiring into ChapterHtmlSlimParser (parser still chunks as today).
  • SECTION_FILE_VERSION bump (no cache-format change).
  • Kinsoku (禁則処理), vertical text, GfxRenderer cluster-aware rendering.

Known pre-existing limitation (follow-up, NOT introduced here)

The focus-reading tokenizer in ParsedText::addWord (the while (ptr < end) loops at
ParsedText.cpp:220 and :270) can infinite-loop on a word containing an embedded NUL byte, because
utf8NextCodepoint returns 0 without advancing the pointer at a NUL. This hazard exists on master
independently of this PR (a Codex review surfaced it). This PR does not introduce or regress it:
isSingleCjkGraphemeCluster's NUL guard (commit 4) returns the CJK-base+NUL case to master's
behavior. The input is practically unreachable from EPUB content (XML 1.0 forbids U+0000; expat
rejects it), so it is deferred to a dedicated focus-tokenizer robustness PR rather than expanding
this behavior-unchanged setup PR.

Verification

  • Host cmake gtest: full suite 117/117 PASS, including the 4 Phase-1 ParsedTextLayout cases
    UNCHANGED (behavior-preservation oracle) + the new Japanese cases + test/utf8 predicates +
    the underline regression. Focus-bypass non-vacuity confirmed by revert-check.
  • Cross-model review: Codex pass over the full diff — read-site mappings, scope guards, and
    no-runtime-CjkBreak all confirmed; the two findings it raised (underline drop, embedded NUL) are
    fixed in commit 4.
  • Firmware build verified by CI (no PlatformIO on the dev machine); clang-format applied by CI.

@mtskf mtskf changed the title Japanese / CJK Line-Breaking — PR E (Phase 2: WordJoin setup) refactor(epub): Japanese CJK line-breaking PR E — WordJoin setup Jun 18, 2026
@mtskf mtskf merged commit b515bc4 into master Jun 18, 2026
7 of 8 checks passed
@mtskf mtskf deleted the feat/cjk-wordjoin-setup branch June 18, 2026 03:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant