Grammar: accept combining marks/ZWNJ/ZWJ as prose content; restrict entity regex to semicolon-terminated names (bd-96fswwce, bd-v8qc9zyc)

claude-notes/plans/2026-08-10-combining-marks-parse.md

@@ -0,0 +1,102 @@
+# Parser rejects combining characters / join controls in prose (bd-96fswwce)
+
+**Date:** 2026-08-10
+**Braid:** bd-96fswwce (bug, P2 — arguably P1, see scope), discovered-from bd-named-entities-w6xbfftj
+**Checkout:** main (on top of the entity-decode work, PR #488)
+**Status:** Implemented 2026-08-10; all verification legs green (piecewise — see Phase 2 caveat).
+
+## Problem
+
+`pandoc_str` in `crates/tree-sitter-qmd/tree-sitter-markdown/grammar.js` is an
+explicit union of content character classes (alphanumerics, curated symbol
+categories, non-ASCII Po/Pc punctuation, smart quotes, non-ASCII whitespace).
+Combining marks (`Mn`/`Mc`/`Me`) and the join controls ZWNJ/ZWJ
+(U+200C/U+200D) are absent, so the lexer finds no token and the parse errors.
+Same bug family and fix shape as bd-6kewx (which added non-ASCII `Po`/`Pc`
+after bare `§` in prose produced parse errors).
+
+All of these are hard parse errors today (verified at `9509f8d2`):
+
+| Input | Class | Note |
+|---|---|---|
+| `x ≂̸ y` (U+2242 U+0338) | Mn after symbol | what `≂̸` decodes to |
+| `cafe` + U+0301 | Mn mid-word | any NFD/decomposed text |
+| `का` (क + U+093E) | Mc | **any Hindi text with vowel signs** |
+| `a` + U+20DD | Me | enclosing marks |
+| `ab` + U+200C + `cd` | Cf (ZWNJ) | Persian/Indic joining control |
+| `ab` + U+200D + `cd` | Cf (ZWJ) | outside emoji sequences |
+
+Pandoc parity (verified with system pandoc): every one of these folds into
+`Str` verbatim (`cafe\769`, `ab\8204cd`, `a\8413`, …).
+
+## Fix
+
+In `grammar.js`, define a content class for combining marks + join controls
+(`\p{M}\u{200C}\u{200D}`) and add it to `PANDOC_REGEX_STR`:
+
+1. as a new single-char alternative (mark after a symbol token / after space /
+   at line start — the AST converter already merges adjacent `Str`s), and
+2. inside the word-continuation class (so `cafe` + U+0301, `का`, `ab‌cd` stay
+   single tokens).
+
+ZWJ interplay with `EMOJI_REGEX` (which uses U+200D inside emoji sequences) is
+safe: the lexer takes the longest match, so emoji ZWJ sequences still win.
+
+Then `tree-sitter generate; tree-sitter build; tree-sitter test` in
+`crates/tree-sitter-qmd/tree-sitter-markdown/` (build.rs recompiles the
+generated `parser.c` into the Rust crate).
+
+## Work items
+
+### Phase 0 — Tests first (TDD)
+
+- [x] Corpus tests `test/corpus/combining_marks.txt` (new file only): the six
+      cases above; verified all 6 fail pre-fix (ERROR nodes).
+- [x] Rust coverage tests in
+      `crates/pampa/tests/integration/test_treesitter_coverage.rs` asserting
+      verbatim `Str` content for each case; verified all 6 fail pre-fix.
+- [x] Roundtrip: re-added `≂̸` to
+      `tests/roundtrip_tests/qmd-json-qmd/named_entities.qmd` and added a
+      literal `combining_marks.qmd` fixture.
+
+### Phase 1 — Grammar fix
+
+- [x] Added `PANDOC_COMBINING_MARKS = "\\p{M}\\u{200C}\\u{200D}"` to
+      `grammar.js` (single-char alternative + word-continuation class);
+      `tree-sitter generate`; `tree-sitter build`; full corpus suite green
+      (551/551, incl. the 6 new — `\p{M}` compound category accepted).
+- [x] Rust tests pass (coverage + roundtrip + entity tests: 14/14).
+
+### Phase 2 — Verification + bookkeeping
+
+- [x] `cargo nextest run --workspace`: 11237 passed, 197 skipped.
+- [x] End-to-end `q2 render` of a fixture with literal `x ≂̸ y`, Hindi
+      `का matra`, NFD `cafe`+U+0301, and ZWNJ/ZWJ words: all render verbatim;
+      NFD sequence byte-verified in the HTML (`6361 6665 cc81` — not
+      normalized).
+- [x] Verification of all `cargo xtask verify` legs — **piecewise** (see
+      caveat): lints/clippy + workspace build green in every attempt; Rust
+      workspace tests 11237/11237 post-change; hub-client `build:all` (incl.
+      WASM from the regenerated parser) green; hub-client `test:ci` 131/131
+      green; ts-packages build green.
+      **Caveat:** three consecutive single-shot `cargo xtask verify` runs
+      failed on *different*, unrelated network-dependent tests (quarto-hub
+      auth, q2-preview listener, sync-client websocket), each passing in
+      isolation. Root cause is environmental: ~2,300 orphaned Jupyter
+      ipykernel processes (started 2026-08-06, PPID 1) hold 13,973 of the
+      16,384 ephemeral ports, so parallel test bursts hit EADDRNOTAVAIL on
+      loopback. Cleanup (`pkill -f ipykernel_launcher`) left to the user —
+      it would also kill any live notebook kernels. CI on the PR provides the
+      independent single-shot check.
+- [x] Commit; close bd-96fswwce.
+
+## Risks
+
+- Adding `\p{M}` to the *continuation* class only (not `startStrRegex`) keeps
+  emphasis/underscore boundary logic untouched; mark-initial runs lex via the
+  single-char alternative instead.
+- `tree-sitter generate` must support `\p{M}` (compound category). The grammar
+  already uses `\p{L}`/`\p{N}`/`\p{So}`; if compound `M` is rejected, fall
+  back to `\p{Mn}\p{Mc}\p{Me}`.
+- State-count growth in the generated parser is possible; watch `parser.c`
+  size and generation time.

claude-notes/plans/2026-08-10-entity-regex-legacy-names.md

@@ -0,0 +1,66 @@
+# Grammar html_entity_regex() mangles legacy no-semicolon entity names (bd-v8qc9zyc)
+
+**Date:** 2026-08-10
+**Braid:** bd-v8qc9zyc (bug, P3), discovered-from bd-named-entities-w6xbfftj
+**Checkout:** main (stacked on the combining-marks work, PR #489)
+**Status:** Implemented 2026-08-10; full verify green.
+
+## Problem
+
+`html_entity_regex()` in `crates/tree-sitter-qmd/common/common.js` builds the
+`entity_reference` regex alternatives with `name.substring(1, name.length - 1)`.
+That strips `&`…`;` correctly for the 2,125 semicolon-terminated keys of
+`html_entities.json`, but the WHATWG table also carries 106 legacy keys
+*without* a trailing semicolon (`&AMP`, `&AElig`, …), whose last **letter**
+gets stripped instead — producing bogus alternatives (`AM`, `AEli`, …). Net
+effect (verified against the current parser):
+
+- `&AM;` lexes as `entity_reference` (then hits the converter's verbatim
+  fallback — output text is right, but the CST is wrong and the regex carries
+  106 garbage alternatives);
+- bare `&AMP` does **not** match — which is correct: CommonMark recognizes
+  only semicolon-terminated entities in markdown, so the legacy keys should
+  not be in the regex at all.
+
+No user-visible AST change results from the fix (`&AM;` merges to the same
+`Str "&AM;"` either way); this is grammar hygiene, pinned at the CST level.
+
+## Fix
+
+Filter the keys to semicolon-terminated ones before mapping:
+`Object.keys(html_entities).filter(name => name.endsWith(';'))`. Regenerate
+(`tree-sitter generate; tree-sitter build`), corpus tests. The converter's
+verbatim fallback in `crates/pampa/src/pandoc/treesitter_utils/entity_reference.rs`
+stays as defense-in-depth; its comment (which cites this strand as the only
+reachable-miss source) gets updated to reflect that misses are now
+unreachable from grammar-produced nodes.
+
+## Work items
+
+### Phase 0 — Tests first (TDD)
+
+- [x] Corpus tests `test/corpus/entity_reference_legacy.txt` (new file):
+      `&` → `entity_reference` (guard), `&AM;` → plain `pandoc_str`s
+      (fails pre-fix), bare `&AMP` → plain `pandoc_str`s (guard).
+      Verified pre-fix: 2 pass, `&AM;` case fails producing
+      `(entity_reference)`.
+- [x] Existing Rust guard: `test_entity_reference_unknown_emits_verbatim`
+      ("A &AM; B" → text "A &AM; B") was written to hold across this fix —
+      keeps passing before and after.
+
+### Phase 1 — Grammar fix
+
+- [x] Filter in `html_entity_regex()`; `tree-sitter generate`;
+      `tree-sitter build`; full corpus suite green (554/554); `parser.c`
+      shrank net −121 lines.
+- [x] Update the stale comments in pampa (`entity_reference.rs`,
+      `test_treesitter_coverage.rs`) that describe the pre-fix behavior.
+
+### Phase 2 — Verification + bookkeeping
+
+- [x] pampa + tree-sitter-qmd suites 4326/4326; workspace suite
+      11285/11285; full single-shot `cargo xtask verify` green (environment
+      recovered after the Jupyter-kernel port leak was cleaned up). One
+      unrelated macOS-only flake surfaced and was filed as bd-zazptk5s
+      (automerge storage splay-prefix case collision).
+- [x] Commit; close bd-v8qc9zyc.

crates/pampa/src/pandoc/treesitter_utils/entity_reference.rs

@@ -36,10 +36,11 @@ fn entity_table() -> &'static HashMap<String, String> {
 /// Process named entity references to their character values:
 /// `&gt;` => `>`, `&nbsp;` => U+00A0, `&copy;` => `©`, etc.
 ///
-/// A name missing from the table passes through verbatim. The regex the
-/// grammar matched with is generated from the same table, so misses are only
-/// reachable through its mangled legacy-name alternatives (bd-v8qc9zyc) —
-/// never from well-formed input — hence no diagnostic.
+/// A name missing from the table passes through verbatim, with no diagnostic.
+/// The regex the grammar matched with is generated from the same table
+/// (semicolon-terminated names only, since bd-v8qc9zyc), so a miss is not
+/// reachable from grammar-produced nodes; the fallback is defense-in-depth
+/// against the two data sources drifting apart.
 pub fn process_entity_reference(
     node: &tree_sitter::Node,
     input_bytes: &[u8],

crates/pampa/tests/integration/test_treesitter_coverage.rs

@@ -720,10 +720,11 @@ fn test_entity_reference_multi_codepoint() {
 
 #[test]
 fn test_entity_reference_unknown_emits_verbatim() {
-    // "&AM;" currently parses as entity_reference (truncated legacy alternative
-    // from the grammar regex, see bd-v8qc9zyc) but is not a valid entity name;
-    // it must survive as literal text. This assertion also holds after the
-    // grammar fix, when it becomes plain text.
+    // "&AM;" is not a valid entity name and must survive as literal text.
+    // Before bd-v8qc9zyc it lexed as entity_reference (a truncated legacy
+    // alternative in the grammar regex) and exercised the converter's
+    // verbatim fallback; since the regex fix it is plain text. The assertion
+    // holds either way, guarding both layers.
     let pandoc = parse_qmd("A &AM; B");
     assert_eq!(para_text(&pandoc, 0), "A &AM; B");
 }
@@ -747,3 +748,50 @@ fn test_entity_reference_inside_emphasis() {
     };
     assert_eq!(inlines_text(&emph.content), "A > B");
 }
+
+// ============================================================================
+// Combining marks / join controls in prose (bd-96fswwce)
+// Pandoc folds all of these into Str verbatim; they must not be parse errors.
+// ============================================================================
+
+#[test]
+fn test_combining_mark_after_symbol() {
+    // U+2242 U+0338 — what ≂̸ decodes to, written literally
+    let pandoc = parse_qmd("x \u{2242}\u{0338} y");
+    assert_eq!(para_text(&pandoc, 0), "x \u{2242}\u{0338} y");
+}
+
+#[test]
+fn test_combining_mark_inside_word() {
+    // Decomposed (NFD) accent: cafe + U+0301
+    let pandoc = parse_qmd("cafe\u{0301} fin");
+    assert_eq!(para_text(&pandoc, 0), "cafe\u{0301} fin");
+}
+
+#[test]
+fn test_spacing_combining_mark_devanagari() {
+    // का = क (U+0915) + ा (U+093E, Mc) — any Hindi text with vowel signs
+    let pandoc = parse_qmd("\u{0915}\u{093E} matra");
+    assert_eq!(para_text(&pandoc, 0), "\u{0915}\u{093E} matra");
+}
+
+#[test]
+fn test_enclosing_combining_mark() {
+    // a + U+20DD (Me, combining enclosing circle)
+    let pandoc = parse_qmd("a\u{20DD} circled");
+    assert_eq!(para_text(&pandoc, 0), "a\u{20DD} circled");
+}
+
+#[test]
+fn test_zero_width_non_joiner_in_word() {
+    // U+200C between letters (Persian/Indic joining control)
+    let pandoc = parse_qmd("ab\u{200C}cd");
+    assert_eq!(para_text(&pandoc, 0), "ab\u{200C}cd");
+}
+
+#[test]
+fn test_zero_width_joiner_in_word() {
+    // U+200D between letters (outside emoji sequences)
+    let pandoc = parse_qmd("ab\u{200D}cd");
+    assert_eq!(para_text(&pandoc, 0), "ab\u{200D}cd");
+}

crates/pampa/tests/roundtrip_tests/qmd-json-qmd/combining_marks.qmd

@@ -0,0 +1,3 @@
+Symbol: x ≂̸ y.
+
+Word: café and का and ab‌cd.

crates/pampa/tests/roundtrip_tests/qmd-json-qmd/named_entities.qmd

@@ -1,5 +1,7 @@
 Named: A > B < C & D "E" F G © H.
 
+Multi-codepoint: x ≂̸ y.
+
 Numeric: A > B < C.
 
 Numeric quote: A "E" B.

crates/tree-sitter-qmd/common/common.js

@@ -174,8 +174,16 @@ function html_entity_regex() {
     // A file with all html entities, should be kept up to date with
     // https://html.spec.whatwg.org/multipage/entities.json
     let html_entities = require("./html_entities.json");
+    // Only semicolon-terminated names: CommonMark recognizes entities in
+    // markdown only with the trailing `;`. The table's 106 legacy keys
+    // without one (`&AMP`, `&AElig`, ...) must not contribute alternatives —
+    // substring() would strip their last *letter* instead of the `;`,
+    // yielding bogus names like `AM` (bd-v8qc9zyc).
     let s = '&(';
-    s += Object.keys(html_entities).map(name => name.substring(1, name.length - 1)).join('|');
+    s += Object.keys(html_entities)
+        .filter(name => name.endsWith(';'))
+        .map(name => name.substring(1, name.length - 1))
+        .join('|');
     s += ');';
     return new RegExp(s);
 }

crates/tree-sitter-qmd/tree-sitter-markdown/grammar.js

@@ -93,6 +93,16 @@ const regexOr = (...groups) => regexBracket(groups.join("|"));
 const PANDOC_NON_ASCII_WHITESPACE =
     "\\u{00A0}\\u{1680}\\u{2000}-\\u{200A}\\u{2028}\\u{2029}\\u{202F}\\u{205F}\\u{3000}";
 
+// Combining marks (Mn nonspacing, Mc spacing, Me enclosing) plus the join
+// controls ZWNJ (U+200C) and ZWJ (U+200D). Pandoc folds all of these into the
+// surrounding `Str` verbatim: decomposed accents (`cafe` + U+0301), Indic
+// vowel signs (`का` = U+0915 + U+093E), enclosing marks, and ZWNJ/ZWJ between
+// letters (Persian/Indic joining control) are content, not markup. Without
+// this class a bare mark in prose produced a parse ERROR (bd-96fswwce) — the
+// same bug family as bd-6kewx above. ZWJ inside emoji sequences is unaffected:
+// EMOJI_REGEX matches those as a longer token, which wins.
+const PANDOC_COMBINING_MARKS = "\\p{M}\\u{200C}\\u{200D}";
+
 const startStrRegex = regexOr(
     "[" + PANDOC_NON_ASCII_WHITESPACE + PANDOC_ALPHA_NUM + PANDOC_SMART_QUOTES + "-]");
 const afterUnderscoreRegex = "[" + PANDOC_ALPHA_NUM + "]";
@@ -113,10 +123,11 @@ const PANDOC_REGEX_STR =
             "[" + PANDOC_PUNCTUATION + "]",
             "[" + PANDOC_VALID_OTHER_PUNCTUATION + "]",
             "[" + PANDOC_VALID_SYMBOLS + "]",
+            "[" + PANDOC_COMBINING_MARKS + "]",
             "[>.,;!?]",
             startStrRegex +
             regexOr(
-                "[!,.;?" + PANDOC_NON_ASCII_WHITESPACE + PANDOC_ALPHA_NUM + PANDOC_SMART_QUOTES + "-]",
+                "[!,.;?" + PANDOC_NON_ASCII_WHITESPACE + PANDOC_ALPHA_NUM + PANDOC_SMART_QUOTES + PANDOC_COMBINING_MARKS + "-]",
                 // "\\\\.",
                 "['\\u{2018}\\u{2019}][\\p{L}\\p{N}]",
                 regexBracket("[_]" + afterUnderscoreRegex)