diff --git a/README.md b/README.md index 208c99d4..4c30bcac 100644 --- a/README.md +++ b/README.md @@ -2,9 +2,9 @@ [![GitHub](https://img.shields.io/badge/GitHub-181717?logo=github&logoColor=white)](https://github.com/ExaDev/documents.js) [![npm](https://img.shields.io/badge/npm-CB3837?logo=npm&logoColor=white)](https://www.npmjs.com/package/documents.js) [![Release](https://img.shields.io/github/v/release/ExaDev/documents.js)](https://github.com/ExaDev/documents.js/releases/latest) [![CI](https://img.shields.io/github/actions/workflow/status/ExaDev/documents.js/ci.yml?branch=main)](https://github.com/ExaDev/documents.js/actions) -> Converts between any two compatible document formats through a shared content/layout pivot. docx, pptx, odt, odp, ods, odg, xlsx, csv (TSV is the same format with a tab delimiter), svg, and markdown all read into and build from the same `ContentDocument`/`LayoutDocument` model, with PDF as the one format every variant can reach. A composition engine (`convertDocument`) routes 111 (source, target) pairs across the ten content formats and PDF, including twenty PDF-pivot round trips (the eight layout-engine formats, plus xlsx and csv composing through ods), twenty-four cross-format bridge functions (same-variant direct copies, cross-variant semantic transforms, and PDF-composed), plus special-case conversions for `.odm` master documents, `.odb` database front-ends (HSQLDB and Firebird, four storage tiers), standalone `.odf` formula documents, and a bounded SQL/rpt-formula engine for `.odb` reports. Also includes: read-and-write live-view editors for all six editable formats, docx comment/footnote/header-footer exposure via `readDocxExtras`, real font resolution (source-embedded faces ahead of caller-supplied, vendored substitutes, and the standard 14), a hand-written MathML typesetting engine with embedded-font PDF rendering and a matching MathML ⇄ OMML translator, and a fully hand-written PDF codec. Built on [ooxml.js](https://github.com/ExaDev/ooxml.js), [odf.js](https://github.com/ExaDev/odf.js), [pdf-codec](https://github.com/ExaDev/pdf-codec), [markdown-codec](https://github.com/ExaDev/markdown-codec), and [document-schema.js](https://github.com/ExaDev/document-schema.js). +> Converts between any two compatible document formats through a shared content/layout pivot. docx, pptx, odt, odp, ods, odg, xlsx, csv (TSV is the same format with a tab delimiter), svg, and markdown all read into and build from the same `ContentDocument`/`LayoutDocument` model, with PDF as the one format every variant can reach. A composition engine (`convertDocument`) routes 111 (source, target) pairs across the ten content formats and PDF, including twenty PDF-pivot round trips (the eight layout-engine formats, plus xlsx and csv composing through ods), twenty-four cross-format bridge functions (same-variant direct copies, cross-variant semantic transforms, and PDF-composed), plus special-case conversions for `.odm` master documents, `.odb` database front-ends (HSQLDB and Firebird, four storage tiers), standalone `.odf` formula documents, and a bounded SQL/rpt-formula engine for `.odb` reports. Also includes: read-and-write live-view editors for all six editable formats, docx comment/footnote/header-footer exposure via `readDocxExtras`, real font resolution (source-embedded faces ahead of caller-supplied, vendored substitutes, and the standard 14), a hand-written MathML typesetting engine with embedded-font PDF rendering and a matching MathML ⇄ OMML translator, LaTeX lowering into the schema's two-layer semantic math core (pinned temml parser, symbol tables from prose, a coherence lint), and a fully hand-written PDF codec. Built on [ooxml.js](https://github.com/ExaDev/ooxml.js), [odf.js](https://github.com/ExaDev/odf.js), [pdf-codec](https://github.com/ExaDev/pdf-codec), [markdown-codec](https://github.com/ExaDev/markdown-codec), and [document-schema.js](https://github.com/ExaDev/document-schema.js). -`documents.js` extends `ooxml.js` in two directions `ooxml.js` deliberately does not cover: full PDF support (parsing and generating, via `pdf-codec`), and a read-**and-write** manipulation API for docx/pptx content — `ooxml.js`'s own typed readers are one-way. The PDF codec is hand-written against ISO 32000-1, with no external PDF library as a dependency — see [Fidelity](#fidelity) and pdf-codec's own README for the honest trade-off (not as robust against adversarial PDFs as a 15+-year-hardened library; fully auditable and dependency-free instead). `src/mathml/` (the MathML typesetting engine) stays in this package and is hand-written too, for the same supply-chain reason. +`documents.js` extends `ooxml.js` in two directions `ooxml.js` deliberately does not cover: full PDF support (parsing and generating, via `pdf-codec`), and a read-**and-write** manipulation API for docx/pptx content — `ooxml.js`'s own typed readers are one-way. The PDF codec is hand-written against ISO 32000-1, with no external PDF library as a dependency — see [Fidelity](#fidelity) and pdf-codec's own README for the honest trade-off (not as robust against adversarial PDFs as a 15+-year-hardened library; fully auditable and dependency-free instead). `src/mathml/` (the MathML typesetting engine) stays in this package and is hand-written too, for the same supply-chain reason. The one deliberate exception on the math side is the LaTeX parser: `src/latex/` lowers LaTeX into the schema's semantic core over a pinned exact-version [temml](https://temml.org) dependency — see [LaTeX lowering into the semantic core](#latex-lowering-into-the-semantic-core) for why a LaTeX grammar is the one component not worth hand-writing and what the pin guarantees. ```mermaid graph TD @@ -463,6 +463,30 @@ const editor = openDocx(existingDocxBytes); const { diagnostics: ommlDiagnostics } = editor.body.appendParagraph().appendOfficeMath(mathml); ``` +### LaTeX lowering into the semantic core + +A formula in the 3.2.0 schema carries two co-equal layers: `presentation` (a verbatim LaTeX string, rendering-authoritative) and `content` (a `MathExpression` semantic tree, computation-authoritative). Neither is stored derived from the other. This package owns the string-to-tree half — the lowering — and runs it wherever LaTeX enters the model: + +- **Parsing** happens at the format edge through [temml](https://temml.org) (MIT, zero dependencies), pinned to the **exact version recorded in `package.json`** — `"temml": "0.13.4"`, no caret. The pin is load-bearing: the lowering consumes temml's internal parse-node API, which carries no stability guarantee across releases, and the two-layer contract says a stored presentation string has one defined parse. Bumping the pin is a deliberate act that must re-run `src/latex/lower.test.ts`, whose table cases pin the parse-node shapes the lowering consumes. temml is the one math component this ecosystem deliberately does not hand-write (a LaTeX grammar is a large surface with none of the supply-chain payoff the hand-written MathML engine has); it is pure JavaScript, its parser never touches the DOM, and the workerd suite proves the whole lowering path in a Cloudflare Workers isolate. +- **Lowering** is mechanical exactly where notation is unambiguous: `\frac` → `math:divide`, radicals → `math:sqrt` / an exact `1/n` exponent, a scripted Sigma or Product with limits → a `sum`/`prod` binder owning the rest of its term, numeric literals → exact rationals (`3.14` → `157/50`, BigInt-exact at any length), subscripts → distinct symbol identities through the symbol table (`x_1` is never `x` times `1`), superscripts → `math:pow` unless the table already curates the scripted form as one symbol. Named functions (`\sin`, `\ln`, ...) consume their argument the way binders consume their summand. +- **Everything context-starved degrades to visible data**: juxtaposition (`mc^2`, `f(x)`, `2(x+1)` — multiplication and function application are both defensible readings, and LaTeX cannot say which), overloaded operators (`\pm`, `\approx`), integrals (the grammar's binders are exactly sum and prod), `\text` prose, compound subscripts (`a_{i+1}`), binomials, `align`/`cases` environments — each becomes an `unparsed` node carrying the verbatim source span plus a named diagnostic from `LATEX_DIAGNOSTIC_CODES`. Never a parse failure, never a silent guess; a degraded juxtaposition is exactly what the round-trip-safe semantic editing the schema defines is for. +- **Symbol tables** come from the document's own prose: sentence-level "where R is…" / "let x be…" definitions seed curated entries (conservatively — precision over recall, no quantity kind is ever guessed), and glyphs nobody defined are minted so every `sym` reference resolves. The markdown read pass builds the table automatically. +- **The markdown read path runs the whole pipeline**: markdown-codec hands `$$` display blocks and `\( \)` inline spans through as raw LaTeX text, and `readMarkdownContent` lowers them into embedded formula blocks (position, content, presentation MathML from the same parse — so `markdownToPdf` typesets real math through the STIX engine, `markdownToDocx` writes real OMML, and `markdownToOdt` writes real embedded formula sub-documents). The write side reconstructs the same markdown math syntax from the verbatim presentation layer. The pass's diagnostics surface through `readMarkdownContent`'s third parameter. +- **The coherence lint** (`lintMathCoherence`) re-parses and re-lowers every stored presentation string against the document's own symbol table and compares with the stored content layer — divergence means somebody edited one layer deliberately, so it reports a **warning carrying provenance** and re-derives nothing. + +```ts +import { latexToFormula, lintMathCoherence, lowerLatex } from 'documents.js'; + +const { expression, diagnostics, mintedSymbols } = lowerLatex('\\sum_{i=1}^{n} \\frac{1}{i^2}'); +// expression: { kind: 'sum', binder: 'i', lower: {kind:'num',numerator:'1',denominator:'1'}, ... } +// diagnostics: [] — fully mechanical; '2x' would degrade to unparsed + 'latex/juxtaposition-unparsed' + +const { formula } = latexToFormula('x^2', { symbolEntries: table.symbols, source: 'my:pipeline' }); +// formula: { mathml, presentation: { latex: 'x^2' }, content, provenance } — ready to embed + +const warnings = lintMathCoherence(pkg); // [{ code: 'math/coherence-divergence', severity: 'warning', provenance, detail }] +``` + ## Fonts Every X → PDF conversion resolves each typeface through a real `FontRegistry`, in this order: @@ -509,7 +533,8 @@ The package is layered from generic primitives outward to the two conversion dir - **`src/omml/`** — the MathML ⇄ OMML structural translator, both directions. `write.ts` covers the identical construct set `src/mathml/layout.ts` typesets; `read.ts` covers strictly more (reads what Word authored, not just what this package writes). Lives outside `src/mathml/` because its I/O type is `ooxml.js`'s `XmlElement` and `src/mathml/` imports no package. - **`src/ooxml/`** — thin adapters over `ooxml.js`'s own `readDocx`/`readPptx`, wrapping results into `ContentDocument`. `docx/formula.ts` is the one local reading pass (splicing OOXML math equations). `docx/extras.ts`'s `readDocxExtras` returns comments/footnotes/headers/footers/numbering. - **`src/odf/`** — ODF-side counterparts: `readOdtContent`/`readOdpContent`/`readOdsContent`/`readOdgContent` are thin adapters over `odf.js`. `formula/read.ts`/`formula/detect.ts` handle embedded formula detection (genuinely new work with no `odf.js`-side equivalent). -- **`src/markdown/`** — third adapter family, via `markdown-codec`. `readMarkdownContent` passes `readMarkdown`'s result straight through (it already produces a full `ContentDocument`). `buildMarkdownText` wraps `writeMarkdown`. `text.ts` is the byte↔text boundary. `MarkdownEditor` holds a mutable in-memory `ContentDocument`. +- **`src/latex/`** — the LaTeX presentation → `MathExpression` lowering: `temml.ts` is the pinned-parser boundary (exact-version temml, its internal parse API guarded behind structural type guards), `lower.ts` the mechanical rules and their degradations, `symbols.ts` the glyph/command map and the prose definition scanner, `rational.ts` the exact-rational helpers, `lint.ts` the coherence lint. See [LaTeX lowering into the semantic core](#latex-lowering-into-the-semantic-core). +- **`src/markdown/`** — third adapter family, via `markdown-codec`. `readMarkdownContent` passes `readMarkdown`'s result through the math-lowering pass (`math.ts` — markdown-codec's preserved `$$` display blocks and `\( \)` inline spans become two-layer formula blocks, with the document's symbol table seeded from its own prose). `buildMarkdownText` wraps `writeMarkdown`, reconstructing markdown math syntax from formula blocks carrying a presentation layer. `text.ts` is the byte↔text boundary. `MarkdownEditor` holds a mutable in-memory `ContentDocument`. - **`src/csv/`** — fourth adapter family, sharing the spreadsheet variant with xlsx/ods. `records.ts` is the RFC 4180 record parser/writer (one shared `quoteCsvField`, also used by the `.odb` CSV exporter); `text.ts` is the byte↔text boundary, rejecting malformed UTF-8; `read.ts` turns records into a spreadsheet `ContentDocument` (first record as verbatim string header, data cells through the same cell-typing heuristic `pdfToOds` uses); `write.ts` turns one sheet of a spreadsheet `ContentDocument` back into records via each cell's `displayText`. TSV is the same format with `{ delimiter: '\t' }` on either side. - **`src/svg/`** — fifth adapter family, sharing the drawing variant with odg. `text.ts` is the byte↔text boundary, rejecting malformed UTF-8; `read.ts` maps the six SVG shape primitives (rect/circle/ellipse/line/polyline/polygon/path) onto a one-page drawing `ContentDocument`, with transform lists composed as 2×3 affines and CSS lengths and the viewBox map resolved into page points; `write.ts` writes the six primitives back out, one shape element each; `path.ts` is the full SVG path-data grammar (M/L/H/V/C/S/Z plus Q/T/A and the relative forms — S/Q/T convert exactly, A is the one bounded approximation at ≤90° per cubic); `transform.ts` parses and composes the transform attribute and classifies the result by frame representability; `units.ts` resolves CSS length units and the viewBox; `paint.ts` resolves fill/stroke presentation attributes and dash styles; `diagnostics.ts` is the shared scope-limit vocabulary. - **`src/layout/`** — the pure conversion algorithms: `engine.ts` (wordprocessing → layout: flow, line-breaking, pagination), `slides.ts` (presentation → layout: direct placement), `sheets.ts` (spreadsheet → layout: grid, print settings, the first algorithm accepting `AbortSignal`), `drawing.ts` (drawing → layout: vector primitives + shape reuse), `reconstruct.ts` (layout → content: baseline clustering for wordprocessing/presentation, near-1:1 mapping for drawing, gridline-lattice-or-text-clustering for spreadsheet). @@ -620,11 +645,11 @@ To run a single test file: `pnpm vitest run src/path/to/file.test.ts`. - **`convertSpreadsheetToLayout` returns `{ document, formulas }`** — formula CID-font glyph runs can't travel through `LayoutDocument.pages[].items`. - **`formulaSizePtForFrame` is one shared two-pass fit** — lay out once at reference size, rescale to fit both frame width and height, floored at 8pt. docx OMML (no geometry) uses height alone. - **Embedded-formula detection in odt/odp is genuinely new work** — `collectFormulaFrames`/`collectSlideFormulaFrames` mirror `odf.js`'s own walks. ods needs no detection pass (`odf.js` 2.2.0 classifies formula sub-documents directly). -- **A formula that cannot typeset degrades to its plain-text stand-in, never to nothing.** `buildDocxPackage` writes real OMML; `buildOdtPackage` writes real embedded formula sub-documents. The markdown writer is the only stand-in-only path. `odmToPdf` carries formulas through as ordinary blocks. +- **A formula that cannot typeset degrades to its plain-text stand-in, never to nothing.** `buildDocxPackage` writes real OMML; `buildOdtPackage` writes real embedded formula sub-documents. The markdown writer reconstructs real `$$`/`\( \)` math for formulas carrying a presentation layer and falls back to the plain-text stand-in (StarMath, the verbatim presentation LaTeX, else `[formula]`) only for formulas with no LaTeX at all. `odmToPdf` carries formulas through as ordinary blocks. - **OMML read/write are deliberately asymmetric** — the reader covers more (`m:d`, `m:nary`, `m:acc`, `m:bar`, `m:func`, `m:sPre`) because it must read what Word wrote. `docx → odt → docx` round trips keep the mathematics but may change the OMML construct. - **The OMML translator covers exactly what `src/mathml/layout.ts` typesets.** A stretchy fence diverges: PDF stretches it, docx writes it at base size. `munderover` becomes nested `m:limUpp`/`m:limLow` (no operand scope in MathML). - **`sourcePath` traces a `LayoutItem` to its `ContentDocument` origin, but only within one read+layout pass** — not an edit-tracking mechanism. Since the frames fusion it survives as traceability only: the authoritative node↔position association is each content node's own `frames`, stamped at the moment of layout (or of reconstruction) rather than re-matched by string afterwards. -- **`readMarkdownContent` passes `readMarkdown`'s result straight through** — `markdown-codec` already produces a full `ContentDocument`. +- **`readMarkdownContent` runs markdown-codec's result through the math-lowering pass** — `markdown-codec` already produces a full `ContentDocument`, but it deliberately carries `$$` display blocks and `\( \)` inline spans through as raw LaTeX text (styled paragraphs and marker runs); the pass lowers that LaTeX into two-layer formula blocks so markdown math typesets, edits, and computes like math from any other format (see [LaTeX lowering into the semantic core](#latex-lowering-into-the-semantic-core)). - **Every markdown construct-mapping gap is a documented `MarkdownDiagnosticCodes` entry** (`md/invented-page-geometry`, `md/nested-emphasis-flattened`, `md/link-title-dropped`, `md/code-block-info-string-dropped`, `md/blockquote-nested-depth`, `md/list-item-block-unlisted`, `md/list-item-multi-block-flattened`, `md/image-unresolved`, `md/raw-html-preserved-as-text`/`md/raw-html-dropped`, `md/front-matter-key-unmapped`, `md/heading-level-clamped`, `md/adjacent-links-merged`, `md/code-span-as-monospace-run`, `md/paragraph-indent-dropped`, `md/list-numid-fallback`, `md/table-cell-formatting-dropped`, `md/table-cell-multi-paragraph-joined`) — never a silent approximation. - **`buildMarkdownText` throws for non-`'wordprocessing'` `ContentDocument`.** - **`decodeMarkdownText` throws on malformed UTF-8** rather than producing U+FFFD. diff --git a/package.json b/package.json index 98c5a490..30239695 100644 --- a/package.json +++ b/package.json @@ -90,6 +90,7 @@ "odf.js": "^3.0.1", "ooxml.js": "^2.16.0", "pdf-codec": "^2.2.35", + "temml": "0.13.4", "zod": "^4.4.3" }, "devDependencies": { diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 3636bc9f..055631a1 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -33,6 +33,9 @@ importers: pdf-codec: specifier: ^2.2.35 version: 2.2.35 + temml: + specifier: 0.13.4 + version: 0.13.4 zod: specifier: ^4.4.3 version: 4.4.3 @@ -2832,6 +2835,10 @@ packages: resolution: {integrity: sha512-yEFYrVhod+hdNyx7g5Bnkkb0G6si8HJurOoOEgC8B/O0uXLHlaey/65KRv6cuWBNhBgHKAROVpc7QyYqE5gFng==} engines: {node: '>=20'} + temml@0.13.4: + resolution: {integrity: sha512-k1yolMBswx34Jw9hZn5Xh2/GBlwqlW+wiN7QaUYUMhSa1ypfti6rlCfSmCxWyooxH1G32C/Z+qVvgAsBOELZBQ==} + engines: {node: '>=18.13.0'} + temp-dir@3.0.0: resolution: {integrity: sha512-nHc6S/bwIilKHNRgK/3jlhDoIHcp45YgyiwcAk46Tr0LfEqGBVpmiAyuiuxeVE44m3mXnEeVhaipLOEWmH+Njw==} engines: {node: '>=14.16'} @@ -5761,6 +5768,8 @@ snapshots: tagged-tag@1.0.0: {} + temml@0.13.4: {} + temp-dir@3.0.0: {} tempy@3.2.0: diff --git a/src/index.ts b/src/index.ts index a0f9d8b3..2db3ecb6 100644 --- a/src/index.ts +++ b/src/index.ts @@ -316,6 +316,17 @@ export { buildOfficeMath, buildOfficeMathParagraph } from './omml/write'; export type { OmmlReadResult } from './omml/read'; export { collectOfficeMathElements, readOfficeMath } from './omml/read'; +// --- LaTeX presentation -> the MathExpression semantic core (src/latex/): the string-to-tree half of document-schema.js's two-layer math model, over a pinned temml parser. lowerLatex is the direct entry point for a caller holding a LaTeX string; latexToFormula wraps it into a whole ContentFormula (verbatim presentation + presentation-MathML + lowered content + provenance) ready to embed the way every other formula in this package travels. The lowering is mechanical where notation is unambiguous (\frac -> math:divide, radicals -> math:sqrt/n-th root, scripted Sigma/Product -> sum/prod binders, numeric literals -> exact rationals, subscripts -> distinct symbol identities) and degrades everything context-starved to visible `unparsed` nodes carrying the verbatim source plus a named diagnostic -- never a parse failure, never a silent guess. The markdown read path runs this lowering automatically over markdown-codec's preserved $$ display blocks and \( \) inline spans (readMarkdownContent's third parameter surfaces the diagnostics). lintMathCoherence is the model's read-only audit: it re-parses and re-lowers every stored presentation string and reports divergence from the stored content layer as a warning carrying provenance -- a deliberate layer edit, never something this package re-derives. --- +export type { LatexDiagnostic, LatexDiagnosticCode, LatexDiagnosticSink } from './latex/diagnostics'; +export { LATEX_DIAGNOSTIC_CODES, MATH_LINT_CODES } from './latex/diagnostics'; +export type { MathLintDiagnostic, MathLintCode } from './latex/diagnostics'; +export type { LatexFormulaOptions, LatexFormulaResult, LatexLoweringResult, LowerLatexOptions } from './latex/lower'; +export { latexToFormula, lowerLatex } from './latex/lower'; +export { lintMathCoherence } from './latex/lint'; +export type { MarkdownMathLoweringOptions } from './markdown/math'; +export { lowerMarkdownMath } from './markdown/math'; + + // --- Format <-> ContentDocument readers and layout algorithms, each independently usable rather than only reachable through the ergonomic conversions below. --- export type { ReadDocxContentOptions } from './ooxml/docx/read'; export { readDocxContent } from './ooxml/docx/read'; diff --git a/src/latex/diagnostics.ts b/src/latex/diagnostics.ts new file mode 100644 index 00000000..b5eeea10 --- /dev/null +++ b/src/latex/diagnostics.ts @@ -0,0 +1,58 @@ +// The LaTeX lowering's degrade-with-diagnostic channel, following the same three-tier failure policy pdf-codec established and src/svg/diagnostics.ts restates for the svg read: input temml cannot parse at all degrades the whole expression to one `unparsed` node, notation the grammar covers degrades construct-by-construct, and nothing is ever a silent guess -- every degradation leaves the verbatim source visible inside the expression tree (the schema's `unparsed` variant) and names itself here. Every code below names one deliberate scope limit of the lowering rules in src/latex/lower.ts; a formula whose every construct is mechanical lowers with zero diagnostics. + +export const LATEX_DIAGNOSTIC_CODES = [ + // temml's parser rejected the string outright (an unknown command, an unmatched brace): the whole expression becomes one `unparsed` node carrying the full verbatim source. + 'latex/parse-error', + // A construct the lowering grammar has no rule for (an integral, an accent, an overline): that construct becomes an `unparsed` node carrying its verbatim source span. + 'latex/construct-unparsed', + // Two operands sit adjacent with no operator between them (juxtaposition -- `mc^2`, `f(x)`, `2(x+1)`): multiplication and function application are the two conventional readings and LaTeX notation cannot say which, so the whole run becomes one `unparsed` node rather than guessing. + 'latex/juxtaposition-unparsed', + // A \text{...} node: prose inside mathematics has no MathExpression reading, so its verbatim source becomes an `unparsed` node. + 'latex/text-unparsed', + // A subscript that is not a simple symbol suffix (`a_{i+1}`, `x_{(n)}`): the grammar has no indexed-access operator, so the whole scripted construct becomes an `unparsed` node. + 'latex/subscript-unparsed', + // A \sum/\prod whose bound is written as a bare glyph or a non-relation (`\sum_i`, `\sum_{i \in S}`) rather than `name = expression`: the binder still lowers, with the missing bound itself an `unparsed` node. + 'latex/binder-bound-implicit', + // A \sum/\prod whose subscript could not be read as a bound at all: the whole binder becomes an `unparsed` node. + 'latex/binder-bound-unreadable', + // A binary/relation operator with no mapping in the core registry (\pm, \approx, \cup, \to): the sequence around it becomes one `unparsed` node rather than dropping or guessing the operator. + 'latex/operator-unmapped', + // A subscript or superscript whose base is itself an `unparsed` construct: scripts attach to nothing lowerable, so the whole scripted span degrades with it. + 'latex/script-base-unparsed', + // An array environment other than the plain matrix family (align, cases, aligned): layout-semantic environments have no MathExpression reading, so the whole environment becomes one `unparsed` node. + 'latex/array-environment-unparsed', + // A binomial or other generalised fraction drawn with delimiters or without a bar: only the plain stacked fraction is unambiguously division. + 'latex/genfrac-unparsed', + // A binary/relation operator this lowering cannot place: a leading operator other than the one unary minus reading, or an operator with no operand on one side (`a + + b`, a trailing `+`). + 'latex/operator-placement-unparsed', + // The prose scanner found and seeded a symbol-table definition -- an informational audit channel, not a degradation: one diagnostic per definition found, so a caller can see exactly what the scanner inferred from the document's own sentences. + 'symbols/prose-definition-found', +] as const; + +export type LatexDiagnosticCode = (typeof LATEX_DIAGNOSTIC_CODES)[number]; + +export interface LatexDiagnostic { + readonly code: LatexDiagnosticCode; + // The verbatim source construct the diagnostic is about -- the same string the corresponding `unparsed` node carries, so a diagnostic and the visible gap in the tree always agree on what degraded. + readonly detail?: string; +} + +export type LatexDiagnosticSink = (diagnostic: LatexDiagnostic) => void; + +// The coherence lint's own vocabulary, separate from the lowering's because it reports a different phenomenon: not "this construct would not lower" but "this formula's two stored layers no longer agree", which means somebody edited one layer deliberately since the content was last derived. +export const MATH_LINT_CODES = [ + // Re-lowering the stored presentation string produced a different expression tree than the stored content layer: a warning carrying provenance, never an automatic re-derivation -- the schema's atomic pair-edit rule says the edit was deliberate and the stored layers stay exactly as stored. + 'math/coherence-divergence', + // A stored presentation string no longer parses at all while the stored content layer holds a lowered (non-unparsed-root) tree: a stronger form of the divergence above, since the presentation layer's own text has become unreadable to the pinned parser. + 'math/coherence-unparseable-presentation', +] as const; + +export type MathLintCode = (typeof MATH_LINT_CODES)[number]; + +export interface MathLintDiagnostic { + readonly code: MathLintCode; + readonly severity: 'warning'; + // The formula's stored provenance (its source and edit trail), carried into the warning so the reader can see who last touched either layer before judging which side is stale. + readonly provenance?: string; + readonly detail?: string; +} diff --git a/src/latex/lint.test.ts b/src/latex/lint.test.ts new file mode 100644 index 00000000..cd3da045 --- /dev/null +++ b/src/latex/lint.test.ts @@ -0,0 +1,97 @@ +import type { ContentBlock, DocumentPackage } from 'document-schema.js'; +import { CONTENT_FORMAT_VERSION, DOCUMENT_PACKAGE_FORMAT_VERSION } from 'document-schema.js'; +import { describe, expect, it } from 'vitest'; +import { latexToFormula } from './lower'; +import { lintMathCoherence } from './lint'; +import { buildFormulaBlock } from '../model/formula'; + +// The coherence lint's contract: re-parse, re-lower, compare -- and report divergence as a warning carrying the stored provenance, never as an automatic re-derivation (the schema's atomic pair-edit rule: the layers stay exactly as stored). These tests also pin that the lint WRITES nothing: every assertion re-reads the same package object after linting. + +function packageOf(blocks: readonly ContentBlock[]): DocumentPackage { + return { + formatVersion: DOCUMENT_PACKAGE_FORMAT_VERSION, + content: { + kind: 'wordprocessing', + formatVersion: CONTENT_FORMAT_VERSION, + metadata: {}, + sections: [{ pageSize: { widthPt: 595, heightPt: 842 }, margins: { topPt: 20, rightPt: 20, bottomPt: 20, leftPt: 20 }, blocks: [...blocks] }], + }, + }; +} + +function mathBlockOf(latex: string): ContentBlock { + return buildFormulaBlock(latexToFormula(latex, { source: 'test:lint' }).formula, { xPt: 0, yPt: 0, widthPt: 0, heightPt: 22 }, 'test:lint'); +} + +describe('lintMathCoherence', () => { + it('a package whose stored content is exactly the mechanical re-lowering of its presentation stays silent', () => { + const pkg = packageOf([mathBlockOf('\\sum_{i=1}^{n} \\frac{1}{i^2}')]); + expect(lintMathCoherence(pkg)).toEqual([]); + }); + + it('a deliberately edited content layer diverges: a warning carrying provenance, and the stored layers are untouched', () => { + const block = mathBlockOf('E = mc^2'); + const pkg = packageOf([block]); + // Someone resolved the mc^2 juxtaposition by hand into an explicit multiplication -- a better reading, stored deliberately next to the unchanged presentation. + if (block.kind !== 'embeddedObject' || block.document.kind !== 'formula') { + throw new Error('expected a formula block'); + } + block.document.formula.content = { + kind: 'app', + operator: 'math:eq', + args: [ + { kind: 'sym', id: 'symbols:E' }, + { kind: 'app', operator: 'math:multiply', args: [{ kind: 'sym', id: 'symbols:m' }, { kind: 'app', operator: 'math:pow', args: [{ kind: 'sym', id: 'symbols:c' }, { kind: 'num', numerator: '2', denominator: '1' }] }] }, + ], + }; + block.document.formula.provenance = { source: 'test:lint', editTrail: ['human edit: resolved the mc^2 juxtaposition into an explicit multiply'] }; + const warnings = lintMathCoherence(pkg); + expect(warnings).toHaveLength(1); + expect(warnings[0]?.code).toBe('math/coherence-divergence'); + expect(warnings[0]?.severity).toBe('warning'); + expect(warnings[0]?.provenance).toBe('test:lint -> human edit: resolved the mc^2 juxtaposition into an explicit multiply'); + expect(warnings[0]?.detail).toContain('E = mc^2'); + // The lint re-derived nothing: the stored content is still the hand-edited tree, byte for byte. + expect(block.document.formula.content).toEqual({ + kind: 'app', + operator: 'math:eq', + args: [ + { kind: 'sym', id: 'symbols:E' }, + { kind: 'app', operator: 'math:multiply', args: [{ kind: 'sym', id: 'symbols:m' }, { kind: 'app', operator: 'math:pow', args: [{ kind: 'sym', id: 'symbols:c' }, { kind: 'num', numerator: '2', denominator: '1' }] }] }, + ], + }); + }); + + it('a stored non-reduced rational still agrees with the reduced re-lowering -- value canonicalisation, not string equality', () => { + // '\frac{0.5}{2}' re-lowers the decimal to the reduced 1/2; the stored content spells the same value as an unreduced 2/4. Same expression, different spelling -- the lint compares cross-reduced values and stays silent. + const block = mathBlockOf('\\frac{0.5}{2}'); + if (block.kind !== 'embeddedObject' || block.document.kind !== 'formula') { + throw new Error('expected a formula block'); + } + block.document.formula.content = { kind: 'app', operator: 'math:divide', args: [{ kind: 'num', numerator: '2', denominator: '4' }, { kind: 'num', numerator: '2', denominator: '1' }] }; + expect(lintMathCoherence(packageOf([block]))).toEqual([]); + }); + + it('an unparseable stored presentation warns only when the stored content is a real lowering', () => { + const degraded = mathBlockOf('\\notacommand'); + const pkgDegraded = packageOf([degraded]); + // Stored content is itself an unparsed root (the lowering degraded too), so this stays silent. + expect(lintMathCoherence(pkgDegraded)).toEqual([]); + const edited = mathBlockOf('\\notacommand'); + if (edited.kind !== 'embeddedObject' || edited.document.kind !== 'formula') { + throw new Error('expected a formula block'); + } + edited.document.formula.content = { kind: 'sym', id: 'symbols:x' }; + const warnings = lintMathCoherence(packageOf([edited])); + expect(warnings.map((warning) => warning.code)).toEqual(['math/coherence-unparseable-presentation']); + }); + + it('walks formula blocks inside table cells and skips formulas carrying only one layer', () => { + const presentationOnly = buildFormulaBlock({ mathml: [], presentation: { latex: 'x^2' } }, { xPt: 0, yPt: 0, widthPt: 0, heightPt: 22 }, 'test'); + const pkg = packageOf([ + { kind: 'table', columnWidthsPt: [100], rows: [{ cells: [{ blocks: [mathBlockOf('a + b')] }] }] }, + presentationOnly, + ]); + expect(lintMathCoherence(pkg)).toEqual([]); + }); +}); diff --git a/src/latex/lint.ts b/src/latex/lint.ts new file mode 100644 index 00000000..a7200f96 --- /dev/null +++ b/src/latex/lint.ts @@ -0,0 +1,113 @@ +import type { ContentBlock, ContentFormula, DocumentPackage, MathExpression, MathSymbolEntry } from 'document-schema.js'; +import type { MathLintDiagnostic } from './diagnostics'; +import { lowerLatex } from './lower'; +import { reduceRational } from './rational'; + +// The coherence lint: the two-layer model's read-only audit. For every formula carrying BOTH a presentation string and a content tree, re-parse the stored presentation with the same pinned parser, re-run the same lowering against the document's own symbol table, and compare the result with the stored content. Divergence means somebody edited one layer deliberately since the content was last derived -- the schema's atomic pair-edit rule guarantees the layers never drift by accident -- so the finding is a WARNING carrying the stored provenance (where the formula came from and what has touched it, per the edit trail), never an automatic re-derivation: this function computes a derived comparison view at comparison time and writes nothing back, exactly the discipline the schema's own comment prescribes. +// +// What agreement MEANS here is deliberately strict: the re-lowered tree and the stored tree must be structurally identical after canonicalisation (key-order-normalised objects; exact rationals compared by cross-multiplication, the schema's own exact-equality rule for producers that skip lowest-terms reduction). A stored content tree someone hand-curated into a BETTER reading than the mechanical lowering (resolving an mc^2 juxtaposition into an explicit multiply, say) will diverge from the mechanical re-lowering -- correctly, and reported as a warning: the lint's job is to surface that a deliberate edit happened, not to judge whether the edit was an improvement. + +function isRecord(value: unknown): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} + +// Canonical comparison view of an expression: objects rebuilt with sorted keys (so a hand-edited JSON blob and a freshly lowered tree compare by structure, not by key order) and rationals reduced to lowest terms (so 1/2 and 2/4 agree, per the schema's exact-equality rule for non-reduced producers -- lowest terms is the canonical spelling both sides reduce to). Everything else compares structurally, recursively. +function canonical(value: unknown): unknown { + if (Array.isArray(value)) { + return value.map(canonical); + } + if (!isRecord(value)) { + return value; + } + if (value.kind === 'num' && typeof value.numerator === 'string' && typeof value.denominator === 'string') { + return { kind: 'num', ...reduceRational(BigInt(value.numerator), BigInt(value.denominator)) }; + } + const entries = Object.entries(value).map(([key, element]) => [key, canonical(element)] as const); + entries.sort(([left], [right]) => (left < right ? -1 : left > right ? 1 : 0)); + return Object.fromEntries(entries); +} + +function expressionsAgree(left: MathExpression, right: MathExpression): boolean { + return JSON.stringify(canonical(left)) === JSON.stringify(canonical(right)); +} + +// Whether a formula's content layer is itself a whole-expression degradation -- one root `unparsed` node. Such a stored tree agrees with any re-lowering that also degraded, and a stored degradation paired with an unparseable presentation is a quieter situation than a real lowering paired with an unreadable string, so only the latter warns. +function isUnparsedRoot(expression: MathExpression): boolean { + return expression.kind === 'unparsed'; +} + +function lintFormula(formula: ContentFormula, locate: string, symbolEntries: readonly MathSymbolEntry[] | undefined, warnings: MathLintDiagnostic[]): void { + const presentation = formula.presentation; + const content = formula.content; + if (presentation === undefined || content === undefined) { + return; + } + const provenance = formula.provenance === undefined ? undefined : [formula.provenance.source, ...formula.provenance.editTrail].join(' -> '); + const relowered = lowerLatex(presentation.latex, { symbolEntries }); + const detail = `${locate}: ${presentation.latex}`; + if (relowered.diagnostics.some((diagnostic) => diagnostic.code === 'latex/parse-error')) { + if (!isUnparsedRoot(content)) { + warnings.push({ code: 'math/coherence-unparseable-presentation', severity: 'warning', ...(provenance === undefined ? {} : { provenance }), detail }); + } + return; + } + if (!expressionsAgree(relowered.expression, content)) { + warnings.push({ code: 'math/coherence-divergence', severity: 'warning', ...(provenance === undefined ? {} : { provenance }), detail }); + } +} + +// Every formula the lint can see in a block flow, including inside table cells. +function collectBlockFormulas(blocks: readonly ContentBlock[], locate: string, formulas: { formula: ContentFormula; locate: string }[]): void { + for (const [index, block] of blocks.entries()) { + if (block.kind === 'embeddedObject' && block.objectKind === 'formula' && block.document.kind === 'formula') { + formulas.push({ formula: block.document.formula, locate: `${locate}/blocks[${String(index)}]` }); + continue; + } + if (block.kind === 'table') { + for (const [rowIndex, row] of block.rows.entries()) { + for (const [cellIndex, cell] of row.cells.entries()) { + collectBlockFormulas(cell.blocks, `${locate}/rows[${String(rowIndex)}].cells[${String(cellIndex)}]`, formulas); + } + } + } + } +} + +// Lint every formula carrying both layers in a package. The walk covers every arm formulas actually travel through: the wordprocessing sections' block flow, presentation slides and drawing pages (both via their shapes' own block flows), the spreadsheet arm's own embeddedObjects array, and the formula arm itself (a standalone formula document). +export function lintMathCoherence(pkg: DocumentPackage): readonly MathLintDiagnostic[] { + const warnings: MathLintDiagnostic[] = []; + const symbolEntries = pkg.content.symbolTable?.symbols; + const found: { formula: ContentFormula; locate: string }[] = []; + const content = pkg.content; + if (content.kind === 'formula') { + found.push({ formula: content.formula, locate: 'formula' }); + } else if (content.kind === 'wordprocessing') { + for (const [index, section] of content.sections.entries()) { + collectBlockFormulas(section.blocks, `sections[${String(index)}]`, found); + } + } else if (content.kind === 'presentation') { + for (const [slideIndex, slide] of content.slides.entries()) { + for (const [shapeIndex, shape] of slide.shapes.entries()) { + collectBlockFormulas(shape.blocks, `slides[${String(slideIndex)}].shapes[${String(shapeIndex)}]`, found); + } + } + } else if (content.kind === 'drawing') { + for (const [pageIndex, page] of content.pages.entries()) { + for (const [shapeIndex, shape] of page.shapes.entries()) { + collectBlockFormulas(shape.blocks, `pages[${String(pageIndex)}].shapes[${String(shapeIndex)}]`, found); + } + } + } else if (content.kind === 'spreadsheet') { + for (const [sheetIndex, sheet] of content.sheets.entries()) { + for (const [objectIndex, object] of (sheet.embeddedObjects ?? []).entries()) { + if (object.objectKind === 'formula' && object.document.kind === 'formula') { + found.push({ formula: object.document.formula, locate: `sheets[${String(sheetIndex)}].embeddedObjects[${String(objectIndex)}]` }); + } + } + } + } + for (const { formula, locate } of found) { + lintFormula(formula, locate, symbolEntries, warnings); + } + return warnings; +} diff --git a/src/latex/lower.test.ts b/src/latex/lower.test.ts new file mode 100644 index 00000000..dd28e764 --- /dev/null +++ b/src/latex/lower.test.ts @@ -0,0 +1,158 @@ +import type { MathExpression, MathSymbolEntry } from 'document-schema.js'; +import { describe, expect, it } from 'vitest'; +import { latexToFormula, lowerLatex } from './lower'; + +// The lowering table: every construct whose reading is mechanical, pinned as an expected MathExpression tree. These cases double as the parse-node-shape pin for the temml version recorded in package.json -- a temml bump that reshapes a node the lowering consumes shows up here first, which is exactly when the bump's re-verification is meant to happen (see src/latex/temml.ts's own top-of-file comment). +describe('lowerLatex mechanical rules', () => { + interface Case { readonly latex: string; readonly expected: MathExpression } + const cases: readonly Case[] = [ + { latex: 'x', expected: { kind: 'sym', id: 'symbols:x' } }, + { latex: '\\alpha', expected: { kind: 'sym', id: 'symbols:α' } }, + { latex: '\\infty', expected: { kind: 'sym', id: 'symbols:∞' } }, + { latex: '42', expected: { kind: 'num', numerator: '42', denominator: '1' } }, + { latex: '3.14', expected: { kind: 'num', numerator: '157', denominator: '50' } }, + { latex: '.5', expected: { kind: 'num', numerator: '1', denominator: '2' } }, + { latex: '\\frac{a}{b}', expected: { kind: 'app', operator: 'math:divide', args: [{ kind: 'sym', id: 'symbols:a' }, { kind: 'sym', id: 'symbols:b' }] } }, + { latex: 'a/b', expected: { kind: 'app', operator: 'math:divide', args: [{ kind: 'sym', id: 'symbols:a' }, { kind: 'sym', id: 'symbols:b' }] } }, + { latex: '\\sqrt{x}', expected: { kind: 'app', operator: 'math:sqrt', args: [{ kind: 'sym', id: 'symbols:x' }] } }, + { latex: '\\sqrt[3]{x}', expected: { kind: 'app', operator: 'math:pow', args: [{ kind: 'sym', id: 'symbols:x' }, { kind: 'app', operator: 'math:divide', args: [{ kind: 'num', numerator: '1', denominator: '1' }, { kind: 'num', numerator: '3', denominator: '1' }] }] } }, + { latex: 'x^2', expected: { kind: 'app', operator: 'math:pow', args: [{ kind: 'sym', id: 'symbols:x' }, { kind: 'num', numerator: '2', denominator: '1' }] } }, + { latex: 'x_1', expected: { kind: 'sym', id: 'symbols:x_1' } }, + { latex: 'x_i^2', expected: { kind: 'app', operator: 'math:pow', args: [{ kind: 'sym', id: 'symbols:x_i' }, { kind: 'num', numerator: '2', denominator: '1' }] } }, + { latex: 'a + b = c', expected: { kind: 'app', operator: 'math:eq', args: [{ kind: 'app', operator: 'math:add', args: [{ kind: 'sym', id: 'symbols:a' }, { kind: 'sym', id: 'symbols:b' }] }, { kind: 'sym', id: 'symbols:c' }] } }, + { latex: 'a - b - c', expected: { kind: 'app', operator: 'math:subtract', args: [{ kind: 'app', operator: 'math:subtract', args: [{ kind: 'sym', id: 'symbols:a' }, { kind: 'sym', id: 'symbols:b' }] }, { kind: 'sym', id: 'symbols:c' }] } }, + { latex: '-x + y', expected: { kind: 'app', operator: 'math:add', args: [{ kind: 'app', operator: 'math:negate', args: [{ kind: 'sym', id: 'symbols:x' }] }, { kind: 'sym', id: 'symbols:y' }] } }, + { latex: 'a \\leq b', expected: { kind: 'app', operator: 'math:leq', args: [{ kind: 'sym', id: 'symbols:a' }, { kind: 'sym', id: 'symbols:b' }] } }, + { latex: '\\sin(x)', expected: { kind: 'app', operator: 'math:sin', args: [{ kind: 'sym', id: 'symbols:x' }] } }, + { latex: '\\sin x + 1', expected: { kind: 'app', operator: 'math:add', args: [{ kind: 'app', operator: 'math:sin', args: [{ kind: 'sym', id: 'symbols:x' }] }, { kind: 'num', numerator: '1', denominator: '1' }] } }, + { latex: '\\left( a + b \\right)^2', expected: { kind: 'app', operator: 'math:pow', args: [{ kind: 'app', operator: 'math:add', args: [{ kind: 'sym', id: 'symbols:a' }, { kind: 'sym', id: 'symbols:b' }] }, { kind: 'num', numerator: '2', denominator: '1' }] } }, + { latex: '\\begin{pmatrix} a & b \\\\ c & d \\end{pmatrix}', expected: { kind: 'matrix', rows: [[{ kind: 'sym', id: 'symbols:a' }, { kind: 'sym', id: 'symbols:b' }], [{ kind: 'sym', id: 'symbols:c' }, { kind: 'sym', id: 'symbols:d' }]] } }, + { latex: '\\sum_{i=1}^{n} i^2', expected: { kind: 'sum', binder: 'i', lower: { kind: 'num', numerator: '1', denominator: '1' }, upper: { kind: 'sym', id: 'symbols:n' }, body: { kind: 'app', operator: 'math:pow', args: [{ kind: 'sym', id: 'i' }, { kind: 'num', numerator: '2', denominator: '1' }] } } }, + { latex: '\\sum_{i=1}^{\\infty} \\frac{1}{i^2}', expected: { kind: 'sum', binder: 'i', lower: { kind: 'num', numerator: '1', denominator: '1' }, upper: { kind: 'sym', id: 'symbols:∞' }, body: { kind: 'app', operator: 'math:divide', args: [{ kind: 'num', numerator: '1', denominator: '1' }, { kind: 'app', operator: 'math:pow', args: [{ kind: 'sym', id: 'i' }, { kind: 'num', numerator: '2', denominator: '1' }] }] } } }, + { latex: '\\prod_{k=0}^{n} k', expected: { kind: 'prod', binder: 'k', lower: { kind: 'num', numerator: '0', denominator: '1' }, upper: { kind: 'sym', id: 'symbols:n' }, body: { kind: 'sym', id: 'k' } } }, + { latex: '\\sum_i x_i', expected: { kind: 'sum', binder: 'i', lower: { kind: 'unparsed', latex: '' }, upper: { kind: 'unparsed', latex: '' }, body: { kind: 'sym', id: 'symbols:x_i' } } }, + ]; + for (const { latex, expected } of cases) { + it(`lowers ${latex} mechanically`, () => { + const result = lowerLatex(latex); + expect(result.diagnostics.filter((diagnostic) => diagnostic.code !== 'latex/binder-bound-implicit')).toEqual([]); + expect(result.expression).toEqual(expected); + }); + } + + it('sums nested in one term: the outer binder owns the inner binder and its summand', () => { + const result = lowerLatex('\\sum_{i=1}^{n} \\sum_{j=1}^{m} i j + 1'); + // The `i j` summand is juxtaposition and degrades inside the body -- the +1 still folds outside the binder, exactly the conventional precedence. + expect(result.expression).toEqual({ + kind: 'app', + operator: 'math:add', + args: [ + { kind: 'sum', binder: 'i', lower: { kind: 'num', numerator: '1', denominator: '1' }, upper: { kind: 'sym', id: 'symbols:n' }, body: { kind: 'sum', binder: 'j', lower: { kind: 'num', numerator: '1', denominator: '1' }, upper: { kind: 'sym', id: 'symbols:m' }, body: { kind: 'unparsed', latex: 'i j' } } }, + { kind: 'num', numerator: '1', denominator: '1' }, + ], + }); + }); + + it('binds the binder name lexically: the bound variable shadows the table inside the body only', () => { + const entries: readonly MathSymbolEntry[] = [{ glyph: 'i', scope: 'document', id: 'curated:imaginary-unit' }]; + const inside = lowerLatex('\\sum_{i=1}^{n} i', { symbolEntries: entries }); + expect(inside.expression).toEqual({ kind: 'sum', binder: 'i', lower: { kind: 'num', numerator: '1', denominator: '1' }, upper: { kind: 'sym', id: 'symbols:n' }, body: { kind: 'sym', id: 'i' } }); + const outside = lowerLatex('i', { symbolEntries: entries }); + expect(outside.expression).toEqual({ kind: 'sym', id: 'curated:imaginary-unit' }); + }); + + it('resolves a curated table entry by glyph instead of minting a duplicate', () => { + const entries: readonly MathSymbolEntry[] = [{ glyph: 'R', scope: 'document', id: 'curated:resistance', quantityKind: 'si:resistance' }]; + const result = lowerLatex('R^2', { symbolEntries: entries }); + expect(result.expression).toEqual({ kind: 'app', operator: 'math:pow', args: [{ kind: 'sym', id: 'curated:resistance' }, { kind: 'num', numerator: '2', denominator: '1' }] }); + expect(result.mintedSymbols).toEqual([]); + }); + + it('a curated scripted glyph is one symbol -- exponentiation stands down for the table\'s judgement', () => { + const entries: readonly MathSymbolEntry[] = [{ glyph: 'x^2', scope: 'document', id: 'curated:square-symbol' }]; + const result = lowerLatex('x^2', { symbolEntries: entries }); + expect(result.expression).toEqual({ kind: 'sym', id: 'curated:square-symbol' }); + }); + + it('mints table entries for every unresolved glyph so every emitted sym reference resolves', () => { + const result = lowerLatex('a + b'); + expect(result.mintedSymbols).toEqual([ + { glyph: 'a', scope: 'document', id: 'symbols:a' }, + { glyph: 'b', scope: 'document', id: 'symbols:b' }, + ]); + }); +}); + +// The degradation table: context-starved and out-of-scope constructs stay visible data -- an `unparsed` node carrying the verbatim source plus a named diagnostic -- never a throw, never a silent guess. +describe('lowerLatex degradations', () => { + interface Case { readonly latex: string; readonly code: string; readonly unparsedLatex?: string } + const cases: readonly Case[] = [ + // Juxtaposition: two defensible readings (multiplication, function application), no mechanical one. + { latex: '2x', code: 'latex/juxtaposition-unparsed', unparsedLatex: '2x' }, + { latex: 'f(x)', code: 'latex/juxtaposition-unparsed', unparsedLatex: 'f(x' }, + // Text prose inside math. + { latex: '\\text{where } R', code: 'latex/text-unparsed' }, + // A compound subscript has no indexed-access reading in the grammar. + { latex: 'x_{i+1}', code: 'latex/subscript-unparsed', unparsedLatex: 'x_{i+1}' }, + // Integrals: the grammar's binders are exactly sum and prod. + { latex: '\\int_0^1 f(x) \\, dx', code: 'latex/subscript-unparsed' }, + // An operator with no mapping in the core registry. + { latex: 'a \\pm b', code: 'latex/operator-unmapped', unparsedLatex: 'a \\pm b' }, + // A binomial is a generalised fraction with delimiters, not a division. + { latex: '\\binom{n}{k}', code: 'latex/genfrac-unparsed' }, + // Layout-semantic array environments. + { latex: '\\begin{cases} a & b \\end{cases}', code: 'latex/array-environment-unparsed' }, + // A string the pinned parser cannot read at all: the whole expression is one unparsed node. + { latex: '\\notacommand', code: 'latex/parse-error', unparsedLatex: '\\notacommand' }, + ]; + for (const { latex, code, unparsedLatex } of cases) { + it(`degrades ${latex} to unparsed with ${code}`, () => { + const result = lowerLatex(latex); + expect(result.diagnostics.some((diagnostic) => diagnostic.code === code)).toBe(true); + if (unparsedLatex !== undefined) { + expect(result.expression).toEqual({ kind: 'unparsed', latex: unparsedLatex }); + } + }); + } + + it('the unparsed node carries the verbatim source, not a re-serialisation', () => { + const result = lowerLatex('a \\pm b'); + expect(result.expression).toEqual({ kind: 'unparsed', latex: 'a \\pm b' }); + }); + + it('a juxtaposition inside a larger relation degrades only the juxtaposed run -- the relation itself still lowers around it', () => { + const result = lowerLatex('E = mc^2'); + expect(result.expression).toEqual({ kind: 'app', operator: 'math:eq', args: [{ kind: 'sym', id: 'symbols:E' }, { kind: 'unparsed', latex: 'mc^2' }] }); + }); + + it('an empty string lowers to an empty unparsed node with no diagnostics', () => { + const result = lowerLatex(''); + expect(result.expression).toEqual({ kind: 'unparsed', latex: '' }); + expect(result.diagnostics).toEqual([]); + }); + + it('streams diagnostics through the sink as they are emitted', () => { + const seen: string[] = []; + lowerLatex('2x', { sink: (diagnostic) => seen.push(diagnostic.code) }); + expect(seen).toEqual(['latex/juxtaposition-unparsed']); + }); +}); + +describe('latexToFormula', () => { + it('builds the two-layer ContentFormula: verbatim presentation, presentation-MathML, lowered content, provenance', () => { + const result = latexToFormula('\\frac{1}{2}', { source: 'test:probe' }); + expect(result.formula.presentation).toEqual({ latex: '\\frac{1}{2}' }); + expect(result.formula.content).toEqual({ kind: 'app', operator: 'math:divide', args: [{ kind: 'num', numerator: '1', denominator: '1' }, { kind: 'num', numerator: '2', denominator: '1' }] }); + expect(result.formula.provenance).toEqual({ source: 'test:probe', editTrail: [] }); + const root = result.formula.mathml[0]; + expect(root?.type).toBe('element'); + expect(root?.type === 'element' ? root.tag : undefined).toBe('math'); + }); + + it('a parse failure still carries the presentation verbatim with an empty MathML array -- the schema-anticipated state, never a throw', () => { + const result = latexToFormula('\\notacommand'); + expect(result.formula.presentation).toEqual({ latex: '\\notacommand' }); + expect(result.formula.mathml).toEqual([]); + expect(result.formula.content).toEqual({ kind: 'unparsed', latex: '\\notacommand' }); + }); +}); diff --git a/src/latex/lower.ts b/src/latex/lower.ts new file mode 100644 index 00000000..9f7be63d --- /dev/null +++ b/src/latex/lower.ts @@ -0,0 +1,675 @@ +import type { ContentFormula, MathExpression, MathMlNode, MathSymbolEntry } from 'document-schema.js'; +import type { LatexDiagnostic, LatexDiagnosticSink } from './diagnostics'; +import { isTemmlNode, parseLatex, type LatexParseResult, type TemmlNode } from './temml'; +import { glyphOfSymbolText, SymbolResolver } from './symbols'; +import { decimalToRational } from './rational'; + +// LaTeX presentation -> MathExpression, the string-to-tree half of the two-layer math model (document-schema.js src/math.ts states the contract: this direction is total -- any input at least degrades to an `unparsed` node -- while tree-to-string rendering is partial, which is why storage carries both layers verbatim). The rules are mechanical exactly where notation is unambiguous and degrade to visible `unparsed` data everywhere else, per the design the issue records: `\frac` is always division, a radical is always a root, a scripted Sigma with limits is always a binder, and juxtaposition -- the one construct with two defensible readings (multiplication, function application) -- is NEVER guessed, because a wrong guess is indistinguishable from a correct lowering until someone computes with it. +// +// The input tree is temml's KaTeX-style parse tree (src/latex/temml.ts, the pinned parser). Everything here reads it through structural guards, so a temml release that reshapes a node changes a type-guard failure in the test suite rather than silently mis-lowering. + +// -- The operator registries this lowering emits into -- +// +// The core arithmetic registry ('math:' prefix): every operator below is one the schema names the grammar's reference consumers implement. Binary operators fold strictly left-to-right, one application per source operator, so the stored tree mirrors the source's own structure (a - b - c is subtract(subtract(a, b), c), not a variadic rewrite -- associativity is a semantics-layer judgement, not this lowering's to make). + +const BINARY_ATOM_OPERATORS: Readonly> = { + '+': 'math:add', + '-': 'math:subtract', + '\\cdot': 'math:multiply', + '\\times': 'math:multiply', + '\\div': 'math:divide', +}; + +const RELATION_ATOM_OPERATORS: Readonly> = { + '=': 'math:eq', + '\\neq': 'math:neq', + '\\ne': 'math:neq', + '<': 'math:lt', + '\\leq': 'math:leq', + '\\le': 'math:leq', + '>': 'math:gt', + '\\geq': 'math:geq', + '\\ge': 'math:geq', +}; + +// The subtraction operator, named so the unary-minus reading below can reference the SOURCE operator it applies to (a leading token mapping to math:subtract) distinctly from the operator it EMITS (math:negate). +const SUBTRACT_OPERATOR = 'math:subtract'; +const UNARY_MINUS_OPERATOR = 'math:negate'; + +// Named single-argument functions. A function name is not reused as a variable in any convention these rules cover, so \sin applied to what follows is mechanical in a way bare f(x) is not -- which is exactly why f(x) degrades (juxtaposition) while \sin(x) lowers. Deliberately excludes variadic and ordering-sensitive names (\min, \max, \arg): their argument-list semantics have no representation in a single-argument registry entry, and half a variadic reading is a wrong reading. +const NAMED_FUNCTION_OPERATORS: Readonly> = { + '\\sin': 'math:sin', + '\\cos': 'math:cos', + '\\tan': 'math:tan', + '\\cot': 'math:cot', + '\\sec': 'math:sec', + '\\csc': 'math:csc', + '\\arcsin': 'math:arcsin', + '\\arccos': 'math:arccos', + '\\arctan': 'math:arctan', + '\\sinh': 'math:sinh', + '\\cosh': 'math:cosh', + '\\tanh': 'math:tanh', + '\\exp': 'math:exp', + '\\log': 'math:log', + '\\ln': 'math:ln', +}; + +// temml node types that carry no mathematics at all -- spacing commands, the zero-size `rule` artefact temml inserts as a radical's vinculum, kerns. Skipping them is not a degradation (there is nothing to degrade); everything else unknown degrades visibly instead. +const PRESENTATION_ONLY_TYPES: ReadonlySet = new Set(['spacing', 'rule', 'kern']); + +// Node types whose whole job is to wrap a body in presentation styling -- unwrapped, with the body lowered in place. Not in PRESENTATION_ONLY_TYPES because their body is mathematics. +const WRAPPER_TYPES: ReadonlySet = new Set(['styling', 'color']); + +function isRecord(value: unknown): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} + +// A temml node's `text` field when it holds a plain written character or a symbol command -- the things a glyph can be built from. +function nodeText(node: TemmlNode): string | undefined { + return typeof node.text === 'string' ? node.text : undefined; +} + +// The written glyph this node carries, for glyph-shaped nodes only (a mathord, or a textord holding a symbol command): the raw character for plain text, the mapped Unicode glyph for a command. +function glyphOfNode(node: TemmlNode): string | undefined { + const text = nodeText(node); + return text === undefined ? undefined : glyphOfSymbolText(text); +} + +// One lowerable item in a term: either a temml node, or a numeric literal folded out of a run of digit textords ('4' '2' -> 42; '3' '.' '1' '4' -> 3.14). temml emits each digit as its own textord, so the fold is what makes decimal notation visible to the lowering at all; the folded literal is validated and reduced through src/latex/rational.ts, keeping the exact-rational contract bit-exact for arbitrarily long literals. A folded item keeps the textords it came from so degradation spans stay verbatim-complete ('2x' degrades carrying the 2, not just the x). +type TermItem = { readonly kind: 'node'; readonly node: TemmlNode } | { readonly kind: 'number'; readonly literal: string; readonly nodes: readonly TemmlNode[] }; + +function numericText(text: string): boolean { + return text.length === 1 && text >= '0' && text <= '9'; +} + +// The per-run state every lowering level threads: the verbatim input (source spans slice into it), the glyph resolver, the lexically scoped binder names (a bound variable shadows the table inside its binder's body), and the diagnostics this run accumulated. +interface LoweringContext { + readonly input: string; + readonly resolver: SymbolResolver; + readonly binders: readonly string[]; + readonly diagnostics: LatexDiagnostic[]; + readonly sink?: LatexDiagnosticSink; +} + +function diagnose(context: LoweringContext, code: LatexDiagnostic['code'], detail?: string): void { + context.diagnostics.push({ code, ...(detail === undefined ? {} : { detail }) }); + context.sink?.({ code, ...(detail === undefined ? {} : { detail }) }); +} + +function unparsed(latex: string): MathExpression { + return { kind: 'unparsed', latex }; +} + +function app(operator: string, args: readonly MathExpression[]): MathExpression { + return { kind: 'app', operator, args: [...args] }; +} + +// The verbatim source substring a set of nodes came from -- what every degradation carries, per the schema's contract that a coverage gap stays visible as the exact source that resisted lowering. temml attaches positions to tokens and groups but not to the wrapper nodes built over them (supsub, genfrac, sqrt), so the walk descends: a wrapper without a position of its own is covered by the outermost span of its descendants. Nodes temml synthesised with no position and no positioned descendants (the radical-vinculum rule) contribute nothing; an all-synthetic set yields ''. +function spanOfNodes(context: LoweringContext, nodes: readonly TemmlNode[]): string { + let start: number | undefined; + let end: number | undefined; + const visit = (node: TemmlNode): void => { + const loc = node.loc; + if (isRecord(loc) && typeof loc.start === 'number' && typeof loc.end === 'number') { + if (start === undefined || loc.start < start) { + start = loc.start; + } + if (end === undefined || loc.end > end) { + end = loc.end; + } + } + for (const value of Object.values(node)) { + if (isTemmlNode(value)) { + visit(value); + } else if (Array.isArray(value)) { + for (const element of value) { + if (isTemmlNode(element)) { + visit(element); + } + } + } + } + }; + for (const node of nodes) { + visit(node); + } + return start === undefined || end === undefined ? '' : context.input.slice(start, end); +} + +// A sup/sub value as a node list: temml hands scripts as single nodes, with braces becoming an ordgroup -- unwrapped here so `^{n+1}` and `^n` reach the same lowering path. Styling wrappers unwrap the same way. +function nodesOfScript(script: unknown): readonly TemmlNode[] | undefined { + if (!isTemmlNode(script)) { + return undefined; + } + if (script.type === 'ordgroup' || WRAPPER_TYPES.has(script.type)) { + const body = script.body; + return Array.isArray(body) && body.every(isTemmlNode) ? body : undefined; + } + return [script]; +} + +// -- The lowering levels -- + +// A node list with binary/relation operators folded: partition at operator atoms, lower each run as a term, then fold left-to-right. This is where `a + b = c` becomes eq(add(a, b), c), and where unmapped operators (\pm, \approx, \to) and malformed operator placement (a trailing '+', an empty middle run) degrade the whole list to one `unparsed` node -- there is no mechanical reading of a sequence this function cannot fold. +function lowerNodeList(nodes: readonly TemmlNode[], context: LoweringContext): MathExpression { + const present = nodes.filter((node) => !PRESENTATION_ONLY_TYPES.has(node.type)); + const segments: TemmlNode[][] = []; + const operators: string[] = []; + let current: TemmlNode[] = []; + let unmappable: string | undefined; + for (const node of present) { + const operator = operatorOfNode(node); + if (operator !== undefined) { + segments.push(current); + operators.push(operator); + current = []; + continue; + } + if (isOperatorAtom(node)) { + unmappable = unmappable ?? (nodeText(node) ?? node.type); + continue; + } + current.push(node); + } + segments.push(current); + if (unmappable !== undefined) { + const detail = spanOfNodes(context, present) || unmappable; + diagnose(context, 'latex/operator-unmapped', detail); + return unparsed(detail); + } + const [firstSegment = [], ...restSegments] = segments; + if (operators.length === 0) { + return lowerTerm(firstSegment, context); + } + const detail = spanOfNodes(context, present); + if (firstSegment.length === 0) { + const leading = operators[0]; + const secondSegment = restSegments[0] ?? []; + if (leading !== SUBTRACT_OPERATOR || secondSegment.length === 0) { + diagnose(context, 'latex/operator-placement-unparsed', detail); + return unparsed(detail); + } + return fold(app(UNARY_MINUS_OPERATOR, [lowerTerm(secondSegment, context)]), operators.slice(1), restSegments.slice(1), context, detail); + } + return fold(lowerTerm(firstSegment, context), operators, restSegments, context, detail); +} + +// Whether this node is an atom in the bin/rel families carrying an operator glyph this registry does not map -- the unmapped ones (\pm between two operands) that must degrade the sequence rather than be dropped. +function isOperatorAtom(node: TemmlNode): boolean { + return node.type === 'atom' && (node.family === 'bin' || node.family === 'rel'); +} + +function operatorOfNode(node: TemmlNode): string | undefined { + // Operator mapping keys off the atom's own text, not its family: TeX relabels a binary operator's atom by position (a leading '-' arrives as family 'open', an operator before another operator as 'ord'), which is a RENDERING convention about spacing, not a statement that the glyph stopped being an operator -- '-' at the head of `-x + y` is still subtraction-shaped and still lowers through the unary-minus reading below. + if (node.type === 'atom') { + const text = nodeText(node); + if (text === undefined) { + return undefined; + } + return BINARY_ATOM_OPERATORS[text] ?? RELATION_ATOM_OPERATORS[text]; + } + // '/' is a textord, not an atom, but a/b is as mechanically division as \frac{a}{b} -- the same operator, reached by the inline spelling. + if (node.type === 'textord' && nodeText(node) === '/') { + return 'math:divide'; + } + return undefined; +} + +function fold(first: MathExpression, operators: readonly string[], segments: readonly TemmlNode[][], context: LoweringContext, detail: string): MathExpression { + let folded = first; + for (let index = 0; index < operators.length; index += 1) { + const operator = operators[index]; + const segmentNodes = segments[index]; + if (operator === undefined || segmentNodes === undefined) { + throw new Error('operator and segment lists diverged while folding a lowered sequence'); + } + if (segmentNodes.length === 0) { + diagnose(context, 'latex/operator-placement-unparsed', detail); + return unparsed(detail); + } + folded = app(operator, [folded, lowerTerm(segmentNodes, context)]); + } + return folded; +} + +// A run of nodes with no binary/relation operator inside: binders and named functions consume the rest of the run, digit runs fold into one numeric literal, and ANY remaining adjacency degrades to one `unparsed` node -- the juxtaposition rule. Juxtaposition is where the issue draws the line between mechanical and context-starved: `mc^2`, `f(x)`, `2(x+1)` all have multiplication AND function application as defensible readings, and LaTeX notation cannot say which, so the run stays visible data with a diagnostic instead of becoming a guess. +function lowerTerm(nodes: readonly TemmlNode[], context: LoweringContext): MathExpression { + return lowerTermItems(termItems(nodes), context); +} + +function lowerTermItems(items: readonly TermItem[], context: LoweringContext): MathExpression { + const first = items[0]; + if (first === undefined) { + diagnose(context, 'latex/construct-unparsed'); + return unparsed(''); + } + if (first.kind === 'node') { + const binder = readBinder(first.node, context); + if (binder.status === 'binder') { + const rest = items.slice(1); + const body = rest.length === 0 ? implicitArgument(context) : lowerTermItems(rest, { ...context, binders: [binder.binder, ...context.binders] }); + return { kind: binder.kind, binder: binder.binder, lower: binder.lower, upper: binder.upper, body }; + } + if (binder.status === 'degraded') { + return unparsed(binder.detail); + } + const namedFunction = namedFunctionOfOp(first.node); + if (namedFunction !== undefined) { + const rest = items.slice(1); + const argument = rest.length === 0 ? implicitArgument(context) : lowerTermItems(rest, context); + return app(namedFunction, [argument]); + } + } + const lowered = lowerTermItem(first, context); + if (items.length > 1) { + const detail = spanOfNodes(context, nodesOfItems(items)); + diagnose(context, 'latex/juxtaposition-unparsed', detail); + return unparsed(detail); + } + return lowered; +} + +function nodesOfItems(items: readonly TermItem[]): readonly TemmlNode[] { + const nodes: TemmlNode[] = []; + for (const item of items) { + if (item.kind === 'node') { + nodes.push(item.node); + } else { + nodes.push(...item.nodes); + } + } + return nodes; +} + +// A binder or named function whose run held nothing after it: the application still lowers, with the missing operand itself an `unparsed` node so the gap is data. +function implicitArgument(context: LoweringContext): MathExpression { + diagnose(context, 'latex/construct-unparsed'); + return unparsed(''); +} + +// Fold runs of digit textords into single numeric items, leaving every other node as its own item. +function termItems(nodes: readonly TemmlNode[]): readonly TermItem[] { + const items: TermItem[] = []; + let digits: string[] = []; + let digitNodes: TemmlNode[] = []; + const flushDigits = (): void => { + if (digits.length > 0) { + items.push({ kind: 'number', literal: digits.join(''), nodes: digitNodes }); + digits = []; + digitNodes = []; + } + }; + for (const node of nodes) { + const text = nodeText(node); + if (node.type === 'textord' && text !== undefined && (numericText(text) || text === '.')) { + digits.push(text); + digitNodes.push(node); + continue; + } + flushDigits(); + items.push({ kind: 'node', node }); + } + flushDigits(); + return items; +} + +function lowerTermItem(item: TermItem, context: LoweringContext): MathExpression { + if (item.kind === 'number') { + const rational = decimalToRational(item.literal); + if (rational === undefined) { + diagnose(context, 'latex/construct-unparsed', item.literal); + return unparsed(item.literal); + } + return { kind: 'num', numerator: rational.numerator, denominator: rational.denominator }; + } + return lowerNode(item.node, context); +} + +// A single node. Everything that is not one of the mechanical constructs below degrades to its own verbatim span with a construct diagnostic -- the total-by-degradation contract. +function lowerNode(node: TemmlNode, context: LoweringContext): MathExpression { + switch (node.type) { + case 'mathord': { + const glyph = glyphOfNode(node); + return glyph === undefined ? degradeNode(node, context) : symbolExpression(glyph, context); + } + case 'textord': { + const text = nodeText(node); + if (text === undefined) { + return degradeNode(node, context); + } + if (/^[0-9]$/.test(text)) { + return { kind: 'num', numerator: text, denominator: '1' }; + } + const glyph = glyphOfSymbolText(text); + return glyph === undefined ? degradeNode(node, context) : symbolExpression(glyph, context); + } + case 'atom': { + // bin/rel atoms that reach here sit where no operator folding applies (a lone '=', a '+' with nothing around it); punct/open/close atoms are interval-and-list notation the grammar has no reading for. All degrade. + return degradeNode(node, context); + } + case 'genfrac': + return lowerGenfrac(node, context); + case 'sqrt': + return lowerSqrt(node, context); + case 'supsub': + return lowerSupsub(node, context); + case 'op': + // Ops that reach here have no argument following them in their run (a bare \sum or \sin); the term level handles every scripted binder and every applied function. + return degradeNode(node, context); + case 'ordgroup': + case 'styling': + case 'color': { + const body = node.body; + if (!Array.isArray(body) || !body.every(isTemmlNode)) { + return degradeNode(node, context); + } + return lowerNodeList(body, context); + } + case 'delimiter': + case 'leftright': + return lowerGrouping(node, context); + case 'array': + return lowerArray(node, context); + case 'text': { + const detail = spanOfNodes(context, [node]); + diagnose(context, 'latex/text-unparsed', detail); + return unparsed(detail); + } + default: + return degradeNode(node, context); + } +} + +function degradeNode(node: TemmlNode, context: LoweringContext): MathExpression { + const span = spanOfNodes(context, [node]); + const detail = span !== '' ? span : (nodeText(node) ?? node.type); + diagnose(context, 'latex/construct-unparsed', detail); + return unparsed(detail); +} + +function symbolExpression(glyph: string, context: LoweringContext): MathExpression { + // The lexical rule: an in-scope binder's name shadows everything else -- the bound variable is local to the binder's body, and its id is the binder name itself rather than a table reference. + if (context.binders.includes(glyph)) { + return { kind: 'sym', id: glyph }; + } + return { kind: 'sym', id: context.resolver.resolve(glyph) }; +} + +// \frac -- the one generalised fraction that is unambiguously division: a bar, no delimiters. \binom and friends (delimiters drawn around them) and the bar-less \genfrac forms degrade rather than become a division they do not assert. temml spells "no delimiter" as null on the genfrac node, so absence is null-or-undefined on both fields. +function lowerGenfrac(node: TemmlNode, context: LoweringContext): MathExpression { + const hasBar = node.hasBarLine !== false; + const delimited = (node.leftDelim !== null && node.leftDelim !== undefined) || (node.rightDelim !== null && node.rightDelim !== undefined); + if (!hasBar || delimited || !isTemmlNode(node.numer) || !isTemmlNode(node.denom)) { + const detail = spanOfNodes(context, [node]); + diagnose(context, 'latex/genfrac-unparsed', detail); + return unparsed(detail); + } + return app('math:divide', [lowerNode(node.numer, context), lowerNode(node.denom, context)]); +} + +// Radicals: \sqrt{x} is math:sqrt; \sqrt[n]{x} is x raised to the exact rational 1/n -- the mechanical identity between radical index and rational exponent, with the exponent itself built as a division so n never rounds. temml spells "no index" as null on the sqrt node. +function lowerSqrt(node: TemmlNode, context: LoweringContext): MathExpression { + const index = node.index === null ? undefined : node.index; + if (!isTemmlNode(node.body) || (index !== undefined && !isTemmlNode(index))) { + return degradeNode(node, context); + } + const body = lowerNode(node.body, context); + if (index === undefined) { + return app('math:sqrt', [body]); + } + return app('math:pow', [body, app('math:divide', [{ kind: 'num', numerator: '1', denominator: '1' }, lowerNode(index, context)])]); +} + +// A grouping construct -- bare parenthesised (content) arrives as a 'delimiter' node, \left(...\right) as 'leftright'. Both lower their inner sequence, so (a + b)^2 becomes pow(add(a, b), 2); a grouping adjacent to anything else was already degraded by the term level's juxtaposition rule. A grouping wrapping exactly one array node is a bracketed matrix (pmatrix, bmatrix) -- the wrapper is presentation, the array is the content. +function lowerGrouping(node: TemmlNode, context: LoweringContext): MathExpression { + const body = node.body; + if (!Array.isArray(body) || !body.every(isTemmlNode)) { + return degradeNode(node, context); + } + if (body.length === 1) { + const only = body[0]; + if (only?.type === 'array') { + return lowerArray(only, context); + } + } + return lowerNodeList(body, context); +} + +// The matrix environments: rows of cells, each cell lowered whole. Layout-semantic environments (align, cases, aligned -- anything carrying envClasses) and column-spec arrays with separators degrade, and so does a ragged body, because the schema's matrix demands equal row widths and inventing padding cells would be a silent guess. +function lowerArray(node: TemmlNode, context: LoweringContext): MathExpression { + const separated = Array.isArray(node.cols) && node.cols.some((col) => !isTemmlNode(col) || col.type === 'separator'); + const body = node.body; + const envClasses = node.envClasses; + const layoutEnvironment = Array.isArray(envClasses) && envClasses.length > 0; + if (layoutEnvironment || separated || !Array.isArray(body)) { + const detail = spanOfNodes(context, [node]); + diagnose(context, layoutEnvironment ? 'latex/array-environment-unparsed' : 'latex/construct-unparsed', detail); + return unparsed(detail); + } + const rows: MathExpression[][] = []; + for (const row of body) { + if (!Array.isArray(row) || !row.every(isTemmlNode)) { + return degradeNode(node, context); + } + rows.push(row.map((cell) => lowerNode(cell, context))); + } + if (rows.length > 0 && new Set(rows.map((row) => row.length)).size > 1) { + const detail = spanOfNodes(context, [node]); + diagnose(context, 'latex/construct-unparsed', detail); + return unparsed(detail); + } + return { kind: 'matrix', rows }; +} + +// -- Scripts -- + +// The one place presentation is allowed to change SEMANTICS by lookup: a subscript makes a distinct symbol identity (x_1 is never x times 1 -- subscripting is how notation spells "another symbol"), resolved through the symbol table like any other glyph; a superscript is exponentiation UNLESS the table already curates the scripted form as one symbol (a document where an embellished pair is a single named quantity -- the table says so, the notation cannot). +function lowerSupsub(node: TemmlNode, context: LoweringContext): MathExpression { + const binder = readBinder(node, context); + if (binder.status === 'binder') { + // A scripted Sigma reached as a bare node rather than a term head: it owns no summand here, and that gap stays an unparsed body. + return { kind: binder.kind, binder: binder.binder, lower: binder.lower, upper: binder.upper, body: implicitArgument(context) }; + } + if (binder.status === 'degraded') { + return unparsed(binder.detail); + } + const detail = spanOfNodes(context, [node]); + const base = isTemmlNode(node.base) ? node.base : undefined; + const sub = node.sub === undefined ? undefined : nodesOfScript(node.sub); + const sup = node.sup === undefined ? undefined : nodesOfScript(node.sup); + if (sub !== undefined) { + const subWritten = simpleScriptGlyph(sub); + const baseGlyph = base !== undefined && (base.type === 'mathord' || base.type === 'textord') ? glyphOfNode(base) : undefined; + if (baseGlyph === undefined || subWritten === undefined) { + diagnose(context, 'latex/subscript-unparsed', detail); + return unparsed(detail); + } + const subscriptedGlyph = `${baseGlyph}_${subWritten}`; + if (sup === undefined) { + return symbolExpression(subscriptedGlyph, context); + } + const tripleGlyph = `${subscriptedGlyph}^${scriptWrittenForm(sup, context)}`; + if (context.resolver.isCurated(tripleGlyph)) { + return symbolExpression(tripleGlyph, context); + } + return app('math:pow', [symbolExpression(subscriptedGlyph, context), lowerNodeList(sup, context)]); + } + if (sup === undefined) { + return degradeNode(node, context); + } + const exponent = lowerNodeList(sup, context); + if (base === undefined) { + return degradeNode(node, context); + } + if (base.type !== 'mathord' && base.type !== 'textord') { + const loweredBase = lowerNode(base, context); + if (loweredBase.kind === 'unparsed') { + diagnose(context, 'latex/script-base-unparsed', detail); + return unparsed(detail); + } + return app('math:pow', [loweredBase, exponent]); + } + const baseGlyph = glyphOfNode(base); + if (baseGlyph === undefined) { + return degradeNode(node, context); + } + const scriptedGlyph = `${baseGlyph}^${scriptWrittenForm(sup, context)}`; + if (context.resolver.isCurated(scriptedGlyph)) { + return symbolExpression(scriptedGlyph, context); + } + return app('math:pow', [symbolExpression(baseGlyph, context), exponent]); +} + +// The written form of a superscript run, for the combined glyphs the table curates ('x' + '2' -> 'x^2'): the verbatim source slice when the nodes carry positions, else the concatenated glyph texts. Verbatim first because the glyph field is "the written form as it appears in presentation". +function scriptWrittenForm(nodes: readonly TemmlNode[], context: LoweringContext): string { + const span = spanOfNodes(context, nodes); + if (span !== '') { + return span; + } + return nodes.map((node) => glyphOfNode(node) ?? nodeText(node) ?? '').join(''); +} + +// Whether a subscript run is a simple symbol suffix -- every node a plain glyph (letter, digit, or symbol command) with nothing structural inside. 'max', 'ij', '1' qualify; 'i+1', '(n)' do not, and their construct degrades rather than becoming a mangled identity. +function simpleScriptGlyph(nodes: readonly TemmlNode[]): string | undefined { + const parts: string[] = []; + for (const node of nodes) { + const text = nodeText(node); + if (node.type === 'mathord') { + const glyph = glyphOfNode(node); + if (glyph === undefined) { + return undefined; + } + parts.push(glyph); + continue; + } + if (node.type === 'textord' && text !== undefined && /^[0-9A-Za-z]$/.test(text)) { + parts.push(text); + continue; + } + return undefined; + } + return parts.length === 0 ? undefined : parts.join(''); +} + +// -- Binders and named functions -- + +// Reading a scripted big operator. A supsub whose base is a scripted Sigma or Product is a binder that OWNS the rest of its term (the summand/product term) -- the term level consults this before anything else, which is what makes \sum_{i=1}^{n} i^2 lower as one binder rather than a Sigma juxtaposed against its summand. \int is an op too but never a binder: the grammar's binders are exactly sum and prod, so integrals degrade as constructs and stay visible. +type BinderRead = + | { readonly status: 'binder'; readonly kind: 'sum' | 'prod'; readonly binder: string; readonly lower: MathExpression; readonly upper: MathExpression } + | { readonly status: 'degraded'; readonly detail: string } + | { readonly status: 'not-a-binder' }; + +function readBinder(node: TemmlNode, context: LoweringContext): BinderRead { + if (node.type !== 'supsub') { + return { status: 'not-a-binder' }; + } + const base = isTemmlNode(node.base) ? node.base : undefined; + if (base?.type !== 'op' || base?.symbol !== true || typeof base?.name !== 'string') { + return { status: 'not-a-binder' }; + } + if (base.name !== '\\sum' && base.name !== '\\prod') { + return { status: 'not-a-binder' }; + } + const kind = base.name === '\\sum' ? 'sum' : 'prod'; + const detail = spanOfNodes(context, [node]); + const sub = node.sub === undefined ? undefined : nodesOfScript(node.sub); + const sup = node.sup === undefined ? undefined : nodesOfScript(node.sup); + const upper = sup === undefined ? implicitBound(context) : lowerNodeList(sup, context); + // `_{name = expression}` + if ((sub?.length ?? 0) >= 3) { + const first = sub?.[0]; + const relation = sub?.[1]; + const rest = sub?.slice(2) ?? []; + const firstGlyph = first?.type === 'mathord' ? glyphOfNode(first) : undefined; + if (firstGlyph !== undefined && relation?.type === 'atom' && nodeText(relation) === '=') { + return { status: 'binder', kind, binder: firstGlyph, lower: lowerNodeList(rest, context), upper }; + } + } + // A bare bound glyph (`\sum_i`): the binder still lowers, the missing range stays visible. + if ((sub?.length ?? 0) === 1 && sub !== undefined) { + const singleGlyph = simpleScriptGlyph(sub); + if (singleGlyph !== undefined) { + diagnose(context, 'latex/binder-bound-implicit', detail); + return { status: 'binder', kind, binder: singleGlyph, lower: unparsed(''), upper }; + } + } + diagnose(context, 'latex/binder-bound-unreadable', detail); + return { status: 'degraded', detail }; +} + +// An absent upper bound: the binder node still carries the slot, filled with an `unparsed` node so the gap is data, plus the diagnostic naming it. +function implicitBound(context: LoweringContext): MathExpression { + diagnose(context, 'latex/binder-bound-implicit'); + return unparsed(''); +} + +// A named function op in head position (sin, log, exp): it consumes the rest of its run as its single argument, the same ownership rule a binder plays -- \sin x + 1 is add(sin(x), 1) because the run is split at '+' before the function ever looks. +function namedFunctionOfOp(node: TemmlNode): string | undefined { + if (node.type !== 'op' || node.symbol === true) { + return undefined; + } + const name = typeof node.name === 'string' ? node.name : undefined; + return name === undefined ? undefined : NAMED_FUNCTION_OPERATORS[name]; +} + +// -- The public surface -- + +export interface LowerLatexOptions { + // The document symbol table's entries this lowering resolves glyphs against (a formula's `sym` references stay small because definitions live in the table once per document). + readonly symbolEntries?: readonly MathSymbolEntry[]; + // A sink receiving every diagnostic as it is emitted, alongside the aggregated copy on the result. + readonly sink?: LatexDiagnosticSink; +} + +export interface LatexLoweringResult { + // The lowered expression -- always defined, worst case one `unparsed` node carrying the verbatim source. + readonly expression: MathExpression; + readonly diagnostics: readonly LatexDiagnostic[]; + // Table entries minted for glyphs no supplied entry covered, merge-ready for the document's symbolTable so every emitted `sym` reference resolves. + readonly mintedSymbols: readonly MathSymbolEntry[]; +} + +// Lower one LaTeX string to a MathExpression. Total: a string the pinned parser cannot read returns an `unparsed` root with a parse-error diagnostic, never a throw. +export function lowerLatex(latex: string, options?: LowerLatexOptions): LatexLoweringResult { + return lowerParsed(latex, parseLatex(latex), options); +} + +function lowerParsed(latex: string, parsed: LatexParseResult, options?: LowerLatexOptions): LatexLoweringResult { + if (latex.trim() === '') { + return { expression: unparsed(''), diagnostics: [], mintedSymbols: [] }; + } + const resolver = new SymbolResolver(options?.symbolEntries ?? []); + const context: LoweringContext = { input: latex, resolver, binders: [], diagnostics: [], sink: options?.sink }; + if (parsed.status === 'unparseable') { + diagnose(context, 'latex/parse-error', parsed.message); + return { expression: unparsed(latex), diagnostics: context.diagnostics, mintedSymbols: resolver.mintedEntries() }; + } + const expression = lowerNodeList(parsed.nodes, context); + return { expression, diagnostics: context.diagnostics, mintedSymbols: resolver.mintedEntries() }; +} + +export interface LatexFormulaOptions extends LowerLatexOptions { + // The provenance source recorded on the formula (a pipeline stage such as 'lowered:latex', or a format origin such as 'markdown:math-block'); the edit trail starts empty because lowering is the birth of the pair, not an edit to it. + readonly source?: string; +} + +export interface LatexFormulaResult { + readonly formula: ContentFormula; + readonly diagnostics: readonly LatexDiagnostic[]; + readonly mintedSymbols: readonly MathSymbolEntry[]; +} + +// Lower one LaTeX string into a whole ContentFormula: the verbatim presentation layer, the presentation-MathML tree (so the formula renders through the existing MathML engine instead of degrading to text), the lowered content layer, and provenance. Both layers are stored as-authoritative per the schema -- nothing here derives one from the other at rest. +export function latexToFormula(latex: string, options?: LatexFormulaOptions): LatexFormulaResult { + const parsed = parseLatex(latex); + const lowering = lowerParsed(latex, parsed, options); + const mathml: MathMlNode[] = parsed.status === 'parsed' ? [...parsed.mathml] : []; + const formula: ContentFormula = { + mathml, + presentation: { latex }, + content: lowering.expression, + provenance: { source: options?.source ?? 'lowered:latex', editTrail: [] }, + }; + return { formula, diagnostics: lowering.diagnostics, mintedSymbols: lowering.mintedSymbols }; +} diff --git a/src/latex/rational.ts b/src/latex/rational.ts new file mode 100644 index 00000000..00d9122b --- /dev/null +++ b/src/latex/rational.ts @@ -0,0 +1,32 @@ +// Exact-rational arithmetic over the schema's canonical decimal-integer strings, shared by the lowering (decimal literals -> lowest-terms rationals) and the coherence lint (normalising stored rationals for comparison). BigInt throughout: Number loses integer exactness above 2^53 and exactness is the entire point of the string-encoded rational the schema defines. + +function gcd(a: bigint, b: bigint): bigint { + let x = a; + let y = b; + while (y !== 0n) { + const next = x % y; + x = y; + y = next; + } + // gcd(0, 0) is defined as 1 here so 0/0-shaped degenerates reduce to 0/1 rather than dividing by zero -- the schema's patterns keep 0's denominator at '1', and this keeps the arithmetic total on the same convention. + return x === 0n ? 1n : x; +} + +// A decimal literal (digits with at most one point) as a lowest-terms rational: '3.14' -> 157/50, '42' -> 42/1. Undefined for a malformed literal (two points, empty digits), which the lowering degrades visibly rather than repair. +export function decimalToRational(literal: string): { numerator: string; denominator: string } | undefined { + const point = literal.indexOf('.'); + const digits = point === -1 ? literal : literal.slice(0, point) + literal.slice(point + 1); + if (!/^\d+$/.test(digits) || literal.includes('.', point + 1)) { + return undefined; + } + const denominatorPower = point === -1 ? 0 : literal.length - point - 1; + const numerator = BigInt(digits); + const denominator = 10n ** BigInt(denominatorPower); + return reduceRational(numerator, denominator); +} + +// Reduce numerator/denominator to lowest terms as the schema's canonical producer convention, so string equality between two reduced forms is value equality. +export function reduceRational(numerator: bigint, denominator: bigint): { numerator: string; denominator: string } { + const divisor = gcd(numerator, denominator); + return { numerator: String(numerator / divisor), denominator: String(denominator / divisor) }; +} diff --git a/src/latex/symbols.test.ts b/src/latex/symbols.test.ts new file mode 100644 index 00000000..2eb0e084 --- /dev/null +++ b/src/latex/symbols.test.ts @@ -0,0 +1,79 @@ +import type { ContentDocument } from 'document-schema.js'; +import { CONTENT_FORMAT_VERSION } from 'document-schema.js'; +import { describe, expect, it } from 'vitest'; +import type { LatexDiagnostic } from './diagnostics'; +import { extractSymbolDefinitionsFromProse } from './symbols'; + +// The prose scanner's conservatism is the point (precision over recall): every case below pins a boundary the matcher must respect -- the two where/let forms it reads, and the shapes it declines rather than mis-seeding the table. + +function wordprocessing(paragraphs: readonly string[]): ContentDocument { + return { + kind: 'wordprocessing', + formatVersion: CONTENT_FORMAT_VERSION, + metadata: {}, + sections: [{ pageSize: { widthPt: 595, heightPt: 842 }, margins: { topPt: 20, rightPt: 20, bottomPt: 20, leftPt: 20 }, blocks: paragraphs.map((text) => ({ kind: 'paragraph' as const, runs: [{ text }] })) }], + }; +} + +describe('extractSymbolDefinitionsFromProse', () => { + it('seeds a table entry from "where R is the resistance per unit length"', () => { + const document = wordprocessing(['Consider a line. Where R is the resistance per unit length, the loss grows.']); + const diagnostics: LatexDiagnostic[] = []; + const entries = extractSymbolDefinitionsFromProse(document, (diagnostic) => diagnostics.push(diagnostic)); + expect(entries).toEqual([ + { + glyph: 'R', + scope: 'document', + id: 'symbols:R', + definitionSource: 'Where R is the resistance per unit length, the loss grows.', + }, + ]); + expect(diagnostics).toEqual([{ code: 'symbols/prose-definition-found', detail: '"R" from: Where R is the resistance per unit length, the loss grows.' }]); + }); + + it('seeds from "let x be the voltage" and Greek-letter subjects', () => { + const document = wordprocessing(['Let x be the voltage across the load. Let α denote the attenuation constant.']); + const entries = extractSymbolDefinitionsFromProse(document); + expect(entries.map((entry) => entry.glyph)).toEqual(['x', 'α']); + expect(entries.map((entry) => entry.id)).toEqual(['symbols:x', 'symbols:α']); + }); + + it('seeds an underscore-subscripted glyph whole (m_e), the table\'s own written-form convention', () => { + const entries = extractSymbolDefinitionsFromProse(wordprocessing(['where m_e is the electron mass'])); + expect(entries.map((entry) => entry.glyph)).toEqual(['m_e']); + }); + + it('declines whole words -- "where the resistance is high" seeds nothing', () => { + expect(extractSymbolDefinitionsFromProse(wordprocessing(['where the resistance is high']))).toEqual([]); + }); + + it('declines non-defining verbs -- "where R varies along the line" seeds nothing', () => { + expect(extractSymbolDefinitionsFromProse(wordprocessing(['where R varies along the line']))).toEqual([]); + }); + + it('declines definitions buried mid-sentence without a where/let head', () => { + expect(extractSymbolDefinitionsFromProse(wordprocessing(['The quantity R is the resistance per unit length.']))).toEqual([]); + }); + + it('keeps the first definition of a repeated glyph and reports each find through the sink', () => { + const diagnostics: LatexDiagnostic[] = []; + const entries = extractSymbolDefinitionsFromProse(wordprocessing(['where R is the resistance per unit length. where R is something else entirely.']), (diagnostic) => diagnostics.push(diagnostic)); + expect(entries).toHaveLength(1); + expect(diagnostics).toHaveLength(1); + }); + + it('scans every section\'s paragraphs and ignores non-wordprocessing arms entirely', () => { + const multiSection: ContentDocument = { + kind: 'wordprocessing', + formatVersion: CONTENT_FORMAT_VERSION, + metadata: {}, + sections: [ + { pageSize: { widthPt: 595, heightPt: 842 }, margins: { topPt: 20, rightPt: 20, bottomPt: 20, leftPt: 20 }, blocks: [{ kind: 'paragraph', runs: [{ text: 'where a is one thing' }] }] }, + { pageSize: { widthPt: 595, heightPt: 842 }, margins: { topPt: 20, rightPt: 20, bottomPt: 20, leftPt: 20 }, blocks: [{ kind: 'paragraph', runs: [{ text: 'unrelated prose' }] }, { kind: 'pageBreak' }] }, + ], + }; + expect(extractSymbolDefinitionsFromProse(multiSection).map((entry) => entry.glyph)).toEqual(['a']); + const formula: ContentDocument = { kind: 'formula', formatVersion: CONTENT_FORMAT_VERSION, metadata: {}, formula: { mathml: [] } }; + expect(extractSymbolDefinitionsFromProse(formula)).toEqual([]); + }); +}); diff --git a/src/latex/symbols.ts b/src/latex/symbols.ts new file mode 100644 index 00000000..8094ca95 --- /dev/null +++ b/src/latex/symbols.ts @@ -0,0 +1,170 @@ +import type { ContentDocument, MathSymbolEntry } from 'document-schema.js'; +import type { LatexDiagnostic } from './diagnostics'; + +// The symbol-table half of the lowering: the map from LaTeX's written symbol commands to the glyphs a symbol table keys on, the id-minting scheme that keeps a formula's `sym` references resolvable, and the conservative prose scanner that seeds a document's table from its own defining sentences. The symbol table is the curation layer the schema defines (document-schema.js src/math.ts): presentation-inert, never consulted by rendering, and the thing that makes a lowered equation computable -- a `sym` node is a reference, and this module is where references acquire targets. + +// A LaTeX symbol command to its single written glyph, the form the symbol table's own `glyph` field expects ("the written form as it appears in presentation"). temml hands symbol commands through as mathord/textord nodes whose text is the raw command ('\\alpha', '\\infty'); the table keys on what a reader actually sees, so the map is to the Unicode character. Greek (both cases, plus the variant letters), the letterlike symbols, and the few common blackboard/structural glyphs engineers actually write -- anything outside the map degrades to an `unparsed` node rather than entering the table under a command name no human wrote. +const COMMAND_GLYPHS: Readonly> = { + '\\alpha': 'α', + '\\beta': 'β', + '\\gamma': 'γ', + '\\delta': 'δ', + '\\epsilon': 'ε', + '\\varepsilon': 'ε', + '\\zeta': 'ζ', + '\\eta': 'η', + '\\theta': 'θ', + '\\vartheta': 'ϑ', + '\\iota': 'ι', + '\\kappa': 'κ', + '\\lambda': 'λ', + '\\mu': 'μ', + '\\nu': 'ν', + '\\xi': 'ξ', + '\\pi': 'π', + '\\varpi': 'ϖ', + '\\rho': 'ρ', + '\\varrho': 'ϱ', + '\\sigma': 'σ', + '\\varsigma': 'ς', + '\\tau': 'τ', + '\\upsilon': 'υ', + '\\phi': 'φ', + '\\varphi': 'φ', + '\\chi': 'χ', + '\\psi': 'ψ', + '\\omega': 'ω', + '\\Gamma': 'Γ', + '\\Delta': 'Δ', + '\\Theta': 'Θ', + '\\Lambda': 'Λ', + '\\Xi': 'Ξ', + '\\Pi': 'Π', + '\\Sigma': 'Σ', + '\\Upsilon': 'Υ', + '\\Phi': 'Φ', + '\\Psi': 'Ψ', + '\\Omega': 'Ω', + '\\infty': '∞', + '\\partial': '∂', + '\\nabla': '∇', + '\\ell': 'ℓ', + '\\hbar': 'ℏ', + '\\Re': 'ℜ', + '\\Im': 'ℑ', + '\\aleph': 'ℵ', +}; + +// The written glyph a symbol-command text carries, or undefined for a command outside the map. A plain (non-command) character is its own glyph and passes through unchanged. +export function glyphOfSymbolText(text: string): string | undefined { + if (!text.startsWith('\\')) { + return text; + } + return COMMAND_GLYPHS[text]; +} + +// The id a table entry for this glyph gets when nobody curated one: 'symbols:' plus the glyph itself. Deliberately the same scheme for every producer in this package (the prose scanner below, the lowering's auto-minting), so an entry seeded from prose and a reference auto-minted from a formula converge on one id instead of minting two spellings of the same symbol -- a curated table can override the id, and then every lookup goes through the curated entry, because minting only happens for glyphs the table does not already carry. +export function mintedSymbolId(glyph: string): string { + return `symbols:${glyph}`; +} + +// Glyph-to-id resolution state shared by one lowering run and the table it feeds. `curated` is the lookup over the table the caller supplied (or the prose-seeded table the markdown pass built); `minted` accumulates entries for glyphs no table entry covers, returned to the caller to merge so every `sym` reference the lowering emitted resolves against the resulting table. Duplicate glyphs in a supplied table are a curatorial error the schema declines to enforce; this resolver takes the FIRST entry per glyph deterministically rather than picking silently among them, so a lowering is reproducible either way. +export class SymbolResolver { + private readonly curated = new Map(); + private readonly minted = new Map(); + + constructor(entries: readonly MathSymbolEntry[]) { + for (const entry of entries) { + if (!this.curated.has(entry.glyph)) { + this.curated.set(entry.glyph, entry.id); + } + } + } + + // Whether the caller-curated table already carries this glyph -- the lowering's only consultation of the table's JUDGEMENT (as opposed to its id mapping): a scripted form the table curates as one symbol is one symbol, and exponentiation stands down. + isCurated(glyph: string): boolean { + return this.curated.has(glyph); + } + + // The id a `sym` node for this glyph references: the curated table's entry when one exists, else a freshly minted entry recorded for the caller to merge. Binder-local names (a `sum`/`prod` bound variable) are resolved by the lowering itself before it gets here -- they shadow the table inside the binder's body and never mint entries, because the bound variable's identity is lexical, not curated. + resolve(glyph: string): string { + const curatedId = this.curated.get(glyph); + if (curatedId !== undefined) { + return curatedId; + } + const existing = this.minted.get(glyph); + if (existing !== undefined) { + return existing.id; + } + const id = mintedSymbolId(glyph); + this.minted.set(glyph, { glyph, scope: 'document', id }); + return id; + } + + // Every glyph this run minted an entry for, in first-mint order -- merge-ready for the document's symbolTable alongside whatever the caller already curated (dedup by glyph is the caller's: this list never contains a glyph the curated entries carried). + mintedEntries(): readonly MathSymbolEntry[] { + return [...this.minted.values()]; + } +} + +// -- Symbol definitions from document prose -- + +// The shape a prose-defined symbol's written form may take: one Latin or Greek letter, optionally with an underscore subscript run ("R", "m_e", "x_1", "α"). A whole word ("where the resistance is...") does not match -- multi-letter runs are words, not symbols, and excluding them is most of what keeps this scanner conservative. +const PROSE_SYMBOL_PATTERN = /[A-Za-zΑ-ω](?:_[A-Za-z0-9]+)?/; + +// Sentence-level definition patterns, the two forms technical prose actually writes: "where R is the resistance per unit length" and "let x be the voltage". The verb set is deliberately small (is/are/be/denotes/denote/represents/stands for) -- "where R varies..." is not a definition, and matching it would mint a wrong quantity identity, which is precisely the failure precision-over-recall is here to avoid. Matches are case-insensitive on the keyword and verb only; the symbol itself is case-sensitive because R and r are different quantities. +const PROSE_DEFINITION_PATTERNS: readonly RegExp[] = [ + /\bwhere\s+([^\s,.;:]+)\s+(?:is|are|denotes?|represents?|stands\s+for)\b/i, + /\blet\s+([^\s,.;:]+)\s+(?:be|denote|represent|stand\s+for)\b/i, +]; + +// Rough sentence segmentation for the scanner: split at sentence-ending punctuation followed by whitespace or end of text. Deliberately rough -- a definition sentence mis-split at an abbreviation at worst misses one definition, and a curatorial pass over the table is where that gets fixed, not a grammar model here. +function sentencesOf(text: string): string[] { + return text + .split(/(?<=[.!?])\s+/) + .map((sentence) => sentence.trim()) + .filter((sentence) => sentence.length > 0); +} + +// Scans a wordprocessing document's own prose for symbol definitions and seeds table entries from them -- the "where equations acquire computability at all" half of the pipeline: a symbol the prose defines gets a table entry (with the defining sentence recorded as its definitionSource), and a formula's reference to that glyph then resolves to the curated entry instead of an auto-mint. Conservative by construction: only the two sentence-level where/let forms, only single-letter(+optional subscript) symbol shapes, and every hit is reported through the sink so a caller can audit what was seeded. No quantityKind or preferredUnit is ever inferred -- prose says what a symbol is in words, and mapping those words onto a quantity vocabulary is curation this package does not attempt. +export function extractSymbolDefinitionsFromProse(document: ContentDocument, sink?: (diagnostic: LatexDiagnostic) => void): MathSymbolEntry[] { + if (document.kind !== 'wordprocessing') { + return []; + } + const entries: MathSymbolEntry[] = []; + const seen = new Set(); + for (const section of document.sections) { + for (const block of section.blocks) { + if (block.kind !== 'paragraph') { + continue; + } + const text = block.runs.map((run) => run.text).join(''); + for (const sentence of sentencesOf(text)) { + const entry = proseDefinitionIn(sentence); + if (entry === undefined || seen.has(entry.glyph)) { + continue; + } + seen.add(entry.glyph); + entries.push(entry); + sink?.({ code: 'symbols/prose-definition-found', detail: `"${entry.glyph}" from: ${sentence}` }); + } + } + } + return entries; +} + +// One sentence's definition, or undefined when it holds no where/let definition pattern. definitionSource carries the defining sentence itself rather than a locator: for prose-sourced definitions the sentence IS the provenance a curator needs, and a paragraph index would rot the moment the document is edited while the sentence stays findable. +function proseDefinitionIn(sentence: string): MathSymbolEntry | undefined { + for (const pattern of PROSE_DEFINITION_PATTERNS) { + const match = pattern.exec(sentence); + const candidate = match?.[1]; + if (candidate === undefined) { + continue; + } + if (PROSE_SYMBOL_PATTERN.exec(candidate)?.[0] !== candidate) { + continue; + } + return { glyph: candidate, scope: 'document', id: mintedSymbolId(candidate), definitionSource: sentence }; + } + return undefined; +} diff --git a/src/latex/temml.ts b/src/latex/temml.ts new file mode 100644 index 00000000..469fabce --- /dev/null +++ b/src/latex/temml.ts @@ -0,0 +1,132 @@ +import type { MathMlNode } from 'document-schema.js'; +import temml from 'temml'; + +// The pinned LaTeX parser. temml (https://temml.org, MIT, zero dependencies of its own) is the one component of the two-layer math model this ecosystem deliberately does not hand-write -- a LaTeX grammar is a large, fiddly surface with no supply-chain-averse payoff the way the hand-written MathML typesetting engine has one -- so it is a real dependency, pinned to the EXACT version recorded in package.json ("temml": "0.13.4", no caret). The pin is load-bearing, not tidiness: this module consumes temml's underscore-prefixed internal API (__parse, the KaTeX-style parse-node tree, and __renderToMathMLTree, the virtual MathML tree), which carries no stability guarantee across releases, and the two-layer model's storage contract says a stored presentation string has ONE defined parse. A caret range would silently change that defined meaning under a consumer's feet; the exact pin makes "which parse does this stored string have" a function of the package's own version. Bumping the pin is a deliberate act that must re-run src/latex/lower.test.ts, whose lowering table cases pin the parse-node shapes this version produces. +// +// Worker-isomorphism holds: temml is pure JavaScript with no dependencies, its parser and virtual-MathML tree builder never touch the DOM (only the optional render/renderMathInElement entry points do, and they feature-detect `document` before using it -- this module never calls them), and test/workers/ proves the whole lowering path under workerd. The eslint no-restricted-imports guard enforcing the rest of src/'s isomorphism applies here unchanged. + +// What the parse and MathML passes are invoked with: throwOnError true because the lowering wants real parse failures surfaced (an unknown command degrades the whole expression to one `unparsed` node with a diagnostic, via the union below), not temml's error-coloured fallback rendering; maxExpand left at its default, which already bounds macro expansion against pathological input. +const TEMML_OPTIONS = { throwOnError: true } as const; + +// A temml parse node, seen only through the structural fields src/latex/lower.ts consumes. temml's own type declarations hand the tree back as `any`, so this package re-declares the surface it reads and narrows into it with the guards below -- the same treatment every loosely-typed third-party value gets at this package's boundaries. `type` is the node's discriminant ('mathord', 'genfrac', 'supsub', ...); every other field is `unknown` until a guard narrows it. +export interface TemmlNode { + readonly type: string; + readonly [field: string]: unknown; +} + +function isRecord(value: unknown): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} + +export function isTemmlNode(value: unknown): value is TemmlNode { + return isRecord(value) && typeof value.type === 'string'; +} + +export function isTemmlNodeArray(value: unknown): value is TemmlNode[] { + return Array.isArray(value) && value.every(isTemmlNode); +} + +// A node's source span: the verbatim substring of the parsed string this node came from, used wherever the lowering degrades something to an `unparsed` node -- the schema's contract is that a coverage gap stays visible as the exact source that resisted lowering, and a re-serialisation of the tree would not be that. Returns undefined only for nodes temml synthesised without a source position (its internal spacing/rule artefacts), which the lowering either skips as presentation-only or degrades with the enclosing construct's own span. +export interface TemmlSourceSpan { + readonly start: number; + readonly end: number; +} + +export function sourceSpanOf(node: TemmlNode): TemmlSourceSpan | undefined { + const loc = node.loc; + if (!isRecord(loc)) { + return undefined; + } + const start = loc.start; + const end = loc.end; + if (typeof start !== 'number' || typeof end !== 'number' || !Number.isInteger(start) || !Number.isInteger(end) || start < 0 || end < start) { + return undefined; + } + return { start, end }; +} + +export function sourceSlice(input: string, span: TemmlSourceSpan | undefined): string { + if (span === undefined) { + return ''; + } + return input.slice(span.start, span.end); +} + +// The result of handing one LaTeX string to the pinned parser: either the parse-node list plus the presentation-MathML tree (both derived from the same single parse, so the two views can never disagree about what the string said), or the parser's own failure message. +export type LatexParseResult = + | { readonly status: 'parsed'; readonly nodes: readonly TemmlNode[]; readonly mathml: readonly MathMlNode[] } + | { readonly status: 'unparseable'; readonly message: string }; + +export function parseLatex(latex: string): LatexParseResult { + try { + const nodes: unknown = temml.__parse(latex, TEMML_OPTIONS); + if (!isTemmlNodeArray(nodes)) { + return { status: 'unparseable', message: 'parser returned a shape this package does not recognise' }; + } + const mathml = mathmlOf(latex); + return { status: 'parsed', nodes, mathml }; + } catch (error) { + return { status: 'unparseable', message: errorMessage(error) }; + } +} + +function errorMessage(error: unknown): string { + return error instanceof Error ? error.message : String(error); +} + +// The parse-node list re-expressed as presentation MathML nodes -- the `mathml` field a lowered ContentFormula carries, so a LaTeX-authored formula renders through the existing MathML typesetting engine (src/mathml/layout.ts) exactly like an ODF-sourced one instead of degrading to its text stand-in. Derived from the same pinned parser at parse time and stored beside the verbatim latex: both are the presentation layer (the schema's rendering-authoritative half), so deriving one from the other at rest violates nothing -- what the schema forbids is re-deriving EITHER from the semantic layer. Returns an empty list when the tree holds a node shape this walk does not recognise, which is the schema-anticipated empty-mathml state (rendering falls back to the plain-text stand-in); the presentation string itself stays authoritative either way. +function mathmlOf(latex: string): MathMlNode[] { + let root: unknown; + try { + root = temml.__renderToMathMLTree(latex, TEMML_OPTIONS); + } catch { + return []; + } + return toMathMlNodes(root) ?? []; +} + +// temml's virtual MathML tree has three node shapes: MathNode (a tag string, a plain-object attribute map, child nodes), TextNode (a text payload), and DocumentFragment (a transparent grouping with children and nothing else -- temml wraps some trees' content in one). The walk maps them onto document-schema.js's MathMlNode (itself a transcription of odf.js's generic XML node shape), flattening fragments into their parent's children (MathML has no fragment construct) and dropping MathNode's classes/style/label fields -- those are temml-internal presentation metadata with no meaning to this package's MathML consumer, which keys off element names and the MathML spec's own attributes. An unrecognised child makes the whole conversion return undefined (the caller's [] degradation above): half a MathML tree is worse than none, because a renderer would typeset a silently amputated formula. +function toMathMlNodes(value: unknown): MathMlNode[] | undefined { + if (!isRecord(value)) { + return undefined; + } + if (typeof value.text === 'string' && value.type === undefined) { + return [{ type: 'text', value: value.text }]; + } + if (!Array.isArray(value.children)) { + return undefined; + } + const children: MathMlNode[] = []; + for (const child of value.children) { + const converted = toMathMlNodes(child); + if (converted === undefined) { + return undefined; + } + children.push(...converted); + } + if (value.type === undefined) { + // A DocumentFragment: transparent, its converted children stand in for it directly. + return children; + } + if (typeof value.type !== 'string') { + return undefined; + } + const attributes = attributesOf(value.attributes); + if (attributes === undefined) { + return undefined; + } + return [{ type: 'element', tag: value.type, attributes, children }]; +} + +function attributesOf(value: unknown): { readonly name: string; readonly value: string }[] | undefined { + if (!isRecord(value)) { + return undefined; + } + const attributes: { name: string; value: string }[] = []; + for (const [name, attributeValue] of Object.entries(value)) { + if (typeof attributeValue === 'string' || typeof attributeValue === 'number' || typeof attributeValue === 'boolean') { + attributes.push({ name, value: String(attributeValue) }); + } + } + return attributes; +} diff --git a/src/markdown/math.test.ts b/src/markdown/math.test.ts new file mode 100644 index 00000000..a815bef0 --- /dev/null +++ b/src/markdown/math.test.ts @@ -0,0 +1,163 @@ +import { describe, expect, it } from 'vitest'; +import type { LatexDiagnostic } from '../latex/diagnostics'; +import { lintMathCoherence } from '../latex/lint'; +import { buildMarkdownText } from './write'; +import { lowerMarkdownMath } from './math'; +import { readMarkdownContent } from './read'; + +// The markdown math-lowering pass end to end: markdown-codec's preserved raw LaTeX ($$ display blocks, \( \) inline spans) becomes the two-layer ContentFormula every format in this family shares, the document's own prose seeds the symbol table, and the write side reconstructs the same markdown math syntax from the presentation layer. These tests pin the whole pipeline the issue describes -- "parse at the format edge, lower at the model level, so every input format that can carry LaTeX benefits from one lowering implementation". + +const MATH_MARKDOWN = [ + '# Math document', + '', + 'Where R is the resistance per unit length.', + '', + '$$', + '\\sum_{i=1}^{n} \\frac{1}{i^2}', + '$$', + '', + 'An inline span \\(x^2 + 1\\) mid-sentence.', + '', + '$$', + '2x', + '$$', +].join('\n'); + +describe('readMarkdownContent math lowering', () => { + it('lowers a $$ display block into an embedded formula block carrying presentation, content, MathML, and provenance', () => { + const content = readMarkdownContent(MATH_MARKDOWN); + if (content.kind !== 'wordprocessing') { + throw new Error('expected a wordprocessing ContentDocument'); + } + const formulaBlock = content.sections[0]?.blocks.find((block) => block.kind === 'embeddedObject'); + if (formulaBlock?.kind !== 'embeddedObject' || formulaBlock.document.kind !== 'formula') { + throw new Error('expected an embedded formula block'); + } + const formula = formulaBlock.document.formula; + expect(formula.presentation).toEqual({ latex: '\\sum_{i=1}^{n} \\frac{1}{i^2}' }); + expect(formula.content).toEqual({ + kind: 'sum', + binder: 'i', + lower: { kind: 'num', numerator: '1', denominator: '1' }, + upper: { kind: 'sym', id: 'symbols:n' }, + body: { kind: 'app', operator: 'math:divide', args: [{ kind: 'num', numerator: '1', denominator: '1' }, { kind: 'app', operator: 'math:pow', args: [{ kind: 'sym', id: 'i' }, { kind: 'num', numerator: '2', denominator: '1' }] }] }, + }); + expect(formula.provenance).toEqual({ source: 'markdown:math-block', editTrail: [] }); + const root = formula.mathml[0]; + expect(root?.type === 'element' ? root.tag : undefined).toBe('math'); + }); + + it('seeds the document symbol table from prose, and a formula referencing the prose-defined glyph resolves to the curated entry', () => { + const content = readMarkdownContent('Where R is the resistance per unit length.\n\n$$\nR^2\n$$\n'); + if (content.kind !== 'wordprocessing') { + throw new Error('expected a wordprocessing ContentDocument'); + } + const table = content.symbolTable; + expect(table?.units).toEqual([]); + const curated = table?.symbols.find((entry) => entry.glyph === 'R'); + expect(curated).toEqual({ glyph: 'R', scope: 'document', id: 'symbols:R', definitionSource: 'Where R is the resistance per unit length.' }); + const formulaBlock = content.sections[0]?.blocks.find((block) => block.kind === 'embeddedObject'); + if (formulaBlock?.kind !== 'embeddedObject' || formulaBlock.document.kind !== 'formula') { + throw new Error('expected an embedded formula block'); + } + expect(formulaBlock.document.formula.content).toEqual({ kind: 'app', operator: 'math:pow', args: [{ kind: 'sym', id: 'symbols:R' }, { kind: 'num', numerator: '2', denominator: '1' }] }); + }); + + it('a document with no math and no prose definitions carries no symbol table at all', () => { + const content = readMarkdownContent('# Plain\n\nJust prose, no symbols defined and no math.\n'); + if (content.kind !== 'wordprocessing') { + throw new Error('expected a wordprocessing ContentDocument'); + } + expect(content.symbolTable).toBeUndefined(); + }); + + it('an inline \\( \\) span leaves its paragraph and follows it as a formula block', () => { + const content = readMarkdownContent('An inline span \\(x^2 + 1\\) mid-sentence.\n'); + if (content.kind !== 'wordprocessing') { + throw new Error('expected a wordprocessing ContentDocument'); + } + const blocks = content.sections[0]?.blocks ?? []; + expect(blocks).toHaveLength(2); + const paragraph = blocks[0]; + expect(paragraph?.kind).toBe('paragraph'); + expect(paragraph?.kind === 'paragraph' ? paragraph.runs.map((run) => run.text) : undefined).toEqual(['An inline span ', ' mid-sentence.']); + const formulaBlock = blocks[1]; + if (formulaBlock?.kind !== 'embeddedObject' || formulaBlock.document.kind !== 'formula') { + throw new Error('expected an embedded formula block after the paragraph'); + } + expect(formulaBlock.document.formula.presentation).toEqual({ latex: 'x^2 + 1' }); + expect(formulaBlock.document.formula.provenance).toEqual({ source: 'markdown:math-inline', editTrail: [] }); + }); + + it('a paragraph that is nothing but an inline span is consumed by its formula block', () => { + const content = readMarkdownContent('\\(E\\)\n'); + if (content.kind !== 'wordprocessing') { + throw new Error('expected a wordprocessing ContentDocument'); + } + const blocks = content.sections[0]?.blocks ?? []; + expect(blocks).toHaveLength(1); + expect(blocks[0]?.kind).toBe('embeddedObject'); + }); + + it('a context-starved construct degrades to unparsed inside the formula and surfaces its diagnostic through the sink', () => { + const diagnostics: LatexDiagnostic[] = []; + const content = readMarkdownContent('$$\n2x\n$$\n', undefined, { onDiagnostic: (diagnostic) => diagnostics.push(diagnostic) }); + if (content.kind !== 'wordprocessing') { + throw new Error('expected a wordprocessing ContentDocument'); + } + const formulaBlock = content.sections[0]?.blocks[0]; + if (formulaBlock?.kind !== 'embeddedObject' || formulaBlock.document.kind !== 'formula') { + throw new Error('expected an embedded formula block'); + } + expect(formulaBlock.document.formula.content).toEqual({ kind: 'unparsed', latex: '2x' }); + expect(diagnostics.some((diagnostic) => diagnostic.code === 'latex/juxtaposition-unparsed')).toBe(true); + }); + + it('an unparseable $$ block still becomes a formula carrying the verbatim presentation and an unparsed root -- the text is never lost', () => { + const content = readMarkdownContent('$$\n\\notacommand{x}\n$$\n'); + if (content.kind !== 'wordprocessing') { + throw new Error('expected a wordprocessing ContentDocument'); + } + const formulaBlock = content.sections[0]?.blocks[0]; + if (formulaBlock?.kind !== 'embeddedObject' || formulaBlock.document.kind !== 'formula') { + throw new Error('expected an embedded formula block'); + } + expect(formulaBlock.document.formula.presentation).toEqual({ latex: '\\notacommand{x}' }); + expect(formulaBlock.document.formula.mathml).toEqual([]); + expect(formulaBlock.document.formula.content).toEqual({ kind: 'unparsed', latex: '\\notacommand{x}' }); + }); + + it('math inside a table cell is lowered too', () => { + // Inline math, not a $$ block -- markdown-codec's block-math syntax needs its own delimiter lines, which cannot occur inside a single-line table cell, so the in-cell shape is the \\( \\) span. + const content = readMarkdownContent('| head |\n| --- |\n| cell \\(a + b\\) text |\n'); + if (content.kind !== 'wordprocessing') { + throw new Error('expected a wordprocessing ContentDocument'); + } + const table = content.sections[0]?.blocks.find((block) => block.kind === 'table'); + if (table?.kind !== 'table') { + throw new Error('expected a table block'); + } + const cellBlocks = table.rows[1]?.cells[0]?.blocks ?? []; + expect(cellBlocks[0]?.kind).toBe('paragraph'); + expect(cellBlocks[1]?.kind).toBe('embeddedObject'); + }); +}); + +describe('buildMarkdownText math reconstruction', () => { + it('round-trips display and inline math from the presentation layer, verbatim', () => { + const content = readMarkdownContent(MATH_MARKDOWN); + const text = buildMarkdownText(content); + expect(text).toContain('$$\n\\sum_{i=1}^{n} \\frac{1}{i^2}\n$$'); + expect(text).toContain('\\(x^2 + 1\\)'); + expect(text).toContain('$$\n2x\n$$'); + // The re-read document lowers the identical presentation strings again -- the two-layer model round-trips through markdown without touching the semantic layer. + const reread = readMarkdownContent(text); + expect(lintMathCoherence({ formatVersion: 2, content: reread })).toEqual([]); + }); + + it('a formula with no presentation layer still flattens to its plain-text stand-in', () => { + const content = lowerMarkdownMath(readMarkdownContent('no math here\n')); + const text = buildMarkdownText(content); + expect(text).toContain('no math here'); + }); +}); diff --git a/src/markdown/math.ts b/src/markdown/math.ts new file mode 100644 index 00000000..4415c8e8 --- /dev/null +++ b/src/markdown/math.ts @@ -0,0 +1,112 @@ +import type { ContentBlock, ContentDocument, ContentRun, MathSymbolEntry } from 'document-schema.js'; +import type { LatexDiagnosticSink } from '../latex/diagnostics'; +import { latexToFormula } from '../latex/lower'; +import { extractSymbolDefinitionsFromProse } from '../latex/symbols'; +import { buildFormulaBlock } from '../model/formula'; + +// The markdown read path's math-lowering pass: markdown-codec recognises math syntax ($$ display blocks and \( \) inline spans, its issue #53) but deliberately stops at carrying the raw LaTeX through as styled text -- a MathBlock-styled paragraph for display math, a Cambria-Math-marked run for inline math, with diagnostics saying "it is not parsed as LaTeX or converted to MathML by this package". This pass is that conversion, living here rather than inside markdown-codec because it is a documents.js question (the issue trail is explicit: markdown-codec#53 deferred to documents.js#563, which settled on model-level lowering in #572): every formula shape is lowered once into the two-layer ContentFormula every format in this family shares, so markdown math becomes typesettable (MathML), editable (OMML via the docx writer), and computable (the MathExpression content layer) without markdown-codec knowing any of that exists. +// +// The two marker strings markdown-codec's lowered runs/paragraphs carry are mirrored here as literals: markdown-codec re-exports its style-constant vocabulary for exactly this sibling-package use, but MATH_BLOCK_STYLE_ID ('MathBlock') and MATH_INLINE_FONT_MARKER ('Cambria Math') are not among the re-exported names. They are stable, documented conventions of markdown-codec's own lower/emit pair (the same "standard, not invented" naming its Courier New code-span marker plays), and this package's read pass (recognition) and write pass (src/markdown/write.ts's reconstruction) agree on them exactly as markdown-codec's own two halves do. + +const MATH_BLOCK_STYLE_ID = 'MathBlock'; +const MATH_INLINE_FONT_MARKER = 'Cambria Math'; + +// The stand-in frame for a formula markdown never gave geometry: markdown records no page geometry at all (src/markdown/read.ts's own pageSize note), so there is no source box to carry. Mirrors the docx OMML recovery's own convention (src/ooxml/docx/formula.ts): width 0 tells the layout fit (formulaSizePtForFrame) that height alone drives the rendered size, and the height is twice the size a body-text formula renders at -- Word's own 11pt body default, since markdown states no size either. +const STAND_IN_FRAME = { xPt: 0, yPt: 0, widthPt: 0, heightPt: 22 }; + +// Provenance sources the pass stamps, distinguishing the two markdown math shapes so the write path (src/markdown/write.ts) can reconstruct the same shape back out: display blocks re-emit as $$ blocks, inline spans as marker runs. +const MATH_BLOCK_SOURCE = 'markdown:math-block'; +const MATH_INLINE_SOURCE = 'markdown:math-inline'; + +export interface MarkdownMathLoweringOptions { + // Receives every diagnostic the pass emits -- prose definitions seeded, per-formula degradations -- as they happen. + readonly onDiagnostic?: LatexDiagnosticSink; +} + +// One paragraph's inline math extraction: the LaTeX of each Cambria-Math-marked run, in order, paired with the paragraph as it stands minus those runs. An all-math paragraph (nothing but math runs) is consumed entirely -- the same "a paragraph carrying nothing but its equation IS the equation" rule the docx and odf recoveries play -- so the formula blocks replace it in place rather than leaving an empty paragraph behind. +interface InlineMathExtraction { + readonly latexRuns: readonly string[]; + readonly remainingRuns: readonly ContentRun[]; + readonly consumed: boolean; +} + +function extractInlineMath(runs: readonly ContentRun[]): InlineMathExtraction { + const latexRuns: string[] = []; + const remainingRuns: ContentRun[] = []; + for (const run of runs) { + if (run.fontFamily === MATH_INLINE_FONT_MARKER && run.text.trim() !== '') { + latexRuns.push(run.text); + continue; + } + remainingRuns.push(run); + } + const consumed = latexRuns.length > 0 && remainingRuns.every((run) => run.text === ''); + return { latexRuns, remainingRuns: consumed ? [] : remainingRuns, consumed }; +} + +// Lower a wordprocessing ContentDocument's markdown-carried math in place-free fashion: returns a NEW document (nothing of the input is mutated -- this is a read adapter's projection, not a layout pass stamping frames). Display-math paragraphs become embedded formula blocks carrying presentation+content+MathML+provenance; inline math runs leave their paragraph and follow it as formula blocks, the position convention the odf/docx recoveries already established for inline equations. The document's symbol table is seeded from its own prose definitions first, so a formula referencing a prose-defined glyph resolves to the curated entry; glyphs nobody defined are minted into the table so every reference resolves. +export function lowerMarkdownMath(document: ContentDocument, options?: MarkdownMathLoweringOptions): ContentDocument { + if (document.kind !== 'wordprocessing') { + return document; + } + const sink = options?.onDiagnostic; + const proseEntries = extractSymbolDefinitionsFromProse(document, sink); + const known = new Map(proseEntries.map((entry) => [entry.glyph, entry])); + const minted = new Map(); + const lowerOne = (latex: string, source: string): ContentBlock | undefined => { + if (latex.trim() === '') { + return undefined; + } + const result = latexToFormula(latex, { symbolEntries: [...known.values()], source }); + for (const diagnostic of result.diagnostics) { + sink?.(diagnostic); + } + for (const entry of result.mintedSymbols) { + if (!known.has(entry.glyph) && !minted.has(entry.glyph)) { + minted.set(entry.glyph, entry); + } + } + return buildFormulaBlock(result.formula, STAND_IN_FRAME, source); + }; + const lowerBlocks = (blocks: readonly ContentBlock[]): ContentBlock[] => { + const out: ContentBlock[] = []; + for (const block of blocks) { + if (block.kind === 'table') { + out.push({ ...block, rows: block.rows.map((row) => ({ ...row, cells: row.cells.map((cell) => ({ ...cell, blocks: lowerBlocks(cell.blocks) })) })) }); + continue; + } + if (block.kind !== 'paragraph') { + out.push(block); + continue; + } + if (block.styleId === MATH_BLOCK_STYLE_ID) { + const latex = block.runs.map((run) => run.text).join(''); + const formulaBlock = lowerOne(latex, MATH_BLOCK_SOURCE); + out.push(formulaBlock ?? block); + continue; + } + const extraction = extractInlineMath(block.runs); + if (extraction.latexRuns.length === 0) { + out.push(block); + continue; + } + if (!extraction.consumed) { + out.push({ ...block, runs: [...extraction.remainingRuns] }); + } + for (const latex of extraction.latexRuns) { + const formulaBlock = lowerOne(latex, MATH_INLINE_SOURCE); + if (formulaBlock !== undefined) { + out.push(formulaBlock); + } + } + } + return out; + }; + const sections = document.sections.map((section) => ({ ...section, blocks: lowerBlocks(section.blocks) })); + const symbols = [...known.values(), ...minted.values()]; + if (symbols.length === 0) { + // No prose definitions and no minted glyphs (a formula of pure numbers mints nothing): the table stays unset rather than carried empty -- the schema's own "a document with no lowered math content simply omits it". + return { ...document, sections }; + } + return { ...document, sections, symbolTable: { symbols, units: [] } }; +} diff --git a/src/markdown/read.ts b/src/markdown/read.ts index 32b1faff..d796a4cc 100644 --- a/src/markdown/read.ts +++ b/src/markdown/read.ts @@ -1,15 +1,17 @@ import type { ReadMarkdownOptions } from 'markdown-codec'; import { readMarkdown } from 'markdown-codec'; import type { ContentDocument } from 'document-schema.js'; +import { lowerMarkdownMath } from './math'; +import type { MarkdownMathLoweringOptions } from './math'; // markdown text -> ContentDocument (the wordprocessing variant). A thin adapter over markdown-codec's own readMarkdown, mirroring src/ooxml/docx/read.ts's readDocxContent / src/odf/ods/read.ts's readOdsContent: markdown-codec's own readMarkdown produces a document-schema.js ContentDocument (kind/formatVersion/metadata/sections) directly. This used to be two nominally distinct ContentDocument types (markdown-codec independently pinned document-schema.js at ^1.5.3, a pre-2.0.0 release, one major behind the 2.0.0+ this package depends on directly), which forced a re-parse through this package's own ContentDocumentSchema to sidestep the recursively-nested mismatch. That version skew is gone: markdown-codec now depends on document-schema.js@^2.2.4, the same range this package depends on directly, and pnpm resolves both to the single installed copy (`pnpm why document-schema.js` shows exactly one) -- so readMarkdown's return value is now genuinely, nominally the same ContentDocument type this function returns, and the plain pass-through this comment used to anticipate is what's below. // -// Unlike readOdtContent/readOdpContent, this adapter runs no second embedded-formula detection pass over its own source: CommonMark/GFM has no embedded-object or formula construct at all for one to find, so there is nothing to detect even speculatively. (A formula reaching markdown from the OTHER direction still degrades to its own plain-text stand-in -- see src/markdown/write.ts's own markdownBlock.) -export function readMarkdownContent(text: string, options?: ReadMarkdownOptions): ContentDocument { +// Unlike readOdtContent/readOdpContent, this adapter runs no second embedded-formula detection pass over its own source: CommonMark/GFM has no embedded-object or formula construct at all for one to find, so there is nothing to detect even speculatively. (A formula reaching markdown from the OTHER direction still degrades to its own plain-text stand-in -- see src/markdown/write.ts's own markdownBlock.) What this adapter DOES add is the math-lowering pass (src/markdown/math.ts): markdown-codec hands $$ display blocks and \( \) inline spans through as raw LaTeX text, and the pass lowers that LaTeX into the two-layer ContentFormula every format in this family shares -- the model-level placement documents.js#563 settled on -- so markdown-carried math typesets, edits, and computes like math from any other format. +export function readMarkdownContent(text: string, options?: ReadMarkdownOptions, math?: MarkdownMathLoweringOptions): ContentDocument { const { document } = readMarkdown(text, { frontMatter: true, ...options }); // readMarkdown's declared return type is the full ContentDocument union, even though it always produces the wordprocessing variant in practice (markdown has no presentation/spreadsheet/drawing/formula equivalent to lower into) -- this both documents and enforces that, mirroring every other readXContent adapter's own kind guard in this package (readDocxContent, readOdtContent, readOdsContent, readOdgContent). if (document.kind !== 'wordprocessing') { throw new Error('readMarkdown returned a non-wordprocessing ContentDocument'); } - return document; + return lowerMarkdownMath(document, math); } diff --git a/src/markdown/write.ts b/src/markdown/write.ts index a206513f..f63505a9 100644 --- a/src/markdown/write.ts +++ b/src/markdown/write.ts @@ -6,6 +6,22 @@ import { formulaOfBlock, formulaPlaceholderText } from '../model/formula'; // markdown-codec's own writeMarkdown parameter type is now the SAME document-schema.js ContentDocument this package imports: markdown-codec bumped its own document-schema.js dependency to ^2.2.4 (matching this package's own ^2.2.4 direct dependency), and pnpm resolves both to the single installed copy (`pnpm why document-schema.js` shows exactly one). The version-skew this file used to reshape around (markdown-codec independently pinned at ^1.5.3, a pre-2.0.0 release with CONTENT_FORMAT_VERSION 1, TypeScript treating the two import("document-schema.js")-sourced types as nominally unrelated all the way down through ContentBlock -> ContentEmbeddedObjectBlock -> document) no longer exists, so there is no LegacyDocument mirror type and no field-by-field document rebuild needed any more -- a real ContentDocument goes straight into writeMarkdown. // // What remains is a genuine semantic transformation writeMarkdown does not do on its own: markdown-codec's own emit path drops every embeddedObject block uniformly, regardless of what it nests (see that package's own src/write.ts -> emit/emit.ts, renderTopLevelBlock's `case "embeddedObject": return ""`). That is exactly right for a recovered drawing -- a rect/ellipse/line/path carries no text to stand in for, and CommonMark/GFM has no vector construct anyway -- but wrong for an embedded formula, whose own text this package still wants preserved rather than silently discarded. markdownBlock below is that one transformation (flatten a formula block to its own plain-text stand-in) and nothing else; every other block, a recovered drawing included, is passed through unchanged and left to writeMarkdown's own uniform embeddedObject handling to drop. +// A formula carrying a presentation LaTeX string is reconstructed as markdown MATH rather than flattened to plain text: markdown-codec's own math vocabulary (its issue #53) round-trips a $$ display block as a MathBlock-styled paragraph and an inline \( \) span as a run marked with the Cambria Math fontFamily, and its emit path regenerates both shapes from exactly those markers -- so the formula's verbatim presentation string (rendering-authoritative, never re-derived from the semantic layer) goes back out as the same syntax it arrived in. Which of the two shapes is chosen by the formula's recorded provenance source: a span that arrived inline goes back inline (in a paragraph of its own -- the block model never retained its position inside the source paragraph, the same position loss every inline-equation recovery in this package already has), everything else goes back as a display block. The marker strings are mirrored here as literals for the same reason src/markdown/math.ts mirrors its read-side pair: markdown-codec documents them as stable lower/emit conventions but does not re-export these two among its public style constants. +const MATH_BLOCK_STYLE_ID = 'MathBlock'; +const MATH_INLINE_FONT_MARKER = 'Cambria Math'; +const MATH_INLINE_SOURCE = 'markdown:math-inline'; + +function formulaParagraph(formula: NonNullable>): ContentBlock { + const latex = formula.presentation?.latex; + if (latex === undefined) { + return { kind: 'paragraph', runs: [{ text: formulaPlaceholderText(formula) }] }; + } + if (formula.provenance?.source === MATH_INLINE_SOURCE) { + return { kind: 'paragraph', runs: [{ text: latex, fontFamily: MATH_INLINE_FONT_MARKER }] }; + } + return { kind: 'paragraph', runs: [{ text: latex }], styleId: MATH_BLOCK_STYLE_ID }; +} + function markdownBlock(block: ContentBlock): ContentBlock { // A table is recursive too -- ContentTableCell.blocks is itself ContentBlock[], so a table cell could carry its own nested embeddedObject (or another nested table) needing the identical treatment as a top-level block. if (block.kind === 'table') { @@ -15,7 +31,7 @@ function markdownBlock(block: ContentBlock): ContentBlock { // An embedded formula is FLATTENED to a paragraph carrying its own plain-text stand-in -- its StarMath annotation, or the literal "[formula]" -- rather than left to writeMarkdown's own default embeddedObject handling, which would silently drop it just like a drawing. Flattening is what actually gets the formula's own text into the markdown. CommonMark/GFM has no math construct to do better with, and this package writes no MathML into docx or odt either in the cases where it still falls back to text (buildDocxPackage/buildOdtPackage's own appendEmbeddedObject, which degrade the identical way when a formula carries no MathML nodes at all). const formula = formulaOfBlock(block); if (formula !== undefined) { - return { kind: 'paragraph', runs: [{ text: formulaPlaceholderText(formula) }] }; + return formulaParagraph(formula); } // A recovered drawing (or any other embeddedObject kind) is passed through unchanged -- writeMarkdown's own uniform embeddedObject handling drops it regardless of what it nests, so there is nothing further to do here. return block; diff --git a/src/model/formula.ts b/src/model/formula.ts index 62f9a13a..0aa78780 100644 --- a/src/model/formula.ts +++ b/src/model/formula.ts @@ -22,8 +22,8 @@ export function formulaOfBlock(block: ContentEmbeddedObjectBlock): ContentFormul return block.document.kind === 'formula' ? block.document.formula : undefined; } -// The plain-text stand-in for a formula a consumer cannot typeset: its own StarMath annotation when the source carried one, else a literal marker. Never an empty string -- an empty stand-in is indistinguishable from the formula having been silently dropped. +// The plain-text stand-in for a formula a consumer cannot typeset: its own StarMath annotation when the source carried one, else the verbatim presentation LaTeX when the formula carries one (a LaTeX-authored formula's own source text is the most faithful thing to show -- and the only thing to show when the pinned parser could not read it, since such a formula's MathML array is empty and its rendering would otherwise collapse to a bare marker), else a literal marker. Never an empty string -- an empty stand-in is indistinguishable from the formula having been silently dropped. export function formulaPlaceholderText(formula: ContentFormula): string { - return formula.starMath ?? '[formula]'; + return formula.starMath ?? formula.presentation?.latex ?? '[formula]'; } diff --git a/test/workers/latex.test.ts b/test/workers/latex.test.ts new file mode 100644 index 00000000..81b93b50 --- /dev/null +++ b/test/workers/latex.test.ts @@ -0,0 +1,45 @@ +import { describe, expect, it } from 'vitest'; +import { buildMarkdownText, latexToFormula, lintMathCoherence, readMarkdownContent } from '../../src'; + +// Proves the whole LaTeX lowering path (the pinned temml parser through the lowering, the markdown read pass, the write reconstruction, and the coherence lint) executes inside a Cloudflare Workers isolate with no Node-only API usage -- temml is pure JavaScript whose DOM-touching entry points (render/renderMathInElement) are never called by this package, and whose parser/MathML-tree builder feature-detect `document` before using it (see src/latex/temml.ts's own top-of-file comment). If temml's parse path (or anything this feature pulls in) touched node:fs/Buffer/process at module top level or during a call, the workerd isolate would throw at import or fail these assertions rather than pass. +describe('the LaTeX lowering under the Cloudflare Workers runtime', () => { + it('lowerLatex\'s pinned parser parses and lowers real math inside workerd', async () => { + const { lowerLatex } = await import('../../src/latex/lower'); + const result = lowerLatex('\\sum_{i=1}^{n} \\frac{1}{i^2}'); + expect(result.expression).toEqual({ + kind: 'sum', + binder: 'i', + lower: { kind: 'num', numerator: '1', denominator: '1' }, + upper: { kind: 'sym', id: 'symbols:n' }, + body: { kind: 'app', operator: 'math:divide', args: [{ kind: 'num', numerator: '1', denominator: '1' }, { kind: 'app', operator: 'math:pow', args: [{ kind: 'sym', id: 'i' }, { kind: 'num', numerator: '2', denominator: '1' }] }] }, + }); + expect(result.diagnostics).toEqual([]); + }); + + it('latexToFormula produces a presentation-MathML tree inside workerd', () => { + const result = latexToFormula('\\frac{a}{b}'); + const root = result.formula.mathml[0]; + expect(root?.type).toBe('element'); + expect(root?.type === 'element' ? root.tag : undefined).toBe('math'); + }); + + it('the markdown read pass lowers $$ display math into a two-layer formula block inside workerd', () => { + const content = readMarkdownContent('Where R is the resistance.\n\n$$\nR^2\n$$\n'); + if (content.kind !== 'wordprocessing') { + throw new Error('expected a wordprocessing ContentDocument'); + } + const formulaBlock = content.sections[0]?.blocks.find((block) => block.kind === 'embeddedObject'); + if (formulaBlock === undefined || formulaBlock.kind !== 'embeddedObject' || formulaBlock.document.kind !== 'formula') { + throw new Error('expected an embedded formula block'); + } + expect(formulaBlock.document.formula.presentation).toEqual({ latex: 'R^2' }); + expect(formulaBlock.document.formula.content).toEqual({ kind: 'app', operator: 'math:pow', args: [{ kind: 'sym', id: 'symbols:R' }, { kind: 'num', numerator: '2', denominator: '1' }] }); + // The write side reconstructs the same markdown math from the presentation layer, all inside the isolate. + expect(buildMarkdownText(content)).toContain('$$\nR^2\n$$'); + }); + + it('the coherence lint runs over a package inside workerd', () => { + const content = readMarkdownContent('$$\nx^2\n$$\n'); + expect(lintMathCoherence({ formatVersion: 2, content })).toEqual([]); + }); +});