diff --git a/README.md b/README.md index 16977659f..fcdeb308a 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ [![GitHub](https://img.shields.io/badge/GitHub-181717?logo=github&logoColor=white)](https://github.com/ExaDev/documents.js) [![npm](https://img.shields.io/badge/npm-CB3837?logo=npm&logoColor=white)](https://www.npmjs.com/package/documents.js) [![Release](https://img.shields.io/github/v/release/ExaDev/documents.js)](https://github.com/ExaDev/documents.js/releases/latest) [![CI](https://img.shields.io/github/actions/workflow/status/ExaDev/documents.js/ci.yml?branch=main)](https://github.com/ExaDev/documents.js/actions) -> Converts between any two compatible document formats through a shared content/layout pivot. docx, pptx, odt, odp, ods, odg, xlsx, and markdown all read into and build from the same `ContentDocument`/`LayoutDocument` model, with PDF as the one format every variant can reach. A composition engine (`convertDocument`) routes 73 (source, target) pairs across the eight content formats and PDF, including fourteen PDF-pivot round trips, sixteen cross-format bridges (same-variant direct copies, cross-variant semantic transforms, and PDF-composed), plus special-case conversions for `.odm` master documents, `.odb` database front-ends (HSQLDB and Firebird, four storage tiers), standalone `.odf` formula documents, and a bounded SQL/rpt-formula engine for `.odb` reports. Also includes: read-and-write live-view editors for all six editable formats, docx comment/footnote/header-footer exposure via `readDocxExtras`, real font resolution (source-embedded faces ahead of caller-supplied, vendored substitutes, and the standard 14), a hand-written MathML typesetting engine with embedded-font PDF rendering and a matching MathML ⇄ OMML translator, and a fully hand-written PDF codec. Built on [ooxml.js](https://github.com/ExaDev/ooxml.js), [odf.js](https://github.com/ExaDev/odf.js), [pdf-codec](https://github.com/ExaDev/pdf-codec), [markdown-codec](https://github.com/ExaDev/markdown-codec), and [document-schema.js](https://github.com/ExaDev/document-schema.js). +> Converts between any two compatible document formats through a shared content/layout pivot. docx, pptx, odt, odp, ods, odg, xlsx, csv (TSV is the same format with a tab delimiter), and markdown all read into and build from the same `ContentDocument`/`LayoutDocument` model, with PDF as the one format every variant can reach. A composition engine (`convertDocument`) routes 91 (source, target) pairs across the nine content formats and PDF, including eighteen PDF-pivot round trips (the seven layout-engine formats, plus xlsx and csv composing through ods), twenty-two cross-format bridge functions (same-variant direct copies, cross-variant semantic transforms, and PDF-composed), plus special-case conversions for `.odm` master documents, `.odb` database front-ends (HSQLDB and Firebird, four storage tiers), standalone `.odf` formula documents, and a bounded SQL/rpt-formula engine for `.odb` reports. Also includes: read-and-write live-view editors for all six editable formats, docx comment/footnote/header-footer exposure via `readDocxExtras`, real font resolution (source-embedded faces ahead of caller-supplied, vendored substitutes, and the standard 14), a hand-written MathML typesetting engine with embedded-font PDF rendering and a matching MathML ⇄ OMML translator, and a fully hand-written PDF codec. Built on [ooxml.js](https://github.com/ExaDev/ooxml.js), [odf.js](https://github.com/ExaDev/odf.js), [pdf-codec](https://github.com/ExaDev/pdf-codec), [markdown-codec](https://github.com/ExaDev/markdown-codec), and [document-schema.js](https://github.com/ExaDev/document-schema.js). `documents.js` extends `ooxml.js` in two directions `ooxml.js` deliberately does not cover: full PDF support (parsing and generating, via `pdf-codec`), and a read-**and-write** manipulation API for docx/pptx content — `ooxml.js`'s own typed readers are one-way. The PDF codec is hand-written against ISO 32000-1, with no external PDF library as a dependency — see [Fidelity](#fidelity) and pdf-codec's own README for the honest trade-off (not as robust against adversarial PDFs as a 15+-year-hardened library; fully auditable and dependency-free instead). `src/mathml/` (the MathML typesetting engine) stays in this package and is hand-written too, for the same supply-chain reason. @@ -72,7 +72,7 @@ npm install documents.js ### The generic entry point: `convertDocument` -A single function, `convertDocument`, sits behind every named conversion and reaches every pair the composition engine can route — all 73 supported (source, target) combinations. The named functions below are thin one-line forwarders to it; they remain the ergonomic layer for a caller who wants a fixed pair and autocomplete discovery, while `convertDocument` is the first-class entry point for a caller working from a runtime format pair (CLI, MCP tool, matrix enumeration). +A single function, `convertDocument`, sits behind every named conversion and reaches every pair the composition engine can route — all 91 supported (source, target) combinations. The named functions below are thin one-line forwarders to it; they remain the ergonomic layer for a caller who wants a fixed pair and autocomplete discovery, while `convertDocument` is the first-class entry point for a caller working from a runtime format pair (CLI, MCP tool, matrix enumeration). ```ts import { convertDocument } from 'documents.js'; @@ -89,10 +89,10 @@ const odtBytes = convertDocument('docx', 'odt', docxBytes, { onMathDiagnostic: ( ### PDF-pivot conversions -The fourteen round-trip ergonomic conversions between the formats with their own layout engine and PDF (docx/pptx/odt/odp/ods/odg/markdown ⇄ PDF, all round-tripping both ways), plus `xlsxToPdf`/`pdfToXlsx` (composing the ods⇄xlsx bridge with the ods⇄pdf layout pair internally): +The fourteen round-trip ergonomic conversions between the formats with their own layout engine and PDF (docx/pptx/odt/odp/ods/odg/markdown ⇄ PDF, all round-tripping both ways), plus `xlsxToPdf`/`pdfToXlsx` and `csvToPdf`/`pdfToCsv` (each composing its ods bridge with the ods⇄pdf layout pair internally — neither xlsx nor csv has a layout engine of its own): ```ts -import { docxToPdf, markdownToPdf, odgToPdf, odpToPdf, odsToPdf, odtToPdf, pdfToDocx, pdfToMarkdown, pdfToOdg, pdfToOdp, pdfToOds, pdfToOdt, pdfToPptx, pdfToXlsx, pptxToPdf, xlsxToPdf } from 'documents.js'; +import { csvToPdf, docxToPdf, markdownToPdf, odgToPdf, odpToPdf, odsToPdf, odtToPdf, pdfToCsv, pdfToDocx, pdfToMarkdown, pdfToOdg, pdfToOdp, pdfToOds, pdfToOdt, pdfToPptx, pdfToXlsx, pptxToPdf, xlsxToPdf } from 'documents.js'; const pdfBytes = docxToPdf(docxBytes); const docxBytes2 = pdfToDocx(pdfBytes); @@ -117,13 +117,16 @@ const xlsxBytes2 = pdfToXlsx(pdfFromXlsx); // composes pdfToOds -> odsToXlsx int const pdfFromMarkdown = markdownToPdf(markdownBytes); const markdownBytes2 = pdfToMarkdown(pdfFromMarkdown); // the lossiest conversion in the whole package -- see Fidelity + +const pdfFromCsv = csvToPdf(csvBytes); // composes csvToOds -> odsToPdf internally +const csvBytes2 = pdfToCsv(pdfFromCsv); // composes pdfToOds -> odsToCsv internally; recovers what was printed, then heuristically re-types it ``` Each accepts an optional `signal` (`AbortSignal`) and either `onSubstitution` (X → PDF, called per character not representable in a standard-14 font) or `sink` (PDF → X, called per recoverable parse diagnostic). Every X → PDF conversion additionally accepts `fonts` (extra `ProvidedFont` faces) and `onFontSubstitution` (per family+weight+style that resolved to something else). Neither is needed for the common case — see [Fonts](#fonts). ### Cross-format bridges -Sixteen bridge functions across eight pairs bypass the PDF pivot entirely. Five same-variant direct-copy pairs (`odtToDocx`/`docxToOdt`, `odpToPptx`/`pptxToOdp`, `odsToXlsx`/`xlsxToOds`, `markdownToDocx`/`docxToMarkdown`, `markdownToOdt`/`odtToMarkdown`) compose a direct `readXContent` → `buildYPackage` pivot copy. Two cross-variant semantic-transform pairs (`docxToPptx`/`pptxToDocx`, `odtToOdp`/`odpToOdt`) go through `src/convert/variant-bridges.ts`. One PDF-composed pair (`xlsxToMarkdown`/`markdownToXlsx`) routes through PDF internally — the single lossiest conversion in the package. +Twenty-two bridge functions across eleven pairs bypass the PDF pivot where a direct path exists. Seven same-variant direct-copy pairs (`odtToDocx`/`docxToOdt`, `odpToPptx`/`pptxToOdp`, `odsToXlsx`/`xlsxToOds`, `csvToOds`/`odsToCsv`, `csvToXlsx`/`xlsxToCsv`, `markdownToDocx`/`docxToMarkdown`, `markdownToOdt`/`odtToMarkdown`) compose a direct `readXContent` → `buildYPackage` pivot copy — the csv pairs are one hop to its spreadsheet siblings, so csv never needs PDF to reach ods or xlsx. Two cross-variant semantic-transform pairs (`docxToPptx`/`pptxToDocx`, `odtToOdp`/`odpToOdt`) go through `src/convert/variant-bridges.ts`. Two PDF-composed pairs (`xlsxToMarkdown`/`markdownToXlsx`, `csvToMarkdown`/`markdownToCsv`) route through PDF internally — the lossiest conversions in the package. ```ts import { odtToDocx, docxToOdt, markdownToDocx, docxToMarkdown } from 'documents.js'; @@ -135,7 +138,7 @@ const docxFromMarkdown = markdownToDocx(markdownBytes); const markdownBytes3 = docxToMarkdown(docxFromMarkdown); // colour, font family/size, and explicit alignment have no markdown source construct -- dropped on this hop ``` -Each takes an optional `{ signal }` — no `onSubstitution`/`sink`, since there is no font substitution or PDF-parse degradation. `odtToDocx`/`markdownToDocx`/`docxToOdt`/`docxToMarkdown` additionally take `onMathDiagnostic`, called per formula construct that degraded crossing the bridge. +Each takes an optional `{ signal }` — no `onSubstitution`/`sink`, since there is no font substitution or PDF-parse degradation. `odtToDocx`/`markdownToDocx`/`docxToOdt`/`docxToMarkdown` additionally take `onMathDiagnostic`, called per formula construct that degraded crossing the bridge. The csv-sourced bridges (`csvToOds`, `csvToXlsx`, `csvToMarkdown`, `csvToPdf`) take `{ delimiter }` — `'\t'` parses the same format as TSV, since a delimiter is a parse option, not a different document format — and `onCellTypeInference`, the per-decision audit channel the read shares with `pdfToOds`. The csv-target bridges (`odsToCsv`, `xlsxToCsv`, `markdownToCsv`, `pdfToCsv`) take `{ delimiter, sheet }`: csv has no second sheet, so writing a multi-sheet source refuses with `CsvSheetNotSpecifiedError` naming every sheet until a caller selects one. ### The `DocumentConverter` port @@ -151,12 +154,12 @@ const { document, diagnostics } = await converter.convert( ); ``` -`DocumentFormat` includes `docx`/`pptx`/`xlsx`/`odt`/`odp`/`ods`/`odg`/`odf`/`markdown`/`pdf` — ten members. The port's `conversions` list is derived from `resolveCompositionPlan` plus the `odf`→`pdf` special case — 73 pairs total. `DocumentFormat` is inferred from `DocumentFormatSchema` (a real Zod schema); `DOCUMENT_FORMATS` is exported as a plain array derived from the same schema: +`DocumentFormat` includes `docx`/`pptx`/`xlsx`/`odt`/`odp`/`ods`/`odg`/`odf`/`csv`/`markdown`/`pdf` — eleven members. The port's `conversions` list is derived from `resolveCompositionPlan` plus the `odf`→`pdf` special case — 91 pairs total. `DocumentFormat` is inferred from `DocumentFormatSchema` (a real Zod schema); `DOCUMENT_FORMATS` is exported as a plain array derived from the same schema: ```ts import { DOCUMENT_FORMATS, DocumentFormatSchema } from 'documents.js'; -console.log(DOCUMENT_FORMATS); // ['docx', 'pptx', 'xlsx', 'odt', 'odp', 'ods', 'odg', 'odf', 'markdown', 'pdf'] +console.log(DOCUMENT_FORMATS); // ['docx', 'pptx', 'xlsx', 'odt', 'odp', 'ods', 'odg', 'odf', 'csv', 'markdown', 'pdf'] DocumentFormatSchema.parse(userSuppliedFormat); // throws a ZodError for anything outside that list ``` @@ -202,7 +205,7 @@ const docxBytesAgain = buildDocumentBytes(captured, 'docx'); ### Package decode/encode, metadata, and deep imports -`decodeDocumentPackage`/`encodeDocumentPackage` dispatch docx/pptx/xlsx through `ooxml.js`'s OPC codec and odt/odp/ods/odg/odf through `odf.js`'s ODF codec, throwing `UnsupportedPackageFormatError` for `markdown`/`pdf`. `decodeOdbPackage` is the `.odb`-specific sibling (`.odb` is not a `DocumentFormat` member): +`decodeDocumentPackage`/`encodeDocumentPackage` dispatch docx/pptx/xlsx through `ooxml.js`'s OPC codec and odt/odp/ods/odg/odf through `odf.js`'s ODF codec, throwing `UnsupportedPackageFormatError` for `markdown`/`csv`/`pdf` (none of the three is a package — they are plain text and bytes respectively). `decodeOdbPackage` is the `.odb`-specific sibling (`.odb` is not a `DocumentFormat` member): ```ts import { decodeDocumentPackage, decodeOdbPackage, encodeDocumentPackage } from 'documents.js'; @@ -212,7 +215,7 @@ const docxBytesAgain = encodeDocumentPackage('docx', pkg); const odbPkg = decodeOdbPackage(odbBytes); ``` -`readDocumentMetadata`/`setDocumentMetadata` read or patch metadata across any `DocumentFormat`. `setDocumentMetadata` patches in place (source/target formats must match); `odf` is rejected in both directions. `readDocumentMetadata('xlsx', ...)` is a named exception: it renders via `xlsxToPdf` and reads the PDF's metadata, because a direct read and the PDF-preview path genuinely disagree on `createdIso`/`modifiedIso`/`producer`. +`readDocumentMetadata`/`setDocumentMetadata` read or patch metadata across any `DocumentFormat`. `setDocumentMetadata` patches in place (source/target formats must match); `odf` is rejected in both directions, and `csv` is rejected in both directions too (RFC 4180 text has no metadata container) — `readDocumentMetadata('csv', ...)` answers an empty `LayoutMetadata` for the same reason. `readDocumentMetadata('xlsx', ...)` is a named exception: it renders via `xlsxToPdf` and reads the PDF's metadata, because a direct read and the PDF-preview path genuinely disagree on `createdIso`/`modifiedIso`/`producer`. ```ts import { readDocumentMetadata, setDocumentMetadata } from 'documents.js'; @@ -230,7 +233,7 @@ import { buildOdtPackage } from 'documents.js/edit/odt/content'; ### Reading and building xlsx content directly -Every other content format has its own standalone `readXContent`-shaped entry point (`readDocxContent`, `readPptxContent`, `readOdtContent`, `readOdpContent`, `readOdsContent`, `readOdgContent`) — xlsx is no longer the exception. `readXlsxContent`/`buildXlsxPackage` are `ooxml.js`'s own spreadsheet `ContentDocument` read/build pair — the same one the `ods⇄xlsx` bridge and every xlsx metadata-rebuild path already use internally — re-exported here directly rather than wrapped, since `readXlsxContent` already produces the right shape on its own: +Every other content format has its own standalone `readXContent`-shaped entry point (`readDocxContent`, `readPptxContent`, `readOdtContent`, `readOdpContent`, `readOdsContent`, `readOdgContent`) — xlsx is no longer the exception. `readXlsxContent`/`buildXlsxPackage` are `ooxml.js`'s own spreadsheet `ContentDocument` read/build pair — the same one the `ods⇄xlsx` bridge and every xlsx metadata-rebuild path already use internally — re-exported here directly rather than wrapped, since `readXlsxContent` already produces the right shape on its own. csv's `readCsvContent`/`buildCsvText` are the same kind of directly-exported stage pair, one level further in: they operate on RFC 4180 text rather than a decoded package (see `src/csv/` under Architecture). ```ts import { buildXlsxPackage, decodeDocumentPackage, encodeDocumentPackage, readXlsxContent } from 'documents.js'; @@ -326,7 +329,7 @@ const layout = readPdf(pdfBytes); // -> LayoutDocument: pages of positioned text const bytes = writePdf(layout); ``` -The nine PDF round trips and ten PDF-bypassing bridges are also available as schema-validated [`z.codec()`](https://zod.dev) pairs (`pdfCodec`, `docxPdfCodec`, `pptxPdfCodec`, `odtPdfCodec`, `odpPdfCodec`, `odsPdfCodec`, `odgPdfCodec`, `xlsxPdfCodec`, `markdownPdfCodec`, `odtDocxCodec`, `odpPptxCodec`, `odsXlsxCodec`, `markdownDocxCodec`, `markdownOdtCodec`) — the no-options form, adding automatic two-way schema validation: +The ten PDF round trips and fourteen PDF-bypassing bridge directions are also available as schema-validated [`z.codec()`](https://zod.dev) pairs (`pdfCodec`, `docxPdfCodec`, `pptxPdfCodec`, `odtPdfCodec`, `odpPdfCodec`, `odsPdfCodec`, `odgPdfCodec`, `xlsxPdfCodec`, `csvPdfCodec`, `markdownPdfCodec`, `odtDocxCodec`, `odpPptxCodec`, `odsXlsxCodec`, `odsCsvCodec`, `xlsxCsvCodec`, `markdownDocxCodec`, `markdownOdtCodec`) — the no-options form, adding automatic two-way schema validation. The two PDF-composed pairs have codec forms too (`xlsxMarkdownCodec`, `csvMarkdownCodec`): ```ts import { z } from 'zod'; @@ -504,6 +507,7 @@ The package is layered from generic primitives outward to the two conversion dir - **`src/ooxml/`** — thin adapters over `ooxml.js`'s own `readDocx`/`readPptx`, wrapping results into `ContentDocument`. `docx/formula.ts` is the one local reading pass (splicing OOXML math equations). `docx/extras.ts`'s `readDocxExtras` returns comments/footnotes/headers/footers/numbering. - **`src/odf/`** — ODF-side counterparts: `readOdtContent`/`readOdpContent`/`readOdsContent`/`readOdgContent` are thin adapters over `odf.js`. `formula/read.ts`/`formula/detect.ts` handle embedded formula detection (genuinely new work with no `odf.js`-side equivalent). - **`src/markdown/`** — third adapter family, via `markdown-codec`. `readMarkdownContent` passes `readMarkdown`'s result straight through (it already produces a full `ContentDocument`). `buildMarkdownText` wraps `writeMarkdown`. `text.ts` is the byte↔text boundary. `MarkdownEditor` holds a mutable in-memory `ContentDocument`. +- **`src/csv/`** — fourth adapter family, sharing the spreadsheet variant with xlsx/ods. `records.ts` is the RFC 4180 record parser/writer (one shared `quoteCsvField`, also used by the `.odb` CSV exporter); `text.ts` is the byte↔text boundary, rejecting malformed UTF-8; `read.ts` turns records into a spreadsheet `ContentDocument` (first record as verbatim string header, data cells through the same cell-typing heuristic `pdfToOds` uses); `write.ts` turns one sheet of a spreadsheet `ContentDocument` back into records via each cell's `displayText`. TSV is the same format with `{ delimiter: '\t' }` on either side. - **`src/layout/`** — the pure conversion algorithms: `engine.ts` (wordprocessing → layout: flow, line-breaking, pagination), `slides.ts` (presentation → layout: direct placement), `sheets.ts` (spreadsheet → layout: grid, print settings, the first algorithm accepting `AbortSignal`), `drawing.ts` (drawing → layout: vector primitives + shape reuse), `reconstruct.ts` (layout → content: baseline clustering for wordprocessing/presentation, near-1:1 mapping for drawing, gridline-lattice-or-text-clustering for spreadsheet). - **`src/hsqldb/`** — `.odb` decoders, four tiers: `script.ts` (TEXT-script DDL/DML parser), `rowformat.ts`/`cache.ts` (CACHED binary row-store), `binary-script.ts` (BINARY/COMPRESSED whole-script). All import only `document-schema.js` — no odf.js knowledge. - **`src/firebird/`** — Tier 3: gbak logical-backup reader. `reader.ts` (attribute framing + RLE decompression + XDR decoding), `schema.ts`/`data.ts` (table/row walking). No ratified spec — built against Firebird's own engine source. @@ -550,6 +554,7 @@ To run a single test file: `pnpm vitest run src/path/to/file.test.ts`. - **ODF text getters must call `decodeOdfText`.** See the dedicated gotcha above. - **`readPdf` recovers rect/ellipse/line as their own `LayoutRect`/`LayoutEllipse`/`LayoutLine` kinds** via pdf-codec's shape-pattern detection — an axis-aligned closed four-corner subpath is a rect, four kappa-ratio cubics at cardinal points is an ellipse, an open single straight stroke is a line. A false positive changes kind, never geometry. Off-axis rotations, freeform curves, and multi-subpath figures narrow to `LayoutPath`. - **`pdfToOds` re-types cells heuristically — this is probabilistic, not a fidelity guarantee.** A rendered PDF never carries a cell's typed value, only the printed string. Re-typing fires only where the string has exactly one defensible reading: the decimal must be exactly representable as a JS number; separators must be unambiguous (`"1,234"` is declined — competing European reading is 1.234); leading zeros decline (`"007"`); dates must self-state their component roles (ISO or named month accepted; `"01/02/2024"` declined). `TRUE`/`FALSE` re-type as booleans; `Yes`/`No` are declined. `displayText` always carries the rendered string verbatim. `onCellTypeInference` reports every decision. A formula is never claimed. +- **The csv read shares `pdfToOds`'s cell-typing heuristic, with the same decision-only audit channel.** The first record is a verbatim string header (never re-typed, even when it looks like data); data cells re-type through `inferCellValue` exactly as the PDF reconstructor does — declines keep the plain string, `displayText` always carries the raw field text, and `onCellTypeInference` fires per decision, staying silent for header cells and no-candidate text. The parser drops blank records, so a record of one empty field alone cannot round-trip. Writing csv takes exactly one sheet: a multi-sheet source refuses with `CsvSheetNotSpecifiedError` naming every sheet until `{ sheet }` selects one. TSV is not a separate format — `{ delimiter: '\t' }` on either side parses or writes the same grid. - **`reconstructWordprocessing`/`reconstructPresentation` recover vector primitives too**, in a nested drawing document — a rule under a heading, an underline, a cell background are all recovered as vectors (intended — discarding real content because it might be incidental is ruled out). A table's gridlines are excluded from vector recovery when the lattice claims them. - **Recovered vectors round-trip through all four readers** — `buildDocxPackage`/`buildPptxPackage` write real DrawingML; `buildOdtPackage`/`buildOdpPackage` write real `draw:rect`/`draw:ellipse`/`draw:line`/`draw:path`. The six PDF-bypassing bridges carry vector geometry across too. - **Each format wraps a vector shape differently.** OOXML: pptx gets a plain `p:sp`; docx gets a `w:drawing`/`wp:anchor` with `behindDoc="1"`/`wp:wrapNone` carrying a `wps:wsp`. ODF: odp appends to `draw:page`; odt anchors in a `text:p` with `style:horizontal-rel`/`style:vertical-rel="page"` (page-absolute coordinates) and `style:run-through="background"`. @@ -618,20 +623,21 @@ To run a single test file: `pnpm vitest run src/path/to/file.test.ts`. Read as **row → column**. `✓` lossless, `~` bounded, `✗` lossy, `✗✗` severe, `→` one-way, `–` no conversion. `.odm`/`.odb` sit outside this table. -| ↓ from \ to → | docx | pptx | xlsx | odt | odp | ods | odg | odf | markdown | pdf | -| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| **docx** | — | ~ | – | ✓ | – | – | – | – | ✗ | ~ | -| **pptx** | ~ | — | – | – | ✓ | – | – | – | – | ~ | -| **xlsx** | – | – | — | – | – | ~ | – | – | ✗✗ | ~ | -| **odt** | ✓ | – | – | — | ~ | – | – | – | ✗ | ~ | -| **odp** | – | ✓ | – | ~ | — | – | – | – | – | ~ | -| **ods** | – | – | ~ | – | – | — | – | – | – | ~ | -| **odg** | – | – | – | – | – | – | — | – | – | ~ | -| **odf** | – | – | – | – | – | – | – | — | – | → | -| **markdown** | ~ | – | ✗✗ | ~ | – | – | – | – | — | ~ | -| **pdf** | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | – | ✗✗ | — | - -73 of 90 directional pairs are routable. The `ContentDocument`/`LayoutDocument` pivots are the hub, not PDF — fourteen bridges bypass PDF entirely. +| ↓ from \ to → | docx | pptx | xlsx | odt | odp | ods | odg | odf | markdown | csv | pdf | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| **docx** | — | ~ | – | ✓ | – | – | – | – | ✗ | ✗ | ~ | +| **pptx** | ~ | — | – | – | ✓ | – | – | – | – | ✗ | ~ | +| **xlsx** | – | – | — | – | – | ~ | – | – | ✗✗ | ~ | ~ | +| **odt** | ✓ | – | – | — | ~ | – | – | – | ✗ | ✗ | ~ | +| **odp** | – | ✓ | – | ~ | — | – | – | – | – | ✗ | ~ | +| **ods** | – | – | ~ | – | – | — | – | – | – | ~ | ~ | +| **odg** | – | – | – | – | – | – | — | – | – | ✗ | ~ | +| **odf** | – | – | – | – | – | – | – | — | – | – | → | +| **markdown** | ~ | – | ✗✗ | ~ | – | – | – | – | — | ✗✗ | ~ | +| **csv** | ✗ | ✗ | ✓ | ✗ | ✗ | ✓ | ✗ | – | ✗✗ | — | ~ | +| **pdf** | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | – | ✗✗ | ✗ | — | + +91 of 110 directional pairs are routable. The `ContentDocument`/`LayoutDocument` pivots are the hub, not PDF — eighteen bridges bypass PDF entirely. **X → PDF** is a genuine layout render: positioned text, images, tables, lists, vector primitives, styled through the full cascade. It is a faithful visual approximation, not pixel-identical — closeness depends on font availability. @@ -643,9 +649,9 @@ Read as **row → column**. `✓` lossless, `~` bounded, `✗` lossy, `✗✗` s **PDF → ods** recovers what was printed, not what was entered. The printed string always survives in `displayText`; re-typed `value` is explicitly probabilistic inference. -**`markdownToPdf`/`pdfToMarkdown`** is the lossiest round trip: `markdownToPdf` is faithful, but `pdfToMarkdown` stacks reconstruction lossiness PLUS markdown's coarser vocabulary (no colour, font, size, alignment). +**`markdownToPdf`/`pdfToMarkdown`** is the lossiest round trip: `markdownToPdf` is faithful, but `pdfToMarkdown` stacks reconstruction lossiness PLUS markdown's coarser vocabulary (no colour, font, size, alignment). The PDF-composed markdown bridges (`xlsxToMarkdown`/`markdownToXlsx`, `csvToMarkdown`/`markdownToCsv`) stack the same two losses in both directions — hence their `✗✗` cells. -**The first three bridge pairs** (odt⇄docx, odp⇄pptx, ods⇄xlsx) bypass PDF entirely — no layout engine, no reconstruction. Text, styling, tables, lists, rotated shapes survive completely. `ods⇄xlsx` has small format-boundary limits (time cells, formula dialects). Embedded formulas survive `odtToDocx` as real OOXML math. +**The five same-variant bridge pairs** (odt⇄docx, odp⇄pptx, ods⇄xlsx, csv⇄ods, csv⇄xlsx) bypass PDF entirely — no layout engine, no reconstruction. Text, styling, tables, lists, rotated shapes survive completely. `ods⇄xlsx` has small format-boundary limits (time cells, formula dialects). Embedded formulas survive `odtToDocx` as real OOXML math. The csv pairs are bounded by what csv itself carries: toward ods/xlsx nothing the csv had is lost, while writing to csv collapses each cell to its `displayText` — formulas become their rendered values, formatting disappears, and a multi-sheet source must name the sheet it wants. **The two markdown bridge pairs** bypass PDF too, but markdown's grammar has no construct for colour/font/size/alignment — `docxToMarkdown`/`odtToMarkdown` drop them (format-boundary loss, not approximation). diff --git a/src/codecs/registry.test.ts b/src/codecs/registry.test.ts index d8bbfd546..db3c52369 100644 --- a/src/codecs/registry.test.ts +++ b/src/codecs/registry.test.ts @@ -16,7 +16,7 @@ import { DOCUMENT_FORMAT_CODECS } from './registry'; // Proves each DOCUMENT_FORMAT_CODECS entry's own read/write pair is wired correctly on its own terms -- not merely that readDocumentMetadata/setDocumentMetadata/buildDocumentBytes happen to still work after being refactored onto this registry (their own test files cover that). Every format with both a content.read and a content.write is exercised as a genuine read -> write -> read round trip: the content a fresh read produces after writing back out must equal the content that went in. -function requireContentCodec(format: 'docx' | 'pptx' | 'odt' | 'odp' | 'ods' | 'odg' | 'markdown' | 'xlsx') { +function requireContentCodec(format: 'docx' | 'pptx' | 'odt' | 'odp' | 'ods' | 'odg' | 'markdown' | 'xlsx' | 'csv') { const content = DOCUMENT_FORMAT_CODECS[format].content; if (!content?.write) { throw new Error(`expected DOCUMENT_FORMAT_CODECS.${format}.content.write to be defined`); @@ -101,6 +101,15 @@ describe('DOCUMENT_FORMAT_CODECS: content read/write round trips', () => { expect(codec.read(rebuiltBytes)).toEqual(content); }); + // csv's round trip is exact-equality like markdown's, for the same stability reason: write emits each cell's displayText, and read re-types that text heuristically -- but re-typing a cell that already went through inferCellValue once lands on the identical value again (a re-typed number prints back as the same digits, a declined string stays a string), so a second read cannot drift from the first. + it('csv: read -> write -> read round-trips the ContentDocument', () => { + const codec = requireContentCodec('csv'); + const csvBytes = new TextEncoder().encode('Name,Amount,Active\nWidget,42.5,TRUE\nGadget,7,\n'); + const content = codec.read(csvBytes); + const rebuiltBytes = codec.write!(content); + expect(codec.read(rebuiltBytes)).toEqual(content); + }); + // xlsx's own column-width unit conversion (ooxml.js's ptToColumnWidthChars/columnWidthCharsToPt, src/typed/xlsx/units.ts) is a best-effort algebraic inverse, not an exact one -- src/convert/bridges.test.ts's own COLUMN_WIDTH_TOLERANCE_PT documents up to ~1pt of drift per pt<->character-width hop. This registry round trip is a second such hop on top of whatever odsToXlsx's own bridge already introduced building the fixture, so widths are checked within tolerance rather than exact equality; every other field (sheet name, cell values/kinds/formula/merges) is checked exactly, since none of those go through a lossy unit conversion. it('xlsx: read -> write -> read carries sheet cell values, kinds, formulas, and merges through exactly, and column widths within tolerance', () => { const codec = requireContentCodec('xlsx'); @@ -161,3 +170,10 @@ describe('DOCUMENT_FORMAT_CODECS: xlsx has a content codec, no layout codec', () expect(DOCUMENT_FORMAT_CODECS.xlsx.layout).toBeUndefined(); }); }); + +describe('DOCUMENT_FORMAT_CODECS: csv has a content codec, no layout codec', () => { + it('csv has a content entry and no layout entry', () => { + expect(DOCUMENT_FORMAT_CODECS.csv.content).toBeDefined(); + expect(DOCUMENT_FORMAT_CODECS.csv.layout).toBeUndefined(); + }); +}); diff --git a/src/codecs/registry.ts b/src/codecs/registry.ts index 101c409b0..cb0907403 100644 --- a/src/codecs/registry.ts +++ b/src/codecs/registry.ts @@ -12,6 +12,9 @@ import { decodeMarkdownText, encodeMarkdownText } from '../markdown/text'; import type { MarkdownImageResolver } from 'markdown-codec'; import { readMarkdownContent } from '../markdown/read'; import { buildMarkdownText } from '../markdown/write'; +import { decodeCsvText, encodeCsvText } from '../csv/text'; +import { readCsvContent } from '../csv/read'; +import { buildCsvText } from '../csv/write'; import { readOdfFormulaContent } from '../odf/formula/read'; import { readOdgContent } from '../odf/odg/read'; import { readOdpContent } from '../odf/odp/read'; @@ -116,6 +119,13 @@ export const DOCUMENT_FORMAT_CODECS: Readonly encodeMarkdownText(buildMarkdownText(content)), }, }, + // The csv entry is the markdown entry's structural twin: decode straight from bytes to text (no package), read into a ContentDocument, write the reverse. DocumentCodecOptions carries no delimiter/sheet, so this codec reads and writes the default comma dialect over a lone sheet -- a caller wanting TSV output or a named sheet of a multi-sheet document uses the named conversions (convert.ts's xlsxToCsv/odsToCsv/pdfToCsv), which thread { delimiter, sheet } through UnifiedConversionOptions; a multi-sheet write through THIS codec throws buildCsvText's own CsvSheetNotSpecifiedError rather than silently truncating. decodeCsvText is a fatal decoder with no loop of its own, so no separate signal check is needed -- the same reasoning as the markdown entry beside it. + csv: { + content: { + read: (bytes) => readCsvContent(decodeCsvText(bytes)), + write: (content) => encodeCsvText(buildCsvText(content)), + }, + }, pdf: { layout: { read: (bytes, options) => readPdf(requireArrayBufferBytes(bytes), { signal: options?.signal }), diff --git a/src/convert/capability.test.ts b/src/convert/capability.test.ts index 0719c0a89..c125756b0 100644 --- a/src/convert/capability.test.ts +++ b/src/convert/capability.test.ts @@ -17,13 +17,15 @@ describe('FORMAT_CAPABILITIES', () => { expect(new Set(byVariant.get('wordprocessing'))).toEqual(new Set(['docx', 'odt', 'markdown'])); expect(new Set(byVariant.get('presentation'))).toEqual(new Set(['pptx', 'odp'])); - expect(new Set(byVariant.get('spreadsheet'))).toEqual(new Set(['xlsx', 'ods'])); + expect(new Set(byVariant.get('spreadsheet'))).toEqual(new Set(['xlsx', 'ods', 'csv'])); expect(new Set(byVariant.get('drawing'))).toEqual(new Set(['odg'])); }); - it('is the one node sharing a variant with a layout-path sibling but having no layout path of its own', () => { + it('marks xlsx and csv as the spreadsheet members with no layout path of their own (ods carries the layout edge)', () => { expect(FORMAT_CAPABILITIES.xlsx.variant).toBe('spreadsheet'); expect(FORMAT_CAPABILITIES.xlsx.hasLayoutPath).toBe(false); + expect(FORMAT_CAPABILITIES.csv.variant).toBe('spreadsheet'); + expect(FORMAT_CAPABILITIES.csv.hasLayoutPath).toBe(false); expect(FORMAT_CAPABILITIES.ods.variant).toBe('spreadsheet'); expect(FORMAT_CAPABILITIES.ods.hasLayoutPath).toBe(true); }); @@ -86,6 +88,28 @@ describe('resolveCompositionPlan', () => { expect(plan!.hops.map((h) => h.executor)).toEqual(['fromPdf', 'bridge']); }); + it('composes csv -> pdf through ods (bridge then toPdf), since csv has no layout engine of its own', () => { + const plan = resolveCompositionPlan('csv', 'pdf'); + expect(plan).toBeDefined(); + expect(plan!.hops.map((h) => h.executor)).toEqual(['bridge', 'toPdf']); + expect(plan!.hops[0]!.from).toBe('csv'); + expect(plan!.hops[0]!.to).toBe('ods'); + expect(plan!.hops[1]!.from).toBe('ods'); + expect(plan!.hops[1]!.to).toBe('pdf'); + }); + + it('composes pdf -> csv through ods (fromPdf then bridge)', () => { + const plan = resolveCompositionPlan('pdf', 'csv'); + expect(plan).toBeDefined(); + expect(plan!.hops.map((h) => h.executor)).toEqual(['fromPdf', 'bridge']); + }); + + it('composes csv -> markdown through ods and pdf (three hops), mirroring the xlsx -> markdown last-resort route', () => { + const plan = resolveCompositionPlan('csv', 'markdown'); + expect(plan).toBeDefined(); + expect(plan!.hops).toHaveLength(3); + }); + it('composes xlsx -> markdown through ods and pdf (three hops), the lossiest route in the package', () => { const plan = resolveCompositionPlan('xlsx', 'markdown'); expect(plan).toBeDefined(); diff --git a/src/convert/capability.ts b/src/convert/capability.ts index 422e03eb5..ff81a7f31 100644 --- a/src/convert/capability.ts +++ b/src/convert/capability.ts @@ -1,6 +1,6 @@ import type { DocumentFormat } from './port'; -// This module models the real ContentDocument-variant compatibility this family already has (wordprocessing = {docx, odt, markdown}, presentation = {pptx, odp}, spreadsheet = {xlsx, ods}, drawing = {odg alone}) plus which nodes have a direct layout-engine path to/from LayoutDocument (FORMAT_CAPABILITIES below). The composition engine (src/convert/composition.ts) consumes FORMAT_CAPABILITIES' variant declarations to build its composition graph, and UnsupportedConversionError is thrown by convertDocument/local.ts for any pair the pathfinder cannot route. The former DIRECT_EDGES list and resolveConversionPath resolver have been superseded by the pathfinder (resolveCompositionPlan in composition.ts), which derives every resolvable pair from the same registry data. +// This module models the real ContentDocument-variant compatibility this family already has (wordprocessing = {docx, odt, markdown}, presentation = {pptx, odp}, spreadsheet = {xlsx, ods, csv}, drawing = {odg alone}) plus which nodes have a direct layout-engine path to/from LayoutDocument (FORMAT_CAPABILITIES below). The composition engine (src/convert/composition.ts) consumes FORMAT_CAPABILITIES' variant declarations to build its composition graph, and UnsupportedConversionError is thrown by convertDocument/local.ts for any pair the pathfinder cannot route. The former DIRECT_EDGES list and resolveConversionPath resolver have been superseded by the pathfinder (resolveCompositionPlan in composition.ts), which derives every resolvable pair from the same registry data. // All five of document-schema.js's own ContentDocument kinds. 'formula' is a genuine member rather than a forward-looking one: readOdfFormulaContent produces a real `{kind:'formula', ...}` ContentDocument and odfToPdf consumes one, so `odf` below models it. Unlike the other four, it is a variant of exactly ONE format -- there is no second 'formula'-variant format to bridge it to, which is why a shared variant does not by itself imply a bridge edge exists. export type ContentVariant = 'wordprocessing' | 'presentation' | 'spreadsheet' | 'drawing' | 'formula'; @@ -21,6 +21,8 @@ export const FORMAT_CAPABILITIES: Readonly pdf through the ods bridge + ods's own layout engine rather than being a genuine ContentDocument -> LayoutDocument pipeline of xlsx's own. xlsx: { format: 'xlsx', variant: 'spreadsheet', hasLayoutPath: false }, + // csv shares the spreadsheet variant with xlsx/ods (readCsvContent/buildCsvText, src/csv/) and follows xlsx's routing exactly: plain text carries no layout of its own, so csv <-> pdf goes through the ods bridge + ods's layout engine. TSV is this same member with { delimiter: '\t' }, not a separate format -- see port.ts's own csv comment. + csv: { format: 'csv', variant: 'spreadsheet', hasLayoutPath: false }, odg: { format: 'odg', variant: 'drawing', hasLayoutPath: true }, // odf reads into the 'formula' ContentDocument variant (readOdfFormulaContent), but hasLayoutPath stays false: odfToPdf renders its formula through writePdf's own separate formula positioning rather than a ContentDocument -> LayoutDocument layout engine, and there is no reverse pdf -> odf at all (see odfToPdf's own module comment in convert.ts). odf: { format: 'odf', variant: 'formula', hasLayoutPath: false }, diff --git a/src/convert/codec.test.ts b/src/convert/codec.test.ts index c4ad3ba9a..15cc4aab8 100644 --- a/src/convert/codec.test.ts +++ b/src/convert/codec.test.ts @@ -1,6 +1,8 @@ import { decodePackage as decodeOoxmlPackage, readXlsxContent } from 'ooxml.js'; import { z } from 'zod'; import { describe, expect, it } from 'vitest'; +import { decodeCsvText, encodeCsvText } from '../csv/text'; +import { parseCsvRecords } from '../csv/records'; import { createDocx, openDocx } from '../edit/docx/editor'; import { openOdg } from '../edit/odg/editor'; import { openOdp } from '../edit/odp/editor'; @@ -14,7 +16,7 @@ import { minimalOdpBytes } from '../test-support/odp'; import { gridOdsBytes } from '../test-support/ods'; import { minimalOdtBytes } from '../test-support/odt'; import { odsToXlsx } from './convert'; -import { docxPdfCodec, markdownDocxCodec, markdownOdtCodec, markdownPdfCodec, odgPdfCodec, odpPdfCodec, odsPdfCodec, odtPdfCodec, pptxPdfCodec, xlsxPdfCodec } from './codec'; +import { csvMarkdownCodec, csvPdfCodec, docxPdfCodec, markdownDocxCodec, markdownOdtCodec, markdownPdfCodec, odgPdfCodec, odpPdfCodec, odsCsvCodec, odsPdfCodec, odtPdfCodec, pptxPdfCodec, xlsxCsvCodec, xlsxPdfCodec } from './codec'; function pdfHeader(bytes: Uint8Array): string { return new TextDecoder('latin1').decode(bytes.subarray(0, 5)); @@ -239,6 +241,99 @@ describe('markdownPdfCodec', () => { }); }); +// csvToPdf composes the csv -> ods bridge with ods -> pdf internally (csv has no layout engine of its own, exactly like xlsxPdfCodec above), and pdfToCsv composes pdf -> ods -> csv -- so this pair carries the same stacked-reconstruction caveat as xlsxPdfCodec, with csv read's heuristic re-typing on top. +describe('csvPdfCodec', () => { + it('z.decode produces valid PDF bytes from csv bytes', () => { + const pdfBytes = z.decode(csvPdfCodec, encodeCsvText('Name,Amount\nWidget,42.5\n')); + expect(pdfHeader(pdfBytes)).toBe('%PDF-'); + }); + + it('z.encode then z.decode round-trips text content, like csvToPdf/pdfToCsv', () => { + const pdfBytes = z.decode(csvPdfCodec, encodeCsvText('Name,Amount\nWidget,42.5\n')); + const csvBytes = z.encode(csvPdfCodec, pdfBytes); + const records = parseCsvRecords(decodeCsvText(csvBytes)); + expect(records[0]).toEqual(['Name', 'Amount']); + expect(records[1]?.[0]).toBe('Widget'); + }); + + it('rejects decode input with malformed UTF-8 before ever reaching csvToPdf', () => { + expect(() => z.decode(csvPdfCodec, new Uint8Array([0xff, 0xfe, 0x00]))).toThrow(z.core.$ZodError); + }); + + it('rejects encode input with no %PDF- header before ever reaching pdfToCsv', () => { + expect(() => z.encode(csvPdfCodec, new TextEncoder().encode('not a pdf'))).toThrow(z.core.$ZodError); + }); +}); + +// The same-variant spreadsheet bridges: direct ContentDocument pivot copies with no layout engine and no reconstruction, exactly like odsXlsxCodec. The csv boundary is displayText-only -- a typed ods/xlsx cell re-reads as whatever inferCellValue re-types its printed text as on the way back through the encode side. +describe('odsCsvCodec', () => { + it('z.decode produces csv text carrying every rendered cell of the ods fixture', () => { + const csvBytes = z.decode(odsCsvCodec, gridOdsBytes()); + const records = parseCsvRecords(decodeCsvText(csvBytes)); + expect(records[0]).toEqual(['Alpha', 'Beta', 'Gamma']); + expect(records[1]).toEqual(['One', 'Two', 'Three']); + }); + + it('z.encode then z.decode round-trips the parsed records, like csvToOds/odsToCsv', () => { + const odsBytes = z.encode(odsCsvCodec, encodeCsvText('Name,Amount\nWidget,42.5\n')); + const csvBytes = z.decode(odsCsvCodec, odsBytes); + expect(parseCsvRecords(decodeCsvText(csvBytes))).toEqual([['Name', 'Amount'], ['Widget', '42.5']]); + }); + + it('rejects decode input whose first zip entry is not a stored ods mimetype part before ever reaching odsToCsv', () => { + expect(() => z.decode(odsCsvCodec, new TextEncoder().encode('not an ods'))).toThrow(z.core.$ZodError); + }); + + it('rejects encode input with malformed UTF-8 before ever reaching csvToOds', () => { + expect(() => z.encode(odsCsvCodec, new Uint8Array([0xff, 0xfe, 0x00]))).toThrow(z.core.$ZodError); + }); +}); + +describe('xlsxCsvCodec', () => { + it('z.decode produces csv text carrying every rendered cell of the xlsx fixture', () => { + const csvBytes = z.decode(xlsxCsvCodec, odsToXlsx(gridOdsBytes())); + const records = parseCsvRecords(decodeCsvText(csvBytes)); + expect(records[0]).toEqual(['Alpha', 'Beta', 'Gamma']); + }); + + it('z.encode then z.decode round-trips the parsed records, like csvToXlsx/xlsxToCsv', () => { + const xlsxBytes = z.encode(xlsxCsvCodec, encodeCsvText('Name,Amount\nWidget,42.5\n')); + const csvBytes = z.decode(xlsxCsvCodec, xlsxBytes); + expect(parseCsvRecords(decodeCsvText(csvBytes))).toEqual([['Name', 'Amount'], ['Widget', '42.5']]); + }); + + it('rejects decode input with no ZIP local-file-header before ever reaching xlsxToCsv', () => { + expect(() => z.decode(xlsxCsvCodec, new TextEncoder().encode('not an xlsx'))).toThrow(z.core.$ZodError); + }); + + it('rejects encode input with malformed UTF-8 before ever reaching csvToXlsx', () => { + expect(() => z.encode(xlsxCsvCodec, new Uint8Array([0xff, 0xfe, 0x00]))).toThrow(z.core.$ZodError); + }); +}); + +// The pdf-composed last-resort pair, mirroring xlsxMarkdownCodec's own shape: neither direction is remotely round-trip-lossless (spreadsheet render stacked on markdown reconstruction and back), so the assertions are structural -- valid output of the target schema carrying real text on the decode side. +describe('csvMarkdownCodec', () => { + it('z.decode produces markdown carrying the rendered cell text, like csvToMarkdown', () => { + const markdownBytes = z.decode(csvMarkdownCodec, encodeCsvText('Name,Amount\nWidget,42.5\n')); + const text = decodeMarkdownText(markdownBytes); + expect(text).toContain('Name'); + expect(text).toContain('Widget'); + }); + + it('z.encode produces csv bytes that parse as well-formed RFC 4180 records, like markdownToCsv', () => { + const csvBytes = z.encode(csvMarkdownCodec, encodeMarkdownText('| A | B |\n| --- | --- |\n| one | two |\n')); + expect(parseCsvRecords(decodeCsvText(csvBytes)).length).toBeGreaterThan(0); + }); + + it('rejects decode input with malformed UTF-8 before ever reaching csvToMarkdown', () => { + expect(() => z.decode(csvMarkdownCodec, new Uint8Array([0xff, 0xfe, 0x00]))).toThrow(z.core.$ZodError); + }); + + it('rejects encode input with malformed UTF-8 before ever reaching markdownToCsv', () => { + expect(() => z.encode(csvMarkdownCodec, new Uint8Array([0xff, 0xfe, 0x00]))).toThrow(z.core.$ZodError); + }); +}); + describe('markdownDocxCodec', () => { it('z.decode produces valid docx bytes from markdown bytes', () => { const docxBytes = z.decode(markdownDocxCodec, encodeMarkdownText(richMarkdownText())); diff --git a/src/convert/codec.ts b/src/convert/codec.ts index b90ecc306..9e6f40892 100644 --- a/src/convert/codec.ts +++ b/src/convert/codec.ts @@ -1,20 +1,27 @@ import { z } from 'zod'; -import { DocxBytesSchema, MarkdownBytesSchema, OdgBytesSchema, OdpBytesSchema, OdsBytesSchema, OdtBytesSchema, PdfBytesSchema, PptxBytesSchema, XlsxBytesSchema } from '../model/bytes'; +import { CsvBytesSchema, DocxBytesSchema, MarkdownBytesSchema, OdgBytesSchema, OdpBytesSchema, OdsBytesSchema, OdtBytesSchema, PdfBytesSchema, PptxBytesSchema, XlsxBytesSchema } from '../model/bytes'; import { + csvToMarkdown, + csvToOds, + csvToPdf, + csvToXlsx, docxToMarkdown, docxToOdt, docxToPdf, + markdownToCsv, markdownToDocx, markdownToOdt, markdownToPdf, odgToPdf, odpToPdf, odpToPptx, + odsToCsv, odsToPdf, odsToXlsx, odtToDocx, odtToMarkdown, odtToPdf, + pdfToCsv, pdfToDocx, pdfToMarkdown, pdfToOdg, @@ -25,6 +32,7 @@ import { pdfToXlsx, pptxToOdp, pptxToPdf, + xlsxToCsv, xlsxToOds, xlsxToPdf, xlsxToMarkdown, @@ -105,3 +113,26 @@ export const xlsxMarkdownCodec = z.codec(XlsxBytesSchema, MarkdownBytesSchema, { decode: (xlsxBytes) => xlsxToMarkdown(xlsxBytes), encode: (markdownBytes) => markdownToXlsx(markdownBytes), }); + +// csv bytes <-> PDF bytes: the no-options form over csvToPdf/pdfToCsv (convert.ts), schema-validated both ways. csvToPdf composes the csv -> ods bridge with ods -> pdf internally (csv has no layout engine of its own, exactly like xlsx), so this pair carries the same "not round-trip-lossless" caveat every PDF-pivot codec above carries, with csv's own additional boundaries on both sides: decode re-types the parsed cells heuristically (inferCellValue), encode reconstructs from geometry. +export const csvPdfCodec = z.codec(CsvBytesSchema, PdfBytesSchema, { + decode: (csvBytes) => csvToPdf(csvBytes), + encode: (pdfBytes) => pdfToCsv(pdfBytes), +}); + +// ods bytes <-> csv bytes and xlsx bytes <-> csv bytes: schema-validated z.codec() pairs over the same-variant spreadsheet bridges (convert.ts) -- direct ContentDocument pivot copies with no layout engine and no reconstruction, exactly like odsXlsxCodec above. Two honest csv-boundary caveats rather than codec lossiness: the csv side is displayText-only, so a typed ods/xlsx cell (currency, percentage, date) re-reads as whatever inferCellValue re-types its printed text as on the way back; and a multi-sheet ods/xlsx source throws CsvSheetNotSpecifiedError from the no-options build, naming every sheet, rather than silently truncating -- the identical contract the registry codec (src/codecs/registry.ts) documents for its own csv write. +export const odsCsvCodec = z.codec(OdsBytesSchema, CsvBytesSchema, { + decode: (odsBytes) => odsToCsv(odsBytes), + encode: (csvBytes) => csvToOds(csvBytes), +}); + +export const xlsxCsvCodec = z.codec(XlsxBytesSchema, CsvBytesSchema, { + decode: (xlsxBytes) => xlsxToCsv(xlsxBytes), + encode: (csvBytes) => csvToXlsx(csvBytes), +}); + +// csv bytes <-> markdown bytes: the no-options form over the two pdf-composed bridge functions (convert.ts), routing csv -> ods -> pdf -> markdown and markdown -> pdf -> ods -> csv. Lossy in the same stacked way as xlsxMarkdownCodec above -- the spreadsheet render and the markdown reconstruction each add their own loss -- with csv read's heuristic re-typing on top on the decode side. +export const csvMarkdownCodec = z.codec(CsvBytesSchema, MarkdownBytesSchema, { + decode: (csvBytes) => csvToMarkdown(csvBytes), + encode: (markdownBytes) => markdownToCsv(markdownBytes), +}); diff --git a/src/convert/composition-plans.test.ts b/src/convert/composition-plans.test.ts index 444dc315f..aa1153b6a 100644 --- a/src/convert/composition-plans.test.ts +++ b/src/convert/composition-plans.test.ts @@ -32,6 +32,7 @@ describe('resolveCompositionPlan route verification', () => { const sameVariant: [DocumentFormat, DocumentFormat][] = [ ['docx', 'odt'], ['odt', 'docx'], ['docx', 'markdown'], ['odt', 'markdown'], ['markdown', 'docx'], ['markdown', 'odt'], + ['csv', 'ods'], ['ods', 'csv'], ['csv', 'xlsx'], ['xlsx', 'csv'], ]; for (const [s, t] of sameVariant) { const plan = resolveCompositionPlan(s, t)!; @@ -61,6 +62,13 @@ describe('resolveCompositionPlan route verification', () => { expect(pdfToXlsx.hops.map((h) => h.executor)).toEqual(['fromPdf', 'bridge']); }); + it('csv <-> pdf composes through ods (2 hops) exactly like xlsx, since csv has no layout engine of its own either', () => { + const csvToPdf = resolveCompositionPlan('csv', 'pdf')!; + expect(csvToPdf.hops.map((h) => h.executor)).toEqual(['bridge', 'toPdf']); + const pdfToCsv = resolveCompositionPlan('pdf', 'csv')!; + expect(pdfToCsv.hops.map((h) => h.executor)).toEqual(['fromPdf', 'bridge']); + }); + it('xlsx -> markdown composes through ods and pdf (3 hops)', () => { const plan = resolveCompositionPlan('xlsx', 'markdown')!; expect(plan.hops).toHaveLength(3); diff --git a/src/convert/composition.ts b/src/convert/composition.ts index 572c62b34..6a2972597 100644 --- a/src/convert/composition.ts +++ b/src/convert/composition.ts @@ -23,6 +23,10 @@ import { readOdtContent } from '../odf/odt/read'; import { buildMarkdownText } from '../markdown/write'; import { decodeMarkdownText, encodeMarkdownText } from '../markdown/text'; import { readMarkdownContent } from '../markdown/read'; +import { decodeCsvText, encodeCsvText } from '../csv/text'; +import { readCsvContent } from '../csv/read'; +import { buildCsvText } from '../csv/write'; +import type { CellTypeInferenceSink } from '../layout/cell-typing'; import { convertDrawingToLayout } from '../layout/drawing'; import { convertWordprocessingToLayout } from '../layout/engine'; import { reconstructDrawing, reconstructPresentation, reconstructSpreadsheet, reconstructWordprocessing, type ReconstructOptions } from '../layout/reconstruct'; @@ -53,21 +57,25 @@ export interface UnifiedConversionOptions { readonly sink?: PdfDiagnosticSink; readonly onMathDiagnostic?: (diagnostic: OmmlDiagnostic, context: { readonly sourcePath?: string }) => void; readonly images?: MarkdownImageResolver; + // The csv hops' own knobs, on the shared shape for the same reason onMathDiagnostic (docx/pptx only) and images (markdown only) already sit here: the pathfinder picks the hops, so per-format options must ride the one options object every hop receives, and each hop's registry closure reads only the fields it knows -- csv read consumes delimiter/onCellTypeInference, csv build consumes delimiter/sheet, every non-csv hop ignores all three. + readonly delimiter?: string; + readonly sheet?: string; + readonly onCellTypeInference?: CellTypeInferenceSink; readonly clock?: ClockPort; } // --- Registry: declarative per-format primitive wiring ----------------------------------------- -// The eight content formats this engine routes between (pdf is the layout pivot, reached via toPdf/fromPdf edges; odf is special, excluded entirely -- see the module doc). -export type ContentFormat = 'docx' | 'pptx' | 'xlsx' | 'odt' | 'odp' | 'ods' | 'odg' | 'markdown'; +// The nine content formats this engine routes between (pdf is the layout pivot, reached via toPdf/fromPdf edges; odf is special, excluded entirely -- see the module doc). +export type ContentFormat = 'docx' | 'pptx' | 'xlsx' | 'odt' | 'odp' | 'ods' | 'odg' | 'csv' | 'markdown'; // The explicit, typed list of content formats, kept in sync with FORMAT_NODES' own keys. Used for iteration in the graph builder in place of `Object.keys(FORMAT_NODES)` (which returns `string[]` and would need a cast back to ContentFormat), so the registry stays cast-free end to end. -const CONTENT_FORMATS: readonly ContentFormat[] = ['docx', 'pptx', 'xlsx', 'odt', 'odp', 'ods', 'odg', 'markdown']; +const CONTENT_FORMATS: readonly ContentFormat[] = ['docx', 'pptx', 'xlsx', 'odt', 'odp', 'ods', 'odg', 'csv', 'markdown']; // The four ContentDocument variants a layout engine exists for. 'formula' is the fifth ContentVariant member but has no layout engine of its own (odfToPdf renders through writePdf's formula positioning, not a ContentDocument -> LayoutDocument pass), so it is excluded from this engine's layout/reconstruct registries. type LayoutVariant = Exclude; -// Every content format's node in the composition graph: the decode -> read -> build -> encode primitive chain, plus the ContentDocument variant every format reads into and builds from. A discriminated union on `family` keeps the package (SourcePackage) and markdown (string) halves' decode/read/build/encode signatures concrete and cast-free: the executors narrow on `family === 'markdown'` to select the right shape. hasSourcePackage drives the font-registry choice in executeToPdf (createDocumentFontRegistry for a package, createFontRegistry for markdown text), mirroring markdownToPdf's own documented divergence from docxToPdf. +// Every content format's node in the composition graph: the decode -> read -> build -> encode primitive chain, plus the ContentDocument variant every format reads into and builds from. A discriminated union keeps the package (SourcePackage) and plain-text (string) halves' decode/read/build/encode signatures concrete and cast-free: the executors narrow through isTextFormatNode (below) to select the right shape. hasSourcePackage is the boolean-literal discriminant that split rests on -- and it also drives the font-registry choice in executeToPdf (createDocumentFontRegistry for a package, createFontRegistry for text), mirroring markdownToPdf's own documented divergence from docxToPdf. interface PackageFormatNode { readonly variant: LayoutVariant; readonly family: 'ooxml' | 'odf'; @@ -78,19 +86,25 @@ interface PackageFormatNode { readonly hasSourcePackage: true; } -interface MarkdownFormatNode { +// The plain-text half of the union: markdown and csv both decode straight from bytes to a string and read/build through their own text-level codecs -- no zip package, no font embedding, no source-package concept at all. family names the text dialect so a format can never be a member of both halves. build takes options because csv's build consumes { delimiter, sheet } from UnifiedConversionOptions; markdown's build ignores them. +interface TextFormatNode { readonly variant: LayoutVariant; - readonly family: 'markdown'; + readonly family: 'markdown' | 'csv'; readonly decode: (bytes: Uint8Array) => string; readonly read: (text: string, options?: UnifiedConversionOptions) => ContentDocument; - readonly build: (content: ContentDocument) => string; + readonly build: (content: ContentDocument, options?: UnifiedConversionOptions) => string; readonly encode: (text: string) => Uint8Array; readonly hasSourcePackage: false; } -export type FormatNode = PackageFormatNode | MarkdownFormatNode; +export type FormatNode = PackageFormatNode | TextFormatNode; -// The single source of truth for "which primitives does each format use". read/build closures thread their own per-format option subset internally: docx and pptx read/build both pull onMathDiagnostic (mirroring readDocxContent's/readPptxContent's own `{ onMathDiagnostic }` and buildDocxPackage's/buildPptxPackage's own option -- ExaDev/documents.js#563 gave pptx the identical OMML degrade-diagnostic channel docx already had), markdown read pulls signal/images (mirroring readMarkdownContent's ReadMarkdownOptions), and every other format's read/build accept and ignore the thread. docxToPdf's openDocx(bytes).toPackage() and decodeOoxmlPackage(bytes) produce the identical Package (openDocx wraps decodeOoxmlPackage and toPackage returns it unmutated), so decode uses the package codec directly for uniformity -- byte-identical to docxToPdf at every downstream call site. +// Narrowing on the boolean-literal hasSourcePackage discriminant (not on family), so the text half stays open to further plain-text families without touching any executor: TypeScript narrows a discriminated union on literal true/false just as it does on string literals. +function isTextFormatNode(node: FormatNode): node is TextFormatNode { + return !node.hasSourcePackage; +} + +// The single source of truth for "which primitives does each format use". read/build closures thread their own per-format option subset internally: docx and pptx read/build both pull onMathDiagnostic (mirroring readDocxContent's/readPptxContent's own `{ onMathDiagnostic }` and buildDocxPackage's/buildPptxPackage's own option -- ExaDev/documents.js#563 gave pptx the identical OMML degrade-diagnostic channel docx already had), markdown read pulls signal/images (mirroring readMarkdownContent's ReadMarkdownOptions), csv read pulls delimiter/onCellTypeInference and csv build pulls delimiter/sheet (mirroring readCsvContent's ReadCsvContentOptions and buildCsvText's BuildCsvTextOptions), and every other format's read/build accept and ignore the thread. docxToPdf's openDocx(bytes).toPackage() and decodeOoxmlPackage(bytes) produce the identical Package (openDocx wraps decodeOoxmlPackage and toPackage returns it unmutated), so decode uses the package codec directly for uniformity -- byte-identical to docxToPdf at every downstream call site. export const FORMAT_NODES: Readonly> = { docx: { variant: 'wordprocessing', @@ -164,9 +178,18 @@ export const FORMAT_NODES: Readonly> = { encode: (text) => encodeMarkdownText(text), hasSourcePackage: false, }, + csv: { + variant: 'spreadsheet', + family: 'csv', + decode: (bytes) => decodeCsvText(bytes), + read: (text, options) => readCsvContent(text, { delimiter: options?.delimiter, onCellTypeInference: options?.onCellTypeInference }), + build: (content, options) => buildCsvText(content, { delimiter: options?.delimiter, sheet: options?.sheet }), + encode: (text) => encodeCsvText(text), + hasSourcePackage: false, + }, }; -// The formats that have a direct layout-engine path to/from PDF (convertXToLayout + writePdf). xlsx is deliberately absent: it has no layout engine of its own, so the pathfinder routes xlsx <-> pdf through ods instead (xlsx -> ods bridge, then ods -> pdf toPdf), reproducing the composed route xlsxToPdf/pdfToXlsx already hard-code in convert.ts. +// The formats that have a direct layout-engine path to/from PDF (convertXToLayout + writePdf). xlsx and csv are deliberately absent: neither has a layout engine of its own, so the pathfinder routes each <-> pdf through ods instead (e.g. csv -> ods bridge, then ods -> pdf toPdf), reproducing the composed route xlsxToPdf/pdfToXlsx already hard-code in convert.ts. const LAYOUT_CAPABLE: ReadonlySet = new Set(['docx', 'pptx', 'odt', 'odp', 'ods', 'odg', 'markdown']); // Cross-variant transforms keyed by `${fromVariant}->${toVariant}`. Each wrapper narrows its input with a runtime kind guard so the underlying transform receives its exact concrete variant type -- the same "no cast, narrow at the boundary" discipline every read/build closure above follows. Today wordprocessing <-> presentation and drawing <-> presentation transforms exist (src/convert/variant-bridges.ts); the pathfinder derives its cross-variant edges from this object's keys, so adding a transform here is the single change needed to teach both the pathfinder and the bridge executor a new variant crossing. @@ -220,9 +243,9 @@ function executeBridge(source: ContentFormat, target: ContentFormat, bytes: Uint const sourceNode = FORMAT_NODES[source]; const targetNode = FORMAT_NODES[target]; - // Decode + read the source, branching on family so the package (SourcePackage) and markdown (string) decoded shapes stay concrete. + // Decode + read the source, branching on hasSourcePackage so the package (SourcePackage) and text (string) decoded shapes stay concrete. let content: ContentDocument; - if (sourceNode.family === 'markdown') { + if (isTextFormatNode(sourceNode)) { const text = sourceNode.decode(bytes); content = sourceNode.read(text, options); } else { @@ -248,8 +271,8 @@ function executeBridge(source: ContentFormat, target: ContentFormat, bytes: Uint options?.onDocument?.({ formatVersion: DOCUMENT_PACKAGE_FORMAT_VERSION, content: buildContent }); // Build + encode the target. A bridge never runs a layout engine, so the reported DocumentPackage carries content only, with no pages array and no node frames -- the identical layoutless shape convert.ts's own bridges report. - if (targetNode.family === 'markdown') { - const text = targetNode.build(buildContent); + if (isTextFormatNode(targetNode)) { + const text = targetNode.build(buildContent, options); return targetNode.encode(text); } const pkg = targetNode.build(buildContent, options); @@ -265,7 +288,7 @@ function executeToPdf(format: ContentFormat, bytes: Uint8Array, opt let content: ContentDocument; let fonts: FontRegistry; - if (node.family === 'markdown') { + if (isTextFormatNode(node)) { throwIfAborted(options?.signal); const text = node.decode(bytes); const read = node.read(text, options); @@ -325,17 +348,17 @@ function executeToPdf(format: ContentFormat, bytes: Uint8Array, opt return writePdf(layout, { signal: options?.signal, onSubstitution: options?.onSubstitution, formulas, fonts }); } -// readPdf -> reconstruct(target variant) -> build(target) -> encode(target), reproducing the exact sequence and option-threading of convert.ts's pdfTo* functions (pdfToDocx/pdfToOdt/pdfToOdp/pdfToOds/pdfToOdg/pdfToMarkdown). sink reaches readPdf; signal reaches both readPdf and the reconstructor. The reconstructor's onCellTypeInference (reconstructSpreadsheet's audit channel) is deliberately left unset -- UnifiedConversionOptions does not carry it today, matching every pdfToOds caller in convert.ts. build is called with no options, matching pdfTo*'s own `buildXPackage(content)` calls (no clock, no onMathDiagnostic threaded on this direction). +// readPdf -> reconstruct(target variant) -> build(target) -> encode(target), reproducing the exact sequence and option-threading of convert.ts's pdfTo* functions (pdfToDocx/pdfToOdt/pdfToOdp/pdfToOds/pdfToOdg/pdfToMarkdown). sink reaches readPdf; signal reaches both readPdf and the reconstructor. onCellTypeInference reaches the reconstructor (reconstructSpreadsheet's audit channel) AND csv's read -- the two places a cell re-typing decision can happen, threaded on the one options object every hop receives; the ergonomic pdfTo* functions do not declare it, so a caller wanting that audit channel on a pdf -> spreadsheet route passes it to convertDocument directly (the first-class entry point every named function forwards to). The package build half is called with no options, matching pdfTo*'s own `buildXPackage(content)` calls (no clock, no onMathDiagnostic threaded on this direction); the text build half receives options because csv's build consumes { delimiter, sheet }. function executeFromPdf(target: ContentFormat, bytes: Uint8Array, options?: UnifiedConversionOptions): Uint8Array { const node = FORMAT_NODES[target]; const layout = readPdf(bytes, { signal: options?.signal, sink: options?.sink }); - const content = RECONSTRUCTORS[node.variant](layout, { signal: options?.signal }); + const content = RECONSTRUCTORS[node.variant](layout, { signal: options?.signal, onCellTypeInference: options?.onCellTypeInference }); // The pages half derives from the read LayoutDocument's own pages -- every rendered page's size, indexed to match the frames the reconstructor attached to the content it built. const pages = layout.pages.map((page) => ({ widthPt: page.widthPt, heightPt: page.heightPt })); options?.onDocument?.({ formatVersion: DOCUMENT_PACKAGE_FORMAT_VERSION, content, pages }); - if (node.family === 'markdown') { - const text = node.build(content); + if (isTextFormatNode(node)) { + const text = node.build(content, options); return node.encode(text); } const pkg = node.build(content); @@ -361,7 +384,7 @@ interface GraphEdge { readonly cost: number; } -// Builds the composition graph's adjacency list from the registry, with fidelity-ordered edge costs: a same-variant bridge (cost 1, lossless) always beats a cross-variant transform (cost 2, approximate), which always beats a toPdf/fromPdf edge (cost 3, geometry-based render or reconstruction). Edges are bidirectional with symmetric costs. The toPdf/fromPdf edges cover exactly LAYOUT_CAPABLE (xlsx absent -- it routes through ods), and cross-variant transform edges are derived from TRANSFORMS' own keys so the graph cannot drift from the registered transforms. +// Builds the composition graph's adjacency list from the registry, with fidelity-ordered edge costs: a same-variant bridge (cost 1, lossless) always beats a cross-variant transform (cost 2, approximate), which always beats a toPdf/fromPdf edge (cost 3, geometry-based render or reconstruction). Edges are bidirectional with symmetric costs. The toPdf/fromPdf edges cover exactly LAYOUT_CAPABLE (xlsx and csv absent -- each routes through ods), and cross-variant transform edges are derived from TRANSFORMS' own keys so the graph cannot drift from the registered transforms. function buildCompositionGraph(): ReadonlyMap { const adj = new Map(); const addDirected = (from: DocumentFormat, to: DocumentFormat, cost: number): void => { @@ -410,7 +433,7 @@ function buildCompositionGraph(): ReadonlyMap = buildCompositionGraph(); -// Standard Dijkstra over the small (<= 9-node) composition graph. Returns the ordered node path from source to target, or undefined if source === target or target is unreachable. +// Standard Dijkstra over the small (<= 10-node) composition graph. Returns the ordered node path from source to target, or undefined if source === target or target is unreachable. function shortestPath(source: DocumentFormat, target: DocumentFormat): readonly DocumentFormat[] | undefined { if (source === target) { return undefined; @@ -490,7 +513,7 @@ export function resolveCompositionPlan(source: DocumentFormat, target: DocumentF return { hops }; } -// Narrows a DocumentFormat to the ContentFormat union (the eight formats with a FORMAT_NODES entry). pdf and odf are excluded: pdf is the layout pivot reached only via toPdf/fromPdf edges, and odf is the special-case format this engine does not route at all. Used by convertDocument to narrow a hop's DocumentFormat endpoints to the ContentFormat the executors are typed against. +// Narrows a DocumentFormat to the ContentFormat union (the nine formats with a FORMAT_NODES entry). pdf and odf are excluded: pdf is the layout pivot reached only via toPdf/fromPdf edges, and odf is the special-case format this engine does not route at all. Used by convertDocument to narrow a hop's DocumentFormat endpoints to the ContentFormat the executors are typed against. function isContentFormat(format: DocumentFormat): format is ContentFormat { return format !== 'pdf' && format !== 'odf'; } diff --git a/src/convert/convert.ts b/src/convert/convert.ts index 4b02cafb4..d10c5af85 100644 --- a/src/convert/convert.ts +++ b/src/convert/convert.ts @@ -30,8 +30,9 @@ import { throwIfAborted } from '../ports/abort'; import { resolveMetadataTimestamps } from '../model/metadata'; import type { ClockPort } from '../ports/clock'; import { convertDocument } from './composition'; +import type { CellTypeInferenceSink } from '../layout/cell-typing'; -// Twelve ergonomic conversions (docx/pptx/odt/odp/ods/odg <-> PDF, all now round-trip both ways). Each is now a thin forwarder to convertDocument (src/convert/composition.ts), which resolves the composition plan for the pair and runs the real decode/read/layout/build/encode primitives -- reproducing the exact sequence and option-threading the hand-written body below used to inline. The four special cases further down this file (odfToPdf, odmToPdf, odbToXlsx, odbToCsv) stay as their real hand-written bodies, since none is a composition-graph pair (formula one-way, resolver callback, table extraction). +// The ergonomic X <-> PDF conversions (docx/pptx/odt/odp/ods/odg/markdown/csv, all round-trip both ways). Each is a thin forwarder to convertDocument (src/convert/composition.ts), which resolves the composition plan for the pair and runs the real decode/read/layout/build/encode primitives -- reproducing the exact sequence and option-threading the hand-written body below used to inline. The four special cases further down this file (odfToPdf, odmToPdf, odbToXlsx, odbToCsv) stay as their real hand-written bodies, since none is a composition-graph pair (formula one-way, resolver callback, table extraction). // `fonts` and `onFontSubstitution` are inherited from DocumentFontRegistryOptions (src/fonts/registry.ts) rather than redeclared here, so the ergonomic conversions and createDocumentFontRegistry describe the same two options in exactly one place. Every X-to-PDF conversion below builds a real FontRegistry from the SOURCE PACKAGE'S OWN embedded faces first, then those caller-supplied faces, then pdf-codec's vendored Carlito/Caladea substitutes, then the standard 14 -- which is why a document that embeds nothing, and asks for no family a vendored substitute covers, still writes byte-identical output to the standard-font-only pipeline this package had before (see convert-fonts.test.ts's own byte-identity proof). export interface DocumentToPdfOptions extends DocumentFontRegistryOptions { @@ -83,6 +84,26 @@ export function markdownToPdf(bytes: Uint8Array, options?: Document return convertDocument('markdown', 'pdf', bytes, options); } +// The two csv option groups the named csv conversions below intersect into the shared options type each already uses, rather than a csv-specific options type per function: every csv-SOURCED conversion parses with { delimiter, onCellTypeInference } (readCsvContent's own two options, src/csv/read.ts) and every csv-TARGET one writes with { delimiter, sheet } (buildCsvText's own two, src/csv/write.ts) -- exactly the fields UnifiedConversionOptions (src/convert/composition.ts) threads to the csv node's read/build legs. Declaring the two groups once here keeps each ergonomic signature honest about which csv knobs that particular function consumes without duplicating the field comments eight times over. +export interface CsvReadOptions { + // The field delimiter to parse with -- ',' (RFC 4180's own default, and readCsvContent's own default when omitted) or '\t' for TSV. TSV is deliberately a delimiter choice on the SAME 'csv' format rather than a second DocumentFormat member, since a delimiter is a parse option, not a different document format (see port.ts's own csv comment). + readonly delimiter?: string; + // Called once per data cell whose text inferCellValue re-typed (or considered and declined) while the csv was being read into spreadsheet cells -- the audit channel for the one lossy step a csv read performs, reported at the read boundary where the re-typing actually happens. + readonly onCellTypeInference?: CellTypeInferenceSink; +} + +export interface CsvWriteOptions { + // The field delimiter to write with -- ',' by default (RFC 4180), '\t' for TSV output. Feeds quoteCsvField's own quoting decision too, so a field containing the delimiter is quoted and a field containing a comma under a tab delimiter is not. + readonly delimiter?: string; + // Selects which sheet of a multi-sheet spreadsheet document to write as csv. Required whenever the document has more than one sheet -- omitting it then throws CsvSheetNotSpecifiedError naming every available sheet, rather than guessing (the identical contract odbToCsv's own `table` option follows for a multi-table .odb). May be omitted when the document has exactly one sheet. + readonly sheet?: string; +} + +// csv bytes -> PDF bytes: csv has no layout engine of its own (exactly like xlsx), so convertDocument's pathfinder resolves this as [csv -> ods bridge, ods -> pdf toPdf] -- the reader parses RFC 4180 text into a spreadsheet ContentDocument (first record as the header row, data cells re-typed by inferCellValue with onCellTypeInference auditing each decision), then ods's own layout engine renders it. `onDocument` reports the last hop's package under the composition engine's own "fires exactly once, on the last hop" convention: the odsToPdf hop's content+layout. +export function csvToPdf(bytes: Uint8Array, options?: DocumentToPdfOptions & CsvReadOptions): Uint8Array { + return convertDocument('csv', 'pdf', bytes, options); +} + const STANDALONE_FORMULA_SIZE_PT = 18; // larger than a typical embedded formula (see engine.ts's own formulaSizePtForFrame), since a standalone .odf's own formula is usually the whole document's content, not a small inline element. const STANDALONE_FORMULA_MARGIN_PT = 72; // 1 inch @@ -159,7 +180,12 @@ export function pdfToMarkdown(bytes: Uint8Array, options?: PdfToDoc return convertDocument('pdf', 'markdown', bytes, options); } -// Ten cross-format bridges, five pairs (odt<->docx, odp<->pptx, ods<->xlsx, and -- further down this section -- markdown<->docx, markdown<->odt), each bypassing PDF entirely. Every conversion above this point pivots through a LayoutDocument; these five pairs don't have that problem: both formats in each pair already read into and build from the identical ContentDocument variant, so the bridge is nothing more than reader -> writer, with no layout engine, no font measurement, and no geometry-based reconstruction in between. Each forwarder below hands the pair to convertDocument (src/convert/composition.ts), whose pathfinder resolves it as a single same-variant bridge hop and runs the identical decode/read/build/encode sequence. +// pdf bytes -> csv bytes: the reverse of csvToPdf above -- convertDocument's pathfinder resolves this as [pdf -> ods fromPdf, ods -> csv bridge], reconstructing the pdf's tabular layout into a spreadsheet ContentDocument and then writing its lone or selected sheet as RFC 4180 text. `onDocument` reports the last hop's package under the composition engine's own "fires exactly once, on the last hop" convention: the odsToCsv bridge hop's content-only package. +export function pdfToCsv(bytes: Uint8Array, options?: PdfToDocumentOptions & CsvWriteOptions): Uint8Array { + return convertDocument('pdf', 'csv', bytes, options); +} + +// The same-variant cross-format bridges (odt<->docx, odp<->pptx, ods<->xlsx, csv<->ods, csv<->xlsx, and -- further down this section -- markdown<->docx, markdown<->odt), each bypassing PDF entirely. Every conversion above this point pivots through a LayoutDocument; these pairs don't have that problem: both formats in each pair already read into and build from the identical ContentDocument variant, so the bridge is nothing more than reader -> writer, with no layout engine, no font measurement, and no geometry-based reconstruction in between. Each forwarder below hands the pair to convertDocument (src/convert/composition.ts), whose pathfinder resolves it as a single same-variant bridge hop and runs the identical decode/read/build/encode sequence. export interface DocumentBridgeOptions { readonly signal?: AbortSignal; // Called exactly once, synchronously, with the DocumentPackage this bridge built internally, before the function returns its bytes -- mirroring DocumentToPdfOptions/PdfToDocumentOptions's own onDocument. A bridge never runs a layout engine (see this section's own top-of-block comment), so `layout` is always left undefined here -- DocumentPackageSchema already models layout as optional for exactly this case, and running a layout conversion purely to populate a field no caller asked for would be wasted work. @@ -170,7 +196,7 @@ export interface DocumentBridgeOptions { readonly images?: MarkdownImageResolver; } -// Options for a composed edge whose two formats share no ContentDocument variant, so the only route is through PDF (today: xlsx <-> markdown). These are 'bridge' hops from the composition engine's point of view (neither endpoint is pdf), but internally they compose a toPdf leg -- which lays content out, so fonts/onFontSubstitution/onSubstitution/clock reach it -- with a fromPdf leg, which reconstructs, so sink reaches it. That is a wider shape than the PDF-bypassing DocumentBridgeOptions above, and every field is optional deliberately: a DocumentBridgeOptions (what local.ts's port passes to any bridge hop) is assignable to this, which is exactly what lets these run as ordinary bridge hops without a new hop kind -- through the port they run with fonts/sink undefined (the defaults), while a direct ergonomic caller can supply them for finer control over the layout and reconstruction legs. +// Options for a composed edge whose two formats share no ContentDocument variant, so the only route is through PDF (today: xlsx <-> markdown, csv <-> markdown). These are 'bridge' hops from the composition engine's point of view (neither endpoint is pdf), but internally they compose a toPdf leg -- which lays content out, so fonts/onFontSubstitution/onSubstitution/clock reach it -- with a fromPdf leg, which reconstructs, so sink reaches it. That is a wider shape than the PDF-bypassing DocumentBridgeOptions above, and every field is optional deliberately: a DocumentBridgeOptions (what local.ts's port passes to any bridge hop) is assignable to this, which is exactly what lets these run as ordinary bridge hops without a new hop kind -- through the port they run with fonts/sink undefined (the defaults), while a direct ergonomic caller can supply them for finer control over the layout and reconstruction legs. export interface ComposedDocumentOptions extends DocumentFontRegistryOptions { readonly signal?: AbortSignal; readonly onSubstitution?: (substitution: WinAnsiSubstitution, context: { readonly pageIndex: number }) => void; @@ -230,6 +256,28 @@ export function odtToMarkdown(bytes: Uint8Array, options?: Document return convertDocument('odt', 'markdown', bytes, options); } +// csv <-> xlsx and csv <-> ods: csv is the spreadsheet family's plain-text member -- it reads into and builds from the same spreadsheet ContentDocument variant as xlsx/ods (see capability.ts), so these resolve as single same-variant bridge hops, the csv side decoding/encoding RFC 4180 text with no package in between. The csv-sourced directions intersect CsvReadOptions (delimiter/onCellTypeInference reach readCsvContent's own parse) and the csv-target ones CsvWriteOptions (delimiter/sheet reach buildCsvText's own writer); every non-csv field of DocumentBridgeOptions threads exactly as it does for the bridges above. + +// Forwards to convertDocument (src/convert/composition.ts). +export function csvToXlsx(bytes: Uint8Array, options?: DocumentBridgeOptions & CsvReadOptions): Uint8Array { + return convertDocument('csv', 'xlsx', bytes, options); +} + +// Forwards to convertDocument (src/convert/composition.ts). +export function xlsxToCsv(bytes: Uint8Array, options?: DocumentBridgeOptions & CsvWriteOptions): Uint8Array { + return convertDocument('xlsx', 'csv', bytes, options); +} + +// Forwards to convertDocument (src/convert/composition.ts). +export function csvToOds(bytes: Uint8Array, options?: DocumentBridgeOptions & CsvReadOptions): Uint8Array { + return convertDocument('csv', 'ods', bytes, options); +} + +// Forwards to convertDocument (src/convert/composition.ts). +export function odsToCsv(bytes: Uint8Array, options?: DocumentBridgeOptions & CsvWriteOptions): Uint8Array { + return convertDocument('ods', 'csv', bytes, options); +} + // Four cross-variant content bridges (wordprocessing <-> presentation), two pairs: docx <-> pptx and odt <-> odp. These cross a VARIANT BOUNDARY via a real semantic transform (src/convert/variant-bridges.ts), which convertDocument's pathfinder resolves as a single cross-variant bridge hop. Both directions are APPROXIMATIONS -- a flow document has no real slide boundaries, and a deck has no flow -- but the blocks themselves (paragraphs, tables, images, list membership, run styling) survive intact through both transforms. // Forwards to convertDocument (src/convert/composition.ts). @@ -276,7 +324,19 @@ export function markdownToXlsx(bytes: Uint8Array, options?: Compose return convertDocument('markdown', 'xlsx', bytes, options); } -// odmToPdf -- the fourteenth conversion, and one of the four SPECIAL cases that stays as its real hand-written body rather than forwarding to convertDocument: a .odm master document doesn't carry its chapters' own content at all (readOdm's own module -- see odf.js's implementation report -- confirmed against real LibreOffice output that a text:section-source is always a bare external reference, never an embedded or cached copy), so producing a PDF requires a caller-supplied resolveSubDocument callback to hand back each chapter's own .odt bytes given its href. This is why odmToPdf takes an options object shape the other conversions don't, and why it is not wired into the DocumentConverter port (src/convert/port.ts) -- that port's convert(request, options) contract is fixed single-bytes-in/bytes-out, and widening it with a resolver parameter for this one format would leak an odm-specific concern into every other conversion's own request shape. A caller wanting odmToPdf behind the port can wrap it in their own adapter. convertDocument's own bytes-in/bytes-out contract cannot express the resolver callback either, which is why this stays hand-written. +// csv <-> markdown: csv and markdown share no ContentDocument variant (spreadsheet vs wordprocessing), so convertDocument's pathfinder resolves these the same three-hop way as xlsx <-> markdown above -- csvToMarkdown as [csv -> ods, ods -> pdf, pdf -> markdown], markdownToCsv as [markdown -> pdf, pdf -> ods, ods -> csv] -- with both legs' lossiness inherited in full. onDocument reports the last hop's package under the composition engine's own "fires exactly once, on the last hop" convention. + +// Forwards to convertDocument (src/convert/composition.ts). +export function csvToMarkdown(bytes: Uint8Array, options?: ComposedDocumentOptions & CsvReadOptions): Uint8Array { + return convertDocument('csv', 'markdown', bytes, options); +} + +// Forwards to convertDocument (src/convert/composition.ts). +export function markdownToCsv(bytes: Uint8Array, options?: ComposedDocumentOptions & CsvWriteOptions): Uint8Array { + return convertDocument('markdown', 'csv', bytes, options); +} + +// odmToPdf -- one of the four SPECIAL cases that stays as its real hand-written body rather than forwarding to convertDocument: a .odm master document doesn't carry its chapters' own content at all (readOdm's own module -- see odf.js's implementation report -- confirmed against real LibreOffice output that a text:section-source is always a bare external reference, never an embedded or cached copy), so producing a PDF requires a caller-supplied resolveSubDocument callback to hand back each chapter's own .odt bytes given its href. This is why odmToPdf takes an options object shape the other conversions don't, and why it is not wired into the DocumentConverter port (src/convert/port.ts) -- that port's convert(request, options) contract is fixed single-bytes-in/bytes-out, and widening it with a resolver parameter for this one format would leak an odm-specific concern into every other conversion's own request shape. A caller wanting odmToPdf behind the port can wrap it in their own adapter. convertDocument's own bytes-in/bytes-out contract cannot express the resolver callback either, which is why this stays hand-written. export interface OdmToPdfOptions extends DocumentToPdfOptions { // Called once per section whose chapter content could not be read inline from the master document itself, with that section's own href (e.g. "../chapter1.odt"). Returns that chapter's own .odt bytes, or undefined if the caller has no bytes for it -- an undefined result is not itself an error here; odmToPdf collects every section that ends up unresolved (no inline content AND no bytes from this callback, or no callback at all) and throws exactly once, naming all of them, rather than surfacing only the first the loop happens to reach. readonly resolveSubDocument?: (href: string) => Uint8Array | undefined; diff --git a/src/convert/csv.test.ts b/src/convert/csv.test.ts new file mode 100644 index 000000000..b4eaa67a9 --- /dev/null +++ b/src/convert/csv.test.ts @@ -0,0 +1,106 @@ +import { buildXlsxPackage, decodePackage as decodeOoxmlPackage, encodePackage as encodeOoxmlPackage, readXlsxContent } from 'ooxml.js'; +import { describe, expect, it } from 'vitest'; +import { decodeCsvText, encodeCsvText } from '../csv/text'; +import { TSV_DELIMITER, parseCsvRecords } from '../csv/records'; +import { CsvSheetNotSpecifiedError } from '../csv/write'; +import { csvToMarkdown, csvToOds, csvToPdf, csvToXlsx, markdownToCsv, odsToCsv, pdfToCsv, xlsxToCsv } from './convert'; +import { createLocalDocumentConverter } from './local'; + +const CSV_TEXT = 'Name,Amount\nWidget,42.5\nGadget,7\n'; +const csvBytes = (): Uint8Array => encodeCsvText(CSV_TEXT); + +describe('csv composition: same-variant bridges', () => { + it('csvToXlsx produces a real xlsx whose cells carry the header verbatim and the re-typed data values', () => { + const xlsxBytes = csvToXlsx(csvBytes()); + const content = readXlsxContent(decodeOoxmlPackage(xlsxBytes)); + if (content.kind !== 'spreadsheet') { + throw new Error('expected a spreadsheet ContentDocument'); + } + const sheet = content.sheets[0]!; + const cellAt = (row: number, column: number) => sheet.cells.find((cell) => cell.row === row && cell.column === column); + expect(cellAt(0, 0)?.value).toEqual({ kind: 'string', value: 'Name' }); + expect(cellAt(1, 0)?.value).toEqual({ kind: 'string', value: 'Widget' }); + expect(cellAt(1, 1)?.value).toEqual({ kind: 'number', value: 42.5 }); + expect(cellAt(2, 1)?.value).toEqual({ kind: 'number', value: 7 }); + }); + + it('csvToOds accepts the TSV delimiter option, parsing the identical grid from tab-separated text', () => { + const tsvBytes = encodeCsvText('Name\tAmount\nWidget\t42.5\nGadget\t7\n'); + const csvFromTsv = odsToCsv(csvToOds(tsvBytes, { delimiter: TSV_DELIMITER })); + expect(parseCsvRecords(decodeCsvText(csvFromTsv))).toEqual(parseCsvRecords(CSV_TEXT)); + }); + + it('odsToCsv/xlsxToCsv emit the rendered cells of real ods and xlsx fixtures', () => { + expect(parseCsvRecords(decodeCsvText(odsToCsv(csvToOds(csvBytes()))))).toEqual(parseCsvRecords(CSV_TEXT)); + expect(parseCsvRecords(decodeCsvText(xlsxToCsv(csvToXlsx(csvBytes()))))).toEqual(parseCsvRecords(CSV_TEXT)); + }); + + it('xlsxToCsv on a multi-sheet source throws CsvSheetNotSpecifiedError naming every sheet, and { sheet } selects one', () => { + const oneSheet = csvToXlsx(encodeCsvText('A\n1\n')); + // Build a genuine two-sheet xlsx through ooxml.js's own builder rather than mutating bridge output. + const sheetOne = readXlsxContent(decodeOoxmlPackage(oneSheet)); + if (sheetOne.kind !== 'spreadsheet') { + throw new Error('expected a spreadsheet ContentDocument'); + } + const twoSheets = { ...sheetOne, sheets: [...sheetOne.sheets, { ...sheetOne.sheets[0]!, name: 'Second' }] }; + const xlsxBytes = encodeOoxmlPackage(buildXlsxPackage(twoSheets)); + expect(() => xlsxToCsv(xlsxBytes)).toThrow(CsvSheetNotSpecifiedError); + expect(parseCsvRecords(decodeCsvText(xlsxToCsv(xlsxBytes, { sheet: 'Second' })))).toEqual([['A'], ['1']]); + }); +}); + +describe('csv composition: PDF pivot', () => { + it('csvToPdf produces valid PDF bytes (composed csv -> ods -> pdf, since csv has no layout engine of its own)', () => { + const pdfBytes = csvToPdf(csvBytes()); + expect(new TextDecoder('latin1').decode(pdfBytes.subarray(0, 5))).toBe('%PDF-'); + }); + + it('pdfToCsv round-trips the rendered header and cell text back to records', () => { + const csvRoundTripped = pdfToCsv(csvToPdf(csvBytes())); + const records = parseCsvRecords(decodeCsvText(csvRoundTripped)); + expect(records[0]).toEqual(['Name', 'Amount']); + expect(records[1]?.[0]).toBe('Widget'); + expect(records[1]?.[1]).toBe('42.5'); + }); + + it('csvToPdf threads onCellTypeInference through to the read hop, exposing the audit channel the ergonomic layer previously lacked', () => { + const events: string[] = []; + csvToPdf(csvBytes(), { onCellTypeInference: (event) => events.push(`${event.row}:${event.column}:${event.outcome}`) }); + // The sink reports decisions only: the header (never re-typed) and the plain-text names (no typing candidate) fire nothing, leaving exactly the two numeric retypes. + expect(events).toEqual(['1:1:retyped', '2:1:retyped']); + }); +}); + +describe('csv composition: pdf-composed last-resort pair', () => { + it('csvToMarkdown produces markdown carrying the rendered cell text', () => { + const markdown = new TextDecoder().decode(csvToMarkdown(csvBytes())); + expect(markdown).toContain('Name'); + expect(markdown).toContain('Widget'); + }); + + it('markdownToCsv produces well-formed RFC 4180 records from a markdown table', () => { + const markdownBytes = new TextEncoder().encode('| A | B |\n| --- | --- |\n| one | two |\n'); + const csvFromMarkdown = markdownToCsv(markdownBytes); + expect(parseCsvRecords(decodeCsvText(csvFromMarkdown)).length).toBeGreaterThan(0); + }); +}); + +describe('csv through the DocumentConverter port', () => { + it('routes csv -> xlsx and reports the target format', async () => { + const converter = createLocalDocumentConverter(); + const result = await converter.convert({ source: { format: 'csv', bytes: csvBytes() }, targetFormat: 'xlsx' }, { signal: new AbortController().signal }); + expect(result.document.format).toBe('xlsx'); + const content = readXlsxContent(decodeOoxmlPackage(result.document.bytes)); + if (content.kind !== 'spreadsheet') { + throw new Error('expected a spreadsheet ContentDocument'); + } + expect(content.sheets[0]?.cells.find((cell) => cell.row === 1 && cell.column === 1)?.value).toEqual({ kind: 'number', value: 42.5 }); + }); + + it('routes csv -> pdf and produces valid PDF bytes', async () => { + const converter = createLocalDocumentConverter(); + const result = await converter.convert({ source: { format: 'csv', bytes: csvBytes() }, targetFormat: 'pdf' }, { signal: new AbortController().signal }); + expect(result.document.format).toBe('pdf'); + expect(new TextDecoder('latin1').decode(result.document.bytes.subarray(0, 5))).toBe('%PDF-'); + }); +}); diff --git a/src/convert/document-fonts.ts b/src/convert/document-fonts.ts index f3190e741..d57f4dc9e 100644 --- a/src/convert/document-fonts.ts +++ b/src/convert/document-fonts.ts @@ -7,7 +7,7 @@ import type { DocumentFormat } from './port'; // Lives beside convert.ts's own DocumentFormat-dispatching functions (buildDocumentBytes, odbReportToPdf, ...) rather than in src/fonts/registry.ts alongside extractSourceFonts/FontSourcePackage: this package's own dependency-direction convention (see the README's Architecture section) states that `fonts` imports no local module at all, so it can sit beside `layout` rather than under it -- importing DocumentFormat from src/convert/port.ts here would break that invariant for no real gain, since `convert` already legitimately depends on `fonts` in the other direction. -// The subset of DocumentFormat that can declare a source-embedded font face at all: docx/pptx via OOXML's own fontTable.xml/embeddedFontLst, odt/odp/ods/odg via ODF's own office:font-face-decls. xlsx has no OOXML font-embedding vocabulary of its own; pdf/markdown carry no source-package concept to embed a font declaration in; a standalone .odf formula document embeds only the STIX Two Math font pdf-codec itself carries, never a caller-resolvable face; .odb has no font concept at all (and is not a DocumentFormat member regardless). +// The subset of DocumentFormat that can declare a source-embedded font face at all: docx/pptx via OOXML's own fontTable.xml/embeddedFontLst, odt/odp/ods/odg via ODF's own office:font-face-decls. xlsx has no OOXML font-embedding vocabulary of its own; pdf/markdown/csv carry no source-package concept to embed a font declaration in; a standalone .odf formula document embeds only the STIX Two Math font pdf-codec itself carries, never a caller-resolvable face; .odb has no font concept at all (and is not a DocumentFormat member regardless). const FONT_SOURCE_FORMATS: Readonly> = { docx: true, pptx: true, @@ -21,7 +21,7 @@ function isFontSourceFormat(format: DocumentFormat): format is keyof typeof FONT return format in FONT_SOURCE_FORMATS; } -// A recognised DocumentFormat that nonetheless has no source-embedded-font concept at all (xlsx, pdf, markdown, odf) -- a named class, matching this package's own convention for every other "recognised but unsupported" input across the .odb/odm surface (OdbTableNotSpecifiedError, OdbUnsupportedFormatError, ...), so a caller can narrow on it with instanceof rather than string-matching a thrown Error's own message. +// A recognised DocumentFormat that nonetheless has no source-embedded-font concept at all (xlsx, pdf, markdown, csv, odf) -- a named class, matching this package's own convention for every other "recognised but unsupported" input across the .odb/odm surface (OdbTableNotSpecifiedError, OdbUnsupportedFormatError, ...), so a caller can narrow on it with instanceof rather than string-matching a thrown Error's own message. export class UnsupportedFontSourceFormatError extends Error { readonly format: DocumentFormat; diff --git a/src/convert/local.test.ts b/src/convert/local.test.ts index 3334be392..27235a04c 100644 --- a/src/convert/local.test.ts +++ b/src/convert/local.test.ts @@ -35,8 +35,18 @@ describe('createLocalDocumentConverter: shape', () => { const converter = createLocalDocumentConverter(); // 5, not 4: convert()'s own ConversionOptions gained clock (a ClockPort), forwarded to every X-to-PDF conversion's /CreationDate and /ModDate stamping -- see port.ts's own contractVersion comment on what does and does not warrant a bump. expect(converter.contractVersion).toBe(5); - // SUPPORTED_CONVERSIONS is now derived from the composition pathfinder (resolveCompositionPlan) rather than a hand-maintained DIRECT_EDGES list. The pathfinder routes every pair of non-odf formats (each reaches all 8 others within the 3-hop cap), plus the special-case odf -> pdf pair -- 73 pairs total, sorted by source then target for determinism. + // SUPPORTED_CONVERSIONS is now derived from the composition pathfinder (resolveCompositionPlan) rather than a hand-maintained DIRECT_EDGES list. The pathfinder routes every pair of non-odf formats (each reaches all 9 others within the 3-hop cap), plus the special-case odf -> pdf pair -- 91 pairs total, sorted by source then target for determinism. csv joins as a full spreadsheet-variant member: same-variant bridges to ods/xlsx directly, everything else composed through the identical ods pivot xlsx uses. expect(converter.conversions).toEqual([ + { source: 'csv', target: 'docx' }, + { source: 'csv', target: 'markdown' }, + { source: 'csv', target: 'odg' }, + { source: 'csv', target: 'odp' }, + { source: 'csv', target: 'ods' }, + { source: 'csv', target: 'odt' }, + { source: 'csv', target: 'pdf' }, + { source: 'csv', target: 'pptx' }, + { source: 'csv', target: 'xlsx' }, + { source: 'docx', target: 'csv' }, { source: 'docx', target: 'markdown' }, { source: 'docx', target: 'odg' }, { source: 'docx', target: 'odp' }, @@ -45,6 +55,7 @@ describe('createLocalDocumentConverter: shape', () => { { source: 'docx', target: 'pdf' }, { source: 'docx', target: 'pptx' }, { source: 'docx', target: 'xlsx' }, + { source: 'markdown', target: 'csv' }, { source: 'markdown', target: 'docx' }, { source: 'markdown', target: 'odg' }, { source: 'markdown', target: 'odp' }, @@ -54,6 +65,7 @@ describe('createLocalDocumentConverter: shape', () => { { source: 'markdown', target: 'pptx' }, { source: 'markdown', target: 'xlsx' }, { source: 'odf', target: 'pdf' }, + { source: 'odg', target: 'csv' }, { source: 'odg', target: 'docx' }, { source: 'odg', target: 'markdown' }, { source: 'odg', target: 'odp' }, @@ -62,6 +74,7 @@ describe('createLocalDocumentConverter: shape', () => { { source: 'odg', target: 'pdf' }, { source: 'odg', target: 'pptx' }, { source: 'odg', target: 'xlsx' }, + { source: 'odp', target: 'csv' }, { source: 'odp', target: 'docx' }, { source: 'odp', target: 'markdown' }, { source: 'odp', target: 'odg' }, @@ -70,6 +83,7 @@ describe('createLocalDocumentConverter: shape', () => { { source: 'odp', target: 'pdf' }, { source: 'odp', target: 'pptx' }, { source: 'odp', target: 'xlsx' }, + { source: 'ods', target: 'csv' }, { source: 'ods', target: 'docx' }, { source: 'ods', target: 'markdown' }, { source: 'ods', target: 'odg' }, @@ -78,6 +92,7 @@ describe('createLocalDocumentConverter: shape', () => { { source: 'ods', target: 'pdf' }, { source: 'ods', target: 'pptx' }, { source: 'ods', target: 'xlsx' }, + { source: 'odt', target: 'csv' }, { source: 'odt', target: 'docx' }, { source: 'odt', target: 'markdown' }, { source: 'odt', target: 'odg' }, @@ -86,6 +101,7 @@ describe('createLocalDocumentConverter: shape', () => { { source: 'odt', target: 'pdf' }, { source: 'odt', target: 'pptx' }, { source: 'odt', target: 'xlsx' }, + { source: 'pdf', target: 'csv' }, { source: 'pdf', target: 'docx' }, { source: 'pdf', target: 'markdown' }, { source: 'pdf', target: 'odg' }, @@ -94,6 +110,7 @@ describe('createLocalDocumentConverter: shape', () => { { source: 'pdf', target: 'odt' }, { source: 'pdf', target: 'pptx' }, { source: 'pdf', target: 'xlsx' }, + { source: 'pptx', target: 'csv' }, { source: 'pptx', target: 'docx' }, { source: 'pptx', target: 'markdown' }, { source: 'pptx', target: 'odg' }, @@ -102,6 +119,7 @@ describe('createLocalDocumentConverter: shape', () => { { source: 'pptx', target: 'odt' }, { source: 'pptx', target: 'pdf' }, { source: 'pptx', target: 'xlsx' }, + { source: 'xlsx', target: 'csv' }, { source: 'xlsx', target: 'docx' }, { source: 'xlsx', target: 'markdown' }, { source: 'xlsx', target: 'odg' }, @@ -113,10 +131,12 @@ describe('createLocalDocumentConverter: shape', () => { ]); }); - // A dedicated, order-independent assertion for the special-case odf -> pdf pair and the composed xlsx <-> pdf pair, on top of the exact-array assertion above -- these keep working even if SUPPORTED_CONVERSIONS' own order ever changes. - it('includes the special-case odf->pdf and composed xlsx<->pdf pairs', () => { + // A dedicated, order-independent assertion for the special-case odf -> pdf pair and the composed csv/xlsx <-> pdf pairs, on top of the exact-array assertion above -- these keep working even if SUPPORTED_CONVERSIONS' own order ever changes. + it('includes the special-case odf->pdf and composed csv/xlsx<->pdf pairs', () => { const converter = createLocalDocumentConverter(); expect(converter.conversions).toContainEqual({ source: 'odf', target: 'pdf' }); + expect(converter.conversions).toContainEqual({ source: 'csv', target: 'pdf' }); + expect(converter.conversions).toContainEqual({ source: 'pdf', target: 'csv' }); expect(converter.conversions).toContainEqual({ source: 'xlsx', target: 'pdf' }); expect(converter.conversions).toContainEqual({ source: 'pdf', target: 'xlsx' }); }); diff --git a/src/convert/local.ts b/src/convert/local.ts index 1cd68ab5a..44b2d7a47 100644 --- a/src/convert/local.ts +++ b/src/convert/local.ts @@ -75,7 +75,7 @@ export function createLocalDocumentConverter(): DocumentConverter { return Promise.reject(new UnsupportedConversionError(source.format, targetFormat)); } - // convertDocument runs the resolved plan end to end, threading the port's ConversionOptions through to whichever hop consumes each field: fonts/onFontSubstitution/onSubstitution/clock reach any toPdf hop (the only kind that lays text out and resolves a face), sink reaches any fromPdf hop (the only kind that reads a PDF and can report parse diagnostics), and signal/images reach every hop. The onDocument callback captures the DocumentPackage the first content-producing hop builds, mirroring the per-edge onDocument wiring the previous direct-edge path threaded into each edge kind individually. + // convertDocument runs the resolved plan end to end, threading the port's ConversionOptions through to whichever hop consumes each field: fonts/onFontSubstitution/onSubstitution/clock reach any toPdf hop (the only kind that lays text out and resolves a face), sink reaches any fromPdf hop (the only kind that reads a PDF and can report parse diagnostics), delimiter/sheet reach any csv hop, and signal/images reach every hop. The onDocument callback captures the DocumentPackage the first content-producing hop builds, mirroring the per-edge onDocument wiring the previous direct-edge path threaded into each edge kind individually. const bytes = convertDocument(source.format, targetFormat, source.bytes, { signal: options.signal, fonts: options.fonts, @@ -83,6 +83,8 @@ export function createLocalDocumentConverter(): DocumentConverter { onSubstitution: (s, c) => diagnostics.push(substitutionDiagnostic(s, c)), sink: (d) => diagnostics.push(fromPdfDiagnostic(d)), images: options.images, + delimiter: options.delimiter, + sheet: options.sheet, clock: options.clock, onDocument, }); diff --git a/src/convert/port.test.ts b/src/convert/port.test.ts index 5f723d5f5..e6a4ff65d 100644 --- a/src/convert/port.test.ts +++ b/src/convert/port.test.ts @@ -3,7 +3,7 @@ import { DOCUMENT_FORMATS, DocumentFormatSchema } from './port'; describe('DocumentFormatSchema / DOCUMENT_FORMATS', () => { it('DOCUMENT_FORMATS lists every DocumentFormat member, matching the schema exactly', () => { - expect(DOCUMENT_FORMATS).toEqual(['docx', 'pptx', 'xlsx', 'odt', 'odp', 'ods', 'odg', 'odf', 'markdown', 'pdf']); + expect(DOCUMENT_FORMATS).toEqual(['docx', 'pptx', 'xlsx', 'odt', 'odp', 'ods', 'odg', 'odf', 'csv', 'markdown', 'pdf']); expect(DOCUMENT_FORMATS).toEqual(DocumentFormatSchema.options); }); diff --git a/src/convert/port.ts b/src/convert/port.ts index 13158dc58..e1c7d3fb2 100644 --- a/src/convert/port.ts +++ b/src/convert/port.ts @@ -6,10 +6,10 @@ import { z } from 'zod'; // The conversion behaviour modelled as a swappable port/contract, not a hard-wired function -- this workspace's standing "portable runtime and storage boundaries" convention, even though the only implementation today (local.ts) is entirely synchronous under the hood. `convert()` itself stays async and takes a mandatory abort signal regardless of that: it's a portability contract for a future non-local adapter (a remote conversion service, say), not a reflection of the local implementation's own synchronicity. -// 'odf' (an ODF formula document) has exactly one direction wired into this port (odf -> pdf, via odfToPdf -- see local.ts) -- unlike every other member here, there is deliberately no pdf -> odf entry: odmToPdf's own README/gotcha explains why that reverse direction is not attempted (recovering structured MathML from rendered glyphs is a categorically different, OCR-adjacent problem, not a geometry-reconstruction one). 'markdown' shares the wordprocessing ContentDocument variant with docx/odt (see capability.ts's own FORMAT_CAPABILITIES.markdown) -- it has a genuine two-way layout-engine edge to/from pdf (markdownToPdf/pdfToMarkdown), plus direct same-variant bridges to docx and odt, exactly like odt already has to docx. +// 'odf' (an ODF formula document) has exactly one direction wired into this port (odf -> pdf, via odfToPdf -- see local.ts) -- unlike every other member here, there is deliberately no pdf -> odf entry: odmToPdf's own README/gotcha explains why that reverse direction is not attempted (recovering structured MathML from rendered glyphs is a categorically different, OCR-adjacent problem, not a geometry-reconstruction one). 'markdown' shares the wordprocessing ContentDocument variant with docx/odt (see capability.ts's own FORMAT_CAPABILITIES.markdown) -- it has a genuine two-way layout-engine edge to/from pdf (markdownToPdf/pdfToMarkdown), plus direct same-variant bridges to docx and odt, exactly like odt already has to docx. 'csv' shares the spreadsheet ContentDocument variant with xlsx/ods -- like xlsx it has no layout engine of its own (the composition engine routes csv <-> pdf through the ods bridge), and TSV is the SAME member with { delimiter: '\t' } rather than a second enum entry, since a delimiter is a parse option, not a different document format. // // Zod-first, matching this package's own convention (src/model/bytes.ts and every document-schema.js-sourced union re-exported above): the schema is the source of truth, DocumentFormat is inferred from it rather than hand-written, and DOCUMENT_FORMATS (below) is derived from the same schema rather than a second, independently-typed literal array that could drift out of sync with it. -export const DocumentFormatSchema = z.enum(['docx', 'pptx', 'xlsx', 'odt', 'odp', 'ods', 'odg', 'odf', 'markdown', 'pdf']); +export const DocumentFormatSchema = z.enum(['docx', 'pptx', 'xlsx', 'odt', 'odp', 'ods', 'odg', 'odf', 'csv', 'markdown', 'pdf']); export type DocumentFormat = z.infer; // Every DocumentFormat member as a plain readonly array, for a caller that wants to enumerate or validate against the full format set without constructing its own Zod schema -- e.g. a CLI's own usage-error text, or an MCP tool's JSON-schema `enum` input. export const DOCUMENT_FORMATS: readonly DocumentFormat[] = DocumentFormatSchema.options; @@ -48,6 +48,10 @@ export interface ConversionOptions { readonly onFontSubstitution?: (substitution: FontSubstitution) => void; // A synchronous resolver for markdown images with a non-data: destination (a relative path, a bare URL) -- the same live-callback shape as onFontSubstitution, with the same remote-adapter caveat: the local implementation honours it (threading it through to markdown-codec's MarkdownImageResolver port for the markdown-sourced conversions), a remote adapter would have no way to call back into the caller's process and would instead leave non-data: images degraded to alt-text. Only the markdown-sourced conversions consult it; every other conversion ignores it. readonly images?: MarkdownImageResolver; + // The single-character field delimiter the csv-sourced hops of a conversion parse with, and the csv-target hops write with -- ',' by default (records.ts's DEFAULT_CSV_DELIMITER), '\t' (TSV_DELIMITER) for TSV. A plain string, so a remote adapter honours it exactly as the local one does; every non-csv hop ignores it. + readonly delimiter?: string; + // Selects which sheet of a multi-sheet spreadsheet a csv-TARGET hop writes -- csv has no second sheet, so writing one is a caller decision (see buildCsvText's own CsvSheetNotSpecifiedError). Every non-csv-target hop ignores it. + readonly sheet?: string; // An injectable clock forwarded to the X-to-PDF conversion's own /CreationDate and /ModDate stamping (see DocumentToPdfOptions.clock) -- a fixed instant is fully serialisable, so a remote adapter honours it the same way the local one does. readonly clock?: ClockPort; } diff --git a/src/convert/roundtrip-matrix.test.ts b/src/convert/roundtrip-matrix.test.ts index 730fa35e0..672f60a83 100644 --- a/src/convert/roundtrip-matrix.test.ts +++ b/src/convert/roundtrip-matrix.test.ts @@ -2,6 +2,9 @@ import { decodePackage as decodeOdfPackage } from 'odf.js'; import { decodePackage as decodeOoxmlPackage, readXlsxContent } from 'ooxml.js'; import { readPdf } from 'pdf-codec'; import { describe, expect, it } from 'vitest'; +import { encodeCsvText } from '../csv/text'; +import { parseCsvRecords } from '../csv/records'; +import { CsvSheetNotSpecifiedError } from '../csv/write'; import { createDocx, openDocx } from '../edit/docx/editor'; import { openOdp } from '../edit/odp/editor'; import { openOdt } from '../edit/odt/editor'; @@ -17,7 +20,7 @@ import { gridOdsBytes, minimalOdsBytes, richOdsBytes } from '../test-support/ods import { minimalOdtBytes } from '../test-support/odt'; import { minimalDocxBytes } from '../test-support/docx'; import { minimalPptxBytes } from '../test-support/pptx'; -import { docxToMarkdown, docxToOdt, docxToPdf, docxToPptx, markdownToDocx, markdownToOdt, markdownToPdf, odfToPdf, odgToPdf, odpToPdf, odpToPptx, odpToOdt, odsToPdf, odsToXlsx, odtToDocx, odtToMarkdown, odtToOdp, odtToPdf, pdfToDocx, pdfToMarkdown, pdfToOdg, pdfToOdp, pdfToOds, pdfToOdt, pdfToPptx, pdfToXlsx, pptxToDocx, pptxToOdp, pptxToPdf, xlsxToOds, xlsxToPdf, xlsxToMarkdown, markdownToXlsx } from './convert'; +import { csvToOds, docxToMarkdown, docxToOdt, docxToPdf, docxToPptx, markdownToDocx, markdownToOdt, markdownToPdf, odfToPdf, odgToPdf, odpToPdf, odpToPptx, odpToOdt, odsToCsv, odsToPdf, odsToXlsx, odtToDocx, odtToMarkdown, odtToOdp, odtToPdf, pdfToDocx, pdfToMarkdown, pdfToOdg, pdfToOdp, pdfToOds, pdfToOdt, pdfToPptx, pdfToXlsx, pptxToDocx, pptxToOdp, pptxToPdf, xlsxToOds, xlsxToPdf, xlsxToMarkdown, markdownToXlsx } from './convert'; import { createLocalDocumentConverter } from './local'; import type { DocumentFormat } from './port'; @@ -246,6 +249,20 @@ const MATRIX_ENTRIES: readonly MatrixEntry[] = [ expect(cellAt(4, 0)?.value).toEqual({ kind: 'string', value: 'Merged Cell' }); }, }, + { + name: 'csv <-> ods (bridge)', + edges: [ + { source: 'csv', target: 'ods' }, + { source: 'ods', target: 'csv' }, + ], + // The csv <-> ods bridge is a direct ContentDocument pivot copy like xlsx <-> ods above -- no layout engine, no reconstruction. The one csv-boundary transformation is read-side re-typing: '42.5' parses into a number cell whose displayText prints back as the identical digits, so a plain-text fixture round-trips field-for-field. The source text is parsed here rather than compared as a raw string, because the writer emits the RFC 4180 CRLF line breaks the fixture's \n spelling is equivalent to. + run: () => { + const csvText = 'Name,Amount\nWidget,42.5\nGadget,7\n'; + const odsBytes = csvToOds(encodeCsvText(csvText)); + const roundTrippedText = new TextDecoder().decode(odsToCsv(odsBytes)); + expect(parseCsvRecords(roundTrippedText)).toEqual(parseCsvRecords(csvText)); + }, + }, { name: 'ods <-> pdf', edges: [ @@ -449,6 +466,8 @@ function fixtureBytes(format: DocumentFormat): Uint8Array { return odsToXlsx(minimalOdsBytes()); case 'markdown': return encodeMarkdownText('# Heading\n\nA paragraph of text.'); + case 'csv': + return encodeCsvText('Name,Amount\nWidget,42.5\nGadget,7\n'); case 'odf': return odfFormulaBytes(FRACTION_FORMULA); case 'pdf': @@ -475,6 +494,7 @@ function isValidOutput(format: DocumentFormat, bytes: Uint8Array): return bytes[0] === 0x50 && bytes[1] === 0x4b; } case 'markdown': + case 'csv': return bytes.length > 0; default: return false; @@ -487,10 +507,16 @@ describe.each(ALL_SUPPORTED_PAIRS.map((pair) => [`${pair.source}->${pair.target} it('produces valid output of the target format without throwing', async () => { const converter = createLocalDocumentConverter(); const sourceBytes = fixtureBytes(pair.source); - const result = await converter.convert( - { source: { format: pair.source, bytes: sourceBytes }, targetFormat: pair.target }, - { signal: new AbortController().signal }, - ); + // A csv target whose intermediate content carries more than one sheet (odp -> csv reconstructs one sheet per slide) is a caller decision, not a conversion failure: without a selection the conversion refuses with CsvSheetNotSpecifiedError naming every sheet, and re-running it with { sheet: } produces the csv. Both halves of that contract are asserted here rather than swallowing the refusal. The async wrapper matters: the local converter throws synchronously inside convert() (its body runs the whole pipeline before constructing the return promise), so without it the refusal would bypass .catch entirely. + const run = async (options: { sheet?: string } = {}) => + converter.convert({ source: { format: pair.source, bytes: sourceBytes }, targetFormat: pair.target }, { signal: new AbortController().signal, ...options }); + const result = await run().catch((error: unknown) => { + if (!(error instanceof CsvSheetNotSpecifiedError)) { + throw error; + } + expect(error.availableSheets.length).toBeGreaterThan(1); + return run({ sheet: error.availableSheets[0]! }); + }); expect(result.document.format).toBe(pair.target); expect(isValidOutput(pair.target, result.document.bytes)).toBe(true); }); diff --git a/src/csv/read-write.test.ts b/src/csv/read-write.test.ts new file mode 100644 index 000000000..d87ac32b3 --- /dev/null +++ b/src/csv/read-write.test.ts @@ -0,0 +1,197 @@ +import { describe, expect, it } from 'vitest'; +import type { ContentDocument } from 'document-schema.js'; +import { CONTENT_FORMAT_VERSION } from 'document-schema.js'; +import { CsvInvalidUtf8Error, decodeCsvText, encodeCsvText } from './text'; +import { TSV_DELIMITER, parseCsvRecords } from './records'; +import { readCsvContent } from './read'; +import type { CellTypeInference } from '../layout/cell-typing'; +import { buildCsvText } from './write'; +import { CsvSheetNotFoundError, CsvSheetNotSpecifiedError, CsvUnsupportedDocumentKindError } from './write'; + +interface SheetFixture { + readonly name: string; + readonly rows: readonly (readonly string[])[]; +} + +function spreadsheetDocument(sheets: readonly SheetFixture[]): ContentDocument { + return { + kind: 'spreadsheet', + formatVersion: CONTENT_FORMAT_VERSION, + metadata: {}, + sheets: sheets.map(({ name, rows }) => ({ + name, + images: [], + columns: Array.from({ length: rows.reduce((max, row) => Math.max(max, row.length), 0) }, (_unused, index) => ({ index, widthPt: 64 })), + rows: Array.from({ length: rows.length }, (_unused, index) => ({ index, heightPt: 15 })), + printSettings: { pageSize: { widthPt: 595, heightPt: 842 }, margins: { topPt: 0, rightPt: 0, bottomPt: 0, leftPt: 0 }, gridlines: false, headers: false, pageOrder: 'downThenOver' }, + cells: rows.flatMap((row, rowIndex) => row.map((field, columnIndex) => ({ row: rowIndex, column: columnIndex, value: { kind: 'string' as const, value: field }, displayText: field }))), + })), + }; +} + +describe('readCsvContent', () => { + it('writes the first record as verbatim string header cells and never re-types them', () => { + // A header of "007" and "TRUE" -- text that inferCellValue would decline/re-type as data -- must stay a plain verbatim string at row 0, because headers are labels, not data. + const document = readCsvContent('007,TRUE\n42.5,x\n'); + expect(document.kind).toBe('spreadsheet'); + if (document.kind !== 'spreadsheet') { + throw new Error('expected a spreadsheet ContentDocument'); + } + const [sheet] = document.sheets; + const cellAt = (row: number, column: number) => sheet?.cells.find((cell) => cell.row === row && cell.column === column); + expect(cellAt(0, 0)?.value).toEqual({ kind: 'string', value: '007' }); + expect(cellAt(0, 1)?.value).toEqual({ kind: 'string', value: 'TRUE' }); + }); + + it('re-types data cells through the shared cell-typing heuristic and declines exactly where it declines', () => { + // "1,234" arrives as one quoted field, so the grouping-ambiguity decline is genuinely exercised rather than pre-empted by field splitting. + const document = readCsvContent('h1,h2,h3,h4,h5,h6\n42.5,TRUE,2024-01-15,007,"1,234",plain\n'); + if (document.kind !== 'spreadsheet') { + throw new Error('expected a spreadsheet ContentDocument'); + } + const [sheet] = document.sheets; + const valueAt = (column: number) => sheet?.cells.find((cell) => cell.row === 1 && cell.column === column)?.value; + expect(valueAt(0)).toEqual({ kind: 'number', value: 42.5 }); + expect(valueAt(1)).toEqual({ kind: 'boolean', value: true }); + expect(valueAt(2)?.kind).toBe('date'); + // All three declines stay plain strings: a leading-zero part number, a grouping-ambiguous number (the typing heuristic cannot distinguish 1234 from 1.234), and ordinary text. + expect(valueAt(3)).toEqual({ kind: 'string', value: '007' }); + expect(valueAt(4)).toEqual({ kind: 'string', value: '1,234' }); + expect(valueAt(5)).toEqual({ kind: 'string', value: 'plain' }); + }); + + it('carries the raw field text in displayText for every cell, independent of the inferred kind', () => { + const document = readCsvContent('h1,h2\n42.5,007\n'); + if (document.kind !== 'spreadsheet') { + throw new Error('expected a spreadsheet ContentDocument'); + } + for (const cell of document.sheets[0]!.cells) { + expect(cell.displayText).toBe(cell.row === 0 ? (cell.column === 0 ? 'h1' : 'h2') : cell.column === 0 ? '42.5' : '007'); + } + }); + + it('maps an empty data field to the empty cell and pads a short record to the grid width with empty cells', () => { + // Row 2 has one field where the grid is three wide: columns 1 and 2 are genuine empty cells, not holes. + const document = readCsvContent('a,b,c\n1,,3\nsolo\n'); + if (document.kind !== 'spreadsheet') { + throw new Error('expected a spreadsheet ContentDocument'); + } + const [sheet] = document.sheets; + const valueAt = (row: number, column: number) => sheet?.cells.find((cell) => cell.row === row && cell.column === column)?.value; + expect(valueAt(1, 1)).toEqual({ kind: 'empty' }); + expect(valueAt(2, 0)).toEqual({ kind: 'string', value: 'solo' }); + expect(valueAt(2, 1)).toEqual({ kind: 'empty' }); + expect(valueAt(2, 2)).toEqual({ kind: 'empty' }); + }); + + it('names the lone sheet Sheet1 and emits exactly one sheet, since a csv file is one table by construction', () => { + const document = readCsvContent('a,b\n1,2\n'); + if (document.kind !== 'spreadsheet') { + throw new Error('expected a spreadsheet ContentDocument'); + } + expect(document.sheets).toHaveLength(1); + expect(document.sheets[0]?.name).toBe('Sheet1'); + }); + + it('parses with the TSV delimiter when asked', () => { + const document = readCsvContent('a,b\t42.5\n', { delimiter: TSV_DELIMITER }); + if (document.kind !== 'spreadsheet') { + throw new Error('expected a spreadsheet ContentDocument'); + } + const [sheet] = document.sheets; + expect(sheet?.cells.find((cell) => cell.row === 0 && cell.column === 0)?.value).toEqual({ kind: 'string', value: 'a,b' }); + expect(sheet?.cells.find((cell) => cell.row === 0 && cell.column === 1)?.value).toEqual({ kind: 'string', value: '42.5' }); + }); + + it('fires onCellTypeInference exactly where inferCellValue reaches a decision, never for header cells or no-candidate text', () => { + const events: CellTypeInference[] = []; + readCsvContent('h1,h2\n007,42.5\nYes,x\n', { onCellTypeInference: (event) => events.push(event) }); + // Header cells are never re-typed, so the header's own "007" fires nothing. "x" matches no typing rule at all, which is not a decision and fires nothing either -- the sink reports decisions (retypes and named-ambiguity declines), not every cell. + expect(events).toEqual([ + { sheetIndex: 0, row: 1, column: 0, displayText: '007', outcome: 'declined', reason: 'leading-zero-digits' }, + { sheetIndex: 0, row: 1, column: 1, displayText: '42.5', outcome: 'retyped', value: { kind: 'number', value: 42.5 }, rule: 'plain-number' }, + { sheetIndex: 0, row: 2, column: 0, displayText: 'Yes', outcome: 'declined', reason: 'ambiguous-boolean-word' }, + ]); + }); +}); + +describe('buildCsvText', () => { + it('writes the lone sheet by default, emitting displayText, and joins with CRLF plus a trailing CRLF', () => { + const text = buildCsvText(spreadsheetDocument([{ name: 'Data', rows: [['Name', 'Amount'], ['Widget', '42.5']] }])); + expect(text).toBe('Name,Amount\r\nWidget,42.5\r\n'); + }); + + it('requires a sheet name for a multi-sheet document, naming every sheet rather than silently truncating', () => { + const document = spreadsheetDocument([ + { name: 'First', rows: [['a']] }, + { name: 'Second', rows: [['b']] }, + ]); + expect(() => buildCsvText(document)).toThrow(CsvSheetNotSpecifiedError); + try { + buildCsvText(document); + } catch (error) { + if (error instanceof CsvSheetNotSpecifiedError) { + expect(error.availableSheets).toEqual(['First', 'Second']); + } + } + expect(buildCsvText(document, { sheet: 'Second' })).toBe('b\r\n'); + }); + + it('throws CsvSheetNotFoundError for a named sheet that does not exist, and for a document with no sheets at all', () => { + const document = spreadsheetDocument([{ name: 'Only', rows: [['a']] }]); + expect(() => buildCsvText(document, { sheet: 'Missing' })).toThrow(CsvSheetNotFoundError); + expect(() => buildCsvText(spreadsheetDocument([]))).toThrow(CsvSheetNotFoundError); + }); + + it('throws CsvUnsupportedDocumentKindError for a non-spreadsheet ContentDocument', () => { + const wordprocessing: ContentDocument = { kind: 'wordprocessing', formatVersion: CONTENT_FORMAT_VERSION, metadata: {}, sections: [] }; + expect(() => buildCsvText(wordprocessing)).toThrow(CsvUnsupportedDocumentKindError); + }); + + it('writes the TSV delimiter when asked, quoting on tab rather than comma', () => { + const text = buildCsvText(spreadsheetDocument([{ name: 'Data', rows: [['a,b', 'c\td']] }]), { delimiter: TSV_DELIMITER }); + // The comma stays bare: under the tab delimiter a comma is ordinary field text, not a split hazard. + expect(text).toBe('a,b\t"c\td"\r\n'); + }); + + it('writes empty fields for unpopulated grid positions, so a sparse sheet stays a uniform rectangle', () => { + // Row 1 populates only column 1; the dense grid writes column 0 as an empty field rather than collapsing the row. + const document: ContentDocument = { + kind: 'spreadsheet', + formatVersion: CONTENT_FORMAT_VERSION, + metadata: {}, + sheets: [{ + name: 'Sparse', + images: [], + columns: [{ index: 0, widthPt: 64 }, { index: 1, widthPt: 64 }], + rows: [{ index: 0, heightPt: 15 }, { index: 1, heightPt: 15 }], + printSettings: { pageSize: { widthPt: 595, heightPt: 842 }, margins: { topPt: 0, rightPt: 0, bottomPt: 0, leftPt: 0 }, gridlines: false, headers: false, pageOrder: 'downThenOver' }, + cells: [ + { row: 0, column: 0, value: { kind: 'string', value: 'h0' }, displayText: 'h0' }, + { row: 0, column: 1, value: { kind: 'string', value: 'h1' }, displayText: 'h1' }, + { row: 1, column: 1, value: { kind: 'string', value: 'only' }, displayText: 'only' }, + ], + }], + }; + expect(buildCsvText(document)).toBe('h0,h1\r\n,only\r\n'); + }); +}); + +describe('readCsvContent -> buildCsvText round trip', () => { + it('round-trips the parsed records field-for-field, since a re-typed value prints back as the identical digits', () => { + const csvText = 'Name,Amount,Active\nWidget,42.5,TRUE\nGadget,7,No\n'; + const document = readCsvContent(csvText); + const rebuilt = buildCsvText(document); + expect(parseCsvRecords(rebuilt)).toEqual(parseCsvRecords(csvText)); + }); +}); + +describe('decodeCsvText / encodeCsvText', () => { + it('round-trips text through the byte boundary', () => { + expect(decodeCsvText(encodeCsvText('a,b\r\ncafé,42.5\r\n'))).toBe('a,b\r\ncafé,42.5\r\n'); + }); + + it('throws CsvInvalidUtf8Error on malformed UTF-8 rather than producing U+FFFD replacement characters', () => { + expect(() => decodeCsvText(new Uint8Array([0xff, 0xfe, 0x00]))).toThrow(CsvInvalidUtf8Error); + }); +}); diff --git a/src/csv/read.ts b/src/csv/read.ts new file mode 100644 index 000000000..305f7b421 --- /dev/null +++ b/src/csv/read.ts @@ -0,0 +1,67 @@ +import type { ContentDocument, ContentSheet, ContentSheetCell, ContentSheetColumn, ContentSheetPrintSettings, ContentSheetRow } from 'document-schema.js'; +import { CONTENT_FORMAT_VERSION, PAGE_SIZE_A4 } from 'document-schema.js'; +import type { CellTypeInferenceSink } from '../layout/cell-typing'; +import { inferCellValue } from '../layout/cell-typing'; +import { DEFAULT_CSV_DELIMITER, parseCsvRecords } from './records'; + +// The csv read half: RFC 4180 records -> one spreadsheet ContentDocument, on the identical pattern src/odb/spreadsheet.ts's own odbTablesToSpreadsheetDocument/tableToSheet already established for another untyped tabular source (.odb tables). The first record is the header row, written as verbatim string cells at row 0 -- headers are labels, never data to re-type, so inferCellValue never sees them. Every data cell runs through the SAME cell-typing heuristic the pdf->ods reconstructor uses (src/layout/cell-typing.ts's inferCellValue), with the identical confidence bar and the identical guarantee: displayText carries the raw field text verbatim independent of what value.kind was inferred, so a part number printed as "007" stays recoverable even though its value stays a string. +// +// A csv file is one sheet by construction -- there is no second table in the format -- so the sheet is named 'Sheet1', matching the name both Excel and LibreOffice give a lone spreadsheet sheet. + +export interface ReadCsvContentOptions { + // The single-character field delimiter the text is parsed with -- DEFAULT_CSV_DELIMITER (',') for csv proper, '\t' (records.ts's TSV_DELIMITER) for TSV. + readonly delimiter?: string; + // The same audit channel the pdf->ods reconstructor exposes: fires once per DATA cell where inferCellValue reached a decision (retyped or declined), carrying { sheetIndex: 0, row, column, displayText } merged with the decision. Header cells never fire -- they are not re-typed at all. + readonly onCellTypeInference?: CellTypeInferenceSink; +} + +const HEADER_ROW_INDEX = 0; +const SHEET_NAME = 'Sheet1'; +// Fallback sizing only -- csv records carry no column-width/row-height information at all. Mirrors src/layout/sheets.ts's own DEFAULT_COLUMN_WIDTH_PT/DEFAULT_ROW_HEIGHT_PT fallback values exactly, the same values src/odb/spreadsheet.ts restates locally for the identical reason (that module keeps them private). +const COLUMN_WIDTH_PT = 64; +const ROW_HEIGHT_PT = 15; +// 2cm margins on an A4 page -- the identical src/odb/spreadsheet.ts fallback, for the identical reason: a table built from nothing but text fields has no real page layout to read a margin from. +const MARGIN_PT = 56.69291338582677; +const DEFAULT_PRINT_SETTINGS: ContentSheetPrintSettings = { + pageSize: PAGE_SIZE_A4, + margins: { topPt: MARGIN_PT, rightPt: MARGIN_PT, bottomPt: MARGIN_PT, leftPt: MARGIN_PT }, + gridlines: true, + headers: true, + pageOrder: 'downThenOver', +}; + +// One data field to a typed cell: empty text is the empty cell (kind 'empty' has no value field), anything inferCellValue re-types carries the inferred value, and everything else -- declines and plain text alike -- stays a string. displayText is the raw field text in every branch, matching the cell-typing module's own contract. +function dataCell(rowIndex: number, columnIndex: number, field: string, onCellTypeInference: CellTypeInferenceSink | undefined): ContentSheetCell { + const inference = field === '' ? undefined : inferCellValue(field); + if (inference !== undefined) { + onCellTypeInference?.({ sheetIndex: 0, row: rowIndex, column: columnIndex, displayText: field, ...inference }); + } + const value = field === '' ? { kind: 'empty' as const } : inference?.outcome === 'retyped' ? inference.value : { kind: 'string' as const, value: field }; + return { row: rowIndex, column: columnIndex, value, displayText: field }; +} + +export function readCsvContent(text: string, options?: ReadCsvContentOptions): ContentDocument { + const records = parseCsvRecords(text, options?.delimiter ?? DEFAULT_CSV_DELIMITER); + const onCellTypeInference = options?.onCellTypeInference; + // The header record may be shorter or longer than a data record (trailing empty fields are unrepresentable in csv text and dropped by producers); the grid is the widest record, and every shorter row pads with empty cells so the sheet stays a uniform rectangle -- the shape every ContentSheet consumer (layout engine, ods/xlsx builders) already expects. + const columnCount = records.reduce((max, record) => Math.max(max, record.length), 0); + + const cells: ContentSheetCell[] = []; + records.forEach((record, rowIndex) => { + for (let columnIndex = 0; columnIndex < columnCount; columnIndex++) { + // An absent field at a trailing grid position is an empty cell, not a hole: a record shorter than the grid means its authoring row simply ended there. + const field = record[columnIndex] ?? ''; + if (rowIndex === HEADER_ROW_INDEX) { + cells.push({ row: rowIndex, column: columnIndex, value: { kind: 'string', value: field }, displayText: field }); + } else { + cells.push(dataCell(rowIndex, columnIndex, field, onCellTypeInference)); + } + } + }); + + const columns: ContentSheetColumn[] = Array.from({ length: columnCount }, (_unused, index) => ({ index, widthPt: COLUMN_WIDTH_PT })); + const rows: ContentSheetRow[] = Array.from({ length: records.length }, (_unused, index) => ({ index, heightPt: ROW_HEIGHT_PT })); + const sheet: ContentSheet = { name: SHEET_NAME, cells, columns, rows, images: [], printSettings: DEFAULT_PRINT_SETTINGS }; + + return { kind: 'spreadsheet', formatVersion: CONTENT_FORMAT_VERSION, metadata: {}, sheets: [sheet] }; +} diff --git a/src/csv/records.test.ts b/src/csv/records.test.ts new file mode 100644 index 000000000..82a6ad0e5 --- /dev/null +++ b/src/csv/records.test.ts @@ -0,0 +1,99 @@ +import { describe, expect, it } from 'vitest'; +import { CsvParseError, DEFAULT_CSV_DELIMITER, TSV_DELIMITER, parseCsvRecords, quoteCsvField } from './records'; + +describe('parseCsvRecords', () => { + it('parses plain comma-delimited records', () => { + expect(parseCsvRecords('a,b,c\r\n1,2,3\r\n')).toEqual([ + ['a', 'b', 'c'], + ['1', '2', '3'], + ]); + }); + + it('accepts CRLF, LF, and CR record breaks alike -- the writer always emits CRLF, but real-world files arrive LF-only or CR-only', () => { + const expected = [['a', 'b'], ['1', '2']]; + expect(parseCsvRecords('a,b\r\n1,2\r\n')).toEqual(expected); + expect(parseCsvRecords('a,b\n1,2\n')).toEqual(expected); + expect(parseCsvRecords('a,b\r1,2\r')).toEqual(expected); + }); + + it('parses a quoted field containing the delimiter, and a doubled quote inside a quoted field as one literal quote (RFC 4180 2.7)', () => { + expect(parseCsvRecords('"a,b",c\r\n"say ""hi""",d\r\n')).toEqual([ + ['a,b', 'c'], + ['say "hi"', 'd'], + ]); + }); + + it('parses a quoted field spanning a record break as one field with an embedded newline', () => { + expect(parseCsvRecords('"line one\r\nline two",b\r\n')).toEqual([['line one\r\nline two', 'b']]); + }); + + it('takes a quote appearing mid-field as a literal character, matching what spreadsheet exporters emit for text like 5" drive', () => { + expect(parseCsvRecords('5" drive,42\r\n')).toEqual([['5" drive', '42']]); + }); + + it('drops a blank line entirely rather than yielding a one-empty-field record', () => { + expect(parseCsvRecords('a,b\r\n\r\n1,2\r\n')).toEqual([['a', 'b'], ['1', '2']]); + }); + + it('ends the final record at end of input even without a trailing record break', () => { + expect(parseCsvRecords('a,b')).toEqual([['a', 'b']]); + }); + + it('parses with the TSV delimiter, where a comma is ordinary field text and a tab splits', () => { + expect(parseCsvRecords('a,b\tc\r\n', TSV_DELIMITER)).toEqual([['a,b', 'c']]); + }); + + it('throws CsvParseError for an unterminated quoted field, naming the field so far', () => { + expect(() => parseCsvRecords('"never closed,x\r\n')).toThrow(CsvParseError); + expect(() => parseCsvRecords('"never closed,x\r\n')).toThrow(/unterminated quoted field/); + }); + + it('throws CsvParseError for a multi-character or empty delimiter, which could never match the per-character scanner', () => { + expect(() => parseCsvRecords('a;b\r\n', ';;')).toThrow(/delimiter must be exactly one character/); + expect(() => parseCsvRecords('a;b\r\n', '')).toThrow(/delimiter must be exactly one character/); + }); +}); + +describe('parseCsvRecords/quoteCsvField round trips', () => { + const fieldsCases: readonly (readonly string[])[] = [ + ['plain', 'fields', 'only'], + ['contains,comma', 'second'], + ['say "hi"', 'doubled "" quote'], + ['multi\r\nline', 'field'], + ['5" drive', 'mid-field quote'], + ['trailing empty', ''], + ]; + + it('every field case round-trips through quoteCsvField joined with CRLF and back', () => { + // A record of one empty field alone is excluded: the parser drops blank records, so such a record cannot round-trip (documented behaviour, covered above). + for (const fields of fieldsCases) { + const encoded = `${fields.map((field) => quoteCsvField(field)).join(',')}\r\n`; + expect(parseCsvRecords(encoded)).toEqual([fields]); + } + }); + + it('round-trips under the TSV delimiter, quoting on tab rather than comma', () => { + const fields = ['contains,comma', 'contains\ttab', 'plain']; + const encoded = `${fields.map((field) => quoteCsvField(field, TSV_DELIMITER)).join(TSV_DELIMITER)}\r\n`; + expect(encoded).toBe('contains,comma\t"contains\ttab"\tplain\r\n'); + expect(parseCsvRecords(encoded, TSV_DELIMITER)).toEqual([fields]); + }); +}); + +describe('quoteCsvField', () => { + it('writes a field with no delimiter, quote, or line break bare', () => { + expect(quoteCsvField('plain')).toBe('plain'); + expect(quoteCsvField('plain,with,commas', TSV_DELIMITER)).toBe('plain,with,commas'); + }); + + it('wraps and doubles exactly when a bare field would re-parse as more than one field or a truncated one', () => { + expect(quoteCsvField('a,b')).toBe('"a,b"'); + expect(quoteCsvField('say "hi"')).toBe('"say ""hi"""'); + expect(quoteCsvField('line\r\nbreak')).toBe('"line\r\nbreak"'); + }); + + it('defaults to the package-wide csv delimiter constant', () => { + expect(DEFAULT_CSV_DELIMITER).toBe(','); + expect(quoteCsvField('a\tb')).toBe('a\tb'); + }); +}); diff --git a/src/csv/records.ts b/src/csv/records.ts new file mode 100644 index 000000000..0c93f5c8b --- /dev/null +++ b/src/csv/records.ts @@ -0,0 +1,103 @@ +// The shared RFC 4180 record layer for csv: one parser (parseCsvRecords) and one quoting writer (quoteCsvField), used by the read path (src/csv/read.ts), the write path (src/csv/write.ts), and src/odb/csv.ts's odbToCsv alike. Before this module existed the quoting lived privately in odb/csv.ts as csvField/CSV_QUOTE_NEEDED_RE; generalising it to also carry the active delimiter (a tab, for TSV) and moving it here means every csv emitter in the package writes the identical dialect by construction -- one implementation, no drift. +// +// The dialect, RFC 4180 (https://www.rfc-editor.org/rfc/rfc4180) with two deliberate tolerances: a field containing the delimiter, a double quote, CR, or LF is wrapped in double quotes with embedded double quotes doubled; records are joined with CRLF. Tolerance one: RFC 4180 mandates CRLF record breaks, but real-world files arrive LF-only (and classic Mac exports arrive CR-only), so the parser accepts all three as a break while the writer always emits CRLF -- accepting a lone break never mis-parses a conforming file. Tolerance two: RFC 4180 allows a quote only immediately after a record break or delimiter; a quote appearing mid-field is taken as a literal character rather than a parse error, matching what spreadsheet exporters actually emit for text like {5" drive}. + +export const DEFAULT_CSV_DELIMITER = ','; +export const TSV_DELIMITER = '\t'; + +export class CsvParseError extends Error { + constructor(message: string) { + super(message); + this.name = 'CsvParseError'; + } +} + +// A delimiter other than exactly one character can never match the scanner's per-character comparison, so a multi-character or empty delimiter would silently parse the whole file as one giant field per line -- rejected here, at the boundary, instead. +function requireSingleCharacterDelimiter(delimiter: string): void { + if (delimiter.length !== 1) { + throw new CsvParseError(`delimiter must be exactly one character, got ${JSON.stringify(delimiter)}`); + } +} + +// A record consisting of exactly one empty field is a blank line -- RFC 4180 gives it no meaning, and every spreadsheet importer drops it. Filtered after parsing so the reader never sees phantom rows. +function isBlankRecord(record: readonly string[]): boolean { + return record.length === 1 && record[0] === ''; +} + +export function parseCsvRecords(text: string, delimiter: string = DEFAULT_CSV_DELIMITER): string[][] { + requireSingleCharacterDelimiter(delimiter); + const records: string[][] = []; + let record: string[] = []; + let field = ''; + let inQuotedField = false; + let fieldStarted = false; + + const endField = (): void => { + record.push(field); + field = ''; + fieldStarted = false; + }; + const endRecord = (): void => { + endField(); + records.push(record); + record = []; + }; + + let index = 0; + while (index < text.length) { + const ch = text[index]; + if (inQuotedField) { + if (ch === '"') { + // A doubled quote inside a quoted field is one literal quote (RFC 4180 2.7); a lone quote closes the field. + if (text[index + 1] === '"') { + field += '"'; + index += 2; + continue; + } + inQuotedField = false; + index += 1; + continue; + } + field += ch; + index += 1; + continue; + } + if (ch === '"' && !fieldStarted) { + inQuotedField = true; + fieldStarted = true; + index += 1; + continue; + } + if (ch === delimiter) { + endField(); + index += 1; + continue; + } + if (ch === '\r' || ch === '\n') { + if (ch === '\r' && text[index + 1] === '\n') { + index += 2; + } else { + index += 1; + } + endRecord(); + continue; + } + field += ch; + fieldStarted = true; + index += 1; + } + + if (inQuotedField) { + throw new CsvParseError(`unterminated quoted field: no closing double quote before end of input (field so far: ${field.slice(0, 40)})`); + } + // A trailing record break already ended the final record above; input not ending in a break leaves a partial field (or a field-only record) to end here. + if (fieldStarted || field !== '' || record.length > 0) { + endRecord(); + } + return records.filter((candidate) => !isBlankRecord(candidate)); +} + +// The writer's half of the dialect: quote exactly when a bare field would re-parse as more than one field or a truncated one, otherwise write it bare. Delimiter-parameterised so TSV output quotes on tab rather than comma. +export function quoteCsvField(field: string, delimiter: string = DEFAULT_CSV_DELIMITER): string { + return field.includes(delimiter) || field.includes('"') || field.includes('\r') || field.includes('\n') ? `"${field.replaceAll('"', '""')}"` : field; +} diff --git a/src/csv/text.ts b/src/csv/text.ts new file mode 100644 index 000000000..b32bf0592 --- /dev/null +++ b/src/csv/text.ts @@ -0,0 +1,21 @@ +// decodeCsvText/encodeCsvText: the byte <-> text boundary for csv, exactly mirroring src/markdown/text.ts's own pair for markdown. csv has no upstream codec package (unlike markdown-codec), so both the decode/encode pair and the invalid-UTF-8 error live here. A fresh TextDecoder/TextEncoder is constructed per call, never module-level cached -- this package's own sideEffects:false convention (package.json) means nothing here creates shared mutable state at import time. +// +// decodeCsvText uses a fatal-mode TextDecoder specifically so a non-UTF-8 input throws here, at the boundary, rather than silently producing U+FFFD replacement characters that would then corrupt every downstream cell value the read path produces -- the identical reasoning src/model/bytes.ts's own CsvBytesSchema documents for the schema-validation path. This function is the enforcement point for the ergonomic conversions (csvToPdf/csvToXlsx/csvToOds/csvToMarkdown) that bypass the schema and call readCsvContent directly on already-checked bytes. +export class CsvInvalidUtf8Error extends Error { + constructor() { + super('csv text must be well-formed UTF-8'); + this.name = 'CsvInvalidUtf8Error'; + } +} + +export function decodeCsvText(bytes: Uint8Array): string { + try { + return new TextDecoder('utf-8', { fatal: true }).decode(bytes); + } catch { + throw new CsvInvalidUtf8Error(); + } +} + +export function encodeCsvText(text: string): Uint8Array { + return new TextEncoder().encode(text); +} diff --git a/src/csv/write.ts b/src/csv/write.ts new file mode 100644 index 000000000..0e68e7e55 --- /dev/null +++ b/src/csv/write.ts @@ -0,0 +1,102 @@ +import type { ContentDocument, ContentSheet } from 'document-schema.js'; +import { DEFAULT_CSV_DELIMITER, quoteCsvField } from './records'; + +// The csv write half: a spreadsheet ContentDocument -> RFC 4180 text, emitting each cell's displayText (the cell's own printed form, independent of value.kind -- a currency cell writes "£42.50", not its numeric value). The sheet-selection contract mirrors src/odb/csv.ts's own odbToCsv table selection exactly: a named sheet must exist, a multi-sheet document requires a name, and a lone sheet is selected by default -- csv has no representation for a second sheet, so writing one is a caller decision, never a silent truncation. + +export class CsvUnsupportedDocumentKindError extends Error { + readonly kind: ContentDocument['kind']; + + constructor(kind: ContentDocument['kind']) { + super(`buildCsvText: expected a spreadsheet ContentDocument, got kind '${kind}'`); + this.name = 'CsvUnsupportedDocumentKindError'; + this.kind = kind; + } +} + +export class CsvSheetNotSpecifiedError extends Error { + readonly availableSheets: readonly string[]; + + constructor(availableSheets: readonly string[]) { + super(`buildCsvText: this document has more than one sheet (${availableSheets.join(', ')}) -- pass { sheet: '' } to select one`); + this.name = 'CsvSheetNotSpecifiedError'; + this.availableSheets = availableSheets; + } +} + +export class CsvSheetNotFoundError extends Error { + readonly sheet: string; + readonly availableSheets: readonly string[]; + + constructor(sheet: string, availableSheets: readonly string[]) { + super(`buildCsvText: sheet "${sheet}" not found -- available sheet(s): ${availableSheets.length === 0 ? '(none)' : availableSheets.join(', ')}`); + this.name = 'CsvSheetNotFoundError'; + this.sheet = sheet; + this.availableSheets = availableSheets; + } +} + +export interface BuildCsvTextOptions { + // The single-character field delimiter to write with -- ',' (DEFAULT_CSV_DELIMITER) for csv, '\t' (records.ts's TSV_DELIMITER) for TSV. + readonly delimiter?: string; + // Selects which sheet of a multi-sheet document is written. Optional only when the document has exactly one sheet. + readonly sheet?: string; +} + +function selectSheet(sheets: readonly ContentSheet[], sheetName: string | undefined): ContentSheet { + const availableNames = sheets.map((candidate) => candidate.name); + if (sheetName !== undefined) { + const found = sheets.find((candidate) => candidate.name === sheetName); + if (found === undefined) { + throw new CsvSheetNotFoundError(sheetName, availableNames); + } + return found; + } + if (sheets.length === 0) { + throw new CsvSheetNotFoundError('(unspecified)', availableNames); + } + if (sheets.length > 1) { + throw new CsvSheetNotSpecifiedError(availableNames); + } + const only = sheets[0]; + if (only === undefined) { + throw new CsvSheetNotFoundError('(unspecified)', availableNames); + } + return only; +} + +export function buildCsvText(content: ContentDocument, options?: BuildCsvTextOptions): string { + if (content.kind !== 'spreadsheet') { + throw new CsvUnsupportedDocumentKindError(content.kind); + } + const sheet = selectSheet(content.sheets, options?.sheet); + const delimiter = options?.delimiter ?? DEFAULT_CSV_DELIMITER; + + // The sheet model is sparse (cells addressed by row/column); csv text is a dense rectangle. The grid spans every populated cell AND every declared column/row index, whichever is wider/taller, so a declared-but-empty trailing row or column still writes its empty line/fields. + const fieldsByRow = new Map>(); + let maxRowIndex = -1; + let maxColumnIndex = -1; + for (const cell of sheet.cells) { + let row = fieldsByRow.get(cell.row); + if (row === undefined) { + row = new Map(); + fieldsByRow.set(cell.row, row); + } + row.set(cell.column, cell.displayText); + maxRowIndex = Math.max(maxRowIndex, cell.row); + maxColumnIndex = Math.max(maxColumnIndex, cell.column); + } + const rowCount = Math.max(sheet.rows.length, maxRowIndex + 1); + const columnCount = Math.max(sheet.columns.length, maxColumnIndex + 1); + + const lines: string[] = []; + for (let rowIndex = 0; rowIndex < rowCount; rowIndex++) { + const row = fieldsByRow.get(rowIndex); + const fields: string[] = []; + for (let columnIndex = 0; columnIndex < columnCount; columnIndex++) { + // An unpopulated grid position is an empty field: the sheet genuinely has no cell there, which is exactly what a bare empty csv field says. + fields.push(quoteCsvField(row?.get(columnIndex) ?? '', delimiter)); + } + lines.push(fields.join(delimiter)); + } + return lines.length === 0 ? '' : `${lines.join('\r\n')}\r\n`; +} diff --git a/src/index.ts b/src/index.ts index f755779ce..621ce5e07 100644 --- a/src/index.ts +++ b/src/index.ts @@ -1,4 +1,4 @@ -// documents.js's public surface: bidirectional docx/pptx/odt/odp/ods/odg <-> PDF conversion, a read+write live-view editor for docx/pptx/odt/odp/ods/odg, and a hand-written PDF codec, built on ooxml.js's lossless OOXML core and odf.js's lossless ODF core. +// documents.js's public surface: bidirectional conversion among docx/pptx/xlsx/odt/odp/ods/odg/markdown/csv and PDF (every content format's own PDF round trip plus the cross-format bridges), a read+write live-view editor for docx/pptx/odt/odp/ods/odg, and a hand-written PDF codec, built on ooxml.js's lossless OOXML core, odf.js's lossless ODF core, and markdown-codec. // --- ooxml.js's lossless OOXML core, re-exported so consumers need only this one dependency. Its own typed readers readDocx/readPptx, and the separate lossy cell-values-only readXlsx, are deliberately NOT re-exported here: readDocxContent/readPptxContent (below) already wrap readDocx/readPptx into ContentDocument, so exposing both the wrapper and the thing it wraps would be a trap -- two overlapping entry points to the same underlying read. readXlsxContent/buildXlsxPackage are the one exception -- re-exported directly further below, in the Format <-> ContentDocument readers section, rather than from here or behind a documents.js-local wrapper: unlike readDocx/readPptx, readXlsxContent already reads (and buildXlsxPackage already builds) a real spreadsheet ContentDocument on its own, so there is no wrapper to write and no second, overlapping entry point to trap a caller into picking the wrong one. readDocx's own comments/footnotes/headers/footers/numbering, which ContentDocument doesn't model at all, are not lost, though -- see readDocxExtras below, which exposes that data as its own real return type rather than by re-exporting readDocx itself. Comment/Footnote/NumberingDefinitions (ooxml.js's own types, reused by readDocxExtras' own return shape) are re-exported here since they're genuinely just data types, not a second entry point to the same read. --- export { @@ -170,8 +170,8 @@ export type { Color as LayoutColor } from 'document-schema.js'; export { COLOR_BLACK, rgbHexToColor } from 'document-schema.js'; export type { Alignment, LayoutFont } from 'document-schema.js'; export { DEFAULT_LAYOUT_FONT } from 'document-schema.js'; -// Magic-byte-validated Uint8Array schemas, so a caller passing the wrong format -- to these functions directly, or as the input/output schema half of a z.codec() below -- gets a clear Zod validation error instead of a confusing failure three layers down. The Odt/Ods/Odp/Odg schemas check the package's actual declared media type (see src/model/bytes.ts), a stronger check than Docx/PptxBytesSchema's generic ZIP-signature check. MarkdownBytesSchema is architecturally different from every other schema here -- it checks only well-formed UTF-8, since markdown has no magic bytes or format-level header of its own to check (see src/model/bytes.ts's own comment). -export { DocxBytesSchema, MarkdownBytesSchema, OdgBytesSchema, OdpBytesSchema, OdsBytesSchema, OdtBytesSchema, PdfBytesSchema, PptxBytesSchema, XlsxBytesSchema } from './model/bytes'; +// Magic-byte-validated Uint8Array schemas, so a caller passing the wrong format -- to these functions directly, or as the input/output schema half of a z.codec() below -- gets a clear Zod validation error instead of a confusing failure three layers down. The Odt/Ods/Odp/Odg schemas check the package's actual declared media type (see src/model/bytes.ts), a stronger check than Docx/PptxBytesSchema's generic ZIP-signature check. MarkdownBytesSchema and CsvBytesSchema are architecturally different from every other schema here -- each checks only well-formed UTF-8, since markdown and csv are plain text with no magic bytes or format-level header of their own to check (see src/model/bytes.ts's own comment). +export { CsvBytesSchema, DocxBytesSchema, MarkdownBytesSchema, OdgBytesSchema, OdpBytesSchema, OdsBytesSchema, OdtBytesSchema, PdfBytesSchema, PptxBytesSchema, XlsxBytesSchema } from './model/bytes'; // --- The live-view read+write editors: a real manipulation API for docx/pptx content, since ooxml.js's own typed readers explicitly forbid write-back. --- export type { CreateEmptyDocxPackageOptions } from './edit/docx/scaffold'; @@ -334,6 +334,15 @@ export { readOdfEmbeddedFormula, readOdfFormulaContent } from './odf/formula/rea export { decodeMarkdownText, encodeMarkdownText } from './markdown/text'; export { readMarkdownContent } from './markdown/read'; export { buildMarkdownText } from './markdown/write'; +// csv <-> ContentDocument -- the same independently-usable pipeline stages the other adapter families above expose: decodeCsvText/encodeCsvText are the byte<->text boundary, readCsvContent parses RFC 4180 text into a one-sheet spreadsheet ContentDocument (first record as the header row, data cells heuristically re-typed by inferCellValue -- the identical heuristic pdfToOds applies, exported standalone further above), buildCsvText writes a sheet back out emitting each cell's displayText. TSV is a delimiter option on these same stages ({ delimiter: '\t' }), not a separate format. The errors are exported beside them so a caller composing the stages directly can branch on them by class: CsvInvalidUtf8Error (malformed bytes at the decode boundary), CsvParseError (RFC 4180 violations -- an unterminated quoted field, a multi-character delimiter), CsvUnsupportedDocumentKindError (a non-spreadsheet ContentDocument), and the multi-sheet selection pair CsvSheetNotSpecifiedError/CsvSheetNotFoundError -- the identical contract odbToCsv's own table selection follows. +export { decodeCsvText, encodeCsvText } from './csv/text'; +export { CsvInvalidUtf8Error } from './csv/text'; +export { readCsvContent } from './csv/read'; +export type { ReadCsvContentOptions } from './csv/read'; +export { buildCsvText } from './csv/write'; +export type { BuildCsvTextOptions } from './csv/write'; +export { CsvSheetNotFoundError, CsvSheetNotSpecifiedError, CsvUnsupportedDocumentKindError } from './csv/write'; +export { CsvParseError } from './csv/records'; // The one-way ContentDocument -> Markdown text renderer covering all five ContentDocument kinds, not just 'wordprocessing' -- buildMarkdownText/writeMarkdown above throw MarkdownUnsupportedDocumentKindError for the other four. renderContentDocumentToMarkdown delegates straight to buildMarkdownText for 'wordprocessing' and otherwise flattens slides/sheets/drawing pages/a bare formula into the same ContentBlock vocabulary first, reporting every degrade decision through its own onDiagnostic option (see src/markdown/render.ts's own module comment). export type { MarkdownRenderDiagnostic, @@ -367,24 +376,24 @@ export { inferCellValue } from './layout/cell-typing'; export type { GridLattice } from './layout/lattice'; export { detectGridLattice } from './layout/lattice'; -// --- Fourteen ergonomic conversions (docx/pptx/odt/odp/ods/odg/xlsx/markdown <-> PDF, all round-trip both ways). xlsx<->pdf (xlsxToPdf/pdfToXlsx) composes the ods<->xlsx bridge with the ods<->pdf layout pair internally -- xlsx has no layout engine of its own -- but is a real, direct, single-call conversion pair from a caller's own point of view, matching the other thirteen's own options shape exactly. markdown<->pdf (markdownToPdf/pdfToMarkdown) DOES lay markdown out directly, reusing convertWordprocessingToLayout/reconstructWordprocessing completely unmodified -- but pdfToMarkdown is the single lossiest conversion in the whole package (see convert.ts's own top-of-file comment and the README's Fidelity section). --- -export type { DocumentToPdfOptions, PdfToDocumentOptions } from './convert/convert'; -export { docxToPdf, markdownToPdf, odgToPdf, odpToPdf, odsToPdf, odtToPdf, pdfToDocx, pdfToMarkdown, pdfToOdg, pdfToOdp, pdfToOds, pdfToOdt, pdfToPptx, pdfToXlsx, pptxToPdf, xlsxToPdf } from './convert/convert'; +// --- The ergonomic X <-> PDF conversions (docx/pptx/odt/odp/ods/odg/xlsx/markdown/csv <-> PDF, all round-trip both ways). xlsx<->pdf and csv<->pdf each compose their same-variant ods bridge with the ods<->pdf layout pair internally -- neither xlsx nor csv has a layout engine of its own -- but both are real, direct, single-call conversion pairs from a caller's own point of view. The csv pairs intersect convert.ts's own CsvReadOptions (delimiter, onCellTypeInference) / CsvWriteOptions (delimiter, sheet) into the shared options type each already uses -- the two option groups every csv-sourced/csv-targeted conversion consumes. markdown<->pdf (markdownToPdf/pdfToMarkdown) DOES lay markdown out directly, reusing convertWordprocessingToLayout/reconstructWordprocessing completely unmodified -- but pdfToMarkdown is the single lossiest conversion in the whole package (see convert.ts's own top-of-file comment and the README's Fidelity section). --- +export type { CsvReadOptions, CsvWriteOptions, DocumentToPdfOptions, PdfToDocumentOptions } from './convert/convert'; +export { csvToPdf, docxToPdf, markdownToPdf, odgToPdf, odpToPdf, odsToPdf, odtToPdf, pdfToCsv, pdfToDocx, pdfToMarkdown, pdfToOdg, pdfToOdp, pdfToOds, pdfToOdt, pdfToPptx, pdfToXlsx, pptxToPdf, xlsxToPdf } from './convert/convert'; -// --- odf (a standalone ODF formula document) -> PDF: not one of the thirteen round-trip conversions above (there is no pdfToOdf -- see convert.ts's own module comment on odfToPdf for why: recovering structured MathML from rendered glyphs is a categorically different, OCR-adjacent problem, not a geometry-reconstruction one). Renders via src/mathml's layoutFormula and the embedded STIX Two Math font, the same pipeline the odt/odp embedded-formula paths use. --- +// --- odf (a standalone ODF formula document) -> PDF: not one of the round-trip conversions above (there is no pdfToOdf -- see convert.ts's own module comment on odfToPdf for why: recovering structured MathML from rendered glyphs is a categorically different, OCR-adjacent problem, not a geometry-reconstruction one). Renders via src/mathml's layoutFormula and the embedded STIX Two Math font, the same pipeline the odt/odp embedded-formula paths use. --- export { odfToPdf } from './convert/convert'; -// Schema-validated z.codec() pairs over the conversions above (docx/pptx/odt/odp/ods/odg/xlsx/markdown bytes <-> PDF bytes), the no-extra-options form -- use docxToPdf/pdfToDocx/pptxToPdf/pdfToPptx/odtToPdf/pdfToOdt/odpToPdf/pdfToOdp/odsToPdf/pdfToOds/odgToPdf/pdfToOdg/xlsxToPdf/pdfToXlsx/markdownToPdf/pdfToMarkdown directly for cancellation or diagnostics. -export { docxPdfCodec, markdownPdfCodec, odgPdfCodec, odpPdfCodec, odsPdfCodec, odtPdfCodec, pptxPdfCodec, xlsxPdfCodec } from './convert/codec'; +// Schema-validated z.codec() pairs over the conversions above (docx/pptx/odt/odp/ods/odg/xlsx/markdown/csv bytes <-> PDF bytes), the no-extra-options form -- use the named conversion functions directly for cancellation, diagnostics, or the csv delimiter/sheet options. +export { csvPdfCodec, docxPdfCodec, markdownPdfCodec, odgPdfCodec, odpPdfCodec, odsPdfCodec, odtPdfCodec, pptxPdfCodec, xlsxPdfCodec } from './convert/codec'; -// --- Ten cross-format bridges, five pairs (odt<->docx, odp<->pptx, ods<->xlsx, markdown<->docx, markdown<->odt), bypassing PDF entirely -- see convert.ts's own module comment on this section for why these carry substantially higher fidelity than the fourteen PDF-pivot conversions above. markdownToDocx/docxToMarkdown and markdownToOdt/odtToMarkdown are hand-written bridge functions -- the composition engine's pathfinder routes them as same-variant bridge hops, and convertDocument's bridge executor runs the identical decode/read/build/encode sequence these functions already hard-code. --- +// --- The cross-format bridges: same-variant direct copies (odt<->docx, odp<->pptx, ods<->xlsx, csv<->ods, csv<->xlsx, markdown<->docx, markdown<->odt) and cross-variant semantic transforms (docx<->pptx, odt<->odp) bypass PDF entirely -- see convert.ts's own module comment on this section for why those carry substantially higher fidelity than the PDF-pivot conversions above. The remaining two pairs (xlsx<->markdown, csv<->markdown) are PDF-composed internally, the last-resort routes the pathfinder picks when no shorter path exists. --- export type { DocumentBridgeOptions } from './convert/convert'; -export { docxToMarkdown, docxToOdt, markdownToDocx, markdownToOdt, odpToPptx, odsToXlsx, odtToDocx, odtToMarkdown, pptxToOdp, xlsxToOds, docxToPptx, pptxToDocx, odtToOdp, odpToOdt, xlsxToMarkdown, markdownToXlsx } from './convert/convert'; +export { csvToMarkdown, csvToOds, csvToXlsx, docxToMarkdown, docxToOdt, markdownToCsv, markdownToDocx, markdownToOdt, odpToPptx, odsToCsv, odsToXlsx, odtToDocx, odtToMarkdown, pptxToOdp, xlsxToCsv, xlsxToOds, docxToPptx, pptxToDocx, odtToOdp, odpToOdt, xlsxToMarkdown, markdownToXlsx } from './convert/convert'; -// Schema-validated z.codec() pairs over the ten bridges above (odt bytes <-> docx bytes, odp bytes <-> pptx bytes, ods bytes <-> xlsx bytes, markdown bytes <-> docx bytes, markdown bytes <-> odt bytes), the no-extra-options form -- use odtToDocx/docxToOdt/odpToPptx/pptxToOdp/odsToXlsx/xlsxToOds/markdownToDocx/docxToMarkdown/markdownToOdt/odtToMarkdown directly for cancellation. -export { markdownDocxCodec, markdownOdtCodec, odpPptxCodec, odsXlsxCodec, odtDocxCodec, xlsxMarkdownCodec } from './convert/codec'; +// Schema-validated z.codec() pairs over the bridges above (odt bytes <-> docx bytes, odp bytes <-> pptx bytes, ods bytes <-> xlsx bytes, ods/xlsx bytes <-> csv bytes, markdown bytes <-> docx/odt bytes, csv bytes <-> markdown bytes), the no-extra-options form -- use the named bridge functions directly for cancellation or the csv delimiter/sheet options. +export { csvMarkdownCodec, markdownDocxCodec, markdownOdtCodec, odsCsvCodec, odpPptxCodec, odsXlsxCodec, odtDocxCodec, xlsxCsvCodec, xlsxMarkdownCodec } from './convert/codec'; -// --- odm (ODF master document, multiple chapters) -> PDF, the one conversion in this package shaped differently from every other: a .odm's chapters are external references (odf.js's readOdm never inlines them -- see odmToPdf's own module comment), so producing a PDF needs a caller-supplied resolveSubDocument callback to hand back each chapter's own .odt bytes. Not part of the twelve-conversion or six-bridge groups above, and not wired into the DocumentConverter port below -- see odmToPdf's own module comment for why. --- +// --- odm (ODF master document, multiple chapters) -> PDF, the one conversion in this package shaped differently from every other: a .odm's chapters are external references (odf.js's readOdm never inlines them -- see odmToPdf's own module comment), so producing a PDF needs a caller-supplied resolveSubDocument callback to hand back each chapter's own .odt bytes. Not part of the conversion or bridge groups above, and not wired into the DocumentConverter port below -- see odmToPdf's own module comment for why. --- export type { OdmToPdfOptions } from './convert/convert'; export { odmToPdf, OdmUnresolvedSectionError } from './convert/convert'; @@ -442,7 +451,7 @@ export { FirebirdBackupParseError } from './firebird/reader'; // --- The swappable conversion port, for a caller that wants to inject a different (e.g. remote) implementation later without changing call sites. --- export type { ConversionOptions, ConversionRequest, ConversionResult, Diagnostic, DocumentConverter, DocumentFormat, DocumentPayload } from './convert/port'; -// DocumentFormat's own Zod schema, and every member as a plain runtime array derived from it -- for a caller that wants to enumerate or validate against the full format set (a CLI's own usage-error text, an MCP tool's JSON-schema `enum` input) without hand-writing a second copy of the ten format literals that could drift out of sync with DocumentFormat itself. +// DocumentFormat's own Zod schema, and every member as a plain runtime array derived from it -- for a caller that wants to enumerate or validate against the full format set (a CLI's own usage-error text, an MCP tool's JSON-schema `enum` input) without hand-writing a second copy of the format literals that could drift out of sync with DocumentFormat itself. export { DocumentFormatSchema, DOCUMENT_FORMATS } from './convert/port'; export { createLocalDocumentConverter } from './convert/local'; @@ -453,13 +462,13 @@ export type { UnifiedConversionOptions, ConversionPlan, CompositionHop } from '. // --- A DocumentPackage (content + its fused positions) -> any DocumentFormat's own bytes -- the reverse of what every ergonomic X-to-PDF/PDF-to-X conversion's own onDocument callback hands back -- plus the frames-to-layout inverse the pdf target rebuilds through (exported for a caller wanting the pdf-codec view of a package's positions without writing bytes). --- export { buildDocumentBytes, layoutDocumentFromPackage } from './convert/from-package'; -// --- Raw package decode/encode, dispatched by DocumentFormat -- the format-aware counterpart to ooxml.js's/odf.js's own decodePackage/encodePackage, for a caller holding a format + bytes rather than already knowing which of the two underlying codecs applies. Covers docx/pptx/xlsx (ooxml.js's OPC container) and odt/odp/ods/odg/odf (odf.js's ODF container); markdown and pdf have no raw-package concept at all and throw UnsupportedPackageFormatError. decodeOdbPackage is the .odb-specific sibling: 'odb' is deliberately not a DocumentFormat member (see src/odb/'s own Architecture/Gotchas entries), but its bytes are an ordinary ODF package decoded by the identical odf.js decodePackage every readOdb*/odbTo* function in this package already starts from -- there is no encodeOdbPackage, since nothing here ever writes a new .odb file. --- +// --- Raw package decode/encode, dispatched by DocumentFormat -- the format-aware counterpart to ooxml.js's/odf.js's own decodePackage/encodePackage, for a caller holding a format + bytes rather than already knowing which of the two underlying codecs applies. Covers docx/pptx/xlsx (ooxml.js's OPC container) and odt/odp/ods/odg/odf (odf.js's ODF container); markdown, csv, and pdf have no raw-package concept at all and throw UnsupportedPackageFormatError. decodeOdbPackage is the .odb-specific sibling: 'odb' is deliberately not a DocumentFormat member (see src/odb/'s own Architecture/Gotchas entries), but its bytes are an ordinary ODF package decoded by the identical odf.js decodePackage every readOdb*/odbTo* function in this package already starts from -- there is no encodeOdbPackage, since nothing here ever writes a new .odb file. --- export { decodeDocumentPackage, decodeOdbPackage, encodeDocumentPackage, UnsupportedPackageFormatError } from './package-codec'; // --- Every DocumentFormat's own source-embedded font faces, dispatched by format -- the DocumentFormat-aware counterpart to extractSourceFonts/FontSourcePackage above, for a caller holding a format + bytes rather than an already-decoded Package. --- export { extractSourceFontsForFormat, UnsupportedFontSourceFormatError } from './convert/document-fonts'; -// --- Cross-format metadata read/write: a document's own title/author/subject/keywords/creator/producer/created/modified, resolved (and, for setDocumentMetadata, patched) by DocumentFormat across all ten formats this package supports. --- +// --- Cross-format metadata read/write: a document's own title/author/subject/keywords/creator/producer/created/modified, resolved (and, for setDocumentMetadata, patched) by DocumentFormat across every format this package supports -- csv reads honestly empty (RFC 4180 text has no metadata container) and is rejected as a setDocumentMetadata source/target for exactly that reason. --- export type { ReadDocumentMetadataOptions } from './metadata/read'; export { readDocumentMetadata } from './metadata/read'; export type { MetadataOverrides, SetDocumentMetadataOptions } from './metadata/write'; diff --git a/src/metadata/write.test.ts b/src/metadata/write.test.ts index bd5cd83c1..3c1d5d0f6 100644 --- a/src/metadata/write.test.ts +++ b/src/metadata/write.test.ts @@ -81,6 +81,10 @@ describe('setDocumentMetadata: rejected formats', () => { expect(() => setDocumentMetadata('odf', 'odf', new Uint8Array(), {})).toThrow(/no write path back out/); }); + it('rejects csv as a source or target, naming that RFC 4180 text has no metadata container', () => { + expect(() => setDocumentMetadata('csv', 'csv', new TextEncoder().encode('A,B\n1,2\n'), {})).toThrow(/no metadata container/); + }); + it('rejects a source that is neither pdf nor a rebuild format, even when target is a rebuild format', () => { expect(() => setDocumentMetadata('pdf', 'docx', new Uint8Array(), {})).toThrow(/must be the same format \(or both 'pdf'\)/); }); diff --git a/src/metadata/write.ts b/src/metadata/write.ts index 1d8667563..c92f0efb2 100644 --- a/src/metadata/write.ts +++ b/src/metadata/write.ts @@ -6,7 +6,7 @@ import type { DocumentCodecOptions } from '../codecs/registry'; import type { DocumentFormat } from '../convert/port'; import { throwIfAborted } from '../ports/abort'; -// Every format whose own ContentDocument setDocumentMetadata can patch a metadata field on and rebuild from scratch through -- the eight formats sharing the readXContent -> buildXPackage round trip. xlsx joined this set once DOCUMENT_FORMAT_CODECS.xlsx.content gained a real read/write pair (src/codecs/registry.ts): it now fits the identical shape docx/pptx/odt/odp/ods/odg/markdown already share, so there is no reason left to special-case it out. Deliberately does NOT include 'pdf': a PDF's metadata is patched directly on its own LayoutDocument (see setDocumentMetadata below), never through this ContentDocument rebuild path at all. +// Every format whose own ContentDocument setDocumentMetadata can patch a metadata field on and rebuild from scratch through -- the eight formats sharing the readXContent -> buildXPackage round trip. xlsx joined this set once DOCUMENT_FORMAT_CODECS.xlsx.content gained a real read/write pair (src/codecs/registry.ts): it now fits the identical shape docx/pptx/odt/odp/ods/odg/markdown already share, so there is no reason left to special-case it out. Deliberately does NOT include 'pdf': a PDF's metadata is patched directly on its own LayoutDocument (see setDocumentMetadata below), never through this ContentDocument rebuild path at all. Nor 'csv': a csv round trip technically exists through the registry codec, but RFC 4180 text has no metadata container at all -- a rebuild would "succeed" and silently drop the override -- so classifyWritePath rejects it explicitly below with that reason rather than letting it fall through to the generic format-mismatch message. const REBUILD_FORMATS: Readonly> = { docx: true, pptx: true, @@ -70,6 +70,9 @@ function classifyWritePath(source: DocumentFormat, target: DocumentFormat): Writ if (target === 'odf' || source === 'odf') { return { errorMessage: "'odf' (a standalone formula document) is not a supported setDocumentMetadata source or target -- it has no write path back out at all" }; } + if (target === 'csv' || source === 'csv') { + return { errorMessage: "'csv' is not a supported setDocumentMetadata source or target -- RFC 4180 text has no metadata container, so a rebuild would silently drop the override. Convert to or from csv first, then patch metadata on the package format." }; + } if (!isRebuildFormat(source) || !isRebuildFormat(target)) { return { errorMessage: `setDocumentMetadata only patches metadata in place; it does not convert format -- source ('${source}') and target ('${target}') must be the same format (or both 'pdf'). Convert first if you need a different target format.` }; } @@ -85,7 +88,7 @@ export interface SetDocumentMetadataOptions { readonly images?: MarkdownImageResolver; } -// Patches a document's own title/author/subject/keywords, leaving every other field and every other flag as-is. Two write paths: a pdf source/target patches the metadata directly on the parsed PDF (writePdf), with no layout engine involved at all -- genuinely lossless for everything else on the page. Every other supported format (docx, pptx, odt, odp, ods, odg, markdown) rebuilds a fresh package from that format's own ContentDocument -- see classifyWritePath's own comment for exactly what that costs for docx specifically. Overrides are applied via mergeMetadata: a field omitted from `overrides` is left exactly as the source document already had it. +// Patches a document's own title/author/subject/keywords, leaving every other field and every other flag as-is. Two write paths: a pdf source/target patches the metadata directly on the parsed PDF (writePdf), with no layout engine involved at all -- genuinely lossless for everything else on the page. Every other supported format (docx, pptx, xlsx, odt, odp, ods, odg, markdown) rebuilds a fresh package from that format's own ContentDocument -- see classifyWritePath's own comment for exactly what that costs for docx specifically. Overrides are applied via mergeMetadata: a field omitted from `overrides` is left exactly as the source document already had it. export function setDocumentMetadata(sourceFormat: DocumentFormat, targetFormat: DocumentFormat, bytes: Uint8Array, overrides: MetadataOverrides, options?: SetDocumentMetadataOptions): Uint8Array { const writePath = classifyWritePath(sourceFormat, targetFormat); if ('errorMessage' in writePath) { diff --git a/src/model/bytes.test.ts b/src/model/bytes.test.ts index dbfad61ac..78bc8bf7e 100644 --- a/src/model/bytes.test.ts +++ b/src/model/bytes.test.ts @@ -1,6 +1,6 @@ import { ODF_MEDIA_TYPES, zipPackage } from 'odf.js'; import { describe, expect, it } from 'vitest'; -import { DocxBytesSchema, MarkdownBytesSchema, OdgBytesSchema, OdpBytesSchema, OdsBytesSchema, OdtBytesSchema, PdfBytesSchema, PptxBytesSchema } from './bytes'; +import { CsvBytesSchema, DocxBytesSchema, MarkdownBytesSchema, OdgBytesSchema, OdpBytesSchema, OdsBytesSchema, OdtBytesSchema, PdfBytesSchema, PptxBytesSchema } from './bytes'; const zipBytes = new Uint8Array([0x50, 0x4b, 0x03, 0x04, 0, 0, 0, 0]); const pdfBytes = new TextEncoder().encode('%PDF-1.7\n%\xe2\xe3\xcf\xd3\n'); @@ -111,4 +111,16 @@ describe('bytes', () => { expect(MarkdownBytesSchema.safeParse(zipBytes).success).toBe(true); expect(MarkdownBytesSchema.safeParse(pdfBytes).success).toBe(true); }); + + // CsvBytesSchema shares MarkdownBytesSchema's architecture exactly: RFC 4180 defines no magic bytes either, so the schema checks only well-formed UTF-8 -- the same validation gap, stated here for the same reason. + it('CsvBytesSchema accepts well-formed UTF-8 csv text, including bytes that would fail every structure-checking schema above', () => { + const csvTextBytes = new TextEncoder().encode('Name,Amount\nWidget,42.5\n'); + expect(CsvBytesSchema.parse(csvTextBytes)).toBe(csvTextBytes); + expect(DocxBytesSchema.safeParse(csvTextBytes).success).toBe(false); + expect(PdfBytesSchema.safeParse(csvTextBytes).success).toBe(false); + }); + + it('CsvBytesSchema rejects malformed UTF-8', () => { + expect(CsvBytesSchema.safeParse(new Uint8Array([0xff, 0xfe, 0x00])).success).toBe(false); + }); }); diff --git a/src/model/bytes.ts b/src/model/bytes.ts index 58e6c84be..9afab2e86 100644 --- a/src/model/bytes.ts +++ b/src/model/bytes.ts @@ -130,3 +130,6 @@ function isWellFormedUtf8Text(bytes: Uint8Array): boolean { } export const MarkdownBytesSchema = z.instanceof(Uint8Array).refine(isWellFormedUtf8Text, { message: 'not well-formed UTF-8 text' }); + +// CsvBytesSchema rests on the identical no-magic-bytes architecture as MarkdownBytesSchema above: csv shares markdown's plain-text nature (no header, no reserved byte sequence -- RFC 4180 text is just fields and delimiters), so well-formed UTF-8 is the one honest bytes-level check, and the same fatal-decode refinement covers both. The parse errors that ARE specific to csv (an unterminated quoted field) surface as CsvParseError from parseCsvRecords, which is where the text is actually understood. +export const CsvBytesSchema = z.instanceof(Uint8Array).refine(isWellFormedUtf8Text, { message: 'not well-formed UTF-8 text' }); diff --git a/src/odb/csv.ts b/src/odb/csv.ts index de575dd01..b14d20a01 100644 --- a/src/odb/csv.ts +++ b/src/odb/csv.ts @@ -1,7 +1,8 @@ import type { HsqldbTable } from '../hsqldb/script'; import { displayTextFor } from '../hsqldb/script'; +import { quoteCsvField } from '../csv/records'; -// Writes exactly one named HsqldbTable as CSV bytes -- no ContentSheet/xlsx machinery involved at all, since CSV needs nothing beyond the table's own column names and each cell's own display text. RFC 4180-style quoting: a field containing a comma, double quote, or newline is wrapped in double quotes with any embedded double quote doubled; every other field is written bare. +// Writes exactly one named HsqldbTable as CSV bytes -- no ContentSheet/xlsx machinery involved at all, since CSV needs nothing beyond the table's own column names and each cell's own display text. RFC 4180 quoting is the shared src/csv/records.ts quoteCsvField (the identical function src/csv/write.ts writes ContentSheet cells through), so every csv this package emits speaks one dialect by construction. export class OdbTableNotSpecifiedError extends Error { readonly availableTables: readonly string[]; @@ -25,12 +26,6 @@ export class OdbTableNotFoundError extends Error { } } -const CSV_QUOTE_NEEDED_RE = /[",\n\r]/; - -function csvField(value: string): string { - return CSV_QUOTE_NEEDED_RE.test(value) ? `"${value.replace(/"/g, '""')}"` : value; -} - function selectTable(tables: readonly HsqldbTable[], tableName: string | undefined): HsqldbTable { const availableNames = tables.map((table) => table.tableName); if (tableName !== undefined) { @@ -55,9 +50,9 @@ function selectTable(tables: readonly HsqldbTable[], tableName: string | undefin export function buildOdbTableCsv(tables: readonly HsqldbTable[], tableName: string | undefined): Uint8Array { const table = selectTable(tables, tableName); - const lines: string[] = [table.columns.map((column) => csvField(column.name)).join(',')]; + const lines: string[] = [table.columns.map((column) => quoteCsvField(column.name)).join(',')]; for (const row of table.rows) { - lines.push(row.map((cell) => csvField(displayTextFor(cell))).join(',')); + lines.push(row.map((cell) => quoteCsvField(displayTextFor(cell))).join(',')); } return new TextEncoder().encode(`${lines.join('\r\n')}\r\n`); } diff --git a/src/package-codec.ts b/src/package-codec.ts index 24849871d..b3fb5b277 100644 --- a/src/package-codec.ts +++ b/src/package-codec.ts @@ -27,7 +27,7 @@ function isOdfPackageFormat(format: DocumentFormat): format is keyof typeof ODF_ return format in ODF_PACKAGE_FORMATS; } -// A recognised DocumentFormat that nonetheless has no raw-package concept at all (markdown is plain text, not a zip container; pdf is its own binary format, not OPC/ODF) -- a named class, matching this package's own convention for every other "recognised but unsupported" input (UnsupportedFontSourceFormatError, OdbUnsupportedFormatError, ...), so a caller can narrow on it with instanceof rather than string-matching a thrown Error's own message. +// A recognised DocumentFormat that nonetheless has no raw-package concept at all (markdown and csv are plain text, not zip containers; pdf is its own binary format, not OPC/ODF) -- a named class, matching this package's own convention for every other "recognised but unsupported" input (UnsupportedFontSourceFormatError, OdbUnsupportedFormatError, ...), so a caller can narrow on it with instanceof rather than string-matching a thrown Error's own message. export class UnsupportedPackageFormatError extends Error { readonly format: DocumentFormat; @@ -38,7 +38,7 @@ export class UnsupportedPackageFormatError extends Error { } } -// Decodes a DocumentFormat's own raw bytes into its underlying Package (parts records -- the ooxml.js/odf.js container both packages independently define, confirmed structurally interchangeable by src/interop.test.ts, so one Package type genuinely covers both branches here). docx/pptx/xlsx decode through ooxml.js's own OPC decoder; odt/odp/ods/odg/odf through odf.js's. markdown and pdf have no raw-package concept at all and throw UnsupportedPackageFormatError. +// Decodes a DocumentFormat's own raw bytes into its underlying Package (parts records -- the ooxml.js/odf.js container both packages independently define, confirmed structurally interchangeable by src/interop.test.ts, so one Package type genuinely covers both branches here). docx/pptx/xlsx decode through ooxml.js's own OPC decoder; odt/odp/ods/odg/odf through odf.js's. markdown, csv, and pdf have no raw-package concept at all and throw UnsupportedPackageFormatError. export function decodeDocumentPackage(format: DocumentFormat, bytes: Uint8Array): Package { if (isOoxmlPackageFormat(format)) { return decodeOoxmlPackage(bytes);