Skip to content

feat(epub): metadata front matter and figure caption dedupe - #4

Open
mathurshubham wants to merge 1 commit into
firecrawl:mainfrom
mathurshubham:feat/epub-front-matter-clean-output
Open

feat(epub): metadata front matter and figure caption dedupe#4
mathurshubham wants to merge 1 commit into
firecrawl:mainfrom
mathurshubham:feat/epub-front-matter-clean-output

Conversation

@mathurshubham

@mathurshubham mathurshubham commented Aug 4, 2026

Copy link
Copy Markdown

What

Two EPUB-focused output improvements, targeting the completeness and
cleanliness axes where EPUB trailed the other formats in the benchmark
(README: EPUB 74, the only format anydoc did not lead).

A third change — stripping the whole-heading **bold** that pandoc's EPUB
CSS forced onto headings — was originally part of this PR but landed
upstream independently (2d3f923), so it has been dropped here. This PR now
covers metadata front matter and figure-caption dedupe only.

1. OPF metadata → YAML front matter (was: duplicate title, dropped metadata)

epub/mod.rs read only <title> and injected it as a body H1. Most books
already carry their own title page in the content, so the output was
double-titled; author, language, date, and publisher were dropped entirely.

Now Dublin Core metadata renders as a ----fenced YAML front matter block:

  • New DocumentMeta on the model (title, authors, language, date,
    publisher, description), mirrored in the Node and Python bindings.
  • Parses the OPF <metadata> descendants (so EPUB2's <dc-metadata>
    nesting is covered), scoped to <metadata> so a stray <title> elsewhere
    can't leak in.
  • Main title chosen via the EPUB3 refinement
    <meta property="title-type" refines="#id">main</meta>, falling back to
    the first non-empty dc:title.
  • Keeps every dc:creator; a single author is a scalar, multiple render as
    a YAML list. Repeated dc:date (EPUB2 opf:event) takes the first
    non-empty.
  • All values are quoted and escaped, so colons/quotes/dashes stay valid YAML.

Only EPUB populates meta; every other format leaves it default and renders
no front matter.

2. Duplicated figure captions removed

pandoc wraps a lone image as
<figure><img alt="x"><figcaption>x</figcaption></figure>, which the
converter walked transparently, printing the text twice (tiny dot tiny dot).

<figure> now has its own handler: a <figcaption> matching a contained
image's alt (normalized: trimmed, whitespace-collapsed, case-folded) is
dropped; a distinct caption is kept as its own paragraph with its own CSS
styling, display: none respected, and its id kept as an anchor. Image
alts are collected from figure descendants, so wrapped <a><img></a>
counts.

Real-world before/after

Converted three Project Gutenberg EPUBs with main and with this branch;
every changed line is one of the intended improvements, with no other drift.
Conversion time is unchanged.

Alice's Adventures in Wonderland — author/date/language were dropped
entirely before:

-# Alice's Adventures in Wonderland
+---
+title: "Alice's Adventures in Wonderland"
+author: "Lewis Carroll"
+language: "en"
+date: "2008-06-27"
+---

Pride and Prejudice and Frankenstein gain their author, date, and
language the same way. (These books contain no <figcaption>, so the
caption dedupe is exercised by the fixtures and the pandoc-built book.epub
snapshot instead.)

Tests

  • Rust: 202 tests pass (cargo test --locked); fmt + clippy -D warnings
    clean.
    • New renderer unit tests: front-matter shape / escaping / multi-author
      list / empty-meta.
    • New html.rs tests: caption dedupe, distinct caption kept, empty alt,
      wrapped <a><img>, hidden figcaption, figcaption's own CSS styling.
    • New integration test: epub_metadata_is_parsed_into_the_model.
  • Node bindings: pass, index.d.ts regenerated with DocumentMeta.
  • Python bindings: pass, incl. the stub-coverage guard (DocumentMeta
    registered in lib.rs, .pyi, and __init__.__all__).
  • New handmade fixtures handmade-metadata.epub and handmade-figure.epub.

Snapshot changes

  • EPUB snapshots gain front matter and lose the injected title H1; the
    book.epub image line loses its doubled caption.
  • No other format's output moved — the new Document.meta field is inert
    everywhere except EPUB.

Notes

  • Benchmark movement is a goal, not a gate: the EPUB judge/metrics ground
    truth is built from source XHTML and excludes OPF metadata, so front matter
    may not raise the score even though the output is more complete.
  • book.epub's snapshot now includes pandoc's build timestamp
    (date: "2026-07-30T23:11:58Z", from the dc:date pandoc stamps at build
    time), so regenerating that fixture will drift this one snapshot line. That
    is inherent to reading dc:date, not a bug.
  • The bench corpus isn't redistributable, so no before/after score is
    included; happy to iterate if a corpus run surfaces anything.

@mathurshubham
mathurshubham force-pushed the feat/epub-front-matter-clean-output branch from d31f7b0 to 7a5327a Compare August 5, 2026 07:19
Three EPUB-focused output improvements, targeting the completeness and
cleanliness axes where EPUB trailed in the benchmark.

- OPF Dublin Core metadata now renders as a YAML front matter block instead
  of a synthetic title heading. Adds `DocumentMeta` to the model (mirrored in
  the Node and Python bindings), parses the OPF `<metadata>` descendants
  (EPUB2 nesting included), selects the main title via the EPUB3 `title-type`
  refinement, keeps every `dc:creator`, and takes the first `dc:date`. Most
  books already carry their own title page, so injecting the OPF title as a
  body heading double-titled them.

- Headings no longer emit a redundant whole-heading bold/italic that the `#`
  prefix already conveys. Done in the Markdown renderer, so it cleans headings
  from every format (doc, docx, odt, rtf, epub); partial emphasis is kept and
  the model still carries the resolved style.

- A `<figure>`'s `<figcaption>` is dropped when it merely repeats a contained
  image's alt text (a common export shape that otherwise printed the caption
  twice); a distinct caption is kept as its own paragraph. Image alts are
  collected from figure descendants, so wrapped `<a><img></a>` counts.

New handmade fixtures cover metadata and figure handling. Snapshots updated;
only EPUB output gains front matter, and the heading cleanup accounts for the
cross-format snapshot changes.
@mathurshubham
mathurshubham force-pushed the feat/epub-front-matter-clean-output branch from 7a5327a to 8ca81e9 Compare August 6, 2026 08:40
@mathurshubham

Copy link
Copy Markdown
Author

@tomsideguide would appreciate a look at this one when you get a chance, I've kept it current with main and it's ready to review.
Since it was opened, your commit 2d3f923 landed the heading-emphasis cleanup that was originally this PR's second change, so I've dropped that part and rebased; the PR is now just the two EPUB pieces, both still unaddressed on main:

  • OPF metadata → YAML front matter — EPUB is the one format in the README benchmark anydoc doesn't lead, and completeness is where it loses: author, language, and date are currently dropped entirely, and the title gets duplicated as a synthetic body H1. For the LLM-ingestion use case the README leads with, front matter is the difference between a consumer knowing what document it's reading and guessing.

  • Figure-caption dedupe — pandoc-produced EPUBs (a large share of real ones) print every lone image's text twice (tiny dot tiny dot).

One design note worth your eyes specifically: the DocumentMeta model field is deliberately format-agnostic — docx (core.xml), ODF (meta.xml), and RTF (\info) all carry the same Dublin Core-ish metadata, so this is groundwork any of those formats can populate later without another model change. Node and Python bindings are mirrored so to_document stays consistent across surfaces.

Happy to split, trim, or rework any part of it if you'd rather take a different shape.

@mathurshubham mathurshubham changed the title feat(epub): metadata front matter, cleaner headings and figure captions feat(epub): metadata front matter and figure caption dedupe Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant