A headline about things
The first paragraph has a sentence that ends here, with commas, and more commas, for scoring.
diff --git a/apps/reader/src/use-web-capture.ts b/apps/reader/src/use-web-capture.ts index affa665e..67577344 100644 --- a/apps/reader/src/use-web-capture.ts +++ b/apps/reader/src/use-web-capture.ts @@ -54,7 +54,7 @@ export function useWebCapture( setOffer(null); setNotice(null); try { - // Readability, the metadata extractors and PDFium load only when a source is added. + // The article extractor, metadata extractors and PDFium load only when a source is added. const [flow, { readerSourceAdditionServices }] = await Promise.all([ import("./source-addition-flow.js"), import("./source-addition-services.js"), diff --git a/docs/extension-capture-improvements.md b/docs/extension-capture-improvements.md index 075f1ecd..c4ee719f 100644 --- a/docs/extension-capture-improvements.md +++ b/docs/extension-capture-improvements.md @@ -164,7 +164,7 @@ Selection capture targets the top-level page, not cross-origin embedded frames o shadow roots (capture includes it; the live selection index does not). Chrome's PDF viewer does not expose selections, so PDFs are highlighted in Reader. Live highlights are a projection, not the canonical annotation. Same-panel retry identity is not cross-window atomic deduplication. Missing -passages in Readability's saved copy require opening that copy in Reader and selecting the +passages in the extracted reading copy require opening that copy in Reader and selecting the intended text there; Reader will not silently substitute the live page or overwrite the archive. The service worker opens its own Connect session for page status; concurrent use with the panel relies on the SDK's grants tolerating two sessions in one extension origin. diff --git a/docs/web-capture-extraction.md b/docs/web-capture-extraction.md new file mode 100644 index 00000000..929c5908 --- /dev/null +++ b/docs/web-capture-extraction.md @@ -0,0 +1,64 @@ +# Web capture extraction + +New captures in Reader and the MV3 extension share `webCaptureImport` in +`packages/web-capture`. It loads the pinned `defuddle@0.19.5` browser core only +when saving, never `defuddle/full` or `defuddle/node`. The browser bundle is +self-contained; npm also installs CLI/optional dependencies, but those are not +part of either application's extractor chunk. + +## Preserve the reading evidence + +Defuddle runs synchronously on a clone of the captured document. Our scholarly +metadata, JSON-LD and publisher-tag pipeline still reads the untouched original; +publisher authors retain their existing precedence. Output remains HTML passed +through `prepareHtmlDocument`, with a separate unchanged archive. + +We disable standardization, content-pattern removal, partial-selector removal, +hidden-element removal and small-image removal. The comparison found real losses +with the defaults: truncated Wikipedia sections, code figures and captions in +the Rust Book, arXiv tables/short summaries, and hidden sidenotes. These settings +prefer complete evidence and original citation/code/math structure over extra +cleanup. Exact-selector cleanup and content scoring remain enabled. Some +boilerplate and duplicate math representations can consequently survive. + +As a last-resort safety check, an extraction with fewer than half the page's +words (at least 400 page words, excluding navigation, header/footer, scripts, +styles, templates and explicitly hidden elements) tries the separately lazy, +pinned Readability 0.6.0. It switches only if Readability recovers more than both +1.5 times the extracted words and 200 additional words. This is a truncation +heuristic, not proof that every passage survived, and does not detect small losses. + +`capture.extractor` records `defuddle@0.19.5` or `mozilla-readability@0.6.0` on the +new source note. The field is optional for historical captures. There is no +backfill, opening-time extraction, or migration of existing `readable.html`: +replacing that text would invalidate its highlights. + +## Network audit (0.19.5) + +The installed browser-core sources were searched for `fetch`, XHR and other +network transports. All extractor HTTP calls are in these async paths: + +| Extractor | Network paths | +| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | +| X oEmbed | FxTwitter API (including articles), then Twitter publish/oEmbed | +| Reddit | old.reddit HTML, then the Reddit Atom feed (`/.rss`) | +| YouTube | InnerTube player (Android, iOS, web clients), next/chapters, caption-track XML URLs; async transcript-panel interaction can also initiate page requests | +| Bilibili | view metadata, player WBI/v2 and v2 variants, subtitle-track URLs | +| C2 wiki | `c2.com/wiki/remodel/pages/` JSON | + +They are reachable through `parseAsync()` or `fetchAsyncVariables()`, neither of +which we call. We additionally set `useAsync: false` and supply a throwing fetch +override rather than letting any extractor use global fetch. The Node/CLI +`fetch.js` has direct fetch/proxy HTTP transports, but is not in the browser +entry point. No extractor XHR calls were found. + +Tests spy on fetch/XHR for all five async-extractor families, including empty +local shells and inline YouTube caption data. Resource/script sanitization and +original-document immutability are also covered. A separate real Chromium check +observed zero external requests while extracting the 39-page comparison corpus, +the regression fixtures and empty X/Reddit/YouTube/Bilibili/C2 shells. Happy DOM's +iframe loading is disabled in the unit tests to model browsers' inert parsed +documents; Chromium also checked that generated embeds do not initiate requests. + +When upgrading Defuddle, repeat the source audit and comparison rather than +assuming the version's network or cleanup behavior remains unchanged. diff --git a/eslint.config.js b/eslint.config.js index eb9ae9d1..d37d6631 100644 --- a/eslint.config.js +++ b/eslint.config.js @@ -15,6 +15,7 @@ export default tseslint.config( "**/coverage/**", "**/node_modules/**", "**/.wrangler/**", + "**/.scratch/**", "apps/capacitor/android/**", "apps/capacitor/ios/**", ], diff --git a/packages/connect/src/source-frontmatter.ts b/packages/connect/src/source-frontmatter.ts index e9df00e1..4c2fdca4 100644 --- a/packages/connect/src/source-frontmatter.ts +++ b/packages/connect/src/source-frontmatter.ts @@ -77,6 +77,7 @@ function captureFields( captured_at: capture.retrievedAt, submitted_url: capture.submittedUrl, canonical_url: capture.canonicalUrl, + ...(capture.extractor ? { extractor: capture.extractor } : {}), }, }; } diff --git a/packages/connect/src/source-imports-web-capture.test.ts b/packages/connect/src/source-imports-web-capture.test.ts index 604deb6e..a8d55995 100644 --- a/packages/connect/src/source-imports-web-capture.test.ts +++ b/packages/connect/src/source-imports-web-capture.test.ts @@ -55,66 +55,70 @@ describe("ConnectSourceImportRepository web capture provenance", () => { ); }); - it("preserves raw and readable web representations with capture metadata", async () => { - const create = vi.fn(() => Promise.resolve(success(recordDocument()))); - const upload = vi.fn((path: string) => - Promise.resolve( - fileDescriptor({ - fileId: path.includes("archive") ? "file-archive" : "file-readable", - path, - mediaType: "text/html", - }), - ), - ); - const repository = new ConnectSourceImportRepository( - { create } as unknown as ReaderConnectClient, - { upload }, - ); + it.each([undefined, "defuddle@0.19.5", "mozilla-readability@0.6.0"])( + "preserves raw and readable web representations with extractor %s", + async (extractor) => { + const create = vi.fn(() => Promise.resolve(success(recordDocument()))); + const upload = vi.fn((path: string) => + Promise.resolve( + fileDescriptor({ + fileId: path.includes("archive") ? "file-archive" : "file-readable", + path, + mediaType: "text/html", + }), + ), + ); + const repository = new ConnectSourceImportRepository( + { create } as unknown as ReaderConnectClient, + { upload }, + ); - await repository.commitFile(webCapturePlan()); + await repository.commitFile(webCapturePlan(extractor)); - expect(upload.mock.calls.map(([path]) => path)).toEqual([ - "files/reader/src_import/example.archive.html", - "files/reader/src_import/example.readable.html", - ]); - expect(create).toHaveBeenCalledWith( - expect.objectContaining({ - frontmatter: expect.objectContaining({ - kind: "webpage", - url: "https://example.com/canonical", - original_url: "https://example.com/submitted", - authors: ["Jane Example"], - published: "2026-08-01", - description: "A captured essay.", - language: "en", - site: "Example Review", - capture: { - submitted_url: "https://example.com/submitted", - canonical_url: "https://example.com/canonical", - captured_at: "2026-08-10T11:59:00.000Z", - method: "url", - application: "dev.mdbase.reader", - }, - documents: [ - expect.objectContaining({ - role: "primary", - file_id: "file-readable", - derived_from_file_id: "file-archive", - }), - expect.objectContaining({ - role: "archive", - file_id: "file-archive", - origin_url: "https://example.com/canonical", - retrieved_at: "2026-08-10T11:59:00.000Z", - }), - ], + expect(upload.mock.calls.map(([path]) => path)).toEqual([ + "files/reader/src_import/example.archive.html", + "files/reader/src_import/example.readable.html", + ]); + expect(create).toHaveBeenCalledWith( + expect.objectContaining({ + frontmatter: expect.objectContaining({ + kind: "webpage", + url: "https://example.com/canonical", + original_url: "https://example.com/submitted", + authors: ["Jane Example"], + published: "2026-08-01", + description: "A captured essay.", + language: "en", + site: "Example Review", + capture: { + submitted_url: "https://example.com/submitted", + canonical_url: "https://example.com/canonical", + captured_at: "2026-08-10T11:59:00.000Z", + method: "url", + application: "dev.mdbase.reader", + ...(extractor ? { extractor } : {}), + }, + documents: [ + expect.objectContaining({ + role: "primary", + file_id: "file-readable", + derived_from_file_id: "file-archive", + }), + expect.objectContaining({ + role: "archive", + file_id: "file-archive", + origin_url: "https://example.com/canonical", + retrieved_at: "2026-08-10T11:59:00.000Z", + }), + ], + }), }), - }), - ); - }); + ); + }, + ); }); -function webCapturePlan(): PlannedSourceFileImport { +function webCapturePlan(extractor?: string): PlannedSourceFileImport { const base = plan(); const representation = base.representations[0]!; return { @@ -131,6 +135,7 @@ function webCapturePlan(): PlannedSourceFileImport { submittedUrl: "https://example.com/submitted", canonicalUrl: "https://example.com/canonical", retrievedAt: dateTime("2026-08-10T11:59:00.000Z"), + ...(extractor ? { extractor } : {}), }, representations: [ { diff --git a/packages/core/src/application/import-source-file.test.ts b/packages/core/src/application/import-source-file.test.ts index 4aecfd93..9467aa4b 100644 --- a/packages/core/src/application/import-source-file.test.ts +++ b/packages/core/src/application/import-source-file.test.ts @@ -121,6 +121,7 @@ describe("importSourceFile web and duplicate safeguards", () => { submittedUrl: "https://example.com/story#part", canonicalUrl: "https://www.example.com/story", retrievedAt: dateTime("2026-08-10T11:59:00.000Z"), + extractor: "defuddle@0.19.5", }, archive: { name: "example-com.archive.html", @@ -136,6 +137,7 @@ describe("importSourceFile web and duplicate safeguards", () => { submittedUrl: "https://example.com/story", canonicalUrl: "https://www.example.com/story", retrievedAt: "2026-08-10T11:59:00.000Z", + extractor: "defuddle@0.19.5", }, representations: [ expect.objectContaining({ role: "primary", derivedFromRole: "archive" }), diff --git a/packages/core/src/application/source-import-ports.ts b/packages/core/src/application/source-import-ports.ts index d1c281a0..be5af474 100644 --- a/packages/core/src/application/source-import-ports.ts +++ b/packages/core/src/application/source-import-ports.ts @@ -56,6 +56,8 @@ export interface SourceCaptureProvenance { readonly submittedUrl: string; readonly canonicalUrl: string; readonly retrievedAt: DateTime; + /** Extractor used for this new reading copy; absent on older captures. */ + readonly extractor?: string; } export interface SourceCaptureArchive { diff --git a/packages/core/src/application/source-import-values.ts b/packages/core/src/application/source-import-values.ts index 547dd747..95321344 100644 --- a/packages/core/src/application/source-import-values.ts +++ b/packages/core/src/application/source-import-values.ts @@ -21,7 +21,13 @@ export function validateCapture( ): NonNullableSmall structures, durable evidence
++ We study a compact example whose paragraphs and evidence should survive extraction. A reader + needs every assertion, not only the longest paragraphs, to make useful and durable + annotations. +
+-
+
-
+
+ Language model pre-training. This short summary is substantive evidence rather than a + related article card. It explains the first stage of our experiment and belongs in the + reading copy. +
+
+ -
+
+ Language model post-training. The second stage adds feedback to our experiment. Removing + this short summary would silently lose real content needed to understand the result. +
+
+
+ The inline expression + + describes the next observation. See the accompanying measurements for exact values and the + note for the assumptions.1 +
+
+
+
+
+ Measurements retain their row and column relationships.
+
+ | Observation | +Value | +
|---|---|
| Baseline measurement | +42 | +
| Follow-up measurement | +43 | +
def measure(x):
+ return x + 1
++ All structures above form part of this small article. Capturing them as HTML preserves the + relationship between prose, mathematical notation, code, and measured values for later + reading. +
+-
+
+
Ownership guide
++ This guide explains how a small program owns a value and passes it to another function. The + examples are part of the explanation, and their captions carry information the reader may + need to highlight. +
++ In the first example we retain the complete program, including its main function. Some + documentation sites wrap code in figures, so those containers must not be mistaken for + decorative images and removed. +
+fn main() {
+ let value = String::from("durable reading copy");
+ println!("{}", value);
+}+ After the call returns, the owning function remains responsible for the value. These + explanatory paragraphs connect the code to the diagram and should remain in their original + order without synthetic whitespace changes. +
+fn length(value: &str) -> usize {
+ value.len()
+}+ The helper borrows rather than takes ownership. This final statement completes the + explanation and provides a useful paragraph for checking that highlighting still locates the + exact text in the saved reading copy. +
+Durable evidence
+
+
+
+
+
+
+
+
+
+
+ + Durable evidence is material preserved with its original context and structure. This + synthetic encyclopedia article demonstrates the relationships between source text, + references, and data tables that a reader must retain. +
+Background
++ A reading copy supports later research when its paragraphs remain available. Extraction + must distinguish navigation from substantive passages, even when a passage is short or + includes several links to references. +
++ We preserve a citation exactly as captured[1] + so the footnote and its original anchor remain connected. Citations should not be + rewritten into new numbering schemes when the surrounding prose can be preserved. +
+Measurements
++ Data tables provide a compact account of the observations. Each row belongs to a + different condition; extracting only the prose would remove important evidence even + though the remaining word count might appear sufficient. +
+| Condition | +Observation | +
|---|---|
| Original | +Complete source text | +
| Captured | +Stable reading copy | +
References
+-
+
- + Back Example Author. A synthetic reference with enough + context to identify the evidence. + +
Only the opening survived.
" } })); +vi.mock("defuddle", () => ({ + default: class { + parse(): { readonly content: string; readonly title: string } { + return { ...output, title: "A clipped article" }; + } + }, +})); + +const paragraph = + "The preserved article discusses a reproducible observation and explains its implications for later research. Its evidence belongs in the reading copy so readers can select and annotate the full passage without losing important context. "; +const html = `A complete article
${Array.from({ length: 20 }, (_, i) => `Observation ${String(i)}. ${paragraph.repeat(2)}
`).join("")}A brief note.
"), + ); + expect(result.capture.extractor).toBe(WEB_CAPTURE_EXTRACTOR); + }); + + it("does not switch merely because the page has a large navigation menu", async () => { + const result = await webCaptureImport( + capture(`A brief note.
`), + ); + expect(result.capture.extractor).toBe(WEB_CAPTURE_EXTRACTOR); + }); + + it("keeps Defuddle when Readability cannot recover substantially more", async () => { + const prior = output.content; + output.content = `${paragraph.repeat(35)}
`; + try { + const result = await webCaptureImport( + capture(`${html}`), + ); + expect(result.capture.extractor).toBe(WEB_CAPTURE_EXTRACTOR); + } finally { + output.content = prior; + } + }); +}); diff --git a/packages/web-capture/src/web-capture.test.ts b/packages/web-capture/src/web-capture.test.ts index ab25a822..12fcec2b 100644 --- a/packages/web-capture/src/web-capture.test.ts +++ b/packages/web-capture/src/web-capture.test.ts @@ -1,7 +1,31 @@ // @vitest-environment happy-dom -import { describe, expect, it } from "vitest"; +// @vitest-environment-options {"settings":{"disableIframePageLoading":true}} +import { readFileSync } from "node:fs"; -import { webCaptureImport } from "./web-capture.js"; +import { htmlSelectionDraft, locateHtmlTarget } from "@mdbase-reader/renderer-html"; +import { afterEach, describe, expect, it, vi } from "vitest"; + +import { + WEB_CAPTURE_EXTRACTOR, + webCaptureImport, + type CapturedWebDocument, +} from "./web-capture.js"; + +const capture = ( + html: string, + canonicalUrl = "https://example.com/article", +): CapturedWebDocument => ({ + submittedUrl: canonicalUrl, + canonicalUrl, + retrievedAt: "2026-08-14T00:00:00.000Z", + html, +}); + +const fixture = (name: string): string => readFileSync(`src/fixtures/${name}.html`, "utf8"); +const readingDocument = (bytes: Uint8Array): Document => + new DOMParser().parseFromString(new TextDecoder().decode(bytes), "text/html"); + +afterEach(() => vi.restoreAllMocks()); describe("web capture extraction", () => { it("extracts the article and metadata while retaining a separate archive", async () => { @@ -22,6 +46,7 @@ describe("web capture extraction", () => { const archive = new TextDecoder().decode(result.archive.bytes); expect(result.title).toBe("A useful essay"); + expect(result.capture.extractor).toBe(WEB_CAPTURE_EXTRACTOR); expect(result.metadata).toMatchObject({ authors: ["Jane Example"], published: "2026-08-01", @@ -33,4 +58,137 @@ describe("web capture extraction", () => { expect(readable).not.toContain("