From a1c258a139277c5414e5cdb12b13ba9903896cb6 Mon Sep 17 00:00:00 2001 From: Gordon Woodhull Date: Sat, 3 Oct 2026 21:28:46 -0400 Subject: [PATCH 1/2] Expand shortcodes in footnote definitions and lst-cap (bd-xjg7vl6c) Rust shortcode_resolve skipped NoteDefinitionPara, NoteDefinitionFencedBlock and the lst-cap attribute of listing code blocks; Quarto 1 expands all three. Add a parameterized all-contexts test; contexts Q1 also leaves literal are pinned as such. --- .../2026-10-03-disable-lua-shortcodes.md | 165 +++++ ...26-10-04-shortcode-footnote-defs-lstcap.md | 69 ++ .../src/transforms/shortcode_resolve.rs | 59 +- crates/quarto-core/tests/integration/main.rs | 1 + .../integration/shortcode_all_contexts.rs | 650 ++++++++++++++++++ 5 files changed, 937 insertions(+), 7 deletions(-) create mode 100644 claude-notes/plans/2026-10-03-disable-lua-shortcodes.md create mode 100644 claude-notes/plans/2026-10-04-shortcode-footnote-defs-lstcap.md create mode 100644 crates/quarto-core/tests/integration/shortcode_all_contexts.rs diff --git a/claude-notes/plans/2026-10-03-disable-lua-shortcodes.md b/claude-notes/plans/2026-10-03-disable-lua-shortcodes.md new file mode 100644 index 000000000..8eed32f0a --- /dev/null +++ b/claude-notes/plans/2026-10-03-disable-lua-shortcodes.md @@ -0,0 +1,165 @@ +# Disable the vendored Lua shortcodes pass (escaped shortcodes expanded twice) + +**Branch:** `issue-brand-shortcode` (from `feature/pandoc-wasm` @ `6d03c76ac`), worktree `workspace-7`. +**Strands:** bug `bd-2uva9urq` (this plan); design question `bd-qwgu94f4` ("should pandoc Lua execute shortcodes?"). Related: `bd-xfqx2tuc`, `bd-qnylgu69`. +**Status:** REVIEWED 2026-10-03 (one edit round applied); not started. No code changed yet. +**Blocked by:** `bd-xjg7vl6c` (Rust never expands shortcodes in footnote definitions). T2 must not land +before it, see Prerequisite. + +## Problem + +`docs/guides/authoring/brand.qmd` fails to render through pandoc.wasm, and natively with +`q2 render --to typst`. The failing text is documentation of the syntax (lines 859-864), written with the +escape: ``### Shortcode - `{{{< brand >}}}` `` and ``Use `{{{< brand color COLOR_NAME VARIANT >}}}` …``. +Errors: `shortcodes-handlers.lua:111` (`brandCommand` is nil, string concat) for the bare form; +`modules/brand/brand.lua:20` (`assert`, mode `"VARIANT"`) for the argument form. +The e2e test `hub-client/e2e/pandoc-warm.harness.spec.ts` ("larger fixtures") hides it by replacing the +shortcodes with `BRAND` via regex. + +## Root cause (verified) + +Two shortcode passes run over the same text, and expansion is not idempotent: + +1. **Rust** (`crates/quarto-core/src/transforms/shortcode_resolve.rs`): `Inline::Code` (≈L2159) and + `Block::CodeBlock` (≈L1899) call `expand_text_in_place` → `parse_text_shortcodes` + (`shortcode_text.rs:63-67`), which turns `{{{< x >}}}` into the literal `{{< x >}}` ("applied once"). + Port of Q1's `apply_code_shortcode` (bd-fz6gwfq0); covered by + `crates/quarto-core/tests/integration/shortcode_text_contexts.rs`. +2. **Pandoc Lua** (vendored Q1 stack): `pre-shortcodes-filter` → `shortcodes_filter()` + (`filters/customnodes/shortcodes.lua:208`), whose `code_handler`/`attr_handler` run Q1's + `apply_code_shortcode` over code and attribute text. It now sees the already-unescaped `{{< … >}}` + and expands it as live. + +Confirmed by dumping the retained `pandoc-input.json`: the `Code` node text reaches Lua as +`{{< brand color COLOR_NAME VARIANT >}}`. Reproduced natively again at review (`q2 render x.qmd --to typst`, rc=1, crash at `shortcodes.lua:305`) with three +minimal files: `` `{{{< brand >}}}` `` and `` `{{{< brand color COLOR_NAME VARIANT >}}}` `` fail; a +live `{{< brand color primary >}}` does **not** crash (Rust warns "Shortcode `brand` is not recognized" +and Lua never sees it). + +Exposure: text contexts only (code, raw, math, attribute values). Escaped shortcodes in ordinary +inlines become a `Str`, which Lua does not expand. Pandoc paths only: native HTML does not run the Lua +stack, so `shortcode_text_contexts.rs` passes today. + +## Decision (Gordon, 2026-10-03) + +Disable shortcode execution on the pandoc Lua side for now. Rationale: + +- Rust already resolves every shortcode node before Pandoc (including user-extension Lua shortcodes via + its own `lua_engine`; unresolved ones become `?key` plus a Q-16-5 diagnostic). The Lua pass's only + reachable input is text contexts that Rust has just processed, so the pass can only double-expand. +- It is the smallest fix and makes native HTML, native typst and wasm consistent: one pass, owned by Rust. +- Cost: Lua-only handlers (`brand`) stop working in code text. They did not work in live position + anyway (Rust does not know `brand`). `bd-qnylgu69` still owns whether Q2 supports `brand` at all. +- The permanent ownership question is parked in `bd-qwgu94f4`. + +Rejected for now: (1) Rust leaves escapes in text contexts for Lua to unescape (couples Rust output to +whether Lua runs afterwards); (2) marker attribute so Lua skips Rust-processed text (extra machinery, +edits vendored Lua handlers); (3) teach Rust `brand` and split ownership by name (largest; belongs to +`bd-qwgu94f4`). + +## Prerequisite: Rust skips footnote definitions (found in review) + +The Lua pass is not purely redundant. Rust's `shortcode_resolve` treats `Block::NoteDefinitionPara` and +`Block::NoteDefinitionFencedBlock` as leaves (L1136, L1928), so a shortcode in `[^1]: Note {{< meta author >}}.` +is never expanded by Rust. Native HTML shows `?meta` there. Pandoc paths resolve it today only because Lua +catches it. Measured against a scratch share tree: with the Lua pass off, pandoc exits rc=83 on that input (an +unresolved `Shortcode` custom node reaches `render`, which calls `internal_error()`); every other context tried +(tables, captions, callout titles, links, headers, attributes, math, raw, code, quotes, definition lists, fig divs) +is identical with the pass off. Gordon's call: independent bug, filed as `bd-xjg7vl6c` (it also recommends an +audit of every AST context Rust skips). **Order:** `bd-xjg7vl6c` lands before T2, otherwise disabling is a crash +regression for footnote definitions. Not part of this plan's checklist. + +**Status (2026-10-04): fixed** on `issue-brand-shortcode` (see +`2026-10-04-shortcode-footnote-defs-lstcap.md`). Rust now expands footnote definitions (single-paragraph and +`::: ^id`) and the `lst-cap` attribute. The audit found no other qmd-reachable gap that Q1 expands; the +remaining unexpanded contexts (link/image titles, Code/CodeBlock/Table attr values, cite prefix/suffix) match +Q1 and are pinned by `shortcode_all_contexts`. The "other walkers in non-shortcode transforms" question was +not checked. + +The earlier assumption "Rust already resolves every shortcode node before Pandoc" is therefore true only +modulo that bug (and anything the audit finds). + +## Off-switch design + +Two registrations of `shortcodes_filter()`: + +- `resources/pandoc-filters/filters/main.lua:319-323` (`pre-shortcodes-filter`, flag `has_shortcodes`, `traverser = 'jog'`). +- `resources/pandoc-filters/filters/crossref/crossref.lua:169-171` (same name, no traverser). + +**Chosen: keep the `shortcodes_filter()` call, gate the entry off.** Change the entry's flag in both files from +`flags = { "has_shortcodes" }` to a flag nothing ever sets (`flags = { "q2_lua_shortcodes_disabled" }`; the +runner skips a filter whose flags are all unset, `ast/runemulation.lua:54-70`), with a `QUARTO2-PATCH` comment +naming this plan. One-line change per file, trivially reversible, and `bd-qwgu94f4` can re-enable it. + +Why not remove the entries (the first draft): `shortcodes_filter()` is evaluated when the filter list table is +built (load time) and assigns the module-local `_shortcodes_filter`. `process_shortcodes()` +(`customnodes/shortcodes.lua:204`) walks with that value, and `quarto-post/foldcode.lua:27` calls it for +`code-summary` in HTML-family pandoc output. Removing the call leaves it `nil`. Keeping the call also keeps the +custom-node handler registration (that is at import time, independent). `normalize/flags.lua` keeps computing +`has_shortcodes`; harmless. Replayed against captured pandoc inputs with each variant: removing the entries and +the flag-gate both fix all three repros and produce byte-identical typst for `shortcode-passthrough` (Rust already +turns passthrough into literal text); only the flag-gate keeps `process_shortcodes` working. + +Unresolved `Shortcode` nodes Rust leaves behind: none in the contexts tried except footnote definitions (above). +After that bug is fixed, any remaining leftover would hit `Shortcode.render` -> `internal_error()` rather than +`?key`; the audit in `bd-xjg7vl6c` is the guard. `contents_shortcode_filter` stays but is inert: `contents` is +Lua-only and Rust never resolves it (it already yields `?contents` before Lua runs). + +Do **not** edit `customnodes/shortcodes.lua` or `quarto-pre/shortcodes-handlers.lua`. + +README: add entries under "Ours vs. pinned" in `resources/pandoc-filters/README.md` for `main.lua` (extend +the existing entry) and `crossref/crossref.lua`, in the exact `` - `` — description `` shape the +`vendored-pandoc-filters` lint requires. + +## Tasks + +Gates per task: `cargo clippy -p --all-targets -- -D warnings` and `cargo nextest run -p `; +one `cargo nextest run --workspace` at the end (capture to a log; compare to the live baseline, not an old +figure). Known red before this work: `pandoc_request_prepare::golden_file_is_structurally_current` and +`::typst_request_matches_the_recorded_native_run` (lane R recording drift). + +- [ ] **T1 Failing test first.** New `crates/quarto-core/tests/integration/shortcode_pandoc_escapes.rs` + (register in `main.rs`, alphabetical). Use `render_document_to_file` with `native` (AST dump: no writer escaping, + no typst compile; model on `gfm_shortcode_round_trip` in `pandoc_long_tail_formats.rs`). Cases: the bare + `` `{{{< brand >}}}` ``, the arguments form, and an escaped shortcode in a fenced code block; assert the render + succeeds and the output contains the literal `{{< brand ... >}}`. Add a control: a live `{{< brand color primary >}}` + still only warns. Practical in CI: the existing long-tail tests already run real native pandoc with no skip. + Confirm it fails (rc 83) before T2. Note `shortcode_text_contexts.rs` is HTML-only, so it never covered this. +- [ ] **T2 Off-switch** (after `bd-xjg7vl6c`): flag-gate the entry in `main.lua` and `crossref.lua` per the design above; README entries. +- [ ] **T3 Dependents (surveyed in review; re-confirm, then no changes expected).** No smoke-all or golden + fixture uses `brand` or `contents` live; the only users are `docs/guides/authoring/brand.qmd` and the + `pandoc-warm` e2e. `crates/pampa/tests/wasm_lua.rs` tests pampa's own `LuaShortcodeEngine` (extension + shortcodes), not the vendored pandoc Lua: unaffected, and no pampa Lua file is edited so the wasm rule does not + trigger. No Lua unit tests exist in `resources/`. `gfm_shortcode_round_trip` uses an escaped shortcode in plain + text (a `Str`), which Lua never expanded: unaffected. Also grep `^\[\^.*\]:.*\{\{<` in fixtures (none found). + The wasm path builds its request from the same stage list as native (`build_pandoc_prefix_stages`, then + `PandocPrepareStage`), so one Rust-resolved AST feeds both; T5 still verifies it. +- [ ] **T4 Goldens.** Editing `main.lua`/`crossref.lua` changes `share_tree_version` + (`pandoc_request/share.rs:54`, a hash of the embedded tree). `golden_file_is_structurally_current` asserts it + equals `schemas/pandoc-request.golden.json`, so it **will go red because of this change**: regenerate with + `Q2_REGENERATE_GOLDEN=1 cargo nextest run -p quarto-core golden_file` and commit the golden (job_id and + share_tree_version change), then check `hub-client` `goldenParity.wasm.test.ts` (compares the golden to the + wasm's live share tree). The recordings under `pandoc-recordings/recordings/share/b540fcb3e42f6fb5/` embed the + old filter sources but are self-contained (`recordingParity` and `check_against_recordings` use the recorded + tree, not the live one) and none of the fixtures hit a changed path, so they stay green and are **not** + re-recorded; note the staleness in the commit message. Correction: at review (HEAD `e38bbe55f`) both + `golden_file_is_structurally_current` and `typst_request_matches_the_recorded_native_run` **pass** in this + worktree (13/13 `pandoc_request_prepare::`), so they are not known-red here; re-measure the workspace baseline + at T7 instead of trusting the "known red" list. +- [ ] **T5 Fixture.** Remove the `BRAND` substitution in `hub-client/e2e/pandoc-warm.harness.spec.ts` + so `brand.qmd` renders as written. `npm run build:wasm`, then `VITE_E2E=1 npm run build` and + `npx playwright test --config playwright.harness.config.ts --project=chromium --workers=1 pandoc-warm`. +- [ ] **T6 Native sweep.** `q2 render` to typst and html for `docs/guides/authoring/brand.qmd` and `docs/guides/authoring/shortcodes.qmd` (its table of `{{{< meta key >}}}` code spans was silently double-expanded too, with no crash); confirm the + escaped examples show literally and note what the live `brand` shortcodes (lines 887, 898) now do. + Report; do not fix `brand` support here (`bd-qnylgu69`). +- [ ] **T7 Workspace run, strand updates.** Record outcome on `bd-2uva9urq`; leave `bd-qwgu94f4` open. + Reconcile this checklist with reality before handing off. + +## Risks / open items + +- `bd-xjg7vl6c` must land first (see Prerequisite); if it slips, T2 waits. +- `foldcode.lua` `process_shortcodes` still runs on `code-summary` text (HTML-family pandoc output only) and + could double-expand an already-unescaped summary: the same class of bug, out of scope; belongs to `bd-qwgu94f4`. +- Wasm verification is T5 only (the e2e); there is no native proxy for the wasm Lua stdlib. +- Untracked strays in this worktree (`ff.mjs`, `serve-ff.mjs`, `hub-client/h9-demo-check.mjs`): never + stage them; use explicit paths. diff --git a/claude-notes/plans/2026-10-04-shortcode-footnote-defs-lstcap.md b/claude-notes/plans/2026-10-04-shortcode-footnote-defs-lstcap.md new file mode 100644 index 000000000..aca93d84b --- /dev/null +++ b/claude-notes/plans/2026-10-04-shortcode-footnote-defs-lstcap.md @@ -0,0 +1,69 @@ +# Shortcodes in footnote definitions and `lst-cap` (bd-xjg7vl6c) + +Strand: `bd-xjg7vl6c`. Blocks `claude-notes/plans/2026-10-03-disable-lua-shortcodes.md` +(bd-2uva9urq): that plan's T2 must not land until this is fixed. + +## Problem + +Rust's `shortcode_resolve` does not expand `{{< meta x >}}` in three qmd-reachable contexts where +Quarto 1 does: + +1. `NoteDefinitionPara` (`[^1]: Note {{< meta author >}}.`). +2. `NoteDefinitionFencedBlock` (q2's fenced form `::: ^id`). +3. The `lst-cap` attribute on a listing code block (`{#lst-a lst-cap="… {{< meta author >}}"}`). + +Cause of 1 and 2: both are "leaf" arms in the two body walkers (`resolve_block`, `stamp_block`). +`FootnotesTransform` runs after `ShortcodeResolveTransform` and moves the definition content into +`Inline::Note` unchanged, so nothing resolves it later. Inline notes (`Inline::Note`) were already walked. + +Cause of 3: `CodeBlock` attribute values are not expanded (matching Q1), but +`FloatRefTargetSugarTransform` runs after the shortcode pass and parses the `lst-cap` string into the +caption, so a literal `{{< … >}}` becomes an unresolved `Inline::Shortcode` that nothing resolves. +Q1 expands it ("Listing 1: LC Ann"). + +The pandoc path hides 1 and 2 today only because the vendored Lua shortcodes pass catches them. + +## Evidence (Q1 parity, 2026-10-04) + +Read from Q1's `shortcodes.lua` (the vendored copy is byte-identical) and confirmed with renders of +a Q1 dev build: + +| Context | Q1 expands? | +|---|---| +| Footnote definition `[^1]: …` | yes | +| `lst-cap` | yes | +| Link/image title, CodeBlock/Code attr values, Table attr values, Cite prefix/suffix | **no** (leaks a placeholder, drops the value, or fails to parse as a citation) | +| Div/Span/Header/Link/Image attrs, link URL, image src | yes (q2 already matches) | + +So only items 1-3 are regressions. The others match Q1 and stay unexpanded. + +Out of scope: the `::: ^id` fenced footnote syntax is a deliberate q2 difference (indented +multi-paragraph footnotes are a code block in q2, Q-2-35). It is not changed; the fix only makes its +body expand like the single-paragraph form. + +## Detection test + +`crates/quarto-core/tests/integration/shortcode_all_contexts.rs` (one parameterized test, ~65 qmd +contexts rendered to native HTML with `author: Ann`). Cases carry `known_gap`; a failing case without a +marker, or a marked case that passes, fails the test. + +## Checklist + +- [x] Parameterized detection test written and green with the gaps marked (`known_gap`). +- [x] Q1 parity check for every gap the test found. +- [x] Walk `NoteDefinitionPara` / `NoteDefinitionFencedBlock` in `resolve_block` (`resolve_inlines` / + `resolve_blocks`) and in `stamp_block` (provenance), and fix the "don't contain inlines" comment. +- [x] Expand shortcodes in the `lst-cap` attribute value of `CodeBlock` in `resolve_block`. Only that + key; other code-block attrs stay literal (Q1 parity). Not gated by the code-text opt-out + (`shortcodes="false"` / `.cell-code`), since the caption is prose, not code text. +- [x] Retag the test: footnote and `lst-cap` cases lose `known_gap`; the Q1-parity cases (titles, + CodeBlock/Code/Table attrs, cite prefix/suffix) become "expected unexpanded (matches Q1)" assertions + that fail if they start expanding (marker `Q1_LITERAL`). A `#| lst-cap:` cell-option case was + not added: it needs an executed cell (engine), which a native-HTML test cannot assume. +- [x] Unknown shortcode / unknown meta key in a footnote definition gives `?name` / `?meta:key` plus + Q-16-3 / Q-16-5 (the strand text said Q-16-5 for an unknown shortcode name; the real code is Q-16-3). +- [x] Gates (workspace: 15733 run, 15733 passed, 202 skipped; identical to the HEAD-plus-test baseline): `cargo clippy -p quarto-core --all-targets -- -D warnings`, + `cargo nextest run -p quarto-core`, then one `cargo nextest run --workspace` against a baseline + measured at HEAD. +- [x] Commit (explicit paths only), then comment on and close `bd-xjg7vl6c`, noting the corrected + Q-code and the Q1-parity decisions; update the disable-Lua plan's prerequisite note. diff --git a/crates/quarto-core/src/transforms/shortcode_resolve.rs b/crates/quarto-core/src/transforms/shortcode_resolve.rs index 3d650c6a2..3dcf54486 100644 --- a/crates/quarto-core/src/transforms/shortcode_resolve.rs +++ b/crates/quarto-core/src/transforms/shortcode_resolve.rs @@ -1129,12 +1129,20 @@ fn stamp_block(block: &mut Block, name: &str, token_arc: &Arc) { } } // Leaves — no nested AST to walk. + Block::NoteDefinitionPara(def) => { + for child in def.content.iter_mut() { + stamp_inline(child, name, token_arc); + } + } + Block::NoteDefinitionFencedBlock(def) => { + for child in def.content.iter_mut() { + stamp_block(child, name, token_arc); + } + } Block::CodeBlock(_) | Block::RawBlock(_) | Block::HorizontalRule(_) | Block::BlockMetadata(_) - | Block::NoteDefinitionPara(_) - | Block::NoteDefinitionFencedBlock(_) | Block::CaptionBlock(_) => {} } } @@ -1897,6 +1905,23 @@ fn resolve_block<'a>( // Text contexts — Q1 substitutes shortcodes textually in // code and raw text (`apply_code_shortcode`; bd-fz6gwfq0). Block::CodeBlock(code_block) => { + // Q1 expands no code-block attribute values, with one + // exception in effect: `lst-cap` becomes the listing + // caption (parsed after this pass by + // `FloatRefTargetSugarTransform`), where Q1 expands + // shortcodes. Expand just that value (bd-xjg7vl6c). + if let Some(cap) = code_block.attr.2.get_mut("lst-cap") { + expand_text_in_place( + cap, + transform, + metadata, + &code_block.source_info, + diagnostics, + lua_engine, + UnhandledInclude::Report, + ) + .await; + } if !code_shortcode_opt_out(&code_block.attr) { expand_text_in_place( &mut code_block.text, @@ -1922,12 +1947,32 @@ fn resolve_block<'a>( ) .await; } + // Footnote definitions (`[^id]: …` and the fenced `::: ^id` + // form) become `Inline::Note`s in `FootnotesTransform`, which + // runs after this pass and moves the content over unchanged, + // so they must be resolved here (bd-xjg7vl6c). + Block::NoteDefinitionPara(def) => { + resolve_inlines( + &mut def.content, + transform, + metadata, + diagnostics, + lua_engine, + ) + .await; + } + Block::NoteDefinitionFencedBlock(def) => { + resolve_blocks( + &mut def.content, + transform, + metadata, + diagnostics, + lua_engine, + ) + .await; + } // These blocks don't contain inlines that could have shortcodes - Block::HorizontalRule(_) - | Block::BlockMetadata(_) - | Block::NoteDefinitionPara(_) - | Block::NoteDefinitionFencedBlock(_) - | Block::CaptionBlock(_) => {} + Block::HorizontalRule(_) | Block::BlockMetadata(_) | Block::CaptionBlock(_) => {} } }) } diff --git a/crates/quarto-core/tests/integration/main.rs b/crates/quarto-core/tests/integration/main.rs index 38fc745fe..81ee38f40 100644 --- a/crates/quarto-core/tests/integration/main.rs +++ b/crates/quarto-core/tests/integration/main.rs @@ -131,6 +131,7 @@ pub mod revealjs_features; pub mod revealjs_format; pub mod sass_cache_key; pub mod secondary_nav_pipeline; +pub mod shortcode_all_contexts; pub mod shortcode_config_pipeline; pub mod shortcode_text_contexts; pub mod sidebar_pipeline; diff --git a/crates/quarto-core/tests/integration/shortcode_all_contexts.rs b/crates/quarto-core/tests/integration/shortcode_all_contexts.rs new file mode 100644 index 000000000..a5e9b9b11 --- /dev/null +++ b/crates/quarto-core/tests/integration/shortcode_all_contexts.rs @@ -0,0 +1,650 @@ +/* + * tests/integration/shortcode_all_contexts.rs + * Copyright (c) 2026 Posit, PBC + * + * Detection test: a live `{{< meta author >}}` shortcode in every + * qmd-reachable AST context, rendered to native HTML (bd-xjg7vl6c). + */ + +//! Each case renders a separate document whose front matter defines +//! `author: Ann` and whose body places `{{< meta author >}}` in one +//! context. A case passes when every `needles` entry appears in the +//! HTML, no `forbid` entry appears, and (unless `unresolved_ok`) the +//! page carries no `quarto-unresolved-shortcode` / `?meta` marker. +//! +//! Cases carry an optional `known_gap` marker. Today the only marker is +//! `Q1_LITERAL`: contexts where Quarto 1 does not expand shortcodes either, +//! asserted here as "stays unexpanded": +//! +//! - an unmarked case that fails => test fails (regression/new gap) +//! - a `known_gap` case that passes => test fails (q2 now differs from Q1) +//! +//! All cases are rendered and reported together; the full results +//! table is printed on any failure. +//! +//! Not reachable from qmd syntax (no case here; covered only by +//! building the AST directly): `TableBody.head` rows (q2 rejects grid +//! tables, Q-2-39), `CaptionBlock`, `BlockMetadata`, attr values on +//! `Row` / `Cell`, and indented multi-paragraph footnote definitions +//! (indented text is a code block in q2, Q-2-35). + +use std::sync::Arc; + +use tempfile::TempDir; + +use quarto_core::render_to_file::{RenderToFileOptions, render_to_file}; +use quarto_system_runtime::{NativeRuntime, SystemRuntime}; + +/// Contexts where Quarto 1 does NOT expand shortcodes (verified against Q1's +/// `shortcodes.lua` and renders, 2026-10-04; see +/// claude-notes/plans/2026-10-04-shortcode-footnote-defs-lstcap.md), so q2 +/// leaves them unexpanded for parity. The case must keep failing; if it +/// starts passing, q2 diverged from Q1 and that should be a deliberate choice. +const Q1_LITERAL: Option<&str> = Some("q1-literal"); + +struct Case { + name: &'static str, + /// Extra front-matter lines (after `author: Ann`). + meta: &'static str, + body: &'static str, + /// Substrings that must all appear in the HTML. + needles: &'static [&'static str], + /// Substrings that must not appear in the HTML. + forbid: &'static [&'static str], + /// Skip the generic "no unresolved marker" check (unknown-shortcode case). + unresolved_ok: bool, + /// Q-code that must appear among the render diagnostics. + expect_diag: Option<&'static str>, + known_gap: Option<&'static str>, +} + +const BASE: Case = Case { + name: "", + meta: "", + body: "", + needles: &[], + forbid: &[], + unresolved_ok: false, + expect_diag: None, + known_gap: None, +}; + +fn cases() -> Vec { + vec![ + // ── Block text contexts ───────────────────────────────────── + Case { + name: "paragraph (control)", + body: "P-CTL {{< meta author >}}.\n", + needles: &["P-CTL Ann."], + ..BASE + }, + Case { + name: "heading text", + body: "## H-TXT {{< meta author >}}\n\nBody.\n", + needles: &["H-TXT Ann"], + ..BASE + }, + Case { + name: "heading attr value", + body: "## Head {data-x=\"H-ATTR {{< meta author >}}\"}\n\nBody.\n", + needles: &["data-x=\"H-ATTR Ann\""], + ..BASE + }, + Case { + name: "bullet list item", + body: "- LI-B {{< meta author >}}\n- other\n", + needles: &["LI-B Ann"], + ..BASE + }, + Case { + name: "ordered list item", + body: "1. LI-O {{< meta author >}}\n2. other\n", + needles: &["LI-O Ann"], + ..BASE + }, + Case { + name: "task list item", + body: "- [ ] LI-T {{< meta author >}}\n- [x] other\n", + needles: &["LI-T Ann"], + ..BASE + }, + Case { + name: "blockquote", + body: "> BQ {{< meta author >}}\n", + needles: &["BQ Ann"], + ..BASE + }, + Case { + name: "definition list term", + body: "::: {.definition-list}\n- DL-TERM {{< meta author >}}\n - Definition body.\n:::\n", + needles: &["DL-TERM Ann"], + ..BASE + }, + Case { + name: "definition list definition", + body: "::: {.definition-list}\n- Term\n - DL-DEF {{< meta author >}}\n:::\n", + needles: &["DL-DEF Ann"], + ..BASE + }, + Case { + name: "line block", + body: "| LB-1 {{< meta author >}}\n| second line\n", + needles: &["LB-1 Ann"], + ..BASE + }, + Case { + name: "div content", + body: "::: {.note}\nDIV-C {{< meta author >}}\n:::\n", + needles: &["DIV-C Ann"], + ..BASE + }, + Case { + name: "div attr value", + body: "::: {data-x=\"DIV-A {{< meta author >}}\"}\nInside.\n:::\n", + needles: &["data-x=\"DIV-A Ann\""], + ..BASE + }, + // ── Inline contexts ───────────────────────────────────────── + Case { + name: "span content", + body: "A [SPAN-C {{< meta author >}}]{.hl} b.\n", + needles: &["SPAN-C Ann"], + ..BASE + }, + Case { + name: "span attr value", + body: "A [word]{data-x=\"SPAN-A {{< meta author >}}\"} b.\n", + needles: &["data-x=\"SPAN-A Ann\""], + ..BASE + }, + Case { + name: "emph", + body: "*EMPH {{< meta author >}}*\n", + needles: &["EMPH Ann"], + ..BASE + }, + Case { + name: "strong", + body: "**STRONG {{< meta author >}}**\n", + needles: &["STRONG Ann"], + ..BASE + }, + Case { + name: "strikeout", + body: "~~STRIKE {{< meta author >}}~~\n", + needles: &["STRIKE Ann"], + ..BASE + }, + Case { + name: "superscript", + body: "x^SUP {{< meta author >}}^\n", + needles: &["SUP Ann"], + ..BASE + }, + Case { + name: "subscript", + body: "x~SUB {{< meta author >}}~\n", + needles: &["SUB Ann"], + ..BASE + }, + Case { + name: "underline span", + body: "[UND {{< meta author >}}]{.underline}\n", + needles: &["UND Ann"], + ..BASE + }, + Case { + name: "smallcaps span", + body: "[SMC {{< meta author >}}]{.smallcaps}\n", + needles: &["SMC Ann"], + ..BASE + }, + Case { + name: "quoted", + body: "He said \"QUO {{< meta author >}}\" today.\n", + needles: &["QUO Ann"], + ..BASE + }, + Case { + name: "link text", + body: "[LNK-T {{< meta author >}}](https://example.com/x)\n", + needles: &["LNK-T Ann"], + ..BASE + }, + Case { + name: "link URL", + body: "[t](https://example.com/{{< meta author >}})\n", + needles: &["https://example.com/Ann"], + ..BASE + }, + Case { + name: "link title", + body: "[t](https://example.com/x \"LNK-TTL {{< meta author >}}\")\n", + needles: &["title=\"LNK-TTL Ann\""], + known_gap: Q1_LITERAL, + ..BASE + }, + Case { + name: "link attr value", + body: "[t](https://example.com/x){data-x=\"LNK-A {{< meta author >}}\"}\n", + needles: &["data-x=\"LNK-A Ann\""], + ..BASE + }, + Case { + name: "image alt", + body: "![IMG-ALT {{< meta author >}}](img.png)\n", + needles: &["IMG-ALT Ann"], + ..BASE + }, + Case { + name: "image title", + body: "![a](img.png \"IMG-TTL {{< meta author >}}\")\n", + needles: &["IMG-TTL Ann"], + known_gap: Q1_LITERAL, + ..BASE + }, + Case { + name: "image attr value", + body: "![a](img.png){data-x=\"IMG-A {{< meta author >}}\"}\n", + needles: &["data-x=\"IMG-A Ann\""], + ..BASE + }, + Case { + name: "image src", + body: "![a]({{< meta author >}}.png)\n", + needles: &["src=\"Ann.png\""], + ..BASE + }, + Case { + name: "bracketed span attr", + body: "[plain]{#sp data-x=\"BSP {{< meta author >}}\"}\n", + needles: &["data-x=\"BSP Ann\""], + ..BASE + }, + // ── Figures / tables ──────────────────────────────────────── + Case { + name: "figure caption", + body: "![FIG-CAP {{< meta author >}}](img.png){#fig-a}\n", + needles: &["FIG-CAP Ann"], + ..BASE + }, + Case { + name: "figure attr value", + body: "![cap](img.png){#fig-b data-x=\"FIG-A {{< meta author >}}\"}\n", + needles: &["data-x=\"FIG-A Ann\""], + ..BASE + }, + Case { + name: "div fig- id caption", + body: "::: {#fig-c}\n![](img.png)\n\nDFIG-CAP {{< meta author >}}\n:::\n", + needles: &["DFIG-CAP Ann"], + ..BASE + }, + Case { + name: "pipe table cell", + body: "| H |\n|---|\n| TBL-CELL {{< meta author >}} |\n", + needles: &["TBL-CELL Ann"], + ..BASE + }, + Case { + name: "pipe table header", + body: "| TBL-HDR {{< meta author >}} |\n|---|\n| x |\n", + needles: &["TBL-HDR Ann"], + ..BASE + }, + Case { + name: "table caption", + body: "| H |\n|---|\n| x |\n\n: TBL-CAP {{< meta author >}}\n", + needles: &["TBL-CAP Ann"], + ..BASE + }, + Case { + name: "table caption (tbl- id)", + body: "| H |\n|---|\n| x |\n\n: TBL-CAP2 {{< meta author >}} {#tbl-a}\n", + needles: &["TBL-CAP2 Ann"], + ..BASE + }, + Case { + name: "table attr value", + body: "| H |\n|---|\n| x |\n\n: Cap {data-x=\"TBL-A {{< meta author >}}\"}\n", + needles: &["data-x=\"TBL-A Ann\""], + known_gap: Q1_LITERAL, + ..BASE + }, + // ── Code / raw / math ─────────────────────────────────────── + Case { + name: "code block text", + body: "```\nCB {{< meta author >}}\n```\n", + needles: &["CB Ann"], + ..BASE + }, + Case { + name: "code block attr value", + body: "```{.txt data-x=\"CB-A {{< meta author >}}\"}\nx\n```\n", + needles: &["data-x=\"CB-A Ann\""], + known_gap: Q1_LITERAL, + ..BASE + }, + Case { + name: "code block opt-out (shortcodes=false)", + body: "```{.markdown shortcodes=\"false\"}\nCB-OFF {{< meta author >}}\n```\n", + needles: &["CB-OFF {{< meta author >}}"], + forbid: &["CB-OFF Ann"], + ..BASE + }, + Case { + name: "code block opt-out (cell-code)", + body: "```{.cell-code}\nCB-CELL {{< meta author >}}\n```\n", + needles: &["CB-CELL {{< meta author >}}"], + forbid: &["CB-CELL Ann"], + ..BASE + }, + Case { + name: "inline code", + body: "Run `IC {{< meta author >}}` now.\n", + needles: &["IC Ann"], + ..BASE + }, + Case { + name: "inline code attr value", + body: "Run `x`{data-x=\"IC-A {{< meta author >}}\"} now.\n", + needles: &["data-x=\"IC-A Ann\""], + known_gap: Q1_LITERAL, + ..BASE + }, + Case { + name: "raw block", + body: "```{=html}\nRAW-B {{< meta author >}}\n```\n", + needles: &["RAW-B Ann"], + ..BASE + }, + Case { + name: "raw inline", + body: "x `RAW-I {{< meta author >}}`{=html} y\n", + needles: &["RAW-I Ann"], + ..BASE + }, + Case { + name: "inline math", + body: "Val $MI = {{< meta author >}}$ here.\n", + needles: &["MI = Ann"], + ..BASE + }, + Case { + name: "display math", + body: "$$\nMD = {{< meta author >}}\n$$\n", + needles: &["MD = Ann"], + ..BASE + }, + // ── Callouts / tabsets / cells ────────────────────────────── + Case { + name: "callout title attr", + body: "::: {.callout-note title=\"CO-T {{< meta author >}}\"}\nBody.\n:::\n", + needles: &["CO-T Ann"], + ..BASE + }, + Case { + name: "callout title heading", + body: "::: {.callout-note}\n## CO-H {{< meta author >}}\n\nBody.\n:::\n", + needles: &["CO-H Ann"], + ..BASE + }, + Case { + name: "callout body", + body: "::: {.callout-note}\nCO-B {{< meta author >}}\n:::\n", + needles: &["CO-B Ann"], + ..BASE + }, + Case { + name: "tabset title", + body: "::: {.panel-tabset}\n## TAB-T {{< meta author >}}\n\nContent A.\n\n## Other\n\nContent B.\n:::\n", + needles: &["TAB-T Ann"], + ..BASE + }, + Case { + name: "tabset body", + body: "::: {.panel-tabset}\n## A\n\nTAB-B {{< meta author >}}\n\n## B\n\nContent B.\n:::\n", + needles: &["TAB-B Ann"], + ..BASE + }, + Case { + name: "listing caption attr (lst-cap)", + body: "```{#lst-a .python lst-cap=\"LST-CAP {{< meta author >}}\"}\nx = 1\n```\n", + needles: &["LST-CAP Ann"], + ..BASE + }, + Case { + name: "code block attr value (lst- listing id)", + body: "```{#lst-b .python data-x=\"CELL-A {{< meta author >}}\"}\nx = 1\n```\n", + needles: &["data-x=\"CELL-A Ann\""], + known_gap: Q1_LITERAL, + ..BASE + }, + // ── Cite ──────────────────────────────────────────────────── + Case { + name: "cite prefix", + body: "[see CITE-P {{< meta author >}} @nokey]\n", + needles: &["CITE-P Ann"], + known_gap: Q1_LITERAL, + ..BASE + }, + Case { + name: "cite suffix", + body: "[@nokey, p. 3 CITE-S {{< meta author >}}]\n", + needles: &["CITE-S Ann"], + known_gap: Q1_LITERAL, + ..BASE + }, + // ── Footnotes ─────────────────────────────────────────────── + Case { + name: "inline footnote ^[...]", + body: "Text.^[FN-INL {{< meta author >}}.]\n", + needles: &["FN-INL Ann."], + ..BASE + }, + Case { + name: "footnote definition (single para)", + body: "Text.[^1]\n\n[^1]: FN-DEF {{< meta author >}}.\n", + needles: &["FN-DEF Ann."], + ..BASE + }, + Case { + name: "footnote definition (fenced block)", + body: "Text.[^my]\n\n::: ^my\nFN-FB {{< meta author >}}.\n\nSecond.\n:::\n", + needles: &["FN-FB Ann."], + ..BASE + }, + Case { + name: "footnote definition (fenced, 2nd para)", + body: "Text.[^my]\n\n::: ^my\nFirst.\n\nFN-FB2 {{< meta author >}}.\n:::\n", + needles: &["FN-FB2 Ann."], + ..BASE + }, + Case { + name: "footnote definition: unknown shortcode", + body: "Text.[^1]\n\n[^1]: FN-UNK {{< nosuchthing >}}.\n", + needles: &["?nosuchthing"], + unresolved_ok: true, + expect_diag: Some("Q-16-3"), + ..BASE + }, + Case { + name: "body: unknown shortcode (control)", + body: "UNK-CTL {{< nosuchthing >}}.\n", + needles: &["?nosuchthing"], + unresolved_ok: true, + expect_diag: Some("Q-16-3"), + ..BASE + }, + Case { + name: "footnote definition: unknown meta key", + body: "Text.[^1]\n\n[^1]: FN-KEY {{< meta nosuchkey >}}.\n", + needles: &["?meta:nosuchkey"], + unresolved_ok: true, + expect_diag: Some("Q-16-5"), + ..BASE + }, + Case { + name: "body: unknown meta key (control)", + body: "KEY-CTL {{< meta nosuchkey >}}.\n", + needles: &["?meta:nosuchkey"], + unresolved_ok: true, + expect_diag: Some("Q-16-5"), + ..BASE + }, + // ── YAML metadata values (rendered into the page) ─────────── + Case { + name: "yaml title", + meta: "title: \"YT {{< meta author >}}\"\n", + body: "Body.\n", + needles: &["YT Ann"], + ..BASE + }, + Case { + name: "yaml subtitle", + meta: "subtitle: \"YS {{< meta author >}}\"\n", + body: "Body.\n", + needles: &["YS Ann"], + ..BASE + }, + Case { + name: "yaml abstract", + meta: "abstract: \"YA {{< meta author >}}\"\n", + body: "Body.\n", + needles: &["YA Ann"], + ..BASE + }, + // These two redefine `author:` as a list/map, so the shortcode + // reads a separate `who: Ann` key instead. + Case { + name: "yaml author field (list of strings)", + meta: "who: Ann\nauthor:\n - Ann\n - \"YL {{< meta who >}}\"\n", + body: "Body.\n", + needles: &["YL Ann"], + ..BASE + }, + Case { + name: "yaml author nested map (affiliation name)", + meta: "who: Ann\nauthor:\n - name: \"Bob\"\n affiliation:\n - name: \"YN {{< meta who >}}\"\n", + body: "Body.\n", + needles: &["YN Ann"], + ..BASE + }, + Case { + name: "yaml description", + meta: "description: \"YD {{< meta author >}}\"\n", + body: "Body.\n", + needles: &["YD Ann"], + ..BASE + }, + Case { + name: "yaml date", + meta: "date: \"YDT {{< meta author >}}\"\n", + body: "Body.\n", + needles: &["YDT Ann"], + ..BASE + }, + ] +} + +fn render(case: &Case) -> Result<(String, Vec), String> { + let temp = TempDir::new().unwrap(); + let qmd_path = temp.path().join("doc.qmd"); + // A case's `meta` may itself define `author`; only add the default + // when it does not. + let author = if case.meta.contains("author:") { + "" + } else { + "author: Ann\n" + }; + let title = if case.meta.contains("title:") { + "" + } else { + "title: All Contexts\n" + }; + std::fs::write( + &qmd_path, + format!("---\n{title}{author}{}---\n\n{}", case.meta, case.body), + ) + .unwrap(); + let runtime: Arc = Arc::new(NativeRuntime::new()); + let result = render_to_file(&qmd_path, "html", &RenderToFileOptions::default(), runtime) + .map_err(|e| format!("render error: {e}"))?; + let html = + std::fs::read_to_string(&result.output_path).map_err(|e| format!("read output: {e}"))?; + let codes = result + .render_output + .diagnostics + .iter() + .filter_map(|d| d.code.clone()) + .collect(); + Ok((html, codes)) +} + +/// `Ok(())` when the case's expectation holds, otherwise why not. +fn check(case: &Case) -> Result<(), String> { + let (html, codes) = render(case)?; + let mut problems = Vec::new(); + for n in case.needles { + if !html.contains(n) { + problems.push(format!("missing {n:?}")); + } + } + for f in case.forbid { + if html.contains(f) { + problems.push(format!("unexpected {f:?}")); + } + } + if !case.unresolved_ok + && (html.contains("quarto-unresolved-shortcode") || html.contains("?meta")) + { + problems.push("unresolved-shortcode marker present".to_string()); + } + if let Some(code) = case.expect_diag + && !codes.iter().any(|c| c == code) + { + problems.push(format!("missing diagnostic {code} (got {codes:?})")); + } + if problems.is_empty() { + Ok(()) + } else { + Err(problems.join("; ")) + } +} + +#[test] +fn shortcode_expands_in_every_context() { + let mut rows = Vec::new(); + let mut bad = Vec::new(); + for case in cases() { + let result = check(&case); + let status = match (&result, case.known_gap) { + (Ok(()), None) => "PASS".to_string(), + (Err(_), Some(id)) => format!("KNOWN-GAP({id})"), + (Err(why), None) => { + bad.push(format!("{}: FAILED: {why}", case.name)); + "FAIL".to_string() + } + (Ok(()), Some(id)) => { + bad.push(format!( + "{}: known_gap {id} now PASSES; q2 now expands where Q1 does not \ + (decide deliberately, then remove the marker)", + case.name + )); + "UNEXPECTED-PASS".to_string() + } + }; + let detail = match &result { + Err(why) if case.known_gap.is_some() || status == "FAIL" => format!(" [{why}]"), + _ => String::new(), + }; + rows.push(format!("{status:<24} {}{detail}", case.name)); + } + assert!( + bad.is_empty(), + "\n{} problem(s):\n {}\n\nFull results:\n{}\n", + bad.len(), + bad.join("\n "), + rows.join("\n") + ); +} From 3b727a960c2732bb30def570a22078e66cd6861f Mon Sep 17 00:00:00 2001 From: Gordon Woodhull Date: Sun, 4 Oct 2026 02:56:57 -0400 Subject: [PATCH 2/2] Disable the vendored Lua shortcodes pass; Rust owns shortcode expansion (bd-2uva9urq) Escaped shortcodes in code spans/blocks were unescaped once by Rust and then expanded as live by the Lua pass, crashing pandoc (exit 83) on handlers Rust does not know (brand). Flag-gate the pre-shortcodes-filter entry in main.lua and crossref.lua off (q2_lua_shortcodes_disabled); shortcodes_filter() is still called because foldcode.lua's process_shortcodes needs _shortcodes_filter. - New shortcode_pandoc_escapes integration test (native writer); it failed with exit 83 before the flag-gate. - pandoc_filters marker test now ignores the unrelated shortcodes QUARTO2-PATCH marker. - README 'Ours vs. pinned' entries for main.lua and crossref/crossref.lua. --- .../2026-10-03-disable-lua-shortcodes.md | 40 +++++++--- crates/quarto-core/tests/integration/main.rs | 1 + .../tests/integration/pandoc_filters.rs | 8 +- .../integration/shortcode_pandoc_escapes.rs | 78 +++++++++++++++++++ resources/pandoc-filters/README.md | 12 ++- .../filters/crossref/crossref.lua | 4 +- resources/pandoc-filters/filters/main.lua | 8 +- 7 files changed, 136 insertions(+), 15 deletions(-) create mode 100644 crates/quarto-core/tests/integration/shortcode_pandoc_escapes.rs diff --git a/claude-notes/plans/2026-10-03-disable-lua-shortcodes.md b/claude-notes/plans/2026-10-03-disable-lua-shortcodes.md index 8eed32f0a..881b142ea 100644 --- a/claude-notes/plans/2026-10-03-disable-lua-shortcodes.md +++ b/claude-notes/plans/2026-10-03-disable-lua-shortcodes.md @@ -2,9 +2,8 @@ **Branch:** `issue-brand-shortcode` (from `feature/pandoc-wasm` @ `6d03c76ac`), worktree `workspace-7`. **Strands:** bug `bd-2uva9urq` (this plan); design question `bd-qwgu94f4` ("should pandoc Lua execute shortcodes?"). Related: `bd-xfqx2tuc`, `bd-qnylgu69`. -**Status:** REVIEWED 2026-10-03 (one edit round applied); not started. No code changed yet. -**Blocked by:** `bd-xjg7vl6c` (Rust never expands shortcodes in footnote definitions). T2 must not land -before it, see Prerequisite. +**Status:** EXECUTED 2026-10-04 (T1-T7 done; commits `7af327711` T1/T2/T4, `aecaef501` T5; not pushed). +**Was blocked by:** `bd-xjg7vl6c` (Rust never expands shortcodes in footnote definitions); fixed in `9f6c35190` before T2. ## Problem @@ -118,15 +117,15 @@ one `cargo nextest run --workspace` at the end (capture to a log; compare to the figure). Known red before this work: `pandoc_request_prepare::golden_file_is_structurally_current` and `::typst_request_matches_the_recorded_native_run` (lane R recording drift). -- [ ] **T1 Failing test first.** New `crates/quarto-core/tests/integration/shortcode_pandoc_escapes.rs` +- [x] **T1 Failing test first.** New `crates/quarto-core/tests/integration/shortcode_pandoc_escapes.rs` (register in `main.rs`, alphabetical). Use `render_document_to_file` with `native` (AST dump: no writer escaping, no typst compile; model on `gfm_shortcode_round_trip` in `pandoc_long_tail_formats.rs`). Cases: the bare `` `{{{< brand >}}}` ``, the arguments form, and an escaped shortcode in a fenced code block; assert the render succeeds and the output contains the literal `{{< brand ... >}}`. Add a control: a live `{{< brand color primary >}}` still only warns. Practical in CI: the existing long-tail tests already run real native pandoc with no skip. Confirm it fails (rc 83) before T2. Note `shortcode_text_contexts.rs` is HTML-only, so it never covered this. -- [ ] **T2 Off-switch** (after `bd-xjg7vl6c`): flag-gate the entry in `main.lua` and `crossref.lua` per the design above; README entries. -- [ ] **T3 Dependents (surveyed in review; re-confirm, then no changes expected).** No smoke-all or golden +- [x] **T2 Off-switch** (after `bd-xjg7vl6c`): flag-gate the entry in `main.lua` and `crossref.lua` per the design above; README entries. +- [x] **T3 Dependents (surveyed in review; re-confirm, then no changes expected).** No smoke-all or golden fixture uses `brand` or `contents` live; the only users are `docs/guides/authoring/brand.qmd` and the `pandoc-warm` e2e. `crates/pampa/tests/wasm_lua.rs` tests pampa's own `LuaShortcodeEngine` (extension shortcodes), not the vendored pandoc Lua: unaffected, and no pampa Lua file is edited so the wasm rule does not @@ -134,7 +133,7 @@ figure). Known red before this work: `pandoc_request_prepare::golden_file_is_str text (a `Str`), which Lua never expanded: unaffected. Also grep `^\[\^.*\]:.*\{\{<` in fixtures (none found). The wasm path builds its request from the same stage list as native (`build_pandoc_prefix_stages`, then `PandocPrepareStage`), so one Rust-resolved AST feeds both; T5 still verifies it. -- [ ] **T4 Goldens.** Editing `main.lua`/`crossref.lua` changes `share_tree_version` +- [x] **T4 Goldens.** Editing `main.lua`/`crossref.lua` changes `share_tree_version` (`pandoc_request/share.rs:54`, a hash of the embedded tree). `golden_file_is_structurally_current` asserts it equals `schemas/pandoc-request.golden.json`, so it **will go red because of this change**: regenerate with `Q2_REGENERATE_GOLDEN=1 cargo nextest run -p quarto-core golden_file` and commit the golden (job_id and @@ -146,15 +145,36 @@ figure). Known red before this work: `pandoc_request_prepare::golden_file_is_str `golden_file_is_structurally_current` and `typst_request_matches_the_recorded_native_run` **pass** in this worktree (13/13 `pandoc_request_prepare::`), so they are not known-red here; re-measure the workspace baseline at T7 instead of trusting the "known red" list. -- [ ] **T5 Fixture.** Remove the `BRAND` substitution in `hub-client/e2e/pandoc-warm.harness.spec.ts` +- [x] **T5 Fixture.** Remove the `BRAND` substitution in `hub-client/e2e/pandoc-warm.harness.spec.ts` so `brand.qmd` renders as written. `npm run build:wasm`, then `VITE_E2E=1 npm run build` and `npx playwright test --config playwright.harness.config.ts --project=chromium --workers=1 pandoc-warm`. -- [ ] **T6 Native sweep.** `q2 render` to typst and html for `docs/guides/authoring/brand.qmd` and `docs/guides/authoring/shortcodes.qmd` (its table of `{{{< meta key >}}}` code spans was silently double-expanded too, with no crash); confirm the +- [x] **T6 Native sweep.** `q2 render` to typst and html for `docs/guides/authoring/brand.qmd` and `docs/guides/authoring/shortcodes.qmd` (its table of `{{{< meta key >}}}` code spans was silently double-expanded too, with no crash); confirm the escaped examples show literally and note what the live `brand` shortcodes (lines 887, 898) now do. Report; do not fix `brand` support here (`bd-qnylgu69`). -- [ ] **T7 Workspace run, strand updates.** Record outcome on `bd-2uva9urq`; leave `bd-qwgu94f4` open. +- [x] **T7 Workspace run, strand updates.** Record outcome on `bd-2uva9urq`; leave `bd-qwgu94f4` open. Reconcile this checklist with reality before handing off. +## Outcome (2026-10-04) + +- Baseline at `9f6c35190`: 15733 run / 15733 passed / 202 skipped. Final: 15737 / 15737 / 202 skipped. Delta +4 = + the four `shortcode_pandoc_escapes` tests; no other count moved. +- T1: three escaped-shortcode cases failed with pandoc exit 83 (`brandCommand` nil, `shortcodes-handlers.lua:111`) + before T2; the live-`brand` control passed throughout. All four green after. +- T2 deviation from the plan: `pandoc_filters::test_shim_group_shape_and_patch_markers` pins exactly two + `QUARTO2-PATCH` markers in `main.lua`, each naming the shim group boundaries. The new marker made it red; the + test now ignores marker lines mentioning `bd-2uva9urq`. +- T4: golden regenerated (`job_id`, `share_tree_version` only). `goldenParity.wasm.test.ts` and + `pandocRequest.wasm.test.ts` pass (52 tests) once `quarto-output-extract` is built (the fixtures skip without + it). Recordings not re-recorded (stale relative to the live filter tree, unaffected). +- T5: `build:wasm`, `VITE_E2E=1 npm run build`, playwright `pandoc-warm`: 9 passed, 4 skipped; `brand.qmd` + renders unmodified (77301 bytes, warm equals fresh). +- T6 (scratch copy of `docs/guides/authoring`): `brand.qmd` html rc=0; escaped examples show literally in html + and typst. `brand.qmd` to typst fails for reasons unrelated to shortcodes (missing `docs/authoring/images/*.png` + screenshots, which typst treats as fatal and html only warns on; then a dangling `` link label). + The "live" `{{< brand ... >}}` at old lines 887/898 are inside fenced blocks marked `shortcodes="false"`, so no + live brand shortcode occurs in prose and none renders `?brand`. `shortcodes.qmd`: html and typst rc=0, the + `{{{< meta key >}}}` code-span table now shows literal `{{< meta key >}}`. Brand support still `bd-qnylgu69`. + ## Risks / open items - `bd-xjg7vl6c` must land first (see Prerequisite); if it slips, T2 waits. diff --git a/crates/quarto-core/tests/integration/main.rs b/crates/quarto-core/tests/integration/main.rs index 81ee38f40..cf6e72cf9 100644 --- a/crates/quarto-core/tests/integration/main.rs +++ b/crates/quarto-core/tests/integration/main.rs @@ -133,6 +133,7 @@ pub mod sass_cache_key; pub mod secondary_nav_pipeline; pub mod shortcode_all_contexts; pub mod shortcode_config_pipeline; +pub mod shortcode_pandoc_escapes; pub mod shortcode_text_contexts; pub mod sidebar_pipeline; pub mod synth_engines_e2e; diff --git a/crates/quarto-core/tests/integration/pandoc_filters.rs b/crates/quarto-core/tests/integration/pandoc_filters.rs index 1051b1d89..04b21ecb1 100644 --- a/crates/quarto-core/tests/integration/pandoc_filters.rs +++ b/crates/quarto-core/tests/integration/pandoc_filters.rs @@ -244,7 +244,11 @@ fn test_shim_group_shape_and_patch_markers() { .expect("main.lua not found in FILTERS_DIR"); let content = std::str::from_utf8(main_lua.contents()).expect("main.lua is not valid UTF-8"); - let marker_count = content.matches("QUARTO2-PATCH").count(); + // The shortcodes flag-gate (bd-2uva9urq) is an unrelated QUARTO2-PATCH + // that has nothing to do with the shim group boundaries; this test only + // covers the shim's two markers. + let is_shim_marker = |l: &str| l.contains("QUARTO2-PATCH") && !l.contains("bd-2uva9urq"); + let marker_count = content.lines().filter(|l| is_shim_marker(l)).count(); assert_eq!( marker_count, 2, "expected exactly two QUARTO2-PATCH markers (import region + tappend region), got {marker_count}" @@ -257,7 +261,7 @@ fn test_shim_group_shape_and_patch_markers() { let marker_line_indices: Vec = lines .iter() .enumerate() - .filter(|(_, l)| l.contains("QUARTO2-PATCH")) + .filter(|(_, l)| is_shim_marker(l)) .map(|(idx, _)| idx) .collect(); assert_eq!(marker_line_indices.len(), 2); diff --git a/crates/quarto-core/tests/integration/shortcode_pandoc_escapes.rs b/crates/quarto-core/tests/integration/shortcode_pandoc_escapes.rs new file mode 100644 index 000000000..79b62f201 --- /dev/null +++ b/crates/quarto-core/tests/integration/shortcode_pandoc_escapes.rs @@ -0,0 +1,78 @@ +/* + * tests/integration/shortcode_pandoc_escapes.rs + * Copyright (c) 2026 Posit, PBC + * + * bd-2uva9urq: escaped shortcodes in text contexts (code spans, code + * blocks) must survive the pandoc path as the literal `{{< ... >}}` the + * author asked for. Rust unescapes `{{{< x >}}}` once; a second, Lua-side + * shortcodes pass used to see the unescaped text and expand it as live, + * crashing on handlers (`brand`) that Rust does not know. + * + * Renders to `native` (pandoc AST dump) so there is no writer escaping and + * no typst compile. `shortcode_text_contexts.rs` is HTML-only and never ran + * the pandoc Lua stack. + */ + +use std::sync::Arc; + +use tempfile::TempDir; + +use quarto_core::render_to_file::{RenderToFileOptions, render_document_to_file}; +use quarto_system_runtime::{NativeRuntime, SystemRuntime}; + +/// Render `body` (no front matter beyond a title) to `native` and return the output. +fn render_native(body: &str) -> String { + let temp = TempDir::new().unwrap(); + let project_dir = temp.path().canonicalize().unwrap(); + let input_path = project_dir.join("esc.qmd"); + std::fs::write(&input_path, format!("---\ntitle: Esc\n---\n\n{body}\n")).unwrap(); + + let runtime: Arc = Arc::new(NativeRuntime::new()); + let result = render_document_to_file( + &input_path, + "native", + &RenderToFileOptions::default(), + None, + runtime, + None, + None, + None, + ) + .expect("native render should succeed"); + std::fs::read_to_string(&result.output_path).expect("native output readable") +} + +#[test] +fn escaped_bare_brand_in_code_span_stays_literal() { + let out = render_native("Use `{{{< brand >}}}` here."); + assert!( + out.contains("{{< brand >}}"), + "literal shortcode expected: {out}" + ); +} + +#[test] +fn escaped_brand_with_arguments_in_code_span_stays_literal() { + let out = render_native("Use `{{{< brand color COLOR_NAME VARIANT >}}}` here."); + assert!( + out.contains("{{< brand color COLOR_NAME VARIANT >}}"), + "literal shortcode expected: {out}" + ); +} + +#[test] +fn escaped_brand_in_fenced_code_block_stays_literal() { + let out = render_native("```\n{{{< brand color COLOR_NAME VARIANT >}}}\n```"); + assert!( + out.contains("{{< brand color COLOR_NAME VARIANT >}}"), + "literal shortcode expected: {out}" + ); +} + +/// Control: a live `brand` shortcode is unknown to Rust (warning only) and +/// was never reached by Lua; it must keep rendering without a crash. +#[test] +fn live_brand_shortcode_still_only_warns() { + let out = render_native("Color: {{< brand color primary >}}"); + assert!(!out.is_empty()); +} diff --git a/resources/pandoc-filters/README.md b/resources/pandoc-filters/README.md index 7eada7667..96bb4d783 100644 --- a/resources/pandoc-filters/README.md +++ b/resources/pandoc-filters/README.md @@ -117,7 +117,12 @@ The following files and modifications are *not* from `v1.11.3` and should be pre **Format for entries:** List items must be shaped exactly as `` - `` — description `` (backtick immediately after `- `) so the `cargo xtask lint` rule `vendored-pandoc-filters` can verify the path exists. The rule checks that every path listed here actually exists on disk. -- `resources/pandoc-filters/filters/main.lua` — patched twice, independently. +- `resources/pandoc-filters/filters/main.lua` — patched three times, independently. + (0) bd-2uva9urq: the `pre-shortcodes-filter` entry's flag is changed from + `has_shortcodes` to `q2_lua_shortcodes_disabled` (never set), marked + `QUARTO2-PATCH`. The `shortcodes_filter()` call stays, since + `process_shortcodes()` needs the `_shortcodes_filter` it assigns. + Re-enable by restoring the flag (see `bd-qwgu94f4`). (1) Task 8: two edits marked `QUARTO2-PATCH`: an `import("./quarto2-shim.lua")` line after the `customnodes/*.lua` import block, and a `tappend(quarto_filter_list, quarto_pandoc_shim_filters)` @@ -129,6 +134,11 @@ The following files and modifications are *not* from `v1.11.3` and should be pre `_quarto.modules.crossref_numbering.assign_crossref_numbers()`, and add a fail-fast guard rejecting `crossref-numbering: external` combined with a LaTeX-family or Typst target. +- `resources/pandoc-filters/filters/crossref/crossref.lua` — patched (ours, + bd-2uva9urq; no upstream PR — marked `QUARTO2-PATCH`). The + `pre-shortcodes-filter` entry's flag is changed from `has_shortcodes` to + `q2_lua_shortcodes_disabled`, the same flag-gate as in `main.lua`; Rust owns + shortcode expansion, and the Lua pass double-expanded escaped shortcodes. - `resources/pandoc-filters/filters/quarto2-shim.lua` — new (ours, Task 8; body implemented across P5 Tasks 1-8, plus a post-review fix wave). Decodes Q2's wire-format `CustomNode` scaffold back into real Q1 nodes, diff --git a/resources/pandoc-filters/filters/crossref/crossref.lua b/resources/pandoc-filters/filters/crossref/crossref.lua index 0629197a8..891631b7c 100644 --- a/resources/pandoc-filters/filters/crossref/crossref.lua +++ b/resources/pandoc-filters/filters/crossref/crossref.lua @@ -166,9 +166,11 @@ local quarto_pre_filters = { { name = "flags", filter = compute_flags() }, + -- QUARTO2-PATCH (bd-2uva9urq, claude-notes/plans/2026-10-03-disable-lua-shortcodes.md): + -- disabled the same way as in main.lua; keep the shortcodes_filter() call. { name = "pre-shortcodes-filter", filter = shortcodes_filter(), - flags = { "has_shortcodes" } }, + flags = { "q2_lua_shortcodes_disabled" } }, } local quarto_crossref_filters = { diff --git a/resources/pandoc-filters/filters/main.lua b/resources/pandoc-filters/filters/main.lua index d81fbb32d..e45aebeab 100644 --- a/resources/pandoc-filters/filters/main.lua +++ b/resources/pandoc-filters/filters/main.lua @@ -316,9 +316,15 @@ local quarto_pre_filters = { traverser = 'jog', }, + -- QUARTO2-PATCH (bd-2uva9urq, claude-notes/plans/2026-10-03-disable-lua-shortcodes.md): + -- Q2 resolves shortcodes in Rust before pandoc runs; this second pass only + -- double-expanded the already-unescaped text in code/attributes. The flag + -- below is never set, so the runner skips the entry. `shortcodes_filter()` + -- must still be called: it assigns the `_shortcodes_filter` that + -- `process_shortcodes()` (quarto-post/foldcode.lua) walks with. { name = "pre-shortcodes-filter", filter = shortcodes_filter(), - flags = { "has_shortcodes" }, + flags = { "q2_lua_shortcodes_disabled" }, traverser = 'jog', },