Let Rust own shortcode expansion: fix escaped shortcodes crashing pandoc, and expand them in footnote definitions - #789
Merged
Conversation
Rust shortcode_resolve skipped NoteDefinitionPara, NoteDefinitionFencedBlock and the lst-cap attribute of listing code blocks; Quarto 1 expands all three. Add a parameterized all-contexts test; contexts Q1 also leaves literal are pinned as such.
…on (bd-2uva9urq) Escaped shortcodes in code spans/blocks were unescaped once by Rust and then expanded as live by the Lua pass, crashing pandoc (exit 83) on handlers Rust does not know (brand). Flag-gate the pre-shortcodes-filter entry in main.lua and crossref.lua off (q2_lua_shortcodes_disabled); shortcodes_filter() is still called because foldcode.lua's process_shortcodes needs _shortcodes_filter. - New shortcode_pandoc_escapes integration test (native writer); it failed with exit 83 before the flag-gate. - pandoc_filters marker test now ignores the unrelated shortcodes QUARTO2-PATCH marker. - README 'Ours vs. pinned' entries for main.lua and crossref/crossref.lua.
Contributor
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Escaped shortcodes such as
`{{{< brand >}}}`crashed pandoc with exit 83 whenever the page was rendered through the pandoc path. Rust unescapes{{{< x >}}}once, and the vendored Lua shortcodes pass then expanded the result as a live shortcode. For handlers that Rust doesn't know (brand), the Lua side hit a nilbrandCommandatshortcodes-handlers.lua:111.The fix makes Rust the only place shortcodes are expanded. Two commits:
lst-cap(bd-xjg7vl6c).shortcode_resolveskippedNoteDefinitionPara,NoteDefinitionFencedBlockand thelst-capattribute of listing code blocks, which the Lua pass had been covering. Quarto 1 expands all three, so Rust now does too. This has to land first. Without it, turning Lua off would leave[^1]: Note {{< meta author >}}.unexpanded.pre-shortcodes-filterentries inmain.luaandcrossref/crossref.luanow requireq2_lua_shortcodes_disabled, a flag that is never set, so the runner skips them. This is a flag-gate rather than a removal.shortcodes_filter()must still be called becausefoldcode.lua'sprocess_shortcodesuses_shortcodes_filter. Both edits carry aQUARTO2-PATCHcomment, and the README's "Ours vs. pinned" section documents them.Tests
shortcode_all_contexts: a parameterized test over every context where a shortcode can appear. Contexts that Quarto 1 also leaves literal are pinned as literal.shortcode_pandoc_escapes: renders tonativeand checks that escaped brand shortcodes stay literal in a code span, a code span with arguments and a fenced block. A control confirms that a live{{< brand >}}still only warns. Three of the four cases failed with exit 83 before the flag-gate.pandoc_filters::test_shim_group_shape_and_patch_markerspinned exactly twoQUARTO2-PATCHmarkers inmain.lua. It now ignores the unrelated shortcodes marker.Not in this PR
{{< brand >}}still doesn't render. That stays with bd-qnylgu69.pandoc-request.golden.jsonstores a hash of the embedded filter tree (share_tree_version), so editing the Lua changes it. That file only exists onfeature/pandoc-wasm. Whichever branch lands second needs oneQ2_REGENERATE_GOLDEN=1regeneration. The pandoc recordings are self-contained and don't need re-recording.Test plan
cargo nextest run --workspace(running locally; result will be posted as a comment)