fix(md-exec): heredoc body containing a ``` line is mistaken for the end of the code block - #259
Conversation
osr21
left a comment
There was a problem hiding this comment.
Full executable verification this time (Python is in my environment, unlike the Rust/shell gaps I've disclosed elsewhere) — every claim reproduces:
- Bug confirmed on
main: your exact repro file fails with the exact warning quoted (here-document at line 3 delimited by end-of-file (wanted 'EOF')), 0 passed 1 failed. - Fix confirmed on your branch: same repro passes; the parser now correctly carries the full heredoc body — including the embedded
```line — into the command. - Self-test suite: 29/29 on main, 30/30 on your branch — and this time the pre-existing count in the PR body is right too.
- Real-world exposure confirmed:
docs/running-an-arc-node.mdhas 3 heredocs on main, andmake test-unit-contractdoes runmd-exec.pyovercrates/quake/tests(Makefile line 262), so the parser is live test infrastructure, not just self-documentation. - Scope check: the diff is exactly the one condition (
and heredoc_delimiter is None) plus an explanatory comment and the regression case inmd-exec.md. The state machine already clearsheredoc_delimiteron the delimiter line before the next fence check, so the fix composes correctly with the existing tracking — no new state introduced.
One edge case worth a sentence in the comment, though I don't consider it blocking: if a code block contains a heredoc whose delimiter line never appears (malformed markdown), the parser now also skips the real closing fence and silently swallows the rest of the file into that command. Pre-fix, the same malformed input produced a confusing-but-visible failure. That's a strictly-broken-input scenario and the old behavior wasn't good either, so I'd merge as-is — but if you want to harden it, resetting heredoc_delimiter with a warning when EOF is reached inside an open heredoc would make the failure mode loud again.
Also appreciated the restraint in the notes: not "fixing" the 4-dot ellipsis ambiguity you found while fuzzing, because the spec only defines ..., is the right call — a behavior-defining discussion first is exactly how that should go.
LGTM. Small, correctly-scoped, honestly-documented, and verified end-to-end.
fix(md-exec): heredoc body containing a ``` line is mistaken for the end of the code block
The bug
scripts/md-exec.py'sMarkdownParser.parse()detects code-fenceboundaries by checking whether a line, stripped, starts with
```.This check runs unconditionally on every line inside a code block —
including lines that are part of an open heredoc body being fed to a
shell command. If a heredoc body contains a line starting with
```(for example, a heredoc that writes out a markdown file, or a snippet that
itself documents markdown syntax), the parser treats that line as the end
of the code fence, cutting the command short mid-heredoc. The shell then
receives an unterminated heredoc.
Reproduction — this file:
fails today with:
heredoc_delimiteris already tracked by the parser (set when a<< DELIMcommand starts, cleared when the delimiter line is seen), but thefence-detection check doesn't consult it.
This isn't just a contrived case:
docs/running-an-arc-node.mdalreadyuses heredocs to write out full files (
cat << "EOF" > ~/.arc_env, twosudo tee ... <<EOFsystemd unit blocks), andmake test-unit-contractruns
python3 scripts/md-exec.py crates/quake/testsas part of the testsuite — so this is live infrastructure, not just the tool's own
self-documentation.
The fix
One-line change: only treat a
```-prefixed line as a fence boundarywhen we're not currently inside an open heredoc body.
Testing
scripts/md-exec.md(the tool's ownself-test suite, run via
python3 md-exec.py md-exec.md) covering aheredoc body with an embedded ``` line.
git stashthe.pychange, rerun — reproduces the exact bash heredoc warning above) and
passes with it.
regressions.
python3 -m py_compile scripts/md-exec.py— compiles cleanly.Notes for reviewers
touched.
check_match) functionseparately looking for bugs there; found one genuinely ambiguous edge
case (4+ consecutive literal dots, e.g.
foo....bar), but the docs onlydefine behavior for exactly
..., so I didn't treat that as a bug orinclude a "fix" for it here — happy to open a separate discussion/PR if
you'd like that behavior formally defined.