Skip to content

FIX: preserve non-ASCII whitespace in headings and paragraphs - #469

Open
simpleqt wants to merge 1 commit into
executablebooks:masterfrom
simpleqt:fix/preserve-non-ascii-whitespace-in-blocks
Open

simpleqt wants to merge 1 commit into
executablebooks:masterfrom
simpleqt:fix/preserve-non-ascii-whitespace-in-blocks

Conversation

@simpleqt

Copy link
Copy Markdown

Port of markdown-it #1074 (released in markdown-it 14.2.0): non-ASCII whitespace at the edges of block content is content, not trimming.

What is wrong

The block rules (heading, lheading, paragraph) trim with mdTrim, whose character set includes U+00A0 and the U+2000-U+200A and U+3000 space characters. CommonMark only treats ASCII whitespace as trimmable there, so "# h\u00a0" currently renders as <h1>h</h1>, while markdown-it 15.0.3 renders <h1>h\u00a0</h1>. The same loss happens in setext headings and paragraphs.

Fix

Add the upstream ASCII-only asciiTrim to markdown_it/common/utils.py and use it in those three block rules. mdTrim is left in place where upstream fully trims (table cells, fence info strings), so this PR does not change those behaviours.

Tests

  • New parametrized cases in tests/test_port/test_whitespace.py for U+00A0 and U+3000 preserved in headings and paragraphs, plus the upstream-#1074 fixture case.
  • The existing expectations in that file encoded the 14.1.0 behaviour (these spaces stripped); they are updated to the 14.2.0+ upstream output.

Evidence on base d504583 with the new tests applied and the four source files reverted:

  • pytest tests/test_port/test_whitespace.py -q -> 3 failed, 15 passed: test_non_ascii_whitespace_is_preserved_in_blocks[\xa0], [\u3000] and test_non_ascii_trim_matches_upstream_fixture.
  • With the fix: same command -> 18 passed; full pytest tests -q -> 1160 passed, 33 skipped; ruff check / ruff format --check clean; mypy -> Success: no issues found in 66 source files.
  • A differential run over 4000 CommonMark inputs against node markdown-it 15.0.3 reduced divergences from 266 to 18, none of them block-whitespace related; the remaining diffs are the port's known intentional differences (FEFF handling, fence info trimming, lowercase inline doctype).

Signed-off-by: simpleqt 2015900818@qq.com

Port markdown-it #1074 (released in 14.2.0): block content is trimmed with ASCII whitespace only, so Unicode spaces such as U+00A0 and U+3000, which are content per CommonMark, are no longer dropped from headings, setext headings and paragraphs.
Copilot AI balanced review requested due to automatic review settings November 14, 2026 15:39

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants