Skip to content

Use ASCII case folding for HTML element and attribute names - #200

Merged
wRAR merged 2 commits into
scrapy:masterfrom
ac1982:fix/ascii-html-name-folding
Oct 14, 2026
Merged

wRAR merged 2 commits into
scrapy:masterfrom
ac1982:fix/ascii-html-name-folding

Conversation

@ac1982

@ac1982 ac1982 commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

HTMLTranslator currently Unicode-lowercases element and attribute names. Given <x-Ä id="upper"></x-Ä><x-ä id="lower"></x-ä>, selecting x-Ä incorrectly returns lower. [DATA-Ä] likewise selects a data-ä attribute instead of data-Ä.

Use the existing ascii_lower helper for HTML name normalization, consistent with HTML/CSS ASCII case-matching rules. ASCII portions still fold (X-Ä matches x-Ä), while non-ASCII characters retain their identity. XML/XHTML and the attribute-value extension point are unchanged.

Two regressions fail against master and pass with the fix; they assert the generated XPath directly, covering ordinary, mixed-ASCII-case, and wildcard-namespace selectors. The original lxml-parsed HTML reproductions were also rerun separately against the revised tree and passed. This is unrelated to #50 (Python 2 byte-string input), #189/#191 (tokenization), and the recent attribute-value i flag support.

Validation on Python 3.14.4 / Linux arm64:

  • Review revision: 29 tests/docs passed, mypy, pylint and all pre-commit hooks passed; both simplified regressions fail against unchanged master.
  • Original submission checks (builds not rerun for this test-only revision):
  • Full tests including documentation examples: 29 passed; XPath module 100% coverage, total 99%.
  • Ruff lint/format, mypy (5 source files), pylint, complete pre-commit hooks, Sphinx build with -W, sdist build / twine check, and git diff --check passed.
  • Other Python versions, PyPy, macOS and Windows were not run locally.

AI assistance: OpenAI Codex investigated, implemented, and ran these checks.

@AdrianAtZyte

Copy link
Copy Markdown
Contributor

Could you simplify the tests to assert on the generated XPath, like most tests in that file do? e.g. assert xpath("X-Ä") == "descendant-or-self::x-Ä".

@codecov

codecov Bot commented Oct 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.79%. Comparing base (805705c) to head (fa9244e).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #200   +/-   ##
=======================================
  Coverage   99.79%   99.79%           
=======================================
  Files           3        3           
  Lines         987      987           
  Branches      161      161           
=======================================
  Hits          985      985           
  Misses          1        1           
  Partials        1        1           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@AdrianAtZyte
AdrianAtZyte requested review from wRAR and a balanced review from Copilot and removed request for Copilot October 13, 2026 11:28
@wRAR
wRAR merged commit 99ec093 into scrapy:master Oct 14, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants