✨ Add downstream seams: opt-in numbering, node factories, app lexers, markup helpers

[chrisjsewell]

The bespoke "example" directives in myst-parser and sphinx-needs can be
replaced by a subclass of SyntaxExampleDirective — but only if the subclass
reaches exact parity with what those projects already ship, since neither will
accept a regression in its own rendered docs. Auditing both against v0.1.1 turned
up five gaps. This PR closes them. Every change is additive: default behaviour is
byte-identical, and the existing tests pass unchanged.

Opt-in numbering — syntax_example_numbering

sphinx-needs is renaming its need-example usages to the canonical
syntax-example directive, and wants its numbered rubrics back as a feature
rather than a subclass hack. New config value (bool, default False, env
rebuild category — it changes title text written into doctrees, so toggling it
must re-read sources):

.. syntax-example::          → "Example 1"
.. syntax-example:: Custom   → "Example 2: Custom"

which is exactly the shape need-example produced ("Example 5: filter
option"). A matching option is being added to ubCode's Rust engine under the
same config name, to produce the same output.

Design decisions worth reviewing:

• The config check lives in format_title's canonical body, so a subclass that
  overrides format_title opts out wholesale rather than being numbered behind
  its own back.
• Numbering decorates the label, so a non-empty default_title is a
  precondition. A subclass with default_title = "" is untouched by the config:
  no argument still means no rubric, an argument still means exactly that
  argument.
• The counter is keyed by the registered directive name, so a downstream alias
  numbers independently of syntax-example in the same document.

Honesty pass on the prose. The README and module docstring claimed "no
auto-numbering and no cross-document state / a pure function of the directive
instance". Numbering is now possible, so those paragraphs are rewritten around
what is actually true and actually load-bearing: the state is per document
(env.temp_data, which Sphinx replaces for each source file it reads), and a
document is exactly the unit of both parallel reading and incremental
rebuilding. Re-reading one changed file reproduces the numbers of a full build,
and nothing crosses between parallel workers. No cross-document state, ever.

build_wrapper_node() / build_render_node()

myst-parser's docs wrap examples in sphinx-design-style divs (is_div=True,
design_component="div"), so that extension's overridden container visitor
emits <div class="myst-example docutils"> without the container class —
which would otherwise attract Bootstrap and pydata-theme .container layout
rules and break the frame. Previously the only way to get a non-container node
was to re-implement run(), i.e. to fork the whole assembly. These two factories
are the seam; run() still sets the source info on whatever the wrapper factory
returns.

The two responsibilities are kept orthogonal: a factory decides the node
type and any extra attributes, and run() applies wrapper_classes /
render_classes to whatever comes back (merging with any class the factory set,
without duplicating) — consistent with how it already applies source_classes
to the literal block. The class attributes therefore remain the single source of
the CSS classes: an override neither has to repeat them nor can silently drop
them and render unstyled. The default factories return a bare
nodes.container(), and default output is byte-identical.

Sphinx-app-registered lexer awareness

_lexer_available() consulted only Pygments' global registry, so a lexer added
with app.add_lexer() was invisible: myst-parser's docs register a local
MystLexer under "myst", and :highlight: myst silently fell back to the
inferred language (as did default_language's myst probe). It now checks
sphinx.highlighting's lexer_classes and lexers first — the same registries
PygmentsBridge.get_lexer resolves against — then Pygments. Both module
attributes are read via getattr and checked for mapping-ness, so a rename or
reshape in a future Sphinx degrades to the Pygments-only probe rather than
crashing; verified against the supported floor (7.2) and current (9.x). The probe
stays a dict membership test, read-only and side-effect free.

Side effect worth noting: Sphinx's own built-in aliases live in the same
registry, so :highlight: none — a valid Sphinx highlight language that Pygments
does not resolve — is now honoured instead of falling back. Documented in the
changelog and covered by a test.

per_document_number(key=None)

The counter behind syntax_example_numbering, exposed for a format_title
override that numbers on its own terms — the seam sphinx-needs' need-example
needs for its numbered rubrics. Default key is namespaced by the registered
directive name, case-folded, since reStructuredText directive names are
case-insensitive and two spellings of one directive must share a run. The
README's numbering recipe, which hand-rolled a temp_data counter, is
simplified to a one-liner using it.

nested_parse_text(text, container)

render_into receives the directive's content as a pre-built line list; an
override that renders something else — myst-parser's :alt-output:, where the
render pane shows alternative markup while the source pane shows the verbatim
content — has only a string. This helper wraps it in a StringList with the
directive's own source and content offset, so a warning raised by the parsed
markup points into the directive rather than at line 1 of an anonymous block, and
routes it through state.nested_parse — the host document's parser, so the
same call renders reStructuredText in an .rst file and MyST in a .md one,
with no format sniffing.

Tests

Seventeen new tests in the existing style (build-level bar one), plus an autouse
fixture that snapshots and restores sphinx.highlighting's lexer registries
around each test (app.add_lexer writes into module globals, which — unlike the
docutils registrations docutils_namespace already handles — would leak between
in-process builds).

Covered: is_div attributes surviving into the doctree from both factories with
the class attributes still applied and source info still set on the wrapper, and
a factory-set class merged rather than replaced; an app.add_lexer'd lexer
honoured via :highlight: under warningiserror=True, with the name asserted
unknown to Pygments before and after, plus :highlight: none; the counter
resetting across a two-document project and shared across two spellings of one
directive name; nested_parse_text in both an RST and a MyST host document, plus
its warning attribution in each; and, for numbering, default-off, on,
per-document reset, alias independence, an overridden format_title untouched,
and a labelless subclass untouched.

Each new test was mutation-checked: reverting the behaviour it covers makes it
fail. Note that the MyST attribution test pins the lineno-vs-content_offset
confusion (5 and 0 there, against 4 and 5 in the reStructuredText equivalent),
not 0 vs content_offset — MyST's own value is 0 for a plain fence, which the
test docstring says.

Verified: pytest (34 passed) on Sphinx 7.2.6, 8.2.3 and 9.1.0; ruff check,
ruff format --check, mypy --strict against both the 7.2 floor and 9.x;
sphinx-build -nW --keep-going on docs/.

No version bump — this is the 0.2.0 feature set staged for a later release
commit.

[chrisjsewell]

Adversarial review — correctness, performance, durability, complexity, mission fit

Verdict: solid, merge-worthy work — with one genuine test gap that should be fixed first, and one stale docstring claim. Everything the PR body asserts that I could execute, I did execute, and it held up. The one claim that didn't fully hold is "each test was mutation-checked" — true for the new behaviors, but the refactor quietly un-covered an old one.

What was verified (not just read)

• 34/34 tests pass on Sphinx 9.0.4 and on the true floor 7.2.6 (docutils 0.20.1, myst-parser 4.0.1); ruff + mypy --strict clean against both ends via the dual pre-commit hooks; docs build -nW clean.
• Default behavior is byte-identical: all 17 pre-existing tests pass unchanged.
• The temp_data scoping claims are real, not prose. Probes run: a 3-document numbered project builds correctly; appending an example to one doc and rebuilding incrementally reproduces full-build numbers everywhere (unchanged docs untouched, re-read doc correct); a parallel=2 read produces correct per-document numbers; env is the right rebuild category.
• The durability guard works: deleting both sphinx.highlighting registries (simulating a future rename) degrades _lexer_available to the Pygments-only probe — python still resolves, none cleanly stops resolving, no crash.

Findings

1. (should-fix) The Pygments-resolution path is now a mutation survivor. Coverage fell 100% → 99.18%, and the uncovered line is _lexer_available's return True after a successful get_lexer_by_name (src/sphinx_syntax_example/__init__.py:161). The cause is subtle: python is in sphinx.highlighting.lexer_classes, so test_highlight_override — which used to exercise the Pygments path — now short-circuits at the new Sphinx-registry check. Verified empirically: hard-breaking the Pygments branch (return True → return False) passes all 34 tests. A regression in the code path every plain-Pygments :highlight: value (ruby, json, …) depends on would ship green. One-line fix: a build test with :highlight: ruby, or assert _lexer_available("ruby") alongside the existing registry asserts.

2. (should-fix, one sentence) The honesty pass missed a sentence. The module docstring still claims the extension "relies only on the stable public get_filetype / config.source_suffix API" — no longer true now that a documented feature reads sphinx.highlighting.lexer_classes / lexers (semi-private, name-guarded). The guard is exactly right; the sentence isn't.

3. Nitpicks. Docstring and PR body say the counter key is "case-folded" but the code uses .lower() (identical for realistic names — align the word or the code). per_document_number(key="") silently falls into the default key via key or …; key is None would be stricter. The README's DivExample snippet uses nodes.container without showing the nodes import, and AltOutputExample calls an undefined self.alternative_markup() — fine as illustration, breaks on paste. test_app_registered_lexer_does_not_leak is order-dependent (vacuous when run alone) — acceptable as a fixture canary, worth a comment saying so.

Design observations (accepted, not defects)

• Read-time numbering means content under ifconfig/only consumes numbers even when excluded from output — inherent to the approach and the same as sphinx-needs.
• A titled example still consumes a number (Example 2: Custom) — intended, matches the need-example shape, tested.
• The source pane gets no factory (literal_block is hard-coded) — asymmetric with the two new seams, but no motivating case; noting it so the asymmetry is a decision, not an accident.
• If Sphinx ever renames the registries, app-lexer :highlight: values degrade silently (verbose note only). That's the documented tradeoff of the guard, and the right one for a -W-safe directive.

Correctness, performance, complexity

format_title's numbering logic handles all the edge combinations correctly (off/on × arg/no-arg × empty default_title — all tested); _apply_classes merges in place without duplicates (tested, and the orthogonal factory/classes contract is genuinely good design — the stylesheet can't be silently orphaned by an override). nested_parse_text's attribution is honest about being approximate, and the MyST-vs-RST offset test is unusually careful. Performance is a non-issue: the new probes are dict lookups that short-circuit before Pygments' plugin scan, and numbering is one temp_data get/set per directive. Complexity growth is real (config value + two factories + two helpers) but each addition is small, orthogonal, and named after a specific downstream consumer.

Mission fit

Yes — with conviction. The module docstring has named need-example-style numbering and myst-parser's :alt-output: as the intended downstream cases since v0.1.0; this PR builds precisely those seams and nothing speculative. The old "pure function / no auto-numbering" pillar is formally weakened (output can now depend on document position), but the rewrite keeps the load-bearing guarantees — parallel safety and incremental reproducibility — verified here empirically rather than taken on the prose's word. The rewritten paragraphs are accurate; finding 2 is the only spot the honesty pass missed.

Recommendation: address findings 1 and 2 (both are minutes of work), then merge. The nitpicks can ride along or wait.

---

Generated by Claude Code

[chrisjsewell]

Both should-fix findings and the nitpicks are addressed in 52b7567:

• Finding 1 — added test_pygments_only_lexer_is_honoured (:highlight: ruby: in Pygments, in neither Sphinx registry) to cover the probe's Pygments arm. Coverage 99.18% → 100%, and re-running your mutant (hard-breaking the return True after get_lexer_by_name) now fails exactly that one test. The test asserts its own precondition (ruby absent from both sphinx.highlighting registries), so a future Sphinx registration can't silently un-cover the arm the same way python did.
• Finding 2 — the module docstring now names sphinx.highlighting's lexer registries as the one deliberate non-public dependency, read by name and mapping-checked, and states the degradation contract: a rename costs the app.add_lexer awareness and nothing more (probe falls back to Pygments alone, never raises).
• Nitpicks — "case-folded" → "lowercased" to match the str.lower() the code calls; per_document_number now uses key is not None, so an explicit "" is a real key rather than silently the default; both README snippets are paste-runnable (imports added, and AltOutputExample takes its markup from an :alt-output: option instead of an undefined method) with the alt-output recipe pinned by a new test, since it doubles as the myst-parser parity claim; the leak test is documented as an order-dependent canary for the registry-isolation fixture.

Re-verified on the head commit: 36 tests pass on Sphinx 7.2.6 / 8.2.3 / 9.1.0, ruff check + ruff format --check clean, mypy --strict clean on both the 7.2 floor and 9.x, and the docs build -nW clean.