AI junk

[zhangj23]

Markup.striptags removes tags with value = f"{value[:start]}{value[end + 1:]}" inside a loop, which copies the whole remaining string once per tag. That makes it quadratic in the number of tags.

Tags are now collected in a single left-to-right pass and joined once.

That is safe because the tag start mark is a single character: removing a tag can never join two neighbouring characters into a new <, so one pass finds exactly the same tags that repeatedly searching from the beginning did. The comment-removal loop just above is deliberately left alone, since <!-- is multi-character and splicing there genuinely can forge a new mark.

Numbers

time.process_time, alternating base and patched subprocesses, min of 5, on a paragraph of <span class='x'>word</span> repeated n times:

tags	main	this branch	speedup	main growth	branch growth
500	0.611 ms	0.165 ms	3.7x	x2.33	x1.83
1000	6.415 ms	0.365 ms	17.6x	x10.50	x2.21
2000	30.368 ms	0.889 ms	34.2x	x4.73	x2.44

The growth columns are the point. main grows much faster than 2x per doubling while this stays near 2x, so what changes is the growth rate rather than the constant factor, and the gap keeps widening with tag count.

striptags is not in _speedups.c, so this is the code path that actually runs whether or not the C extension is compiled. I checked that before measuring.

Two costs, both measured

Input with exactly one tag is about 6 percent slower (0.939x), because a single tag cannot amortise the list allocation and the join. Two tags is already 1.012x, four is 1.057x, and a ten-tag bio string is 1.131x. So the crossover is at two tags.

I tried adding a single-tag fast path to remove that. It worked for one tag and simply moved the regression to the two-tag case (0.898x), because the lookahead it needs is itself unamortised there. So this looks inherent to batching at very small tag counts rather than something to engineer away, and I would rather report it than keep trading it around.

Peak memory is higher on tag-dense input, because the chunk list coexists with the input until the join. On Markup("<>" * 200000), tracemalloc peak goes from 1,200,119 to 2,072,916 bytes, about 1.7x. main's peak is bounded by roughly 2x the input; this is bounded by that plus about 8 bytes per tag for the list slot. For a template variable that is negligible, but it is a real change in the shape of the memory use and worth stating.

Behaviour

Differential testing against main: 34 hand-written cases plus 40,000 randomized fuzz cases over an alphabet of <, >, <a>, </a>, <!--, -->, newline, ampersand and letters, comparing exact output. 0 mismatches.

The cases cover unclosed tags, comments interleaved with tags, bare <> pairs, nested tags, a tag containing a quoted >, non-ASCII inside a tag, and the entity-looking strings <a> and A. I also checked the route a user actually reaches this by, Jinja's |striptags filter, on ordinary markup.

Checks

pytest gives 79 passed, 1 skipped, exit code 0, matching main. ruff check src and ruff format --check src are both clean.

[davidism]

https://palletsprojects.com/contributing/llm-ai