[SizzleUnrlsd]

Summary

Closes #288.

Each position in a template was counted from its start. _position read the template up to the offset of each filter it applies, of each include and extends, and of each {% load %} the engine cannot resolve. A template applying n filters was therefore read n times, which is quadratic in its size: 160,000 filters on one line (2 MB) took 49 s to collect, where Django compiles and renders the template in 2.7 s.

The start of each line of a template is now found once, when it is read, and the line of each position by bisection:

• One line index for the engine. The index that places the values of JSON and TOML documents, _Lines of source/positions.py, moves to source/manager.py, beside LINE_BREAK and lines_of, whose count of lines it follows. It becomes LineIndex, exported by coretrace_python.source.
• Positions found while a template is read. _read_scopes builds the template's index once. Its filters, and now its includes, extends and loads, get their SourceSpan from it. project_templates takes the line of an unread entry from that span. _position is removed.

What does not change.

• Positions. Templates are read with universal newlines, so their text holds only \n, where LineIndex and _position count alike. The filters, their order and the unread entries, lines included, stay the same.
• Interfaces. _read_scopes keeps its signature. LineIndex is the only new public name. The Enterprise plugins use none of the internals that change.

The design and its measurements are in the issue.

Commit	Change
refactor(source): share the line index of JSON and TOML positions	LineIndex in source/manager.py, exported; tests/test_line_index.py
refactor(templates): give each include, extends and load of a template its span	spans read with the template, still counted by _position
fix(templates): find the positions in a template from a line index built once	the template's LineIndex in _read_scopes, _position removed; tests/test_template_positions.py
docs(changelog): announce the positions found in linear time	CHANGELOG.md, under Performance

Results

Same results. On 436 projects, this branch gives the filters, their order and the unread entries of main, positions included: 11,648 filters and 3,132 unread entries.

• Local projects: 36 (14 Django packages, Django 5.1.4 and the 21 regression projects).
• Generated projects: 400 projects mixing filters, includes, extends, loads, comments and {% filter %} blocks, with \n, \r\n and lone \r line ends, and characters outside ASCII.

Time. project_templates on one template, on Python 3.12.3, an AMD Ryzen 9 9950X3D under Ubuntu 24.04 (WSL2), minimum of three runs:

Template	main	This branch
{{ v|upper }} × 10,000, on one line (126 KB)	0.26 s	0.08 s
× 40,000 (507 KB)	3.26 s	0.37 s
× 160,000 (2 MB)	48.92 s	2.21 s
{% include v %} × 80,000, one per line	13.97 s	0.52 s
{% load missing %}{{ v|upper }} × 80,000, one per line	27.87 s	0.79 s

• Full analysis (--check) of the 2 MB template: 3.26 s instead of 49.74 s.
• Local projects: project_templates takes 2.72 s on the 36 projects, as on main (2.69 s).

Tests

New files. No existing test changes.

• tests/test_line_index.py: the line and column of every offset of a text broken by \n, \r\n and a lone \r, holding a character outside ASCII, and each offset back from its line and column.
• tests/test_template_positions.py: one template of 200,000 units, collected in a subprocess bounded at 60 s, as the time tests of templates: collecting filter calls takes time exponential in nested includes #279 and templates: the filter walk is exponential in a cycle of includes, and cuts a cycle Django renders through #284 do. Filters on one line, the last at line 1, column 2,599,993; includes named by an expression, then loads of a library not found, one per line, the last unread entry at line 200,000.

Before the change.

• On main: tests/test_line_index.py cannot import LineIndex, and each case of tests/test_template_positions.py is stopped at 60 s.
• On the second commit, just before the fix: the three cases are still stopped at 60 s.
• With the fix: the filters take 1.69 s, the includes 1.16 s and the loads 1.66 s.

Mutations. Each of these makes tests fail:

• the line of an offset found by bisect_left, which puts an offset at a line start on the line before: tests/test_line_index.py and 7 tests of JSON and TOML positions;
• the line index counting only \n: both tests of tests/test_line_index.py;
• the line of an unread entry taken from its column: 2 tests of tests/test_template_filter_calls.py and the includes case;
• the line index built again for each position: the filters case, stopped at 60 s.

Verification

Python 3.12 (macOS), on main at 07adc12:

• ruff check . and mypy: clean on each commit;
• unit suite: 2,671, 2,671, 2,674 and 2,674 passed, commit after commit (2,669 on main);
• regression: 19 of 19, snapshots unchanged;
• Enterprise plugins at 1b80fce: 978 passed.