Report churn in changed characters alongside line counts

[quantecon-services]

Closes #4. Phase 1 of the work plan in #8: line-level counting over-credits one-character fixes, so churn is now also emitted in changed characters at every level the line counts exist.

What changed

pairs.changed_chars(old, new) — a small helper returning (deleted, added) character counts from SequenceMatcher opcodes over the paired lines' raw text.

Per pair (pairs.jsonl): every row carries chars_changed — opcode counts for modified pairs, the full line length for addition/deletion rows.

Per commit (commits.jsonl): prose_chars_added / prose_chars_deleted, computed by pairing each hunk's lines and counting characters only where a side is prose. Emitted for every post-translation commit (machine tiers included), so machine-delivered characters are available downstream as the F1 denominator, mirroring how prose_adds/prose_dels already work. Hunks are now paired once per commit and reused by the pair-categorisation pass, which previously re-paired them.

Per document (documents.json): prose_char_churn_by_tier alongside prose_churn_by_tier.

docs/method.md (F1 row and the last-toucher limit) and the README limit now describe the reported counts and the counting rule; CHANGELOG entry added. run.json is unchanged.

Validation

• ruff check clean; unit tests 8 passed including new test_changed_chars (a one-character punctuation fix counts (1, 1), not the line).
• Regression test run locally against the pinned lecture-intro.zh-cn checkout at 0466562: passed — all compared study quantities (per-lecture translation dates, ai-initial/ai-sync shares, similarities, prose-line total, pair count) are unchanged; the new fields are purely additive.
• Sanity scan of the same checkout: a commit touching 7+7 prose lines reports 14 added / 25 deleted characters; punctuation-width pairs report chars_changed of 1–2 against 37–78-character lines.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MhveTMPqgH3AhmJui3h41m

---

Generated by Claude Code

[Copilot]

Pull request overview

This PR implements Phase 1 of the Stage 1 roadmap (issue #8) and closes issue #4 by reporting churn in changed characters alongside existing line counts, addressing the "last-toucher" over-crediting problem where a one-character fix is credited with the whole line. Character counts are emitted at every level the line counts already exist — per pair, per commit, and per document — so the F1 metric can be normalised per 1,000 characters as well as per 1,000 lines. The change is purely additive and leaves run.json and the regression-tested study quantities unchanged.

Changes:

• New pairs.changed_chars(old, new) helper returning (deleted, added) character counts from SequenceMatcher opcodes.
• scan.py now pairs hunks once per commit and reuses them to emit chars_changed per pair, prose_chars_added/prose_chars_deleted per commit (all post-translation tiers), and prose_char_churn_by_tier per document.
• Documentation (docs/method.md, README.md, CHANGELOG.md) updated to describe the reported character counts and counting rule; new unit test test_changed_chars.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/textstrata/pairs.py	Adds changed_chars helper computing deleted/added char counts from opcodes.
src/textstrata/scan.py	Pairs hunks once per commit; adds char-churn fields per pair, commit, and document; reuses paired in the categorisation pass.
tests/test_units.py	Imports and unit-tests changed_chars including a one-character punctuation fix.
docs/method.md	Updates F1 row and last-toucher limit to describe reported character counts and counting rule.
README.md	Rewords the last-toucher limit to state character churn is now reported.
CHANGELOG.md	Records the new character-level churn fields.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.