Skip to content

Fix name_usage index defects found reviewing #712's fixes - #714

Merged
derek73 merged 1 commit into
masterfrom
fix/name-usage-index-review-712
Oct 18, 2026
Merged

derek73 merged 1 commit into
masterfrom
fix/name-usage-index-review-712

Conversation

@derek73

@derek73 derek73 commented Oct 18, 2026 •

Copy link
Copy Markdown
Owner

Follow-up to #712, which was merged before this round so another session could proceed. A second review of #712's fix commits found these defects in tools/name_usage.py's CJK index. All are in the dev tool; none changes parsing.

Fixes

  • A complete index read as missing. A batch that overran under a new listing was cached as halves, but the old whole-batch file stayed beside them and was read first. A complete index then loaded as missing offline, and every online run re-sent the overrunning query. The halved marker is now read first, and the old file is removed when a batch is halved.
  • Batches were keyed by their first and last item. A re-listing that changed an item in between raised "unusable", or silently missed the new item. Each batch file now carries a digest of its whole membership and is fetched again when that changes.
  • --refresh-index failed silently with only Latin words. It deleted the index and, if the rebuild failed, exited 0 with nothing said. The failure is now printed and the exit status is 2.
  • HTTPErrors are closed on every path where the report discards one: the Wikidata row, the ends-with row, a pairing count and a source loader. Python 3.14 warns on one left open, and Peel the #671 credential words off a glued name, except 部長; name_usage answers ends-with for CJK #712 closed only the retry and overrun paths.
  • Extension H. _CJK_CHARS now reaches Extension H, and a redundant range is dropped. INDEX_VERSION is now 3, so an existing cache is rebuilt.

Tests

The new tests cover:

  • a cache built before versioning (no version file);
  • a batch whose first, last or middle item changed;
  • the stale whole batch beside its halves;
  • corrupt batch shapes, and a listing with no item column;
  • the pairing-count union with the larger chunk first, which a "last chunk wins" bug would pass;
  • closed HTTPErrors on each path;
  • the refresh failure.

Results:

  • Tool tests: 124 pass on 3.11, 3.14 and 3.15.
  • Full suite: passes; mypy and ruff are clean.

Index rebuild

The version-3 rebuild ran clean: it detected the old cache and rebuilt it, and one batch that timed out was split in half. The results match #712's: 215,820 entries; 殿 ends 13 name items; the only name item ending in any #707 word is 二世's mislabel 厄德二世 (Q18339792); and every family/given pair counts 0 people.

🤖 Generated with Claude Code

- A batch that overran under a new listing and was cached as halves
  left the old whole-batch file beside them, and that file was read
  first: a complete index loaded as missing offline and re-sent the
  overrunning query on every run. The halved marker is read first now,
  and the old file is removed when a batch is halved.
- Batches were keyed by their first and last item, so a re-listing
  that changed an item in between raised "unusable" or silently missed
  the new item. Each batch file now carries a digest of its whole
  membership and is fetched again when that changes.
- --refresh-index with only Latin words deleted the index and, if the
  rebuild failed, exited 0 with nothing said. The failure is printed and
  the exit status is 2.
- HTTPErrors are closed wherever the report lets go of one (the
  Wikidata row, the ends-with row, a pairing, a source loader), not only
  in the retry and overrun paths; 3.14 warns on one left open.
- _CJK_CHARS reaches Extension H; the separate compatibility-supplement
  range sat inside it. INDEX_VERSION 3, so the cache is rebuilt.

Tests: a pre-versioning cache (no version file), each end and the
middle of a batch changing, the stale whole batch beside its halves,
corrupt batch shapes ([] and short or non-string rows), a listing with
no item column, the pairing union with the larger chunk first, closed
HTTPErrors on each path, and the refresh failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73 derek73 added the bug label Oct 18, 2026
@derek73 derek73 self-assigned this Oct 18, 2026
@codecov

codecov Bot commented Oct 18, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.03%. Comparing base (901ab66) to head (5d87b22).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #714   +/-   ##
=======================================
  Coverage   99.03%   99.03%           
=======================================
  Files          46       46           
  Lines        4557     4557           
=======================================
  Hits         4513     4513           
  Misses         44       44           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@derek73 derek73 added this to the 2.5 milestone Oct 18, 2026
@derek73
derek73 merged commit d791bb6 into master Oct 18, 2026
12 checks passed
@derek73
derek73 deleted the fix/name-usage-index-review-712 branch October 18, 2026 05:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant