From dfd24feaae01530bd502575dac708152b20317b2 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 30 Aug 2026 00:53:58 -0700 Subject: [PATCH 01/12] fix(render): initials() honors FOLDED_TAG's order (#408) O3's fold and P6's attachment tag rather than move a token, and `_types._text_for` partitions on that tag for every role it renders -- so `parse("der, y van")` reads family 'van der'. `initials()` walked `tokens_for(role)` in written order and never asked, so a view and the field beside it gave two orders for one parse: initials 'y. d. v.'. Mirrors `_text_for`'s partition, per role rather than scoped to FAMILY. Both producers re-role to FAMILY, so the GIVEN and MIDDLE arms are unreachable today; they are there so the two views cannot diverge again, the same uniformity `_post_rules`' UNJOINED_TAG loop takes. Measured over the 1094-name deduped corpus, 6564 parses across three name_order values x middle_as_family off/on: 660 move. At the default order it is 1 without middle_as_family and 71 with it, and the change is strictly parity-restoring -- 54 of those 71 return to exactly what the released 1.4.0 wheel gives and none moves away from it. The facade's initials had been folding all along, through its own *_list views, so the core view was out of step with the field, the facade and v1 at once, with nothing in the suite touching it. `initials()` is not a compared surface, so the gate cannot see this: run at all three baselines on the pre-change tree and on this one, the output is identical to the byte. No ledger rule was added or needed. Co-Authored-By: Claude Opus 5 --- docs/design/decisions.md | 10 ++++++ docs/design/mechanisms.md | 4 +-- docs/design/rules.md | 13 +++++-- docs/release_log.rst | 2 ++ nameparser/_render.py | 25 ++++++++++++-- nameparser/_types.py | 7 ++-- tests/v2/cases.py | 8 +++-- tests/v2/test_render.py | 72 +++++++++++++++++++++++++++++++++++++-- 8 files changed, 126 insertions(+), 15 deletions(-) diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 8776d743..06ae8aba 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -722,6 +722,16 @@ Declined: - 2026-08-18 — the GROUPING half of #404: a particle run that joins nothing does not chain, so "Jong van der" would split into middle 'van' plus family 'der'. Measured and rejected, though NOT for the reason first recorded here. The first draft said the split makes the family "no longer all-particle so the base fix stops firing" — false, and `der` and `la` are both shipped particles, so a family of either IS all-particle and the rule fires on it ("Juan Smith der" gives base 'der'). What the split actually costs is the SCOPE of the base and a stray particle relocated: grouping can decline to merge but cannot keep the words apart, because roles re-assemble them and two adjacent same-role pieces are one part at the field level. "Juan Smith van der" becomes middle 'Smith van', family 'der' — a base of 'der' rather than 'van der', and a middle name nobody wrote. Keeping the run whole in one part is what gives the base its full extent. The split reading needs the leftover distribution to know these are separate units, which is mechanisms.md#UNIT-PARTITION's problem. +### R3 — initials honor the fold + +- 2026-08-30 #408 — DONE: `initials()` orders folded words first, as every other view of the same parse already did. O3's fold and P6's attachment both TAG rather than move a token (mechanisms.md#FOLDED_TAG — spans cannot reorder), and `_types._text_for` partitions on that tag for every role it renders, so the `family` field reads `van der` where the tokens sit `der ... van`. `initials()` walked `tokens_for(role)` in written order and never asked, so the two views disagreed about one parse: `parse("der, y van")` gave family `van der` and initials `y. d. v.`, and it gives `y. v. d.` now. This is mechanisms.md#RENDER-HONORS-THE-PARSE's second shape, filed as its worked instance, so the design is not re-argued here — only what the measurement added to it. + WHAT THE MEASUREMENT ADDED, and it reframes the issue from a consistency fix to a parity restoration. v1 had this feature as `CONSTANTS.middle_name_as_last` (`_config_shim` maps it to `Policy.middle_as_family`) and had NO general `name_order`, so only the two default-order policies have a v1 reference at all. Measured on the released 1.4.0 wheel over the 1094-name deduped corpus, from a script run outside the worktree that asserts `nameparser.__version__` first: at the default policy 1 name moves and agrees with 1.4.0 neither before nor after; under `middle_as_family` 71 move, of which **54 come into exact 1.4.0 agreement and NOT ONE moves away from it**. Worked pair: `Doe, Dr. John A.` gives `J. A. D.` at 1.4.0, gave `J. D. A.` here, and gives `J. A. D.` again; `Brundridge, Contessa A` likewise `C. A. B.` / `C. B. A.` / `C. A. B.`. The facade had been giving the folded answer all along — `HumanName.initials()` reads `first_list`/`middle_list`/`last_list`, and `_facade._list_for` prepends the carriers exactly as `_text_for` does — so the core view was simultaneously out of step with the field beside it, with the facade above it, and with 1.4.0 behind it. A view that stopped honoring a record had been reproducing a v1 bug v1 did not have. + THE 17 THAT AGREE WITH 1.4.0 NEITHER WAY, recorded because the obvious explanation is wrong and was written down before it was checked. It is not that their parse already differs: 14 of the 17 have SEVEN BYTE-IDENTICAL FIELDS at 1.4.0 and here, and differ only in v1's initials GRANULARITY — v1 initials one group per `*_list` element, so a run its parse joined into one element (`Velasquez y Garcia`, `abdul salam`) yields `V G.` where 2.x, one token per word, yields `V. G.`. Strip the periods and spaces and all 14 come into 1.4.0's letter ORDER after this change and none was in it before (`JVGQ` → `JQVG`). Only 3 — `der, y van`, `Smith, John V.` and `남궁민수 지훈` — parse differently from 1.4.0, and those are #379/#404 and #432 territory, not this change's. So the parity claim is stronger than "54 of 71 and the rest unexplained": 68 of the 71 come into 1.4.0's ordering, 54 of them exactly, and the residue is a separate divergence this change neither causes nor closes. + BLAST RADIUS by policy, over the same corpus × three `name_order` values × `middle_as_family` off/on (6564 parses, 660 move). Default order: 1 / 71. FAMILY_FIRST: 1 / 294. FAMILY_FIRST_GIVEN_LAST: 1 / 292. The `middle_as_family` half is where the reach is, which follows from the producers: O3 folds every middle word, while P6's attachment moves initials only where the attached run is READMITTED by the unjoined mark — a family that is all particles — so it reaches one corpus name in each order and the same one. Recompute by parsing the four `tools/differential/corpus*.jsonl` files deduped under each of the six policies and comparing `initials()` against a folded-first partition of `tokens_for(role)`. + THE VERIFICATION LIMIT, and it is decisions.md#R4's, restated because a plausible reading of "one name moves at the default policy" is that the ledger owes it a rule. It does not, and could not: `tools/differential/compare.py` compares the seven role fields plus `_ambiguities`, and `initials()` is not among them — the same blindness recorded for `capitalized()` under R4. Measured rather than reasoned: the gate was run at all three baselines on the pre-change tree (exported with `git archive HEAD` into a scratch directory, so nothing in the working tree moved) and on the changed one, and the two outputs are byte-identical apart from the tree path and the baseline-worker temp path — 1094 corpus names, 229 / 194 / 102 intentional diffs, `unexplained: 0`, and 67 / 58 / 51 headings with identical per-heading counts, at 1.4.0 / 2.0.0 / 2.1.0. NO LEDGER RULE WAS ADDED, and none was needed. What stands in its place is `rules.md#R3`'s example line and `tests/v2/test_render.py`, both mutation-checked. + WHY R3 GAINED A SENTENCE rather than leaning on R1. R1 already says folded family words "render before the rest of the family wherever they stood in the string" — but its subject is "Every FIELD", and `initials()` is not a field: R3 calls it "this view", and it takes a format spec and two separators rather than being read as an attribute. So R1 does not reach it by its own words, and #408 is precisely what leaving that to inference costs. R3's new clause states the order in R3's own voice, carries the O3 example line (`"Hassan, Mohamad Ahmad Ali"` under `middle_as_family` → `initials="M. A. A. H."`, reusing R1's own input so `corpus_rules.jsonl` is unchanged at 241 names), and R1 and R3 now name each other in `interacts:` — the field says which order, the view says it follows the field. P6's half gets no example line, for the same reason R1 carries none: the only input that witnesses it is `der, y van`, whose PARSE is itself contested, so pinning its initials normatively would pin a value another open question can move. It is pinned in the unit test instead, which is where today's-behavior pins belong. + THE PARTITION IS PER ROLE, not scoped to FAMILY, and this is a deliberate choice rather than a copied line. Both producers re-role to FAMILY, so FAMILY is the only role a parse can put the tag on today and the GIVEN and MIDDLE arms are unreachable. They are there because `_text_for` partitions for every role it renders and the two views must not diverge again — the same uniformity `_post_rules`' own UNJOINED_TAG loop takes for its three roles, "for uniformity with the rule, not because all three are observable". A producer that ever folded into another part would otherwise reopen #408 there with nothing to fail. `test_initials_folds_in_every_role_it_renders` pins it from a hand-built name, since no input string can; mutation-checked by scoping the partition to FAMILY, which fails that test and only that test. + ### R4 — case repair reads the unjoined mark - 2026-08-29 #407 — DECIDED: case repair consults the mark #404 introduced, and consults it for the WHOLE PART rather than for a particle standing alone. Until now three views read it and the fourth did not, which is how `parse("ANH DO")` came to report family_base 'DO' and initials 'A. D.' while `capitalized()` returned 'Anh do' — the same word treated as an ordinary name word by three readers and as a particle by the fourth. The scope question is the only real choice here, and standing-alone loses on the first Vietnamese name that tests it: `anh van do` has family `van do`, two particle-vocabulary words and neither of them alone, so a rule keyed on aloneness would capitalize `Anh Do` and leave `Anh van do`, making the same surname read differently for having a given-name word in front of it. rules.md#R2 states the test as "Every word", not "standing alone", and this is the case that shows why it is stated that way. One predicate, four views. diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 4e82b735..3678c7ad 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -15,7 +15,7 @@ Problem shape. A later stage needs to refer to "that word." Contract statement. ## FOLDED_TAG — reorder at render time, not parse time -Problem shape. A rule wants words to RENDER in a different order than they sit in the string. Contract statement. Tokens never move: a rule that needs different rendering order tags the token, and the rendering views consult the tag — family views order folded tokens first. How it works. Reordering the token tuple would break span math and reintroduce the #100 family. Parse state stays in string order; only the view reorders (rule R1, rule O3's render clause). Lives in. nameparser/_types.py (FOLDED_TAG, the family view), nameparser/_pipeline/_post_rules.py (its producer), and one deliberate CONSUMER-side strip: Parser.revise removes the tag from harvested tokens (a revised value must not inherit fold ordering) — losing that strip is this mechanism's measured hazard, a family rendering "García Gabriel Márquez". Reach for it when. A new rule needs "X renders before Y" and you are tempted to swap tokens. Don't swap. Tag. +Problem shape. A rule wants words to RENDER in a different order than they sit in the string. Contract statement. Tokens never move: a rule that needs different rendering order tags the token, and the rendering views consult the tag — EVERY view that renders the affected role, ordering folded tokens first, not the field views alone. How it works. Reordering the token tuple would break span math and reintroduce the #100 family. Parse state stays in string order; only the view reorders (rule R1, rule R3's order clause, rule O3's render clause). The consumer list is where this mechanism fails, and it has failed there twice in opposite directions: a consumer that never read the tag (`initials()` walked written order through 2.0, 2.1 and 2.2 — #408, filed as an instance of RENDER-HONORS-THE-PARSE and fixed 2026-08-30), and a consumer that read it when it should not have (the revise strip below). Adding a producer is cheap; adding a view is where the roster has to be walked. Lives in. nameparser/_types.py (FOLDED_TAG, `_text_for` — every role it renders, not FAMILY alone), nameparser/_render.py (`initials`, the same partition per role), nameparser/_facade.py (`_list_for`, the v1 `*_list` views, which prepend the carriers the same way), nameparser/_pipeline/_post_rules.py (its two producers, O3's fold and P6's attachment, both of which re-role to FAMILY — so FAMILY is the only role a parse can put the tag on today, and the views partitioning every role is uniformity with this contract rather than reachable behavior), and one deliberate CONSUMER-side strip: Parser.revise removes the tag from harvested tokens (a revised value must not inherit fold ordering) — losing that strip is this mechanism's measured hazard, a family rendering "García Gabriel Márquez". Reach for it when. A new rule needs "X renders before Y" and you are tempted to swap tokens. Don't swap. Tag. ## VOCAB-TAGS — the vocabulary layer speaks once @@ -59,7 +59,7 @@ Problem shape. Two stages need the same answer about the same input, and the one ## RENDER-HONORS-THE-PARSE — the parse decides it, the views honor it -Problem shape. A render view needs a fact the parse already settled — whether a word is the conjunction or an initial, whether a particle is acting as one, which word renders first. Contract statement. The parse decides it; the render views honor those decisions and never re-evaluate them. Two directions break that, and each has been found here as a defect: a view RE-DERIVES the answer from the text, keeping its own copy of a pipeline predicate — that one shipped through 2.0 and 2.1 — or a view honors the record and then OVERRIDES it, readmitting what the decision excluded, which is filed and open here. How it works. Re-deriving fails because the two copies stop being the same question long before anyone notices they are two: `_cap_word` re-ran the conjunction-versus-initial decision from the word's spelling against a hand-maintained copy of the pipeline's `_INITIAL` pattern while classify had already answered it and recorded it on the token, and `_classify.py` asks `is_initial()`, the shape test ANDed with a script-repertoire test since #320, where `_render.py` asked the bare pattern, and asked it per WORD of a token's text rather than per token, so `juan e-f smith` repaired to `Juan e-F Smith` (#458). Overriding fails more directly, and this entry's instance of it is OPEN rather than closed: `initials()` honors the `conjunction` tag through `_SKIP_TAGS` and then readmits the token whenever it also carries UNJOINED_TAG, the mark of an all-particle part, where rules.md#R3 excludes a conjunction "even then" — so under a caller's vocabulary that puts a word in both sets the view readmits what the rule excluded, and under `Lexicon.default().add(particles={'y'})` — the lexicon those two readings need, and they hold under no other — `parse("Anh y Van")` initialed `A. y. V.` and `parse("Juan de y")` `J. d. y.` on 2026-08-29. Read those two values scoped to that lexicon or they are simply false: under the DEFAULT vocabulary the same strings give `A. V.` and `J.`, and the second is a rules.md#R3 example line the doc runner asserts. #461 is where that stands, and it is worth reading before re-fixing it: the narrowing was written, measured and BACKED OUT in the same PR, because honoring R3 there cost `initials()` its agreement with `family_base`, which reads that same `de y` as the base — this entry's other shape, arrived at from the render side, and the reason the question moved from the code to R3's clause (decisions.md carries the argument). What to carry away is the DIRECTION and not its verdict here: a view that honors a record and then readmits what the record excluded is overriding a decision it never took, whichever way this one settles. Case repair reads that same token and does NOT readmit it — `capitalized(force=True)` on `Anh y Van` gives `Anh y Van` under that same lexicon, R4 carrying the carve-out in its own words but ON R3's authority — its text reads "being no name word in any part — the carve-out R3 states for initials", so a change to R3's clause reaches R4's TEXT. What does NOT follow, though an earlier wording of this sentence asserted it, is that the two stand or fall together in BEHAVIOR: they have already come apart, over the 25 corpus names carrying a conjunction in the GIVEN group — `parse("john and jane smith").capitalized()` keeps `and` lowercase, so R4's carve-out holds there, while `.initials()` gives `j. a. j. s.`, so R3's does not (decisions.md#R2 carries that population, and rules.md#R3 now says so in its own words). The dependency is textual, and only textual — so the two views disagree today about that token exactly as they did before #461 and the backout restores that disagreement knowingly; only the FORCED call witnesses the repair half, R5's gate refusing a mixed-case name before any of this is consulted. This is the CONSUMER-side rule over the producer-side entries — VOCAB-TAGS records what the vocabulary knew, MARK-DONT-STRIP what a stage decided about it, FOLDED_TAG what order to render in — and a view reads what they recorded, whichever kind it is. It sits where ONE-PREDICATE-PER-QUESTION's stated limit leaves off: where two live sites need one answer they share a predicate, but a render view always comes AFTER the decider, so the answer is recorded rather than shared — on the TOKEN, which is the views' equivalent of that entry's `ParseState.order`, no view being able to see a ParseState at all. Known limit, and the half most easily got wrong next: a token the parse never saw carries no decision to honor, so a view falls back to the vocabulary — the tell is `UNCLASSIFIED_TAG`, which `ParsedName.replace()` stamps when it splices raw text into a field and the facade's v1 pickle load stamps when it rebuilds a name from `*_list` strings. It is NOT untaggedness, since an ordinary parsed name word carries no tags either; and it is NOT `span is None`, which was tried and is wrong in the other direction — span-less means SYNTHETIC, and `Parser.revise()` builds span-less tokens from a full sub-parse whose tags it keeps on purpose, so the span reading overrode exactly the tags `revise()` exists to preserve (`revise(middle='e-f')` repaired to `e-F` where the parse gave `E-F`). A hand-built span-less token is unmarked and therefore classified, which is the same tag-driven default every other view applies. A view can only fall back if it is HANDED a vocabulary, and exactly one is: `capitalized(lexicon=...)`. `family_base` and `family_particles` are properties on ParsedName, whose fields are original/tokens/ambiguities and nothing else, so a spliced field empties the particles view and leaves the base the whole field, with `Parser.revise()` the crossing there too (docs/usage.rst says so where it documents the degradation). `initials()` is the near miss and the instructive one: it is a METHOD, so it looks like it could ask, but its signature is `(spec, delimiter, separator)` and carries no lexicon — a fallback there was written and dropped because it had to GUESS `Lexicon.default()`, and the guess erased a whole field under a caller's own vocabulary (decisions.md#R4). `capitalized()` guesses nothing: it reads the lexicon it was handed, and only defaults to `Lexicon.default()` when the caller passes none, which is the documented meaning of omitting the argument rather than a fallback. Within the one view that can fall back, the fallback is drawn per QUESTION (rules.md#R4's Accepted clause). Whether a word is the conjunction or an initial is a property of the word, which a vocabulary answers alone, so case repair asks it. Whether a part is wholly particles is a property of the whole PART, which the pipeline answers once and records as UNJOINED_TAG. What `_cap_word` cannot do is RE-DERIVE that answer where no word of the part carries a tag — it is handed the whole token's tags and gates the particle conjunct on UNJOINED_TAG, so it is not blind to the part, it simply has no evidence to reconstruct one from — so repair leaves that half to plain particle treatment and rules.md#R4's Accepted boundary records the consequence — a spliced field is not repaired as a parsed one is, with `Parser.revise()` as the supported crossing. A fallback is right only while it answers as the pipeline would, and THAT is held by hand rather than mechanically: test_regex_sync pins the two `_INITIAL` copies to each other and to config, while the repertoire half of the pipeline's predicate (#320) is deliberately not carried across, layering forbidding the import. What that divergence can reach is now nothing observable: it needs a caller-added conjunction written initial-SHAPED in a script that has no initials (`씨.`), and case repair is the fallback's only reader, so the two paths differ by `lower()` versus `capitalize()` over a caseless script — the same string either way. `initials()` used to be the reader that could witness it, and no longer falls back at all. A second limit, recorded rather than closed: `_cap_word`'s PARTICLE conjunct still keys on the lexicon handed to the view rather than on the `particle` tag, so a repair run with a lexicon other than the parse's re-decides a word the parse already read — a name parsed under the default vocabulary, where `parse('juan smith vega')` reads `vega` as the family, repairs to `Juan Smith vega` when `capitalized()` is handed `Lexicon.default().add(particles={'vega'})` instead — the divergence needs the two lexicons to differ, and repairing under the parse's own lexicon gives `Juan Smith Vega`. Making it read the tag moves a boundary rules.md#R4 states in prose, so it is a separate decision and not a cleanup (decisions.md#R4, "NOT DONE"). Lives in. nameparser/_render.py (`capitalized`/`_cap_word`, `_reads_as_conjunction`, and `initials`, which honors tags and never falls back) and nameparser/_types.py (`_text_for` and `UNCLASSIFIED_TAG`, with the `ParsedName.replace()` producer beside it) and nameparser/_facade.py (the v1 pickle load, the SECOND producer of that mark — it is named in this list because a change that follows the list into `_types.py` alone leaves it behind, which is the site test_a_restored_pickle_keeps_v1_conjunction_repair exists to protect), reading what nameparser/_pipeline/ recorded — the mark those views read is recomputed producer-side in `_remarked`, which is deliberately silent about text nobody classified and is right to be. Reach for it when. A view is about to consult a Lexicon, a regex or an exception list about a word the parse already saw — or a view and a field disagree about the same parse. #408 is filed as that second shape, and was open on 2026-08-29: `initials()` walks tokens in written order where the family field applies FOLDED_TAG's ordering, so `parse("der, y van")` gave family `van der` and initials `y. d. v.` that day, one view not yet honoring the fold. It belongs here rather than argued again from scratch. +Problem shape. A render view needs a fact the parse already settled — whether a word is the conjunction or an initial, whether a particle is acting as one, which word renders first. Contract statement. The parse decides it; the render views honor those decisions and never re-evaluate them. Two directions break that, and each has been found here as a defect: a view RE-DERIVES the answer from the text, keeping its own copy of a pipeline predicate — that one shipped through 2.0 and 2.1 — or a view honors the record and then OVERRIDES it, readmitting what the decision excluded, which is filed and open here. How it works. Re-deriving fails because the two copies stop being the same question long before anyone notices they are two: `_cap_word` re-ran the conjunction-versus-initial decision from the word's spelling against a hand-maintained copy of the pipeline's `_INITIAL` pattern while classify had already answered it and recorded it on the token, and `_classify.py` asks `is_initial()`, the shape test ANDed with a script-repertoire test since #320, where `_render.py` asked the bare pattern, and asked it per WORD of a token's text rather than per token, so `juan e-f smith` repaired to `Juan e-F Smith` (#458). Overriding fails more directly, and this entry's instance of it is OPEN rather than closed: `initials()` honors the `conjunction` tag through `_SKIP_TAGS` and then readmits the token whenever it also carries UNJOINED_TAG, the mark of an all-particle part, where rules.md#R3 excludes a conjunction "even then" — so under a caller's vocabulary that puts a word in both sets the view readmits what the rule excluded, and under `Lexicon.default().add(particles={'y'})` — the lexicon those two readings need, and they hold under no other — `parse("Anh y Van")` initialed `A. y. V.` and `parse("Juan de y")` `J. d. y.` on 2026-08-29. Read those two values scoped to that lexicon or they are simply false: under the DEFAULT vocabulary the same strings give `A. V.` and `J.`, and the second is a rules.md#R3 example line the doc runner asserts. #461 is where that stands, and it is worth reading before re-fixing it: the narrowing was written, measured and BACKED OUT in the same PR, because honoring R3 there cost `initials()` its agreement with `family_base`, which reads that same `de y` as the base — this entry's other shape, arrived at from the render side, and the reason the question moved from the code to R3's clause (decisions.md carries the argument). What to carry away is the DIRECTION and not its verdict here: a view that honors a record and then readmits what the record excluded is overriding a decision it never took, whichever way this one settles. Case repair reads that same token and does NOT readmit it — `capitalized(force=True)` on `Anh y Van` gives `Anh y Van` under that same lexicon, R4 carrying the carve-out in its own words but ON R3's authority — its text reads "being no name word in any part — the carve-out R3 states for initials", so a change to R3's clause reaches R4's TEXT. What does NOT follow, though an earlier wording of this sentence asserted it, is that the two stand or fall together in BEHAVIOR: they have already come apart, over the 25 corpus names carrying a conjunction in the GIVEN group — `parse("john and jane smith").capitalized()` keeps `and` lowercase, so R4's carve-out holds there, while `.initials()` gives `j. a. j. s.`, so R3's does not (decisions.md#R2 carries that population, and rules.md#R3 now says so in its own words). The dependency is textual, and only textual — so the two views disagree today about that token exactly as they did before #461 and the backout restores that disagreement knowingly; only the FORCED call witnesses the repair half, R5's gate refusing a mixed-case name before any of this is consulted. This is the CONSUMER-side rule over the producer-side entries — VOCAB-TAGS records what the vocabulary knew, MARK-DONT-STRIP what a stage decided about it, FOLDED_TAG what order to render in — and a view reads what they recorded, whichever kind it is. It sits where ONE-PREDICATE-PER-QUESTION's stated limit leaves off: where two live sites need one answer they share a predicate, but a render view always comes AFTER the decider, so the answer is recorded rather than shared — on the TOKEN, which is the views' equivalent of that entry's `ParseState.order`, no view being able to see a ParseState at all. Known limit, and the half most easily got wrong next: a token the parse never saw carries no decision to honor, so a view falls back to the vocabulary — the tell is `UNCLASSIFIED_TAG`, which `ParsedName.replace()` stamps when it splices raw text into a field and the facade's v1 pickle load stamps when it rebuilds a name from `*_list` strings. It is NOT untaggedness, since an ordinary parsed name word carries no tags either; and it is NOT `span is None`, which was tried and is wrong in the other direction — span-less means SYNTHETIC, and `Parser.revise()` builds span-less tokens from a full sub-parse whose tags it keeps on purpose, so the span reading overrode exactly the tags `revise()` exists to preserve (`revise(middle='e-f')` repaired to `e-F` where the parse gave `E-F`). A hand-built span-less token is unmarked and therefore classified, which is the same tag-driven default every other view applies. A view can only fall back if it is HANDED a vocabulary, and exactly one is: `capitalized(lexicon=...)`. `family_base` and `family_particles` are properties on ParsedName, whose fields are original/tokens/ambiguities and nothing else, so a spliced field empties the particles view and leaves the base the whole field, with `Parser.revise()` the crossing there too (docs/usage.rst says so where it documents the degradation). `initials()` is the near miss and the instructive one: it is a METHOD, so it looks like it could ask, but its signature is `(spec, delimiter, separator)` and carries no lexicon — a fallback there was written and dropped because it had to GUESS `Lexicon.default()`, and the guess erased a whole field under a caller's own vocabulary (decisions.md#R4). `capitalized()` guesses nothing: it reads the lexicon it was handed, and only defaults to `Lexicon.default()` when the caller passes none, which is the documented meaning of omitting the argument rather than a fallback. Within the one view that can fall back, the fallback is drawn per QUESTION (rules.md#R4's Accepted clause). Whether a word is the conjunction or an initial is a property of the word, which a vocabulary answers alone, so case repair asks it. Whether a part is wholly particles is a property of the whole PART, which the pipeline answers once and records as UNJOINED_TAG. What `_cap_word` cannot do is RE-DERIVE that answer where no word of the part carries a tag — it is handed the whole token's tags and gates the particle conjunct on UNJOINED_TAG, so it is not blind to the part, it simply has no evidence to reconstruct one from — so repair leaves that half to plain particle treatment and rules.md#R4's Accepted boundary records the consequence — a spliced field is not repaired as a parsed one is, with `Parser.revise()` as the supported crossing. A fallback is right only while it answers as the pipeline would, and THAT is held by hand rather than mechanically: test_regex_sync pins the two `_INITIAL` copies to each other and to config, while the repertoire half of the pipeline's predicate (#320) is deliberately not carried across, layering forbidding the import. What that divergence can reach is now nothing observable: it needs a caller-added conjunction written initial-SHAPED in a script that has no initials (`씨.`), and case repair is the fallback's only reader, so the two paths differ by `lower()` versus `capitalize()` over a caseless script — the same string either way. `initials()` used to be the reader that could witness it, and no longer falls back at all. A second limit, recorded rather than closed: `_cap_word`'s PARTICLE conjunct still keys on the lexicon handed to the view rather than on the `particle` tag, so a repair run with a lexicon other than the parse's re-decides a word the parse already read — a name parsed under the default vocabulary, where `parse('juan smith vega')` reads `vega` as the family, repairs to `Juan Smith vega` when `capitalized()` is handed `Lexicon.default().add(particles={'vega'})` instead — the divergence needs the two lexicons to differ, and repairing under the parse's own lexicon gives `Juan Smith Vega`. Making it read the tag moves a boundary rules.md#R4 states in prose, so it is a separate decision and not a cleanup (decisions.md#R4, "NOT DONE"). Lives in. nameparser/_render.py (`capitalized`/`_cap_word`, `_reads_as_conjunction`, and `initials`, which honors tags and never falls back) and nameparser/_types.py (`_text_for` and `UNCLASSIFIED_TAG`, with the `ParsedName.replace()` producer beside it) and nameparser/_facade.py (the v1 pickle load, the SECOND producer of that mark — it is named in this list because a change that follows the list into `_types.py` alone leaves it behind, which is the site test_a_restored_pickle_keeps_v1_conjunction_repair exists to protect), reading what nameparser/_pipeline/ recorded — the mark those views read is recomputed producer-side in `_remarked`, which is deliberately silent about text nobody classified and is right to be. Reach for it when. A view is about to consult a Lexicon, a regex or an exception list about a word the parse already saw — or a view and a field disagree about the same parse. #408 was that second shape, and is CLOSED (2026-08-30): `initials()` walked tokens in written order where the family field applies FOLDED_TAG's ordering, so `parse("der, y van")` gave family `van der` and initials `y. d. v.` on 2026-08-29 and gives `y. v. d.` now, the view reading the tag as `_text_for` does. Worth keeping as the worked instance of the shape rather than deleting with the fix, and worth two notes on how it read once measured. The disagreement was not a judgment call anyone had taken: the FACADE already ordered folded-first through its own `*_list` views, so the core view was out of step with the field, with v1, and with the facade at once, and nothing in 6125 tests touched it. And the fix was strictly parity-RESTORING, which is not what this entry's other instances have been -- over the 1094-name corpus at the default order, 71 names move under `middle_as_family`, of which 54 return to 1.4.0's answer and none leaves it (decisions.md#R3 carries the rest of the measurement). A view that stopped honoring a record had been quietly reproducing a v1 bug that v1 did not have. ## CLAUSE-CONTENT-OVERRULES-DELIMITER — content wins diff --git a/docs/design/rules.md b/docs/design/rules.md index 27277858..4fe6a9e3 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -1060,7 +1060,7 @@ R1. Rationale: a field is a way of reading the parse, not a stored "Hassan, Mohamad Ahmad Ali" → family="Hassan" · boundary "Smith, MD PhD" → suffix="MD PhD" "John Smith, MD, Bart" → suffix="MD, Bart" - history: decisions.md#C1 · implemented: nameparser/_types.py + history: decisions.md#C1 · interacts: O3, P6, R3 · implemented: nameparser/_types.py R2. Rationale: callers need the surname with and without its particles — sorting wants "Vega", display wants "de la Vega". @@ -1112,9 +1112,18 @@ R3. Rationale: initials abbreviate the person's name words; titles, one per joined name word under the carve-out. Until that is decided the given group's answer is pinned-but-undocumented rather than specified, and no line below asserts it. + Which words a group contributes is one question; the ORDER they + contribute in is a second, and its answer is the field's. Each + group initials in the order its field reads — written order, + except folded family words, which initial before the rest of the + family exactly as they render before it (R1). Stated here rather + than left to R1, whose subject is every FIELD and which this view + is not: initials abbreviate a field, so the two read one parse + and never disagree about it. "Dr. Juan Q. Xavier de la Vega III" → initials="J. Q. X. V." "Anh Do" → initials="A. D." "Nguyen, Van Le" → initials="V. L. N." + "Hassan, Mohamad Ahmad Ali" middle_as_family → initials="M. A. A. H." "Sean O'Connor" → initials="S. O." · boundary A family that is ALL particles therefore contributes its words rather than nothing: they are the base (R2), so they initial. @@ -1145,7 +1154,7 @@ R3. Rationale: initials abbreviate the person's name words; titles, the parser to it. #461 asks the neighbouring question about the all-particle base and does not own this one; decisions.md#R2 carries the population and the measurements. - history: decisions.md#R2 · interacts: P3, R2, R4 · implemented: nameparser/_render.py, nameparser/_facade.py + history: decisions.md#R2 · interacts: O3, P3, P6, R1, R2, R4 · implemented: nameparser/_render.py, nameparser/_facade.py R4. Rationale: case repair is a display concern, applied only on request and never destructively. diff --git a/docs/release_log.rst b/docs/release_log.rst index e85b8301..b244a55a 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -69,6 +69,8 @@ Release Log - Change case repair to read the parser's own ``conjunction`` tag instead of re-deciding, from the word's spelling, whether a word is a conjunction or an initial. The parse answers that question already -- ``"Scott E. Werner"`` reads ``E.`` as an initial rather than the Italian conjunction -- and the other views honor the answer; case repair asked again, with a shape test applied to each word of a token's text rather than to the token. Two spellings of one name disagreed because of it: ``"juan e-f smith"`` capitalized to ``Juan e-F Smith`` while ``"JUAN E-F SMITH"`` gave ``Juan E-F Smith``; both give ``Juan E-F Smith`` now, ``e-f`` being a middle name and no conjunction of the parse's reading. A conjunction written as a word of its own is untouched, and so is the one-letter carve-out where it applies -- ``"juan y garcia"`` still repairs to ``Juan y Garcia``, ``"JUAN Y GARCIA"`` still to ``Juan Y Garcia``. A field assigned after the parse is unaffected: its text was never classified, so there is no reading to honor and repair asks the vocabulary, applying v1's own predicate the way every earlier version applied it everywhere -- ``h.last = "velasquez y garcia"`` still repairs to ``Velasquez y Garcia`` and ``h.middle = "e."`` to ``E.``. That is the predicate over TODAY's vocabulary, which is narrower than parity with 1.4.0 and the difference is real: ``h.last = "хосе и мария сантос"`` gives ``Хосе И Мария Сантос`` on 1.4.0 and ``Хосе и Мария Сантос`` here, because the Cyrillic ``и`` is a 2.x conjunction and was not a 1.4.0 one. What decides which path a token takes is a mark the assignment leaves, not the absence of a span: a value revised through ``Parser.revise()`` is classified by a sub-parse and keeps its tags, so it repairs as the parse does. One reading does change for hand-built ``Token``\ s in the 2.0 API: an untagged token whose text is conjunction vocabulary is now an ordinary name word and capitalizes, where 2.1 lowercased it -- tags are what the views read, and a hand-built token that carries none is a token with nothing to declare. Case repair is not one of the seven role fields the differential harness compares, so no gate run can see this change either way and none of its counts move; measured directly instead, no name of the 1094-name differential corpus moves under ``capitalized()`` or ``capitalized(force=True)``, its uppercased and lowercased spellings included -- 6564 name/spelling/lexicon rows and 13128 calls (closes #458) + - Fix ``initials()`` reading a name in a different order than the fields of the same name. Two rules fold words into the family name and render them before the rest of it -- ``Policy(middle_as_family=True)``, which sends every middle word to the family, and the tussenvoegsel attachment after a family comma -- and both do it by marking the words rather than moving them, since a parsed word keeps the position it was written at. The ``family`` field reads that mark and ``initials()`` did not, so one parse gave two orders: ``HumanName("der, y van")`` gave last ``van der`` and initials ``y. d. v.``, and now gives ``y. v. d.``. This RESTORES v1: ``middle_name_as_last`` is v1's spelling of the same option, so most of what moves has a 1.4.0 answer to be measured against, and measured over the 1094-name differential corpus at the default name order, 71 names move under that option, 54 of them back to exactly what 1.4.0 returns and not one of them away from it -- ``"Doe, Dr. John A."`` gives ``J. A. D.`` again where 2.0 through 2.2 gave ``J. D. A.``, and ``"Brundridge, Contessa A"`` gives ``C. A. B.`` where they gave ``C. B. A.``. Of the 17 that match 1.4.0 neither before nor after, 14 now agree with it on the ORDER and differ only in how v1 grouped initials -- 1.4.0 gives one initial per element of its own ``last_list``, so a conjunction-joined surname yields ``V G.`` where the 2.x view, one initial per word, yields ``V. G.`` -- and the remaining three parse differently from 1.4.0 for reasons that predate this fix. Without the option, one corpus name moves, the ``"der, y van"`` above, whose family is nothing but particles and so contributes its own words. ``HumanName.initials()`` was already right and is unchanged: it reads ``first_list``/``middle_list``/``last_list``, which prepend the folded words as v1 did, so the 2.0 API's view was out of step with the field beside it, with the facade, and with 1.4.0 at once. ``initials()`` is not one of the seven role fields the differential harness compares, so no gate run can see this change: run at all three baselines before and after, the output is identical to the byte -- 1094 corpus names, 229 / 194 / 102 intentional diffs and ``unexplained: 0`` at 1.4.0 / 2.0.0 / 2.1.0, with every per-heading count unchanged. The ``rules.md#R3`` example line and ``tests/v2/test_render.py`` are what pin it (closes #408) + - Fix a tussenvoegsel attached to the family name after a comma deciding a genuinely uncertain reading and reporting nothing. ``"Van Johnson"`` reports a ``PARTICLE_OR_GIVEN`` ambiguity -- ``Van`` is a Dutch particle and a Vietnamese given name, and the parser has to pick one -- while ``"Nguyen, Thi Van"`` picked the same word the same way, silently, and lost the given name doing it. The attachment now reports the fork it decides, in the kind that names the reading it declined. A particle that could be an ordinary name reports ``PARTICLE_OR_GIVEN``: ``"Nguyen, Thi Van"``, ``"Berg, Jan van der"`` and ``"Vega, Juan de la"`` each gain one, the ``detail`` naming the ambiguous word. A particle the parser had already read as a post-nominal reports ``SUFFIX_OR_NAME`` instead, because the credential reading is what the attachment overrode: ``"Berg, Jan vd"`` gains one, ``vd`` being read as *van der* rather than as the Volunteer Decoration. Which kind you get follows the reading that was overridden rather than the word's vocabulary, so ``"Berg, Jan do"`` reports ``PARTICLE_OR_GIVEN`` even though ``do`` is a postnominal too -- it was already being read as a name word, so no credential reading was overridden. A particle where nothing was overridden reports nothing at all: ``"Jong, Piet de"`` and ``"Jong, Anke de"`` are unchanged, ``de`` being no name in any reading and no postnominal either, and so is ``"Berg, Jan de vd"``, whose run was read as name words whole. Worth knowing before you filter on this: ``"Beethoven, Ludwig van"`` -- the textbook Dutch listing, read exactly right -- now carries a report too. It is the same string shape over the same vocabulary as ``"Nguyen, Thi Van"``, and nothing in the input separates them, so a report on one is a report on both. This adds the report and nothing else: every field these names parse to is exactly what the tussenvoegsel fix below already gave them, and ``ambiguities`` is the only value that grows. Seven differential corpus names gain a kind against the 2.0.0 and 2.1.0 baselines, seven of the nine the tussenvoegsel fix below already moved -- five of ``fix(#379)``'s seven and both of ``fix(#380)``'s two, now carrying ``_ambiguities`` in their diff as well; the two left out are the pair named unchanged above, ``Jong, Piet de`` and ``Jong, Anke de`` -- and none against 1.4.0, which had no ambiguity reporting at all (closes #405) - Fix a tussenvoegsel after a family comma being parsed as a middle name. Dutch and Belgian alphabetized listings move the particle behind the given name -- ``"Beethoven, Ludwig van"`` is how ``"Ludwig van Beethoven"`` is filed -- and the trailing particle run was read as a middle name rather than as part of the surname: ``"Beethoven, Ludwig van"`` gave middle ``van``, last ``Beethoven``, and ``"Berg, Jan van der"`` gave middle ``van der``. The run now attaches to the family the comma has already named and renders before it, so those read family ``van Beethoven`` and ``van der Berg`` with the given name unchanged. The derived views move with the parse, so ``family_particles`` is ``van`` and ``family_base`` is ``Beethoven`` where they were empty and ``Beethoven`` before. `#130 `_ asked for the split and got it in 1.3.0 as ``last_base``/``last_prefixes``; 2.0 renamed them ``family_base``/``family_particles``. What was wrong until now was the values they reported for this listing. Both halves of the particle vocabulary attach -- never-given ``de`` and may-be-given ``van`` alike -- because after a comma the family is already named and the particle has no other role to take. Two guards bound it. A name whose only given word is the particle keeps it, so ``"Nguyen, Van"`` still reads given ``Van``: the attachment needs a given word to spare. And where the word is BOTH particle and suffix vocabulary the attachment outranks the post-nominal reading, so ``"Berg, Jan vd"`` reads family ``vd Berg`` where 1.4.0 and 2.1 alike gave suffix ``vd`` -- a trailing abbreviation after a family comma is the tussenvoegsel far more often than the decoration it collides with, and the same shape sweeps in ``mc``, which 2.1 also read as a suffix. ``do`` is in ``SUFFIX_ACRONYMS_AMBIGUOUS`` and 2.1 already read a trailing one as a name word, so it attaches by the plain rule rather than by the override (closes #379, closes #380). Names without the comma are untouched: ``"Ludwig van Beethoven"`` already read family ``van Beethoven`` and is byte-identical. One of the 751 differential corpus names moves, ``"Vega, Juan de la"``, at the 1.4.0, 2.0.0 and 2.1.0 baselines alike diff --git a/nameparser/_render.py b/nameparser/_render.py index d3634669..bf3786ca 100644 --- a/nameparser/_render.py +++ b/nameparser/_render.py @@ -15,8 +15,8 @@ import re from nameparser._lexicon import Lexicon, _normalize -from nameparser._types import (UNCLASSIFIED_TAG, UNJOINED_TAG, Ambiguity, - ParsedName, Role, Token) +from nameparser._types import (FOLDED_TAG, UNCLASSIFIED_TAG, UNJOINED_TAG, + Ambiguity, ParsedName, Role, Token) _SPACES = re.compile(r"\s+") _SPACE_BEFORE_COMMA = re.compile(r"\s+,") @@ -122,7 +122,10 @@ def render(name: ParsedName, spec: str) -> str: def initials(name: ParsedName, spec: str, delimiter: str, separator: str) -> str: """First letter of each contributing token per group, v1 semantics: delimiter follows each initial, separator sits between initials - within a group. Tokens tagged particle/conjunction contribute no + within a group. Each group is ordered the way its FIELD is + ordered -- written order, except folded words, which initial + before the rest of the group (#408). Tokens tagged + particle/conjunction contribute no initial in middle/family (given-name tokens always contribute), and the unjoined mark readmits the words of an all-particle part whichever of those tags they carry; tags come from the pipeline -- @@ -147,6 +150,22 @@ def initials(name: ParsedName, spec: str, delimiter: str, separator: str) -> str tokens = tuple(t for t in tokens if not (_SKIP_TAGS & t.tags) or UNJOINED_TAG in t.tags) + # mechanisms.md#FOLDED_TAG: "a rule that needs different + # rendering order tags the token, and the rendering views + # consult the tag" -- this is a rendering view, so it reads + # the tag the same way _types._text_for does, and for the same + # reason: the fold is an ORDER the parse recorded, not one the + # view is free to take again + # (mechanisms.md#RENDER-HONORS-THE-PARSE: "the parse decides + # it; the render views honor those decisions and never + # re-evaluate them"). Applied to every role this view renders, + # exactly as _text_for applies it -- the pipeline puts the tag + # on FAMILY tokens alone today, so GIVEN and MIDDLE are + # uniformity with the mechanism rather than reachable + # behavior; a producer that ever folds into another part would + # otherwise reopen #408 there. + tokens = (tuple(t for t in tokens if FOLDED_TAG in t.tags) + + tuple(t for t in tokens if FOLDED_TAG not in t.tags)) values[key] = separator.join( t.text[0] + delimiter for t in tokens) return _format_spec(spec, values, "initials", _INITIALS_KEYS) diff --git a/nameparser/_types.py b/nameparser/_types.py index 745ff06d..4836082e 100644 --- a/nameparser/_types.py +++ b/nameparser/_types.py @@ -110,9 +110,10 @@ def __add__(self, other: object) -> NoReturn: # type: ignore[override] #: The one sanctioned view-reorder marker (namespaced = unstable API). #: Tokens cannot reorder (span order is validated), so a role fold that #: must render BEFORE the role's original tokens tags them with this; -#: _text_for and the facade lists prepend carriers. Single-sourced here -#: so the emitter (_pipeline/_post_rules) and the consumers cannot -#: drift. +#: _text_for, _render.initials and the facade lists all prepend the +#: carriers. Single-sourced here so the emitter (_pipeline/_post_rules) +#: and the consumers cannot drift -- initials() was the consumer that +#: did drift, walking written order until #408. FOLDED_TAG = "vocab:folded-middle" #: Raw text spliced into a field after the parse, which no parse ever diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 407221d7..f36bd668 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -102,8 +102,12 @@ def __post_init__(self) -> None: {"given": "Mohamad", "family": "Ahmad Ali Hassan"}, policy=Policy(middle_as_family=True), notes="v1 PREPENDED middle_list to last_list; folded tokens " - "carry vocab:folded-middle and the family view orders " - "them first (spans cannot reorder)"), + "carry vocab:folded-middle and every view of them orders " + "them first (spans cannot reorder). This table has no " + "initials column, so the half #408 moved is pinned in " + "rules.md#R3 and tests/v2/test_render.py: initials were " + "'M. H. A. A.' here and are 'M. A. A. H.' now, which is " + "also 1.4.0's answer"), Case("ambiguous_surname_acronyms", "Jack MA", {"given": "Jack", "family": "MA"}, ambiguities=("suffix-or-name",), diff --git a/tests/v2/test_render.py b/tests/v2/test_render.py index 32f312e0..7f041393 100644 --- a/tests/v2/test_render.py +++ b/tests/v2/test_render.py @@ -1,10 +1,11 @@ import pytest -from nameparser import HumanName, Parser, Policy, parse +from nameparser import FAMILY_FIRST, HumanName, Parser, Policy, parse from nameparser._lexicon import Lexicon from nameparser._render import _collapse, render -from nameparser._types import (UNCLASSIFIED_TAG, UNJOINED_TAG, Ambiguity, - AmbiguityKind, ParsedName, Role, Span, Token) +from nameparser._types import (FOLDED_TAG, UNCLASSIFIED_TAG, UNJOINED_TAG, + Ambiguity, AmbiguityKind, ParsedName, Role, + Span, Token) def test_collapse_is_the_254_algorithm() -> None: @@ -202,6 +203,71 @@ def test_initials_readmits_a_conjunction_in_a_particle_part() -> None: assert parse("Juan de y").initials() == "J." +def test_initials_order_folded_words_first_like_the_family_field() -> None: + """#408: the view and the field must read one parse the same way. + + O3's fold and P6's attachment both tag rather than move (spans + cannot reorder), and the family field reads that tag. `initials()` + walked tokens in written order and did not, so the two views + disagreed about the same parse -- 'der, y van' gave family + 'van der' and initials 'y. d. v.' + (mechanisms.md#RENDER-HONORS-THE-PARSE names #408 as that shape). + + Each assertion pairs the field with the view deliberately: a + regression that reordered BOTH would still be caught by the + literal, and one that reordered neither by the pairing. + """ + # P6's attachment, the one name the DEFAULT policy moves + van_der = parse("der, y van") + assert van_der.family == "van der" + assert van_der.initials() == "y. v. d." + # and the facade, which reached this answer by its own route + # (its *_list views prepend the carriers) -- so the core view was + # the one out of step + assert HumanName("der, y van").initials() == "y. v. d." + + # O3's fold, which is where the reach is (71 of the corpus's 1094 + # names move under this policy at the default order, measured + # 2026-08-30) + maf = Parser(policy=Policy(middle_as_family=True)) + hassan = maf.parse("Hassan, Mohamad Ahmad Ali") + assert hassan.family == "Ahmad Ali Hassan" + assert hassan.initials() == "M. A. A. H." + doe = maf.parse("Doe, Dr. John A.") + assert doe.family == "A. Doe" + assert doe.initials() == "J. A. D." + + # under a non-default order too: the fold is not order-specific + ff = Parser(policy=Policy(name_order=FAMILY_FIRST, + middle_as_family=True)) + vega = ff.parse("Smith Juan Vega") + assert vega.family == "Vega Smith" + assert vega.initials() == "J. V. S." + + +def test_initials_folds_in_every_role_it_renders() -> None: + """The partition is applied per role, exactly as _text_for applies + it -- not scoped to FAMILY, where the pipeline's two producers put + the tag today. + + Both producers (O3's fold, P6's attachment) re-role to FAMILY, so + no input string reaches this and it is pinned from a hand-built + name instead. Uniformity with the mechanism is the point: a + producer that ever folded into another part would otherwise + reopen #408 there, silently, with nothing to fail. + """ + pn = _pn("Ann Bea Cyd", [ + Token("Ann", Span(0, 3), Role.GIVEN), + Token("Bea", Span(4, 7), Role.MIDDLE, frozenset({FOLDED_TAG})), + Token("Cyd", Span(8, 11), Role.GIVEN, frozenset({FOLDED_TAG})), + ]) + # the field already orders this way; the view now agrees + assert pn.given == "Cyd Ann" + assert pn.initials("{given}") == "C. A." + assert pn.middle == "Bea" + assert pn.initials("{middle}") == "B." + + def test_initials_custom_delimiter_and_separator() -> None: assert _bobdole().initials(delimiter="", separator="") == "B A D" From 3195a891c3d807ba2f7e81b785c3b93bd4d28e99 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 30 Aug 2026 01:13:03 -0700 Subject: [PATCH 02/12] fix(tests,docs): the five findings this branch's docs review raised MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All five were measured on the branch, and three were errors in my own measurements rather than in the prose reporting them. 1. The release-log bullet opened on `HumanName("der, y van")`, which never moved -- the facade folds through its own `*_list` views and moves for 0 of the 1094 corpus names. The value that moved is `parse(...)`, the 2.0-API view, which is also the bullet's own argument four sentences later. Corrected, and the facade's invariance is now stated as measured rather than asserted. 2. The 17 names matching 1.4.0 neither before nor after split 15/2, not 14/3: `der, y van` parses BYTE-IDENTICALLY to 1.4.0 under `middle_as_family`, and I had carried its field diff over from the other policy. It also disagrees for a different reason than the 14 -- 1.4.0 contributes nothing for an all-particle family where 2.x contributes its words, which is R3's own decided clause, not an unexplained residue. `남궁민수 지훈` is the hangul surname split (W1), not #379/#404 or #432. 3. `test_initials_folds_in_every_role_it_renders` asserted the MIDDLE arm and pinned only GIVEN: the hand-built name had ONE middle token, and a one-element partition is the identity. Two tokens per role now, and each arm is separately mutation-checked -- skipping the partition for GIVEN, for MIDDLE, or for FAMILY each fails. 4. R3's `history:` still pointed at `decisions.md#R2`, so the entry arguing its new order clause was unreachable from the rule. It points at `#R3` now, and the entry carries the back-pointer. 5. "One name moves at the default policy" carried no denominator. The population is one: exactly one corpus name has a contributing folded token standing behind a contributing unfolded one. Stated at both sites, with the 71-of-72 figure for the folded policy. Co-Authored-By: Claude Opus 5 --- docs/design/decisions.md | 6 ++++-- docs/design/rules.md | 2 +- docs/release_log.rst | 2 +- tests/v2/test_render.py | 16 +++++++++++----- 4 files changed, 17 insertions(+), 9 deletions(-) diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 06ae8aba..e2f6e4ec 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -724,10 +724,12 @@ Declined: ### R3 — initials honor the fold +R3's earlier history is under `decisions.md#R2`, which this entry does not repeat: the #461 backout and the unsettled given-group conjunction question are recorded there because they were decided alongside R2's all-particle clause. This entry is R3's own, and R3's `history:` points here. + - 2026-08-30 #408 — DONE: `initials()` orders folded words first, as every other view of the same parse already did. O3's fold and P6's attachment both TAG rather than move a token (mechanisms.md#FOLDED_TAG — spans cannot reorder), and `_types._text_for` partitions on that tag for every role it renders, so the `family` field reads `van der` where the tokens sit `der ... van`. `initials()` walked `tokens_for(role)` in written order and never asked, so the two views disagreed about one parse: `parse("der, y van")` gave family `van der` and initials `y. d. v.`, and it gives `y. v. d.` now. This is mechanisms.md#RENDER-HONORS-THE-PARSE's second shape, filed as its worked instance, so the design is not re-argued here — only what the measurement added to it. WHAT THE MEASUREMENT ADDED, and it reframes the issue from a consistency fix to a parity restoration. v1 had this feature as `CONSTANTS.middle_name_as_last` (`_config_shim` maps it to `Policy.middle_as_family`) and had NO general `name_order`, so only the two default-order policies have a v1 reference at all. Measured on the released 1.4.0 wheel over the 1094-name deduped corpus, from a script run outside the worktree that asserts `nameparser.__version__` first: at the default policy 1 name moves and agrees with 1.4.0 neither before nor after; under `middle_as_family` 71 move, of which **54 come into exact 1.4.0 agreement and NOT ONE moves away from it**. Worked pair: `Doe, Dr. John A.` gives `J. A. D.` at 1.4.0, gave `J. D. A.` here, and gives `J. A. D.` again; `Brundridge, Contessa A` likewise `C. A. B.` / `C. B. A.` / `C. A. B.`. The facade had been giving the folded answer all along — `HumanName.initials()` reads `first_list`/`middle_list`/`last_list`, and `_facade._list_for` prepends the carriers exactly as `_text_for` does — so the core view was simultaneously out of step with the field beside it, with the facade above it, and with 1.4.0 behind it. A view that stopped honoring a record had been reproducing a v1 bug v1 did not have. - THE 17 THAT AGREE WITH 1.4.0 NEITHER WAY, recorded because the obvious explanation is wrong and was written down before it was checked. It is not that their parse already differs: 14 of the 17 have SEVEN BYTE-IDENTICAL FIELDS at 1.4.0 and here, and differ only in v1's initials GRANULARITY — v1 initials one group per `*_list` element, so a run its parse joined into one element (`Velasquez y Garcia`, `abdul salam`) yields `V G.` where 2.x, one token per word, yields `V. G.`. Strip the periods and spaces and all 14 come into 1.4.0's letter ORDER after this change and none was in it before (`JVGQ` → `JQVG`). Only 3 — `der, y van`, `Smith, John V.` and `남궁민수 지훈` — parse differently from 1.4.0, and those are #379/#404 and #432 territory, not this change's. So the parity claim is stronger than "54 of 71 and the rest unexplained": 68 of the 71 come into 1.4.0's ordering, 54 of them exactly, and the residue is a separate divergence this change neither causes nor closes. - BLAST RADIUS by policy, over the same corpus × three `name_order` values × `middle_as_family` off/on (6564 parses, 660 move). Default order: 1 / 71. FAMILY_FIRST: 1 / 294. FAMILY_FIRST_GIVEN_LAST: 1 / 292. The `middle_as_family` half is where the reach is, which follows from the producers: O3 folds every middle word, while P6's attachment moves initials only where the attached run is READMITTED by the unjoined mark — a family that is all particles — so it reaches one corpus name in each order and the same one. Recompute by parsing the four `tools/differential/corpus*.jsonl` files deduped under each of the six policies and comparing `initials()` against a folded-first partition of `tokens_for(role)`. + THE 17 THAT AGREE WITH 1.4.0 NEITHER WAY, recorded because the obvious explanation is wrong — and the FIRST REPLACEMENT for it was wrong the same way, corrected by this branch's docs review before merge. It is not that their parse already differs: only 2 of the 17 parse differently from 1.4.0 under this policy, `Smith, John V.` (#432's dotted numeral) and `남궁민수 지훈` (the unspaced-hangul surname split and the script order, rules.md#W1). The other 15 have SEVEN BYTE-IDENTICAL FIELDS at 1.4.0 and here — `der, y van` INCLUDED, which an earlier draft of this bullet filed with the two above by carrying over a measurement taken at the OTHER policy, where its parse does differ. Of those 15, 14 differ from 1.4.0 only in v1's initials GRANULARITY — v1 initials one group per `*_list` element, so a run its parse joined into one element (`Velasquez y Garcia`, `abdul salam`) yields `V G.` where 2.x, one token per word, yields `V. G.` — and strip the periods and spaces and all 14 come into 1.4.0's letter ORDER after this change, none having been in it before (`JVGQ` → `JQVG`). The 15th is `der, y van`, and it is worth naming rather than lumping: its fields match 1.4.0 exactly and its initials do not, because 1.4.0 contributes NOTHING for a family that is all particles (`van der` → `y.`) where 2.x contributes its words. That is R3's own adjudicated clause and decisions.md#R2's territory — R3 carries `"Juan van der" → initials="J. v. d."` against 1.4.0's `J.` — so this one name is a divergence this document has already decided in favor of, not an unexplained residue. So the parity claim is stronger than "54 of 71 and the rest unexplained": 68 of the 71 come into 1.4.0's ordering, 54 of them exactly, one more is R2's decided divergence, and 2 have a parse difference that predates this change. + BLAST RADIUS by policy, over the same corpus × three `name_order` values × `middle_as_family` off/on (6564 parses, 660 move). Default order: 1 / 71. FAMILY_FIRST: 1 / 294. FAMILY_FIRST_GIVEN_LAST: 1 / 292. STATE THE POPULATION BESIDE THE COUNT, per this file's own rule: the shape an order change can be observed in is a contributing folded token standing AFTER a contributing unfolded one of the same role, and at the default order the corpus holds exactly 1 such name without `middle_as_family` and 72 with it. So the movers are 1 of 1 and 71 of 72 — a near-total hit rate over a corpus that carries one instance of P6's half, not a blast radius near zero. (The one non-mover is `Chang, Andy C I`, where the folded `C` and the family `Chang` share a letter and the reorder is invisible in initials.) Nine corpus names carry the tag at all at the default order; the other eight have a single contributing family token, which makes the partition the identity. The `middle_as_family` half is where the reach is, which follows from the producers: O3 folds every middle word, while P6's attachment moves initials only where the attached run is READMITTED by the unjoined mark — a family that is all particles — so it reaches one corpus name in each order and the same one. Recompute by parsing the four `tools/differential/corpus*.jsonl` files deduped under each of the six policies and comparing `initials()` against a folded-first partition of `tokens_for(role)`. THE VERIFICATION LIMIT, and it is decisions.md#R4's, restated because a plausible reading of "one name moves at the default policy" is that the ledger owes it a rule. It does not, and could not: `tools/differential/compare.py` compares the seven role fields plus `_ambiguities`, and `initials()` is not among them — the same blindness recorded for `capitalized()` under R4. Measured rather than reasoned: the gate was run at all three baselines on the pre-change tree (exported with `git archive HEAD` into a scratch directory, so nothing in the working tree moved) and on the changed one, and the two outputs are byte-identical apart from the tree path and the baseline-worker temp path — 1094 corpus names, 229 / 194 / 102 intentional diffs, `unexplained: 0`, and 67 / 58 / 51 headings with identical per-heading counts, at 1.4.0 / 2.0.0 / 2.1.0. NO LEDGER RULE WAS ADDED, and none was needed. What stands in its place is `rules.md#R3`'s example line and `tests/v2/test_render.py`, both mutation-checked. WHY R3 GAINED A SENTENCE rather than leaning on R1. R1 already says folded family words "render before the rest of the family wherever they stood in the string" — but its subject is "Every FIELD", and `initials()` is not a field: R3 calls it "this view", and it takes a format spec and two separators rather than being read as an attribute. So R1 does not reach it by its own words, and #408 is precisely what leaving that to inference costs. R3's new clause states the order in R3's own voice, carries the O3 example line (`"Hassan, Mohamad Ahmad Ali"` under `middle_as_family` → `initials="M. A. A. H."`, reusing R1's own input so `corpus_rules.jsonl` is unchanged at 241 names), and R1 and R3 now name each other in `interacts:` — the field says which order, the view says it follows the field. P6's half gets no example line, for the same reason R1 carries none: the only input that witnesses it is `der, y van`, whose PARSE is itself contested, so pinning its initials normatively would pin a value another open question can move. It is pinned in the unit test instead, which is where today's-behavior pins belong. THE PARTITION IS PER ROLE, not scoped to FAMILY, and this is a deliberate choice rather than a copied line. Both producers re-role to FAMILY, so FAMILY is the only role a parse can put the tag on today and the GIVEN and MIDDLE arms are unreachable. They are there because `_text_for` partitions for every role it renders and the two views must not diverge again — the same uniformity `_post_rules`' own UNJOINED_TAG loop takes for its three roles, "for uniformity with the rule, not because all three are observable". A producer that ever folded into another part would otherwise reopen #408 there with nothing to fail. `test_initials_folds_in_every_role_it_renders` pins it from a hand-built name, since no input string can; mutation-checked by scoping the partition to FAMILY, which fails that test and only that test. diff --git a/docs/design/rules.md b/docs/design/rules.md index 4fe6a9e3..50cb7abe 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -1154,7 +1154,7 @@ R3. Rationale: initials abbreviate the person's name words; titles, the parser to it. #461 asks the neighbouring question about the all-particle base and does not own this one; decisions.md#R2 carries the population and the measurements. - history: decisions.md#R2 · interacts: O3, P3, P6, R1, R2, R4 · implemented: nameparser/_render.py, nameparser/_facade.py + history: decisions.md#R3 · interacts: O3, P3, P6, R1, R2, R4 · implemented: nameparser/_render.py, nameparser/_facade.py R4. Rationale: case repair is a display concern, applied only on request and never destructively. diff --git a/docs/release_log.rst b/docs/release_log.rst index b244a55a..c000100d 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -69,7 +69,7 @@ Release Log - Change case repair to read the parser's own ``conjunction`` tag instead of re-deciding, from the word's spelling, whether a word is a conjunction or an initial. The parse answers that question already -- ``"Scott E. Werner"`` reads ``E.`` as an initial rather than the Italian conjunction -- and the other views honor the answer; case repair asked again, with a shape test applied to each word of a token's text rather than to the token. Two spellings of one name disagreed because of it: ``"juan e-f smith"`` capitalized to ``Juan e-F Smith`` while ``"JUAN E-F SMITH"`` gave ``Juan E-F Smith``; both give ``Juan E-F Smith`` now, ``e-f`` being a middle name and no conjunction of the parse's reading. A conjunction written as a word of its own is untouched, and so is the one-letter carve-out where it applies -- ``"juan y garcia"`` still repairs to ``Juan y Garcia``, ``"JUAN Y GARCIA"`` still to ``Juan Y Garcia``. A field assigned after the parse is unaffected: its text was never classified, so there is no reading to honor and repair asks the vocabulary, applying v1's own predicate the way every earlier version applied it everywhere -- ``h.last = "velasquez y garcia"`` still repairs to ``Velasquez y Garcia`` and ``h.middle = "e."`` to ``E.``. That is the predicate over TODAY's vocabulary, which is narrower than parity with 1.4.0 and the difference is real: ``h.last = "хосе и мария сантос"`` gives ``Хосе И Мария Сантос`` on 1.4.0 and ``Хосе и Мария Сантос`` here, because the Cyrillic ``и`` is a 2.x conjunction and was not a 1.4.0 one. What decides which path a token takes is a mark the assignment leaves, not the absence of a span: a value revised through ``Parser.revise()`` is classified by a sub-parse and keeps its tags, so it repairs as the parse does. One reading does change for hand-built ``Token``\ s in the 2.0 API: an untagged token whose text is conjunction vocabulary is now an ordinary name word and capitalizes, where 2.1 lowercased it -- tags are what the views read, and a hand-built token that carries none is a token with nothing to declare. Case repair is not one of the seven role fields the differential harness compares, so no gate run can see this change either way and none of its counts move; measured directly instead, no name of the 1094-name differential corpus moves under ``capitalized()`` or ``capitalized(force=True)``, its uppercased and lowercased spellings included -- 6564 name/spelling/lexicon rows and 13128 calls (closes #458) - - Fix ``initials()`` reading a name in a different order than the fields of the same name. Two rules fold words into the family name and render them before the rest of it -- ``Policy(middle_as_family=True)``, which sends every middle word to the family, and the tussenvoegsel attachment after a family comma -- and both do it by marking the words rather than moving them, since a parsed word keeps the position it was written at. The ``family`` field reads that mark and ``initials()`` did not, so one parse gave two orders: ``HumanName("der, y van")`` gave last ``van der`` and initials ``y. d. v.``, and now gives ``y. v. d.``. This RESTORES v1: ``middle_name_as_last`` is v1's spelling of the same option, so most of what moves has a 1.4.0 answer to be measured against, and measured over the 1094-name differential corpus at the default name order, 71 names move under that option, 54 of them back to exactly what 1.4.0 returns and not one of them away from it -- ``"Doe, Dr. John A."`` gives ``J. A. D.`` again where 2.0 through 2.2 gave ``J. D. A.``, and ``"Brundridge, Contessa A"`` gives ``C. A. B.`` where they gave ``C. B. A.``. Of the 17 that match 1.4.0 neither before nor after, 14 now agree with it on the ORDER and differ only in how v1 grouped initials -- 1.4.0 gives one initial per element of its own ``last_list``, so a conjunction-joined surname yields ``V G.`` where the 2.x view, one initial per word, yields ``V. G.`` -- and the remaining three parse differently from 1.4.0 for reasons that predate this fix. Without the option, one corpus name moves, the ``"der, y van"`` above, whose family is nothing but particles and so contributes its own words. ``HumanName.initials()`` was already right and is unchanged: it reads ``first_list``/``middle_list``/``last_list``, which prepend the folded words as v1 did, so the 2.0 API's view was out of step with the field beside it, with the facade, and with 1.4.0 at once. ``initials()`` is not one of the seven role fields the differential harness compares, so no gate run can see this change: run at all three baselines before and after, the output is identical to the byte -- 1094 corpus names, 229 / 194 / 102 intentional diffs and ``unexplained: 0`` at 1.4.0 / 2.0.0 / 2.1.0, with every per-heading count unchanged. The ``rules.md#R3`` example line and ``tests/v2/test_render.py`` are what pin it (closes #408) + - Fix ``initials()`` reading a name in a different order than the fields of the same name. Two rules fold words into the family name and render them before the rest of it -- ``Policy(middle_as_family=True)``, which sends every middle word to the family, and the tussenvoegsel attachment after a family comma -- and both do it by marking the words rather than moving them, since a parsed word keeps the position it was written at. The ``family`` field reads that mark and ``initials()`` did not, so one parse gave two orders: ``parse("der, y van")`` gave family ``van der`` and initials ``y. d. v.``, and now gives ``y. v. d.``. This RESTORES v1: ``middle_name_as_last`` is v1's spelling of the same option, so most of what moves has a 1.4.0 answer to be measured against, and measured over the 1094-name differential corpus at the default name order, 71 names move under that option, 54 of them back to exactly what 1.4.0 returns and not one of them away from it -- ``"Doe, Dr. John A."`` gives ``J. A. D.`` again where 2.0 through 2.2 gave ``J. D. A.``, and ``"Brundridge, Contessa A"`` gives ``C. A. B.`` where they gave ``C. B. A.``. Of the 17 that match 1.4.0 neither before nor after, 14 now agree with it on the ORDER and differ only in how v1 grouped initials -- 1.4.0 gives one initial per element of its own ``last_list``, so a conjunction-joined surname yields ``V G.`` where the 2.x view, one initial per word, yields ``V. G.``; one more is ``"der, y van"``, whose family is nothing but particles, where 1.4.0 contributes no initial at all and 2.x contributes its words, a difference this release does not touch; and the remaining two parse differently from 1.4.0 for reasons that predate this fix. Without the option, one corpus name moves, the ``"der, y van"`` above -- and it is one of one, since it is the only name in the corpus whose family holds two contributing words with a folded one behind the other, the shape an order change can be seen in at all. ``HumanName.initials()`` was already right and is unchanged, measured: no name of the 1094-name corpus moves through the facade, with the option or without it. It reads ``first_list``/``middle_list``/``last_list``, which prepend the folded words as v1 did, so it is the 2.0 API's ``ParsedName.initials()`` that was out of step -- with the field beside it, with the facade, and with 1.4.0 at once. ``initials()`` is not one of the seven role fields the differential harness compares, so no gate run can see this change: run at all three baselines before and after, the output is identical to the byte -- 1094 corpus names, 229 / 194 / 102 intentional diffs and ``unexplained: 0`` at 1.4.0 / 2.0.0 / 2.1.0, with every per-heading count unchanged. The ``rules.md#R3`` example line and ``tests/v2/test_render.py`` are what pin it (closes #408) - Fix a tussenvoegsel attached to the family name after a comma deciding a genuinely uncertain reading and reporting nothing. ``"Van Johnson"`` reports a ``PARTICLE_OR_GIVEN`` ambiguity -- ``Van`` is a Dutch particle and a Vietnamese given name, and the parser has to pick one -- while ``"Nguyen, Thi Van"`` picked the same word the same way, silently, and lost the given name doing it. The attachment now reports the fork it decides, in the kind that names the reading it declined. A particle that could be an ordinary name reports ``PARTICLE_OR_GIVEN``: ``"Nguyen, Thi Van"``, ``"Berg, Jan van der"`` and ``"Vega, Juan de la"`` each gain one, the ``detail`` naming the ambiguous word. A particle the parser had already read as a post-nominal reports ``SUFFIX_OR_NAME`` instead, because the credential reading is what the attachment overrode: ``"Berg, Jan vd"`` gains one, ``vd`` being read as *van der* rather than as the Volunteer Decoration. Which kind you get follows the reading that was overridden rather than the word's vocabulary, so ``"Berg, Jan do"`` reports ``PARTICLE_OR_GIVEN`` even though ``do`` is a postnominal too -- it was already being read as a name word, so no credential reading was overridden. A particle where nothing was overridden reports nothing at all: ``"Jong, Piet de"`` and ``"Jong, Anke de"`` are unchanged, ``de`` being no name in any reading and no postnominal either, and so is ``"Berg, Jan de vd"``, whose run was read as name words whole. Worth knowing before you filter on this: ``"Beethoven, Ludwig van"`` -- the textbook Dutch listing, read exactly right -- now carries a report too. It is the same string shape over the same vocabulary as ``"Nguyen, Thi Van"``, and nothing in the input separates them, so a report on one is a report on both. This adds the report and nothing else: every field these names parse to is exactly what the tussenvoegsel fix below already gave them, and ``ambiguities`` is the only value that grows. Seven differential corpus names gain a kind against the 2.0.0 and 2.1.0 baselines, seven of the nine the tussenvoegsel fix below already moved -- five of ``fix(#379)``'s seven and both of ``fix(#380)``'s two, now carrying ``_ambiguities`` in their diff as well; the two left out are the pair named unchanged above, ``Jong, Piet de`` and ``Jong, Anke de`` -- and none against 1.4.0, which had no ambiguity reporting at all (closes #405) diff --git a/tests/v2/test_render.py b/tests/v2/test_render.py index 7f041393..69a5981c 100644 --- a/tests/v2/test_render.py +++ b/tests/v2/test_render.py @@ -256,16 +256,22 @@ def test_initials_folds_in_every_role_it_renders() -> None: producer that ever folded into another part would otherwise reopen #408 there, silently, with nothing to fail. """ - pn = _pn("Ann Bea Cyd", [ + pn = _pn("Ann Bea Cyd Dee", [ Token("Ann", Span(0, 3), Role.GIVEN), - Token("Bea", Span(4, 7), Role.MIDDLE, frozenset({FOLDED_TAG})), + Token("Bea", Span(4, 7), Role.MIDDLE), Token("Cyd", Span(8, 11), Role.GIVEN, frozenset({FOLDED_TAG})), + Token("Dee", Span(12, 15), Role.MIDDLE, frozenset({FOLDED_TAG})), ]) - # the field already orders this way; the view now agrees + # the fields already order this way; the view now agrees. Each + # role carries TWO tokens on purpose: a one-token group makes the + # partition the identity, so a single folded middle would have + # left this arm asserted and unpinned (the shape the mutation + # below caught -- scoping the partition to GIVEN and FAMILY + # passed the whole suite against an earlier draft of this test). assert pn.given == "Cyd Ann" assert pn.initials("{given}") == "C. A." - assert pn.middle == "Bea" - assert pn.initials("{middle}") == "B." + assert pn.middle == "Dee Bea" + assert pn.initials("{middle}") == "D. B." def test_initials_custom_delimiter_and_separator() -> None: From 9e429ca4569433be346bca99ba89fcf4b4d5c4e1 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 30 Aug 2026 01:14:03 -0700 Subject: [PATCH 03/12] docs(decisions): record all four mutations, and why the first was not enough The entry named one mutation; finding 3 showed it left the MIDDLE arm unpinned. Records the four that run now, and the general shape -- a test pinning a partition needs two elements in every group it claims, or the claim rides on an identity. Co-Authored-By: Claude Opus 5 --- docs/design/decisions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/decisions.md b/docs/design/decisions.md index e2f6e4ec..04b55d4a 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -732,7 +732,7 @@ R3's earlier history is under `decisions.md#R2`, which this entry does not repea BLAST RADIUS by policy, over the same corpus × three `name_order` values × `middle_as_family` off/on (6564 parses, 660 move). Default order: 1 / 71. FAMILY_FIRST: 1 / 294. FAMILY_FIRST_GIVEN_LAST: 1 / 292. STATE THE POPULATION BESIDE THE COUNT, per this file's own rule: the shape an order change can be observed in is a contributing folded token standing AFTER a contributing unfolded one of the same role, and at the default order the corpus holds exactly 1 such name without `middle_as_family` and 72 with it. So the movers are 1 of 1 and 71 of 72 — a near-total hit rate over a corpus that carries one instance of P6's half, not a blast radius near zero. (The one non-mover is `Chang, Andy C I`, where the folded `C` and the family `Chang` share a letter and the reorder is invisible in initials.) Nine corpus names carry the tag at all at the default order; the other eight have a single contributing family token, which makes the partition the identity. The `middle_as_family` half is where the reach is, which follows from the producers: O3 folds every middle word, while P6's attachment moves initials only where the attached run is READMITTED by the unjoined mark — a family that is all particles — so it reaches one corpus name in each order and the same one. Recompute by parsing the four `tools/differential/corpus*.jsonl` files deduped under each of the six policies and comparing `initials()` against a folded-first partition of `tokens_for(role)`. THE VERIFICATION LIMIT, and it is decisions.md#R4's, restated because a plausible reading of "one name moves at the default policy" is that the ledger owes it a rule. It does not, and could not: `tools/differential/compare.py` compares the seven role fields plus `_ambiguities`, and `initials()` is not among them — the same blindness recorded for `capitalized()` under R4. Measured rather than reasoned: the gate was run at all three baselines on the pre-change tree (exported with `git archive HEAD` into a scratch directory, so nothing in the working tree moved) and on the changed one, and the two outputs are byte-identical apart from the tree path and the baseline-worker temp path — 1094 corpus names, 229 / 194 / 102 intentional diffs, `unexplained: 0`, and 67 / 58 / 51 headings with identical per-heading counts, at 1.4.0 / 2.0.0 / 2.1.0. NO LEDGER RULE WAS ADDED, and none was needed. What stands in its place is `rules.md#R3`'s example line and `tests/v2/test_render.py`, both mutation-checked. WHY R3 GAINED A SENTENCE rather than leaning on R1. R1 already says folded family words "render before the rest of the family wherever they stood in the string" — but its subject is "Every FIELD", and `initials()` is not a field: R3 calls it "this view", and it takes a format spec and two separators rather than being read as an attribute. So R1 does not reach it by its own words, and #408 is precisely what leaving that to inference costs. R3's new clause states the order in R3's own voice, carries the O3 example line (`"Hassan, Mohamad Ahmad Ali"` under `middle_as_family` → `initials="M. A. A. H."`, reusing R1's own input so `corpus_rules.jsonl` is unchanged at 241 names), and R1 and R3 now name each other in `interacts:` — the field says which order, the view says it follows the field. P6's half gets no example line, for the same reason R1 carries none: the only input that witnesses it is `der, y van`, whose PARSE is itself contested, so pinning its initials normatively would pin a value another open question can move. It is pinned in the unit test instead, which is where today's-behavior pins belong. - THE PARTITION IS PER ROLE, not scoped to FAMILY, and this is a deliberate choice rather than a copied line. Both producers re-role to FAMILY, so FAMILY is the only role a parse can put the tag on today and the GIVEN and MIDDLE arms are unreachable. They are there because `_text_for` partitions for every role it renders and the two views must not diverge again — the same uniformity `_post_rules`' own UNJOINED_TAG loop takes for its three roles, "for uniformity with the rule, not because all three are observable". A producer that ever folded into another part would otherwise reopen #408 there with nothing to fail. `test_initials_folds_in_every_role_it_renders` pins it from a hand-built name, since no input string can; mutation-checked by scoping the partition to FAMILY, which fails that test and only that test. + THE PARTITION IS PER ROLE, not scoped to FAMILY, and this is a deliberate choice rather than a copied line. Both producers re-role to FAMILY, so FAMILY is the only role a parse can put the tag on today and the GIVEN and MIDDLE arms are unreachable. They are there because `_text_for` partitions for every role it renders and the two views must not diverge again — the same uniformity `_post_rules`' own UNJOINED_TAG loop takes for its three roles, "for uniformity with the rule, not because all three are observable". A producer that ever folded into another part would otherwise reopen #408 there with nothing to fail. `test_initials_folds_in_every_role_it_renders` pins it from a hand-built name, since no input string can. Mutation-checked FOUR ways, and the fourth is the one that matters: scoping the partition to FAMILY fails that test and only that test, and skipping it for GIVEN, for MIDDLE, or for FAMILY each fails too — which the FIRST draft of the test did not achieve. It carried one token per role, and a one-element partition is the identity, so the MIDDLE arm was asserted and unpinned: skipping the partition for MIDDLE passed the entire suite. Two tokens per role is what closes it, and the general form is worth carrying: a test written to pin a partition, a sort or a dedup needs at least two elements in every group it claims to cover, or the claim rides on a no-op. ### R4 — case repair reads the unjoined mark From 1d9b65101ab6f48bd486aa0f1e8a54be162a1d80 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 30 Aug 2026 01:15:05 -0700 Subject: [PATCH 04/12] docs(api): say in the rendered API reference which order initials come out in ParsedName.initials' docstring IS the reference modules.html renders, so a docs sweep that stops at the .rst files misses half the page (AGENTS.md's release checklist, step 0). The order clause was in _render.initials, which autodoc does not render. Co-Authored-By: Claude Opus 5 --- nameparser/_types.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/nameparser/_types.py b/nameparser/_types.py index 4836082e..d6e7fd1d 100644 --- a/nameparser/_types.py +++ b/nameparser/_types.py @@ -860,6 +860,9 @@ def initials(self, spec: str = "{given} {middle} {family}", delimiter: str = ".", separator: str = " ") -> str: """Initials per group; v1's initials_format/_delimiter/_separator become call-site arguments instead of Config-wide settings. + Each group is ordered the way its own field is ordered, so a + name whose middle words fold into the family initials them + before the family's own words, as `family` renders them. Valid spec keys: given, middle, family.""" import nameparser._render as _render return _render.initials(self, spec, delimiter, separator) From 8e4e7da1ea8753f7798d74cfa85fa5399cf607b3 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 30 Aug 2026 01:16:00 -0700 Subject: [PATCH 05/12] docs(mechanisms): the fixture-arity form of the inert-measurement class The #408 test's one-token-per-role fixture made its MIDDLE arm's mutation a no-op, and the FAMILY-scoped mutation that did fail read as proof the arms were pinned. Recorded as a Verification shape, since the failure arrives through the fixture rather than the predicate and the existing entries all address the predicate. Co-Authored-By: Claude Opus 5 --- docs/design/decisions.md | 2 +- docs/design/mechanisms.md | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 04b55d4a..4f9b6f63 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -732,7 +732,7 @@ R3's earlier history is under `decisions.md#R2`, which this entry does not repea BLAST RADIUS by policy, over the same corpus × three `name_order` values × `middle_as_family` off/on (6564 parses, 660 move). Default order: 1 / 71. FAMILY_FIRST: 1 / 294. FAMILY_FIRST_GIVEN_LAST: 1 / 292. STATE THE POPULATION BESIDE THE COUNT, per this file's own rule: the shape an order change can be observed in is a contributing folded token standing AFTER a contributing unfolded one of the same role, and at the default order the corpus holds exactly 1 such name without `middle_as_family` and 72 with it. So the movers are 1 of 1 and 71 of 72 — a near-total hit rate over a corpus that carries one instance of P6's half, not a blast radius near zero. (The one non-mover is `Chang, Andy C I`, where the folded `C` and the family `Chang` share a letter and the reorder is invisible in initials.) Nine corpus names carry the tag at all at the default order; the other eight have a single contributing family token, which makes the partition the identity. The `middle_as_family` half is where the reach is, which follows from the producers: O3 folds every middle word, while P6's attachment moves initials only where the attached run is READMITTED by the unjoined mark — a family that is all particles — so it reaches one corpus name in each order and the same one. Recompute by parsing the four `tools/differential/corpus*.jsonl` files deduped under each of the six policies and comparing `initials()` against a folded-first partition of `tokens_for(role)`. THE VERIFICATION LIMIT, and it is decisions.md#R4's, restated because a plausible reading of "one name moves at the default policy" is that the ledger owes it a rule. It does not, and could not: `tools/differential/compare.py` compares the seven role fields plus `_ambiguities`, and `initials()` is not among them — the same blindness recorded for `capitalized()` under R4. Measured rather than reasoned: the gate was run at all three baselines on the pre-change tree (exported with `git archive HEAD` into a scratch directory, so nothing in the working tree moved) and on the changed one, and the two outputs are byte-identical apart from the tree path and the baseline-worker temp path — 1094 corpus names, 229 / 194 / 102 intentional diffs, `unexplained: 0`, and 67 / 58 / 51 headings with identical per-heading counts, at 1.4.0 / 2.0.0 / 2.1.0. NO LEDGER RULE WAS ADDED, and none was needed. What stands in its place is `rules.md#R3`'s example line and `tests/v2/test_render.py`, both mutation-checked. WHY R3 GAINED A SENTENCE rather than leaning on R1. R1 already says folded family words "render before the rest of the family wherever they stood in the string" — but its subject is "Every FIELD", and `initials()` is not a field: R3 calls it "this view", and it takes a format spec and two separators rather than being read as an attribute. So R1 does not reach it by its own words, and #408 is precisely what leaving that to inference costs. R3's new clause states the order in R3's own voice, carries the O3 example line (`"Hassan, Mohamad Ahmad Ali"` under `middle_as_family` → `initials="M. A. A. H."`, reusing R1's own input so `corpus_rules.jsonl` is unchanged at 241 names), and R1 and R3 now name each other in `interacts:` — the field says which order, the view says it follows the field. P6's half gets no example line, for the same reason R1 carries none: the only input that witnesses it is `der, y van`, whose PARSE is itself contested, so pinning its initials normatively would pin a value another open question can move. It is pinned in the unit test instead, which is where today's-behavior pins belong. - THE PARTITION IS PER ROLE, not scoped to FAMILY, and this is a deliberate choice rather than a copied line. Both producers re-role to FAMILY, so FAMILY is the only role a parse can put the tag on today and the GIVEN and MIDDLE arms are unreachable. They are there because `_text_for` partitions for every role it renders and the two views must not diverge again — the same uniformity `_post_rules`' own UNJOINED_TAG loop takes for its three roles, "for uniformity with the rule, not because all three are observable". A producer that ever folded into another part would otherwise reopen #408 there with nothing to fail. `test_initials_folds_in_every_role_it_renders` pins it from a hand-built name, since no input string can. Mutation-checked FOUR ways, and the fourth is the one that matters: scoping the partition to FAMILY fails that test and only that test, and skipping it for GIVEN, for MIDDLE, or for FAMILY each fails too — which the FIRST draft of the test did not achieve. It carried one token per role, and a one-element partition is the identity, so the MIDDLE arm was asserted and unpinned: skipping the partition for MIDDLE passed the entire suite. Two tokens per role is what closes it, and the general form is worth carrying: a test written to pin a partition, a sort or a dedup needs at least two elements in every group it claims to cover, or the claim rides on a no-op. + THE PARTITION IS PER ROLE, not scoped to FAMILY, and this is a deliberate choice rather than a copied line. Both producers re-role to FAMILY, so FAMILY is the only role a parse can put the tag on today and the GIVEN and MIDDLE arms are unreachable. They are there because `_text_for` partitions for every role it renders and the two views must not diverge again — the same uniformity `_post_rules`' own UNJOINED_TAG loop takes for its three roles, "for uniformity with the rule, not because all three are observable". A producer that ever folded into another part would otherwise reopen #408 there with nothing to fail. `test_initials_folds_in_every_role_it_renders` pins it from a hand-built name, since no input string can. Mutation-checked FOUR ways, and the fourth is the one that matters: scoping the partition to FAMILY fails that test and only that test, and skipping it for GIVEN, for MIDDLE, or for FAMILY each fails too — which the FIRST draft of the test did not achieve. It carried one token per role, and a one-element partition is the identity, so the MIDDLE arm was asserted and unpinned: skipping the partition for MIDDLE passed the entire suite. Two tokens per role is what closes it, and the general form is carried as mechanisms.md#TWO-ELEMENT-GROUPS rather than left here: a test written to pin a partition, a sort or a dedup needs at least two elements in every group it claims to cover, or the claim rides on a no-op. ### R4 — case repair reads the unjoined mark diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 3678c7ad..c79af151 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -173,6 +173,10 @@ Contract statement. A workaround keyed to a third-party library's measured defec Contract statement. Where a fix has more than one plausible shape, every candidate is measured before one is chosen — simulated in-process against the real pipeline rather than implemented — and the choice cites the resulting table. How it works. The seams already exist, so simulating is cheap: `Parser(lexicon=dataclasses.replace(Lexicon.default(), ...))` for a vocabulary question, a pytest plugin recompiling one stage for a code question, a monkeypatched predicate for a guard. What the table buys is the discovery that the candidates differ in ways argument does not reveal — #400's obvious fix, adding `abd` to the wordlist, provably changed nothing on the common three-word shape, and the two vocabulary alternatives each silently lost a credential spelling; the winner was a fifth option the wordlist framing had hidden. The counterexample is the same PR's ledger `fields`, widened to include `suffix` on one reviewer's finding without pricing what else the widening claimed: it took `{family, suffix}` from a neighbouring rule and two further shapes from UNEXPLAINED, diffs in which the rule's own subject does not move, and was reverted. One table would have shown that before the commit rather than after. Reach for it when. A review names a defect and the fix looks obvious — that is when the option space is least examined, and the obvious fix is the candidate whose cost nobody has priced. Not a mechanism with a home module; the technique IS the seams above, which is why it sits here rather than in the catalog. +### TWO-ELEMENT-GROUPS — a partition's mutation check needs something to permute + +Problem shape. A test pins an ordering, a sort, a dedup or a partition, and its fixture gives one of the affected groups a single element. Contract statement. Every group a test claims to cover carries at least TWO elements, because on a one-element group a partition is the identity — the assertion holds whether the code runs or not, and the mutation that should expose it passes. How it works. This is the inert-measurement class arriving through the FIXTURE rather than through the predicate, and it hides behind a mutation check that looks conclusive: #408's `initials()` partitions three roles, and its first test carried one token per role plus a second GIVEN token. Scoping the partition to FAMILY failed that test, which read as the arms being pinned — but skipping the partition for MIDDLE alone passed the entire suite (measured 2026-08-30, in review, at 6128 passed), because `initials("{middle}")` over one token is the same string either way. The tell is that the mutation which fails is not the mutation the test's docstring promises: a per-role claim wants a per-role mutation, one arm at a time, and only then does the fixture's arity show. Reach for it when. A test asserts on the ORDER or MEMBERSHIP of a rendered group and the fixture was written to be small. + ### Field notes — the traps themselves - Enumerate the rules that BUILD a structure; do not recall them. #395's unit walk was written three times in one PR — P2's chain missing, then the conjunction and bound-given branches absorbing one token where the particle branch absorbed a unit, then the suffix stop — and each miss came from listing the joining rules from memory instead of reading them out of rules.md. Both later misses reproduced the very defect the first fix had just removed, mirrored. From dfa2dbc11180be2b3568985fc911e2be06ff8046 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 30 Aug 2026 01:17:24 -0700 Subject: [PATCH 06/12] test(render): cite the fixture-arity mechanism instead of a dangling pointer The comment referred to 'the mutation below', which lives in decisions.md rather than in this file. Cites mechanisms.md#TWO-ELEMENT-GROUPS under the excerpt discipline instead -- and the citation test caught the first attempt, which had paraphrased the em dash away. Co-Authored-By: Claude Opus 5 --- tests/v2/test_render.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/tests/v2/test_render.py b/tests/v2/test_render.py index 69a5981c..e7f6f6ea 100644 --- a/tests/v2/test_render.py +++ b/tests/v2/test_render.py @@ -263,11 +263,11 @@ def test_initials_folds_in_every_role_it_renders() -> None: Token("Dee", Span(12, 15), Role.MIDDLE, frozenset({FOLDED_TAG})), ]) # the fields already order this way; the view now agrees. Each - # role carries TWO tokens on purpose: a one-token group makes the - # partition the identity, so a single folded middle would have - # left this arm asserted and unpinned (the shape the mutation - # below caught -- scoping the partition to GIVEN and FAMILY - # passed the whole suite against an earlier draft of this test). + # role carries TWO tokens on purpose, per + # mechanisms.md#TWO-ELEMENT-GROUPS: "on a one-element group a + # partition is the identity" and the mutation that should expose + # it passes. An earlier draft gave MIDDLE one token, and skipping + # the partition for MIDDLE alone passed the whole suite. assert pn.given == "Cyd Ann" assert pn.initials("{given}") == "C. A." assert pn.middle == "Dee Bea" From dd1eef55d2cc5294b5f1f010737d376b3de92312 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 30 Aug 2026 01:18:55 -0700 Subject: [PATCH 07/12] docs(decisions): the first test's fixture was two GIVEN and one MIDDLE, not one per role Co-Authored-By: Claude Opus 5 --- docs/design/decisions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 4f9b6f63..7c717051 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -732,7 +732,7 @@ R3's earlier history is under `decisions.md#R2`, which this entry does not repea BLAST RADIUS by policy, over the same corpus × three `name_order` values × `middle_as_family` off/on (6564 parses, 660 move). Default order: 1 / 71. FAMILY_FIRST: 1 / 294. FAMILY_FIRST_GIVEN_LAST: 1 / 292. STATE THE POPULATION BESIDE THE COUNT, per this file's own rule: the shape an order change can be observed in is a contributing folded token standing AFTER a contributing unfolded one of the same role, and at the default order the corpus holds exactly 1 such name without `middle_as_family` and 72 with it. So the movers are 1 of 1 and 71 of 72 — a near-total hit rate over a corpus that carries one instance of P6's half, not a blast radius near zero. (The one non-mover is `Chang, Andy C I`, where the folded `C` and the family `Chang` share a letter and the reorder is invisible in initials.) Nine corpus names carry the tag at all at the default order; the other eight have a single contributing family token, which makes the partition the identity. The `middle_as_family` half is where the reach is, which follows from the producers: O3 folds every middle word, while P6's attachment moves initials only where the attached run is READMITTED by the unjoined mark — a family that is all particles — so it reaches one corpus name in each order and the same one. Recompute by parsing the four `tools/differential/corpus*.jsonl` files deduped under each of the six policies and comparing `initials()` against a folded-first partition of `tokens_for(role)`. THE VERIFICATION LIMIT, and it is decisions.md#R4's, restated because a plausible reading of "one name moves at the default policy" is that the ledger owes it a rule. It does not, and could not: `tools/differential/compare.py` compares the seven role fields plus `_ambiguities`, and `initials()` is not among them — the same blindness recorded for `capitalized()` under R4. Measured rather than reasoned: the gate was run at all three baselines on the pre-change tree (exported with `git archive HEAD` into a scratch directory, so nothing in the working tree moved) and on the changed one, and the two outputs are byte-identical apart from the tree path and the baseline-worker temp path — 1094 corpus names, 229 / 194 / 102 intentional diffs, `unexplained: 0`, and 67 / 58 / 51 headings with identical per-heading counts, at 1.4.0 / 2.0.0 / 2.1.0. NO LEDGER RULE WAS ADDED, and none was needed. What stands in its place is `rules.md#R3`'s example line and `tests/v2/test_render.py`, both mutation-checked. WHY R3 GAINED A SENTENCE rather than leaning on R1. R1 already says folded family words "render before the rest of the family wherever they stood in the string" — but its subject is "Every FIELD", and `initials()` is not a field: R3 calls it "this view", and it takes a format spec and two separators rather than being read as an attribute. So R1 does not reach it by its own words, and #408 is precisely what leaving that to inference costs. R3's new clause states the order in R3's own voice, carries the O3 example line (`"Hassan, Mohamad Ahmad Ali"` under `middle_as_family` → `initials="M. A. A. H."`, reusing R1's own input so `corpus_rules.jsonl` is unchanged at 241 names), and R1 and R3 now name each other in `interacts:` — the field says which order, the view says it follows the field. P6's half gets no example line, for the same reason R1 carries none: the only input that witnesses it is `der, y van`, whose PARSE is itself contested, so pinning its initials normatively would pin a value another open question can move. It is pinned in the unit test instead, which is where today's-behavior pins belong. - THE PARTITION IS PER ROLE, not scoped to FAMILY, and this is a deliberate choice rather than a copied line. Both producers re-role to FAMILY, so FAMILY is the only role a parse can put the tag on today and the GIVEN and MIDDLE arms are unreachable. They are there because `_text_for` partitions for every role it renders and the two views must not diverge again — the same uniformity `_post_rules`' own UNJOINED_TAG loop takes for its three roles, "for uniformity with the rule, not because all three are observable". A producer that ever folded into another part would otherwise reopen #408 there with nothing to fail. `test_initials_folds_in_every_role_it_renders` pins it from a hand-built name, since no input string can. Mutation-checked FOUR ways, and the fourth is the one that matters: scoping the partition to FAMILY fails that test and only that test, and skipping it for GIVEN, for MIDDLE, or for FAMILY each fails too — which the FIRST draft of the test did not achieve. It carried one token per role, and a one-element partition is the identity, so the MIDDLE arm was asserted and unpinned: skipping the partition for MIDDLE passed the entire suite. Two tokens per role is what closes it, and the general form is carried as mechanisms.md#TWO-ELEMENT-GROUPS rather than left here: a test written to pin a partition, a sort or a dedup needs at least two elements in every group it claims to cover, or the claim rides on a no-op. + THE PARTITION IS PER ROLE, not scoped to FAMILY, and this is a deliberate choice rather than a copied line. Both producers re-role to FAMILY, so FAMILY is the only role a parse can put the tag on today and the GIVEN and MIDDLE arms are unreachable. They are there because `_text_for` partitions for every role it renders and the two views must not diverge again — the same uniformity `_post_rules`' own UNJOINED_TAG loop takes for its three roles, "for uniformity with the rule, not because all three are observable". A producer that ever folded into another part would otherwise reopen #408 there with nothing to fail. `test_initials_folds_in_every_role_it_renders` pins it from a hand-built name, since no input string can. Mutation-checked FOUR ways, and the fourth is the one that matters: scoping the partition to FAMILY fails that test and only that test, and skipping it for GIVEN, for MIDDLE, or for FAMILY each fails too — which the FIRST draft of the test did not achieve. It carried two GIVEN tokens and ONE MIDDLE token, and a one-element partition is the identity, so the MIDDLE arm was asserted and unpinned: skipping the partition for MIDDLE passed the entire suite, 6128 tests and all. Two tokens per role is what closes it, and the general form is carried as mechanisms.md#TWO-ELEMENT-GROUPS rather than left here: a test written to pin a partition, a sort or a dedup needs at least two elements in every group it claims to cover, or the claim rides on a no-op. ### R4 — case repair reads the unjoined mark From fc075242d13e6ac6a9ec2f105193b78e6c4fb423 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 30 Aug 2026 01:26:15 -0700 Subject: [PATCH 08/12] test(render): the per-role fixture had no FAMILY token The test named for every role it renders carried two GIVEN and two MIDDLE and no FAMILY, so skipping the partition for FAMILY passed it -- caught only by its siblings and by R3's example line, neither of which is about per-role application. Verified by mutation before and after: the skip-FAMILY mutation now fails this test and did not. Same miss as the first draft's, one role apart. A zero-element group is the identity as surely as a one-element group and reads even less like a gap, which is how it survived the round that had just named the one-element form. decisions.md and mechanisms.md#TWO-ELEMENT-GROUPS both described the fixture wrongly and are corrected; the mechanism's only worked example was certifying as fixed the gap that survived. Co-Authored-By: Claude Opus 5 --- docs/design/decisions.md | 2 +- docs/design/mechanisms.md | 2 +- tests/v2/test_render.py | 18 +++++++++++++----- 3 files changed, 15 insertions(+), 7 deletions(-) diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 7c717051..75972009 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -732,7 +732,7 @@ R3's earlier history is under `decisions.md#R2`, which this entry does not repea BLAST RADIUS by policy, over the same corpus × three `name_order` values × `middle_as_family` off/on (6564 parses, 660 move). Default order: 1 / 71. FAMILY_FIRST: 1 / 294. FAMILY_FIRST_GIVEN_LAST: 1 / 292. STATE THE POPULATION BESIDE THE COUNT, per this file's own rule: the shape an order change can be observed in is a contributing folded token standing AFTER a contributing unfolded one of the same role, and at the default order the corpus holds exactly 1 such name without `middle_as_family` and 72 with it. So the movers are 1 of 1 and 71 of 72 — a near-total hit rate over a corpus that carries one instance of P6's half, not a blast radius near zero. (The one non-mover is `Chang, Andy C I`, where the folded `C` and the family `Chang` share a letter and the reorder is invisible in initials.) Nine corpus names carry the tag at all at the default order; the other eight have a single contributing family token, which makes the partition the identity. The `middle_as_family` half is where the reach is, which follows from the producers: O3 folds every middle word, while P6's attachment moves initials only where the attached run is READMITTED by the unjoined mark — a family that is all particles — so it reaches one corpus name in each order and the same one. Recompute by parsing the four `tools/differential/corpus*.jsonl` files deduped under each of the six policies and comparing `initials()` against a folded-first partition of `tokens_for(role)`. THE VERIFICATION LIMIT, and it is decisions.md#R4's, restated because a plausible reading of "one name moves at the default policy" is that the ledger owes it a rule. It does not, and could not: `tools/differential/compare.py` compares the seven role fields plus `_ambiguities`, and `initials()` is not among them — the same blindness recorded for `capitalized()` under R4. Measured rather than reasoned: the gate was run at all three baselines on the pre-change tree (exported with `git archive HEAD` into a scratch directory, so nothing in the working tree moved) and on the changed one, and the two outputs are byte-identical apart from the tree path and the baseline-worker temp path — 1094 corpus names, 229 / 194 / 102 intentional diffs, `unexplained: 0`, and 67 / 58 / 51 headings with identical per-heading counts, at 1.4.0 / 2.0.0 / 2.1.0. NO LEDGER RULE WAS ADDED, and none was needed. What stands in its place is `rules.md#R3`'s example line and `tests/v2/test_render.py`, both mutation-checked. WHY R3 GAINED A SENTENCE rather than leaning on R1. R1 already says folded family words "render before the rest of the family wherever they stood in the string" — but its subject is "Every FIELD", and `initials()` is not a field: R3 calls it "this view", and it takes a format spec and two separators rather than being read as an attribute. So R1 does not reach it by its own words, and #408 is precisely what leaving that to inference costs. R3's new clause states the order in R3's own voice, carries the O3 example line (`"Hassan, Mohamad Ahmad Ali"` under `middle_as_family` → `initials="M. A. A. H."`, reusing R1's own input so `corpus_rules.jsonl` is unchanged at 241 names), and R1 and R3 now name each other in `interacts:` — the field says which order, the view says it follows the field. P6's half gets no example line, for the same reason R1 carries none: the only input that witnesses it is `der, y van`, whose PARSE is itself contested, so pinning its initials normatively would pin a value another open question can move. It is pinned in the unit test instead, which is where today's-behavior pins belong. - THE PARTITION IS PER ROLE, not scoped to FAMILY, and this is a deliberate choice rather than a copied line. Both producers re-role to FAMILY, so FAMILY is the only role a parse can put the tag on today and the GIVEN and MIDDLE arms are unreachable. They are there because `_text_for` partitions for every role it renders and the two views must not diverge again — the same uniformity `_post_rules`' own UNJOINED_TAG loop takes for its three roles, "for uniformity with the rule, not because all three are observable". A producer that ever folded into another part would otherwise reopen #408 there with nothing to fail. `test_initials_folds_in_every_role_it_renders` pins it from a hand-built name, since no input string can. Mutation-checked FOUR ways, and the fourth is the one that matters: scoping the partition to FAMILY fails that test and only that test, and skipping it for GIVEN, for MIDDLE, or for FAMILY each fails too — which the FIRST draft of the test did not achieve. It carried two GIVEN tokens and ONE MIDDLE token, and a one-element partition is the identity, so the MIDDLE arm was asserted and unpinned: skipping the partition for MIDDLE passed the entire suite, 6128 tests and all. Two tokens per role is what closes it, and the general form is carried as mechanisms.md#TWO-ELEMENT-GROUPS rather than left here: a test written to pin a partition, a sort or a dedup needs at least two elements in every group it claims to cover, or the claim rides on a no-op. + THE PARTITION IS PER ROLE, not scoped to FAMILY, and this is a deliberate choice rather than a copied line. Both producers re-role to FAMILY, so FAMILY is the only role a parse can put the tag on today and the GIVEN and MIDDLE arms are unreachable. They are there because `_text_for` partitions for every role it renders and the two views must not diverge again — the same uniformity `_post_rules`' own UNJOINED_TAG loop takes for its three roles, "for uniformity with the rule, not because all three are observable". A producer that ever folded into another part would otherwise reopen #408 there with nothing to fail. `test_initials_folds_in_every_role_it_renders` pins it from a hand-built name, since no input string can. Mutation-checked four ways — scoping the partition to FAMILY, and skipping it for GIVEN, for MIDDLE, for FAMILY — and each fails that test. It took THREE drafts of the fixture to make that true, and the two misses are the same miss one role apart. The first carried two GIVEN tokens and ONE MIDDLE: a one-element partition is the identity, so the MIDDLE arm was asserted and unpinned, and skipping the partition for MIDDLE passed the entire suite. The second fixed MIDDLE and carried NO FAMILY token at all, in a test named for every role it renders — so skipping the partition for FAMILY passed this test, and was caught only by its siblings and by R3's example line, neither of which is about per-role application. A zero-element group is the identity too, and reads even less like a gap than a one-element one, which is why the second miss survived a review that had just named the first. Two tokens per role is what closes it, and the general form is carried as mechanisms.md#TWO-ELEMENT-GROUPS rather than left here: a test written to pin a partition, a sort or a dedup needs at least two elements in every group it claims to cover, or the claim rides on a no-op. ### R4 — case repair reads the unjoined mark diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index c79af151..b535d6e2 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -175,7 +175,7 @@ Contract statement. Where a fix has more than one plausible shape, every candida ### TWO-ELEMENT-GROUPS — a partition's mutation check needs something to permute -Problem shape. A test pins an ordering, a sort, a dedup or a partition, and its fixture gives one of the affected groups a single element. Contract statement. Every group a test claims to cover carries at least TWO elements, because on a one-element group a partition is the identity — the assertion holds whether the code runs or not, and the mutation that should expose it passes. How it works. This is the inert-measurement class arriving through the FIXTURE rather than through the predicate, and it hides behind a mutation check that looks conclusive: #408's `initials()` partitions three roles, and its first test carried one token per role plus a second GIVEN token. Scoping the partition to FAMILY failed that test, which read as the arms being pinned — but skipping the partition for MIDDLE alone passed the entire suite (measured 2026-08-30, in review, at 6128 passed), because `initials("{middle}")` over one token is the same string either way. The tell is that the mutation which fails is not the mutation the test's docstring promises: a per-role claim wants a per-role mutation, one arm at a time, and only then does the fixture's arity show. Reach for it when. A test asserts on the ORDER or MEMBERSHIP of a rendered group and the fixture was written to be small. +Problem shape. A test pins an ordering, a sort, a dedup or a partition, and its fixture gives one of the affected groups a single element. Contract statement. Every group a test claims to cover carries at least TWO elements, because on a one-element group a partition is the identity — the assertion holds whether the code runs or not, and the mutation that should expose it passes. How it works. This is the inert-measurement class arriving through the FIXTURE rather than through the predicate, and it hides behind a mutation check that looks conclusive: #408's `initials()` partitions three roles, and its first test carried two GIVEN tokens and one MIDDLE, and its second carried two of each and no FAMILY token at all — a group of ZERO is the identity as surely as a group of one, and looks even less like a hole. Scoping the partition to FAMILY failed that test, which read as the arms being pinned — but skipping the partition for MIDDLE alone passed the entire suite (measured 2026-08-30, in review, at 6128 passed), because `initials("{middle}")` over one token is the same string either way. The tell is that the mutation which fails is not the mutation the test's docstring promises: a per-role claim wants a per-role mutation, one arm at a time, and only then does the fixture's arity show. Reach for it when. A test asserts on the ORDER or MEMBERSHIP of a rendered group and the fixture was written to be small. ### Field notes — the traps themselves diff --git a/tests/v2/test_render.py b/tests/v2/test_render.py index e7f6f6ea..cbf7241f 100644 --- a/tests/v2/test_render.py +++ b/tests/v2/test_render.py @@ -256,22 +256,30 @@ def test_initials_folds_in_every_role_it_renders() -> None: producer that ever folded into another part would otherwise reopen #408 there, silently, with nothing to fail. """ - pn = _pn("Ann Bea Cyd Dee", [ + pn = _pn("Ann Bea Cyd Dee Eve Fay", [ Token("Ann", Span(0, 3), Role.GIVEN), Token("Bea", Span(4, 7), Role.MIDDLE), Token("Cyd", Span(8, 11), Role.GIVEN, frozenset({FOLDED_TAG})), Token("Dee", Span(12, 15), Role.MIDDLE, frozenset({FOLDED_TAG})), + Token("Eve", Span(16, 19), Role.FAMILY), + Token("Fay", Span(20, 23), Role.FAMILY, frozenset({FOLDED_TAG})), ]) - # the fields already order this way; the view now agrees. Each - # role carries TWO tokens on purpose, per + # the fields already order this way; the view now agrees. All + # THREE roles carry two tokens, per # mechanisms.md#TWO-ELEMENT-GROUPS: "on a one-element group a # partition is the identity" and the mutation that should expose - # it passes. An earlier draft gave MIDDLE one token, and skipping - # the partition for MIDDLE alone passed the whole suite. + # it passes. Two drafts got this wrong in the same way one role + # apart: the first gave MIDDLE one token, and skipping the + # partition for MIDDLE alone passed the whole suite; the second + # gave FAMILY none, so skipping it for FAMILY passed THIS test + # and was caught only by its siblings. A group with zero elements + # is the identity too, and reads even less like a gap. assert pn.given == "Cyd Ann" assert pn.initials("{given}") == "C. A." assert pn.middle == "Dee Bea" assert pn.initials("{middle}") == "D. B." + assert pn.family == "Fay Eve" + assert pn.initials("{family}") == "F. E." def test_initials_custom_delimiter_and_separator() -> None: From 9c49e0264edec980924da0bb0102e15cc1dab7d1 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 30 Aug 2026 02:13:33 -0700 Subject: [PATCH 09/12] test(render): the folded RUN was initial-degenerate; and a stale release bullet Two findings from the branch review. The partition BUILDS a group, and that group needs the two-element treatment as much as the role group does. The only fixture with a two-element folded run was 'Hassan, Mohamad Ahmad Ali', whose run is 'Ahmad Ali' -- both initial to 'A.', so reversing the folded half alone passed every test in the file. Verified before and after: the reversal mutation now fails test_initials_order_folded_words_ first_like_the_family_field and previously failed nothing. Third occurrence of the arity shape on this branch, one level down each time: a one-element role group, then a zero-element one, now a two-element group whose members the assertion cannot tell apart. mechanisms.md#TWO-ELEMENT-GROUPS says count the elements; this one says count the ones the assertion can DISTINGUISH. Separately, the #385/#402 bullet six lines above the #408 bullet, in the same Unreleased section, still told readers the two APIs order initials differently and that #408 was open. Every clause of it was false as shipped: the values agree at 'y. v. d.', the 87 constructed inputs are 0, and #408 closes below it. No count survives the fix, so none is carried forward. Co-Authored-By: Claude Opus 5 --- docs/release_log.rst | 2 +- tests/v2/test_render.py | 10 ++++++++++ 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/release_log.rst b/docs/release_log.rst index c000100d..bbf22794 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -63,7 +63,7 @@ Release Log - Fix a name opening with a particle that is *never* a given name being split at the particle under a family-first name order -- ``Policy(name_order=FAMILY_FIRST)`` and ``Policy(name_order=FAMILY_FIRST_GIVEN_LAST)`` alike, and identically: ``"de Mesnil"`` read as family ``de``, given ``Mesnil``, and ``"de la Vega"`` as family ``de``, given ``la Vega``. Each is now the whole surname, as it has always been in the default order. The rule enforcing it asked for the particle by the ``GIVEN`` role, which under a family-first order belongs to the token *after* the particle, so the test read the wrong word and declined. It now also asks by position -- the piece that opens the name -- so both shapes of the same rule are caught: where such a particle stands alone as a piece, either opening the name or in the given position, the name is left with no given name at all, the given and the middles folding into the family. Standing *alone* is the whole of it, and the rule claims nothing wider: ``"Juan de la Vega"`` under ``FAMILY_FIRST`` still reports given ``de la Vega``, because there the particle chained onto the words after it rather than standing alone, and a bare ``"de"`` with nothing to fold into is still reported as the given name. The decision behind the fix: a word that can never be a given name leaves ``name_order`` nothing to decide, so declaring family-first is not a reason to make ``de`` a surname on its own. A leading particle that *may* be a given name is genuinely order-dependent and is untouched -- ``"van Gogh"`` still reads as family ``van``, given ``Gogh`` under both family-first orders. This is also what gives ``Lexicon.particles_ambiguous`` an effect outside the default order: taking a word out of it now changes the parsed fields under a family-first order, where before it moved only the ambiguity report. Seven of the 751 differential corpus names move, the same seven under each family-first order; default-order output is byte-identical over all 751, at the 1.4.0, 2.0.0 and 2.1.0 differential baselines alike (closes #359) - - Fix a family name made only of particle words reporting no base on the 2.0 API, so the surname vanished from ``family_base`` and, on both APIs, from the initials: ``parse("Anh Do")`` gave family ``Do`` with ``family_base`` ``''`` and initials ``A.``, and under ``Policy(name_order=FAMILY_FIRST)`` ``"Del Toro"`` gave family ``Del`` the same way. The empty base was a 2.0 regression rather than a longstanding bug -- 1.4.0's own guard kept ``HumanName("Anh Do").last_base`` at ``Do``, and the facade has kept it right throughout; what 1.4.0 and 2.1 shared was the missing initial. A particle earns its name by joining forward to the word it modifies, so a particle standing alone in a name part is not doing a particle's work there and reads as an ordinary name word: it anchors the base, leaves the particles view, and contributes an initial. ``"Anh Do"`` is now base ``Do``, initials ``A. D.``; ``"Juan van der"`` is base ``van der``, initials ``J. v. d.``; ``"Nguyen, Van Le"`` initials ``V. L. N.`` where the middle name used to be dropped. Position decides this, not vocabulary -- whether the word is borne as a surname somewhere does not enter into it, which reverses the reasoning ``rules.md#R2`` carried before. The invariant it exists to hold: a non-empty family always has a non-empty base, because a particle needs a base to attach to. Where the particles DO join a name word nothing changes -- ``"Juan de la Vega"`` keeps base ``Vega``, particles ``de la`` and initials ``J. V.``. The parse fields themselves do not move: ``title``, ``given``, ``middle``, ``family``, ``suffix``, ``nickname`` and ``maiden`` are byte-identical over all 751 differential corpus names in all three name orders, and only the derived views and the initials change -- which is why the differential harness, which compares those seven roles, reports no diffs for this at any baseline. One consequence to know about: where the whole family is particles AND a tussenvoegsel was attached to it (#379), the two APIs order the initials differently -- ``parse("der, y van").initials()`` is ``y. d. v.`` against the facade's ``y. v. d.`` -- because ``initials()`` reads tokens in written order while the family FIELD renders folded words first. That affects 87 constructed inputs and no corpus name; it is a pre-existing ordering gap in ``initials()`` that this change makes visible, tracked at #408 (closes #385, closes #402) + - Fix a family name made only of particle words reporting no base on the 2.0 API, so the surname vanished from ``family_base`` and, on both APIs, from the initials: ``parse("Anh Do")`` gave family ``Do`` with ``family_base`` ``''`` and initials ``A.``, and under ``Policy(name_order=FAMILY_FIRST)`` ``"Del Toro"`` gave family ``Del`` the same way. The empty base was a 2.0 regression rather than a longstanding bug -- 1.4.0's own guard kept ``HumanName("Anh Do").last_base`` at ``Do``, and the facade has kept it right throughout; what 1.4.0 and 2.1 shared was the missing initial. A particle earns its name by joining forward to the word it modifies, so a particle standing alone in a name part is not doing a particle's work there and reads as an ordinary name word: it anchors the base, leaves the particles view, and contributes an initial. ``"Anh Do"`` is now base ``Do``, initials ``A. D.``; ``"Juan van der"`` is base ``van der``, initials ``J. v. d.``; ``"Nguyen, Van Le"`` initials ``V. L. N.`` where the middle name used to be dropped. Position decides this, not vocabulary -- whether the word is borne as a surname somewhere does not enter into it, which reverses the reasoning ``rules.md#R2`` carried before. The invariant it exists to hold: a non-empty family always has a non-empty base, because a particle needs a base to attach to. Where the particles DO join a name word nothing changes -- ``"Juan de la Vega"`` keeps base ``Vega``, particles ``de la`` and initials ``J. V.``. The parse fields themselves do not move: ``title``, ``given``, ``middle``, ``family``, ``suffix``, ``nickname`` and ``maiden`` are byte-identical over all 751 differential corpus names in all three name orders, and only the derived views and the initials change -- which is why the differential harness, which compares those seven roles, reports no diffs for this at any baseline. One consequence to know about: where the whole family is particles AND a tussenvoegsel was attached to it (#379), this change exposed a pre-existing ordering gap between ``initials()`` and the family field of the same name -- fixed in the #408 entry below, so the two agree as shipped (closes #385, closes #402) - Fix case repair lowercasing the words of a family name made only of particle words, where every other view already reads them as ordinary name words: ``HumanName("ANH DO").capitalize()`` gives ``Anh Do`` where it gave ``Anh do``, and ``"anh van do"`` gives ``Anh Van Do`` where it gave ``Anh van do``. A particle earns its name by joining forward to the word it modifies, so a part whose every word is particle vocabulary leaves none of them anything to join; the fix above already made those words anchor ``family_base`` and contribute initials, and case repair now agrees with them rather than reading the same word two ways. The test is the whole part, not a particle standing alone, which is why the two-word family in ``"anh van do"`` moves along with the one-word family in ``"ANH DO"`` -- the same Vietnamese surname, and a standing-alone test would have read it one way behind a given name and another way alone. This DIFFERS FROM 1.4.0 deliberately and does not restore it: 1.4.0 returned ``Anh do``, lowercasing on vocabulary membership alone. The accepted cost is that a degenerate family which is nothing but particles capitalizes too, so ``"juan van der"`` gives ``Juan Van Der`` where 1.4.0 gave ``Juan van der``. A conjunction is untouched by any of this, so ``"der, y van"`` gives ``y Van Der`` -- the family capitalizing while the conjunction keeps the lowercase it always had; and where the particles DO join a name word nothing changes, ``"juan de la vega"`` still giving ``Juan de la Vega``. Case repair is not one of the seven role fields the differential harness compares, so no gate run can see this change either way and none of its counts move; the ``rules.md#R4`` examples and the v1 capitalization tests are what pin it (closes #407) diff --git a/tests/v2/test_render.py b/tests/v2/test_render.py index cbf7241f..f87d59bc 100644 --- a/tests/v2/test_render.py +++ b/tests/v2/test_render.py @@ -233,6 +233,16 @@ def test_initials_order_folded_words_first_like_the_family_field() -> None: hassan = maf.parse("Hassan, Mohamad Ahmad Ali") assert hassan.family == "Ahmad Ali Hassan" assert hassan.initials() == "M. A. A. H." + # The folded RUN needs two elements too, and distinguishable + # ones: hassan's run is `Ahmad Ali`, which both initial to + # `A.`, so reversing the folded half alone passes every test + # above. Same arity trap as the fixture drafts, one level + # down -- the partition BUILDS a group, and that group needs + # the treatment too (mechanisms.md#TWO-ELEMENT-GROUPS). + doe2 = maf.parse("Doe, John A. Kenneth") + assert doe2.family == "A. Kenneth Doe" + assert doe2.initials() == "J. A. K. D." + doe = maf.parse("Doe, Dr. John A.") assert doe.family == "A. Doe" assert doe.initials() == "J. A. D." From aa71f10430c3e052153468c1e2751121074bb240 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 30 Aug 2026 02:16:58 -0700 Subject: [PATCH 10/12] test(render): guard the per-role claim three prose sites make _render.initials, its sibling test and mechanisms.md#FOLDED_TAG all say the pipeline puts the tag on FAMILY tokens alone -- which is why the GIVEN and MIDDLE arms of the partition are uniformity rather than reachable behavior. Measured true (0 of 1094 corpus names carry it elsewhere) and tested nowhere, so a rule that ever folded into another part would falsify all three silently and turn initials()'s 'would otherwise reopen #408 there' into a live gap with nothing to fail. Verified by mutation: re-roling one producer's fold to MIDDLE fails this guard. It also asserts its own floor, so it cannot pass by exercising nothing. Three comment overclaims fixed with it, all the reach-versus-rule shape this branch keeps producing: 'the one name the DEFAULT policy moves' is the one CORPUS name (constructed ones move too -- 'de la, y van', 'der, e van'); skip-FAMILY was caught by the siblings AND R3's example line, not the siblings alone; and draft 1 carried both arity misses, not one. Co-Authored-By: Claude Opus 5 --- tests/v2/test_render.py | 39 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 36 insertions(+), 3 deletions(-) diff --git a/tests/v2/test_render.py b/tests/v2/test_render.py index f87d59bc..ced474e5 100644 --- a/tests/v2/test_render.py +++ b/tests/v2/test_render.py @@ -217,7 +217,9 @@ def test_initials_order_folded_words_first_like_the_family_field() -> None: regression that reordered BOTH would still be caught by the literal, and one that reordered neither by the pairing. """ - # P6's attachment, the one name the DEFAULT policy moves + # P6's attachment, the one CORPUS name the default policy moves + # -- constructed ones move too (`de la, y van`, `der, e van`), + # so the count is the corpus's reach and not the rule's van_der = parse("der, y van") assert van_der.family == "van der" assert van_der.initials() == "y. v. d." @@ -255,6 +257,36 @@ def test_initials_order_folded_words_first_like_the_family_field() -> None: assert vega.initials() == "J. V. S." +def test_folded_tag_lands_only_on_family_today() -> None: + """The guard under the per-role claim three prose sites make. + + `_render.initials`, its sibling test and mechanics.md#FOLDED_TAG all + say the pipeline puts the tag on FAMILY tokens alone, which is why + the GIVEN and MIDDLE arms of the partition are uniformity rather + than reachable behavior. That claim was true and tested nowhere: a + rule that ever folded into another part would falsify all three + silently, and turn `initials()`'s "would otherwise reopen #408 + there" from a hypothetical into a live gap with nothing to fail. + """ + from .cases import CASES + seen = 0 + for policy in (Policy(), Policy(middle_as_family=True)): + p = Parser(policy=policy) + for case in CASES: + if case.locale is not None: + continue + for tok in p.parse(case.text).tokens: + if FOLDED_TAG in tok.tags: + seen += 1 + assert tok.role is Role.FAMILY, ( + f"{case.text!r}: {tok.text!r} carries FOLDED_TAG in " + f"{tok.role}. Both producers re-role to FAMILY today; " + f"if that changed on purpose, the per-role claims in " + f"_render.initials, its sibling test and " + f"mechanisms.md#FOLDED_TAG all move with it") + assert seen, "no case row exercises the fold; this guard is inert" + + def test_initials_folds_in_every_role_it_renders() -> None: """The partition is applied per role, exactly as _text_for applies it -- not scoped to FAMILY, where the pipeline's two producers put @@ -279,10 +311,11 @@ def test_initials_folds_in_every_role_it_renders() -> None: # mechanisms.md#TWO-ELEMENT-GROUPS: "on a one-element group a # partition is the identity" and the mutation that should expose # it passes. Two drafts got this wrong in the same way one role - # apart: the first gave MIDDLE one token, and skipping the + # apart, the first carrying both misses: it gave MIDDLE one token, + # and skipping the # partition for MIDDLE alone passed the whole suite; the second # gave FAMILY none, so skipping it for FAMILY passed THIS test - # and was caught only by its siblings. A group with zero elements + # and was caught only by its siblings and by R3's example line. A group with zero elements # is the identity too, and reads even less like a gap. assert pn.given == "Cyd Ann" assert pn.initials("{given}") == "C. A." From 62d9f6b4b023e20686093aacaa95cb274aecc2be Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 30 Aug 2026 02:27:39 -0700 Subject: [PATCH 11/12] docs: the design-docs review findings on #408, measured Five findings from the docs review of this branch. Each was re-verified at HEAD before editing; all five still held. decisions.md#R3's recompute recipe measured the implementation against itself. "Compare initials() against a folded-first partition of tokens_for(role)" is what initials() now IS, so on the shipped tree the recipe yields 0 differing parses, not 660 -- the inert-measurement shape inside the sentence written to prevent it, and the only recompute path the repo offered for the release log's counts too. The recipe now names the comparator that produces the number (the pre-change rendering: the same contributing tokens left in WRITTEN order) and states that the skip filter comes first, because dropping it counts every skipped word as a difference. All four numbers measured 2026-08-30: 660 of 6564 with the filter and written order, reproducing 1 / 71, 1 / 294, 1 / 292; 0 for the old wording; 1546 for written order without the filter; 1018 for the folded partition without it. mechanisms.md#RENDER-HONORS-THE-PARSE called the fix "strictly parity-RESTORING", which overstates in two directions. 588 of the 660 moving parses are under the two family-first orders, which v1 has no equivalent for, so nothing there is restored or broken -- decisions.md#R3 already says this. And the entry's own lead example is a counterexample: parse("der, y van") gives y. d. v. before, y. v. d. after, and 1.4.0 gives y. (measured on the released wheel). The 54-of-71-and-none-away claim is kept and scoped to the population it was measured over. rules.md#R3's new order clause rested on a false premise -- "initials abbreviate a field" -- which the rule's own first sentence and first example line contradict: the family FIELD of "Dr. Juan Q. Xavier de la Vega III" is de la Vega and its initials are J. Q. X. V. #461 is open and is exactly that membership question, so a normative "the two never disagree" would be the authority for forcing initials() to match family_base -- the reading measured and backed out on the previous branch. The clause now rests on order alone. mechanisms.md#FOLDED_TAG claimed the consumer list "has failed there twice in opposite directions". The second never shipped: the revise strip arrived with Parser.revise in the same commit as the test that pins it, and no issue, decision entry or test names a released defect of that shape. Now one shipped failure and one hazard. Its "Lives in." roster also gave that site as a bare method; it is nameparser/_parser.py:187, and the roster is what a reader walks when adding a view. docs/release_log.rst's #408 bullet carried counts from neither source AGENTS.md sanctions -- the bullet itself says the gate cannot see initials(), and R3's example line witnesses the order without counting. No count is dropped: each carries argument the bullet needs. Instead the bullet now says where each comes from -- what moves is recomputed by decisions.md#R3's now-working recipe, while the 1.4.0 comparisons and the facade sweep are dated snapshots nothing in the repo re-derives. Verified: 6129 passed / 156 skipped / 9 xfailed; mypy, ruff, sphinx doctests and the README doctest clean; corpus rebuild 1094 / 241 with no diff; gate 229 / 194 / 102 with unexplained: 0 at 1.4.0 / 2.0.0 / 2.1.0. Co-Authored-By: Claude Opus 5 --- docs/design/decisions.md | 2 +- docs/design/mechanisms.md | 4 ++-- docs/design/rules.md | 7 +++++-- docs/release_log.rst | 2 +- 4 files changed, 9 insertions(+), 6 deletions(-) diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 75972009..c794ae8a 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -729,7 +729,7 @@ R3's earlier history is under `decisions.md#R2`, which this entry does not repea - 2026-08-30 #408 — DONE: `initials()` orders folded words first, as every other view of the same parse already did. O3's fold and P6's attachment both TAG rather than move a token (mechanisms.md#FOLDED_TAG — spans cannot reorder), and `_types._text_for` partitions on that tag for every role it renders, so the `family` field reads `van der` where the tokens sit `der ... van`. `initials()` walked `tokens_for(role)` in written order and never asked, so the two views disagreed about one parse: `parse("der, y van")` gave family `van der` and initials `y. d. v.`, and it gives `y. v. d.` now. This is mechanisms.md#RENDER-HONORS-THE-PARSE's second shape, filed as its worked instance, so the design is not re-argued here — only what the measurement added to it. WHAT THE MEASUREMENT ADDED, and it reframes the issue from a consistency fix to a parity restoration. v1 had this feature as `CONSTANTS.middle_name_as_last` (`_config_shim` maps it to `Policy.middle_as_family`) and had NO general `name_order`, so only the two default-order policies have a v1 reference at all. Measured on the released 1.4.0 wheel over the 1094-name deduped corpus, from a script run outside the worktree that asserts `nameparser.__version__` first: at the default policy 1 name moves and agrees with 1.4.0 neither before nor after; under `middle_as_family` 71 move, of which **54 come into exact 1.4.0 agreement and NOT ONE moves away from it**. Worked pair: `Doe, Dr. John A.` gives `J. A. D.` at 1.4.0, gave `J. D. A.` here, and gives `J. A. D.` again; `Brundridge, Contessa A` likewise `C. A. B.` / `C. B. A.` / `C. A. B.`. The facade had been giving the folded answer all along — `HumanName.initials()` reads `first_list`/`middle_list`/`last_list`, and `_facade._list_for` prepends the carriers exactly as `_text_for` does — so the core view was simultaneously out of step with the field beside it, with the facade above it, and with 1.4.0 behind it. A view that stopped honoring a record had been reproducing a v1 bug v1 did not have. THE 17 THAT AGREE WITH 1.4.0 NEITHER WAY, recorded because the obvious explanation is wrong — and the FIRST REPLACEMENT for it was wrong the same way, corrected by this branch's docs review before merge. It is not that their parse already differs: only 2 of the 17 parse differently from 1.4.0 under this policy, `Smith, John V.` (#432's dotted numeral) and `남궁민수 지훈` (the unspaced-hangul surname split and the script order, rules.md#W1). The other 15 have SEVEN BYTE-IDENTICAL FIELDS at 1.4.0 and here — `der, y van` INCLUDED, which an earlier draft of this bullet filed with the two above by carrying over a measurement taken at the OTHER policy, where its parse does differ. Of those 15, 14 differ from 1.4.0 only in v1's initials GRANULARITY — v1 initials one group per `*_list` element, so a run its parse joined into one element (`Velasquez y Garcia`, `abdul salam`) yields `V G.` where 2.x, one token per word, yields `V. G.` — and strip the periods and spaces and all 14 come into 1.4.0's letter ORDER after this change, none having been in it before (`JVGQ` → `JQVG`). The 15th is `der, y van`, and it is worth naming rather than lumping: its fields match 1.4.0 exactly and its initials do not, because 1.4.0 contributes NOTHING for a family that is all particles (`van der` → `y.`) where 2.x contributes its words. That is R3's own adjudicated clause and decisions.md#R2's territory — R3 carries `"Juan van der" → initials="J. v. d."` against 1.4.0's `J.` — so this one name is a divergence this document has already decided in favor of, not an unexplained residue. So the parity claim is stronger than "54 of 71 and the rest unexplained": 68 of the 71 come into 1.4.0's ordering, 54 of them exactly, one more is R2's decided divergence, and 2 have a parse difference that predates this change. - BLAST RADIUS by policy, over the same corpus × three `name_order` values × `middle_as_family` off/on (6564 parses, 660 move). Default order: 1 / 71. FAMILY_FIRST: 1 / 294. FAMILY_FIRST_GIVEN_LAST: 1 / 292. STATE THE POPULATION BESIDE THE COUNT, per this file's own rule: the shape an order change can be observed in is a contributing folded token standing AFTER a contributing unfolded one of the same role, and at the default order the corpus holds exactly 1 such name without `middle_as_family` and 72 with it. So the movers are 1 of 1 and 71 of 72 — a near-total hit rate over a corpus that carries one instance of P6's half, not a blast radius near zero. (The one non-mover is `Chang, Andy C I`, where the folded `C` and the family `Chang` share a letter and the reorder is invisible in initials.) Nine corpus names carry the tag at all at the default order; the other eight have a single contributing family token, which makes the partition the identity. The `middle_as_family` half is where the reach is, which follows from the producers: O3 folds every middle word, while P6's attachment moves initials only where the attached run is READMITTED by the unjoined mark — a family that is all particles — so it reaches one corpus name in each order and the same one. Recompute by parsing the four `tools/differential/corpus*.jsonl` files deduped under each of the six policies and comparing `initials()` against a folded-first partition of `tokens_for(role)`. + BLAST RADIUS by policy, over the same corpus × three `name_order` values × `middle_as_family` off/on (6564 parses, 660 move). Default order: 1 / 71. FAMILY_FIRST: 1 / 294. FAMILY_FIRST_GIVEN_LAST: 1 / 292. STATE THE POPULATION BESIDE THE COUNT, per this file's own rule: the shape an order change can be observed in is a contributing folded token standing AFTER a contributing unfolded one of the same role, and at the default order the corpus holds exactly 1 such name without `middle_as_family` and 72 with it. So the movers are 1 of 1 and 71 of 72 — a near-total hit rate over a corpus that carries one instance of P6's half, not a blast radius near zero. (The one non-mover is `Chang, Andy C I`, where the folded `C` and the family `Chang` share a letter and the reorder is invisible in initials.) Nine corpus names carry the tag at all at the default order; the other eight have a single contributing family token, which makes the partition the identity. The `middle_as_family` half is where the reach is, which follows from the producers: O3 folds every middle word, while P6's attachment moves initials only where the attached run is READMITTED by the unjoined mark — a family that is all particles — so it reaches one corpus name in each order and the same one. RECOMPUTE, and the COMPARATOR is the half that has to be stated, because the obvious phrasing of it measures nothing: parse the four `tools/differential/corpus*.jsonl` files deduped under each of the six policies and compare `initials()` against THE PRE-CHANGE RENDERING — the same contributing tokens left in WRITTEN order. What this sentence said until 2026-08-30 was "against a folded-first partition of `tokens_for(role)`", which is what `initials()` now IS: that recipe compares the shipped implementation with itself and yields 0 differing parses, the inert-measurement shape (mechanisms.md's Verification shapes section) written into the very sentence meant to prevent it, and the only recompute path the repository offered for docs/release_log.rst's counts as well. CONTRIBUTING is the other half of the recipe and it comes FIRST: apply this view's own skip filter — outside the given group a token tagged `particle` or `conjunction` contributes nothing unless it also carries UNJOINED_TAG — and compare the two orders only over what survives it. Leave the filter out and every skipped word reads as a difference instead: 1546 rather than 660 against written order, and 1018 rather than 0 against the folded partition. Re-measured this way on 2026-08-30 and reproducing the counts above: 660 of 6564 parses, 1 / 71 at the default order, 1 / 294 and 1 / 292 under the two family-first orders. THE VERIFICATION LIMIT, and it is decisions.md#R4's, restated because a plausible reading of "one name moves at the default policy" is that the ledger owes it a rule. It does not, and could not: `tools/differential/compare.py` compares the seven role fields plus `_ambiguities`, and `initials()` is not among them — the same blindness recorded for `capitalized()` under R4. Measured rather than reasoned: the gate was run at all three baselines on the pre-change tree (exported with `git archive HEAD` into a scratch directory, so nothing in the working tree moved) and on the changed one, and the two outputs are byte-identical apart from the tree path and the baseline-worker temp path — 1094 corpus names, 229 / 194 / 102 intentional diffs, `unexplained: 0`, and 67 / 58 / 51 headings with identical per-heading counts, at 1.4.0 / 2.0.0 / 2.1.0. NO LEDGER RULE WAS ADDED, and none was needed. What stands in its place is `rules.md#R3`'s example line and `tests/v2/test_render.py`, both mutation-checked. WHY R3 GAINED A SENTENCE rather than leaning on R1. R1 already says folded family words "render before the rest of the family wherever they stood in the string" — but its subject is "Every FIELD", and `initials()` is not a field: R3 calls it "this view", and it takes a format spec and two separators rather than being read as an attribute. So R1 does not reach it by its own words, and #408 is precisely what leaving that to inference costs. R3's new clause states the order in R3's own voice, carries the O3 example line (`"Hassan, Mohamad Ahmad Ali"` under `middle_as_family` → `initials="M. A. A. H."`, reusing R1's own input so `corpus_rules.jsonl` is unchanged at 241 names), and R1 and R3 now name each other in `interacts:` — the field says which order, the view says it follows the field. P6's half gets no example line, for the same reason R1 carries none: the only input that witnesses it is `der, y van`, whose PARSE is itself contested, so pinning its initials normatively would pin a value another open question can move. It is pinned in the unit test instead, which is where today's-behavior pins belong. THE PARTITION IS PER ROLE, not scoped to FAMILY, and this is a deliberate choice rather than a copied line. Both producers re-role to FAMILY, so FAMILY is the only role a parse can put the tag on today and the GIVEN and MIDDLE arms are unreachable. They are there because `_text_for` partitions for every role it renders and the two views must not diverge again — the same uniformity `_post_rules`' own UNJOINED_TAG loop takes for its three roles, "for uniformity with the rule, not because all three are observable". A producer that ever folded into another part would otherwise reopen #408 there with nothing to fail. `test_initials_folds_in_every_role_it_renders` pins it from a hand-built name, since no input string can. Mutation-checked four ways — scoping the partition to FAMILY, and skipping it for GIVEN, for MIDDLE, for FAMILY — and each fails that test. It took THREE drafts of the fixture to make that true, and the two misses are the same miss one role apart. The first carried two GIVEN tokens and ONE MIDDLE: a one-element partition is the identity, so the MIDDLE arm was asserted and unpinned, and skipping the partition for MIDDLE passed the entire suite. The second fixed MIDDLE and carried NO FAMILY token at all, in a test named for every role it renders — so skipping the partition for FAMILY passed this test, and was caught only by its siblings and by R3's example line, neither of which is about per-role application. A zero-element group is the identity too, and reads even less like a gap than a one-element one, which is why the second miss survived a review that had just named the first. Two tokens per role is what closes it, and the general form is carried as mechanisms.md#TWO-ELEMENT-GROUPS rather than left here: a test written to pin a partition, a sort or a dedup needs at least two elements in every group it claims to cover, or the claim rides on a no-op. diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index b535d6e2..decf36f2 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -15,7 +15,7 @@ Problem shape. A later stage needs to refer to "that word." Contract statement. ## FOLDED_TAG — reorder at render time, not parse time -Problem shape. A rule wants words to RENDER in a different order than they sit in the string. Contract statement. Tokens never move: a rule that needs different rendering order tags the token, and the rendering views consult the tag — EVERY view that renders the affected role, ordering folded tokens first, not the field views alone. How it works. Reordering the token tuple would break span math and reintroduce the #100 family. Parse state stays in string order; only the view reorders (rule R1, rule R3's order clause, rule O3's render clause). The consumer list is where this mechanism fails, and it has failed there twice in opposite directions: a consumer that never read the tag (`initials()` walked written order through 2.0, 2.1 and 2.2 — #408, filed as an instance of RENDER-HONORS-THE-PARSE and fixed 2026-08-30), and a consumer that read it when it should not have (the revise strip below). Adding a producer is cheap; adding a view is where the roster has to be walked. Lives in. nameparser/_types.py (FOLDED_TAG, `_text_for` — every role it renders, not FAMILY alone), nameparser/_render.py (`initials`, the same partition per role), nameparser/_facade.py (`_list_for`, the v1 `*_list` views, which prepend the carriers the same way), nameparser/_pipeline/_post_rules.py (its two producers, O3's fold and P6's attachment, both of which re-role to FAMILY — so FAMILY is the only role a parse can put the tag on today, and the views partitioning every role is uniformity with this contract rather than reachable behavior), and one deliberate CONSUMER-side strip: Parser.revise removes the tag from harvested tokens (a revised value must not inherit fold ordering) — losing that strip is this mechanism's measured hazard, a family rendering "García Gabriel Márquez". Reach for it when. A new rule needs "X renders before Y" and you are tempted to swap tokens. Don't swap. Tag. +Problem shape. A rule wants words to RENDER in a different order than they sit in the string. Contract statement. Tokens never move: a rule that needs different rendering order tags the token, and the rendering views consult the tag — EVERY view that renders the affected role, ordering folded tokens first, not the field views alone. How it works. Reordering the token tuple would break span math and reintroduce the #100 family. Parse state stays in string order; only the view reorders (rule R1, rule R3's order clause, rule O3's render clause). The consumer list is where this mechanism fails, in two opposite directions — and only ONE of them has ever shipped, which is worth keeping straight because a roster is walked differently when it is guarding against a defect than when it is guarding against a possibility. SHIPPED: a consumer that never read the tag (`initials()` walked written order through 2.0, 2.1 and 2.2 — #408, filed as an instance of RENDER-HONORS-THE-PARSE and fixed 2026-08-30). HAZARD, never released: a consumer that reads the tag where it should not (the revise strip below), which arrived with `Parser.revise` in the same commit as the test that pins it, so no version has gone out without it and no issue, decision entry or test names a defect of that shape. Adding a producer is cheap; adding a view is where the roster has to be walked. Lives in. nameparser/_types.py (FOLDED_TAG, `_text_for` — every role it renders, not FAMILY alone), nameparser/_render.py (`initials`, the same partition per role), nameparser/_facade.py (`_list_for`, the v1 `*_list` views, which prepend the carriers the same way), nameparser/_pipeline/_post_rules.py (its two producers, O3's fold and P6's attachment, both of which re-role to FAMILY — so FAMILY is the only role a parse can put the tag on today, and the views partitioning every role is uniformity with this contract rather than reachable behavior), and nameparser/_parser.py (`Parser.revise`, the one deliberate CONSUMER-side strip: it removes the tag from the tokens it harvests, a revised value not being allowed to inherit fold ordering) — losing that strip is this mechanism's measured hazard, a family rendering "García Gabriel Márquez". A module path and not a method name, because this roster is what a reader walks when adding a view, and a bare `Parser.revise` sends them looking in the wrong file. Reach for it when. A new rule needs "X renders before Y" and you are tempted to swap tokens. Don't swap. Tag. ## VOCAB-TAGS — the vocabulary layer speaks once @@ -59,7 +59,7 @@ Problem shape. Two stages need the same answer about the same input, and the one ## RENDER-HONORS-THE-PARSE — the parse decides it, the views honor it -Problem shape. A render view needs a fact the parse already settled — whether a word is the conjunction or an initial, whether a particle is acting as one, which word renders first. Contract statement. The parse decides it; the render views honor those decisions and never re-evaluate them. Two directions break that, and each has been found here as a defect: a view RE-DERIVES the answer from the text, keeping its own copy of a pipeline predicate — that one shipped through 2.0 and 2.1 — or a view honors the record and then OVERRIDES it, readmitting what the decision excluded, which is filed and open here. How it works. Re-deriving fails because the two copies stop being the same question long before anyone notices they are two: `_cap_word` re-ran the conjunction-versus-initial decision from the word's spelling against a hand-maintained copy of the pipeline's `_INITIAL` pattern while classify had already answered it and recorded it on the token, and `_classify.py` asks `is_initial()`, the shape test ANDed with a script-repertoire test since #320, where `_render.py` asked the bare pattern, and asked it per WORD of a token's text rather than per token, so `juan e-f smith` repaired to `Juan e-F Smith` (#458). Overriding fails more directly, and this entry's instance of it is OPEN rather than closed: `initials()` honors the `conjunction` tag through `_SKIP_TAGS` and then readmits the token whenever it also carries UNJOINED_TAG, the mark of an all-particle part, where rules.md#R3 excludes a conjunction "even then" — so under a caller's vocabulary that puts a word in both sets the view readmits what the rule excluded, and under `Lexicon.default().add(particles={'y'})` — the lexicon those two readings need, and they hold under no other — `parse("Anh y Van")` initialed `A. y. V.` and `parse("Juan de y")` `J. d. y.` on 2026-08-29. Read those two values scoped to that lexicon or they are simply false: under the DEFAULT vocabulary the same strings give `A. V.` and `J.`, and the second is a rules.md#R3 example line the doc runner asserts. #461 is where that stands, and it is worth reading before re-fixing it: the narrowing was written, measured and BACKED OUT in the same PR, because honoring R3 there cost `initials()` its agreement with `family_base`, which reads that same `de y` as the base — this entry's other shape, arrived at from the render side, and the reason the question moved from the code to R3's clause (decisions.md carries the argument). What to carry away is the DIRECTION and not its verdict here: a view that honors a record and then readmits what the record excluded is overriding a decision it never took, whichever way this one settles. Case repair reads that same token and does NOT readmit it — `capitalized(force=True)` on `Anh y Van` gives `Anh y Van` under that same lexicon, R4 carrying the carve-out in its own words but ON R3's authority — its text reads "being no name word in any part — the carve-out R3 states for initials", so a change to R3's clause reaches R4's TEXT. What does NOT follow, though an earlier wording of this sentence asserted it, is that the two stand or fall together in BEHAVIOR: they have already come apart, over the 25 corpus names carrying a conjunction in the GIVEN group — `parse("john and jane smith").capitalized()` keeps `and` lowercase, so R4's carve-out holds there, while `.initials()` gives `j. a. j. s.`, so R3's does not (decisions.md#R2 carries that population, and rules.md#R3 now says so in its own words). The dependency is textual, and only textual — so the two views disagree today about that token exactly as they did before #461 and the backout restores that disagreement knowingly; only the FORCED call witnesses the repair half, R5's gate refusing a mixed-case name before any of this is consulted. This is the CONSUMER-side rule over the producer-side entries — VOCAB-TAGS records what the vocabulary knew, MARK-DONT-STRIP what a stage decided about it, FOLDED_TAG what order to render in — and a view reads what they recorded, whichever kind it is. It sits where ONE-PREDICATE-PER-QUESTION's stated limit leaves off: where two live sites need one answer they share a predicate, but a render view always comes AFTER the decider, so the answer is recorded rather than shared — on the TOKEN, which is the views' equivalent of that entry's `ParseState.order`, no view being able to see a ParseState at all. Known limit, and the half most easily got wrong next: a token the parse never saw carries no decision to honor, so a view falls back to the vocabulary — the tell is `UNCLASSIFIED_TAG`, which `ParsedName.replace()` stamps when it splices raw text into a field and the facade's v1 pickle load stamps when it rebuilds a name from `*_list` strings. It is NOT untaggedness, since an ordinary parsed name word carries no tags either; and it is NOT `span is None`, which was tried and is wrong in the other direction — span-less means SYNTHETIC, and `Parser.revise()` builds span-less tokens from a full sub-parse whose tags it keeps on purpose, so the span reading overrode exactly the tags `revise()` exists to preserve (`revise(middle='e-f')` repaired to `e-F` where the parse gave `E-F`). A hand-built span-less token is unmarked and therefore classified, which is the same tag-driven default every other view applies. A view can only fall back if it is HANDED a vocabulary, and exactly one is: `capitalized(lexicon=...)`. `family_base` and `family_particles` are properties on ParsedName, whose fields are original/tokens/ambiguities and nothing else, so a spliced field empties the particles view and leaves the base the whole field, with `Parser.revise()` the crossing there too (docs/usage.rst says so where it documents the degradation). `initials()` is the near miss and the instructive one: it is a METHOD, so it looks like it could ask, but its signature is `(spec, delimiter, separator)` and carries no lexicon — a fallback there was written and dropped because it had to GUESS `Lexicon.default()`, and the guess erased a whole field under a caller's own vocabulary (decisions.md#R4). `capitalized()` guesses nothing: it reads the lexicon it was handed, and only defaults to `Lexicon.default()` when the caller passes none, which is the documented meaning of omitting the argument rather than a fallback. Within the one view that can fall back, the fallback is drawn per QUESTION (rules.md#R4's Accepted clause). Whether a word is the conjunction or an initial is a property of the word, which a vocabulary answers alone, so case repair asks it. Whether a part is wholly particles is a property of the whole PART, which the pipeline answers once and records as UNJOINED_TAG. What `_cap_word` cannot do is RE-DERIVE that answer where no word of the part carries a tag — it is handed the whole token's tags and gates the particle conjunct on UNJOINED_TAG, so it is not blind to the part, it simply has no evidence to reconstruct one from — so repair leaves that half to plain particle treatment and rules.md#R4's Accepted boundary records the consequence — a spliced field is not repaired as a parsed one is, with `Parser.revise()` as the supported crossing. A fallback is right only while it answers as the pipeline would, and THAT is held by hand rather than mechanically: test_regex_sync pins the two `_INITIAL` copies to each other and to config, while the repertoire half of the pipeline's predicate (#320) is deliberately not carried across, layering forbidding the import. What that divergence can reach is now nothing observable: it needs a caller-added conjunction written initial-SHAPED in a script that has no initials (`씨.`), and case repair is the fallback's only reader, so the two paths differ by `lower()` versus `capitalize()` over a caseless script — the same string either way. `initials()` used to be the reader that could witness it, and no longer falls back at all. A second limit, recorded rather than closed: `_cap_word`'s PARTICLE conjunct still keys on the lexicon handed to the view rather than on the `particle` tag, so a repair run with a lexicon other than the parse's re-decides a word the parse already read — a name parsed under the default vocabulary, where `parse('juan smith vega')` reads `vega` as the family, repairs to `Juan Smith vega` when `capitalized()` is handed `Lexicon.default().add(particles={'vega'})` instead — the divergence needs the two lexicons to differ, and repairing under the parse's own lexicon gives `Juan Smith Vega`. Making it read the tag moves a boundary rules.md#R4 states in prose, so it is a separate decision and not a cleanup (decisions.md#R4, "NOT DONE"). Lives in. nameparser/_render.py (`capitalized`/`_cap_word`, `_reads_as_conjunction`, and `initials`, which honors tags and never falls back) and nameparser/_types.py (`_text_for` and `UNCLASSIFIED_TAG`, with the `ParsedName.replace()` producer beside it) and nameparser/_facade.py (the v1 pickle load, the SECOND producer of that mark — it is named in this list because a change that follows the list into `_types.py` alone leaves it behind, which is the site test_a_restored_pickle_keeps_v1_conjunction_repair exists to protect), reading what nameparser/_pipeline/ recorded — the mark those views read is recomputed producer-side in `_remarked`, which is deliberately silent about text nobody classified and is right to be. Reach for it when. A view is about to consult a Lexicon, a regex or an exception list about a word the parse already saw — or a view and a field disagree about the same parse. #408 was that second shape, and is CLOSED (2026-08-30): `initials()` walked tokens in written order where the family field applies FOLDED_TAG's ordering, so `parse("der, y van")` gave family `van der` and initials `y. d. v.` on 2026-08-29 and gives `y. v. d.` now, the view reading the tag as `_text_for` does. Worth keeping as the worked instance of the shape rather than deleting with the fix, and worth two notes on how it read once measured. The disagreement was not a judgment call anyone had taken: the FACADE already ordered folded-first through its own `*_list` views, so the core view was out of step with the field, with v1, and with the facade at once, and nothing in 6125 tests touched it. And the fix was strictly parity-RESTORING, which is not what this entry's other instances have been -- over the 1094-name corpus at the default order, 71 names move under `middle_as_family`, of which 54 return to 1.4.0's answer and none leaves it (decisions.md#R3 carries the rest of the measurement). A view that stopped honoring a record had been quietly reproducing a v1 bug that v1 did not have. +Problem shape. A render view needs a fact the parse already settled — whether a word is the conjunction or an initial, whether a particle is acting as one, which word renders first. Contract statement. The parse decides it; the render views honor those decisions and never re-evaluate them. Two directions break that, and each has been found here as a defect: a view RE-DERIVES the answer from the text, keeping its own copy of a pipeline predicate — that one shipped through 2.0 and 2.1 — or a view honors the record and then OVERRIDES it, readmitting what the decision excluded, which is filed and open here. How it works. Re-deriving fails because the two copies stop being the same question long before anyone notices they are two: `_cap_word` re-ran the conjunction-versus-initial decision from the word's spelling against a hand-maintained copy of the pipeline's `_INITIAL` pattern while classify had already answered it and recorded it on the token, and `_classify.py` asks `is_initial()`, the shape test ANDed with a script-repertoire test since #320, where `_render.py` asked the bare pattern, and asked it per WORD of a token's text rather than per token, so `juan e-f smith` repaired to `Juan e-F Smith` (#458). Overriding fails more directly, and this entry's instance of it is OPEN rather than closed: `initials()` honors the `conjunction` tag through `_SKIP_TAGS` and then readmits the token whenever it also carries UNJOINED_TAG, the mark of an all-particle part, where rules.md#R3 excludes a conjunction "even then" — so under a caller's vocabulary that puts a word in both sets the view readmits what the rule excluded, and under `Lexicon.default().add(particles={'y'})` — the lexicon those two readings need, and they hold under no other — `parse("Anh y Van")` initialed `A. y. V.` and `parse("Juan de y")` `J. d. y.` on 2026-08-29. Read those two values scoped to that lexicon or they are simply false: under the DEFAULT vocabulary the same strings give `A. V.` and `J.`, and the second is a rules.md#R3 example line the doc runner asserts. #461 is where that stands, and it is worth reading before re-fixing it: the narrowing was written, measured and BACKED OUT in the same PR, because honoring R3 there cost `initials()` its agreement with `family_base`, which reads that same `de y` as the base — this entry's other shape, arrived at from the render side, and the reason the question moved from the code to R3's clause (decisions.md carries the argument). What to carry away is the DIRECTION and not its verdict here: a view that honors a record and then readmits what the record excluded is overriding a decision it never took, whichever way this one settles. Case repair reads that same token and does NOT readmit it — `capitalized(force=True)` on `Anh y Van` gives `Anh y Van` under that same lexicon, R4 carrying the carve-out in its own words but ON R3's authority — its text reads "being no name word in any part — the carve-out R3 states for initials", so a change to R3's clause reaches R4's TEXT. What does NOT follow, though an earlier wording of this sentence asserted it, is that the two stand or fall together in BEHAVIOR: they have already come apart, over the 25 corpus names carrying a conjunction in the GIVEN group — `parse("john and jane smith").capitalized()` keeps `and` lowercase, so R4's carve-out holds there, while `.initials()` gives `j. a. j. s.`, so R3's does not (decisions.md#R2 carries that population, and rules.md#R3 now says so in its own words). The dependency is textual, and only textual — so the two views disagree today about that token exactly as they did before #461 and the backout restores that disagreement knowingly; only the FORCED call witnesses the repair half, R5's gate refusing a mixed-case name before any of this is consulted. This is the CONSUMER-side rule over the producer-side entries — VOCAB-TAGS records what the vocabulary knew, MARK-DONT-STRIP what a stage decided about it, FOLDED_TAG what order to render in — and a view reads what they recorded, whichever kind it is. It sits where ONE-PREDICATE-PER-QUESTION's stated limit leaves off: where two live sites need one answer they share a predicate, but a render view always comes AFTER the decider, so the answer is recorded rather than shared — on the TOKEN, which is the views' equivalent of that entry's `ParseState.order`, no view being able to see a ParseState at all. Known limit, and the half most easily got wrong next: a token the parse never saw carries no decision to honor, so a view falls back to the vocabulary — the tell is `UNCLASSIFIED_TAG`, which `ParsedName.replace()` stamps when it splices raw text into a field and the facade's v1 pickle load stamps when it rebuilds a name from `*_list` strings. It is NOT untaggedness, since an ordinary parsed name word carries no tags either; and it is NOT `span is None`, which was tried and is wrong in the other direction — span-less means SYNTHETIC, and `Parser.revise()` builds span-less tokens from a full sub-parse whose tags it keeps on purpose, so the span reading overrode exactly the tags `revise()` exists to preserve (`revise(middle='e-f')` repaired to `e-F` where the parse gave `E-F`). A hand-built span-less token is unmarked and therefore classified, which is the same tag-driven default every other view applies. A view can only fall back if it is HANDED a vocabulary, and exactly one is: `capitalized(lexicon=...)`. `family_base` and `family_particles` are properties on ParsedName, whose fields are original/tokens/ambiguities and nothing else, so a spliced field empties the particles view and leaves the base the whole field, with `Parser.revise()` the crossing there too (docs/usage.rst says so where it documents the degradation). `initials()` is the near miss and the instructive one: it is a METHOD, so it looks like it could ask, but its signature is `(spec, delimiter, separator)` and carries no lexicon — a fallback there was written and dropped because it had to GUESS `Lexicon.default()`, and the guess erased a whole field under a caller's own vocabulary (decisions.md#R4). `capitalized()` guesses nothing: it reads the lexicon it was handed, and only defaults to `Lexicon.default()` when the caller passes none, which is the documented meaning of omitting the argument rather than a fallback. Within the one view that can fall back, the fallback is drawn per QUESTION (rules.md#R4's Accepted clause). Whether a word is the conjunction or an initial is a property of the word, which a vocabulary answers alone, so case repair asks it. Whether a part is wholly particles is a property of the whole PART, which the pipeline answers once and records as UNJOINED_TAG. What `_cap_word` cannot do is RE-DERIVE that answer where no word of the part carries a tag — it is handed the whole token's tags and gates the particle conjunct on UNJOINED_TAG, so it is not blind to the part, it simply has no evidence to reconstruct one from — so repair leaves that half to plain particle treatment and rules.md#R4's Accepted boundary records the consequence — a spliced field is not repaired as a parsed one is, with `Parser.revise()` as the supported crossing. A fallback is right only while it answers as the pipeline would, and THAT is held by hand rather than mechanically: test_regex_sync pins the two `_INITIAL` copies to each other and to config, while the repertoire half of the pipeline's predicate (#320) is deliberately not carried across, layering forbidding the import. What that divergence can reach is now nothing observable: it needs a caller-added conjunction written initial-SHAPED in a script that has no initials (`씨.`), and case repair is the fallback's only reader, so the two paths differ by `lower()` versus `capitalize()` over a caseless script — the same string either way. `initials()` used to be the reader that could witness it, and no longer falls back at all. A second limit, recorded rather than closed: `_cap_word`'s PARTICLE conjunct still keys on the lexicon handed to the view rather than on the `particle` tag, so a repair run with a lexicon other than the parse's re-decides a word the parse already read — a name parsed under the default vocabulary, where `parse('juan smith vega')` reads `vega` as the family, repairs to `Juan Smith vega` when `capitalized()` is handed `Lexicon.default().add(particles={'vega'})` instead — the divergence needs the two lexicons to differ, and repairing under the parse's own lexicon gives `Juan Smith Vega`. Making it read the tag moves a boundary rules.md#R4 states in prose, so it is a separate decision and not a cleanup (decisions.md#R4, "NOT DONE"). Lives in. nameparser/_render.py (`capitalized`/`_cap_word`, `_reads_as_conjunction`, and `initials`, which honors tags and never falls back) and nameparser/_types.py (`_text_for` and `UNCLASSIFIED_TAG`, with the `ParsedName.replace()` producer beside it) and nameparser/_facade.py (the v1 pickle load, the SECOND producer of that mark — it is named in this list because a change that follows the list into `_types.py` alone leaves it behind, which is the site test_a_restored_pickle_keeps_v1_conjunction_repair exists to protect), reading what nameparser/_pipeline/ recorded — the mark those views read is recomputed producer-side in `_remarked`, which is deliberately silent about text nobody classified and is right to be. Reach for it when. A view is about to consult a Lexicon, a regex or an exception list about a word the parse already saw — or a view and a field disagree about the same parse. #408 was that second shape, and is CLOSED (2026-08-30): `initials()` walked tokens in written order where the family field applies FOLDED_TAG's ordering, so `parse("der, y van")` gave family `van der` and initials `y. d. v.` on 2026-08-29 and gives `y. v. d.` now, the view reading the tag as `_text_for` does. Worth keeping as the worked instance of the shape rather than deleting with the fix, and worth two notes on how it read once measured. The disagreement was not a judgment call anyone had taken: the FACADE already ordered folded-first through its own `*_list` views, so the core view was out of step with the field, with v1, and with the facade at once, and nothing in 6125 tests touched it. And where the change has a v1 reference at all it RESTORES rather than deviates, which is not what this entry's other instances have been -- but read that SCOPED to the population it was measured over, because an unscoped version of this sentence stood here until 2026-08-30 and overstated in both directions. Only the two DEFAULT-ORDER policies have a v1 reference: v1 had `middle_name_as_last` and no general `name_order`, so 588 of the 660 moving parses -- the two family-first orders -- restore nothing and break nothing, there being no v1 answer to come into or leave (decisions.md#R3 says the same and carries the rest of the measurement). Where the reference does exist the claim is exact and worth keeping: over the 1094-name corpus at the default order, 71 names move under `middle_as_family`, of which 54 return to 1.4.0's answer and none leaves it. And it is a claim about THOSE 71 rather than about every name the fix touches -- this entry's own lead example is the counterexample, `parse("der, y van")` giving `y. d. v.` before and `y. v. d.` after where 1.4.0 gives `y.`, v1 contributing nothing at all for a family that is all particles (rules.md#R2's territory, and a divergence decisions.md#R2 has already decided in favor of). A view that stopped honoring a record had been quietly reproducing a v1 bug that v1 did not have. ## CLAUSE-CONTENT-OVERRULES-DELIMITER — content wins diff --git a/docs/design/rules.md b/docs/design/rules.md index 50cb7abe..02347166 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -1118,8 +1118,11 @@ R3. Rationale: initials abbreviate the person's name words; titles, except folded family words, which initial before the rest of the family exactly as they render before it (R1). Stated here rather than left to R1, whose subject is every FIELD and which this view - is not: initials abbreviate a field, so the two read one parse - and never disagree about it. + is not: what the view takes from the field is the ORDER it reads, + and about order the two never disagree. Membership is the first + question above and they can differ there — the family field of + the first example line below is de la Vega, where the initials + take its base — so this clause settles order and nothing else. "Dr. Juan Q. Xavier de la Vega III" → initials="J. Q. X. V." "Anh Do" → initials="A. D." "Nguyen, Van Le" → initials="V. L. N." diff --git a/docs/release_log.rst b/docs/release_log.rst index bbf22794..7a22ec8d 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -69,7 +69,7 @@ Release Log - Change case repair to read the parser's own ``conjunction`` tag instead of re-deciding, from the word's spelling, whether a word is a conjunction or an initial. The parse answers that question already -- ``"Scott E. Werner"`` reads ``E.`` as an initial rather than the Italian conjunction -- and the other views honor the answer; case repair asked again, with a shape test applied to each word of a token's text rather than to the token. Two spellings of one name disagreed because of it: ``"juan e-f smith"`` capitalized to ``Juan e-F Smith`` while ``"JUAN E-F SMITH"`` gave ``Juan E-F Smith``; both give ``Juan E-F Smith`` now, ``e-f`` being a middle name and no conjunction of the parse's reading. A conjunction written as a word of its own is untouched, and so is the one-letter carve-out where it applies -- ``"juan y garcia"`` still repairs to ``Juan y Garcia``, ``"JUAN Y GARCIA"`` still to ``Juan Y Garcia``. A field assigned after the parse is unaffected: its text was never classified, so there is no reading to honor and repair asks the vocabulary, applying v1's own predicate the way every earlier version applied it everywhere -- ``h.last = "velasquez y garcia"`` still repairs to ``Velasquez y Garcia`` and ``h.middle = "e."`` to ``E.``. That is the predicate over TODAY's vocabulary, which is narrower than parity with 1.4.0 and the difference is real: ``h.last = "хосе и мария сантос"`` gives ``Хосе И Мария Сантос`` on 1.4.0 and ``Хосе и Мария Сантос`` here, because the Cyrillic ``и`` is a 2.x conjunction and was not a 1.4.0 one. What decides which path a token takes is a mark the assignment leaves, not the absence of a span: a value revised through ``Parser.revise()`` is classified by a sub-parse and keeps its tags, so it repairs as the parse does. One reading does change for hand-built ``Token``\ s in the 2.0 API: an untagged token whose text is conjunction vocabulary is now an ordinary name word and capitalizes, where 2.1 lowercased it -- tags are what the views read, and a hand-built token that carries none is a token with nothing to declare. Case repair is not one of the seven role fields the differential harness compares, so no gate run can see this change either way and none of its counts move; measured directly instead, no name of the 1094-name differential corpus moves under ``capitalized()`` or ``capitalized(force=True)``, its uppercased and lowercased spellings included -- 6564 name/spelling/lexicon rows and 13128 calls (closes #458) - - Fix ``initials()`` reading a name in a different order than the fields of the same name. Two rules fold words into the family name and render them before the rest of it -- ``Policy(middle_as_family=True)``, which sends every middle word to the family, and the tussenvoegsel attachment after a family comma -- and both do it by marking the words rather than moving them, since a parsed word keeps the position it was written at. The ``family`` field reads that mark and ``initials()`` did not, so one parse gave two orders: ``parse("der, y van")`` gave family ``van der`` and initials ``y. d. v.``, and now gives ``y. v. d.``. This RESTORES v1: ``middle_name_as_last`` is v1's spelling of the same option, so most of what moves has a 1.4.0 answer to be measured against, and measured over the 1094-name differential corpus at the default name order, 71 names move under that option, 54 of them back to exactly what 1.4.0 returns and not one of them away from it -- ``"Doe, Dr. John A."`` gives ``J. A. D.`` again where 2.0 through 2.2 gave ``J. D. A.``, and ``"Brundridge, Contessa A"`` gives ``C. A. B.`` where they gave ``C. B. A.``. Of the 17 that match 1.4.0 neither before nor after, 14 now agree with it on the ORDER and differ only in how v1 grouped initials -- 1.4.0 gives one initial per element of its own ``last_list``, so a conjunction-joined surname yields ``V G.`` where the 2.x view, one initial per word, yields ``V. G.``; one more is ``"der, y van"``, whose family is nothing but particles, where 1.4.0 contributes no initial at all and 2.x contributes its words, a difference this release does not touch; and the remaining two parse differently from 1.4.0 for reasons that predate this fix. Without the option, one corpus name moves, the ``"der, y van"`` above -- and it is one of one, since it is the only name in the corpus whose family holds two contributing words with a folded one behind the other, the shape an order change can be seen in at all. ``HumanName.initials()`` was already right and is unchanged, measured: no name of the 1094-name corpus moves through the facade, with the option or without it. It reads ``first_list``/``middle_list``/``last_list``, which prepend the folded words as v1 did, so it is the 2.0 API's ``ParsedName.initials()`` that was out of step -- with the field beside it, with the facade, and with 1.4.0 at once. ``initials()`` is not one of the seven role fields the differential harness compares, so no gate run can see this change: run at all three baselines before and after, the output is identical to the byte -- 1094 corpus names, 229 / 194 / 102 intentional diffs and ``unexplained: 0`` at 1.4.0 / 2.0.0 / 2.1.0, with every per-heading count unchanged. The ``rules.md#R3`` example line and ``tests/v2/test_render.py`` are what pin it (closes #408) + - Fix ``initials()`` reading a name in a different order than the fields of the same name. Two rules fold words into the family name and render them before the rest of it -- ``Policy(middle_as_family=True)``, which sends every middle word to the family, and the tussenvoegsel attachment after a family comma -- and both do it by marking the words rather than moving them, since a parsed word keeps the position it was written at. The ``family`` field reads that mark and ``initials()`` did not, so one parse gave two orders: ``parse("der, y van")`` gave family ``van der`` and initials ``y. d. v.``, and now gives ``y. v. d.``. This RESTORES v1: ``middle_name_as_last`` is v1's spelling of the same option, so most of what moves has a 1.4.0 answer to be measured against, and measured over the 1094-name differential corpus at the default name order, 71 names move under that option, 54 of them back to exactly what 1.4.0 returns and not one of them away from it -- ``"Doe, Dr. John A."`` gives ``J. A. D.`` again where 2.0 through 2.2 gave ``J. D. A.``, and ``"Brundridge, Contessa A"`` gives ``C. A. B.`` where they gave ``C. B. A.``. Of the 17 that match 1.4.0 neither before nor after, 14 now agree with it on the ORDER and differ only in how v1 grouped initials -- 1.4.0 gives one initial per element of its own ``last_list``, so a conjunction-joined surname yields ``V G.`` where the 2.x view, one initial per word, yields ``V. G.``; one more is ``"der, y van"``, whose family is nothing but particles, where 1.4.0 contributes no initial at all and 2.x contributes its words, a difference this release does not touch; and the remaining two parse differently from 1.4.0 for reasons that predate this fix. Without the option, one corpus name moves, the ``"der, y van"`` above -- and it is one of one, since it is the only name in the corpus whose family holds two contributing words with a folded one behind the other, the shape an order change can be seen in at all. ``HumanName.initials()`` was already right and is unchanged, measured: no name of the 1094-name corpus moves through the facade, with the option or without it. It reads ``first_list``/``middle_list``/``last_list``, which prepend the folded words as v1 did, so it is the 2.0 API's ``ParsedName.initials()`` that was out of step -- with the field beside it, with the facade, and with 1.4.0 at once. ``initials()`` is not one of the seven role fields the differential harness compares, so no gate run can see this change: run at all three baselines before and after, the output is identical to the byte -- 1094 corpus names, 229 / 194 / 102 intentional diffs and ``unexplained: 0`` at 1.4.0 / 2.0.0 / 2.1.0, with every per-heading count unchanged. So where the counts in this bullet come from has to be said, the gate's classified summary not being able to supply them and the ``rules.md#R3`` example line witnessing the order without counting anything: what MOVES is recomputed by the recipe in the ``R3`` entry of ``docs/design/decisions.md``, which compares this view against the pre-change rendering over these same four corpora and reproduces the 71 and the one-of-one; the 1.4.0 comparisons and the facade sweep are dated snapshots rather than re-derivable ones, measured 2026-08-30 against the released 1.4.0 wheel and against the pre-change tree, which nothing in the repository re-runs. The ``rules.md#R3`` example line and ``tests/v2/test_render.py`` are what pin the behavior (closes #408) - Fix a tussenvoegsel attached to the family name after a comma deciding a genuinely uncertain reading and reporting nothing. ``"Van Johnson"`` reports a ``PARTICLE_OR_GIVEN`` ambiguity -- ``Van`` is a Dutch particle and a Vietnamese given name, and the parser has to pick one -- while ``"Nguyen, Thi Van"`` picked the same word the same way, silently, and lost the given name doing it. The attachment now reports the fork it decides, in the kind that names the reading it declined. A particle that could be an ordinary name reports ``PARTICLE_OR_GIVEN``: ``"Nguyen, Thi Van"``, ``"Berg, Jan van der"`` and ``"Vega, Juan de la"`` each gain one, the ``detail`` naming the ambiguous word. A particle the parser had already read as a post-nominal reports ``SUFFIX_OR_NAME`` instead, because the credential reading is what the attachment overrode: ``"Berg, Jan vd"`` gains one, ``vd`` being read as *van der* rather than as the Volunteer Decoration. Which kind you get follows the reading that was overridden rather than the word's vocabulary, so ``"Berg, Jan do"`` reports ``PARTICLE_OR_GIVEN`` even though ``do`` is a postnominal too -- it was already being read as a name word, so no credential reading was overridden. A particle where nothing was overridden reports nothing at all: ``"Jong, Piet de"`` and ``"Jong, Anke de"`` are unchanged, ``de`` being no name in any reading and no postnominal either, and so is ``"Berg, Jan de vd"``, whose run was read as name words whole. Worth knowing before you filter on this: ``"Beethoven, Ludwig van"`` -- the textbook Dutch listing, read exactly right -- now carries a report too. It is the same string shape over the same vocabulary as ``"Nguyen, Thi Van"``, and nothing in the input separates them, so a report on one is a report on both. This adds the report and nothing else: every field these names parse to is exactly what the tussenvoegsel fix below already gave them, and ``ambiguities`` is the only value that grows. Seven differential corpus names gain a kind against the 2.0.0 and 2.1.0 baselines, seven of the nine the tussenvoegsel fix below already moved -- five of ``fix(#379)``'s seven and both of ``fix(#380)``'s two, now carrying ``_ambiguities`` in their diff as well; the two left out are the pair named unchanged above, ``Jong, Piet de`` and ``Jong, Anke de`` -- and none against 1.4.0, which had no ambiguity reporting at all (closes #405) From ab5af8eb1bf161ba9107c281a6c98280933e5060 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 30 Aug 2026 02:53:02 -0700 Subject: [PATCH 12/12] docs(agents): admit a recompute recipe as release-log provenance The rule named two sources for a quantified release bullet: the differential gate's classified summary, or verification against a rules.md example. Neither can reach a claim about a render view. The gate compares the seven role fields and _ambiguities, so a change to initials() or capitalized() leaves its output identical to the byte, and an example line witnesses one output without counting anything. #408's bullet needed counts of exactly that kind, so it named a third source in prose -- a recompute recipe kept with decisions.md#R3 -- with no rule admitting one. Admit it, and carry the failure that recipe hit: its first draft compared initials() against a folded-first partition, which is what initials() now IS, so it reproduced 0 against a claimed 660 while being the only stated provenance for the number. Naming the comparator, and running the recipe as written before shipping the bullet, are what stop that. Cross-version numbers stay dated snapshots under Counting claims. --- AGENTS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index 1236d436..5027712a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ Three committed contributor docs carry the parser's normative rules and their re **Counting claims.** A bare count in prose is either an assertion or a liability, keyed by who observes its staleness: asserted counts (a test holds the number) fail CI at change time — the useful kind; dated snapshots ("51 sites at spec time") cannot go stale; standing present-tense prose counts are the forbidden class — promote to an assertion, add a date, or state the invariant and let a test count. After changing how many times something runs, sweep for counts, not for the thing's name. -**Release-log claims.** Quantified or universal behavior claims in release bullets must come from the differential gate's classified summary or be verified against rules.md examples, never written from memory. Per-rule ledger toml comments asserting PARSER behavior cite rule IDs under the excerpt discipline; free prose is for ledger mechanics only (owned by tools/differential/README.md). +**Release-log claims.** Quantified or universal behavior claims in release bullets must come from the differential gate's classified summary, be verified against rules.md examples, or -- for a view the gate cannot see -- carry a recompute recipe stored with the design entry the bullet cites; never write one from memory. The gate compares the seven role fields and `_ambiguities`, so `capitalized()`, `initials()` and any future render view are invisible to it (decisions.md#R4, #R3) and the first two sources cannot reach a claim about one: a gate run is byte-identical across the change, and an example line witnesses an output without counting anything. A recipe names the corpus files, the policy sweep, and -- the part that is easy to omit and fatal -- THE COMPARATOR, which must be something the shipped tree is not: #408's first recipe said to compare `initials()` against a folded-first partition, which is what `initials()` now IS, so it reproduced 0 where the bullet claimed 660 and was the only stated provenance for the number. Run the recipe as written before shipping the bullet. Cross-version numbers (a released wheel, the pre-change tree) are dated snapshots under Counting claims, since nothing in the repository re-runs them. Per-rule ledger toml comments asserting PARSER behavior cite rule IDs under the excerpt discipline; free prose is for ledger mechanics only (owned by tools/differential/README.md). **Working on docs/design/ has its own AGENTS.md.** `docs/design/AGENTS.md` carries the landing-a-design distillation checklist, the primary-source review rule, the dated-count convention, and the ten review axes. Claude Code loads it automatically when a session reads or edits anything under docs/design/; if your tool does not do nested discovery, read it yourself before touching those files or reviewing a change to them.