From 8e94162c758ed13c9e4e3c49e58a8a9bffa3a5a9 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sat, 29 Aug 2026 10:44:21 -0700 Subject: [PATCH 1/6] docs(design): extract R5, the case-repair gate capitalized() acts only on a name written entirely in one case; a mixed-case name is kept as written, and whether its casing is right does not enter into the decision. rules.md never said so, and R4's statement described the narrower 'an already-correct name comes back unchanged'. R4's boundary did not pin what it appeared to: 'Juan McDonald' passes under force=True as well, so it distinguished the gate from idempotence not at all. R5's witness is 'Shirley Maclaine' -- mixed-case and wrong, and kept -- and its boundary sits on the forced row, where R5's effect is absent rather than present. An earlier draft said the name is kept 'whether or not its casing is correct'. That promises a branch nothing can witness: for a mixed-case name repair would not alter, kept and repaired are the same string, so no example separates a parser with the gate from one without it. Two rows were tried and withdrawn -- one inert, one redundant with the row above it -- before the statement was reworded to claim only what is observable. decisions.md records the arc. No parser change. The two halves of the rule have different ages: the refusal predates the git history (45a1539, 2011-02-03, the initial svn import, already opens capitalize() with the guard), and bf1e0a5 (2016-06-02, shipped in 0.4.0) added the force override around it, so the rule as stated holds from 0.4.0 on. 1.4.0 is where it was measured -- the last v1 release -- not where it came from. The override has two routes, so the statement names neither: besides the per-call argument, the facade honors the v1 constant force_mixed_case_capitalization, and with it set a bare capitalize() repairs mixed case. decisions.md carries the mechanism and a 3.0 removal-candidate line; rules.md stays implementation-free. capitalized_forced is a new example pseudo-field: force is a per-call argument, not a Policy, so it cannot ride the annotation slot. decisions.md also records the debt this leaves. R4 keeps one falsehood -- 'Juan Mcdonald' is refused despite the vocabulary-exceptions promise -- and one ambiguity, 'already-correct' meaning idempotence to the rule and bearer-correct to the reader. The second wants disambiguation, not a carve-out for deliberately single-cased names, which R5's rationale declines. R4 is amended in a later commit; this one touches it only on its pointer line. Three mechanical riders the doc guards and conventions require: _render.py gains a rules.md#R5 citation comment on the gate line (no code change -- implemented: is checked against the modules that cite the rule); corpus_rules.jsonl gains 'Shirley Maclaine', 234 -> 235, with compare.py's _CORPUS_FLOORS comment moved to match and the floor itself untouched. That name is already in corpus.jsonl and the harness dedupes across corpora, so the differential population holds at 1090 and no ledger rule is needed: the gate exits 0 with 0 unexplained at the 1.4.0, 2.0.0 and 2.1.0 baselines. Co-Authored-By: Claude Opus 5 --- docs/design/decisions.md | 10 ++++++++++ docs/design/rules.md | 19 ++++++++++++++++++- nameparser/_render.py | 2 ++ tests/v2/rules_doc.py | 2 +- tests/v2/test_rules_doc.py | 4 ++++ tools/differential/compare.py | 2 +- tools/differential/corpus_rules.jsonl | 1 + 7 files changed, 37 insertions(+), 3 deletions(-) diff --git a/docs/design/decisions.md b/docs/design/decisions.md index c5440b58..1e3720af 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -712,6 +712,15 @@ Declined: - 2026-08-18 — the GROUPING half of #404: a particle run that joins nothing does not chain, so "Jong van der" would split into middle 'van' plus family 'der'. Measured and rejected, though NOT for the reason first recorded here. The first draft said the split makes the family "no longer all-particle so the base fix stops firing" — false, and `der` and `la` are both shipped particles, so a family of either IS all-particle and the rule fires on it ("Juan Smith der" gives base 'der'). What the split actually costs is the SCOPE of the base and a stray particle relocated: grouping can decline to merge but cannot keep the words apart, because roles re-assemble them and two adjacent same-role pieces are one part at the field level. "Juan Smith van der" becomes middle 'Smith van', family 'der' — a base of 'der' rather than 'van der', and a middle name nobody wrote. Keeping the run whole in one part is what gives the base its full extent. The split reading needs the leftover distribution to know these are separate units, which is mechanisms.md#UNIT-PARTITION's problem. +### R5 — the case-repair gate + +- 2026-08-29 (#407 arc) — EXTRACTION, not a decision: the parser is untouched. The two halves of R5 have separate provenance, and conflating them is easy enough that the first draft of this entry did. The REFUSAL — repair skips any name already carrying more than one case — is older than the git history: `git log -S "name == name.upper() or name == name.lower()" --reverse` bottoms out at 45a1539 (2011-02-03), the initial import from svn, where `capitalize()` already opens with that guard and a bare `return`. (A path-filtered search answers 280895b instead, the same-day commit that moved the module into `nameparser/`; the code did not change there.) The OVERRIDE is bf1e0a5, 2016-06-02, which did not add the refusal but wrapped it — `if not (name == name.upper() or ...)` became `if not force and not (...)` — and shipped in 0.4.0 (June 2, 2016; its own release-log line is docs/release_log.rst under that heading). So R5's statement as a whole holds from 0.4.0 on. rules.md had never said any of it, though bf1e0a5's diff shows the API docstring already did: "It will not adjust the case of names entered in mixed case" was there before that commit edited around it. The gap was rules.md's alone. Evidence, measured on the released 1.4.0 wheel — the last v1 release and one of the differential baselines, so a natural thing to measure against and not a release that introduced anything here — and re-measured on this branch today, facade and core agreeing: `HumanName('Shirley Maclaine').capitalize()` leaves `'Shirley Maclaine'` — mixed case, wrong, and kept — while the same name under `force=True` gives `'Shirley MacLaine'`; `HumanName('Juan McDonald').capitalize()` leaves `'Juan McDonald'`. rules.md's own preamble classifies behavior in this position as "pinned-but-undocumented — an extraction gap to close, not a specification", which is why the fix lands in the document rather than in `_render.py`. +- 2026-08-29 — why the gap survived, and what the new boundary buys. R4 promised that "an already-correct name comes back unchanged", which is idempotence: true of everything the gate returns, and silent about the gate. The two readings were indistinguishable through R4's boundary, because `"Juan McDonald"` comes back unchanged under `force=True` as well — an already-correct input cannot witness a rule whose subject is what gets kept REGARDLESS of correctness. `Shirley Maclaine` separates them, so R5's block pins it twice: kept as written on a plain `capitalized=` line, and repaired on the forced line. Under idempotence the plain line would have to fail. The BOUNDARY sits on the forced line, which took a correction to get right — the first draft marked the refusal, and the refusal is R5's effect, not its absence. R5's own stated exception is repair-regardless, so that is the non-firing case (rules.md's preamble: the boundary is "usually the rule's OWN stated exception"). +- 2026-08-29 — WHY R5'S STATEMENT WAS REWORDED RATHER THAN GIVEN A THIRD EXAMPLE, recorded at length because the wrong turn was taken twice and the reasoning is reusable. R5 first said a mixed-case name is returned untouched "whether or not its casing is correct". That phrasing reads as a disjunction with two branches, and invites the question of which example witnesses the already-correct one. Nothing can: for a mixed-case name that repair would not alter, kept and repaired are the SAME STRING by construction, so no example line distinguishes a parser with the gate from one without it. The unwitnessable branch is a property of the claim, not a gap in our choice of names — and the bullet above had already said so ("an already-correct input cannot witness a rule whose subject is what gets kept REGARDLESS of correctness") before two review rounds pushed past it. Two rows were tried and both withdrawn. `"Juan McDonald" → capitalized="Juan McDonald"` was INERT (mechanisms.md's inert-measurement class): it passes with R5's gate deleted, which is the whole failure shape that class names. `"Vincent Van Gogh" → capitalized="Vincent Van Gogh"` did discriminate — measured by comparing the shipped call against gate-off behavior, which the forced call reproduces exactly, since deleting the gate is what `force` already does: shipped `'Vincent Van Gogh'`, gate-off `'Vincent van Gogh'`, the particle rule lowercasing `Van` the moment repair runs — but it was REDUNDANT, not complementary. It asserts the same proposition as the `Shirley Maclaine` row above it: repair would change this name, and the gate keeps it anyway. The two differ only in which repair rule would have fired, and which rule repair applies is R4's subject, not R5's. It was also the wrong name to call bearer-correct: this document writes `Vincent van Gogh` lowercase in P6's own examples, and the tussenvoegsel convention behind that is amended later on this branch, so the entry would have asserted a spelling the rules contradict. The fix was to stop claiming the unwitnessable branch: the statement now says a mixed-case name is kept and that whether its casing is right does not enter into the decision, which `Shirley Maclaine` — casing wrong, kept — witnesses whole. GENERAL LESSON, and the reason this is long: when an example cannot be found for half a rule, suspect the STATEMENT before suspecting the example set. A phrasing that promises more branches than the behavior has will absorb inert examples indefinitely, each one looking like progress. Caution for the commit that reworks R4 and will choose its own mixed-case rows: avoid a name whose family base is wholly particle vocabulary (`Anh Van Do`, base `Van Do`), because the #407 work changes how those capitalize and would neuter such a row silently; `gogh` and `vega` are in neither `particles` nor `particles_ambiguous`, `van` and `do` are in both. +- 2026-08-29 — the override has TWO routes, and the rule states neither, by design. Per-call is the obvious one. The second is a v1 Constants attribute the facade still honors, `force_mixed_case_capitalization` (nameparser/_facade.py resolves it when no per-call value is given; docs/release_log.rst records it as "still honored through the facade"). Measured today: with that attribute set True, `HumanName('Shirley Maclaine')` then a bare `.capitalize()` — no argument at all — LEAVES `'Shirley MacLaine'` (the v1 call mutates in place and returns None, so the name is read back with `str()`). An earlier draft of R5 said "only an explicit request to repair regardless overrides that", which that measurement falsifies; the statement now says repair-regardless was asked for, without saying by what route, because rules.md is implementation-free by its own preamble. +- 2026-08-29 — `capitalized_forced` is a test-side pseudo-field, not a parser field. The route the EXAMPLES use is the per-call argument to `capitalized()`, and an argument is not a policy, locale or extras gate — the only three things rules.md's grammar admits in an example's annotation slot — so it cannot ride that slot, and the doc runner grew a resolver branch instead. (The facade attribute above is a second route to the same behavior, not a second thing to assert; the core takes the argument only.) The R5 block's example lines are asserted by the suite like any others. They do enlarge one file: `corpus_rules.jsonl` is generated from this document's examples, and `Shirley Maclaine` had not been among them, so regenerating adds it (235 lines, from 234). Whether that moves the DIFFERENTIAL population is a separate question with a per-name answer, and the general answer is that a rules.md example CAN move it: the harness dedupes across corpus files, so a new example name costs a population slot exactly when no other corpus already carries that same string. `Shirley Maclaine` was already in `corpus.jsonl`, so it costs nothing and the population holds at 1090 — measured, not assumed. The row withdrawn above is the counter-case, and worth keeping for it: `Vincent Van Gogh` appears in no other corpus (`corpus.jsonl` and `corpus_issues.jsonl` carry `Vincent van Gogh`, a DIFFERENT string that does not dedupe against it), and while it was in the block the population read 1091. Neither name needed a ledger rule, which is a measurement rather than a consequence of the above: the gate exits 0 with 0 unexplained at all three baselines either way. +- 2026-08-29 — DEBT this extraction leaves, named so the next commit inherits an obligation rather than a rediscovery. Pulling the gate out into R5 leaves R4 carrying ONE falsehood and ONE ambiguity — different defects wanting different repairs, and `interacts: R5` carries neither, the field being advisory. FALSE: R4 promises repair "vocabulary exceptions (McDonald) included", but `str(parse('Juan Mcdonald').capitalized())` is `'Juan Mcdonald'` — the gate refuses before any vocabulary is consulted, and only `str(parse('Juan Mcdonald').capitalized(force=True))`, `'Juan McDonald'`, reaches the exception. R4 needs its promise scoped to names the gate admits. AMBIGUOUS, not false: R4's "an already-correct name comes back unchanged" means correct by the repair's own conventions, i.e. idempotence, and under that meaning it is true; a reader hears correct as the bearer writes it, and under THAT meaning `str(parse('bell hooks').capitalized())` — `'Bell Hooks'` — looks like a counterexample. It is not one, because `bell hooks` is not already-correct in R4's sense. What R4 owes is a disambiguation of "correct", NOT a narrowing to spare deliberately single-cased names: that would be new behavior, and R5's own rationale declines it on the ground that single case leaves the repair nothing to read. Also for that commit, and inert as things stand: R4's boundary row `"Juan McDonald" → capitalized="Juan McDonald"` passes with R5's gate deleted, exactly like the R5 row that was withdrawn above; rewriting it to `capitalized_forced=` makes it discriminate for R4's own subject but still witnesses nothing about the already-correct question. This commit adds R5 and touches R4 only on its pointer line, leaving both defects as found rather than half-fixed by a commit whose subject is something else. + ### removed-v1-surface - empty_attribute_default: removed in 2.0 (#255; deprecated in 1.4 per the bridge discipline). Origin #44 (2016): a DB-NULL convenience whose first answer — `name.title or None` — became the migration path. The in-band-signaling bug that sealed it (#254): the 2016 `.replace('None','')` scrub could not tell interpolated None from name text, so "Nonez Smith" rendered @@ -735,6 +744,7 @@ Promoted 2026-08-15 from session memory (Derek's 2026-07-30 ask; promotion appro - (A) nameparser.config removal scope: the 3.0 schedule says "nameparser.config in its entirety" while actually enumerating only the five shim exports; whether the DATA modules keep the package as their home or move under the core is open, and Lexicon's public field docs cross-reference nameparser.config.particles et al. (see the maintainer note in nameparser/config/__init__.py). +- (A) `force_mixed_case_capitalization` as a second route into R5's override (2026-08-29, recorded while extracting R5). The core takes `force` per call and nothing else; the attribute reaches the same behavior only because the facade resolves it when no per-call value is given, which is v1 parity and nothing more. Measured: with it True, a bare `.capitalize()` on `HumanName('Shirley Maclaine')` leaves it `'Shirley MacLaine'` (mutator: read the result back with `str()`). Without the shim there is one route, and R5's statement could say so plainly instead of being written to cover both. Recorded as a removal CANDIDATE — nothing is decided here, and the bridge discipline above applies whenever it is. - (B) Pickle-guard layout breaks landing in minors (2.1's __setstate__ breaks): the guarded-raise design is right regardless; only the in-a-minor friction is shim-era. - (B) Positional-read-when-unlicensed as the safe direction (script_orders fallbacks, #298 dot-suppression granularity): coincides with 1.4 parity but stands alone — family-first is the marked case needing affirmative evidence. - (B) The deprecation-bridge shape (#293/#354) is the template for every remaining 2.x→3.0 shim (the bridge DISCIPLINE — warn in a released version first — predates it, from #223/#224; this entry is the module-__getattr__ mechanics): PEP 562 module __getattr__ PLUS __all__ (star imports never reach __getattr__ — measured: `from ...prefixes import *` bound nothing and leaked the helper); warn per read-location rather than per process (a write-back let a vendored dependency's first read consume the only warning); and the TYPE_CHECKING split, because a module __getattr__ silently disables mypy's attr-defined checking for the whole module (measured on a py.typed package). diff --git a/docs/design/rules.md b/docs/design/rules.md index 0c2eb032..49d9d03e 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -1105,7 +1105,24 @@ R4. Rationale: case repair is a display concern, applied only on already-correct name comes back unchanged. "juan mcdonald" → capitalized="Juan McDonald" "Juan McDonald" → capitalized="Juan McDonald" · boundary - implemented: nameparser/_render.py + interacts: R5 · implemented: nameparser/_render.py + +R5. Rationale: mixed case is evidence that the writer cased the name + deliberately, and a repair cannot tell a deliberate spelling from + a mistaken one. A name written wholly in one case leaves the + repair no such evidence to read either way — the writer who + always writes lowercase is indistinguishable from the writer who + could not be bothered — so repair proceeds there, which is a + choice about how to act absent evidence rather than a claim that + nothing can be lost by it. + Case repair acts only on a name written entirely in one case. A + name written in more than one case is kept as it was written, + and whether that casing is right does not enter into it, unless + repair regardless of how the name is cased was asked for. + "juan mcdonald" → capitalized="Juan McDonald" + "Shirley Maclaine" → capitalized="Shirley Maclaine" + "Shirley Maclaine" → capitalized_forced="Shirley MacLaine" · boundary + history: decisions.md#R5 · interacts: R4 · implemented: nameparser/_render.py ## Construction & configuration diagnostics (D) diff --git a/nameparser/_render.py b/nameparser/_render.py index 487ef096..b4b3c561 100644 --- a/nameparser/_render.py +++ b/nameparser/_render.py @@ -170,6 +170,8 @@ def capitalized(name: ParsedName, lexicon: Lexicon | None, *, raise TypeError(f"lexicon must be a Lexicon or None, got {lexicon!r}") lex = Lexicon.default() if lexicon is None else lexicon joined = " ".join(t.text for t in name.tokens) + # rules.md#R5: "case repair acts only on a name written entirely + # in one case" if not force and joined not in (joined.upper(), joined.lower()): return name new_tokens = tuple( diff --git a/tests/v2/rules_doc.py b/tests/v2/rules_doc.py index a803d383..1b80b2a0 100644 --- a/tests/v2/rules_doc.py +++ b/tests/v2/rules_doc.py @@ -46,7 +46,7 @@ ASSERTABLE_FIELDS = frozenset({ "title", "given", "middle", "family", "suffix", "nickname", "maiden", "family_base", "family_particles", "surnames", "given_names", - "initials", "capitalized", + "initials", "capitalized", "capitalized_forced", "ambiguities", "pieces", "warns", "raises"}) diff --git a/tests/v2/test_rules_doc.py b/tests/v2/test_rules_doc.py index 83bc151e..3de413be 100644 --- a/tests/v2/test_rules_doc.py +++ b/tests/v2/test_rules_doc.py @@ -117,6 +117,10 @@ def _run(example: Example) -> object: return parsed.initials() if example.field == "capitalized": return str(parsed.capitalized()) + # R5: the gate returns mixed-case input untouched, so an example + # asserting what the REPAIR does must ask for it regardless + if example.field == "capitalized_forced": + return str(parsed.capitalized(force=True)) return getattr(parsed, example.field) diff --git a/tools/differential/compare.py b/tools/differential/compare.py index 1955f156..283b8565 100644 --- a/tools/differential/compare.py +++ b/tools/differential/compare.py @@ -373,7 +373,7 @@ def _is_latin_only(name: str) -> bool: "corpus.jsonl": 480, # 486 today, from v1's banks at a pinned ref "corpus_cjk.jsonl": 95, # 98 today, generated from the case table "corpus_issues.jsonl": 370, # 381 today, harvested and append-only - "corpus_rules.jsonl": 150, # 234 today, generated from rules.md + "corpus_rules.jsonl": 150, # 235 today, generated from rules.md } diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index f7543f31..3d0ff383 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -136,6 +136,7 @@ "Sheik abdul Jr Smith" "Sheik abdul salam" "Sheik abdul salam Jr" +"Shirley Maclaine" "Sidorov Ivan Petrovich Jr." "Sir John" "Sir abdul van der Berg" From 96b46466d949213d38b66b8f79b023bdc8b47849 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sat, 29 Aug 2026 11:40:51 -0700 Subject: [PATCH 2/6] fix(render,docs): case repair consults the unjoined-particle mark (#407) `capitalized()` lowercased the words of a family name made only of particle vocabulary, while `family_base`, `family_particles` and `initials()` have read them as ordinary name words since #404 -- `parse("ANH DO").capitalized()` gave 'Anh do' with family_base 'DO' and initials 'A. D.'. _cap_word now sees the token's tags and skips the particle lowercasing when UNJOINED_TAG is present. The mark is consulted for the WHOLE PART, not for a particle standing alone: `anh van do` has family `van do`, two particle words and neither alone, and a standing-alone test would read the same surname one way behind a given name and another way alone. Only the PARTICLE conjunct is gated. Gating the conjunction conjunct too would be a NO-OP, not a mistake: a part is marked only where every word in it carries "particle", `particles` and `conjunctions` are disjoint, so a part holding a conjunction is never marked and no conjunction token can carry the mark. The code comment, the release bullet and the new test all say so now -- an earlier draft of each claimed R3 gives "a conjunction contributing nothing even in an all-particle part", which describes a configuration the parser cannot produce (`der, y van` keeps its lowercase `y` because the `y` is a GIVEN-part conjunction, not a word of the all-particle family). Deliberate 1.4.0 deviation, not a restoration: the released wheel returns 'Anh do'. Accepted cost, agreed beforehand -- a degenerate all-particle family capitalizes too, `juan van der` -> 'Juan Van Der'. docs/design/rules.md#R4 is restated, discharging the two debts decisions.md#R5 recorded: the unscoped vocabulary promise (the gate refuses before any vocabulary is read) and the ambiguous "already-correct", now disambiguated as correct-by-repair's-own- conventions. A third error surfaced while rewriting those words and is fixed too: McDonald is not a vocabulary exception at all -- the shipped exceptions are II/III/IV/M.D./Ph.D. and Mc/Mac is a separate convention -- so the statement now names one of each. R4's boundary moves to `juan de la vega` and its inert `"Juan McDonald"` row becomes a `capitalized_forced=` row that asserts what repair applies. That correction then needed a witness of its own. Naming two mechanisms where the rows only covered one reintroduced the very defect the rewrite was repairing: emptying the exceptions map left all nine R4/R5 rows green, none of their inputs holding a word the map carries. R4 gains `"john smith phd" -> capitalized="John Smith Ph.D."`, which is the only row of the ten that dies when the map is emptied (`John Smith Phd`). Chosen for having the dullest possible parse and a replacement no other rule can produce -- nothing else in the repair inserts periods -- so the row pins the map and nothing else. Reframing, on Derek's correction: case repair is not about correctness. Mixed case is the writer making an explicit choice and repair defers to it rather than judging it right or wrong. R4's closing clause no longer says a bearer's spelling is unspared -- which contradicted R5 -- but that a SINGLE-CASE spelling is repaired even where its bearer meant it, nothing in the text marking it as a choice. decisions.md#R5's withdrawn-witness bullet retires the bearer-correct frame it was written under, and records that needing a contestable claim about Dutch orthography was the symptom of it. R5's override clause is reworded from `unless repair regardless of how the name is cased was asked for` to `unless repair was asked for anyway`. The old nine words carried a second reading -- that the repair disregards the input's casing -- which is exactly the property the entry below measures and falsifies, in nearly its own vocabulary. A reader taking it that way would run the re-casing test, land on `Velasquez y Garcia, Dr. Juan Q.`, and conclude the rule was wrong when only the phrasing was. R5 also gains `"SHIRLEY MACLAINE" -> capitalized="Shirley MacLaine"`. It is the only row in that block that fails when the gate is narrowed to lowercase-only; until now R5 stated that repair acts on a name written entirely in one case and witnessed only the lowercase half. MEASUREMENT CORRECTION, recorded rather than shipped as proposed. The invariant that forcing ignores the given casing does NOT hold. Over the 1094 corpus names, `capitalize(force=True)` differs from uppercasing then repairing for 63 names and from lowercasing for 16: uppercase is the WORSE direction, and 38 of the 63 parse byte-identically and diverge inside the repair, so they are not parse-level. v1's initial carve-out is the mechanism (a one-letter conjunction is initial-shaped once uppercased), and 1.4.0 does the same. The property is pinned in tests over names with no single-letter word whose class case decides, with `juan y garcia` beside it as the recorded exception, and is deliberately NOT stated in rules.md: its examples are keyed on input strings, so stating it would invite the re-casing test that falsifies it. The repair-level half is illustrated with `Velasquez y Garcia, Dr. Juan Q.`, whose partition is byte-identical either side; the space-written form of that name is a member of the 25 whose roles do move. decisions.md#R4 records the scope choice, the deviation, the cost, and the VERIFICATION LIMIT that matters most: compare.py compares the seven role fields plus _ambiguities and never `capitalized()`, so no ledger run can see this deviation at any baseline -- half of mechanisms.md#FACADE-CONTRACT's promise cannot be kept here. The rules.md examples and the v1-facade tests stand in its place; the ones that pin the fix are mutation-checked against deletion of the tag consult, and the `juan de la vega` control passes that mutation by design, which is what makes it a control. A follow-up issue for a capitalization column is left as a forward reference; the entry records why one was not added now (1.4.0 and 2.1.0 already disagree on `capitalize(force=True)` for 139 of 1090 corpus names, nearly all echoes of parse diffs the field comparison explains, so it needs a diff-of-diffs design rather than a column). Gate: corpus 1094 (was 1090 -- `anh van do`, `juan de la vega`, `SHIRLEY MACLAINE` and `john smith phd` are new), intentional diffs 229/194/102 at 1.4.0/2.0.0/2.1.0, unexplained 0 at all three -- unmoved, as expected for a change to an uncompared field. corpus_rules.jsonl 235 -> 240; its floor comment updated, the floor itself unchanged. Ledger guards pass with no reach recount. Co-Authored-By: Claude Opus 5 --- docs/design/decisions.md | 15 +++- docs/design/rules.md | 26 +++++-- docs/release_log.rst | 2 + nameparser/_render.py | 28 +++++-- nameparser/_types.py | 8 +- tests/test_capitalization.py | 106 ++++++++++++++++++++++++++ tools/differential/compare.py | 2 +- tools/differential/corpus_rules.jsonl | 5 ++ 8 files changed, 172 insertions(+), 20 deletions(-) diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 1e3720af..c6385545 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -712,13 +712,24 @@ Declined: - 2026-08-18 — the GROUPING half of #404: a particle run that joins nothing does not chain, so "Jong van der" would split into middle 'van' plus family 'der'. Measured and rejected, though NOT for the reason first recorded here. The first draft said the split makes the family "no longer all-particle so the base fix stops firing" — false, and `der` and `la` are both shipped particles, so a family of either IS all-particle and the rule fires on it ("Juan Smith der" gives base 'der'). What the split actually costs is the SCOPE of the base and a stray particle relocated: grouping can decline to merge but cannot keep the words apart, because roles re-assemble them and two adjacent same-role pieces are one part at the field level. "Juan Smith van der" becomes middle 'Smith van', family 'der' — a base of 'der' rather than 'van der', and a middle name nobody wrote. Keeping the run whole in one part is what gives the base its full extent. The split reading needs the leftover distribution to know these are separate units, which is mechanisms.md#UNIT-PARTITION's problem. +### R4 — case repair reads the unjoined mark + +- 2026-08-29 #407 — DECIDED: case repair consults the mark #404 introduced, and consults it for the WHOLE PART rather than for a particle standing alone. Until now three views read it and the fourth did not, which is how `parse("ANH DO")` came to report family_base 'DO' and initials 'A. D.' while `capitalized()` returned 'Anh do' — the same word treated as an ordinary name word by three readers and as a particle by the fourth. The scope question is the only real choice here, and standing-alone loses on the first Vietnamese name that tests it: `anh van do` has family `van do`, two particle-vocabulary words and neither of them alone, so a rule keyed on aloneness would capitalize `Anh Do` and leave `Anh van do`, making the same surname read differently for having a given-name word in front of it. rules.md#R2 states the test as "Every word", not "standing alone", and this is the case that shows why it is stated that way. One predicate, four views. +- 2026-08-29 — this is a DELIBERATE DEVIATION from 1.4.0, not a parity restoration, said plainly because the surrounding branch work is mostly restorations and a reader will otherwise assume this is one. Measured on the released 1.4.0 wheel: `HumanName('ANH DO').capitalize()` leaves `'Anh do'` and `HumanName('anh van do').capitalize()` leaves `'Anh van do'`; both are `'Anh Do'` and `'Anh Van Do'` on this branch. There was nothing to restore — v1 lowercased on vocabulary membership alone and had no notion of a particle with nothing to join, the mark being a 2.x invention. +- 2026-08-29 — the ACCEPTED COST, agreed before the change rather than discovered after it: a degenerate family that is nothing but particles capitalizes too, so `juan van der` renders `Juan Van Der` where 1.4.0 gave `Juan van der`. This is the same trade rules.md#R2 already took for `family_base` — those words ARE the base, so they are name words, and a parser that called them name words for sorting and particles for display would be the incoherent option. Blast radius over the 1090 names the corpora carried before this change, measured by running both predicates in one process: 4 names move under a plain `capitalized()` — `ANH DO`, `vai la`, `van ma van`, and `der, y van`, which becomes `y Van Der` and keeps its lowercase conjunction. 23 move under `force=True`, the extra 19 being mixed-case inputs the gate holds back (rules.md#R5). +- 2026-08-29 — THE VERIFICATION LIMIT, which matters more than the change. `tools/differential/compare.py` compares the seven role fields plus `_ambiguities` and nothing else, so `capitalized()` is not a compared field and NO ledger run at any baseline can see this deviation: the corpus counts and the three intentional-diff totals hold across it precisely because the harness is blind to it, and a green run is evidence about roles, not about case. mechanisms.md#FACADE-CONTRACT promises that v1-visible changes are "release-log-classified fixes, each verified in the differential ledger". The first half is kept — docs/release_log.rst carries the 2.2.0 bullet — and the SECOND HALF CANNOT BE, for this change or for any other that moves only case. Standing in its place: rules.md#R4's own example rows (`ANH DO`, `anh van do`, and the `juan de la vega` boundary), rules.md#R5's, and the v1-facade tests in tests/test_capitalization.py. The ones that pin the fix are mutation-checked against deletion of the tag consult; the `juan de la vega` control passes that mutation BY DESIGN, which is what makes it a control rather than a witness. A follow-up issue for a capitalization column belongs on the tracker and is not filed yet; until it is, this bullet is the only written record that the ledger's silence here is structural rather than a passing grade. +- 2026-08-29 — WHY THE HARNESS WAS NOT EXTENDED IN THIS COMMIT, measured rather than waved off, because "add a column" is the obvious answer to the bullet above and it is the wrong size. Over those same 1090 names the 1.4.0 and 2.1.0 wheels ALREADY disagree on `capitalize(force=True)` for 139 of them (recompute by running `HumanName(n).capitalize(force=True)` under each wheel over the four corpus files deduped, and diffing the two maps). Nearly every one is an echo of a parse diff the field comparison already classifies — a token that changed role capitalizes differently — so a naive column would re-report the ledger's own explained diffs in a vocabulary its rules cannot match, and would arrive pre-failed by 139 entries. A useful column has to compare case only where the fields agree, which is a diff-of-diffs design and a piece of work in its own right. +- 2026-08-29 — DEBT DISCHARGED, the obligation the last bullet of decisions.md#R5 left this commit; recorded here so the pointer resolves rather than dangling. The FALSEHOOD: R4 promised repair "vocabulary exceptions (McDonald) included" flatly, which `str(parse('Juan Mcdonald').capitalized())` — `'Juan Mcdonald'` — falsifies, the gate refusing before any vocabulary is consulted. The statement now scopes that promise to where repair acts at all and hands the where to R5. A SECOND error in the same eight words turned up while rewriting them, and is worth recording because it had survived unread for as long as the first: McDonald is not a vocabulary exception. The shipped exceptions are five entries — II, III, IV, M.D., Ph.D. (recompute with `Lexicon.default().capitalization_exceptions_map`) — and `Mc`/`Mac` is a separate convention applied to any name shaped like one, so the old parenthetical illustrated the promise with the one mechanism it was not about. The statement now names an exception the vocabulary actually records (Ph.D.) and the Mac/Mc convention separately. The AMBIGUITY: "an already-correct name comes back unchanged" is true read as idempotence and reads as false under bearer-correctness, and the repair was to say WHICH is meant — a name already written the way repair would write it comes back unchanged — plus one clause refusing the inference a reader might otherwise draw, that a spelling its bearer chose is not spared for having been chosen. That clause is deliberately a REFUSAL and not a carve-out: sparing deliberately single-cased names is behavior the parser does not have and that R5's rationale declines on the record, so `str(parse('bell hooks').capitalized())` is still `'Bell Hooks'`. R4's boundary moved too: the old `"Juan McDonald" → capitalized="Juan McDonald"` row was inert — it passes with R5's gate deleted — so it became a `capitalized_forced=` row, which asserts what repair APPLIES rather than what the gate keeps and fails when the Mac/Mc convention is removed (measured by neutering it in one process: `'Juan Mcdonald'`). The boundary is now `juan de la vega`, a working particle run R4's new clause deliberately does not reach. + ### R5 — the case-repair gate - 2026-08-29 (#407 arc) — EXTRACTION, not a decision: the parser is untouched. The two halves of R5 have separate provenance, and conflating them is easy enough that the first draft of this entry did. The REFUSAL — repair skips any name already carrying more than one case — is older than the git history: `git log -S "name == name.upper() or name == name.lower()" --reverse` bottoms out at 45a1539 (2011-02-03), the initial import from svn, where `capitalize()` already opens with that guard and a bare `return`. (A path-filtered search answers 280895b instead, the same-day commit that moved the module into `nameparser/`; the code did not change there.) The OVERRIDE is bf1e0a5, 2016-06-02, which did not add the refusal but wrapped it — `if not (name == name.upper() or ...)` became `if not force and not (...)` — and shipped in 0.4.0 (June 2, 2016; its own release-log line is docs/release_log.rst under that heading). So R5's statement as a whole holds from 0.4.0 on. rules.md had never said any of it, though bf1e0a5's diff shows the API docstring already did: "It will not adjust the case of names entered in mixed case" was there before that commit edited around it. The gap was rules.md's alone. Evidence, measured on the released 1.4.0 wheel — the last v1 release and one of the differential baselines, so a natural thing to measure against and not a release that introduced anything here — and re-measured on this branch today, facade and core agreeing: `HumanName('Shirley Maclaine').capitalize()` leaves `'Shirley Maclaine'` — mixed case, wrong, and kept — while the same name under `force=True` gives `'Shirley MacLaine'`; `HumanName('Juan McDonald').capitalize()` leaves `'Juan McDonald'`. rules.md's own preamble classifies behavior in this position as "pinned-but-undocumented — an extraction gap to close, not a specification", which is why the fix lands in the document rather than in `_render.py`. - 2026-08-29 — why the gap survived, and what the new boundary buys. R4 promised that "an already-correct name comes back unchanged", which is idempotence: true of everything the gate returns, and silent about the gate. The two readings were indistinguishable through R4's boundary, because `"Juan McDonald"` comes back unchanged under `force=True` as well — an already-correct input cannot witness a rule whose subject is what gets kept REGARDLESS of correctness. `Shirley Maclaine` separates them, so R5's block pins it twice: kept as written on a plain `capitalized=` line, and repaired on the forced line. Under idempotence the plain line would have to fail. The BOUNDARY sits on the forced line, which took a correction to get right — the first draft marked the refusal, and the refusal is R5's effect, not its absence. R5's own stated exception is repair-regardless, so that is the non-firing case (rules.md's preamble: the boundary is "usually the rule's OWN stated exception"). -- 2026-08-29 — WHY R5'S STATEMENT WAS REWORDED RATHER THAN GIVEN A THIRD EXAMPLE, recorded at length because the wrong turn was taken twice and the reasoning is reusable. R5 first said a mixed-case name is returned untouched "whether or not its casing is correct". That phrasing reads as a disjunction with two branches, and invites the question of which example witnesses the already-correct one. Nothing can: for a mixed-case name that repair would not alter, kept and repaired are the SAME STRING by construction, so no example line distinguishes a parser with the gate from one without it. The unwitnessable branch is a property of the claim, not a gap in our choice of names — and the bullet above had already said so ("an already-correct input cannot witness a rule whose subject is what gets kept REGARDLESS of correctness") before two review rounds pushed past it. Two rows were tried and both withdrawn. `"Juan McDonald" → capitalized="Juan McDonald"` was INERT (mechanisms.md's inert-measurement class): it passes with R5's gate deleted, which is the whole failure shape that class names. `"Vincent Van Gogh" → capitalized="Vincent Van Gogh"` did discriminate — measured by comparing the shipped call against gate-off behavior, which the forced call reproduces exactly, since deleting the gate is what `force` already does: shipped `'Vincent Van Gogh'`, gate-off `'Vincent van Gogh'`, the particle rule lowercasing `Van` the moment repair runs — but it was REDUNDANT, not complementary. It asserts the same proposition as the `Shirley Maclaine` row above it: repair would change this name, and the gate keeps it anyway. The two differ only in which repair rule would have fired, and which rule repair applies is R4's subject, not R5's. It was also the wrong name to call bearer-correct: this document writes `Vincent van Gogh` lowercase in P6's own examples, and the tussenvoegsel convention behind that is amended later on this branch, so the entry would have asserted a spelling the rules contradict. The fix was to stop claiming the unwitnessable branch: the statement now says a mixed-case name is kept and that whether its casing is right does not enter into the decision, which `Shirley Maclaine` — casing wrong, kept — witnesses whole. GENERAL LESSON, and the reason this is long: when an example cannot be found for half a rule, suspect the STATEMENT before suspecting the example set. A phrasing that promises more branches than the behavior has will absorb inert examples indefinitely, each one looking like progress. Caution for the commit that reworks R4 and will choose its own mixed-case rows: avoid a name whose family base is wholly particle vocabulary (`Anh Van Do`, base `Van Do`), because the #407 work changes how those capitalize and would neuter such a row silently; `gogh` and `vega` are in neither `particles` nor `particles_ambiguous`, `van` and `do` are in both. -- 2026-08-29 — the override has TWO routes, and the rule states neither, by design. Per-call is the obvious one. The second is a v1 Constants attribute the facade still honors, `force_mixed_case_capitalization` (nameparser/_facade.py resolves it when no per-call value is given; docs/release_log.rst records it as "still honored through the facade"). Measured today: with that attribute set True, `HumanName('Shirley Maclaine')` then a bare `.capitalize()` — no argument at all — LEAVES `'Shirley MacLaine'` (the v1 call mutates in place and returns None, so the name is read back with `str()`). An earlier draft of R5 said "only an explicit request to repair regardless overrides that", which that measurement falsifies; the statement now says repair-regardless was asked for, without saying by what route, because rules.md is implementation-free by its own preamble. +- 2026-08-29 — WHY R5'S STATEMENT WAS REWORDED RATHER THAN GIVEN A THIRD EXAMPLE, recorded at length because the wrong turn was taken twice and the reasoning is reusable. R5 first said a mixed-case name is returned untouched "whether or not its casing is correct". That phrasing reads as a disjunction with two branches, and invites the question of which example witnesses the already-correct one. Nothing can: for a mixed-case name that repair would not alter, kept and repaired are the SAME STRING by construction, so no example line distinguishes a parser with the gate from one without it. The unwitnessable branch is a property of the claim, not a gap in our choice of names — and the bullet above had already said so ("an already-correct input cannot witness a rule whose subject is what gets kept REGARDLESS of correctness") before two review rounds pushed past it. Two rows were tried and both withdrawn. `"Juan McDonald" → capitalized="Juan McDonald"` was INERT (mechanisms.md's inert-measurement class): it passes with R5's gate deleted, which is the whole failure shape that class names. `"Vincent Van Gogh" → capitalized="Vincent Van Gogh"` did discriminate — measured by comparing the shipped call against gate-off behavior, which the forced call reproduces exactly, since deleting the gate is what `force` already does: shipped `'Vincent Van Gogh'`, gate-off `'Vincent van Gogh'`, the particle rule lowercasing `Van` the moment repair runs — but it was REDUNDANT, not complementary. It asserts the same proposition as the `Shirley Maclaine` row above it: repair would change this name, and the gate keeps it anyway. The two differ only in which repair rule would have fired, and which rule repair applies is R4's subject, not R5's. And the frame it was chosen under was itself wrong, which is the more useful half of the lesson (reframed 2026-08-29 on Derek's correction, while R4 was being reworked). The row was picked to be bearer-correct — a name whose casing the bearer would endorse — and that frame drags the document into per-name arguments about whose spelling is right: this one needed `Vincent van Gogh` to be the correct spelling, which P6's own examples contradict, and the tussenvoegsel convention behind them is amended later on this branch anyway. CORRECTNESS DOES NOT ENTER INTO IT. Mixed case is the writer making an explicit choice, and repair defers to that choice rather than judging it — a name kept is not a name endorsed. Read that way the withdrawn row needed no claim about Dutch orthography at all, and the reason it needed one is exactly that the frame was wrong. The fix was to stop claiming the unwitnessable branch: the statement now says a mixed-case name is kept and that whether its casing is right does not enter into the decision, which `Shirley Maclaine` — casing wrong, kept — witnesses whole. GENERAL LESSON, and the reason this is long: when an example cannot be found for half a rule, suspect the STATEMENT before suspecting the example set. A phrasing that promises more branches than the behavior has will absorb inert examples indefinitely, each one looking like progress. Caution for the commit that reworks R4 and will choose its own mixed-case rows: avoid a name whose family base is wholly particle vocabulary (`Anh Van Do`, base `Van Do`), because the #407 work changes how those capitalize and would neuter such a row silently; `gogh` and `vega` are in neither `particles` nor `particles_ambiguous`, `van` and `do` are in both. +- 2026-08-29 — the override has TWO routes, and the rule states neither, by design. Per-call is the obvious one. The second is a v1 Constants attribute the facade still honors, `force_mixed_case_capitalization` (nameparser/_facade.py resolves it when no per-call value is given; docs/release_log.rst records it as "still honored through the facade"). Measured today: with that attribute set True, `HumanName('Shirley Maclaine')` then a bare `.capitalize()` — no argument at all — LEAVES `'Shirley MacLaine'` (the v1 call mutates in place and returns None, so the name is read back with `str()`). An earlier draft of R5 said "only an explicit request to repair regardless overrides that", which that measurement falsifies; the statement now says repair was asked for anyway, without saying by what route (it read `repair regardless of how the name is cased was asked for` until 2026-08-29, when that phrasing turned out to have a second reading -- see the override bullet below), because rules.md is implementation-free by its own preamble. - 2026-08-29 — `capitalized_forced` is a test-side pseudo-field, not a parser field. The route the EXAMPLES use is the per-call argument to `capitalized()`, and an argument is not a policy, locale or extras gate — the only three things rules.md's grammar admits in an example's annotation slot — so it cannot ride that slot, and the doc runner grew a resolver branch instead. (The facade attribute above is a second route to the same behavior, not a second thing to assert; the core takes the argument only.) The R5 block's example lines are asserted by the suite like any others. They do enlarge one file: `corpus_rules.jsonl` is generated from this document's examples, and `Shirley Maclaine` had not been among them, so regenerating adds it (235 lines, from 234). Whether that moves the DIFFERENTIAL population is a separate question with a per-name answer, and the general answer is that a rules.md example CAN move it: the harness dedupes across corpus files, so a new example name costs a population slot exactly when no other corpus already carries that same string. `Shirley Maclaine` was already in `corpus.jsonl`, so it costs nothing and the population holds at 1090 — measured, not assumed. The row withdrawn above is the counter-case, and worth keeping for it: `Vincent Van Gogh` appears in no other corpus (`corpus.jsonl` and `corpus_issues.jsonl` carry `Vincent van Gogh`, a DIFFERENT string that does not dedupe against it), and while it was in the block the population read 1091. Neither name needed a ledger rule, which is a measurement rather than a consequence of the above: the gate exits 0 with 0 unexplained at all three baselines either way. +- 2026-08-29 — WHAT THE OVERRIDE DOES AND DOES NOT PROMISE, from Derek's framing of R5 and then measured, because the framing implies a property that is ALMOST true and the gap is the useful part. The framing first, and it supersedes the correctness talk elsewhere in this entry: mixed case is the writer making an explicit choice, and repair defers to that choice instead of judging it. Nothing is being called correct or incorrect — a name kept is a name whose writer said something about it, and a name repaired is one whose writer did not. The property that seems to follow is that asking for repair REGARDLESS should ignore the given casing entirely, so one name repairs to one string however it was written. MEASURED over the 1094 corpus names and it does NOT hold: `capitalize(force=True)` differs from uppercasing the input and calling `capitalize()` for 63 names, and from lowercasing it for 16. Two things about those numbers are the opposite of what one would guess. UPPERCASE IS THE WORSE DIRECTION, not the clean one. And the misses are not merely the pipeline's case-sensitivity leaking in: of the 63, only 25 move a role at all, and the other 38 parse byte-identically and diverge inside the repair. The mechanism is v1's initial carve-out — a conjunction is not lowercased where it is written initial-shaped, and initial-shaped means one CAPITAL letter — so uppercasing turns every one-letter conjunction into an initial (`Velasquez y Garcia, Dr. Juan Q.` forced keeps `y`; the same name uppercased then repaired gives `Y`, with a byte-identical partition either side -- the comma form is the one to cite here, the space-written `Dr. Juan Q. Velasquez y Garcia` being a member of the 25 whose roles DO move), and lowercasing turns a middle initial `E` into the Italian conjunction. 1.4.0 does the same (`JUAN Y GARCIA` capitalizes to `Juan Y Garcia`), so this is inherited, and it is recorded here rather than fixed here. Recompute both directions by running the two forms over the four corpus files deduped and diffing. +- 2026-08-29 — and therefore NOT stated in rules.md, which is a deliberate choice rather than an oversight. The document's examples are keyed on input STRINGS, so any statement of the property invites exactly the test that falsifies it — re-case the input, expect the same output — and the counterexamples are already in the corpora. The property is true of the repair given a parse, and rules.md speaks input-to-output; a rule stating it would be over-broad in the one direction a reader would check. What R5's statement says is enough for the promise that IS kept: a mixed-case name is kept unless repair was asked for anyway. That clause was REWORDED for this, and the reword is the whole point rather than a tidy-up. It read `unless repair regardless of how the name is cased was asked for`, which carries two readings -- the intended one, that the request overrides the keeping, and a second one, that the repair disregards the input's casing, which is this property in nearly this bullet's own words. A reader taking the second reading would run the re-casing test predicted above, land on `Velasquez y Garcia, Dr. Juan Q.` (in the corpus today), and conclude the RULE is wrong when only the phrasing was. Nine words, and they asserted the thing the paragraph exists to deny. The property is pinned in tests/test_capitalization.py instead, over names carrying no single-letter word whose class case decides, with `juan y garcia` beside it as the recorded exception. R5's example block gains `"SHIRLEY MACLAINE" → capitalized="Shirley MacLaine"` from this work, and it earns its place on its own ground rather than as half of a convergence pair: it is the only row in the block that fails when the gate is narrowed to lowercase-only, every other row passing that mutation. Measured three ways — gate deleted (passes, so it does not witness the gate's existence), gate narrowed to accept only all-lowercase (FAILS, and alone in the block), Mac/Mc convention deleted (fails, with the other two rows). Until it was added, R5 stated that repair acts on a name written entirely in one case and witnessed only the lowercase half of it. - 2026-08-29 — DEBT this extraction leaves, named so the next commit inherits an obligation rather than a rediscovery. Pulling the gate out into R5 leaves R4 carrying ONE falsehood and ONE ambiguity — different defects wanting different repairs, and `interacts: R5` carries neither, the field being advisory. FALSE: R4 promises repair "vocabulary exceptions (McDonald) included", but `str(parse('Juan Mcdonald').capitalized())` is `'Juan Mcdonald'` — the gate refuses before any vocabulary is consulted, and only `str(parse('Juan Mcdonald').capitalized(force=True))`, `'Juan McDonald'`, reaches the exception. R4 needs its promise scoped to names the gate admits. AMBIGUOUS, not false: R4's "an already-correct name comes back unchanged" means correct by the repair's own conventions, i.e. idempotence, and under that meaning it is true; a reader hears correct as the bearer writes it, and under THAT meaning `str(parse('bell hooks').capitalized())` — `'Bell Hooks'` — looks like a counterexample. It is not one, because `bell hooks` is not already-correct in R4's sense. What R4 owes is a disambiguation of "correct", NOT a narrowing to spare deliberately single-cased names: that would be new behavior, and R5's own rationale declines it on the ground that single case leaves the repair nothing to read. Also for that commit, and inert as things stand: R4's boundary row `"Juan McDonald" → capitalized="Juan McDonald"` passes with R5's gate deleted, exactly like the R5 row that was withdrawn above; rewriting it to `capitalized_forced=` makes it discriminate for R4's own subject but still witnesses nothing about the already-correct question. This commit adds R5 and touches R4 only on its pointer line, leaving both defects as found rather than half-fixed by a commit whose subject is something else. ### removed-v1-surface diff --git a/docs/design/rules.md b/docs/design/rules.md index 49d9d03e..0fe3330a 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -1100,12 +1100,25 @@ R3. Rationale: initials abbreviate the person's name words; titles, R4. Rationale: case repair is a display concern, applied only on request and never destructively. - Case repair returns a repaired copy — vocabulary exceptions - (McDonald) included — and never mutates the parse; an - already-correct name comes back unchanged. + Case repair returns a repaired copy and never mutates the parse. + Where it acts at all — R5 decides where — the copy honors the + casing a vocabulary entry records (Ph.D.) and the Mac/Mc surname + convention (McDonald), not only ordinary word-by-word casing, and + a part whose every word is particle vocabulary is repaired as + ordinary name words, since none of them is doing a particle's + work there (R2). A name already written the way repair would + write it comes back unchanged, measured by repair's own + conventions rather than by the bearer's. A spelling written in a + single case is repaired even where its bearer meant it, because + nothing in the text marks it as a choice; where the text does + mark one, R5 defers to it. "juan mcdonald" → capitalized="Juan McDonald" - "Juan McDonald" → capitalized="Juan McDonald" · boundary - interacts: R5 · implemented: nameparser/_render.py + "Juan McDonald" → capitalized_forced="Juan McDonald" + "ANH DO" → capitalized="Anh Do" + "anh van do" → capitalized="Anh Van Do" + "john smith phd" → capitalized="John Smith Ph.D." + "juan de la vega" → capitalized="Juan de la Vega" · boundary + history: decisions.md#R4 · interacts: R2, R3, R5 · implemented: nameparser/_render.py R5. Rationale: mixed case is evidence that the writer cased the name deliberately, and a repair cannot tell a deliberate spelling from @@ -1118,8 +1131,9 @@ R5. Rationale: mixed case is evidence that the writer cased the name Case repair acts only on a name written entirely in one case. A name written in more than one case is kept as it was written, and whether that casing is right does not enter into it, unless - repair regardless of how the name is cased was asked for. + repair was asked for anyway. "juan mcdonald" → capitalized="Juan McDonald" + "SHIRLEY MACLAINE" → capitalized="Shirley MacLaine" "Shirley Maclaine" → capitalized="Shirley Maclaine" "Shirley Maclaine" → capitalized_forced="Shirley MacLaine" · boundary history: decisions.md#R5 · interacts: R4 · implemented: nameparser/_render.py diff --git a/docs/release_log.rst b/docs/release_log.rst index da67c853..863ae4e5 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -65,6 +65,8 @@ Release Log - Fix a family name made only of particle words reporting no base on the 2.0 API, so the surname vanished from ``family_base`` and, on both APIs, from the initials: ``parse("Anh Do")`` gave family ``Do`` with ``family_base`` ``''`` and initials ``A.``, and under ``Policy(name_order=FAMILY_FIRST)`` ``"Del Toro"`` gave family ``Del`` the same way. The empty base was a 2.0 regression rather than a longstanding bug -- 1.4.0's own guard kept ``HumanName("Anh Do").last_base`` at ``Do``, and the facade has kept it right throughout; what 1.4.0 and 2.1 shared was the missing initial. A particle earns its name by joining forward to the word it modifies, so a particle standing alone in a name part is not doing a particle's work there and reads as an ordinary name word: it anchors the base, leaves the particles view, and contributes an initial. ``"Anh Do"`` is now base ``Do``, initials ``A. D.``; ``"Juan van der"`` is base ``van der``, initials ``J. v. d.``; ``"Nguyen, Van Le"`` initials ``V. L. N.`` where the middle name used to be dropped. Position decides this, not vocabulary -- whether the word is borne as a surname somewhere does not enter into it, which reverses the reasoning ``rules.md#R2`` carried before. The invariant it exists to hold: a non-empty family always has a non-empty base, because a particle needs a base to attach to. Where the particles DO join a name word nothing changes -- ``"Juan de la Vega"`` keeps base ``Vega``, particles ``de la`` and initials ``J. V.``. The parse fields themselves do not move: ``title``, ``given``, ``middle``, ``family``, ``suffix``, ``nickname`` and ``maiden`` are byte-identical over all 751 differential corpus names in all three name orders, and only the derived views and the initials change -- which is why the differential harness, which compares those seven roles, reports no diffs for this at any baseline. One consequence to know about: where the whole family is particles AND a tussenvoegsel was attached to it (#379), the two APIs order the initials differently -- ``parse("der, y van").initials()`` is ``y. d. v.`` against the facade's ``y. v. d.`` -- because ``initials()`` reads tokens in written order while the family FIELD renders folded words first. That affects 87 constructed inputs and no corpus name; it is a pre-existing ordering gap in ``initials()`` that this change makes visible, tracked at #408 (closes #385, closes #402) + - Fix case repair lowercasing the words of a family name made only of particle words, where every other view already reads them as ordinary name words: ``HumanName("ANH DO").capitalize()`` gives ``Anh Do`` where it gave ``Anh do``, and ``"anh van do"`` gives ``Anh Van Do`` where it gave ``Anh van do``. A particle earns its name by joining forward to the word it modifies, so a part whose every word is particle vocabulary leaves none of them anything to join; the fix above already made those words anchor ``family_base`` and contribute initials, and case repair now agrees with them rather than reading the same word two ways. The test is the whole part, not a particle standing alone, which is why the two-word family in ``"anh van do"`` moves along with the one-word family in ``"ANH DO"`` -- the same Vietnamese surname, and a standing-alone test would have read it one way behind a given name and another way alone. This DIFFERS FROM 1.4.0 deliberately and does not restore it: 1.4.0 returned ``Anh do``, lowercasing on vocabulary membership alone. The accepted cost is that a degenerate family which is nothing but particles capitalizes too, so ``"juan van der"`` gives ``Juan Van Der`` where 1.4.0 gave ``Juan van der``. A conjunction is untouched by any of this, so ``"der, y van"`` gives ``y Van Der`` -- the family capitalizing while the conjunction keeps the lowercase it always had; and where the particles DO join a name word nothing changes, ``"juan de la vega"`` still giving ``Juan de la Vega``. Case repair is not one of the seven role fields the differential harness compares, so no gate run can see this change either way and none of its counts move; the ``rules.md#R4`` examples and the v1 capitalization tests are what pin it (closes #407) + - Fix a tussenvoegsel after a family comma being parsed as a middle name. Dutch and Belgian alphabetized listings move the particle behind the given name -- ``"Beethoven, Ludwig van"`` is how ``"Ludwig van Beethoven"`` is filed -- and the trailing particle run was read as a middle name rather than as part of the surname: ``"Beethoven, Ludwig van"`` gave middle ``van``, last ``Beethoven``, and ``"Berg, Jan van der"`` gave middle ``van der``. The run now attaches to the family the comma has already named and renders before it, so those read family ``van Beethoven`` and ``van der Berg`` with the given name unchanged. The derived views move with the parse, so ``family_particles`` is ``van`` and ``family_base`` is ``Beethoven`` where they were empty and ``Beethoven`` before. `#130 `_ asked for the split and got it in 1.3.0 as ``last_base``/``last_prefixes``; 2.0 renamed them ``family_base``/``family_particles``. What was wrong until now was the values they reported for this listing. Both halves of the particle vocabulary attach -- never-given ``de`` and may-be-given ``van`` alike -- because after a comma the family is already named and the particle has no other role to take. Two guards bound it. A name whose only given word is the particle keeps it, so ``"Nguyen, Van"`` still reads given ``Van``: the attachment needs a given word to spare. And where the word is BOTH particle and suffix vocabulary the attachment outranks the post-nominal reading, so ``"Berg, Jan vd"`` reads family ``vd Berg`` where 1.4.0 and 2.1 alike gave suffix ``vd`` -- a trailing abbreviation after a family comma is the tussenvoegsel far more often than the decoration it collides with, and the same shape sweeps in ``mc``, which 2.1 also read as a suffix. ``do`` is in ``SUFFIX_ACRONYMS_AMBIGUOUS`` and 2.1 already read a trailing one as a name word, so it attaches by the plain rule rather than by the override (closes #379, closes #380). Names without the comma are untouched: ``"Ludwig van Beethoven"`` already read family ``van Beethoven`` and is byte-identical. One of the 751 differential corpus names moves, ``"Vega, Juan de la"``, at the 1.4.0, 2.0.0 and 2.1.0 baselines alike - Add ``abd`` to :data:`~nameparser.config.bound_given_names.BOUND_GIVEN_NAMES`, so the spellings that write the article as its own word join like the others do: ``"abd Allah Smith"`` was given ``abd``, middle ``Allah`` and is now given ``abd Allah``. ``abdul``, ``abdel`` and ``abdal`` were already there, and the Arabic-script ``عبد`` has covered the same word since 2.0, so only the Latin spelling was short. The word is also the postnominal ABD ("All But Dissertation") and stays in ``SUFFIX_ACRONYMS``: position tells the two readings apart, so ``"Jane Smith ABD"``, ``"Jane Smith, ABD"`` and ``"Jane Smith A.B.D."`` all still read the credential as a suffix. Making that work needed one fix beyond the wordlist. The join reserves enough words to leave a family name behind, and the reserve counted every piece except titles and suffixes -- so a word in BOTH vocabularies did not count as a name word even when the rule had already claimed it as one, and the join declined silently on the common three-word shape. The claimed piece now counts, which is what the reserve means: it asks whether enough OTHER words are left to spare. No shipped word other than ``abd`` is in both sets, and for any word that is not, the changed expression reduces to the old one -- so nothing else can move, by arithmetic rather than by sampling. The differential corpus agrees at all 751 names, but it holds no name carrying a dual-membership word, so that agreement tests the corpus rather than the change (#400) diff --git a/nameparser/_render.py b/nameparser/_render.py index b4b3c561..51a24edf 100644 --- a/nameparser/_render.py +++ b/nameparser/_render.py @@ -123,13 +123,26 @@ def initials(name: ParsedName, spec: str, delimiter: str, separator: str) -> str return _format_spec(spec, values, "initials", _INITIALS_KEYS) -def _cap_word(word: str, role: Role, lex: Lexicon) -> str: +def _cap_word(word: str, role: Role, tags: frozenset[str], + lex: Lexicon) -> str: # v1 cap_word order: particle/conjunction rule first, then the # exceptions map, then Mac/Mc, then str.capitalize normalized = _normalize(word) + # rules.md#R4: "a part whose every word is particle vocabulary is + # repaired as ordinary name words, since none of them is doing a + # particle's work there" -- UNJOINED_TAG is that mark (#407). + # Only the PARTICLE conjunct is gated on it, and gating the other + # would be a NO-OP rather than a mistake: the mark is applied to a + # part only when every word in it carries "particle" (_post_rules + # and _types._remarked both), a conjunction carries "conjunction" + # instead, and the two vocabularies are disjoint -- so a part + # holding a conjunction is never marked and no conjunction token + # can carry the mark. Left ungated because that is the smaller + # predicate, not because a case turns on it. # v1's is_conjunction excludes initials: 'E.' in 'Scott E. Werner' # is an initial, not the conjunction 'e' (pinned live 2026-07-17) - if ((normalized in lex.particles and role in (Role.MIDDLE, Role.FAMILY)) + if ((normalized in lex.particles and role in (Role.MIDDLE, Role.FAMILY) + and UNJOINED_TAG not in tags) or (normalized in lex.conjunctions and not _INITIAL.fullmatch(word))): return word.lower() @@ -146,14 +159,15 @@ def _cap_word(word: str, role: Role, lex: Lexicon) -> str: return word.capitalize() -def _cap_text(text: str, role: Role, lex: Lexicon) -> str: +def _cap_text(text: str, role: Role, tags: frozenset[str], + lex: Lexicon) -> str: # word-by-word within the token text: hyphenated names capitalize # both sides ("macdole-eisenhower" -> "MacDole-Eisenhower") - return _WORD.sub(lambda m: _cap_word(m.group(0), role, lex), text) + return _WORD.sub(lambda m: _cap_word(m.group(0), role, tags, lex), text) -# rules.md#R4: "case repair returns a repaired copy — vocabulary -# exceptions (McDonald) included — and never mutates the parse" +# rules.md#R4: "case repair returns a repaired copy and never mutates +# the parse" def capitalized(name: ParsedName, lexicon: Lexicon | None, *, force: bool) -> ParsedName: """Case-fixing transform -> new ParsedName, same spans, new token @@ -175,7 +189,7 @@ def capitalized(name: ParsedName, lexicon: Lexicon | None, *, if not force and joined not in (joined.upper(), joined.lower()): return name new_tokens = tuple( - Token(_cap_text(t.text, t.role, lex), t.span, t.role, t.tags) + Token(_cap_text(t.text, t.role, t.tags, lex), t.span, t.role, t.tags) for t in name.tokens) # equal tokens (possible only for synthetic span=None duplicates) # collapse to one mapping entry -- benign: the rebuilt ambiguity diff --git a/nameparser/_types.py b/nameparser/_types.py index 83de8091..934627fb 100644 --- a/nameparser/_types.py +++ b/nameparser/_types.py @@ -98,10 +98,10 @@ def __add__(self, other: object) -> NoReturn: # type: ignore[override] #: A name part whose every word is particle vocabulary is a part #: where none of them is doing a particle's work -- nothing joins them -#: to a name -- so THREE views read them as ordinary name words: they -#: anchor `family_base`, drop out of `family_particles`, and -#: contribute initials (rules.md#R2). Capitalization does not consult -#: the mark and still lowercases them, which #407 tracks. +#: to a name -- so FOUR views read them as ordinary name words: they +#: anchor `family_base`, drop out of `family_particles`, contribute +#: initials (rules.md#R2), and capitalize like any other name word +#: rather than being lowercased as particles (#407). #: MARKED rather than untagged: `particle` is stable API and says the #: word IS particle vocabulary wherever it lands, which stays true, and #: keeping it leaves a later rule free to report the fork this decides. diff --git a/tests/test_capitalization.py b/tests/test_capitalization.py index fd60c613..012f4ebd 100644 --- a/tests/test_capitalization.py +++ b/tests/test_capitalization.py @@ -142,3 +142,109 @@ def test_capitalize_prefix_clash_on_first_name(self) -> None: hn = HumanName("van nguyen") hn.capitalize() self.m(str(hn), 'Van Nguyen', hn) + + # #407, rules.md#R4. The family is one word and that word is + # particle vocabulary, so nothing joins it to a name and it is not + # doing a particle's work: family_base and initials have read it as + # an ordinary name word since #404, and case repair now agrees. + # Deliberately the v1 facade, and deliberately single-case input -- + # this MOVES v1-visible behavior (1.4.0 returns 'Anh do'), and the + # mixed-case gate would return the input untouched (rules.md#R5). + def test_capitalize_all_particle_family_is_a_name_word(self) -> None: + hn = HumanName('ANH DO') + hn.capitalize() + self.m(str(hn), 'Anh Do', hn) + + # The reason the mark is read for the WHOLE PART rather than for a + # particle standing alone: this family is 'van do', two particle + # words and neither of them alone. A standing-alone rule would + # capitalize the name above and leave this one lowercased, reading + # the same surname two ways. + def test_capitalize_all_particle_family_of_two_words(self) -> None: + hn = HumanName('anh van do') + hn.capitalize() + self.m(str(hn), 'Anh Van Do', hn) + + # The recorded negative control for the two above (AGENTS.md's + # guard-test convention): here 'de la' has 'vega' to join to, so + # the particles are doing a particle's work and stay lowercase. + # This FENCES the scope of #407 rather than pinning the fix -- it + # passes both before and after the change, and fails only if the + # new clause is widened to reach working particle runs. + def test_capitalize_working_particle_stays_lowercase(self) -> None: + hn = HumanName('juan de la vega') + hn.capitalize() + self.m(str(hn), 'Juan de la Vega', hn) + + # The corpus name that carries both halves of the predicate at + # once, and the one the release-log bullet asserts: the family + # `van der` is all particles and capitalizes, while the `y` keeps + # its lowercase. This DOES pin the fix -- deleting the tag consult + # gives 'y van der' -- but it is worth being exact about what it + # does NOT pin, since a reviewer proposed it for that. It cannot + # witness the conjunction conjunct being left ungated, and neither + # can any other name: the mark is applied to a part only where + # EVERY word in it carries "particle" (_pipeline/_post_rules and + # _types._remarked alike), `particles` and `conjunctions` are + # disjoint sets, and so a part holding a conjunction is never + # marked and no conjunction token can ever carry the mark. Gating + # that conjunct too would be a semantic no-op, which is why + # mutating it kills nothing. The `y` here is a GIVEN-part word + # besides, not a word of the all-particle family. + def test_capitalize_all_particle_family_beside_a_conjunction(self) -> None: + hn = HumanName('der, y van') + hn.capitalize() + self.m(str(hn), 'y Van Der', hn) + + # rules.md#R5's override, stated as a property rather than a + # single row: mixed case is the writer making an explicit choice + # and repair defers to it, so asking for repair REGARDLESS should + # ignore the input's case entirely -- one name, one repaired + # string, however it was written. + # + # Measured over the 1094-name differential corpus (2026-08-29), + # because the promise is nearly true and the exceptions are the + # whole story. `capitalize(force=True)` differs from uppercasing + # the input and calling `capitalize()` for 63 of 1094 names, and + # from lowercasing it for 16 -- so UPPERCASE IS THE WORSE + # DIRECTION, not the clean one. Nor are the misses merely + # parse-level: of the 63, only 25 move a role, and the other 38 + # parse byte-identically and differ inside the repair itself. + # Recompute by running both forms over the four corpus files + # deduped and diffing. + # + # The mechanism is v1's initial carve-out, which _cap_word + # documents a few lines from the predicate: a conjunction is not + # lowercased where it is written initial-shaped, and + # initial-shaped means one CAPITAL letter. Uppercase a name and + # every one-letter conjunction becomes an initial; lowercase one + # and a middle initial `E` becomes the Italian conjunction. So + # the property is pinned over names carrying no single-letter + # word whose class case decides, and the exception is pinned + # beside it as data rather than left to be rediscovered. + def test_forcing_repair_ignores_the_case_it_was_given(self) -> None: + for name in ('shirley maclaine', 'juan de la vega', 'anh van do', + 'donovan mcnabb-smith', 'jane smith phd', + 'lt. gen. john a. kenneth doe iv'): + forced = HumanName(name) + forced.capitalize(force=True) + upper = HumanName(name.upper()) + upper.capitalize() + lower = HumanName(name.lower()) + lower.capitalize() + self.m(str(upper), str(forced), upper) + self.m(str(lower), str(forced), lower) + + # The recorded exception to the property above, and the reason it + # is scoped rather than universal. 1.4.0 does exactly this too + # (measured on the released wheel: 'JUAN Y GARCIA' capitalizes to + # 'Juan Y Garcia'), so it is inherited behavior and not a 2.x + # regression -- recorded here, deliberately not fixed here. + def test_a_one_letter_conjunction_is_case_sensitive_to_repair(self) -> None: + lowered = HumanName('juan y garcia') + lowered.capitalize(force=True) + self.m(str(lowered), 'Juan y Garcia', lowered) + uppered = HumanName('JUAN Y GARCIA') + uppered.capitalize() + # 'Y' is initial-shaped, so the conjunction rule declines it + self.m(str(uppered), 'Juan Y Garcia', uppered) diff --git a/tools/differential/compare.py b/tools/differential/compare.py index 283b8565..4d310e6c 100644 --- a/tools/differential/compare.py +++ b/tools/differential/compare.py @@ -373,7 +373,7 @@ def _is_latin_only(name: str) -> bool: "corpus.jsonl": 480, # 486 today, from v1's banks at a pinned ref "corpus_cjk.jsonl": 95, # 98 today, generated from the case table "corpus_issues.jsonl": 370, # 381 today, harvested and append-only - "corpus_rules.jsonl": 150, # 235 today, generated from rules.md + "corpus_rules.jsonl": 150, # 240 today, generated from rules.md } diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 3d0ff383..f98f7bbb 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -5,6 +5,7 @@ "'Smitty' Sir John" "(" ".," +"ANH DO" "Abu Bakar" "Abu Bakar Salim" "Ali Ahmad Vali oglu" @@ -130,6 +131,7 @@ "Nguyễn Thị Minh Khai" "Nguyễn, Thị Vân" "Rev. John Smith" +"SHIRLEY MACLAINE" "Salam, abd Allah" "Sean O'Connor" "Sheik Abu Bakar" @@ -189,6 +191,7 @@ "abdul Smith V" "abdul V Smith" "abdul salam ahmed salem" +"anh van do" "de" "de Mesnil Juan" "de la Cruz Juan Carlos" @@ -196,6 +199,8 @@ "de la Vega y Santos Juan" "de los Santos" "ibn Awf abdul Rahman" +"john smith phd" +"juan de la vega" "juan mcdonald" "mohamad ali smith" "née Jones" From 46185d3854b48891728022b8c20a69ceedfc1dcb Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sat, 29 Aug 2026 12:53:25 -0700 Subject: [PATCH 3/6] fix(post_rules): P6's attachment reports the fork it decides (#405) rules.md#A1 requires a genuinely uncertain reading to carry a report. P6's tussenvoegsel attachment creates such a fork and reported nothing: 'Van Johnson' reports particle-or-given while 'Nguyen, Thi Van' decided the same fork silently and lost the given name. Two kinds, keyed on the reading the attachment OVERRODE rather than on what the words are. Where assign had read the run as a post-nominal, that is what was declined (suffix-or-name); otherwise, where the run holds an ambiguous particle, the name-word reading was declined (particle-or-given); otherwise nothing was decided and nothing is reported. Keying on vocabulary instead gets both collision words backwards: 'do' is suffix vocabulary yet reports particle-or-given, having never been read as a credential, and 'Berg, Jan de vd' reports nothing though 'vd' is. Both are pinned in cases.py. The arms are ordered, not asserted disjoint. The plan's premise -- a suffix-vocabulary particle arrives suffix-roled and IS the whole run -- holds for the shipped vocabulary (0 both-arm runs over 29,400 generated names) but not for the rule: a caller's Lexicon can put one word in both sets, and the assert fired on 'Berg, Jan zz'. A1 forbids raising from a stage, so the suffix arm wins by statement. assign's 'deliberately not emitted' scoping was true of the family -- which the comma fixes -- and not of the particle behind it. post_rules gains ambiguities in the stage ownership map. Ledger: no new rule. Against 2.0.0 and 2.1.0 these seven names already diff in middle/family (or family/suffix) from #379 and #380, and classify matches on `fields` being a superset of the whole diff, so an _ambiguities-only rule explains none of them. The two existing rules are widened instead, in both ledgers; each still passes #452's declared == union check. 1.4.0 needs nothing. Gate output is byte-identical before and after at all three baselines: 1094 names, 229/194/102 intentional, 0 unexplained, no per-heading count moved. Co-Authored-By: Claude Opus 5 --- AGENTS.md | 2 +- docs/design/decisions.md | 8 +++ docs/design/rules.md | 28 ++++++-- docs/release_log.rst | 2 + nameparser/_pipeline/_assign.py | 6 +- nameparser/_pipeline/_post_rules.py | 73 +++++++++++++++++--- nameparser/_pipeline/_state.py | 8 ++- tests/v2/cases.py | 67 ++++++++++++++++-- tests/v2/pipeline/test_post_rules.py | 32 +++++++++ tests/v2/pipeline/test_state.py | 6 +- tests/v2/test_ledger_guards.py | 8 +-- tools/differential/corpus_rules.jsonl | 1 + tools/differential/expected_since_2.0.0.toml | 24 ++++++- tools/differential/expected_since_2.1.0.toml | 33 ++++++++- 14 files changed, 263 insertions(+), 35 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 2be01ec0..137597f4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -249,7 +249,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These - **Method organization**, fixed section order in every class: fields + `__post_init__` validation → alternative constructors → dunders (construction/equality → protocol → operators) → properties → public methods by concern (access → editing → comparison → rendering delegates) → private helpers last, except a helper serving exactly one section may sit at that section's head. Sanctioned deviation, facade layer only: `HumanName` and the shim `Constants` organize by v1 concern groups (`# -- render defaults --`, `# -- config / parsing --`, `# -- fields --`, ..., dunders and pickle last) — the classes mirror v1's own surface and die in 3.0; the canonical order still binds every core type. - **Validation is eager and fail-loud**: every `raise` states the offending value, the expected form, and the fix. Exception taxonomy: wrong type — including wrong element type inside a collection, bare `str` where an iterable of strings is expected, or a `Mapping` where a plain iterable is expected — raises `TypeError`; well-typed but unacceptable values raise `ValueError`; failed enum lookups stay `ValueError` for any input (stdlib `EnumType` precedent). **When the message hands the reader code to paste, that code has to survive a type checker** — nameparser ships `py.typed`. #337's segmenterless warning offered `Policy(segment_scripts=())`, an `arg-type` error, because these fields are annotated with what they STORE rather than everything the constructor accepts. Prefer the `frozenset()` / `()` spellings in messages and docstrings, and pin the offered spelling in a test — the warning tests matched on `ja_segmenter` and never checked the actionable half of the message. **A warning emitted in `Parser.__post_init__` needs `parser_for` to re-emit it from its own frame** (the `catch_warnings(record=True)` block at its return): `__post_init__`'s `stacklevel` is sized for direct `Parser(...)` construction, and through `parser_for`'s extra frame the default one-line rendering attributes the warning to the library's own `return Parser(...)` — the exact call the message tells the user to change becomes invisible. No single stacklevel serves both entry points; a new construction warning gets the re-emission for free, but a new CONSTRUCTION SITE for `Parser` inside this package needs its own re-emission or its callers get library-attributed warnings (#337 review). - **Guard, hint, and emit for the WHOLE family, and parametrize the test over it**: a check added to one member of a set belongs on all of it, and the test must sweep the family, not one example. This session shipped `_reject_str_and_mapping` on `Policy` but not `PolicyPatch`, the bytes decode hint on three of five config entry points, and a regex-sync roster missing four of its copies — each a separate follow-up bug that a `{class} × {field} × {bad-value}` parametrization would have caught and a per-example test hid. When you find you're guarding member N, grep for the other members first. -- **Ambiguities are emitted at the DECISION site**: an `Ambiguity` records a fork the parse had to call, not a token that sits in an ambiguous vocabulary. Emit where the branch is taken — the trailing-suffix peel in `_assign`, the delimiter escape's follow-up in `classify` — never by scanning for a `vocab:*-ambiguous` tag. The same tagged token is a genuine fork in one position and unremarkable in another (`do` mid-name in "Joao da Silva do Amaral de Souza" chooses nothing). **A branch that runs but changes nothing is not a decision either** -- the prefix chain's `merge(k, j)` executes even when `j == k + 1`, folding a piece into itself, and keying on "the code got here" reported a fork for all 39 ambiguous particles on "Do Van Jr." (`Dr.` when that was written, before #367 made a plain title transparent and put the shape out of the loop's reach entirely), where the particle stayed a lone leading name piece — the GIVEN name under the default order, the family name under `FAMILY_FIRST` — and `_assign` reported the same token again. Check that the branch actually claimed something (`j > k + 1`) before recording. Structure also structure often settles the question before it arises, which is why `PARTICLE_OR_GIVEN` is deliberately not emitted on the `FAMILY_COMMA` path and `SUFFIX_OR_NAME` is not emitted for "Ma, Jack". The decision site also has the token index and the detail text in hand, which the tag scan would have to reconstruct. **If a fork's two branches are taken in DIFFERENT stages, every one of them needs the emitter** -- `PARTICLE_OR_GIVEN` is decided in `_assign` when the ambiguous particle stays a lone leading piece and in `_group` when something shifts it off the name's leading piece and the prefix chain claims it, so both report; for two years only the first did. What can still do the shifting is narrow, and #367 is why: a plain title no longer can (`Dr. Van Johnson` reads as `Van Johnson` does and reports from `_assign`), so the `_group` emitter needs a word that is BOTH a title and a particle — `st`, `do` and `freiherr` in the default vocabulary, or any overlap a caller's config creates — standing ahead of the chained particle with nothing but titles before it. That word need not LEAD the input: `Dr. Do van Johnson` reaches the emitter with a plain title in front of it, and `Do St Johnson` reaches it with the chained particle itself in both vocabularies. `Freiherr von Richthofen` is the canonical shape rather than the only one; when checking whether that emitter is dead, a both-vocabulary word is the thing to look for, and the answer is that it is not dead. The stage-ownership map in `tests/v2/pipeline/test_state.py` must list `ambiguities` for each such stage, and it passes vacuously until a case row exercises the path, so add the row too. Report BOTH directions of a two-way fork — "John Smith MA" (read as a suffix) and "Jack MA" (read as the family name) are equally guesses. Every kind needs a trigger in `tests/v2/test_contracts.py::_AMBIGUITY_TRIGGERS` (an explicit `None`, strict-xfail, while reserved), and case-table rows pin expected kinds exactly, so a new emitter shows up in both immediately. **Pin the decision, not the vocabulary**: the only titled-particle test used an UNAMBIGUOUS particle, so it walked the right code path and proved nothing about the branch under test -- two criticals passed 1539 tests. A row contrasting the two readings ("John Smith V" against "John Smith B") is what makes an emitter's absence meaningful. +- **Ambiguities are emitted at the DECISION site**: an `Ambiguity` records a fork the parse had to call, not a token that sits in an ambiguous vocabulary. Emit where the branch is taken — the trailing-suffix peel in `_assign`, the delimiter escape's follow-up in `classify` — never by scanning for a `vocab:*-ambiguous` tag. The same tagged token is a genuine fork in one position and unremarkable in another (`do` mid-name in "Joao da Silva do Amaral de Souza" chooses nothing). **A branch that runs but changes nothing is not a decision either** -- the prefix chain's `merge(k, j)` executes even when `j == k + 1`, folding a piece into itself, and keying on "the code got here" reported a fork for all 39 ambiguous particles on "Do Van Jr." (`Dr.` when that was written, before #367 made a plain title transparent and put the shape out of the loop's reach entirely), where the particle stayed a lone leading name piece — the GIVEN name under the default order, the family name under `FAMILY_FIRST` — and `_assign` reported the same token again. Check that the branch actually claimed something (`j > k + 1`) before recording. Structure also structure often settles the question before it arises, which is why `PARTICLE_OR_GIVEN` is not emitted on the `FAMILY_COMMA` path's WHOLLY-FAMILY read -- the comma fixed which piece is the family -- and `SUFFIX_OR_NAME` is not emitted for "Ma, Jack". Read that scope narrowly: the comma settles nothing about a particle trailing the given name, so P6's attachment in `post_rules` decides that fork on the same path and reports it (#405), in the kind naming the reading it OVERRODE, which is the reading assign made and not the word's vocabulary: `SUFFIX_OR_NAME` where assign had read the run as a post-nominal (`vd`, `mc`), else `PARTICLE_OR_GIVEN` where the run holds an ambiguous particle (`van`, and `do`, which is in the suffix vocabulary too but in its AMBIGUOUS half, so no credential reading was overridden), else silence. The decision site also has the token index and the detail text in hand, which the tag scan would have to reconstruct. **If a fork's two branches are taken in DIFFERENT stages, every one of them needs the emitter** -- `PARTICLE_OR_GIVEN` is decided in `_assign` when the ambiguous particle stays a lone leading piece, in `_group` when something shifts it off the name's leading piece and the prefix chain claims it, and in `post_rules` when P6's attachment takes a trailing particle into the family after a comma, so all three report; for two years only the first did. What can still do the shifting is narrow, and #367 is why: a plain title no longer can (`Dr. Van Johnson` reads as `Van Johnson` does and reports from `_assign`), so the `_group` emitter needs a word that is BOTH a title and a particle — `st`, `do` and `freiherr` in the default vocabulary, or any overlap a caller's config creates — standing ahead of the chained particle with nothing but titles before it. That word need not LEAD the input: `Dr. Do van Johnson` reaches the emitter with a plain title in front of it, and `Do St Johnson` reaches it with the chained particle itself in both vocabularies. `Freiherr von Richthofen` is the canonical shape rather than the only one; when checking whether that emitter is dead, a both-vocabulary word is the thing to look for, and the answer is that it is not dead. The stage-ownership map in `tests/v2/pipeline/test_state.py` must list `ambiguities` for each such stage, and it passes vacuously until a case row exercises the path, so add the row too. Report BOTH directions of a two-way fork — "John Smith MA" (read as a suffix) and "Jack MA" (read as the family name) are equally guesses. Every kind needs a trigger in `tests/v2/test_contracts.py::_AMBIGUITY_TRIGGERS` (an explicit `None`, strict-xfail, while reserved), and case-table rows pin expected kinds exactly, so a new emitter shows up in both immediately. **Pin the decision, not the vocabulary**: the only titled-particle test used an UNAMBIGUOUS particle, so it walked the right code path and proved nothing about the branch under test -- two criticals passed 1539 tests. A row contrasting the two readings ("John Smith V" against "John Smith B") is what makes an emitter's absence meaningful. - **A kind is worth adding only if a reader would hesitate too**: the test is not "does the code take a branch" but whether a person reading that input would genuinely be unsure. "Smith, John V" reads as a middle initial to anyone -- the comma settles it -- so reporting it would be noise that teaches callers to ignore the field, which costs more than the missing report. Reachability of the second branch is necessary, not sufficient. Prefer leaving a fork silent and documenting the omission (see the comma paths in concepts.rst) over emitting on input nobody finds ambiguous. - **Parser owns config-dependent conveniences**: `Parser.matches`/`Parser.capitalized`/`Parser.revise` exist because the `ParsedName` equivalents fall back to DEFAULT config for str/omitted arguments (documented loudly in both docstrings). `revise` harvests tokens from a full sub-parse of each replacement value (tags kept minus `FOLDED_TAG`, roles forced, ambiguities discarded); the merge tail is shared with `replace()` via `ParsedName._with_field_tokens`. `Parser.capitalized` delegates through `name.capitalized(self.lexicon)` specifically so `_parser` never imports `_render` — keep it that way. - **Per-word vocabulary fields warn on multi-word entries** (`_normset`/`_normpairs` via `_warn_dead_entry`, UserWarning, never a raise — see the given_name_titles Gotcha for why raising is wrong). `given_name_titles` is the one multi-word-matched field and is exempt; `_edit` passes `warn=False` (add() warns once via the new instance's `__post_init__`; remove() stores nothing). The default vocabulary and every locale pack must stay warning-free (`test_default_lexicon_builds_warning_free`, `test_pack_vocabulary_entries_are_single_words`). diff --git a/docs/design/decisions.md b/docs/design/decisions.md index c6385545..b15d0df1 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -110,6 +110,14 @@ Open: [#380](https://github.com/derek73/python-nameparser/issues/380) covers "Be - 2026-08-22 (late review of #425's docs, PR #426) — the bound join runs before P6 sees the name, and P6 now says so. P5's new title-word clause took `do`, `freiherr` and `st` — title AND particle vocabulary, until #296's audit took `do` out of TITLES, since when the join takes `do` as the particle it is — away from the attachment: `Berg, abdul do` had read family 'do Berg' since #379 and reads given 'abdul do' now. That is 1.4.0's reading, and it is what every other particle already did: `Berg, abdul van` read given 'abdul van' at 1.4.0, on 2.0 and 2.1, and on master before #426 — the old reserve's title exclusion had held only the title-word trio back. The precedence is the stage order (P5 joins in group; P6 attaches in post_rules, over what is left) rather than a judgement about tussenvoegsels behind bound words, and both rules carry it as Accepted with `Berg, abdul van` as the example, parity at every baseline. The precedence has one exception, found by the review of #424's docs when the Accepted clause claimed the attachment "never sees" the particle: a trailing particle of the unambiguous suffix vocabulary too — `vd`, `mc`; `do` is ambiguous, and a name word to the join — is a suffix piece to the join, which never takes one (#421), so the attachment does see it, and `Berg, abdul vd` reads family 'vd Berg' where 1.4.0 read given 'abdul vd'. Both rules say so, with the example; at 1.4.0 the fix(comma-precomma-family) rule already reaches it (a rule written for it was shadowed, and dropped). +- 2026-08-29 #405 (the A1 deviation, reversed) — the attachment reports the fork it decides, and the Accepted clause saying it does not is deleted rather than re-tracked. What that clause rested on was assign's own scoping comment: after a comma that fixed the family, a leading given-position particle is "not meaningfully ambiguous". True of the FAMILY — the comma names which piece that is — and not of the particle standing behind the given name, which P6 decides on its own a stage later, out of the same vocabulary that makes `Van Johnson` report `particle-or-given` with no comma in sight. The old state was the shape A1 forbids: `Van Johnson` reported the fork, `Nguyen, Thi Van` decided the identical fork silently and lost a given name doing it. + TWO kinds, keyed on WHAT THE ATTACHMENT OVERRODE and not on what the words are, because `mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE` asks the emitted fork to be the one taken. Where assign had read the run as a post-nominal, the overridden reading is the post-nominal — `suffix-or-name`, which is exactly the S2 precedence this rule already claims in writing. Otherwise, where the run holds an ambiguous particle (`van`, `Van`, `la`), the overridden reading is that word as a name word — `particle-or-given`. One kind for both would assert "particle or given" about a word the parse had read as a credential and never weighed as a name, naming a branch nobody considered. + The key is the READING and not the vocabulary, and the docs review caught a first draft that said otherwise ("a word that is also suffix vocabulary declines a post-nominal"). That wording is wrong for both of the words it most obviously describes. `do` is the third member of the particle ∩ suffix-acronym overlap and reports `particle-or-given`: it sits in the AMBIGUOUS acronym half, which had already left it a name word, so no credential reading was ever taken for the attachment to override. And a run of several words is read whole — `Berg, Jan de vd` reports NOTHING though `vd` is suffix vocabulary, because behind `de` the run was read as name words, where the lone `Berg, Jan vd` reports `suffix-or-name`. Both are pinned in tests/v2/cases.py; neither is in any corpus, so nothing else would have caught the wrong statement. + The two arms are ORDERED rather than asserted disjoint, and that is a correction to the plan, which recorded them as disjoint by construction ("a suffix-vocabulary particle arrives suffix-roled and IS the whole run"). That premise is a property of the SHIPPED vocabulary, not of the rule: `vd` and `mc` are the only words in both the particle and the unambiguous suffix sets and neither is ambiguous particle vocabulary, so no shipped input reaches both arms — measured, 0 both-arm runs over 29,400 generated names (every ordered pair of the 70-word particle vocabulary in six family-comma shapes: bare, behind a given word, behind a title, before `Jr.`, before `MD`, and with the pair split across the comma). But a caller's `Lexicon` may put one word in both, and the `assert` the plan asked for DID fire on `Berg, Jan zz` under a Lexicon adding `zz` to `particles_ambiguous` and `suffix_acronyms`. `rules.md#A1` says parsing never fails on any input and names exactly one exception, a caller's segmenter; an `AssertionError` out of a stage would be a second. So the code decides instead: the suffix arm wins, because assign had read the word as a post-nominal and the name-word reading was never on the table for the attachment to decline. Pinned in tests/v2/pipeline/test_post_rules.py. + The never-given half stays SILENT, and that is the substantive half of the decision. `Jong, Piet de` has no live alternative: `de` is not a name in any reading and is not suffix vocabulary either, so the attachment chose between one reading and nothing, which is not a fork. Reporting it would be "the code got here" keying, the failure this project already has on record (a fork reported for every ambiguous particle at once on `Dr. Van Jr.` — the count is left out on purpose, since decisions.md#vocabulary-collisions says to recompute before quoting it). + Measured over the 1094-name differential corpus: P6's attachment fires on 9 names, of which 5 take the ambiguous arm (`Beethoven, Ludwig van`, `Berg, Jan van der`, `Nguyen, Thi Van`, `Vega, Juan de la`, `der, y van`), 2 the suffix arm (`Berg, Jan vd`, `Berg, abdul vd`) and 2 stay silent (`Jong, Anke de`, `Jong, Piet de`). Seven names gain an ambiguity kind and no role field moves that was not already moving. Read that seven as evidence about the corpus rather than about the blast radius, the caution the 2026-08-16 entry above already gives for this rule: 304 of the 1094 names carry a comma and only 13 end in a particle, 11 of those being rows #379, #380 and this change added themselves. The reach is a SHAPE — every family-comma listing ending in an ambiguous particle, which is to say ordinary Dutch, Flemish and unaccented Vietnamese listings — and a caller holding such records sees a report on all of them, not on seven. The LEDGER shape that follows from that is not the one the plan predicted, and the difference is worth recording because it will recur for any report added to a rule whose fields already move. A new `fields = ["_ambiguities"]` rule explains none of the seven: `classify` matches on a rule's `fields` being a SUPERSET of the whole diff, and against 2.0.0 and 2.1.0 these names diff in `middle`/`family` (or `family`/`suffix`) from #379 and #380 as well. So the repair is to WIDEN those two existing rules to `{middle, family, _ambiguities}` and `{family, suffix, _ambiguities}` in both ledgers, not to add a rule. Each widened rule still passes the #452 equality check, since the union over the names it explains is exactly what it declares — the two silent `Jong` names contribute `{middle, family}` and the five ambiguous ones contribute `_ambiguities` on top. 1.4.0 needs nothing at all, having no ambiguity surface to diff. Gate before and after: byte-identical output at all three baselines apart from the provenance header — 1094 names, 229/194/102 intentional, 0 unexplained, and no per-heading count moved. The seven names were already in the diff; only their field sets grew. + The most visible consequence, and it is not the Vietnamese one: `Beethoven, Ludwig van` — the textbook-correct Dutch listing, read exactly right — now carries a report. That is honest rather than regrettable. The parser cannot separate it from `Nguyen, Thi Van`; the two are the same string shape over the same vocabulary, and a report on one is a report on the other. A caller who wants only the doubtful cases has nothing here to filter on, because there is nothing to filter on in the input either. + ### M2 — the maiden-marker rule - 2026-07-03 (maiden-bucket design; #274 filed 2026-07-07, landed in the v2 core, PR #288) — the marker takes everything after it up to a trailing suffix, greedily: "née Jones Smith" is a two-word maiden name, matching how the marker is actually used in running text. The marker itself is dropped as structural, like a delimiter character. diff --git a/docs/design/rules.md b/docs/design/rules.md index 0fe3330a..c594c581 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -397,11 +397,19 @@ P6. Rationale: a particle ending the name has nothing to link traditions write the same string. "Nguyen, Thi Van" → family="Van Nguyen" "Nguyễn, Thị Vân" → family="Nguyễn" - Accepted: no ambiguity is reported for that collision, which A1 - would call for. The fork is decided here, and assign's emitter is - scoped to the no-comma shapes on the reasoning that a comma has - fixed the family — true of the family, not of the particle behind - it. Tracked at #405. + The fork is reported, in the kind naming the reading this + attachment OVERRODE — what the run was read as before it fired, + not what its words are. Where the run was read as a post-nominal, + which is the S2 reading this rule outranks, the report is + suffix-or-name. Otherwise, where the run holds an ambiguous + particle, the reading overridden is that word as a name word, and + the report is particle-or-given. Where the attachment overrode + neither — a never-given particle read as a name word, which it + could never have been — nothing was decided and nothing is + reported. + "Nguyen, Thi Van" → ambiguities=("particle-or-given",) + "Berg, Jan vd" → ambiguities=("suffix-or-name",) + "Jong, Piet de" → ambiguities=() · boundary Accepted: the colliding spelling has a format that reads correctly, and it is ONE order, not both: FAMILY_FIRST still sends the given name to the middle, and only @@ -421,14 +429,20 @@ P6. Rationale: a particle ending the name has nothing to link weighed; mc inherits it, which is the shape's cost and is tracked with the other contested memberships. `do` sits in the AMBIGUOUS acronym half and was already read as a name word - there, so the precedence decides nothing for it. + there, so the precedence decides nothing for it — and because + the report names the reading OVERRIDDEN, `do` reports + particle-or-given while vd and mc report suffix-or-name. The + same distinction decides a run of several words: a run read as + name words reports on its ambiguous member if it has one and + otherwise reports nothing, whatever its words could have been + read as standing alone. Both are pinned in tests/v2/cases.py. Accepted: a bound given word ahead of the trailing particle takes it as its pair first (P5), so the attachment never sees it — unless the particle is of the unambiguous suffix vocabulary too (vd, mc), which the join declines and the attachment then takes. "Berg, abdul van" → given="abdul van" "Berg, abdul vd" → family="vd Berg" - history: decisions.md#P6 · interacts: C1, P1, S2, P5 · implemented: nameparser/_pipeline/_post_rules.py + history: decisions.md#P6 · interacts: A1, C1, P1, S2, P5 · implemented: nameparser/_pipeline/_post_rules.py ## Suffixes: generational & credentials (S) diff --git a/docs/release_log.rst b/docs/release_log.rst index 863ae4e5..839bfc98 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -67,6 +67,8 @@ Release Log - Fix case repair lowercasing the words of a family name made only of particle words, where every other view already reads them as ordinary name words: ``HumanName("ANH DO").capitalize()`` gives ``Anh Do`` where it gave ``Anh do``, and ``"anh van do"`` gives ``Anh Van Do`` where it gave ``Anh van do``. A particle earns its name by joining forward to the word it modifies, so a part whose every word is particle vocabulary leaves none of them anything to join; the fix above already made those words anchor ``family_base`` and contribute initials, and case repair now agrees with them rather than reading the same word two ways. The test is the whole part, not a particle standing alone, which is why the two-word family in ``"anh van do"`` moves along with the one-word family in ``"ANH DO"`` -- the same Vietnamese surname, and a standing-alone test would have read it one way behind a given name and another way alone. This DIFFERS FROM 1.4.0 deliberately and does not restore it: 1.4.0 returned ``Anh do``, lowercasing on vocabulary membership alone. The accepted cost is that a degenerate family which is nothing but particles capitalizes too, so ``"juan van der"`` gives ``Juan Van Der`` where 1.4.0 gave ``Juan van der``. A conjunction is untouched by any of this, so ``"der, y van"`` gives ``y Van Der`` -- the family capitalizing while the conjunction keeps the lowercase it always had; and where the particles DO join a name word nothing changes, ``"juan de la vega"`` still giving ``Juan de la Vega``. Case repair is not one of the seven role fields the differential harness compares, so no gate run can see this change either way and none of its counts move; the ``rules.md#R4`` examples and the v1 capitalization tests are what pin it (closes #407) + - Fix a tussenvoegsel attached to the family name after a comma deciding a genuinely uncertain reading and reporting nothing. ``"Van Johnson"`` reports a ``PARTICLE_OR_GIVEN`` ambiguity -- ``Van`` is a Dutch particle and a Vietnamese given name, and the parser has to pick one -- while ``"Nguyen, Thi Van"`` picked the same word the same way, silently, and lost the given name doing it. The attachment now reports the fork it decides, in the kind that names the reading it declined. A particle that could be an ordinary name reports ``PARTICLE_OR_GIVEN``: ``"Nguyen, Thi Van"``, ``"Berg, Jan van der"`` and ``"Vega, Juan de la"`` each gain one, the ``detail`` naming the ambiguous word. A particle the parser had already read as a post-nominal reports ``SUFFIX_OR_NAME`` instead, because the credential reading is what the attachment overrode: ``"Berg, Jan vd"`` gains one, ``vd`` being read as *van der* rather than as the Volunteer Decoration. Which kind you get follows the reading that was overridden rather than the word's vocabulary, so ``"Berg, Jan do"`` reports ``PARTICLE_OR_GIVEN`` even though ``do`` is a postnominal too -- it was already being read as a name word, so no credential reading was overridden. A particle where nothing was overridden reports nothing at all: ``"Jong, Piet de"`` and ``"Jong, Anke de"`` are unchanged, ``de`` being no name in any reading and no postnominal either, and so is ``"Berg, Jan de vd"``, whose run was read as name words whole. Worth knowing before you filter on this: ``"Beethoven, Ludwig van"`` -- the textbook Dutch listing, read exactly right -- now carries a report too. It is the same string shape over the same vocabulary as ``"Nguyen, Thi Van"``, and nothing in the input separates them, so a report on one is a report on both. This adds the report and nothing else: every field these names parse to is exactly what the tussenvoegsel fix above already gave them, and ``ambiguities`` is the only value that grows. Seven of the 1,094 differential corpus names gain a kind against the 2.0.0 and 2.1.0 baselines -- the same seven the tussenvoegsel fix already moved, now carrying ``_ambiguities`` in their diff as well -- and none against 1.4.0, which had no ambiguity reporting at all (closes #405) + - Fix a tussenvoegsel after a family comma being parsed as a middle name. Dutch and Belgian alphabetized listings move the particle behind the given name -- ``"Beethoven, Ludwig van"`` is how ``"Ludwig van Beethoven"`` is filed -- and the trailing particle run was read as a middle name rather than as part of the surname: ``"Beethoven, Ludwig van"`` gave middle ``van``, last ``Beethoven``, and ``"Berg, Jan van der"`` gave middle ``van der``. The run now attaches to the family the comma has already named and renders before it, so those read family ``van Beethoven`` and ``van der Berg`` with the given name unchanged. The derived views move with the parse, so ``family_particles`` is ``van`` and ``family_base`` is ``Beethoven`` where they were empty and ``Beethoven`` before. `#130 `_ asked for the split and got it in 1.3.0 as ``last_base``/``last_prefixes``; 2.0 renamed them ``family_base``/``family_particles``. What was wrong until now was the values they reported for this listing. Both halves of the particle vocabulary attach -- never-given ``de`` and may-be-given ``van`` alike -- because after a comma the family is already named and the particle has no other role to take. Two guards bound it. A name whose only given word is the particle keeps it, so ``"Nguyen, Van"`` still reads given ``Van``: the attachment needs a given word to spare. And where the word is BOTH particle and suffix vocabulary the attachment outranks the post-nominal reading, so ``"Berg, Jan vd"`` reads family ``vd Berg`` where 1.4.0 and 2.1 alike gave suffix ``vd`` -- a trailing abbreviation after a family comma is the tussenvoegsel far more often than the decoration it collides with, and the same shape sweeps in ``mc``, which 2.1 also read as a suffix. ``do`` is in ``SUFFIX_ACRONYMS_AMBIGUOUS`` and 2.1 already read a trailing one as a name word, so it attaches by the plain rule rather than by the override (closes #379, closes #380). Names without the comma are untouched: ``"Ludwig van Beethoven"`` already read family ``van Beethoven`` and is byte-identical. One of the 751 differential corpus names moves, ``"Vega, Juan de la"``, at the 1.4.0, 2.0.0 and 2.1.0 baselines alike - Add ``abd`` to :data:`~nameparser.config.bound_given_names.BOUND_GIVEN_NAMES`, so the spellings that write the article as its own word join like the others do: ``"abd Allah Smith"`` was given ``abd``, middle ``Allah`` and is now given ``abd Allah``. ``abdul``, ``abdel`` and ``abdal`` were already there, and the Arabic-script ``عبد`` has covered the same word since 2.0, so only the Latin spelling was short. The word is also the postnominal ABD ("All But Dissertation") and stays in ``SUFFIX_ACRONYMS``: position tells the two readings apart, so ``"Jane Smith ABD"``, ``"Jane Smith, ABD"`` and ``"Jane Smith A.B.D."`` all still read the credential as a suffix. Making that work needed one fix beyond the wordlist. The join reserves enough words to leave a family name behind, and the reserve counted every piece except titles and suffixes -- so a word in BOTH vocabularies did not count as a name word even when the rule had already claimed it as one, and the join declined silently on the common three-word shape. The claimed piece now counts, which is what the reserve means: it asks whether enough OTHER words are left to spare. No shipped word other than ``abd`` is in both sets, and for any word that is not, the changed expression reduces to the old one -- so nothing else can move, by arithmetic rather than by sampling. The differential corpus agrees at all 751 names, but it holds no name carrying a dual-membership word, so that agreement tests the corpus rather than the change (#400) diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index 167c2b0c..bd96433d 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -317,7 +317,11 @@ def assign(state: ParseState) -> ParseState: # wholly-family read: after a comma that fixed the family, a # leading given-position particle is not meaningfully # ambiguous, and script_orders is not consulted for the parallel - # reason. The positional read below (a comma followed by no + # reason. Scoped, not silent: the comma fixed WHICH PIECE is the + # family and said nothing about a particle trailing the given + # name, so P6's attachment in post_rules decides that fork and + # reports it there, at the site that takes the branch (#405). + # The positional read below (a comma followed by no # name word) emits and consults both, being the no-comma read # of segment 0; group's chain emitter still does not, since # group runs before assign decides which read applies (recorded diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index c0c18d79..050d9037 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -4,7 +4,8 @@ particle fold reads the opening piece of segment 0, or of segment 1 under a family comma (#359). structure was always read here, for the rotation gate. -Produces: tokens with roles adjusted by the post rules. +Produces: tokens with roles adjusted by the post rules, plus the +ambiguity P6's attachment reports for the fork it decides (#405). Reads: Policy.patronymic_rules, Policy.middle_as_family; Lexicon.given_name_titles. @@ -19,9 +20,13 @@ from nameparser._lexicon import _title_key from nameparser._pipeline._assign import _name_positions -from nameparser._pipeline._state import ParseState, Structure, WorkToken +from nameparser._pipeline._state import ( + ParseState, PendingAmbiguity, Structure, WorkToken, +) from nameparser._policy import PatronymicRule -from nameparser._types import FOLDED_TAG, UNJOINED_TAG, Role +from nameparser._types import ( + FOLDED_TAG, UNJOINED_TAG, AmbiguityKind, Role, +) # Ported verbatim from v1 (nameparser/config/regexes.py) -- layering # forbids the config import; keep in sync by hand. @@ -175,6 +180,7 @@ def _is_lone_never_given_particle(site: tuple[int, ...], def post_rules(state: ParseState) -> ParseState: tokens = list(state.tokens) + ambiguities = list(state.ambiguities) titles = _idx(tokens, Role.TITLE) givens = _idx(tokens, Role.GIVEN) middles = _idx(tokens, Role.MIDDLE) @@ -400,11 +406,59 @@ def post_rules(state: ParseState) -> ParseState: # generated multi-piece runs: `end - k` is never above 1 # where the guard passes. Written as a range because the # guard, not this loop, is what bounds it. - for piece in seg[k:end]: - for i in piece: - tokens[i] = dataclasses.replace( - tokens[i], role=Role.FAMILY, - tags=tokens[i].tags | {FOLDED_TAG}) + run = [i for piece in seg[k:end] for i in piece] + # mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE: "Emit at + # the site that takes the branch, not where an ambiguous + # tag sits" -- this attachment is where the fork is + # decided, so the report is raised here rather than in + # assign, whose own emitter is scoped to the no-comma + # shapes (#405). + # + # Two arms, keyed on what the attachment OVERRODE rather + # than on what the words are, so each names a branch the + # parse actually weighed. `declined_suffix` reads the role + # assign gave the run, which is why both lists are built + # BEFORE the re-roling loop below overwrites it. + # + # ORDERED, not asserted disjoint. The two cannot both hold + # under the shipped vocabulary -- `vd` and `mc` are the + # only words in both the particle and the unambiguous + # suffix vocabularies, and neither is ambiguous particle + # vocabulary -- but a caller's Lexicon may put one word in + # both, and rules.md#A1 says parsing never fails on any + # input, so this decides instead of raising. (Measured: an + # assert here DID fire on `Berg, Jan zz` under a Lexicon + # adding `zz` to particles_ambiguous and suffix_acronyms.) + # The suffix arm wins because it names the reading the + # parse actually took: assign had read the word as a + # post-nominal, so the name-word reading was never on the + # table for the attachment to decline. + ambiguous = [i for i in run + if "vocab:particle-ambiguous" in tokens[i].tags] + declined_suffix = [i for i in run + if tokens[i].role is Role.SUFFIX] + text = " ".join(tokens[i].text for i in run) + if declined_suffix: + ambiguities.append(PendingAmbiguity( + AmbiguityKind.SUFFIX_OR_NAME, + f"{text!r} written without periods is both a " + f"post-nominal and a family-name particle; after a " + f"family comma it is read as the particle rather " + f"than as a post-nominal", + tuple(run))) + elif ambiguous: + word = tokens[ambiguous[0]].text + ambiguities.append(PendingAmbiguity( + AmbiguityKind.PARTICLE_OR_GIVEN, + f"{word!r} is both a family-name particle and an " + f"ordinary given name; after a family comma " + f"{text!r} is read as the family's particle " + f"rather than as a name word", + tuple(run))) + for i in run: + tokens[i] = dataclasses.replace( + tokens[i], role=Role.FAMILY, + tags=tokens[i].tags | {FOLDED_TAG}) # rules.md#O3: "every middle word joins the family name and is # rendered before it" (v1 handle_middle_name_as_last). v1 @@ -440,4 +494,5 @@ def post_rules(state: ParseState) -> ParseState: for i in part: tokens[i] = dataclasses.replace( tokens[i], tags=tokens[i].tags | {UNJOINED_TAG}) - return dataclasses.replace(state, tokens=tuple(tokens)) + return dataclasses.replace(state, tokens=tuple(tokens), + ambiguities=tuple(ambiguities)) diff --git a/nameparser/_pipeline/_state.py b/nameparser/_pipeline/_state.py index a8ea880c..01344203 100644 --- a/nameparser/_pipeline/_state.py +++ b/nameparser/_pipeline/_state.py @@ -98,12 +98,14 @@ class ParseState: in the segment runs shifts by n); classify -> token tags; group -> pieces/piece_tags/dropped AND maiden token roles; assign -> the remaining token roles AND `order`, the effective - order it read them under; post_rules -> roles again. + order it read them under; post_rules -> roles again, and the + ambiguity P6's attachment reports. Ambiguities are recorded by every stage that DECIDES one -- extract (resolved to a token index by tokenize), segment, script_segment, classify, - group, and assign -- since a fork whose branches are taken in - different stages needs an emitter in each. Post-group, segments + group, assign, and post_rules -- since a fork whose branches are + taken in different stages needs an emitter in each. Post-group, + segments may retain indices of dropped tokens -- assign iterates pieces, never segments. This ownership map is pinned by tests/v2/pipeline/test_state.py. diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 3db72716..78b4bb8b 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -324,26 +324,40 @@ def __post_init__(self) -> None: Case("tussenvoegsel_after_family_comma", "Beethoven, Ludwig van", {"given": "Ludwig", "family": "van Beethoven"}, classification="fix(#379)", + ambiguities=("particle-or-given",), notes="1.4.0 gives middle 'van', last 'Beethoven'. The " "particle attaches to the family the comma already " "named and renders before it, so the derived views " "move with it -- family_particles 'van', family_base " - "'Beethoven', which is what #130 asked for"), + "'Beethoven', which is what #130 asked for. The " + "textbook-correct Dutch listing reports the fork all " + "the same (#405): the parser cannot tell it from " + "'Nguyen, Thi Van', which is the same string shape"), Case("tussenvoegsel_multiword", "Berg, Jan van der", {"given": "Jan", "family": "van der Berg"}, classification="fix(#379)", - notes="the whole run attaches, not just its last word"), + ambiguities=("particle-or-given",), + notes="the whole run attaches, not just its last word -- and " + "one report covers the whole run, named for 'van', the " + "word that is ambiguous vocabulary ('der' is never a " + "given name)"), Case("tussenvoegsel_outranks_the_suffix_reading", "Berg, Jan vd", {"given": "Jan", "family": "vd Berg"}, classification="fix(#380)", + ambiguities=("suffix-or-name",), notes="'vd' is particle AND suffix vocabulary, and assign " "read the trailing one as a post-nominal (1.4.0 and " "2.1 alike gave suffix 'vd'). After a family comma the " "tussenvoegsel abbreviation is far more often the " - "reading meant; P6 states that precedence over S2"), + "reading meant; P6 states that precedence over S2. " + "The declined post-nominal is what the report names " + "(#405), so the kind is suffix-or-name and not " + "particle-or-given -- 'vd' is no given name in either " + "reading"), Case("tussenvoegsel_behind_a_post_nominal", "Berg, Jan van Jr.", {"given": "Jan", "family": "van Berg", "suffix": "Jr."}, classification="fix(#379)", + ambiguities=("particle-or-given",), notes="the credential sits BEHIND the tussenvoegsel in this " "listing, so the run is found by walking past it. " "Without that walk the same name parsed two ways on " @@ -367,6 +381,7 @@ def __post_init__(self) -> None: "Berg, Jan van, Jr.", {"given": "Jan", "family": "van Berg", "suffix": "Jr."}, classification="fix(#379)", + ambiguities=("particle-or-given",), notes="the credential in its own comma segment, which is the " "spelling the no-comma row is defined against -- both " "must read the same, and gating the rule on a two" @@ -374,6 +389,7 @@ def __post_init__(self) -> None: Case("tussenvoegsel_behind_a_title", "Berg, Dr. Jan van", {"title": "Dr.", "given": "Jan", "family": "van Berg"}, classification="fix(#379)", + ambiguities=("particle-or-given",), notes="the words-to-spare test asks whether ANY word ahead " "of the run holds a given role, not whether all of " "them do: the title does not, and the rule must still " @@ -381,13 +397,54 @@ def __post_init__(self) -> None: Case("tussenvoegsel_takes_the_vietnamese_reading", "Nguyen, Thi Van", {"given": "Thi", "family": "Van Nguyen"}, classification="fix(#379)", + ambiguities=("particle-or-given",), notes="the accepted cost, pinned so it cannot move without " "someone deciding to move it: Nguyen Thi Van is " "family-middle-given, so the given name Van is lost " "here. The listing is identical to the Dutch one and " "nothing separates them. The comma-LESS family-first " "spelling reads it correctly, which is what makes the " - "loss acceptable -- see rules.md#P6"), + "loss acceptable -- see rules.md#P6. Since #405 the " + "loss is at least REPORTED: 'Van' is ambiguous " + "vocabulary, so the attachment declines a live reading " + "as a name word and says so"), + Case("tussenvoegsel_report_names_the_reading_overridden", + "Berg, Jan do", + {"given": "Jan", "family": "do Berg"}, + classification="fix(#379)", + ambiguities=("particle-or-given",), + notes="`do` is the third word in BOTH the particle and the " + "suffix vocabularies, and it reports the OTHER kind " + "from vd and mc. The report names what the attachment " + "overrode, not what the word is: `do` sits in the " + "AMBIGUOUS acronym half, which already left it a name " + "word, so assign never read it as a post-nominal and " + "there was no credential reading to decline (#405). A " + "rule keyed on 'is also suffix vocabulary' would say " + "suffix-or-name here and be wrong for the one word " + "that most looks like it should"), + Case("tussenvoegsel_multiword_run_reports_on_its_reading", + "Berg, Jan de vd", + {"given": "Jan", "family": "de vd Berg"}, + classification="fix(#379)", + notes="the same point from the other side: `vd` IS suffix " + "vocabulary, but behind `de` the run is read as name " + "words rather than as a post-nominal, so nothing was " + "declined and nothing is reported -- where the lone " + "`Berg, Jan vd` reports suffix-or-name. Neither word " + "is ambiguous vocabulary either, so no arm fires. The " + "arms turn on the READING assign made, which is the " + "only thing that makes them a fork"), + Case("tussenvoegsel_never_given_reports_nothing", "Jong, Piet de", + {"given": "Piet", "family": "de Jong"}, + classification="fix(#379)", + notes="the reporting boundary (#405): 'de' is never-given " + "particle vocabulary and is not suffix vocabulary " + "either, so the attachment declines no reading the " + "parse could have taken and stays SILENT. A single " + "kind covering every attachment would assert " + "'particle or given' about a word that is no given " + "name in any reading"), Case("tussenvoegsel_needs_a_given_word_to_spare", "Nguyen, Van", {"given": "Van", "family": "Nguyen"}, classification="parity", @@ -1058,7 +1115,7 @@ def __post_init__(self) -> None: "something. An ungated marker stop made the chain merge " "nothing for a DIFFERENT reason than the guard assumes, " "silencing the report while still deciding the fork -- " - "the shape A1 forbids and #405 tracks. Pinned because " + "the shape A1 forbids and #405 closed at P6. Pinned because " "removing a report a caller already sees is worse than " "never emitting one"), Case("maiden_marker_particles_on_both_sides", diff --git a/tests/v2/pipeline/test_post_rules.py b/tests/v2/pipeline/test_post_rules.py index 2273d591..475190ac 100644 --- a/tests/v2/pipeline/test_post_rules.py +++ b/tests/v2/pipeline/test_post_rules.py @@ -595,3 +595,35 @@ def test_a_bound_pair_survives_the_leading_particle_stop( lexicon=Lexicon.default(), policy=policy)) assert _by_role(out, Role.FAMILY) == "de Mesnil" assert _by_role(out, Role.GIVEN) == "abd Allah" + + +def test_a_dual_membership_word_decides_rather_than_raising() -> None: + # P6's report has two arms, and the plan that added them (#405) + # recorded them as disjoint by construction -- "a suffix-vocabulary + # particle arrives suffix-roled and IS the whole run". That is true + # of the SHIPPED vocabulary and not of the rule: `vd` and `mc` are + # the only words in both the particle and the unambiguous suffix + # sets, and neither is ambiguous particle vocabulary, so no shipped + # input reaches both arms. A caller's Lexicon can put one word in + # both, and an `assert` written on that premise DID fire here + # before this test existed -- rules.md#A1 says parsing never fails + # on any input, and an AssertionError out of a stage is exactly the + # failure it forbids. + # + # So the arms are ORDERED and the suffix one wins: assign had read + # the word as a post-nominal, so the name-word reading was never on + # the table for the attachment to decline + # (mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE -- report the + # branch actually taken). + lex = dataclasses.replace( + _LEX, + particles=frozenset(_LEX.particles | {"zz"}), + particles_ambiguous=frozenset(_LEX.particles_ambiguous | {"zz"}), + suffix_acronyms=frozenset(_LEX.suffix_acronyms | {"zz"})) + out = run(ParseState(original="Berg, Jan zz", lexicon=lex, + policy=Policy())) + # token order, so the attachment shows as the run joining the + # family rather than as the family view's rendered order + assert _by_role(out, Role.FAMILY) == "Berg zz" + assert _by_role(out, Role.GIVEN) == "Jan" + assert [a.kind.value for a in out.ambiguities] == ["suffix-or-name"] diff --git a/tests/v2/pipeline/test_state.py b/tests/v2/pipeline/test_state.py index 8da7ae8a..ad2cd602 100644 --- a/tests/v2/pipeline/test_state.py +++ b/tests/v2/pipeline/test_state.py @@ -76,7 +76,11 @@ def test_stage_field_ownership() -> None: # assign also records `order`: the effective order it read the # name under, which post_rules needs and must not re-derive "assign": {"tokens", "ambiguities", "order"}, - "post_rules": {"tokens"}, + # post_rules also emits PARTICLE_OR_GIVEN and SUFFIX_OR_NAME: + # P6's attachment takes the family branch of a fork whose other + # branches assign and group take, so each stage reports the + # side it decides + "post_rules": {"tokens", "ambiguities"}, } assert {s.__name__ for s in STAGES} == set(ownership) # Within the tokens themselves the contract is finer: texts and diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index 7ebe2af3..14ffbf2f 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -1666,7 +1666,7 @@ def _claim(rule: dict) -> _Claim: "fix(#429) a wholly-credential segment after a one-word family renders as one entry": _Claim(1, ('suffix', 'title'), "9e0b9e8d5cbe"), "fix(#379) a tussenvoegsel after a family comma attaches to the family": - _Claim(13, ('family', 'middle'), "973617235cda"), + _Claim(13, ('_ambiguities', 'family', 'middle'), "973617235cda"), "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": _Claim(108, ('_ambiguities', 'family', 'given', 'middle'), "9a814f70c2dc"), "fix(#308/#312/#319/#320) glued CJK honorific peeled off the name into suffix": @@ -1684,7 +1684,7 @@ def _claim(rule: dict) -> _Claim: "fix(#367) a title no longer displaces a leading particle out of the leading position": _Claim(3, ('family', 'given'), "724967a4a117"), "fix(#380) a trailing vd after a family comma is the tussenvoegsel, not a post-nominal": - _Claim(2, ('family', 'suffix'), "ec0d45289dc1"), + _Claim(2, ('_ambiguities', 'family', 'suffix'), "ec0d45289dc1"), "fix(#399) a maiden marker bounds the particle chain that swallowed it": _Claim(5, ('family', 'maiden'), "15ec75a89f07"), "fix(#360) mc moved into the never-given particles, so it folds into the family": @@ -1782,13 +1782,13 @@ def _claim(rule: dict) -> _Claim: "fix(#429) a wholly-credential segment after a one-word family renders as one entry": _Claim(1, ('suffix', 'title'), "9e0b9e8d5cbe"), "fix(#379) a tussenvoegsel after a family comma attaches to the family": - _Claim(13, ('family', 'middle'), "973617235cda"), + _Claim(13, ('_ambiguities', 'family', 'middle'), "973617235cda"), "fix(#424) an unlisted abbreviation is as transparent as a listed title to the leading particle": _Claim(1, ('_ambiguities', 'family', 'given'), "ca7b37af6cf8"), "fix(#367) a title no longer displaces a leading particle out of the leading position": _Claim(3, ('family', 'given'), "724967a4a117"), "fix(#380) a trailing vd after a family comma is the tussenvoegsel, not a post-nominal": - _Claim(2, ('family', 'suffix'), "ec0d45289dc1"), + _Claim(2, ('_ambiguities', 'family', 'suffix'), "ec0d45289dc1"), "fix(#399) a maiden marker bounds the particle chain that swallowed it": _Claim(5, ('family', 'maiden'), "15ec75a89f07"), "fix(#360) mc moved into the never-given particles, so it folds into the family": diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index f98f7bbb..ebad1264 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -100,6 +100,7 @@ "Jones née" "Jong Anke de" "Jong, Anke de" +"Jong, Piet de" "Jose E Maria Santos" "Juan & Garcia" "Juan McDonald" diff --git a/tools/differential/expected_since_2.0.0.toml b/tools/differential/expected_since_2.0.0.toml index 3094c535..1e9bba95 100644 --- a/tools/differential/expected_since_2.0.0.toml +++ b/tools/differential/expected_since_2.0.0.toml @@ -240,8 +240,16 @@ issue = "fix(#380) a trailing vd after a family comma is the tussenvoegsel, not # named (rules.md#P6) where 2.0 read it as a post-nominal. Arrived via # corpus_rules.jsonl (#414), which is the arrival fix(#379) below # predicted in writing. +# +# `_ambiguities` joined the diff in #405. Checked against THIS ledger's +# own run rather than copied from the 2.1 one: the two names this rule +# explains here are the same pair -- 'Berg, Jan vd' and 'Berg, abdul +# vd' -- and each moves {family, suffix, _ambiguities}, gaining +# SUFFIX_OR_NAME. 2.0 read the trailing `vd` as a post-nominal exactly +# as 2.1 did, so the diff shape is identical at both baselines; that is +# a measurement here, not an inference from the copy above. name_regex = "(?i)^[\\u0000-\\u024f]*,[\\u0000-\\u024f]*\\svd\\s*$" -fields = ["family", "suffix"] +fields = ["family", "suffix", "_ambiguities"] [[change]] issue = "fix(#399) a maiden marker bounds the particle chain that swallowed it" @@ -893,8 +901,20 @@ issue = "fix(#379) a tussenvoegsel after a family comma attaches to the family" # naming tradition, so a future {middle,family} regression on # 'Nguyen, Van' -- one of the two corpus names it reaches, and the one # that does NOT move -- would classify as intended. +# +# `_ambiguities` joined the diff in #405. Measured on THIS ledger's own +# run: the same seven names classify here as at 2.1, and they split the +# same way -- five move {middle, family, _ambiguities} ('Vega, Juan de +# la', 'Beethoven, Ludwig van', 'Berg, Jan van der', 'der, y van', +# 'Nguyen, Thi Van', each gaining PARTICLE_OR_GIVEN) and two move +# {middle, family} alone ('Jong, Anke de', 'Jong, Piet de', whose `de` +# declines no reading and so reports nothing). The identical split is +# what a run at this baseline shows, not what the 2.1 rule says: 2.0 +# read the trailing run as a middle name as 2.1 did, and no ambiguity +# surface changed between them. The union over the seven is the three +# fields declared. name_regex = "(?i)^[\\u0000-\\u024f]*,[\\u0000-\\u024f]*\\s(de|del|den|der|di|do|dos|du|la|le|los|mc|van|vd|von|zu)(\\s+(de|del|den|der|di|do|dos|du|la|le|los|mc|van|vd|von|zu))*\\s*$" -fields = ["middle", "family"] +fields = ["middle", "family", "_ambiguities"] [[change]] issue = "fix(#412) a connective join no longer absorbs the maiden marker beside it" diff --git a/tools/differential/expected_since_2.1.0.toml b/tools/differential/expected_since_2.1.0.toml index d6d5c236..45d324ef 100644 --- a/tools/differential/expected_since_2.1.0.toml +++ b/tools/differential/expected_since_2.1.0.toml @@ -602,8 +602,20 @@ issue = "fix(#380) a trailing vd after a family comma is the tussenvoegsel, not # corpus name exercises it -- so an mc-after-comma name should arrive # UNEXPLAINED and be read once, the same discipline that produced this # rule. +# +# `_ambiguities` joined the diff in #405, which made the attachment +# report the fork it decides. Measured on THIS ledger's run: both names +# this rule explains -- 'Berg, Jan vd' and 'Berg, abdul vd' -- move +# {family, suffix, _ambiguities}, each gaining SUFFIX_OR_NAME, the kind +# that names the post-nominal reading the attachment declined. Neither +# name moves a field it did not move before, so the widening admits one +# more role and no more names; the union over the two is exactly the +# three declared. A separate fix(#405) rule was tried first and cannot +# work: classify matches on `fields` being a SUPERSET of the whole +# diff, so an `_ambiguities`-only rule matches neither name and this +# rule would go dormant with both of them unexplained. name_regex = "(?i)^[\\u0000-\\u024f]*,[\\u0000-\\u024f]*\\svd\\s*$" -fields = ["family", "suffix"] +fields = ["family", "suffix", "_ambiguities"] [[change]] issue = "fix(#399) a maiden marker bounds the particle chain that swallowed it" @@ -807,8 +819,25 @@ issue = "fix(#379) a tussenvoegsel after a family comma attaches to the family" # naming tradition, so a future {middle,family} regression on # 'Nguyen, Van' -- one of the two corpus names it reaches, and the one # that does NOT move -- would classify as intended. +# +# `_ambiguities` joined the diff in #405, which made the attachment +# report the fork it decides. Measured on THIS ledger's run, the seven +# names split: five move {middle, family, _ambiguities} -- 'Vega, Juan +# de la', 'Beethoven, Ludwig van', 'Berg, Jan van der', 'der, y van' +# and 'Nguyen, Thi Van', each gaining PARTICLE_OR_GIVEN, their particle +# being vocabulary that could have been a name word -- and two still +# move {middle, family} alone: 'Jong, Anke de' and 'Jong, Piet de', +# whose `de` can be no name in any reading, so nothing was declined and +# nothing is reported. The union over the seven is the three fields +# declared, which is what the over-declaration check asks for; narrowing +# to {middle, family} would orphan the five. A separate fix(#405) rule +# cannot take them instead: classify matches on `fields` being a +# SUPERSET of the whole diff, so an `_ambiguities`-only rule matches +# none of the five, and any rule wide enough to match them is this rule +# plus a role -- which then also claims the two silent names and leaves +# this one dormant. name_regex = "(?i)^[\\u0000-\\u024f]*,[\\u0000-\\u024f]*\\s(de|del|den|der|di|do|dos|du|la|le|los|mc|van|vd|von|zu)(\\s+(de|del|den|der|di|do|dos|du|la|le|los|mc|van|vd|von|zu))*\\s*$" -fields = ["middle", "family"] +fields = ["middle", "family", "_ambiguities"] [[change]] issue = "fix(#412) a connective join no longer absorbs the maiden marker beside it" From 6460e7882ccfac90f51721c15b5b27702c56a3fe Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sat, 29 Aug 2026 14:22:40 -0700 Subject: [PATCH 4/6] docs: settle R4's verification limit, sweep the branch's stale prose R4's verification-limit entry pointed forward to an unfiled issue for a capitalization column in the differential harness. That issue is not coming: the harness is scoped to parsing behavior, where a role is emergent across the eight-stage fold and the reach of a change has to be measured because it cannot be read off the diff. Case repair is a pure function of (word, role, tags, lexicon) with four branches behind one gate, so a change to it is legible in the change; a case column would mostly re-report parse diffs the field comparison already explains, and the residue belongs in unit tests. The entry now states that as a scope rather than as a gap, and the following bullet is rewritten as a settled decision instead of a deferral. The load-bearing fact is kept: compare.py compares the seven role fields plus `_ambiguities` and never `capitalized()`. Stale prose the branch's two behavior changes left behind. The AmbiguityKind docstrings are the API reference through autodoc and were written for the pre-#405 world: PARTICLE_OR_GIVEN said the fork is reported for a particle "at the head of a name" in "two shapes", where P6's attachment now makes a third at the tail after a family comma, and SUFFIX_OR_NAME's coverage list did not include the family-comma abbreviation it now reports. docs/concepts.rst told readers that "the comma paths stay quiet on purpose", which #405 made false; it now says what a comma actually settles and names the listing that still reports. No other site had prose to move -- AGENTS.md's ambiguity bullet was already corrected by the #405 commit, and README.rst, docs/usage.rst, docs/customize.rst, docs/migrate.rst and docs/modules.rst carry only working-particle examples that this branch does not reach. corpus_rules.jsonl's count in _CORPUS_FLOORS read 240; the #405 commit added a rules.md example after it was written. It is 241. OUT OF SCOPE FOR THIS BRANCH, but adjacent enough to fix while the files are open: all three expected_since_*.toml headers said `fields` "narrows further and is optional". That stopped being true when #456/#457 landed on 2026-08-29 -- validate_rules already refused a rule with `fields` and no `name_regex` (#451) and now refuses the mirror shape, so both keys are required. Each header is corrected in its own words. Verified at HEAD: 6114 passed / 156 skipped / 9 xfailed, mypy and ruff clean, and the gate green at 1094 names with 229/194/102 intentional diffs and 0 unexplained at all three baselines. Co-Authored-By: Claude Opus 5 --- docs/concepts.rst | 11 ++++++---- docs/design/decisions.md | 4 ++-- nameparser/_types.py | 21 +++++++++++++++----- tools/differential/compare.py | 2 +- tools/differential/expected_since_1.4.0.toml | 10 ++++++---- tools/differential/expected_since_2.0.0.toml | 11 +++++----- tools/differential/expected_since_2.1.0.toml | 9 +++++---- 7 files changed, 43 insertions(+), 25 deletions(-) diff --git a/docs/concepts.rst b/docs/concepts.rst index 59c8fa78..b39cd841 100644 --- a/docs/concepts.rst +++ b/docs/concepts.rst @@ -180,10 +180,13 @@ call it and says so. An empty ``ambiguities`` is therefore not a certificate of certainty. Reporting is deliberately partial: :class:`~nameparser.AmbiguityKind` lists the forks worth flagging, and even those are not reported -everywhere they occur — the comma paths stay quiet on purpose, since a -comma usually settles the structure before the question arises. -Coverage grows over releases. Treat a non-empty ``ambiguities`` as a -signal to act on; do not read an empty one as a guarantee. +everywhere they occur. What a comma buys is narrower than it looks — +it names which words are the surname, and stays quiet about that +much, but it leaves the rest of the name to be read as usual, so +``"Beethoven, Ludwig van"`` still reports the trailing ``van`` it had +to call. Coverage grows over releases. Treat a non-empty +``ambiguities`` as a signal to act on; do not read an empty one as a +guarantee. :class:`Tokens ` also carry tags — a second, independent label alongside their role, recording how a token was classified rather than what part of diff --git a/docs/design/decisions.md b/docs/design/decisions.md index b15d0df1..0b74e2dd 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -725,8 +725,8 @@ Declined: - 2026-08-29 #407 — DECIDED: case repair consults the mark #404 introduced, and consults it for the WHOLE PART rather than for a particle standing alone. Until now three views read it and the fourth did not, which is how `parse("ANH DO")` came to report family_base 'DO' and initials 'A. D.' while `capitalized()` returned 'Anh do' — the same word treated as an ordinary name word by three readers and as a particle by the fourth. The scope question is the only real choice here, and standing-alone loses on the first Vietnamese name that tests it: `anh van do` has family `van do`, two particle-vocabulary words and neither of them alone, so a rule keyed on aloneness would capitalize `Anh Do` and leave `Anh van do`, making the same surname read differently for having a given-name word in front of it. rules.md#R2 states the test as "Every word", not "standing alone", and this is the case that shows why it is stated that way. One predicate, four views. - 2026-08-29 — this is a DELIBERATE DEVIATION from 1.4.0, not a parity restoration, said plainly because the surrounding branch work is mostly restorations and a reader will otherwise assume this is one. Measured on the released 1.4.0 wheel: `HumanName('ANH DO').capitalize()` leaves `'Anh do'` and `HumanName('anh van do').capitalize()` leaves `'Anh van do'`; both are `'Anh Do'` and `'Anh Van Do'` on this branch. There was nothing to restore — v1 lowercased on vocabulary membership alone and had no notion of a particle with nothing to join, the mark being a 2.x invention. - 2026-08-29 — the ACCEPTED COST, agreed before the change rather than discovered after it: a degenerate family that is nothing but particles capitalizes too, so `juan van der` renders `Juan Van Der` where 1.4.0 gave `Juan van der`. This is the same trade rules.md#R2 already took for `family_base` — those words ARE the base, so they are name words, and a parser that called them name words for sorting and particles for display would be the incoherent option. Blast radius over the 1090 names the corpora carried before this change, measured by running both predicates in one process: 4 names move under a plain `capitalized()` — `ANH DO`, `vai la`, `van ma van`, and `der, y van`, which becomes `y Van Der` and keeps its lowercase conjunction. 23 move under `force=True`, the extra 19 being mixed-case inputs the gate holds back (rules.md#R5). -- 2026-08-29 — THE VERIFICATION LIMIT, which matters more than the change. `tools/differential/compare.py` compares the seven role fields plus `_ambiguities` and nothing else, so `capitalized()` is not a compared field and NO ledger run at any baseline can see this deviation: the corpus counts and the three intentional-diff totals hold across it precisely because the harness is blind to it, and a green run is evidence about roles, not about case. mechanisms.md#FACADE-CONTRACT promises that v1-visible changes are "release-log-classified fixes, each verified in the differential ledger". The first half is kept — docs/release_log.rst carries the 2.2.0 bullet — and the SECOND HALF CANNOT BE, for this change or for any other that moves only case. Standing in its place: rules.md#R4's own example rows (`ANH DO`, `anh van do`, and the `juan de la vega` boundary), rules.md#R5's, and the v1-facade tests in tests/test_capitalization.py. The ones that pin the fix are mutation-checked against deletion of the tag consult; the `juan de la vega` control passes that mutation BY DESIGN, which is what makes it a control rather than a witness. A follow-up issue for a capitalization column belongs on the tracker and is not filed yet; until it is, this bullet is the only written record that the ledger's silence here is structural rather than a passing grade. -- 2026-08-29 — WHY THE HARNESS WAS NOT EXTENDED IN THIS COMMIT, measured rather than waved off, because "add a column" is the obvious answer to the bullet above and it is the wrong size. Over those same 1090 names the 1.4.0 and 2.1.0 wheels ALREADY disagree on `capitalize(force=True)` for 139 of them (recompute by running `HumanName(n).capitalize(force=True)` under each wheel over the four corpus files deduped, and diffing the two maps). Nearly every one is an echo of a parse diff the field comparison already classifies — a token that changed role capitalizes differently — so a naive column would re-report the ledger's own explained diffs in a vocabulary its rules cannot match, and would arrive pre-failed by 139 entries. A useful column has to compare case only where the fields agree, which is a diff-of-diffs design and a piece of work in its own right. +- 2026-08-29 — THE VERIFICATION LIMIT, which matters more than the change. `tools/differential/compare.py` compares the seven role fields plus `_ambiguities` and nothing else, so `capitalized()` is not a compared field and NO ledger run at any baseline can see this deviation: the corpus counts and the three intentional-diff totals hold across it precisely because the harness is blind to it, and a green run is evidence about roles, not about case. mechanisms.md#FACADE-CONTRACT promises that v1-visible changes are "release-log-classified fixes, each verified in the differential ledger". The first half is kept — docs/release_log.rst carries the 2.2.0 bullet — and the SECOND HALF CANNOT BE, for this change or for any other that moves only case. Standing in its place: rules.md#R4's own example rows (`ANH DO`, `anh van do`, and the `juan de la vega` boundary), rules.md#R5's, and the v1-facade tests in tests/test_capitalization.py. The ones that pin the fix are mutation-checked against deletion of the tag consult; the `juan de la vega` control passes that mutation BY DESIGN, which is what makes it a control rather than a witness. Read the silence as a SCOPE and not as a gap: the harness compares parses, case repair is a render view, and nothing is owed on the tracker for it — the next bullet is why that is a decision rather than a deferral. +- 2026-08-29 — WHY CASE IS NOT A COMPARED SURFACE, settled here rather than left open, because "add a column" is the obvious answer to the bullet above and it is the wrong instrument. What the harness is FOR is parsing behavior — tools/differential/README.md says to run it "when touching parsing behavior, and before cutting a release" — and what earns it its two-process cost is that a role is EMERGENT: the eight stages fold over one name and each can move what the next sees, so the reach of a vocabulary or rule edit is not derivable by reading the diff and has to be measured over a corpus instead. Case repair is the opposite kind of thing. `_render._cap_word` is a pure function of `(word, role, tags, lexicon)` with four branches — the particle/conjunction lowercasing, the exceptions map, the Mac/Mc convention, and plain `capitalize` — sitting behind R5's single gate, so an edit to it is legible: you can see what you touched. The measurement agrees with the reasoning. Over the 1094 corpus names the 1.4.0 and 2.1.0 wheels ALREADY disagree on `capitalize(force=True)` for 139 of them (recompute by running `HumanName(n).capitalize(force=True)` under each wheel over the four corpus files deduped, and diffing the two maps; the count was the same over the 1090 names the corpora carried before this change). Nearly every one is an echo of a parse diff the field comparison already classifies — a token that changed role capitalizes differently — so a column would re-report the ledger's own explained diffs in a vocabulary its rules cannot match, and would arrive pre-failed by 139 entries. Isolating the residue means comparing case only where the fields agree, a diff-of-diffs, to reach a class that a unit test reaches directly. So the compared surfaces stay the seven roles plus `_ambiguities`, by design and not by omission, and case is pinned where case belongs: the `rules.md#R4` and `#R5` example rows, and tests/test_capitalization.py. - 2026-08-29 — DEBT DISCHARGED, the obligation the last bullet of decisions.md#R5 left this commit; recorded here so the pointer resolves rather than dangling. The FALSEHOOD: R4 promised repair "vocabulary exceptions (McDonald) included" flatly, which `str(parse('Juan Mcdonald').capitalized())` — `'Juan Mcdonald'` — falsifies, the gate refusing before any vocabulary is consulted. The statement now scopes that promise to where repair acts at all and hands the where to R5. A SECOND error in the same eight words turned up while rewriting them, and is worth recording because it had survived unread for as long as the first: McDonald is not a vocabulary exception. The shipped exceptions are five entries — II, III, IV, M.D., Ph.D. (recompute with `Lexicon.default().capitalization_exceptions_map`) — and `Mc`/`Mac` is a separate convention applied to any name shaped like one, so the old parenthetical illustrated the promise with the one mechanism it was not about. The statement now names an exception the vocabulary actually records (Ph.D.) and the Mac/Mc convention separately. The AMBIGUITY: "an already-correct name comes back unchanged" is true read as idempotence and reads as false under bearer-correctness, and the repair was to say WHICH is meant — a name already written the way repair would write it comes back unchanged — plus one clause refusing the inference a reader might otherwise draw, that a spelling its bearer chose is not spared for having been chosen. That clause is deliberately a REFUSAL and not a carve-out: sparing deliberately single-cased names is behavior the parser does not have and that R5's rationale declines on the record, so `str(parse('bell hooks').capitalized())` is still `'Bell Hooks'`. R4's boundary moved too: the old `"Juan McDonald" → capitalized="Juan McDonald"` row was inert — it passes with R5's gate deleted — so it became a `capitalized_forced=` row, which asserts what repair APPLIES rather than what the gate keeps and fails when the Mac/Mc convention is removed (measured by neutering it in one process: `'Juan Mcdonald'`). The boundary is now `juan de la vega`, a working particle run R4's new clause deliberately does not reach. ### R5 — the case-repair gate diff --git a/nameparser/_types.py b/nameparser/_types.py index 934627fb..975d1583 100644 --- a/nameparser/_types.py +++ b/nameparser/_types.py @@ -367,13 +367,17 @@ class AmbiguityKind(StrEnum): #: family name remains; "Jack MA" keeps it as the name because none #: would) and a trailing roman numeral, which is a suffix where any #: other single letter would be a name ("John Smith V" vs "John - #: Smith B"). Which name part was declined depends on position and + #: Smith B"). It also covers a family-comma listing, where a + #: trailing abbreviation that is both a post-nominal and a surname + #: particle joins the surname the comma already named: "Berg, Jan + #: vd" takes ``vd`` for *van der* and declines the decoration. + #: Which name part was declined depends on position and #: ``name_order``, so ``detail`` names it rather than the kind. SUFFIX_OR_NAME = "suffix-or-name" - #: An ambiguous particle at the head of a name is either a - #: particle or a name in its own right -- "Van Johnson" is the - #: actor's given name, a bare "Van Buren" the presidential - #: surname, and the two-word shape cannot distinguish them. Two + #: An ambiguous particle is either a particle or a name in its own + #: right -- "Van Johnson" is the actor's given name, a bare + #: "Van Buren" the presidential surname, and the two-word shape + #: cannot distinguish them. Three #: shapes report this kind, decided in different stages, and #: ``detail`` is what tells them apart. A particle left standing #: alone chained nothing and was assigned a role, which ``detail`` @@ -389,6 +393,13 @@ class AmbiguityKind(StrEnum): #: and what remains is a leading word that is both a title and a #: particle, so it stays a name piece and the particle behind it is #: genuinely not leading ("Freiherr von Richthofen"). + #: The third shape is at the TAIL rather than the head: a family + #: comma names the surname, and a particle written behind the given + #: name after it -- "Beethoven, Ludwig van", the Dutch and Flemish + #: filing convention -- is attached to that surname, over its + #: reading as an ordinary name word. The comma settles which piece + #: is the family and nothing about this, so the fork is real and + #: ``detail`` names the word it turned on. PARTICLE_OR_GIVEN = "particle-or-given" #: A nickname/maiden delimiter opened without closing (or closed #: without opening); the text was kept as literal name content, so diff --git a/tools/differential/compare.py b/tools/differential/compare.py index 4d310e6c..6f679310 100644 --- a/tools/differential/compare.py +++ b/tools/differential/compare.py @@ -373,7 +373,7 @@ def _is_latin_only(name: str) -> bool: "corpus.jsonl": 480, # 486 today, from v1's banks at a pinned ref "corpus_cjk.jsonl": 95, # 98 today, generated from the case table "corpus_issues.jsonl": 370, # 381 today, harvested and append-only - "corpus_rules.jsonl": 150, # 240 today, generated from rules.md + "corpus_rules.jsonl": 150, # 241 today, generated from rules.md } diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index 427a7dbe..0188ec86 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -1,7 +1,9 @@ -# Every rule needs `issue` and `name_regex`; `fields` narrows further -# and is optional. Since #451 validate_rules REJECTS a rule carrying -# `fields` and no `name_regex`, as it already rejected one with -# neither. An unexplained diff is a release blocker until classified +# Every rule needs all three of `issue`, `name_regex` and `fields`. +# `fields` stopped being optional on 2026-08-29: validate_rules had +# refused a rule carrying `fields` and no `name_regex` since #451, and +# #456 closed the mirror shape, so a half-declared rule is now a +# startup error whichever key is the missing one. An unexplained diff +# is a release blocker until classified # (tools/differential/README.md). The citation here used to name a # gitignored spec requirement, and passed the armed legacy-citation # guard only because the line wrapped between the two words. diff --git a/tools/differential/expected_since_2.0.0.toml b/tools/differential/expected_since_2.0.0.toml index 1e9bba95..2d1bc78b 100644 --- a/tools/differential/expected_since_2.0.0.toml +++ b/tools/differential/expected_since_2.0.0.toml @@ -1,10 +1,11 @@ # Ledger for baseline 2.0.0 -- what changes for a user upgrading from # the previous minor. Same rule grammar as expected_since_1.4.0.toml: -# every rule needs `issue` and `name_regex`; `fields` narrows further -# and is optional. Since #451 validate_rules REJECTS a rule carrying -# `fields` and no `name_regex`, as it already rejected one with -# neither. File order is therefore load-bearing; the note below says -# how it falls out here. +# `issue`, `name_regex` and `fields` are all mandatory, none of them +# optional any more. A rule may not narrow on one axis alone: +# validate_rules has refused `fields` without a `name_regex` since +# #451 and refuses `name_regex` without a `fields` since #456. Since +# every rule therefore carries a `name_regex`, file order is +# load-bearing; the note below says how it falls out here. # # `fields` names roles the way Role does, for BOTH compared surfaces, # plus the pseudo-field `_ambiguities`. The roster is deliberately not diff --git a/tools/differential/expected_since_2.1.0.toml b/tools/differential/expected_since_2.1.0.toml index 45d324ef..bcf4c8b5 100644 --- a/tools/differential/expected_since_2.1.0.toml +++ b/tools/differential/expected_since_2.1.0.toml @@ -1,8 +1,9 @@ # Ledger for baseline 2.1.0 -- what changes for a user upgrading from -# the previous minor. Same rule grammar as the other ledgers: every rule -# needs `issue` and `name_regex`; `fields` narrows further and is -# optional. Since #451 validate_rules REJECTS a rule carrying `fields` -# and no `name_regex`, as it already rejected one with neither. Every +# the previous minor. Same rule grammar as the other ledgers: `issue`, +# `name_regex` and `fields` are every one of them required. Neither +# narrowing key may stand on its own -- #451 banned a rule with +# `fields` and no regex, #456 banned the reverse -- so a rule here +# narrows by name AND by role, never by just one. Every # rule therefore sits in one tier, _sorted_rules is stable, and FILE # ORDER decides every tie: write the narrower rule first. # From 766b2e58887660c21a38f855e83fd0e250e146ae Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sat, 29 Aug 2026 14:29:04 -0700 Subject: [PATCH 5/6] docs: drop a drifting count from the decision-site prose 'all 39 ambiguous particles' describes a past bug that fired for every member of the set; the number adds nothing to that point and had already drifted -- particles_ambiguous holds 37 today. Same edit in the three places the phrase appears (AGENTS.md, mechanisms.md and the cases.py row that quotes it), plus the 'only 9 of the 39' ratio in decisions.md's Excluded block, which is the same standing-count shape. decisions.md#vocabulary-collisions keeps its 39: that one is labelled 'Sizes as of 2.2.0dev' and followed by 'recompute before quoting them', which is the dated-snapshot form AGENTS.md's counting rule permits. Co-Authored-By: Claude Opus 5 --- AGENTS.md | 2 +- docs/design/decisions.md | 2 +- docs/design/mechanisms.md | 2 +- tests/v2/cases.py | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 137597f4..a90a2b57 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -249,7 +249,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These - **Method organization**, fixed section order in every class: fields + `__post_init__` validation → alternative constructors → dunders (construction/equality → protocol → operators) → properties → public methods by concern (access → editing → comparison → rendering delegates) → private helpers last, except a helper serving exactly one section may sit at that section's head. Sanctioned deviation, facade layer only: `HumanName` and the shim `Constants` organize by v1 concern groups (`# -- render defaults --`, `# -- config / parsing --`, `# -- fields --`, ..., dunders and pickle last) — the classes mirror v1's own surface and die in 3.0; the canonical order still binds every core type. - **Validation is eager and fail-loud**: every `raise` states the offending value, the expected form, and the fix. Exception taxonomy: wrong type — including wrong element type inside a collection, bare `str` where an iterable of strings is expected, or a `Mapping` where a plain iterable is expected — raises `TypeError`; well-typed but unacceptable values raise `ValueError`; failed enum lookups stay `ValueError` for any input (stdlib `EnumType` precedent). **When the message hands the reader code to paste, that code has to survive a type checker** — nameparser ships `py.typed`. #337's segmenterless warning offered `Policy(segment_scripts=())`, an `arg-type` error, because these fields are annotated with what they STORE rather than everything the constructor accepts. Prefer the `frozenset()` / `()` spellings in messages and docstrings, and pin the offered spelling in a test — the warning tests matched on `ja_segmenter` and never checked the actionable half of the message. **A warning emitted in `Parser.__post_init__` needs `parser_for` to re-emit it from its own frame** (the `catch_warnings(record=True)` block at its return): `__post_init__`'s `stacklevel` is sized for direct `Parser(...)` construction, and through `parser_for`'s extra frame the default one-line rendering attributes the warning to the library's own `return Parser(...)` — the exact call the message tells the user to change becomes invisible. No single stacklevel serves both entry points; a new construction warning gets the re-emission for free, but a new CONSTRUCTION SITE for `Parser` inside this package needs its own re-emission or its callers get library-attributed warnings (#337 review). - **Guard, hint, and emit for the WHOLE family, and parametrize the test over it**: a check added to one member of a set belongs on all of it, and the test must sweep the family, not one example. This session shipped `_reject_str_and_mapping` on `Policy` but not `PolicyPatch`, the bytes decode hint on three of five config entry points, and a regex-sync roster missing four of its copies — each a separate follow-up bug that a `{class} × {field} × {bad-value}` parametrization would have caught and a per-example test hid. When you find you're guarding member N, grep for the other members first. -- **Ambiguities are emitted at the DECISION site**: an `Ambiguity` records a fork the parse had to call, not a token that sits in an ambiguous vocabulary. Emit where the branch is taken — the trailing-suffix peel in `_assign`, the delimiter escape's follow-up in `classify` — never by scanning for a `vocab:*-ambiguous` tag. The same tagged token is a genuine fork in one position and unremarkable in another (`do` mid-name in "Joao da Silva do Amaral de Souza" chooses nothing). **A branch that runs but changes nothing is not a decision either** -- the prefix chain's `merge(k, j)` executes even when `j == k + 1`, folding a piece into itself, and keying on "the code got here" reported a fork for all 39 ambiguous particles on "Do Van Jr." (`Dr.` when that was written, before #367 made a plain title transparent and put the shape out of the loop's reach entirely), where the particle stayed a lone leading name piece — the GIVEN name under the default order, the family name under `FAMILY_FIRST` — and `_assign` reported the same token again. Check that the branch actually claimed something (`j > k + 1`) before recording. Structure also structure often settles the question before it arises, which is why `PARTICLE_OR_GIVEN` is not emitted on the `FAMILY_COMMA` path's WHOLLY-FAMILY read -- the comma fixed which piece is the family -- and `SUFFIX_OR_NAME` is not emitted for "Ma, Jack". Read that scope narrowly: the comma settles nothing about a particle trailing the given name, so P6's attachment in `post_rules` decides that fork on the same path and reports it (#405), in the kind naming the reading it OVERRODE, which is the reading assign made and not the word's vocabulary: `SUFFIX_OR_NAME` where assign had read the run as a post-nominal (`vd`, `mc`), else `PARTICLE_OR_GIVEN` where the run holds an ambiguous particle (`van`, and `do`, which is in the suffix vocabulary too but in its AMBIGUOUS half, so no credential reading was overridden), else silence. The decision site also has the token index and the detail text in hand, which the tag scan would have to reconstruct. **If a fork's two branches are taken in DIFFERENT stages, every one of them needs the emitter** -- `PARTICLE_OR_GIVEN` is decided in `_assign` when the ambiguous particle stays a lone leading piece, in `_group` when something shifts it off the name's leading piece and the prefix chain claims it, and in `post_rules` when P6's attachment takes a trailing particle into the family after a comma, so all three report; for two years only the first did. What can still do the shifting is narrow, and #367 is why: a plain title no longer can (`Dr. Van Johnson` reads as `Van Johnson` does and reports from `_assign`), so the `_group` emitter needs a word that is BOTH a title and a particle — `st`, `do` and `freiherr` in the default vocabulary, or any overlap a caller's config creates — standing ahead of the chained particle with nothing but titles before it. That word need not LEAD the input: `Dr. Do van Johnson` reaches the emitter with a plain title in front of it, and `Do St Johnson` reaches it with the chained particle itself in both vocabularies. `Freiherr von Richthofen` is the canonical shape rather than the only one; when checking whether that emitter is dead, a both-vocabulary word is the thing to look for, and the answer is that it is not dead. The stage-ownership map in `tests/v2/pipeline/test_state.py` must list `ambiguities` for each such stage, and it passes vacuously until a case row exercises the path, so add the row too. Report BOTH directions of a two-way fork — "John Smith MA" (read as a suffix) and "Jack MA" (read as the family name) are equally guesses. Every kind needs a trigger in `tests/v2/test_contracts.py::_AMBIGUITY_TRIGGERS` (an explicit `None`, strict-xfail, while reserved), and case-table rows pin expected kinds exactly, so a new emitter shows up in both immediately. **Pin the decision, not the vocabulary**: the only titled-particle test used an UNAMBIGUOUS particle, so it walked the right code path and proved nothing about the branch under test -- two criticals passed 1539 tests. A row contrasting the two readings ("John Smith V" against "John Smith B") is what makes an emitter's absence meaningful. +- **Ambiguities are emitted at the DECISION site**: an `Ambiguity` records a fork the parse had to call, not a token that sits in an ambiguous vocabulary. Emit where the branch is taken — the trailing-suffix peel in `_assign`, the delimiter escape's follow-up in `classify` — never by scanning for a `vocab:*-ambiguous` tag. The same tagged token is a genuine fork in one position and unremarkable in another (`do` mid-name in "Joao da Silva do Amaral de Souza" chooses nothing). **A branch that runs but changes nothing is not a decision either** -- the prefix chain's `merge(k, j)` executes even when `j == k + 1`, folding a piece into itself, and keying on "the code got here" reported a fork for all ambiguous particles on "Do Van Jr." (`Dr.` when that was written, before #367 made a plain title transparent and put the shape out of the loop's reach entirely), where the particle stayed a lone leading name piece — the GIVEN name under the default order, the family name under `FAMILY_FIRST` — and `_assign` reported the same token again. Check that the branch actually claimed something (`j > k + 1`) before recording. Structure also structure often settles the question before it arises, which is why `PARTICLE_OR_GIVEN` is not emitted on the `FAMILY_COMMA` path's WHOLLY-FAMILY read -- the comma fixed which piece is the family -- and `SUFFIX_OR_NAME` is not emitted for "Ma, Jack". Read that scope narrowly: the comma settles nothing about a particle trailing the given name, so P6's attachment in `post_rules` decides that fork on the same path and reports it (#405), in the kind naming the reading it OVERRODE, which is the reading assign made and not the word's vocabulary: `SUFFIX_OR_NAME` where assign had read the run as a post-nominal (`vd`, `mc`), else `PARTICLE_OR_GIVEN` where the run holds an ambiguous particle (`van`, and `do`, which is in the suffix vocabulary too but in its AMBIGUOUS half, so no credential reading was overridden), else silence. The decision site also has the token index and the detail text in hand, which the tag scan would have to reconstruct. **If a fork's two branches are taken in DIFFERENT stages, every one of them needs the emitter** -- `PARTICLE_OR_GIVEN` is decided in `_assign` when the ambiguous particle stays a lone leading piece, in `_group` when something shifts it off the name's leading piece and the prefix chain claims it, and in `post_rules` when P6's attachment takes a trailing particle into the family after a comma, so all three report; for two years only the first did. What can still do the shifting is narrow, and #367 is why: a plain title no longer can (`Dr. Van Johnson` reads as `Van Johnson` does and reports from `_assign`), so the `_group` emitter needs a word that is BOTH a title and a particle — `st`, `do` and `freiherr` in the default vocabulary, or any overlap a caller's config creates — standing ahead of the chained particle with nothing but titles before it. That word need not LEAD the input: `Dr. Do van Johnson` reaches the emitter with a plain title in front of it, and `Do St Johnson` reaches it with the chained particle itself in both vocabularies. `Freiherr von Richthofen` is the canonical shape rather than the only one; when checking whether that emitter is dead, a both-vocabulary word is the thing to look for, and the answer is that it is not dead. The stage-ownership map in `tests/v2/pipeline/test_state.py` must list `ambiguities` for each such stage, and it passes vacuously until a case row exercises the path, so add the row too. Report BOTH directions of a two-way fork — "John Smith MA" (read as a suffix) and "Jack MA" (read as the family name) are equally guesses. Every kind needs a trigger in `tests/v2/test_contracts.py::_AMBIGUITY_TRIGGERS` (an explicit `None`, strict-xfail, while reserved), and case-table rows pin expected kinds exactly, so a new emitter shows up in both immediately. **Pin the decision, not the vocabulary**: the only titled-particle test used an UNAMBIGUOUS particle, so it walked the right code path and proved nothing about the branch under test -- two criticals passed 1539 tests. A row contrasting the two readings ("John Smith V" against "John Smith B") is what makes an emitter's absence meaningful. - **A kind is worth adding only if a reader would hesitate too**: the test is not "does the code take a branch" but whether a person reading that input would genuinely be unsure. "Smith, John V" reads as a middle initial to anyone -- the comma settles it -- so reporting it would be noise that teaches callers to ignore the field, which costs more than the missing report. Reachability of the second branch is necessary, not sufficient. Prefer leaving a fork silent and documenting the omission (see the comma paths in concepts.rst) over emitting on input nobody finds ambiguous. - **Parser owns config-dependent conveniences**: `Parser.matches`/`Parser.capitalized`/`Parser.revise` exist because the `ParsedName` equivalents fall back to DEFAULT config for str/omitted arguments (documented loudly in both docstrings). `revise` harvests tokens from a full sub-parse of each replacement value (tags kept minus `FOLDED_TAG`, roles forced, ambiguities discarded); the merge tail is shared with `replace()` via `ParsedName._with_field_tokens`. `Parser.capitalized` delegates through `name.capitalized(self.lexicon)` specifically so `_parser` never imports `_render` — keep it that way. - **Per-word vocabulary fields warn on multi-word entries** (`_normset`/`_normpairs` via `_warn_dead_entry`, UserWarning, never a raise — see the given_name_titles Gotcha for why raising is wrong). `given_name_titles` is the one multi-word-matched field and is exempt; `_edit` passes `warn=False` (add() warns once via the new instance's `__post_init__`; remove() stores nothing). The default vocabulary and every locale pack must stay warning-free (`test_default_lexicon_builds_warning_free`, `test_pack_vocabulary_entries_are_single_words`). diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 0b74e2dd..1800940a 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -360,7 +360,7 @@ Declined: Excluded (the never-given / ambiguous particle line, nameparser/config/particles.py — #360 owns the vocabulary question): -- Only 9 of the 39 ambiguous members were ever individually justified; the rest sit there by the conservative default (ambiguous unless argued never-given). +- Only 9 of the ambiguous members were ever individually justified; the rest sit there by the conservative default (ambiguous unless argued never-given). - mc, ste — measured misparses ("Mc Donald" → given "Mc"), tracked in #360; st is inert at the head because TITLES claims it first; mac must stay ambiguous because Mac is a real given name. - Encoding rationale (#293, predating #360's membership questions): the data layer stores the NEVER-GIVEN set and derives the ambiguous one, because that is safe-by-default for new particles — a one-place addition — and the v1 shim translates by one-directional complement. And the constants are FROZEN specifically to kill the cached-Lexicon.default()-vs-fresh- Constants desync that runtime module-constant mutation caused. - Load-bearing dependency: TITLES ∩ ambiguous — {do, freiherr, st} until #296's audit, {freiherr, st} since — is what keeps the particle-or-given ambiguity emitter reachable at all; moving them all would leave the shipped vocabulary unable to reach it, which is why test_the_chained_emitter_is_reachable_by_construction (tests/v2/test_parser.py) pins reachability by construction rather than by shipped vocabulary — an empty intersection fails no test and does not mean the emitter is unreachable. diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 8743b49c..03484464 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -105,7 +105,7 @@ Problem shape. A retired default rides in on old pickles, but a user may have de Problem shape. An ambiguity report should fire exactly when the parse chose between live readings — no more, no less — and the choosing happens in code, not in vocabulary. Contract statement. Emit at the site that takes the branch, not where an ambiguous tag sits; when a fork's two branches are decided in different stages, EVERY deciding stage carries an emitter; and a branch that runs but changes nothing is not a decision and must not report. How it works. PARTICLE_OR_GIVEN fires from assignment for a lone leading particle and from grouping when a title shifts it off the front — for two years only the first site emitted. And keying on "the code got here" instead of "the outcome differed" once reported -a fork for all 39 ambiguous particles on "Dr. Van Jr.". Lives in. The AmbiguityKind emitters across _pipeline/ (rule A1 is the observable contract). Reach for it when. Adding any ambiguous vocabulary or any new fork — count the deciding sites, then count the emitters. +a fork for all ambiguous particles on "Dr. Van Jr.". Lives in. The AmbiguityKind emitters across _pipeline/ (rule A1 is the observable contract). Reach for it when. Adding any ambiguous vocabulary or any new fork — count the deciding sites, then count the emitters. ## MAKE-WRONG-STATES-UNREPRESENTABLE — the house meta-pattern diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 78b4bb8b..407221d7 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -515,7 +515,7 @@ def __post_init__(self) -> None: notes="the piece after the particle is a suffix, so the chain " "scan never advances and the merge is a no-op -- nothing " "was chained, so there is no fork to report (the emitter " - "fired here for all 39 ambiguous particles, and _assign " + "fired here for all ambiguous particles, and _assign " "double-reported the same token). Spelled with 'St' " "since #296's audit took 'do' out of TITLES; before that " "with 'Do', and before 2.2 with 'Dr.': " From e555d2136d66a36e16ebcb0d3b3d0aacf950722d Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sat, 29 Aug 2026 15:43:41 -0700 Subject: [PATCH 6/6] fix(post_rules,render,docs): the four-agent review of PR #460 P6's two ambiguity details said the trailing run "is read as the family's particle", which every other view contradicts: R2's mark runs later in the same stage and can make the whole attached part ordinary name words -- `de la, Jan van` gives family_base 'van de la', initials 'J. d. l. v.' and 'Jan Van De La' forced, while the report said "particle". Both details now name the ATTACHMENT that P6 actually decides -- the run joins the family the comma named rather than standing on its own -- which is true whether or not R2 later marks the part. The emit stays at the decision site. _render._cap_word's conjunction conjunct is correctly left ungated on the unjoined mark; only the comment's reason was false. It claimed `particles` and `conjunctions` are disjoint, so no conjunction could ever carry the mark; both sets are public API, and under `Lexicon.default().add(particles={'y'})` `anh y van` has an all-particle family whose `y` carries the mark and gives 'Anh y Van', where gating the conjunct gives 'Anh Y Van'. The real reason is R3's own carve-out, now stated in rules.md#R4 as well, with the disjointness claim scoped to the shipped vocabulary (measured empty in the default lexicon and all four locale packs). Same claim fixed at tests/test_capitalization.py. Also, all re-measured: - rules.md#R4: "Mac/Mc surname convention" -> "Mac/Mc convention" (_MAC has no role test; `mackenzie smith` capitalizes to 'MacKenzie Smith' with `mackenzie` in the GIVEN role). - release_log #405 bullet: the tussenvoegsel fix moved NINE names at 2.1.0, not seven -- fix(#379) explains 7 and fix(#380) 2; the seven gaining a kind are five of #379's plus both of #380's, and the two left out are the `Jong` pair the same bullet already lists as unchanged. Standing population count dropped. Its "the fix above" pointer was also backwards; the #379 bullet sits below it. - AGENTS.md: TITLES cap particles_ambiguous is {freiherr, st}, not {st, do, freiherr}; and `Dr. Do van Johnson` / `Do St Johnson` emit from `assign`, not `group`, so both worked examples were wrong. Replaced with `Dr. St van Johnson` / `St van Johnson` (reach it) and `Jan Freiherr von Richthofen` / `Jan St Johnson` (do not), measured by stepping STAGES. Dangling concepts.rst cross-reference dropped; "Structure also structure" typo fixed. - tests/v2/pipeline/test_post_rules.py: both arms' detail text and the reported token span are now asserted. Mutation-checked outside the worktree -- constant details and `tuple(run[:1])` each left the full suite green before, and each kills a new test now. - rules.md#R4 + decisions.md#R4: the replace()/revise() boundary, recorded as an Accepted boundary rather than a gap. `_cap_word` keys the particle TEST on lexicon membership and the REPAIR on the mark, and the mark is a parse product, so a family spliced in by `replace()` as 'de la' stays lowercase where a parsed one gives 'De La'. `Parser.revise()` is the supported crossing and already promised it -- measured, it gives 'De La' -- so its enumeration of the tag-driven views gains `capitalized()`, the fourth since #407. Recorded too: for an all-particle part the other three tag-driven views converge across `replace()` and `revise()` (measured over `de la`, `van der`, `do`, `de`, `van de la`), and on a mixed part they diverge while case repair agrees -- which is why the boundary was invisible until #407 made case repair tag-driven. - decisions.md blast radius: 4 and 23 movers over the 1090 names the corpora carried before this change, 5 and 24 over today's 1094, both now stated (`anh van do` is the added mover). - decisions.md / tests/test_capitalization.py: the re-casing figures are 63/25/38 through the v2 core and 62/25/37 through HumanName, whose default render spec omits the maiden name and so cannot see `Jane van der Berg nee y Jones`. Each now names its surface. - decisions.md: the 11 comma-plus-trailing-particle corpus names came from the #379/#380 arc alone; this change added four names and none carries a comma. - _state.py and Parser.revise()'s docstrings reflowed; capitalized() now says it reads token tags, matching initials(); the R4/R5 duplicate example row is kept and recorded as deliberate (five other rows already sit under two rules apiece). Gate green at all three baselines: corpus 1094, intentional diffs 229 / 194 / 102, unexplained 0. corpus_rules.jsonl unchanged at 241. Co-Authored-By: Claude Opus 5 --- AGENTS.md | 4 +- docs/design/decisions.md | 14 +++++-- docs/design/rules.md | 17 +++++++- docs/release_log.rst | 2 +- nameparser/_parser.py | 27 +++++++------ nameparser/_pipeline/_post_rules.py | 20 ++++++++-- nameparser/_pipeline/_state.py | 12 +++--- nameparser/_render.py | 33 +++++++++++---- tests/test_capitalization.py | 46 +++++++++++++-------- tests/v2/pipeline/test_post_rules.py | 60 ++++++++++++++++++++++++++-- 10 files changed, 175 insertions(+), 60 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index a90a2b57..1236d436 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -249,8 +249,8 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These - **Method organization**, fixed section order in every class: fields + `__post_init__` validation → alternative constructors → dunders (construction/equality → protocol → operators) → properties → public methods by concern (access → editing → comparison → rendering delegates) → private helpers last, except a helper serving exactly one section may sit at that section's head. Sanctioned deviation, facade layer only: `HumanName` and the shim `Constants` organize by v1 concern groups (`# -- render defaults --`, `# -- config / parsing --`, `# -- fields --`, ..., dunders and pickle last) — the classes mirror v1's own surface and die in 3.0; the canonical order still binds every core type. - **Validation is eager and fail-loud**: every `raise` states the offending value, the expected form, and the fix. Exception taxonomy: wrong type — including wrong element type inside a collection, bare `str` where an iterable of strings is expected, or a `Mapping` where a plain iterable is expected — raises `TypeError`; well-typed but unacceptable values raise `ValueError`; failed enum lookups stay `ValueError` for any input (stdlib `EnumType` precedent). **When the message hands the reader code to paste, that code has to survive a type checker** — nameparser ships `py.typed`. #337's segmenterless warning offered `Policy(segment_scripts=())`, an `arg-type` error, because these fields are annotated with what they STORE rather than everything the constructor accepts. Prefer the `frozenset()` / `()` spellings in messages and docstrings, and pin the offered spelling in a test — the warning tests matched on `ja_segmenter` and never checked the actionable half of the message. **A warning emitted in `Parser.__post_init__` needs `parser_for` to re-emit it from its own frame** (the `catch_warnings(record=True)` block at its return): `__post_init__`'s `stacklevel` is sized for direct `Parser(...)` construction, and through `parser_for`'s extra frame the default one-line rendering attributes the warning to the library's own `return Parser(...)` — the exact call the message tells the user to change becomes invisible. No single stacklevel serves both entry points; a new construction warning gets the re-emission for free, but a new CONSTRUCTION SITE for `Parser` inside this package needs its own re-emission or its callers get library-attributed warnings (#337 review). - **Guard, hint, and emit for the WHOLE family, and parametrize the test over it**: a check added to one member of a set belongs on all of it, and the test must sweep the family, not one example. This session shipped `_reject_str_and_mapping` on `Policy` but not `PolicyPatch`, the bytes decode hint on three of five config entry points, and a regex-sync roster missing four of its copies — each a separate follow-up bug that a `{class} × {field} × {bad-value}` parametrization would have caught and a per-example test hid. When you find you're guarding member N, grep for the other members first. -- **Ambiguities are emitted at the DECISION site**: an `Ambiguity` records a fork the parse had to call, not a token that sits in an ambiguous vocabulary. Emit where the branch is taken — the trailing-suffix peel in `_assign`, the delimiter escape's follow-up in `classify` — never by scanning for a `vocab:*-ambiguous` tag. The same tagged token is a genuine fork in one position and unremarkable in another (`do` mid-name in "Joao da Silva do Amaral de Souza" chooses nothing). **A branch that runs but changes nothing is not a decision either** -- the prefix chain's `merge(k, j)` executes even when `j == k + 1`, folding a piece into itself, and keying on "the code got here" reported a fork for all ambiguous particles on "Do Van Jr." (`Dr.` when that was written, before #367 made a plain title transparent and put the shape out of the loop's reach entirely), where the particle stayed a lone leading name piece — the GIVEN name under the default order, the family name under `FAMILY_FIRST` — and `_assign` reported the same token again. Check that the branch actually claimed something (`j > k + 1`) before recording. Structure also structure often settles the question before it arises, which is why `PARTICLE_OR_GIVEN` is not emitted on the `FAMILY_COMMA` path's WHOLLY-FAMILY read -- the comma fixed which piece is the family -- and `SUFFIX_OR_NAME` is not emitted for "Ma, Jack". Read that scope narrowly: the comma settles nothing about a particle trailing the given name, so P6's attachment in `post_rules` decides that fork on the same path and reports it (#405), in the kind naming the reading it OVERRODE, which is the reading assign made and not the word's vocabulary: `SUFFIX_OR_NAME` where assign had read the run as a post-nominal (`vd`, `mc`), else `PARTICLE_OR_GIVEN` where the run holds an ambiguous particle (`van`, and `do`, which is in the suffix vocabulary too but in its AMBIGUOUS half, so no credential reading was overridden), else silence. The decision site also has the token index and the detail text in hand, which the tag scan would have to reconstruct. **If a fork's two branches are taken in DIFFERENT stages, every one of them needs the emitter** -- `PARTICLE_OR_GIVEN` is decided in `_assign` when the ambiguous particle stays a lone leading piece, in `_group` when something shifts it off the name's leading piece and the prefix chain claims it, and in `post_rules` when P6's attachment takes a trailing particle into the family after a comma, so all three report; for two years only the first did. What can still do the shifting is narrow, and #367 is why: a plain title no longer can (`Dr. Van Johnson` reads as `Van Johnson` does and reports from `_assign`), so the `_group` emitter needs a word that is BOTH a title and a particle — `st`, `do` and `freiherr` in the default vocabulary, or any overlap a caller's config creates — standing ahead of the chained particle with nothing but titles before it. That word need not LEAD the input: `Dr. Do van Johnson` reaches the emitter with a plain title in front of it, and `Do St Johnson` reaches it with the chained particle itself in both vocabularies. `Freiherr von Richthofen` is the canonical shape rather than the only one; when checking whether that emitter is dead, a both-vocabulary word is the thing to look for, and the answer is that it is not dead. The stage-ownership map in `tests/v2/pipeline/test_state.py` must list `ambiguities` for each such stage, and it passes vacuously until a case row exercises the path, so add the row too. Report BOTH directions of a two-way fork — "John Smith MA" (read as a suffix) and "Jack MA" (read as the family name) are equally guesses. Every kind needs a trigger in `tests/v2/test_contracts.py::_AMBIGUITY_TRIGGERS` (an explicit `None`, strict-xfail, while reserved), and case-table rows pin expected kinds exactly, so a new emitter shows up in both immediately. **Pin the decision, not the vocabulary**: the only titled-particle test used an UNAMBIGUOUS particle, so it walked the right code path and proved nothing about the branch under test -- two criticals passed 1539 tests. A row contrasting the two readings ("John Smith V" against "John Smith B") is what makes an emitter's absence meaningful. -- **A kind is worth adding only if a reader would hesitate too**: the test is not "does the code take a branch" but whether a person reading that input would genuinely be unsure. "Smith, John V" reads as a middle initial to anyone -- the comma settles it -- so reporting it would be noise that teaches callers to ignore the field, which costs more than the missing report. Reachability of the second branch is necessary, not sufficient. Prefer leaving a fork silent and documenting the omission (see the comma paths in concepts.rst) over emitting on input nobody finds ambiguous. +- **Ambiguities are emitted at the DECISION site**: an `Ambiguity` records a fork the parse had to call, not a token that sits in an ambiguous vocabulary. Emit where the branch is taken — the trailing-suffix peel in `_assign`, the delimiter escape's follow-up in `classify` — never by scanning for a `vocab:*-ambiguous` tag. The same tagged token is a genuine fork in one position and unremarkable in another (`do` mid-name in "Joao da Silva do Amaral de Souza" chooses nothing). **A branch that runs but changes nothing is not a decision either** -- the prefix chain's `merge(k, j)` executes even when `j == k + 1`, folding a piece into itself, and keying on "the code got here" reported a fork for all ambiguous particles on "Do Van Jr." (`Dr.` when that was written, before #367 made a plain title transparent and put the shape out of the loop's reach entirely), where the particle stayed a lone leading name piece — the GIVEN name under the default order, the family name under `FAMILY_FIRST` — and `_assign` reported the same token again. Check that the branch actually claimed something (`j > k + 1`) before recording. Structure often settles the question before it arises, which is why `PARTICLE_OR_GIVEN` is not emitted on the `FAMILY_COMMA` path's WHOLLY-FAMILY read -- the comma fixed which piece is the family -- and `SUFFIX_OR_NAME` is not emitted for "Ma, Jack". Read that scope narrowly: the comma settles nothing about a particle trailing the given name, so P6's attachment in `post_rules` decides that fork on the same path and reports it (#405), in the kind naming the reading it OVERRODE, which is the reading assign made and not the word's vocabulary: `SUFFIX_OR_NAME` where assign had read the run as a post-nominal (`vd`, `mc`), else `PARTICLE_OR_GIVEN` where the run holds an ambiguous particle (`van`, and `do`, which is in the suffix vocabulary too but in its AMBIGUOUS half, so no credential reading was overridden), else silence. The decision site also has the token index and the detail text in hand, which the tag scan would have to reconstruct. **If a fork's two branches are taken in DIFFERENT stages, every one of them needs the emitter** -- `PARTICLE_OR_GIVEN` is decided in `_assign` when the ambiguous particle stays a lone leading piece, in `_group` when something shifts it off the name's leading piece and the prefix chain claims it, and in `post_rules` when P6's attachment takes a trailing particle into the family after a comma, so all three report; for two years only the first did. What can still do the shifting is narrow, and #367 is why: a plain title no longer can (`Dr. Van Johnson` reads as `Van Johnson` does and reports from `_assign`), so the `_group` emitter needs a word that is BOTH a title and a particle — measured, `TITLES ∩ particles_ambiguous` is `{freiherr, st}` in the default vocabulary (`do` left TITLES in #296's audit; decisions.md's Excluded block records the before and after), plus any overlap a caller's config creates — standing ahead of the chained particle as the LEADING NAME word. Titles may precede it, so `Dr. St van Johnson` reaches the emitter and `St van Johnson` does too; a given name may not, so `Jan Freiherr von Richthofen` does not reach it while `Freiherr von Richthofen` and `Dr. Freiherr von Richthofen` do. Two shapes that look like they should reach it and do NOT, both measured by stepping `STAGES` and watching where `ambiguities` grows: `Dr. Do van Johnson` and `Do St Johnson` report from `assign`, not `group`, because `do` is no longer a title and so stays the leading name piece assign reports on — a both-vocabulary word CHAINED (`Jan St Johnson`) reports nothing at all. When checking whether that emitter is dead, a both-vocabulary word in the leading name position is the thing to look for, and the answer is that it is not dead. The stage-ownership map in `tests/v2/pipeline/test_state.py` must list `ambiguities` for each such stage, and it passes vacuously until a case row exercises the path, so add the row too. Report BOTH directions of a two-way fork — "John Smith MA" (read as a suffix) and "Jack MA" (read as the family name) are equally guesses. Every kind needs a trigger in `tests/v2/test_contracts.py::_AMBIGUITY_TRIGGERS` (an explicit `None`, strict-xfail, while reserved), and case-table rows pin expected kinds exactly, so a new emitter shows up in both immediately. **Pin the decision, not the vocabulary**: the only titled-particle test used an UNAMBIGUOUS particle, so it walked the right code path and proved nothing about the branch under test -- two criticals passed 1539 tests. A row contrasting the two readings ("John Smith V" against "John Smith B") is what makes an emitter's absence meaningful. +- **A kind is worth adding only if a reader would hesitate too**: the test is not "does the code take a branch" but whether a person reading that input would genuinely be unsure. "Smith, John V" reads as a middle initial to anyone -- the comma settles it -- so reporting it would be noise that teaches callers to ignore the field, which costs more than the missing report. Reachability of the second branch is necessary, not sufficient. Prefer leaving a fork silent and documenting the omission over emitting on input nobody finds ambiguous. - **Parser owns config-dependent conveniences**: `Parser.matches`/`Parser.capitalized`/`Parser.revise` exist because the `ParsedName` equivalents fall back to DEFAULT config for str/omitted arguments (documented loudly in both docstrings). `revise` harvests tokens from a full sub-parse of each replacement value (tags kept minus `FOLDED_TAG`, roles forced, ambiguities discarded); the merge tail is shared with `replace()` via `ParsedName._with_field_tokens`. `Parser.capitalized` delegates through `name.capitalized(self.lexicon)` specifically so `_parser` never imports `_render` — keep it that way. - **Per-word vocabulary fields warn on multi-word entries** (`_normset`/`_normpairs` via `_warn_dead_entry`, UserWarning, never a raise — see the given_name_titles Gotcha for why raising is wrong). `given_name_titles` is the one multi-word-matched field and is exempt; `_edit` passes `warn=False` (add() warns once via the new instance's `__post_init__`; remove() stores nothing). The default vocabulary and every locale pack must stay warning-free (`test_default_lexicon_builds_warning_free`, `test_pack_vocabulary_entries_are_single_words`). - **Invariants guard harm, not no-ops**: add a constructor check when violating it produces a *wrong parse*, not when it produces *nothing*. A false positive costs a working configuration; a true positive on an inert condition costs the user nothing, so that trade is never worth taking. `suffix_acronyms_ambiguous ∩ suffix_words` is guarded because the overlap loses a family name; `given_name_titles` is not, because an unreachable entry is simply never consulted (see Gotchas). Before adding one, construct the config it forbids and check what actually breaks. **Between raise and silence sits the construction-time `UserWarning`**, for a gap that is statically decidable, harmless to SOME deliberate caller, and indistinguishable-from-working for everyone else: the segmenterless activation (#337 — `parser_for(locales.JA)` without a segmenter behaved exactly like a working parser minus the feature) warns rather than raises because the inert JA registration is itself a pinned property, and a warning is filterable by the caller who wants exactly that. The message must carry every applicable remedy and no inapplicable one (the `ja_segmenter` hint fires only when a Japanese script is among the dead ones). Test fuzzers that legitimately construct such configs suppress the warning by MESSAGE, never by category — a blanket `UserWarning` ignore would mask the next construction diagnostic (`_quiet_parser` in `tests/v2/test_properties.py` is the pattern). diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 1800940a..5729111c 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -115,7 +115,7 @@ Open: [#380](https://github.com/derek73/python-nameparser/issues/380) covers "Be The key is the READING and not the vocabulary, and the docs review caught a first draft that said otherwise ("a word that is also suffix vocabulary declines a post-nominal"). That wording is wrong for both of the words it most obviously describes. `do` is the third member of the particle ∩ suffix-acronym overlap and reports `particle-or-given`: it sits in the AMBIGUOUS acronym half, which had already left it a name word, so no credential reading was ever taken for the attachment to override. And a run of several words is read whole — `Berg, Jan de vd` reports NOTHING though `vd` is suffix vocabulary, because behind `de` the run was read as name words, where the lone `Berg, Jan vd` reports `suffix-or-name`. Both are pinned in tests/v2/cases.py; neither is in any corpus, so nothing else would have caught the wrong statement. The two arms are ORDERED rather than asserted disjoint, and that is a correction to the plan, which recorded them as disjoint by construction ("a suffix-vocabulary particle arrives suffix-roled and IS the whole run"). That premise is a property of the SHIPPED vocabulary, not of the rule: `vd` and `mc` are the only words in both the particle and the unambiguous suffix sets and neither is ambiguous particle vocabulary, so no shipped input reaches both arms — measured, 0 both-arm runs over 29,400 generated names (every ordered pair of the 70-word particle vocabulary in six family-comma shapes: bare, behind a given word, behind a title, before `Jr.`, before `MD`, and with the pair split across the comma). But a caller's `Lexicon` may put one word in both, and the `assert` the plan asked for DID fire on `Berg, Jan zz` under a Lexicon adding `zz` to `particles_ambiguous` and `suffix_acronyms`. `rules.md#A1` says parsing never fails on any input and names exactly one exception, a caller's segmenter; an `AssertionError` out of a stage would be a second. So the code decides instead: the suffix arm wins, because assign had read the word as a post-nominal and the name-word reading was never on the table for the attachment to decline. Pinned in tests/v2/pipeline/test_post_rules.py. The never-given half stays SILENT, and that is the substantive half of the decision. `Jong, Piet de` has no live alternative: `de` is not a name in any reading and is not suffix vocabulary either, so the attachment chose between one reading and nothing, which is not a fork. Reporting it would be "the code got here" keying, the failure this project already has on record (a fork reported for every ambiguous particle at once on `Dr. Van Jr.` — the count is left out on purpose, since decisions.md#vocabulary-collisions says to recompute before quoting it). - Measured over the 1094-name differential corpus: P6's attachment fires on 9 names, of which 5 take the ambiguous arm (`Beethoven, Ludwig van`, `Berg, Jan van der`, `Nguyen, Thi Van`, `Vega, Juan de la`, `der, y van`), 2 the suffix arm (`Berg, Jan vd`, `Berg, abdul vd`) and 2 stay silent (`Jong, Anke de`, `Jong, Piet de`). Seven names gain an ambiguity kind and no role field moves that was not already moving. Read that seven as evidence about the corpus rather than about the blast radius, the caution the 2026-08-16 entry above already gives for this rule: 304 of the 1094 names carry a comma and only 13 end in a particle, 11 of those being rows #379, #380 and this change added themselves. The reach is a SHAPE — every family-comma listing ending in an ambiguous particle, which is to say ordinary Dutch, Flemish and unaccented Vietnamese listings — and a caller holding such records sees a report on all of them, not on seven. The LEDGER shape that follows from that is not the one the plan predicted, and the difference is worth recording because it will recur for any report added to a rule whose fields already move. A new `fields = ["_ambiguities"]` rule explains none of the seven: `classify` matches on a rule's `fields` being a SUPERSET of the whole diff, and against 2.0.0 and 2.1.0 these names diff in `middle`/`family` (or `family`/`suffix`) from #379 and #380 as well. So the repair is to WIDEN those two existing rules to `{middle, family, _ambiguities}` and `{family, suffix, _ambiguities}` in both ledgers, not to add a rule. Each widened rule still passes the #452 equality check, since the union over the names it explains is exactly what it declares — the two silent `Jong` names contribute `{middle, family}` and the five ambiguous ones contribute `_ambiguities` on top. 1.4.0 needs nothing at all, having no ambiguity surface to diff. Gate before and after: byte-identical output at all three baselines apart from the provenance header — 1094 names, 229/194/102 intentional, 0 unexplained, and no per-heading count moved. The seven names were already in the diff; only their field sets grew. + Measured over the 1094-name differential corpus: P6's attachment fires on 9 names, of which 5 take the ambiguous arm (`Beethoven, Ludwig van`, `Berg, Jan van der`, `Nguyen, Thi Van`, `Vega, Juan de la`, `der, y van`), 2 the suffix arm (`Berg, Jan vd`, `Berg, abdul vd`) and 2 stay silent (`Jong, Anke de`, `Jong, Piet de`). Seven names gain an ambiguity kind and no role field moves that was not already moving. Read that seven as evidence about the corpus rather than about the blast radius, the caution the 2026-08-16 entry above already gives for this rule: 304 of the 1094 names carry a comma and only 13 end in a particle, 11 of those being rows the #379/#380 arc added itself (this change added four corpus names and not one of them carries a comma, so the 11 is unchanged by it). The reach is a SHAPE — every family-comma listing ending in an ambiguous particle, which is to say ordinary Dutch, Flemish and unaccented Vietnamese listings — and a caller holding such records sees a report on all of them, not on seven. The LEDGER shape that follows from that is not the one the plan predicted, and the difference is worth recording because it will recur for any report added to a rule whose fields already move. A new `fields = ["_ambiguities"]` rule explains none of the seven: `classify` matches on a rule's `fields` being a SUPERSET of the whole diff, and against 2.0.0 and 2.1.0 these names diff in `middle`/`family` (or `family`/`suffix`) from #379 and #380 as well. So the repair is to WIDEN those two existing rules to `{middle, family, _ambiguities}` and `{family, suffix, _ambiguities}` in both ledgers, not to add a rule. Each widened rule still passes the #452 equality check, since the union over the names it explains is exactly what it declares — the two silent `Jong` names contribute `{middle, family}` and the five ambiguous ones contribute `_ambiguities` on top. 1.4.0 needs nothing at all, having no ambiguity surface to diff. Gate before and after: byte-identical output at all three baselines apart from the provenance header — 1094 names, 229/194/102 intentional, 0 unexplained, and no per-heading count moved. The seven names were already in the diff; only their field sets grew. The most visible consequence, and it is not the Vietnamese one: `Beethoven, Ludwig van` — the textbook-correct Dutch listing, read exactly right — now carries a report. That is honest rather than regrettable. The parser cannot separate it from `Nguyen, Thi Van`; the two are the same string shape over the same vocabulary, and a report on one is a report on the other. A caller who wants only the doubtful cases has nothing here to filter on, because there is nothing to filter on in the input either. ### M2 — the maiden-marker rule @@ -724,11 +724,17 @@ Declined: - 2026-08-29 #407 — DECIDED: case repair consults the mark #404 introduced, and consults it for the WHOLE PART rather than for a particle standing alone. Until now three views read it and the fourth did not, which is how `parse("ANH DO")` came to report family_base 'DO' and initials 'A. D.' while `capitalized()` returned 'Anh do' — the same word treated as an ordinary name word by three readers and as a particle by the fourth. The scope question is the only real choice here, and standing-alone loses on the first Vietnamese name that tests it: `anh van do` has family `van do`, two particle-vocabulary words and neither of them alone, so a rule keyed on aloneness would capitalize `Anh Do` and leave `Anh van do`, making the same surname read differently for having a given-name word in front of it. rules.md#R2 states the test as "Every word", not "standing alone", and this is the case that shows why it is stated that way. One predicate, four views. - 2026-08-29 — this is a DELIBERATE DEVIATION from 1.4.0, not a parity restoration, said plainly because the surrounding branch work is mostly restorations and a reader will otherwise assume this is one. Measured on the released 1.4.0 wheel: `HumanName('ANH DO').capitalize()` leaves `'Anh do'` and `HumanName('anh van do').capitalize()` leaves `'Anh van do'`; both are `'Anh Do'` and `'Anh Van Do'` on this branch. There was nothing to restore — v1 lowercased on vocabulary membership alone and had no notion of a particle with nothing to join, the mark being a 2.x invention. -- 2026-08-29 — the ACCEPTED COST, agreed before the change rather than discovered after it: a degenerate family that is nothing but particles capitalizes too, so `juan van der` renders `Juan Van Der` where 1.4.0 gave `Juan van der`. This is the same trade rules.md#R2 already took for `family_base` — those words ARE the base, so they are name words, and a parser that called them name words for sorting and particles for display would be the incoherent option. Blast radius over the 1090 names the corpora carried before this change, measured by running both predicates in one process: 4 names move under a plain `capitalized()` — `ANH DO`, `vai la`, `van ma van`, and `der, y van`, which becomes `y Van Der` and keeps its lowercase conjunction. 23 move under `force=True`, the extra 19 being mixed-case inputs the gate holds back (rules.md#R5). +- 2026-08-29 — the ACCEPTED COST, agreed before the change rather than discovered after it: a degenerate family that is nothing but particles capitalizes too, so `juan van der` renders `Juan Van Der` where 1.4.0 gave `Juan van der`. This is the same trade rules.md#R2 already took for `family_base` — those words ARE the base, so they are name words, and a parser that called them name words for sorting and particles for display would be the incoherent option. Blast radius, measured by running both predicates in one process, and stated over BOTH populations because the neighbouring bullets in this entry measure over 1094 and a bare "4 and 23" would read as measured there too. Over the 1090 names the corpora carried before this change: 4 names move under a plain `capitalized()` — `ANH DO`, `vai la`, `van ma van`, and `der, y van`, which becomes `y Van Der` and keeps its lowercase conjunction — and 23 under `force=True`, the extra 19 being mixed-case inputs the gate holds back (rules.md#R5). Over today's 1094 it is 5 and 24: this change's own R4 example rows added `anh van do`, and that name is itself a mover. - 2026-08-29 — THE VERIFICATION LIMIT, which matters more than the change. `tools/differential/compare.py` compares the seven role fields plus `_ambiguities` and nothing else, so `capitalized()` is not a compared field and NO ledger run at any baseline can see this deviation: the corpus counts and the three intentional-diff totals hold across it precisely because the harness is blind to it, and a green run is evidence about roles, not about case. mechanisms.md#FACADE-CONTRACT promises that v1-visible changes are "release-log-classified fixes, each verified in the differential ledger". The first half is kept — docs/release_log.rst carries the 2.2.0 bullet — and the SECOND HALF CANNOT BE, for this change or for any other that moves only case. Standing in its place: rules.md#R4's own example rows (`ANH DO`, `anh van do`, and the `juan de la vega` boundary), rules.md#R5's, and the v1-facade tests in tests/test_capitalization.py. The ones that pin the fix are mutation-checked against deletion of the tag consult; the `juan de la vega` control passes that mutation BY DESIGN, which is what makes it a control rather than a witness. Read the silence as a SCOPE and not as a gap: the harness compares parses, case repair is a render view, and nothing is owed on the tracker for it — the next bullet is why that is a decision rather than a deferral. - 2026-08-29 — WHY CASE IS NOT A COMPARED SURFACE, settled here rather than left open, because "add a column" is the obvious answer to the bullet above and it is the wrong instrument. What the harness is FOR is parsing behavior — tools/differential/README.md says to run it "when touching parsing behavior, and before cutting a release" — and what earns it its two-process cost is that a role is EMERGENT: the eight stages fold over one name and each can move what the next sees, so the reach of a vocabulary or rule edit is not derivable by reading the diff and has to be measured over a corpus instead. Case repair is the opposite kind of thing. `_render._cap_word` is a pure function of `(word, role, tags, lexicon)` with four branches — the particle/conjunction lowercasing, the exceptions map, the Mac/Mc convention, and plain `capitalize` — sitting behind R5's single gate, so an edit to it is legible: you can see what you touched. The measurement agrees with the reasoning. Over the 1094 corpus names the 1.4.0 and 2.1.0 wheels ALREADY disagree on `capitalize(force=True)` for 139 of them (recompute by running `HumanName(n).capitalize(force=True)` under each wheel over the four corpus files deduped, and diffing the two maps; the count was the same over the 1090 names the corpora carried before this change). Nearly every one is an echo of a parse diff the field comparison already classifies — a token that changed role capitalizes differently — so a column would re-report the ledger's own explained diffs in a vocabulary its rules cannot match, and would arrive pre-failed by 139 entries. Isolating the residue means comparing case only where the fields agree, a diff-of-diffs, to reach a class that a unit test reaches directly. So the compared surfaces stay the seven roles plus `_ambiguities`, by design and not by omission, and case is pinned where case belongs: the `rules.md#R4` and `#R5` example rows, and tests/test_capitalization.py. - 2026-08-29 — DEBT DISCHARGED, the obligation the last bullet of decisions.md#R5 left this commit; recorded here so the pointer resolves rather than dangling. The FALSEHOOD: R4 promised repair "vocabulary exceptions (McDonald) included" flatly, which `str(parse('Juan Mcdonald').capitalized())` — `'Juan Mcdonald'` — falsifies, the gate refusing before any vocabulary is consulted. The statement now scopes that promise to where repair acts at all and hands the where to R5. A SECOND error in the same eight words turned up while rewriting them, and is worth recording because it had survived unread for as long as the first: McDonald is not a vocabulary exception. The shipped exceptions are five entries — II, III, IV, M.D., Ph.D. (recompute with `Lexicon.default().capitalization_exceptions_map`) — and `Mc`/`Mac` is a separate convention applied to any name shaped like one, so the old parenthetical illustrated the promise with the one mechanism it was not about. The statement now names an exception the vocabulary actually records (Ph.D.) and the Mac/Mc convention separately. The AMBIGUITY: "an already-correct name comes back unchanged" is true read as idempotence and reads as false under bearer-correctness, and the repair was to say WHICH is meant — a name already written the way repair would write it comes back unchanged — plus one clause refusing the inference a reader might otherwise draw, that a spelling its bearer chose is not spared for having been chosen. That clause is deliberately a REFUSAL and not a carve-out: sparing deliberately single-cased names is behavior the parser does not have and that R5's rationale declines on the record, so `str(parse('bell hooks').capitalized())` is still `'Bell Hooks'`. R4's boundary moved too: the old `"Juan McDonald" → capitalized="Juan McDonald"` row was inert — it passes with R5's gate deleted — so it became a `capitalized_forced=` row, which asserts what repair APPLIES rather than what the gate keeps and fails when the Mac/Mc convention is removed (measured by neutering it in one process: `'Juan Mcdonald'`). The boundary is now `juan de la vega`, a working particle run R4's new clause deliberately does not reach. +- 2026-08-29 — the CONJUNCTION CARVE-OUT, stated in rules.md#R4 rather than left to a code comment, and the reason the comment gave was false. `_cap_word`'s conjunction conjunct is deliberately NOT gated on the unjoined mark, and the behavior is right: R3 already carries the carve-out for initials ("A CONJUNCTION never initials, so a base that is one contributes nothing even then"), and R4 was the half of the pair that carried it nowhere. What the comment claimed instead was an impossibility — `particles` and `conjunctions` are disjoint, so no conjunction token can ever carry the mark — and disjointness is a property of the SHIPPED vocabulary while both sets are public, configurable API. Measured: under `Lexicon.default().add(particles={'y'})`, `anh y van` parses to an all-particle family whose `y` carries `conjunction`, `particle` and the mark at once, and repairs to `Anh y Van`; gating the conjunct too, in one process, gives `Anh Y Van`. So the clause decides something and is not a no-op. It gets NO example row, and that is a limit of the document rather than a choice: the runner parses with the default vocabulary, over which `particles ∩ conjunctions` is empty in the default lexicon and in all four locale packs, so no input string can witness it. `der, y van` is the nearest thing and does not witness it either — the `y` there is a GIVEN word standing beside the all-particle family, not inside it (tests/test_capitalization.py says so at the row). + +- 2026-08-29 — the REPLACE/REVISE BOUNDARY, raised by the review of this branch as a gap needing a systematic fix and corrected by Derek to what it actually is: the documented boundary, with a supported path across it. `_cap_word` keys the particle TEST on lexicon membership but the REPAIR on the unjoined mark, and the mark is a parse product — so `parse('de la').capitalized(force=True)` is `'De La'` while `base.replace(family='de la').capitalized(force=True)` is `'de la'`, `replace()` splicing raw text into a field without classifying it. `Parser.revise()` is the method that classifies it, and its docstring already promised exactly this: each value gets a full sub-parse "so the stable tags survive and the tag-driven views ... behave as if the text had been parsed". Measured: `Parser().revise(base, family='de la').capitalized(force=True)` is `'De La'`. So there is nothing to fix — rules.md#R4 carries it as an Accepted boundary naming `revise()`, not as a deviation, and `revise()`'s enumeration of the tag-driven views gains `capitalized()`, which #407 made the fourth. (#458 would additionally make `replace()` agree, by keying the test on the `particle` tag rather than re-deriving membership from the word — the same shape it already names for `conjunction`/`initial`. That is a convenience, not the resolution.) It is a limit and not a regression either way: 2.1.0 gives `'de la'` on the spliced path too, so only the parsed path moved. + +- 2026-08-29 — WHY THE BOUNDARY WENT UNNOTICED UNTIL #407, which is where a future reader should look for it. For an ALL-PARTICLE part the other three tag-driven views give the same answer through `replace()` and `revise()` alike: measured over `de la`, `van der`, `do`, `de` and `van de la`, all five agree on `family_particles=''`, on a `family_base` holding the whole part, and on initials from every word. They converge because an UNTAGGED part and a MARKED all-particle part reach the same place by different routes — untagged, no word is recognized as a particle; marked, none is ACTING as one — and all three views only ask which words are particles. Case repair is the one view that asks a second question, since it must also decide whether to lowercase, so it is where the two routes first come apart. The mirror case confirms the reading: on a MIXED part the convergence is the other way round — `de la vega` and `van der berg` diverge in all three views between `replace()` and `revise()` (`replace()` reports particles `''` and base `'de la vega'` where `revise()` reports `'de la'` and `'vega'`) and AGREE on case repair, R4's all-particle clause not reaching them. So before #407 the distinction was invisible on exactly the shape the clause is about, and visible only on shapes the clause does not govern. + ### R5 — the case-repair gate - 2026-08-29 (#407 arc) — EXTRACTION, not a decision: the parser is untouched. The two halves of R5 have separate provenance, and conflating them is easy enough that the first draft of this entry did. The REFUSAL — repair skips any name already carrying more than one case — is older than the git history: `git log -S "name == name.upper() or name == name.lower()" --reverse` bottoms out at 45a1539 (2011-02-03), the initial import from svn, where `capitalize()` already opens with that guard and a bare `return`. (A path-filtered search answers 280895b instead, the same-day commit that moved the module into `nameparser/`; the code did not change there.) The OVERRIDE is bf1e0a5, 2016-06-02, which did not add the refusal but wrapped it — `if not (name == name.upper() or ...)` became `if not force and not (...)` — and shipped in 0.4.0 (June 2, 2016; its own release-log line is docs/release_log.rst under that heading). So R5's statement as a whole holds from 0.4.0 on. rules.md had never said any of it, though bf1e0a5's diff shows the API docstring already did: "It will not adjust the case of names entered in mixed case" was there before that commit edited around it. The gap was rules.md's alone. Evidence, measured on the released 1.4.0 wheel — the last v1 release and one of the differential baselines, so a natural thing to measure against and not a release that introduced anything here — and re-measured on this branch today, facade and core agreeing: `HumanName('Shirley Maclaine').capitalize()` leaves `'Shirley Maclaine'` — mixed case, wrong, and kept — while the same name under `force=True` gives `'Shirley MacLaine'`; `HumanName('Juan McDonald').capitalize()` leaves `'Juan McDonald'`. rules.md's own preamble classifies behavior in this position as "pinned-but-undocumented — an extraction gap to close, not a specification", which is why the fix lands in the document rather than in `_render.py`. @@ -736,8 +742,8 @@ Declined: - 2026-08-29 — WHY R5'S STATEMENT WAS REWORDED RATHER THAN GIVEN A THIRD EXAMPLE, recorded at length because the wrong turn was taken twice and the reasoning is reusable. R5 first said a mixed-case name is returned untouched "whether or not its casing is correct". That phrasing reads as a disjunction with two branches, and invites the question of which example witnesses the already-correct one. Nothing can: for a mixed-case name that repair would not alter, kept and repaired are the SAME STRING by construction, so no example line distinguishes a parser with the gate from one without it. The unwitnessable branch is a property of the claim, not a gap in our choice of names — and the bullet above had already said so ("an already-correct input cannot witness a rule whose subject is what gets kept REGARDLESS of correctness") before two review rounds pushed past it. Two rows were tried and both withdrawn. `"Juan McDonald" → capitalized="Juan McDonald"` was INERT (mechanisms.md's inert-measurement class): it passes with R5's gate deleted, which is the whole failure shape that class names. `"Vincent Van Gogh" → capitalized="Vincent Van Gogh"` did discriminate — measured by comparing the shipped call against gate-off behavior, which the forced call reproduces exactly, since deleting the gate is what `force` already does: shipped `'Vincent Van Gogh'`, gate-off `'Vincent van Gogh'`, the particle rule lowercasing `Van` the moment repair runs — but it was REDUNDANT, not complementary. It asserts the same proposition as the `Shirley Maclaine` row above it: repair would change this name, and the gate keeps it anyway. The two differ only in which repair rule would have fired, and which rule repair applies is R4's subject, not R5's. And the frame it was chosen under was itself wrong, which is the more useful half of the lesson (reframed 2026-08-29 on Derek's correction, while R4 was being reworked). The row was picked to be bearer-correct — a name whose casing the bearer would endorse — and that frame drags the document into per-name arguments about whose spelling is right: this one needed `Vincent van Gogh` to be the correct spelling, which P6's own examples contradict, and the tussenvoegsel convention behind them is amended later on this branch anyway. CORRECTNESS DOES NOT ENTER INTO IT. Mixed case is the writer making an explicit choice, and repair defers to that choice rather than judging it — a name kept is not a name endorsed. Read that way the withdrawn row needed no claim about Dutch orthography at all, and the reason it needed one is exactly that the frame was wrong. The fix was to stop claiming the unwitnessable branch: the statement now says a mixed-case name is kept and that whether its casing is right does not enter into the decision, which `Shirley Maclaine` — casing wrong, kept — witnesses whole. GENERAL LESSON, and the reason this is long: when an example cannot be found for half a rule, suspect the STATEMENT before suspecting the example set. A phrasing that promises more branches than the behavior has will absorb inert examples indefinitely, each one looking like progress. Caution for the commit that reworks R4 and will choose its own mixed-case rows: avoid a name whose family base is wholly particle vocabulary (`Anh Van Do`, base `Van Do`), because the #407 work changes how those capitalize and would neuter such a row silently; `gogh` and `vega` are in neither `particles` nor `particles_ambiguous`, `van` and `do` are in both. - 2026-08-29 — the override has TWO routes, and the rule states neither, by design. Per-call is the obvious one. The second is a v1 Constants attribute the facade still honors, `force_mixed_case_capitalization` (nameparser/_facade.py resolves it when no per-call value is given; docs/release_log.rst records it as "still honored through the facade"). Measured today: with that attribute set True, `HumanName('Shirley Maclaine')` then a bare `.capitalize()` — no argument at all — LEAVES `'Shirley MacLaine'` (the v1 call mutates in place and returns None, so the name is read back with `str()`). An earlier draft of R5 said "only an explicit request to repair regardless overrides that", which that measurement falsifies; the statement now says repair was asked for anyway, without saying by what route (it read `repair regardless of how the name is cased was asked for` until 2026-08-29, when that phrasing turned out to have a second reading -- see the override bullet below), because rules.md is implementation-free by its own preamble. - 2026-08-29 — `capitalized_forced` is a test-side pseudo-field, not a parser field. The route the EXAMPLES use is the per-call argument to `capitalized()`, and an argument is not a policy, locale or extras gate — the only three things rules.md's grammar admits in an example's annotation slot — so it cannot ride that slot, and the doc runner grew a resolver branch instead. (The facade attribute above is a second route to the same behavior, not a second thing to assert; the core takes the argument only.) The R5 block's example lines are asserted by the suite like any others. They do enlarge one file: `corpus_rules.jsonl` is generated from this document's examples, and `Shirley Maclaine` had not been among them, so regenerating adds it (235 lines, from 234). Whether that moves the DIFFERENTIAL population is a separate question with a per-name answer, and the general answer is that a rules.md example CAN move it: the harness dedupes across corpus files, so a new example name costs a population slot exactly when no other corpus already carries that same string. `Shirley Maclaine` was already in `corpus.jsonl`, so it costs nothing and the population holds at 1090 — measured, not assumed. The row withdrawn above is the counter-case, and worth keeping for it: `Vincent Van Gogh` appears in no other corpus (`corpus.jsonl` and `corpus_issues.jsonl` carry `Vincent van Gogh`, a DIFFERENT string that does not dedupe against it), and while it was in the block the population read 1091. Neither name needed a ledger rule, which is a measurement rather than a consequence of the above: the gate exits 0 with 0 unexplained at all three baselines either way. -- 2026-08-29 — WHAT THE OVERRIDE DOES AND DOES NOT PROMISE, from Derek's framing of R5 and then measured, because the framing implies a property that is ALMOST true and the gap is the useful part. The framing first, and it supersedes the correctness talk elsewhere in this entry: mixed case is the writer making an explicit choice, and repair defers to that choice instead of judging it. Nothing is being called correct or incorrect — a name kept is a name whose writer said something about it, and a name repaired is one whose writer did not. The property that seems to follow is that asking for repair REGARDLESS should ignore the given casing entirely, so one name repairs to one string however it was written. MEASURED over the 1094 corpus names and it does NOT hold: `capitalize(force=True)` differs from uppercasing the input and calling `capitalize()` for 63 names, and from lowercasing it for 16. Two things about those numbers are the opposite of what one would guess. UPPERCASE IS THE WORSE DIRECTION, not the clean one. And the misses are not merely the pipeline's case-sensitivity leaking in: of the 63, only 25 move a role at all, and the other 38 parse byte-identically and diverge inside the repair. The mechanism is v1's initial carve-out — a conjunction is not lowercased where it is written initial-shaped, and initial-shaped means one CAPITAL letter — so uppercasing turns every one-letter conjunction into an initial (`Velasquez y Garcia, Dr. Juan Q.` forced keeps `y`; the same name uppercased then repaired gives `Y`, with a byte-identical partition either side -- the comma form is the one to cite here, the space-written `Dr. Juan Q. Velasquez y Garcia` being a member of the 25 whose roles DO move), and lowercasing turns a middle initial `E` into the Italian conjunction. 1.4.0 does the same (`JUAN Y GARCIA` capitalizes to `Juan Y Garcia`), so this is inherited, and it is recorded here rather than fixed here. Recompute both directions by running the two forms over the four corpus files deduped and diffing. -- 2026-08-29 — and therefore NOT stated in rules.md, which is a deliberate choice rather than an oversight. The document's examples are keyed on input STRINGS, so any statement of the property invites exactly the test that falsifies it — re-case the input, expect the same output — and the counterexamples are already in the corpora. The property is true of the repair given a parse, and rules.md speaks input-to-output; a rule stating it would be over-broad in the one direction a reader would check. What R5's statement says is enough for the promise that IS kept: a mixed-case name is kept unless repair was asked for anyway. That clause was REWORDED for this, and the reword is the whole point rather than a tidy-up. It read `unless repair regardless of how the name is cased was asked for`, which carries two readings -- the intended one, that the request overrides the keeping, and a second one, that the repair disregards the input's casing, which is this property in nearly this bullet's own words. A reader taking the second reading would run the re-casing test predicted above, land on `Velasquez y Garcia, Dr. Juan Q.` (in the corpus today), and conclude the RULE is wrong when only the phrasing was. Nine words, and they asserted the thing the paragraph exists to deny. The property is pinned in tests/test_capitalization.py instead, over names carrying no single-letter word whose class case decides, with `juan y garcia` beside it as the recorded exception. R5's example block gains `"SHIRLEY MACLAINE" → capitalized="Shirley MacLaine"` from this work, and it earns its place on its own ground rather than as half of a convergence pair: it is the only row in the block that fails when the gate is narrowed to lowercase-only, every other row passing that mutation. Measured three ways — gate deleted (passes, so it does not witness the gate's existence), gate narrowed to accept only all-lowercase (FAILS, and alone in the block), Mac/Mc convention deleted (fails, with the other two rows). Until it was added, R5 stated that repair acts on a name written entirely in one case and witnessed only the lowercase half of it. +- 2026-08-29 — WHAT THE OVERRIDE DOES AND DOES NOT PROMISE, from Derek's framing of R5 and then measured, because the framing implies a property that is ALMOST true and the gap is the useful part. The framing first, and it supersedes the correctness talk elsewhere in this entry: mixed case is the writer making an explicit choice, and repair defers to that choice instead of judging it. Nothing is being called correct or incorrect — a name kept is a name whose writer said something about it, and a name repaired is one whose writer did not. The property that seems to follow is that asking for repair REGARDLESS should ignore the given casing entirely, so one name repairs to one string however it was written. MEASURED over the 1094 corpus names and it does NOT hold. On the v2 core — `parse(n).capitalized(force=True)` against `parse(n.upper()).capitalized()`, rendering all seven roles — the two differ for 63 names, and the lowercase direction for 16. Naming the surface matters here: through `HumanName` and `str()`, which is what tests/test_capitalization.py uses, the upper count is 62, because the facade's default render spec omits the maiden name and so cannot see `Jane van der Berg née y Jones` (the whole difference, and it is a conjunction inside the maiden name). Two things about those numbers are the opposite of what one would guess. UPPERCASE IS THE WORSE DIRECTION, not the clean one. And the misses are not merely the pipeline's case-sensitivity leaking in: of the 63, only 25 move a role at all, and the other 38 parse byte-identically and diverge inside the repair (25 and 37 through the facade, the missing name being one of the byte-identical ones). The mechanism is v1's initial carve-out — a conjunction is not lowercased where it is written initial-shaped, and initial-shaped means one CAPITAL letter — so uppercasing turns every one-letter conjunction into an initial (`Velasquez y Garcia, Dr. Juan Q.` forced keeps `y`; the same name uppercased then repaired gives `Y`, with a byte-identical partition either side -- the comma form is the one to cite here, the space-written `Dr. Juan Q. Velasquez y Garcia` being a member of the 25 whose roles DO move), and lowercasing turns a middle initial `E` into the Italian conjunction. 1.4.0 does the same (`JUAN Y GARCIA` capitalizes to `Juan Y Garcia`), so this is inherited, and it is recorded here rather than fixed here. Recompute both directions by running the two forms over the four corpus files deduped and diffing. +- 2026-08-29 — and therefore NOT stated in rules.md, which is a deliberate choice rather than an oversight. The document's examples are keyed on input STRINGS, so any statement of the property invites exactly the test that falsifies it — re-case the input, expect the same output — and the counterexamples are already in the corpora. The property is true of the repair given a parse, and rules.md speaks input-to-output; a rule stating it would be over-broad in the one direction a reader would check. What R5's statement says is enough for the promise that IS kept: a mixed-case name is kept unless repair was asked for anyway. That clause was REWORDED for this, and the reword is the whole point rather than a tidy-up. It read `unless repair regardless of how the name is cased was asked for`, which carries two readings -- the intended one, that the request overrides the keeping, and a second one, that the repair disregards the input's casing, which is this property in nearly this bullet's own words. A reader taking the second reading would run the re-casing test predicted above, land on `Velasquez y Garcia, Dr. Juan Q.` (in the corpus today), and conclude the RULE is wrong when only the phrasing was. Nine words, and they asserted the thing the paragraph exists to deny. The property is pinned in tests/test_capitalization.py instead, over names carrying no single-letter word whose class case decides, with `juan y garcia` beside it as the recorded exception. R5's example block gains `"SHIRLEY MACLAINE" → capitalized="Shirley MacLaine"` from this work, and it earns its place on its own ground rather than as half of a convergence pair: it is the only row in the block that fails when the gate is narrowed to lowercase-only, every other row passing that mutation. Measured three ways — gate deleted (passes, so it does not witness the gate's existence), gate narrowed to accept only all-lowercase (FAILS, and alone in the block), Mac/Mc convention deleted (fails, with the other two rows). Until it was added, R5 stated that repair acts on a name written entirely in one case and witnessed only the lowercase half of it. That lowercase half is still `"juan mcdonald"`, which is byte for byte an R4 row as well, and the duplication is deliberate rather than an editing slip: the two rules make different claims about the same line — R4 that the repair honors the Mac/Mc convention, R5 that an all-lowercase name is acted on at all — and dropping it from R5 would leave the gate's lowercase half unwitnessed inside the rule that states the gate. Five other rows already sit under two rules apiece for the same reason (P5/P6 twice, P5/O5, N3/M4, W1/W3). - 2026-08-29 — DEBT this extraction leaves, named so the next commit inherits an obligation rather than a rediscovery. Pulling the gate out into R5 leaves R4 carrying ONE falsehood and ONE ambiguity — different defects wanting different repairs, and `interacts: R5` carries neither, the field being advisory. FALSE: R4 promises repair "vocabulary exceptions (McDonald) included", but `str(parse('Juan Mcdonald').capitalized())` is `'Juan Mcdonald'` — the gate refuses before any vocabulary is consulted, and only `str(parse('Juan Mcdonald').capitalized(force=True))`, `'Juan McDonald'`, reaches the exception. R4 needs its promise scoped to names the gate admits. AMBIGUOUS, not false: R4's "an already-correct name comes back unchanged" means correct by the repair's own conventions, i.e. idempotence, and under that meaning it is true; a reader hears correct as the bearer writes it, and under THAT meaning `str(parse('bell hooks').capitalized())` — `'Bell Hooks'` — looks like a counterexample. It is not one, because `bell hooks` is not already-correct in R4's sense. What R4 owes is a disambiguation of "correct", NOT a narrowing to spare deliberately single-cased names: that would be new behavior, and R5's own rationale declines it on the ground that single case leaves the repair nothing to read. Also for that commit, and inert as things stand: R4's boundary row `"Juan McDonald" → capitalized="Juan McDonald"` passes with R5's gate deleted, exactly like the R5 row that was withdrawn above; rewriting it to `capitalized_forced=` makes it discriminate for R4's own subject but still witnesses nothing about the already-correct question. This commit adds R5 and touches R4 only on its pointer line, leaving both defects as found rather than half-fixed by a commit whose subject is something else. ### removed-v1-surface diff --git a/docs/design/rules.md b/docs/design/rules.md index c594c581..f0b0dcc1 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -1116,11 +1116,13 @@ R4. Rationale: case repair is a display concern, applied only on request and never destructively. Case repair returns a repaired copy and never mutates the parse. Where it acts at all — R5 decides where — the copy honors the - casing a vocabulary entry records (Ph.D.) and the Mac/Mc surname + casing a vocabulary entry records (Ph.D.) and the Mac/Mc convention (McDonald), not only ordinary word-by-word casing, and a part whose every word is particle vocabulary is repaired as ordinary name words, since none of them is doing a particle's - work there (R2). A name already written the way repair would + work there (R2). A CONJUNCTION keeps its lowercase even inside + such a part, being no name word in any part — the carve-out R3 + states for initials. A name already written the way repair would write it comes back unchanged, measured by repair's own conventions rather than by the bearer's. A spelling written in a single case is repaired even where its bearer meant it, because @@ -1132,6 +1134,17 @@ R4. Rationale: case repair is a display concern, applied only on "anh van do" → capitalized="Anh Van Do" "john smith phd" → capitalized="John Smith Ph.D." "juan de la vega" → capitalized="Juan de la Vega" · boundary + Accepted: the clause reaches a part the parser read. A field + spliced in as raw text after the parse carries no reading of its + own, so a family set that way to "de la" stays lowercase where + those same two words parsed from a name are repaired to "De La". + That is the boundary between splicing text into a field and + revising a field through the parser — revise() classifies the + value, so the repair follows it — rather than a gap between them. + It is also a limit and not a regression: a spliced-in family was + repaired exactly this way before the clause existed. Stated + without an example line because every line here names an input + string, and this shape needs a field edited after the parse. history: decisions.md#R4 · interacts: R2, R3, R5 · implemented: nameparser/_render.py R5. Rationale: mixed case is evidence that the writer cased the name diff --git a/docs/release_log.rst b/docs/release_log.rst index 839bfc98..a6b7dd6f 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -67,7 +67,7 @@ Release Log - Fix case repair lowercasing the words of a family name made only of particle words, where every other view already reads them as ordinary name words: ``HumanName("ANH DO").capitalize()`` gives ``Anh Do`` where it gave ``Anh do``, and ``"anh van do"`` gives ``Anh Van Do`` where it gave ``Anh van do``. A particle earns its name by joining forward to the word it modifies, so a part whose every word is particle vocabulary leaves none of them anything to join; the fix above already made those words anchor ``family_base`` and contribute initials, and case repair now agrees with them rather than reading the same word two ways. The test is the whole part, not a particle standing alone, which is why the two-word family in ``"anh van do"`` moves along with the one-word family in ``"ANH DO"`` -- the same Vietnamese surname, and a standing-alone test would have read it one way behind a given name and another way alone. This DIFFERS FROM 1.4.0 deliberately and does not restore it: 1.4.0 returned ``Anh do``, lowercasing on vocabulary membership alone. The accepted cost is that a degenerate family which is nothing but particles capitalizes too, so ``"juan van der"`` gives ``Juan Van Der`` where 1.4.0 gave ``Juan van der``. A conjunction is untouched by any of this, so ``"der, y van"`` gives ``y Van Der`` -- the family capitalizing while the conjunction keeps the lowercase it always had; and where the particles DO join a name word nothing changes, ``"juan de la vega"`` still giving ``Juan de la Vega``. Case repair is not one of the seven role fields the differential harness compares, so no gate run can see this change either way and none of its counts move; the ``rules.md#R4`` examples and the v1 capitalization tests are what pin it (closes #407) - - Fix a tussenvoegsel attached to the family name after a comma deciding a genuinely uncertain reading and reporting nothing. ``"Van Johnson"`` reports a ``PARTICLE_OR_GIVEN`` ambiguity -- ``Van`` is a Dutch particle and a Vietnamese given name, and the parser has to pick one -- while ``"Nguyen, Thi Van"`` picked the same word the same way, silently, and lost the given name doing it. The attachment now reports the fork it decides, in the kind that names the reading it declined. A particle that could be an ordinary name reports ``PARTICLE_OR_GIVEN``: ``"Nguyen, Thi Van"``, ``"Berg, Jan van der"`` and ``"Vega, Juan de la"`` each gain one, the ``detail`` naming the ambiguous word. A particle the parser had already read as a post-nominal reports ``SUFFIX_OR_NAME`` instead, because the credential reading is what the attachment overrode: ``"Berg, Jan vd"`` gains one, ``vd`` being read as *van der* rather than as the Volunteer Decoration. Which kind you get follows the reading that was overridden rather than the word's vocabulary, so ``"Berg, Jan do"`` reports ``PARTICLE_OR_GIVEN`` even though ``do`` is a postnominal too -- it was already being read as a name word, so no credential reading was overridden. A particle where nothing was overridden reports nothing at all: ``"Jong, Piet de"`` and ``"Jong, Anke de"`` are unchanged, ``de`` being no name in any reading and no postnominal either, and so is ``"Berg, Jan de vd"``, whose run was read as name words whole. Worth knowing before you filter on this: ``"Beethoven, Ludwig van"`` -- the textbook Dutch listing, read exactly right -- now carries a report too. It is the same string shape over the same vocabulary as ``"Nguyen, Thi Van"``, and nothing in the input separates them, so a report on one is a report on both. This adds the report and nothing else: every field these names parse to is exactly what the tussenvoegsel fix above already gave them, and ``ambiguities`` is the only value that grows. Seven of the 1,094 differential corpus names gain a kind against the 2.0.0 and 2.1.0 baselines -- the same seven the tussenvoegsel fix already moved, now carrying ``_ambiguities`` in their diff as well -- and none against 1.4.0, which had no ambiguity reporting at all (closes #405) + - Fix a tussenvoegsel attached to the family name after a comma deciding a genuinely uncertain reading and reporting nothing. ``"Van Johnson"`` reports a ``PARTICLE_OR_GIVEN`` ambiguity -- ``Van`` is a Dutch particle and a Vietnamese given name, and the parser has to pick one -- while ``"Nguyen, Thi Van"`` picked the same word the same way, silently, and lost the given name doing it. The attachment now reports the fork it decides, in the kind that names the reading it declined. A particle that could be an ordinary name reports ``PARTICLE_OR_GIVEN``: ``"Nguyen, Thi Van"``, ``"Berg, Jan van der"`` and ``"Vega, Juan de la"`` each gain one, the ``detail`` naming the ambiguous word. A particle the parser had already read as a post-nominal reports ``SUFFIX_OR_NAME`` instead, because the credential reading is what the attachment overrode: ``"Berg, Jan vd"`` gains one, ``vd`` being read as *van der* rather than as the Volunteer Decoration. Which kind you get follows the reading that was overridden rather than the word's vocabulary, so ``"Berg, Jan do"`` reports ``PARTICLE_OR_GIVEN`` even though ``do`` is a postnominal too -- it was already being read as a name word, so no credential reading was overridden. A particle where nothing was overridden reports nothing at all: ``"Jong, Piet de"`` and ``"Jong, Anke de"`` are unchanged, ``de`` being no name in any reading and no postnominal either, and so is ``"Berg, Jan de vd"``, whose run was read as name words whole. Worth knowing before you filter on this: ``"Beethoven, Ludwig van"`` -- the textbook Dutch listing, read exactly right -- now carries a report too. It is the same string shape over the same vocabulary as ``"Nguyen, Thi Van"``, and nothing in the input separates them, so a report on one is a report on both. This adds the report and nothing else: every field these names parse to is exactly what the tussenvoegsel fix below already gave them, and ``ambiguities`` is the only value that grows. Seven differential corpus names gain a kind against the 2.0.0 and 2.1.0 baselines, seven of the nine the tussenvoegsel fix below already moved -- five of ``fix(#379)``'s seven and both of ``fix(#380)``'s two, now carrying ``_ambiguities`` in their diff as well; the two left out are the pair named unchanged above, ``Jong, Piet de`` and ``Jong, Anke de`` -- and none against 1.4.0, which had no ambiguity reporting at all (closes #405) - Fix a tussenvoegsel after a family comma being parsed as a middle name. Dutch and Belgian alphabetized listings move the particle behind the given name -- ``"Beethoven, Ludwig van"`` is how ``"Ludwig van Beethoven"`` is filed -- and the trailing particle run was read as a middle name rather than as part of the surname: ``"Beethoven, Ludwig van"`` gave middle ``van``, last ``Beethoven``, and ``"Berg, Jan van der"`` gave middle ``van der``. The run now attaches to the family the comma has already named and renders before it, so those read family ``van Beethoven`` and ``van der Berg`` with the given name unchanged. The derived views move with the parse, so ``family_particles`` is ``van`` and ``family_base`` is ``Beethoven`` where they were empty and ``Beethoven`` before. `#130 `_ asked for the split and got it in 1.3.0 as ``last_base``/``last_prefixes``; 2.0 renamed them ``family_base``/``family_particles``. What was wrong until now was the values they reported for this listing. Both halves of the particle vocabulary attach -- never-given ``de`` and may-be-given ``van`` alike -- because after a comma the family is already named and the particle has no other role to take. Two guards bound it. A name whose only given word is the particle keeps it, so ``"Nguyen, Van"`` still reads given ``Van``: the attachment needs a given word to spare. And where the word is BOTH particle and suffix vocabulary the attachment outranks the post-nominal reading, so ``"Berg, Jan vd"`` reads family ``vd Berg`` where 1.4.0 and 2.1 alike gave suffix ``vd`` -- a trailing abbreviation after a family comma is the tussenvoegsel far more often than the decoration it collides with, and the same shape sweeps in ``mc``, which 2.1 also read as a suffix. ``do`` is in ``SUFFIX_ACRONYMS_AMBIGUOUS`` and 2.1 already read a trailing one as a name word, so it attaches by the plain rule rather than by the override (closes #379, closes #380). Names without the comma are untouched: ``"Ludwig van Beethoven"`` already read family ``van Beethoven`` and is byte-identical. One of the 751 differential corpus names moves, ``"Vega, Juan de la"``, at the 1.4.0, 2.0.0 and 2.1.0 baselines alike diff --git a/nameparser/_parser.py b/nameparser/_parser.py index 7d2ea875..dd64c928 100644 --- a/nameparser/_parser.py +++ b/nameparser/_parser.py @@ -161,19 +161,20 @@ def revise(self, name: ParsedName, **fields: str) -> ParsedName: """:meth:`ParsedName.replace` with this parser's vocabulary: each value is tokenized and classified by a full sub-parse, so the stable tags survive and the tag-driven views - (family_particles, initials(), the suffix join) behave as if - the text had been parsed. The value is classified ON ITS OWN, - though -- a word whose reading depends on surrounding context - may classify differently than it would in place (a standalone - "B. S." reads as initials, not a suffix run). The sub-parse's - role choices and ambiguities are discarded -- every harvested - token takes the named field's role -- and its structural - behavior applies: delimiter characters do not become tokens, - and a maiden marker is consumed as in parsing -- mid-value - always, and leading a DELIMITED value under a policy routing - that pair to maiden, where "(née Jones)" revises to "Jones" - while the bare "née Jones" keeps its marker, a leading marker - in an undelimited value being no marker at all (#329). + (family_particles, initials(), the suffix join, and since #407 + capitalized()) behave as if the text had been parsed. The + value is classified ON ITS OWN, though -- a word whose reading + depends on surrounding context may classify differently than + it would in place (a standalone "B. S." reads as initials, not + a suffix run). The sub-parse's role choices and ambiguities + are discarded -- every harvested token takes the named field's + role -- and its structural behavior applies: delimiter + characters do not become tokens, and a maiden marker is + consumed as in parsing -- mid-value always, and leading a + DELIMITED value under a policy routing that pair to maiden, + where "(née Jones)" revises to "Jones" while the bare + "née Jones" keeps its marker, a leading marker in an + undelimited value being no marker at all (#329). Tokens are synthetic (span=None); original is unchanged; a value with no name content (empty, whitespace, or punctuation only) clears the field; ambiguities referencing replaced diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 050d9037..1869dae8 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -433,6 +433,18 @@ def post_rules(state: ParseState) -> ParseState: # parse actually took: assign had read the word as a # post-nominal, so the name-word reading was never on the # table for the attachment to decline. + # + # Both details name the ATTACHMENT and stop there. What + # P6 decides is that the run joins the family the comma + # named instead of standing on its own; how the joined + # words then READ is R2's call, taken by the UNJOINED_TAG + # loop at the end of this stage. The two can disagree: + # `de la, Jan van` attaches `van` and leaves an + # all-particle family, which R2 marks, so every other view + # -- `family_base`, the initials, case repair -- reads + # those words as ordinary name words ('Jan Van De La' + # forced). A detail promising "read as the family's + # particle" would contradict all three. ambiguous = [i for i in run if "vocab:particle-ambiguous" in tokens[i].tags] declined_suffix = [i for i in run @@ -443,8 +455,8 @@ def post_rules(state: ParseState) -> ParseState: AmbiguityKind.SUFFIX_OR_NAME, f"{text!r} written without periods is both a " f"post-nominal and a family-name particle; after a " - f"family comma it is read as the particle rather " - f"than as a post-nominal", + f"family comma it joins the family the comma named " + f"rather than standing as a post-nominal", tuple(run))) elif ambiguous: word = tokens[ambiguous[0]].text @@ -452,8 +464,8 @@ def post_rules(state: ParseState) -> ParseState: AmbiguityKind.PARTICLE_OR_GIVEN, f"{word!r} is both a family-name particle and an " f"ordinary given name; after a family comma " - f"{text!r} is read as the family's particle " - f"rather than as a name word", + f"{text!r} joins the family the comma named " + f"rather than standing as a name word of its own", tuple(run))) for i in run: tokens[i] = dataclasses.replace( diff --git a/nameparser/_pipeline/_state.py b/nameparser/_pipeline/_state.py index 01344203..d7da34f3 100644 --- a/nameparser/_pipeline/_state.py +++ b/nameparser/_pipeline/_state.py @@ -102,13 +102,11 @@ class ParseState: ambiguity P6's attachment reports. Ambiguities are recorded by every stage that DECIDES one -- extract (resolved to a token index by tokenize), segment, - script_segment, classify, - group, assign, and post_rules -- since a fork whose branches are - taken in different stages needs an emitter in each. Post-group, - segments - may retain indices of dropped tokens -- assign iterates pieces, - never segments. This ownership map is pinned by - tests/v2/pipeline/test_state.py. + script_segment, classify, group, assign, and post_rules -- since a + fork whose branches are taken in different stages needs an emitter + in each. Post-group, segments may retain indices of dropped tokens + -- assign iterates pieces, never segments. This ownership map is + pinned by tests/v2/pipeline/test_state.py. segmenter belongs to no stage: like original/lexicon/policy it is passed in at construction by Parser.parse and only ever READ (by diff --git a/nameparser/_render.py b/nameparser/_render.py index 51a24edf..4502329d 100644 --- a/nameparser/_render.py +++ b/nameparser/_render.py @@ -131,14 +131,23 @@ def _cap_word(word: str, role: Role, tags: frozenset[str], # rules.md#R4: "a part whose every word is particle vocabulary is # repaired as ordinary name words, since none of them is doing a # particle's work there" -- UNJOINED_TAG is that mark (#407). - # Only the PARTICLE conjunct is gated on it, and gating the other - # would be a NO-OP rather than a mistake: the mark is applied to a - # part only when every word in it carries "particle" (_post_rules - # and _types._remarked both), a conjunction carries "conjunction" - # instead, and the two vocabularies are disjoint -- so a part - # holding a conjunction is never marked and no conjunction token - # can carry the mark. Left ungated because that is the smaller - # predicate, not because a case turns on it. + # Only the PARTICLE conjunct is gated on it, and that is the rule + # rather than an omission: rules.md#R4 carries the carve-out R3 + # already states for initials -- "A CONJUNCTION never initials, so + # a base that is one contributes nothing even then" -- so a + # conjunction keeps conjunction treatment even inside a part the + # mark has turned into ordinary name words. + # No SHIPPED name witnesses the difference: `particles` and + # `conjunctions` are disjoint in the default vocabulary and in + # every locale pack, so no shipped conjunction can sit in an + # all-particle part and carry the mark. That is a property of the + # shipped DATA, not an invariant -- both sets are public, + # configurable API, and a caller's Lexicon may put one word in + # both, the way _pipeline/_post_rules.py's arms allow for. Measured: + # under `Lexicon.default().add(particles={'y'})`, `anh y van` has + # an all-particle family whose `y` carries both tags and the mark, + # and gives 'Anh y Van'; gating this conjunct too would give + # 'Anh Y Van'. # v1's is_conjunction excludes initials: 'E.' in 'Scott E. Werner' # is an initial, not the conjunction 'e' (pinned live 2026-07-17) if ((normalized in lex.particles and role in (Role.MIDDLE, Role.FAMILY) @@ -175,6 +184,14 @@ def capitalized(name: ParsedName, lexicon: Lexicon | None, *, touched unless force=True; the gate reads the joined token texts (not render() output -- the case gate stays decoupled from spec formatting and the #254 collapse). + The repair reads token TAGS as well as texts: a part whose every + word is particle vocabulary is repaired as ordinary name words, + and the mark saying so comes from the pipeline. A hand-built + token, or one replace() splices in, carries no tags and is + repaired as a plain particle instead: a family set that way to + 'de la' stays 'de la' where the same words parsed give 'De La'. + Parser.revise() is the edit that classifies the value, and gives + 'De La' (rules.md#R4's Accepted boundary). Idempotent: without force, a capitalized result is mixed-case and the gate returns it unchanged; with force, every _cap_word rule is a fixpoint on its own output.""" diff --git a/tests/test_capitalization.py b/tests/test_capitalization.py index 012f4ebd..abbe5262 100644 --- a/tests/test_capitalization.py +++ b/tests/test_capitalization.py @@ -183,14 +183,20 @@ def test_capitalize_working_particle_stays_lowercase(self) -> None: # gives 'y van der' -- but it is worth being exact about what it # does NOT pin, since a reviewer proposed it for that. It cannot # witness the conjunction conjunct being left ungated, and neither - # can any other name: the mark is applied to a part only where - # EVERY word in it carries "particle" (_pipeline/_post_rules and - # _types._remarked alike), `particles` and `conjunctions` are - # disjoint sets, and so a part holding a conjunction is never - # marked and no conjunction token can ever carry the mark. Gating - # that conjunct too would be a semantic no-op, which is why - # mutating it kills nothing. The `y` here is a GIVEN-part word - # besides, not a word of the all-particle family. + # can any other name parsed with the SHIPPED vocabulary: the mark + # is applied to a part only where EVERY word in it carries + # "particle" (_pipeline/_post_rules and _types._remarked alike), + # and `particles` and `conjunctions` are disjoint in the default + # lexicon and in all four locale packs, so no shipped conjunction + # ever sits inside a marked part. That is a property of the + # shipped DATA, not an invariant -- both sets are public API, and + # `Lexicon.default().add(particles={'y'})` parses `anh y van` to + # an all-particle family whose `y` carries the mark, giving + # 'Anh y Van' where gating the conjunct too would give + # 'Anh Y Van'. So the ungated conjunct decides something: it is + # rules.md#R4's stated carve-out, matching R3's for initials, + # rather than a no-op. The `y` here is a GIVEN-part word besides, + # not a word of the all-particle family. def test_capitalize_all_particle_family_beside_a_conjunction(self) -> None: hn = HumanName('der, y van') hn.capitalize() @@ -204,14 +210,22 @@ def test_capitalize_all_particle_family_beside_a_conjunction(self) -> None: # # Measured over the 1094-name differential corpus (2026-08-29), # because the promise is nearly true and the exceptions are the - # whole story. `capitalize(force=True)` differs from uppercasing - # the input and calling `capitalize()` for 63 of 1094 names, and - # from lowercasing it for 16 -- so UPPERCASE IS THE WORSE - # DIRECTION, not the clean one. Nor are the misses merely - # parse-level: of the 63, only 25 move a role, and the other 38 - # parse byte-identically and differ inside the repair itself. - # Recompute by running both forms over the four corpus files - # deduped and diffing. + # whole story. On THIS surface -- `HumanName.capitalize()` and + # `str()`, which is what the test below uses -- forcing repair + # differs from uppercasing the input and calling `capitalize()` + # for 62 of the 1094, and from lowercasing it for 16, so + # UPPERCASE IS THE WORSE DIRECTION, not the clean one. Nor are + # the misses merely parse-level: of the 62, only 25 move a role, + # and the other 37 parse byte-identically and differ inside the + # repair itself. Through the v2 core -- + # `parse(n).capitalized(force=True)`, rendering all seven roles + # -- the counts are 63 and 38, which is what decisions.md#R5 + # states. The one name the facade cannot see is + # `Jane van der Berg nee y Jones`, whose conjunction sits in the + # MAIDEN name: `str(HumanName)` renders the default spec, and + # that spec omits the field. Recompute by running both forms over + # the four corpus files deduped and diffing, on whichever surface + # you name. # # The mechanism is v1's initial carve-out, which _cap_word # documents a few lines from the predicate: a conjunction is not diff --git a/tests/v2/pipeline/test_post_rules.py b/tests/v2/pipeline/test_post_rules.py index 475190ac..6f835a99 100644 --- a/tests/v2/pipeline/test_post_rules.py +++ b/tests/v2/pipeline/test_post_rules.py @@ -9,7 +9,7 @@ from nameparser._policy import (FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST, GIVEN_FIRST, PatronymicRule, Policy, Script) -from nameparser._types import STABLE_TAGS, Role +from nameparser._types import STABLE_TAGS, AmbiguityKind, Role # A reduced lexicon, the convention in every pipeline stage module: a # stage test should not move when shipped vocabulary does. What it must @@ -22,10 +22,13 @@ _LEX = Lexicon( titles=frozenset({"mr", "sir", "dr", "md"}), given_name_titles=frozenset({"sir"}), - particles=frozenset({"de", "der", "ibn", "la", "van"}), + particles=frozenset({"de", "der", "ibn", "la", "van", "vd"}), particles_ambiguous=frozenset({"la", "van"}), suffix_words=frozenset({"jr"}), - suffix_acronyms=frozenset({"md"}), + # `vd` mirrors its shipped dual membership -- particle AND + # UNAMBIGUOUS suffix acronym -- which is what puts a name on P6's + # suffix arm below. + suffix_acronyms=frozenset({"md", "vd"}), conjunctions=frozenset({"y"}), bound_given_names=frozenset({"abdul"}), ) @@ -627,3 +630,54 @@ def test_a_dual_membership_word_decides_rather_than_raising() -> None: assert _by_role(out, Role.FAMILY) == "Berg zz" assert _by_role(out, Role.GIVEN) == "Jan" assert [a.kind.value for a in out.ambiguities] == ["suffix-or-name"] + + +def test_the_attachment_report_names_the_whole_run_it_moved() -> None: + # Both arms' detail text and the reported token span, asserted + # here because neither was: mutation-checked before this test + # existed, replacing either f-string with a constant left the + # full suite green, and so did reporting `run[:1]` instead of the + # whole run. _types.py says PARTICLE_OR_GIVEN now has three + # shapes and that `detail` is what tells them apart, and the two + # older shapes DO assert their detail verbatim (test_assign.py), + # so this one owed the same. + # + # A TWO-token run, so the span assertion has something to lose: + # the report is named for `van`, the word that is ambiguous + # vocabulary, and covers `der` with it. cases.py says "one report + # covers the whole run" and nothing pinned it. + # + # The detail names the ATTACHMENT and stops there -- what P6 + # decides is that the run joins the family the comma named. It + # deliberately does not promise how the joined words then READ, + # which is R2's call taken later in the same stage and can go the + # other way ("de la, Jan van" leaves an all-particle family whose + # words every other view reads as name words). + out = _parsed("Berg, Jan van der") + assert _by_role(out, Role.FAMILY) == "Berg van der" + (amb,) = out.ambiguities + assert amb.kind is AmbiguityKind.PARTICLE_OR_GIVEN + assert amb.detail == ( + "'van' is both a family-name particle and an ordinary given " + "name; after a family comma 'van der' joins the family the " + "comma named rather than standing as a name word of its own") + assert [out.tokens[i].text for i in amb.indices] == ["van", "der"] + + +def test_the_suffix_arm_report_names_the_post_nominal_it_declined() -> None: + # The other arm, verbatim for the same reason. `vd` is particle + # AND unambiguous suffix vocabulary, so assign read the trailing + # one as a post-nominal and the attachment overrode that reading + # -- which is the branch the kind names (#405), and the detail + # follows _assign.py's "written without periods" template for the + # same fork. + out = _parsed("Berg, Jan vd") + assert _by_role(out, Role.FAMILY) == "Berg vd" + (amb,) = out.ambiguities + assert amb.kind is AmbiguityKind.SUFFIX_OR_NAME + assert amb.detail == ( + "'vd' written without periods is both a post-nominal and a " + "family-name particle; after a family comma it joins the " + "family the comma named rather than standing as a " + "post-nominal") + assert [out.tokens[i].text for i in amb.indices] == ["vd"]