From c67cdd0d5cffa5ac27062da1fc04c370097b44bd Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sat, 29 Aug 2026 18:19:39 -0700 Subject: [PATCH 01/14] fix(render,docs): case repair reads the conjunction tag (#458) `_cap_word` re-ran the conjunction-versus-initial decision at render time from the word's spelling, instead of reading the tag `classify` had already set. Every other view honors that tag; only case repair asked again. It now reads `"conjunction" in tags`. The two copies were not asking the same question, which is why this is more than a tidy-up. `_classify.py` asks `is_initial(token.text)` -- the shape test ANDed with a repertoire test since #320 -- while `_render.py` asked the bare `_INITIAL` pattern, shape only, and asked it per WORD of a token's text rather than per token. So `juan e-f smith` capitalized to `Juan e-F Smith`, the Italian conjunction `e` lowercased inside a hyphenated middle name that the parse had read as one ordinary name word, while `JUAN E-F SMITH` gave `Juan E-F Smith`. Both give `Juan E-F Smith` now. It also retires the hand-sync obligation the module carried in a comment -- "keep in sync with `nameparser/_pipeline/_vocab.py` by hand". A tag read has nothing to keep in sync, and `_render._INITIAL` had no other reader, so it is deleted rather than kept alive for its own sync assertion: an unread copy pins nothing about behavior, and `test_regex_sync` still pins `_vocab._INITIAL` against the public `REGEXES["initial"]`, which is the relationship worth keeping. Measured, re-derived on this branch: 0 of the 1094 differential-corpus names move under `capitalized()` or `capitalized(force=True)`, and 0 move with each name also re-run uppercased and lowercased (6565 name/variant/lexicon rows; the second lexicon puts `y` in `particles` so a token can carry `conjunction`, `particle` and the unjoined mark at once). The R5 force-versus-case counts hold at 62/16 through the facade and 63 through the core. The gate is blind to case repair either way and its counts are unchanged: 229 / 194 / 102 intentional diffs, unexplained 0, at 1.4.0 / 2.0.0 / 2.1.0. What moves outside that population is the hyphenated shape above, and untagged tokens: a value `replace()` splices in carries no reading for repair to honor, so a family set to `de y` now repairs to `de Y` -- rules.md#R4's Accepted boundary read in the other direction, since `revise()` classifies the value and keeps `de y`. The PARTICLE conjunct still keys on lexicon membership and is untouched; the parenthetical in decisions.md's replace/revise bullet that expected #458 to change it too is answered there. Co-Authored-By: Claude Opus 5 --- docs/design/decisions.md | 2 ++ docs/design/mechanisms.md | 2 +- docs/design/rules.md | 9 +++++++ docs/release_log.rst | 2 ++ nameparser/_render.py | 49 +++++++++++++++++------------------- tests/test_capitalization.py | 26 ++++++++++++++++--- tests/v2/test_regex_sync.py | 24 ++++++++++-------- tests/v2/test_render.py | 41 +++++++++++++++++++++++++----- 8 files changed, 107 insertions(+), 48 deletions(-) diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 5729111c..4f97089d 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -733,6 +733,8 @@ Declined: - 2026-08-29 — the REPLACE/REVISE BOUNDARY, raised by the review of this branch as a gap needing a systematic fix and corrected by Derek to what it actually is: the documented boundary, with a supported path across it. `_cap_word` keys the particle TEST on lexicon membership but the REPAIR on the unjoined mark, and the mark is a parse product — so `parse('de la').capitalized(force=True)` is `'De La'` while `base.replace(family='de la').capitalized(force=True)` is `'de la'`, `replace()` splicing raw text into a field without classifying it. `Parser.revise()` is the method that classifies it, and its docstring already promised exactly this: each value gets a full sub-parse "so the stable tags survive and the tag-driven views ... behave as if the text had been parsed". Measured: `Parser().revise(base, family='de la').capitalized(force=True)` is `'De La'`. So there is nothing to fix — rules.md#R4 carries it as an Accepted boundary naming `revise()`, not as a deviation, and `revise()`'s enumeration of the tag-driven views gains `capitalized()`, which #407 made the fourth. (#458 would additionally make `replace()` agree, by keying the test on the `particle` tag rather than re-deriving membership from the word — the same shape it already names for `conjunction`/`initial`. That is a convenience, not the resolution.) It is a limit and not a regression either way: 2.1.0 gives `'de la'` on the spliced path too, so only the parsed path moved. +- 2026-08-29 #458 — DONE: case repair's conjunction test reads the `conjunction` TAG instead of re-deriving the word's class at render time, so a view honors the decision the parse recorded (mechanisms.md#VOCAB-TAGS) rather than taking it again. The two copies of the question were not the same question, which is the reason this was worth doing over a corpus that cannot see it: `_classify.py` asks `is_initial(token.text)` — the shape test ANDed with a repertoire test since #320 — while `_render.py` asked a bare pattern, shape only, over each WORD of a token's text rather than the token. It also retires the hand-sync obligation the module carried in a comment ("keep in sync with `nameparser/_pipeline/_vocab.py` by hand"): a tag read has nothing to keep in sync, and `_render._INITIAL` is deleted, its last reader gone. MEASURED, re-derived on this branch rather than taken from the issue: 0 of the 1094 corpus names move under `capitalized()` or `capitalized(force=True)`, and 0 move with each name also re-run uppercased and lowercased (6565 name/variant/lexicon rows, the second lexicon being `Lexicon.default().add(particles={'y'})`); the R5 force-versus-case counts recorded above hold at 62/16 through the facade and 63 through the core. Movement EXISTS outside that population, and the issue's "no output moves" is a claim about the corpus only: the old predicate re-decided per word, so `juan e-f smith` capitalized to `Juan e-F Smith` — the Italian conjunction inside a hyphenated middle name — and now gives `Juan E-F Smith`, agreeing with the uppercase spelling it always gave. Untagged tokens move the same way: a family `replace()` splices in as `de y` now repairs to `de Y`, the R4 boundary above read in the other direction (`revise()` classifies and keeps `de y`, agreeing with the parse). NOT DONE, and the parenthetical in the boundary bullet above should be read as still open: the PARTICLE conjunct still keys on lexicon membership, so `replace()` does not agree there. Making it read the `particle` tag is a second change with its own blast radius — it moves a boundary rules.md#R4 states in prose — and #458 as filed does not ask for it. + - 2026-08-29 — WHY THE BOUNDARY WENT UNNOTICED UNTIL #407, which is where a future reader should look for it. For an ALL-PARTICLE part the other three tag-driven views give the same answer through `replace()` and `revise()` alike: measured over `de la`, `van der`, `do`, `de` and `van de la`, all five agree on `family_particles=''`, on a `family_base` holding the whole part, and on initials from every word. They converge because an UNTAGGED part and a MARKED all-particle part reach the same place by different routes — untagged, no word is recognized as a particle; marked, none is ACTING as one — and all three views only ask which words are particles. Case repair is the one view that asks a second question, since it must also decide whether to lowercase, so it is where the two routes first come apart. The mirror case confirms the reading: on a MIXED part the convergence is the other way round — `de la vega` and `van der berg` diverge in all three views between `replace()` and `revise()` (`replace()` reports particles `''` and base `'de la vega'` where `revise()` reports `'de la'` and `'vega'`) and AGREE on case repair, R4's all-particle clause not reaching them. So before #407 the distinction was invisible on exactly the shape the clause is about, and visible only on shapes the clause does not govern. ### R5 — the case-repair gate diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 03484464..4e333d9c 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -21,7 +21,7 @@ Problem shape. A rule wants words to RENDER in a different order than they sit i Problem shape. A later stage needs to know what the vocabulary knew about a word. Contract statement. classify tags every token with what the vocabulary knows about it, and later stages test tags — they never re-look a word up. How it works. One lookup site means one answer: a stage that re-derived vocabulary facts could disagree with the stage before it. Stable tags ("particle", "conjunction", "initial") are API; "vocab:"-namespaced ones are not. -Lives in. nameparser/_pipeline/_classify.py (producer); consumers throughout _group/_assign/_post_rules. Reach for it when. A stage is about to import Lexicon to ask about a word classify already saw. +Lives in. nameparser/_pipeline/_classify.py (producer); consumers throughout _group/_assign/_post_rules, and nameparser/_render.py, which is not a stage but reads the same tags to decide what a view shows (#458 moved case repair's conjunction test onto the tag; initials() had read them since 2.0). Reach for it when. A stage is about to import Lexicon to ask about a word classify already saw — or a view is, which is the harder one to notice, since a view legitimately holds a Lexicon for the questions classify never answered. ## PIECES — joining structure survives assignment diff --git a/docs/design/rules.md b/docs/design/rules.md index f0b0dcc1..ba7a1005 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -1145,6 +1145,15 @@ R4. Rationale: case repair is a display concern, applied only on repaired exactly this way before the clause existed. Stated without an example line because every line here names an input string, and this shape needs a field edited after the parse. + Accepted, the same boundary read the other way: the conjunction + carve-out reaches a word the parse read as a conjunction, and + nothing else. A word of that vocabulary standing inside a longer + written word is not one, and neither is one spliced into a field + after the parse — a family set that way to "de y" repairs to "de + Y" where the parsed name keeps the "y" as written. The two + directions are one fact: repair honors the reading the parse + recorded and takes none of its own. revise() classifies the value + here too, so it agrees with the parse in both directions. history: decisions.md#R4 · interacts: R2, R3, R5 · implemented: nameparser/_render.py R5. Rationale: mixed case is evidence that the writer cased the name diff --git a/docs/release_log.rst b/docs/release_log.rst index a6b7dd6f..1e9880cf 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -67,6 +67,8 @@ Release Log - Fix case repair lowercasing the words of a family name made only of particle words, where every other view already reads them as ordinary name words: ``HumanName("ANH DO").capitalize()`` gives ``Anh Do`` where it gave ``Anh do``, and ``"anh van do"`` gives ``Anh Van Do`` where it gave ``Anh van do``. A particle earns its name by joining forward to the word it modifies, so a part whose every word is particle vocabulary leaves none of them anything to join; the fix above already made those words anchor ``family_base`` and contribute initials, and case repair now agrees with them rather than reading the same word two ways. The test is the whole part, not a particle standing alone, which is why the two-word family in ``"anh van do"`` moves along with the one-word family in ``"ANH DO"`` -- the same Vietnamese surname, and a standing-alone test would have read it one way behind a given name and another way alone. This DIFFERS FROM 1.4.0 deliberately and does not restore it: 1.4.0 returned ``Anh do``, lowercasing on vocabulary membership alone. The accepted cost is that a degenerate family which is nothing but particles capitalizes too, so ``"juan van der"`` gives ``Juan Van Der`` where 1.4.0 gave ``Juan van der``. A conjunction is untouched by any of this, so ``"der, y van"`` gives ``y Van Der`` -- the family capitalizing while the conjunction keeps the lowercase it always had; and where the particles DO join a name word nothing changes, ``"juan de la vega"`` still giving ``Juan de la Vega``. Case repair is not one of the seven role fields the differential harness compares, so no gate run can see this change either way and none of its counts move; the ``rules.md#R4`` examples and the v1 capitalization tests are what pin it (closes #407) + - Change case repair to read the parser's own ``conjunction`` tag instead of re-deciding, from the word's spelling, whether a word is a conjunction or an initial. The parse answers that question already -- ``"Scott E. Werner"`` reads ``E.`` as an initial rather than the Italian conjunction -- and the other views honor the answer; case repair asked again, with a shape test applied to each word of a token's text rather than to the token. Two spellings of one name disagreed because of it: ``"juan e-f smith"`` capitalized to ``Juan e-F Smith`` while ``"JUAN E-F SMITH"`` gave ``Juan E-F Smith``; both give ``Juan E-F Smith`` now, ``e-f`` being a middle name and no conjunction of the parse's reading. A conjunction written as a word of its own is untouched, and so is the one-letter carve-out where it applies -- ``"juan y garcia"`` still repairs to ``Juan y Garcia``, ``"JUAN Y GARCIA"`` still to ``Juan Y Garcia``. One consequence for callers who edit fields: a value ``replace()`` splices in carries no reading for repair to honor, so a family set to ``de y`` now repairs to ``de Y``, where ``Parser.revise()`` classifies the value and keeps ``de y`` as the parsed name does. Case repair is not one of the seven role fields the differential harness compares, so no gate run can see this change either way and none of its counts move; measured directly instead, no name of the 1094-name differential corpus moves under ``capitalized()`` or ``capitalized(force=True)``, its uppercased and lowercased spellings included (closes #458) + - Fix a tussenvoegsel attached to the family name after a comma deciding a genuinely uncertain reading and reporting nothing. ``"Van Johnson"`` reports a ``PARTICLE_OR_GIVEN`` ambiguity -- ``Van`` is a Dutch particle and a Vietnamese given name, and the parser has to pick one -- while ``"Nguyen, Thi Van"`` picked the same word the same way, silently, and lost the given name doing it. The attachment now reports the fork it decides, in the kind that names the reading it declined. A particle that could be an ordinary name reports ``PARTICLE_OR_GIVEN``: ``"Nguyen, Thi Van"``, ``"Berg, Jan van der"`` and ``"Vega, Juan de la"`` each gain one, the ``detail`` naming the ambiguous word. A particle the parser had already read as a post-nominal reports ``SUFFIX_OR_NAME`` instead, because the credential reading is what the attachment overrode: ``"Berg, Jan vd"`` gains one, ``vd`` being read as *van der* rather than as the Volunteer Decoration. Which kind you get follows the reading that was overridden rather than the word's vocabulary, so ``"Berg, Jan do"`` reports ``PARTICLE_OR_GIVEN`` even though ``do`` is a postnominal too -- it was already being read as a name word, so no credential reading was overridden. A particle where nothing was overridden reports nothing at all: ``"Jong, Piet de"`` and ``"Jong, Anke de"`` are unchanged, ``de`` being no name in any reading and no postnominal either, and so is ``"Berg, Jan de vd"``, whose run was read as name words whole. Worth knowing before you filter on this: ``"Beethoven, Ludwig van"`` -- the textbook Dutch listing, read exactly right -- now carries a report too. It is the same string shape over the same vocabulary as ``"Nguyen, Thi Van"``, and nothing in the input separates them, so a report on one is a report on both. This adds the report and nothing else: every field these names parse to is exactly what the tussenvoegsel fix below already gave them, and ``ambiguities`` is the only value that grows. Seven differential corpus names gain a kind against the 2.0.0 and 2.1.0 baselines, seven of the nine the tussenvoegsel fix below already moved -- five of ``fix(#379)``'s seven and both of ``fix(#380)``'s two, now carrying ``_ambiguities`` in their diff as well; the two left out are the pair named unchanged above, ``Jong, Piet de`` and ``Jong, Anke de`` -- and none against 1.4.0, which had no ambiguity reporting at all (closes #405) - Fix a tussenvoegsel after a family comma being parsed as a middle name. Dutch and Belgian alphabetized listings move the particle behind the given name -- ``"Beethoven, Ludwig van"`` is how ``"Ludwig van Beethoven"`` is filed -- and the trailing particle run was read as a middle name rather than as part of the surname: ``"Beethoven, Ludwig van"`` gave middle ``van``, last ``Beethoven``, and ``"Berg, Jan van der"`` gave middle ``van der``. The run now attaches to the family the comma has already named and renders before it, so those read family ``van Beethoven`` and ``van der Berg`` with the given name unchanged. The derived views move with the parse, so ``family_particles`` is ``van`` and ``family_base`` is ``Beethoven`` where they were empty and ``Beethoven`` before. `#130 `_ asked for the split and got it in 1.3.0 as ``last_base``/``last_prefixes``; 2.0 renamed them ``family_base``/``family_particles``. What was wrong until now was the values they reported for this listing. Both halves of the particle vocabulary attach -- never-given ``de`` and may-be-given ``van`` alike -- because after a comma the family is already named and the particle has no other role to take. Two guards bound it. A name whose only given word is the particle keeps it, so ``"Nguyen, Van"`` still reads given ``Van``: the attachment needs a given word to spare. And where the word is BOTH particle and suffix vocabulary the attachment outranks the post-nominal reading, so ``"Berg, Jan vd"`` reads family ``vd Berg`` where 1.4.0 and 2.1 alike gave suffix ``vd`` -- a trailing abbreviation after a family comma is the tussenvoegsel far more often than the decoration it collides with, and the same shape sweeps in ``mc``, which 2.1 also read as a suffix. ``do`` is in ``SUFFIX_ACRONYMS_AMBIGUOUS`` and 2.1 already read a trailing one as a name word, so it attaches by the plain rule rather than by the override (closes #379, closes #380). Names without the comma are untouched: ``"Ludwig van Beethoven"`` already read family ``van Beethoven`` and is byte-identical. One of the 751 differential corpus names moves, ``"Vega, Juan de la"``, at the 1.4.0, 2.0.0 and 2.1.0 baselines alike diff --git a/nameparser/_render.py b/nameparser/_render.py index 4502329d..e6696f20 100644 --- a/nameparser/_render.py +++ b/nameparser/_render.py @@ -39,24 +39,6 @@ #: Not STABLE_TAGS -- that also contains "initial", which must contribute. _SKIP_TAGS = frozenset({"particle", "conjunction"}) -# Ported verbatim from v1 (nameparser/config/regexes.py "initial", -# minus the empty alternative) -- layering forbids importing the -# pipeline here; keep in sync with _pipeline/_vocab.py by hand. -# Deliberately NOT composed with that module's repertoire test (#320): -# layering forbids the import, and nothing here needs it. The only use -# is v1's conjunction carve-out in _cap_word below, which this pattern -# can only reach once `normalized in lex.conjunctions` already holds -- -# and no CJK token reaches that, the shipped vocabulary carrying no CJK -# conjunction or particle in the default lexicon or in any locale pack. -# That is a property of the shipped DATA, not an invariant -- conjunctions -# is public, configurable API -- but the divergence stays harmless if a -# user adds one: CJK is caseless, so the carve-out's word.lower() and the -# fall-through's word.capitalize() return the same string either way. -# So the two copies keep identical PATTERNS and divergent PREDICATES; -# test_regex_sync pins the patterns, which is the promise being kept. -_INITIAL = re.compile(r"^(\w\.|[A-Z])$") - - def _collapse(rendered: str) -> str: """The #254 collapse: empty fields substitute '' and every artifact of that is removed -- dangling empty-nickname wrappers, space runs, @@ -148,12 +130,24 @@ def _cap_word(word: str, role: Role, tags: frozenset[str], # an all-particle family whose `y` carries both tags and the mark, # and gives 'Anh y Van'; gating this conjunct too would give # 'Anh Y Van'. - # v1's is_conjunction excludes initials: 'E.' in 'Scott E. Werner' - # is an initial, not the conjunction 'e' (pinned live 2026-07-17) + # That conjunct reads the TAG, not the word (#458). classify takes + # the conjunction-versus-initial decision once, over the whole + # token -- v1's is_conjunction excludes initials, so 'E.' in + # 'Scott E. Werner' is an initial and is never tagged (pinned live + # 2026-07-17) -- and a view honors that decision rather than taking + # it again from the spelling (mechanisms.md#VOCAB-TAGS: "later + # stages test tags"). Asking again was not even the same question: + # the copy of the initial pattern that stood here was the SHAPE + # half alone, and it re-decided per WORD of a token's text, so + # 'juan e-f smith' capitalized to 'Juan e-F Smith'. What moves the + # other way is a token the parse never saw: one hand-built, or + # spliced in by replace(), carries no decision to honor and is + # repaired as an ordinary name word -- rules.md#R4's Accepted + # boundary, which the particle conjunct above draws in the opposite + # direction because it still asks the vocabulary itself. if ((normalized in lex.particles and role in (Role.MIDDLE, Role.FAMILY) and UNJOINED_TAG not in tags) - or (normalized in lex.conjunctions - and not _INITIAL.fullmatch(word))): + or "conjunction" in tags): return word.lower() # v1 cap_word tries the edge-stripped form, then the period-free # form ('Ph.D.' -> 'ph.d' -> 'phd' hits the exceptions map) @@ -186,10 +180,13 @@ def capitalized(name: ParsedName, lexicon: Lexicon | None, *, formatting and the #254 collapse). The repair reads token TAGS as well as texts: a part whose every word is particle vocabulary is repaired as ordinary name words, - and the mark saying so comes from the pipeline. A hand-built - token, or one replace() splices in, carries no tags and is - repaired as a plain particle instead: a family set that way to - 'de la' stays 'de la' where the same words parsed give 'De La'. + and the mark saying so comes from the pipeline, as does the + reading that a word is a conjunction rather than an initial. A + hand-built token, or one replace() splices in, carries no tags: + it is repaired as a plain particle, and as no conjunction at all. + A family set that way to 'de la' stays 'de la' where the same + words parsed give 'De La', while one set to 'de y' gives 'de Y' + where the parse would keep the conjunction lowercase. Parser.revise() is the edit that classifies the value, and gives 'De La' (rules.md#R4's Accepted boundary). Idempotent: without force, a capitalized result is mixed-case and diff --git a/tests/test_capitalization.py b/tests/test_capitalization.py index abbe5262..3ddbff4b 100644 --- a/tests/test_capitalization.py +++ b/tests/test_capitalization.py @@ -227,10 +227,12 @@ def test_capitalize_all_particle_family_beside_a_conjunction(self) -> None: # the four corpus files deduped and diffing, on whichever surface # you name. # - # The mechanism is v1's initial carve-out, which _cap_word - # documents a few lines from the predicate: a conjunction is not - # lowercased where it is written initial-shaped, and - # initial-shaped means one CAPITAL letter. Uppercase a name and + # The mechanism is v1's initial carve-out, taken in the PARSE + # since #458 and read off the tag by the repair: a word of the + # conjunction vocabulary is not tagged one where it is written + # initial-shaped, and initial-shaped means one CAPITAL letter + # (nameparser/_pipeline/_classify.py, and the tests beside it -- + # the repair no longer asks). Uppercase a name and # every one-letter conjunction becomes an initial; lowercase one # and a middle initial `E` becomes the Italian conjunction. So # the property is pinned over names carrying no single-letter @@ -262,3 +264,19 @@ def test_a_one_letter_conjunction_is_case_sensitive_to_repair(self) -> None: uppered.capitalize() # 'Y' is initial-shaped, so the conjunction rule declines it self.m(str(uppered), 'Juan Y Garcia', uppered) + + # The other half of #458, and the half that MOVED: the decision is + # the whole token's, so a conjunction word inside a longer token is + # not one. `e` is the Italian conjunction and `e-f` is a middle + # name; before #458 the repair re-ran the test over each word of a + # token's text and gave 'Juan e-F Smith'. The uppercase spelling is + # here to show the old answer was not even self-consistent: it read + # `E` as initial-shaped and capitalized, so the same name repaired + # to two different strings depending on how it was written. + def test_a_conjunction_inside_a_longer_token_is_a_name_word(self) -> None: + lowered = HumanName('juan e-f smith') + lowered.capitalize(force=True) + self.m(str(lowered), 'Juan E-F Smith', lowered) + uppered = HumanName('JUAN E-F SMITH') + uppered.capitalize() + self.m(str(uppered), 'Juan E-F Smith', uppered) diff --git a/tests/v2/test_regex_sync.py b/tests/v2/test_regex_sync.py index 61cd4a24..038905e0 100644 --- a/tests/v2/test_regex_sync.py +++ b/tests/v2/test_regex_sync.py @@ -72,16 +72,19 @@ def test_patronymic_patterns_match_config() -> None: assert copy.flags == source.flags, key -def test_initial_copies_agree_with_each_other_and_config() -> None: - # _vocab._INITIAL and _render._INITIAL are both v1's "initial" - # pattern minus its trailing "?" (documented at _render.py's - # _INITIAL definition: the two call sites always fullmatch a +def test_initial_copy_matches_config_minus_the_empty_alternative() -> None: + # _vocab._INITIAL is v1's "initial" pattern minus its trailing "?" + # (documented at the definition: its callers always fullmatch a # non-empty token, so the empty-string alternative is dropped on - # purpose). Assert both the internal-copy agreement and the exact, - # documented relationship to the config source, so a future edit to - # either side that breaks the relationship fails loudly here - # instead of silently drifting. - assert _vocab._INITIAL.pattern == _render._INITIAL.pattern + # purpose). Assert the exact, documented relationship to the config + # source, so a future edit to either side that breaks the + # relationship fails loudly here instead of silently drifting. + # There was a second copy, in _render, and this test also asserted + # the two agreed. It went with #458: case repair stopped asking + # whether a word is initial-shaped -- classify records that, and a + # view reads the tag -- leaving the constant with no reader. A copy + # kept alive for its own sync assertion pins nothing about + # behavior, so it was deleted rather than commented. source = _config.REGEXES["initial"] # config's pattern is the pipeline copy with an extra "?" spliced in # just before the trailing "$", making the whole group optional. @@ -118,8 +121,7 @@ def test_initial_copies_agree_with_each_other_and_config() -> None: ("_vocab", "_PERIOD_NOT_AT_END"): "period_not_at_end", # Deliberately NOT a straight copy -- pinned by the dedicated tests # above, which assert the documented RELATIONSHIP instead: - ("_render", "_INITIAL"): None, # config's pattern minus one "?" - ("_vocab", "_INITIAL"): None, # same + ("_vocab", "_INITIAL"): None, # config's pattern minus one "?" ("_tokenize", "_BIDI"): None, # re_bidi, not a REGEXES key # Mirrors _pipeline._state.COMMA_CHARS, not nameparser.config ("_render", "_COMMA_CHAR"): None, diff --git a/tests/v2/test_render.py b/tests/v2/test_render.py index 224df8cd..922b33e0 100644 --- a/tests/v2/test_render.py +++ b/tests/v2/test_render.py @@ -211,11 +211,12 @@ def test_capitalized_with_explicit_lexicon() -> None: assert out.suffix == "Phd" -def test_capitalized_lowers_conjunctions_but_not_initial_shapes() -> None: - # v1's is_conjunction excludes initial-shaped words: a lowercase - # 'y' lowers, but an uppercase 'Y' looks like an initial and - # capitalizes ('JOSE ORTEGA Y GASSET' -> 'Jose Ortega Y Gasset', - # pinned live against v1.4 2026-07-17) +def test_capitalized_lowers_the_words_the_parse_tagged_conjunction() -> None: + # #458: whether a word is the conjunction or an initial is + # classify's decision, recorded as the tag; repair honors the tag + # and never asks the word. Both halves of that are asserted here, + # since a repair that lowered every conjunction-vocabulary word + # would pass the first alone. # 01234567890123456789 pn = _pn("juan ortega y gasset", [ Token("juan", Span(0, 4), Role.GIVEN), @@ -225,13 +226,41 @@ def test_capitalized_lowers_conjunctions_but_not_initial_shapes() -> None: ]) out = pn.capitalized(force=True) assert out.family == "Ortega y Gasset" + # v1's is_conjunction excludes initial-shaped words, so classify + # withholds the tag from an uppercase 'Y' and repair capitalizes it + # ('JOSE ORTEGA Y GASSET' -> 'Jose Ortega Y Gasset', pinned live + # against v1.4 2026-07-17 and pinned end to end in + # tests/test_capitalization.py). These tokens are what a parse of + # the uppercase name builds. upper = _pn("JUAN ORTEGA Y GASSET", [ Token("JUAN", Span(0, 4), Role.GIVEN), Token("ORTEGA", Span(5, 11), Role.FAMILY), - Token("Y", Span(12, 13), Role.FAMILY, frozenset({"conjunction"})), + Token("Y", Span(12, 13), Role.FAMILY), Token("GASSET", Span(14, 20), Role.FAMILY), ]) assert upper.capitalized(force=True).family == "Ortega Y Gasset" + # The tag decides even against the shape: a token tagged + # conjunction lowers however it is written. Nothing shipped builds + # this token -- that is the point, since the old predicate could + # not have honored it. + tagged = _pn("JUAN ORTEGA Y GASSET", [ + Token("JUAN", Span(0, 4), Role.GIVEN), + Token("ORTEGA", Span(5, 11), Role.FAMILY), + Token("Y", Span(12, 13), Role.FAMILY, frozenset({"conjunction"})), + Token("GASSET", Span(14, 20), Role.FAMILY), + ]) + assert tagged.capitalized(force=True).family == "Ortega y Gasset" + # ... and an untagged word of the conjunction vocabulary is an + # ordinary name word. The reachable shape is a token whose text is + # more than one word, since the repair walks a token's words while + # the tag is the whole token's: 'juan e-f smith' capitalized to + # 'Juan e-F Smith' while the old predicate re-decided per word. + hyphenated = _pn("juan e-f smith", [ + Token("juan", Span(0, 4), Role.GIVEN), + Token("e-f", Span(5, 8), Role.MIDDLE), + Token("smith", Span(9, 14), Role.FAMILY), + ]) + assert hyphenated.capitalized(force=True).middle == "E-F" def test_capitalized_rebuilds_ambiguity_tokens() -> None: From 4a414187bd309846906df5f78d3a0c3d5fad90ca Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sat, 29 Aug 2026 18:23:48 -0700 Subject: [PATCH 02/14] fix(render,docs): a conjunction never initials, even in an all-particle part (#461) The unjoined mark readmits the words of an all-particle part to `initials()`, because none of them is acting as a particle there. It says nothing about a conjunction, and rules.md#R3 excludes one "even then" -- "A CONJUNCTION never initials, so a base that is one contributes nothing even then". The override readmitted both tags, so the rule and the code disagreed. It is now narrowed to the tag the mark is about. The reach is a caller's own vocabulary alone: `particles` and `conjunctions` are disjoint in the defaults and in every locale pack, so 0 of the 1094 corpus names move and the gate holds at 229 / 194 / 102 intentional diffs, unexplained 0, at 1.4.0 / 2.0.0 / 2.1.0. Under `Lexicon.default().add(particles={'y'})` three corpus names move, all in the fixed direction: `Juan de y` J. d. y. -> J. d., `der, y van` d. y. v. -> d. v., and `johnny y` j. y. -> j., the last being R3's "a base that is one contributes nothing" read literally. What makes it worth fixing over a shape nothing shipped can reach is that the two views disagreed about ONE token: `Anh y Van` capitalized as `Anh y Van`, honoring the carve-out, and initialed as `A. y. V.`, ignoring it. So the test asserts both views on the same parse. That also closes a hole on the other side: case repair's conjunction conjunct is deliberately ungated on the mark for exactly this rule, a decision recorded in decisions.md#R4 and argued at the code -- and gating it passed the entire suite until this test existed. R3 gains prose rather than an example line, for the same reason R4's carve-out has none: the doc runner parses with the default vocabulary, over which the shape is unreachable, so the pin lives in tests/v2/test_render.py and the document says where. Co-Authored-By: Claude Opus 5 --- docs/design/decisions.md | 2 ++ docs/design/rules.md | 9 ++++++++ docs/release_log.rst | 2 ++ nameparser/_render.py | 26 ++++++++++++++++----- tests/v2/test_render.py | 49 +++++++++++++++++++++++++++++++++++++++- 5 files changed, 81 insertions(+), 7 deletions(-) diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 4f97089d..044a6385 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -716,6 +716,8 @@ Declined: "Juan van der" stays "J." (no borne name, no base, and initials of a bare particle run would be nonsense). The general lesson, worth more than this instance: a deviates: marker gets written on the rule whose STATEMENT changed, but a rule can change another rule's OUTPUT without touching its statement, and nothing looks for that — the runner asserts per example line, so an unmarked downstream rule stays green precisely because its own examples avoid the affected input. When adding a marker, walk the changed rule's `interacts:` targets and ask whether any of THEIR examples move. +- 2026-08-29 #461 — FIXED: the unjoined mark readmitted BOTH tags to `initials()`, and it is a statement about one of them. R3 says a conjunction never initials "even then" — even inside the all-particle part R2 makes ordinary name words — and the override readmitted a conjunction along with the particles it is about. The reach is a caller's vocabulary alone: `particles` and `conjunctions` are disjoint in the defaults and in every locale pack, so 0 of the 1094 corpus names move, and the gate's counts hold at 229 / 194 / 102 with 0 unexplained. Under `Lexicon.default().add(particles={'y'})` three corpus names do move, all in the fixed direction: `Juan de y` from `J. d. y.` to `J. d.`, `der, y van` from `d. y. v.` to `d. v.`, and `johnny y` from `j. y.` to `j.` — the last being R3's "a base that is one contributes nothing" read literally. What makes this worth fixing over a shape nothing shipped can reach is that the two views disagreed about ONE TOKEN: `Anh y Van` capitalized as `Anh y Van`, honoring the carve-out, and initialed as `A. y. V.`, ignoring it. Case repair's conjunction conjunct was deliberately ungated on the mark for exactly this reason (rules.md#R4, and the CONJUNCTION CARVE-OUT bullet under it) — that decision was recorded, argued and unpinned, and gating the conjunct passed the whole suite until this fix's test went in. It now asserts both views on the same parse, so the pair moves together or fails together. R3 gains prose rather than an example line, for the reason the R4 carve-out gains none: the doc runner parses with the default vocabulary, over which the shape is unreachable. + Declined: - 2026-08-18 — the GROUPING half of #404: a particle run that joins nothing does not chain, so "Jong van der" would split into middle 'van' plus family 'der'. Measured and rejected, though NOT for the reason first recorded here. The first draft said the split makes the family "no longer all-particle so the base fix stops firing" — false, and `der` and `la` are both shipped particles, so a family of either IS all-particle and the rule fires on it ("Juan Smith der" gives base 'der'). What the split actually costs is the SCOPE of the base and a stray particle relocated: grouping can decline to merge but cannot keep the words apart, because roles re-assemble them and two adjacent same-role pieces are one part at the field level. "Juan Smith van der" becomes middle 'Smith van', family 'der' — a base of 'der' rather than 'van der', and a middle name nobody wrote. Keeping the run whole in one part is what gives the base its full extent. The split reading needs the leftover distribution to know these are separate units, which is mechanisms.md#UNIT-PARTITION's problem. diff --git a/docs/design/rules.md b/docs/design/rules.md index ba7a1005..a0dbdeb4 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -1110,6 +1110,15 @@ R3. Rationale: initials abbreviate the person's name words; titles, rather than nothing: they are the base (R2), so they initial. "Juan van der" → initials="J. v. d." "Juan de y" → initials="J." + The conjunction half of that carries no example line, and the + reason is a limit of this document rather than a choice: every + line here names an input string parsed with the default + vocabulary, over which particles and conjunctions are disjoint — + in the defaults and in every locale pack — so no string can put a + conjunction inside an all-particle part. It takes a caller's own + vocabulary, where "Juan de y" is that part and initials "J. d." + while the "y" still contributes nothing, and it is pinned in + tests/v2/test_render.py. history: decisions.md#R2 · interacts: R2 · implemented: nameparser/_render.py, nameparser/_facade.py R4. Rationale: case repair is a display concern, applied only on diff --git a/docs/release_log.rst b/docs/release_log.rst index 1e9880cf..571a8873 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -69,6 +69,8 @@ Release Log - Change case repair to read the parser's own ``conjunction`` tag instead of re-deciding, from the word's spelling, whether a word is a conjunction or an initial. The parse answers that question already -- ``"Scott E. Werner"`` reads ``E.`` as an initial rather than the Italian conjunction -- and the other views honor the answer; case repair asked again, with a shape test applied to each word of a token's text rather than to the token. Two spellings of one name disagreed because of it: ``"juan e-f smith"`` capitalized to ``Juan e-F Smith`` while ``"JUAN E-F SMITH"`` gave ``Juan E-F Smith``; both give ``Juan E-F Smith`` now, ``e-f`` being a middle name and no conjunction of the parse's reading. A conjunction written as a word of its own is untouched, and so is the one-letter carve-out where it applies -- ``"juan y garcia"`` still repairs to ``Juan y Garcia``, ``"JUAN Y GARCIA"`` still to ``Juan Y Garcia``. One consequence for callers who edit fields: a value ``replace()`` splices in carries no reading for repair to honor, so a family set to ``de y`` now repairs to ``de Y``, where ``Parser.revise()`` classifies the value and keeps ``de y`` as the parsed name does. Case repair is not one of the seven role fields the differential harness compares, so no gate run can see this change either way and none of its counts move; measured directly instead, no name of the 1094-name differential corpus moves under ``capitalized()`` or ``capitalized(force=True)``, its uppercased and lowercased spellings included (closes #458) + - Fix ``initials()`` initialing a conjunction that stands inside a family name made only of particle words, where every other reading of that name treats it as the conjunction it is. A part whose every word is particle vocabulary has none of them acting as a particle, so those words initial like ordinary name words (#402 above) -- but a conjunction is not one of them in any part, and the readmission covered it too. One token was then read two ways: with ``y`` configured as a particle as well as the conjunction it already is, ``"Anh y Van"`` capitalized to ``Anh y Van`` and initialed to ``A. y. V.``; it initials to ``A. V.`` now, and ``"Juan de y"`` to ``J. d.`` -- the ``de`` an ordinary name word there, the ``y`` still contributing nothing. Reaching this needs a ``Lexicon`` a caller configured with a word in both ``particles`` and ``conjunctions``: the shipped sets are disjoint, in the default vocabulary and in every locale pack, so no name parsed with the shipped vocabulary changes at all -- none of the 1094 differential corpus names moves, and the gate reports the same 229 / 194 / 102 intentional diffs with nothing unexplained at the 1.4.0, 2.0.0 and 2.1.0 baselines. Filed so the rule and the implementation would not stay silently in disagreement, ``rules.md#R3`` having said this since it was written (closes #461) + - Fix a tussenvoegsel attached to the family name after a comma deciding a genuinely uncertain reading and reporting nothing. ``"Van Johnson"`` reports a ``PARTICLE_OR_GIVEN`` ambiguity -- ``Van`` is a Dutch particle and a Vietnamese given name, and the parser has to pick one -- while ``"Nguyen, Thi Van"`` picked the same word the same way, silently, and lost the given name doing it. The attachment now reports the fork it decides, in the kind that names the reading it declined. A particle that could be an ordinary name reports ``PARTICLE_OR_GIVEN``: ``"Nguyen, Thi Van"``, ``"Berg, Jan van der"`` and ``"Vega, Juan de la"`` each gain one, the ``detail`` naming the ambiguous word. A particle the parser had already read as a post-nominal reports ``SUFFIX_OR_NAME`` instead, because the credential reading is what the attachment overrode: ``"Berg, Jan vd"`` gains one, ``vd`` being read as *van der* rather than as the Volunteer Decoration. Which kind you get follows the reading that was overridden rather than the word's vocabulary, so ``"Berg, Jan do"`` reports ``PARTICLE_OR_GIVEN`` even though ``do`` is a postnominal too -- it was already being read as a name word, so no credential reading was overridden. A particle where nothing was overridden reports nothing at all: ``"Jong, Piet de"`` and ``"Jong, Anke de"`` are unchanged, ``de`` being no name in any reading and no postnominal either, and so is ``"Berg, Jan de vd"``, whose run was read as name words whole. Worth knowing before you filter on this: ``"Beethoven, Ludwig van"`` -- the textbook Dutch listing, read exactly right -- now carries a report too. It is the same string shape over the same vocabulary as ``"Nguyen, Thi Van"``, and nothing in the input separates them, so a report on one is a report on both. This adds the report and nothing else: every field these names parse to is exactly what the tussenvoegsel fix below already gave them, and ``ambiguities`` is the only value that grows. Seven differential corpus names gain a kind against the 2.0.0 and 2.1.0 baselines, seven of the nine the tussenvoegsel fix below already moved -- five of ``fix(#379)``'s seven and both of ``fix(#380)``'s two, now carrying ``_ambiguities`` in their diff as well; the two left out are the pair named unchanged above, ``Jong, Piet de`` and ``Jong, Anke de`` -- and none against 1.4.0, which had no ambiguity reporting at all (closes #405) - Fix a tussenvoegsel after a family comma being parsed as a middle name. Dutch and Belgian alphabetized listings move the particle behind the given name -- ``"Beethoven, Ludwig van"`` is how ``"Ludwig van Beethoven"`` is filed -- and the trailing particle run was read as a middle name rather than as part of the surname: ``"Beethoven, Ludwig van"`` gave middle ``van``, last ``Beethoven``, and ``"Berg, Jan van der"`` gave middle ``van der``. The run now attaches to the family the comma has already named and renders before it, so those read family ``van Beethoven`` and ``van der Berg`` with the given name unchanged. The derived views move with the parse, so ``family_particles`` is ``van`` and ``family_base`` is ``Beethoven`` where they were empty and ``Beethoven`` before. `#130 `_ asked for the split and got it in 1.3.0 as ``last_base``/``last_prefixes``; 2.0 renamed them ``family_base``/``family_particles``. What was wrong until now was the values they reported for this listing. Both halves of the particle vocabulary attach -- never-given ``de`` and may-be-given ``van`` alike -- because after a comma the family is already named and the particle has no other role to take. Two guards bound it. A name whose only given word is the particle keeps it, so ``"Nguyen, Van"`` still reads given ``Van``: the attachment needs a given word to spare. And where the word is BOTH particle and suffix vocabulary the attachment outranks the post-nominal reading, so ``"Berg, Jan vd"`` reads family ``vd Berg`` where 1.4.0 and 2.1 alike gave suffix ``vd`` -- a trailing abbreviation after a family comma is the tussenvoegsel far more often than the decoration it collides with, and the same shape sweeps in ``mc``, which 2.1 also read as a suffix. ``do`` is in ``SUFFIX_ACRONYMS_AMBIGUOUS`` and 2.1 already read a trailing one as a name word, so it attaches by the plain rule rather than by the override (closes #379, closes #380). Names without the comma are untouched: ``"Ludwig van Beethoven"`` already read family ``van Beethoven`` and is byte-identical. One of the 751 differential corpus names moves, ``"Vega, Juan de la"``, at the 1.4.0, 2.0.0 and 2.1.0 baselines alike diff --git a/nameparser/_render.py b/nameparser/_render.py index e6696f20..030d2f4e 100644 --- a/nameparser/_render.py +++ b/nameparser/_render.py @@ -35,10 +35,14 @@ #: Tags whose tokens contribute no initial outside the given group -- #: unless the token also carries UNJOINED_TAG, i.e. the whole part is #: particles, in which case they are the part's only words and do -#: contribute (rules.md#R3, #404). +#: contribute (rules.md#R3, #404). The mark readmits the PARTICLE tag +#: alone: it says those particles are not acting as particles here and +#: nothing about a conjunction, which rules.md#R3 excludes "even then" +#: (#461). #: Not STABLE_TAGS -- that also contains "initial", which must contribute. _SKIP_TAGS = frozenset({"particle", "conjunction"}) + def _collapse(rendered: str) -> str: """The #254 collapse: empty fields substitute '' and every artifact of that is removed -- dangling empty-nickname wrappers, space runs, @@ -85,9 +89,11 @@ def initials(name: ParsedName, spec: str, delimiter: str, separator: str) -> str """First letter of each contributing token per group, v1 semantics: delimiter follows each initial, separator sits between initials within a group. Tokens tagged particle/conjunction contribute no - initial in middle/family (given-name tokens always contribute); - tags come from the pipeline -- hand-built untagged tokens all - contribute. Valid spec keys: given, middle, family.""" + initial in middle/family (given-name tokens always contribute), + and the unjoined mark readmits the particles of an all-particle + part but never a conjunction standing in one; tags come from the + pipeline -- hand-built untagged tokens all contribute. Valid spec + keys: given, middle, family.""" if not isinstance(delimiter, str): raise TypeError(f"delimiter must be a str, got {delimiter!r}") if not isinstance(separator, str): @@ -97,9 +103,14 @@ def initials(name: ParsedName, spec: str, delimiter: str, separator: str) -> str role = Role(key) tokens = name.tokens_for(role) if role is not Role.GIVEN: + # rules.md#R3: "A CONJUNCTION never initials, so a base + # that is one contributes nothing even then" -- "even + # then" being the all-particle part the mark names, so the + # override is narrowed to the tag the mark is about (#461) tokens = tuple(t for t in tokens if not (_SKIP_TAGS & t.tags) - or UNJOINED_TAG in t.tags) + or (UNJOINED_TAG in t.tags + and "conjunction" not in t.tags)) values[key] = separator.join( t.text[0] + delimiter for t in tokens) return _format_spec(spec, values, "initials", _INITIALS_KEYS) @@ -129,7 +140,10 @@ def _cap_word(word: str, role: Role, tags: frozenset[str], # under `Lexicon.default().add(particles={'y'})`, `anh y van` has # an all-particle family whose `y` carries both tags and the mark, # and gives 'Anh y Van'; gating this conjunct too would give - # 'Anh Y Van'. + # 'Anh Y Van'. That is pinned, on the same parse as the initials() + # half it matches, by test_initials_and_repair_agree_on_a_ + # conjunction_in_a_particle_part (#461) -- until which gating it + # passed the whole suite. # That conjunct reads the TAG, not the word (#458). classify takes # the conjunction-versus-initial decision once, over the whole # token -- v1's is_conjunction excludes initials, so 'E.' in diff --git a/tests/v2/test_render.py b/tests/v2/test_render.py index 922b33e0..b2010ba6 100644 --- a/tests/v2/test_render.py +++ b/tests/v2/test_render.py @@ -1,8 +1,10 @@ import pytest +from nameparser import Parser from nameparser._lexicon import Lexicon from nameparser._render import _collapse, render -from nameparser._types import Ambiguity, AmbiguityKind, ParsedName, Role, Span, Token +from nameparser._types import (UNJOINED_TAG, Ambiguity, AmbiguityKind, + ParsedName, Role, Span, Token) def test_collapse_is_the_254_algorithm() -> None: @@ -124,6 +126,51 @@ def test_initials_skips_tagged_particles_outside_given() -> None: assert pn.initials("{family}") == "S." +def test_initials_and_repair_agree_on_a_conjunction_in_a_particle_part() -> None: + """#461, and the one shape that needs a caller's own Lexicon. + + The unjoined mark readmits the words of an all-particle part + because none of them is acting as a particle there -- but it says + nothing about a conjunction, which rules.md#R3 excludes "even + then". Reaching that needs a word in `particles` AND + `conjunctions`; the shipped sets are disjoint, in the default + vocabulary and in every locale pack, so no input string can + witness it and rules.md can carry no example line for it. + + This is also where the two views can be shown to agree, which is + the whole point of the carve-out: before #461, `Anh y Van` + capitalized as `Anh y Van` and initialed as `A. y. V.` -- one + token, two views, opposite readings. The `capitalized` assertions + here are what pins case repair's ungated conjunction conjunct + (rules.md#R4); gating it on the mark passed the entire suite until + this test existed, since no shipped name reaches it either. + """ + # mechanisms.md#VOCABULARY-OVERLAP-AS-PRECONDITION: "assert the + # intersection as a precondition" -- built here rather than found, + # so what has to hold is the half not being added. + assert "y" in Lexicon.default().conjunctions, ( + "'y' left the default conjunctions; this test builds the " + "particle/conjunction overlap it needs from the other side and " + "no longer has one. Pick another shipped conjunction.") + lex = Lexicon.default().add(particles={"y"}) + p = Parser(lexicon=lex) + + van = p.parse("Anh y Van") + tags = [t.tags for t in van.tokens if t.text == "y"][0] + assert {"conjunction", "particle", UNJOINED_TAG} <= tags, sorted(tags) + assert van.initials() == "A. V." + assert van.capitalized(lex, force=True).family == "y Van" + + # R3's own example line, under a lexicon that makes `de y` the + # all-particle part the default vocabulary cannot: `de` initials + # as the ordinary name word R2 makes it, `y` still does not. + assert p.parse("Juan de y").initials() == "J. d." + # and a base that IS the conjunction contributes nothing at all + assert p.parse("johnny y").initials() == "j." + # unchanged where the part is particles alone + assert p.parse("Juan van der").initials() == "J. v. d." + + def test_initials_custom_delimiter_and_separator() -> None: assert _bobdole().initials(delimiter="", separator="") == "B A D" From 11b6cea7c1122451e0114ca4acc5900b87a09f24 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sat, 29 Aug 2026 19:24:30 -0700 Subject: [PATCH 03/14] fix(render,docs): a view falls back to the vocabulary for text the parse never saw Reading the tag and nothing else regressed every ASSIGNED field holding a conjunction, because a value `replace()` splices into a field is never classified and so carries no tag to read. Measured on the released 1.4.0 and 2.1.0 wheels, both agreeing and both broken by the tag-only reading: h.last = 'velasquez y garcia' -> John Velasquez y Garcia h.middle = 'e.' -> John E. Smith h.last = 'smith-y' -> John Smith-y h.first = 'y' -> y Smith That is the canonical Spanish surname and the shape `y` is in the conjunction vocabulary for, so it is a regression to fix rather than a boundary to accept. A synthetic token -- `span is None`, which `ParsedName.replace()` builds and every parsed token has -- was never read, so there is no decision to honor and the view asks the vocabulary, which gives the answer the parser would have given. "Untagged" cannot be the test: an ordinary parsed name word carries no tags either (`velasquez` parses to `tags=[]`). The span is the tell. The same fallback in `initials()` fixes a defect older than #458 and found only beside it: `replace(family='velasquez y garcia').initials()` gave `v. y. g.` and `replace(family='de la vega').initials()` gave `d. l. v.`, where parsing those names gives `v. g.` and `v.`. The v2 core is the whole reach -- `HumanName.initials_list()` computes from the field strings with its own lexicon and already gave both answers at 1.4.0. `initials()` takes no lexicon, so the fallback reads the cached `Lexicon.default()`, and only where a spanless token is present. HOW MUCH each view falls back is not the same, and the difference is the view rather than the question. Whether a word is the conjunction or an initial is a property of the word, so both views ask it -- v1's initial carve-out included, which is why `_render._INITIAL` is back, now with one reader (`_reads_as_conjunction`) whose input is precisely the text `_vocab` never saw, and why test_regex_sync's two-copy assertion matters more rather than less: the fallback is right only while it answers as the pipeline would. Whether a particle is ACTING as a particle is a property of the whole PART. Case repair walks one word at a time and never sees the part, so it does not ask -- widening it would make `replace(family='de la')` give `De La` and thereby reverse the replace/revise boundary rules.md#R4 records, with `revise()` as its documented crossing. `initials()` holds every token of the role at once, so it does ask, by `_types._remarked`'s own test -- every word of the part carries `particle` -- with the vocabulary standing in for the tags a spliced token never got. Scoped per ROLE, not per name: a role the parse classified whole is decided by its tags however the other roles were built. On a fully parsed role the recomputation and the recorded mark agree by construction, which is what makes per-name scoping pass every other test in the suite; a hand-built name pins the difference. Verified, all three agreeing where before the core disagreed on two: assigned family core before core now facade parse de la vega j. d. l. v. j. v. j. v. J. v. de la j. d. l. j. d. l. j. d. l. J. d. l. velasquez y garcia j. v. g. j. v. g. j. v. g. J. v. g. van der berg j. v. d. b. j. b. j. b. J. b. van der j. v. d. j. v. d. j. v. d. J. v. d. smith j. s. j. s. j. s. J. s. `replace(family='velasquez y garcia')` capitalizes to `Velasquez y Garcia`; `replace(family='de la')` still capitalizes to `de la` and `revise(family='de la')` still to `De La`; the parsed half of #458 stands, `juan e-f smith` still capitalizing to `Juan E-F Smith`. Over the whole branch, 0 of the 1094 corpus names move under the shipped vocabulary in any of the four views, and the gate is 229 / 194 / 102 with 0 unexplained at all three baselines. Co-Authored-By: Claude Opus 5 --- docs/design/decisions.md | 6 +- docs/design/rules.md | 36 ++++++-- docs/release_log.rst | 4 +- nameparser/_render.py | 156 ++++++++++++++++++++++++++++------- tests/test_capitalization.py | 25 ++++++ tests/v2/test_regex_sync.py | 32 ++++--- tests/v2/test_render.py | 93 +++++++++++++++++++++ 7 files changed, 300 insertions(+), 52 deletions(-) diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 044a6385..647e7b0b 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -735,7 +735,11 @@ Declined: - 2026-08-29 — the REPLACE/REVISE BOUNDARY, raised by the review of this branch as a gap needing a systematic fix and corrected by Derek to what it actually is: the documented boundary, with a supported path across it. `_cap_word` keys the particle TEST on lexicon membership but the REPAIR on the unjoined mark, and the mark is a parse product — so `parse('de la').capitalized(force=True)` is `'De La'` while `base.replace(family='de la').capitalized(force=True)` is `'de la'`, `replace()` splicing raw text into a field without classifying it. `Parser.revise()` is the method that classifies it, and its docstring already promised exactly this: each value gets a full sub-parse "so the stable tags survive and the tag-driven views ... behave as if the text had been parsed". Measured: `Parser().revise(base, family='de la').capitalized(force=True)` is `'De La'`. So there is nothing to fix — rules.md#R4 carries it as an Accepted boundary naming `revise()`, not as a deviation, and `revise()`'s enumeration of the tag-driven views gains `capitalized()`, which #407 made the fourth. (#458 would additionally make `replace()` agree, by keying the test on the `particle` tag rather than re-deriving membership from the word — the same shape it already names for `conjunction`/`initial`. That is a convenience, not the resolution.) It is a limit and not a regression either way: 2.1.0 gives `'de la'` on the spliced path too, so only the parsed path moved. -- 2026-08-29 #458 — DONE: case repair's conjunction test reads the `conjunction` TAG instead of re-deriving the word's class at render time, so a view honors the decision the parse recorded (mechanisms.md#VOCAB-TAGS) rather than taking it again. The two copies of the question were not the same question, which is the reason this was worth doing over a corpus that cannot see it: `_classify.py` asks `is_initial(token.text)` — the shape test ANDed with a repertoire test since #320 — while `_render.py` asked a bare pattern, shape only, over each WORD of a token's text rather than the token. It also retires the hand-sync obligation the module carried in a comment ("keep in sync with `nameparser/_pipeline/_vocab.py` by hand"): a tag read has nothing to keep in sync, and `_render._INITIAL` is deleted, its last reader gone. MEASURED, re-derived on this branch rather than taken from the issue: 0 of the 1094 corpus names move under `capitalized()` or `capitalized(force=True)`, and 0 move with each name also re-run uppercased and lowercased (6565 name/variant/lexicon rows, the second lexicon being `Lexicon.default().add(particles={'y'})`); the R5 force-versus-case counts recorded above hold at 62/16 through the facade and 63 through the core. Movement EXISTS outside that population, and the issue's "no output moves" is a claim about the corpus only: the old predicate re-decided per word, so `juan e-f smith` capitalized to `Juan e-F Smith` — the Italian conjunction inside a hyphenated middle name — and now gives `Juan E-F Smith`, agreeing with the uppercase spelling it always gave. Untagged tokens move the same way: a family `replace()` splices in as `de y` now repairs to `de Y`, the R4 boundary above read in the other direction (`revise()` classifies and keeps `de y`, agreeing with the parse). NOT DONE, and the parenthetical in the boundary bullet above should be read as still open: the PARTICLE conjunct still keys on lexicon membership, so `replace()` does not agree there. Making it read the `particle` tag is a second change with its own blast radius — it moves a boundary rules.md#R4 states in prose — and #458 as filed does not ask for it. +- 2026-08-29 #458 — DONE: case repair's conjunction test reads the `conjunction` TAG instead of re-deriving the word's class at render time, so a view honors the decision the parse recorded (mechanisms.md#VOCAB-TAGS) rather than taking it again. The two copies of the question were not the same question, which is the reason this was worth doing over a corpus that cannot see it: `_classify.py` asks `is_initial(token.text)` — the shape test ANDed with a repertoire test since #320 — while `_render.py` asked a bare pattern, shape only, over each WORD of a token's text rather than the token. It also retires the hand-sync obligation the module carried in a comment ("keep in sync with `nameparser/_pipeline/_vocab.py` by hand"): a tag read has nothing to keep in sync, and `_render._INITIAL` is deleted, its last reader gone. MEASURED, re-derived on this branch rather than taken from the issue: 0 of the 1094 corpus names move under `capitalized()` or `capitalized(force=True)`, and 0 move with each name also re-run uppercased and lowercased (6565 name/variant/lexicon rows, the second lexicon being `Lexicon.default().add(particles={'y'})`); the R5 force-versus-case counts recorded above hold at 62/16 through the facade and 63 through the core. Movement EXISTS outside that population, and the issue's "no output moves" is a claim about the corpus only: the old predicate re-decided per word, so `juan e-f smith` capitalized to `Juan e-F Smith` — the Italian conjunction inside a hyphenated middle name — and now gives `Juan E-F Smith`, agreeing with the uppercase spelling it always gave. An earlier draft of this change moved untagged tokens the same way — a family `replace()` splices in as `de y` repairing to `de Y` — and that was a REGRESSION, caught in review before it shipped and fixed in the bullet below; read this bullet with that one. NOT DONE, and the parenthetical in the boundary bullet above should be read as still open: the PARTICLE conjunct still keys on lexicon membership, so `replace()` does not agree there. Making it read the `particle` tag is a second change with its own blast radius — it moves a boundary rules.md#R4 states in prose — and #458 as filed does not ask for it. + +- 2026-08-29 (#458 review) — THE FALLBACK FOR TEXT THE PARSE NEVER SAW, and why it is scoped to one of the two questions `_cap_word` asks. Reading the tag and nothing else regressed every ASSIGNED field holding a conjunction, because a value `replace()` splices in is never classified and so carries no tag to read. Measured on the released 1.4.0 and 2.1.0 wheels, all agreeing and all broken by the tag-only reading: `h.last = 'velasquez y garcia'` then `capitalize(force=True)` gives `John Velasquez y Garcia`, `h.middle = 'e.'` gives `John E. Smith`, `h.last = 'smith-y'` gives `John Smith-y`, `h.first = 'y'` gives `y Smith`. That is the canonical Spanish surname and the shape `y` is in the vocabulary FOR, so an Accepted clause was not available: it had to be fixed. RULE: a synthetic token — `span is None`, which `ParsedName.replace()` builds and every parsed token has — was never read, so there is no decision to honor and the view falls back to the vocabulary. Note what the discriminator is NOT: an ordinary parsed name word carries no tags either (`velasquez` parses to `tags=[]`), so "untagged" would have swept in the whole parsed corpus. The span is the tell. SCOPED TO THE PER-WORD QUESTION, and this is the durable half. *Is this word a conjunction rather than an initial* is a property of the word and its spelling: an unread token has no ambiguity the parse would have resolved differently, so the vocabulary gives the answer the parser would have given, v1's initial carve-out included — which is why `_render._INITIAL` came back and why test_regex_sync's two-copy assertion matters MORE now, the fallback being right only while it answers as `_vocab` would. *Is this particle acting as a particle here* is a property of the whole PART, and the two views hold different amounts of it — which is why the scope differs per VIEW and not per question, a distinction the first draft of this bullet got wrong. Case repair walks one word of one token at a time and never sees the part, so it cannot answer it and does not ask: the fallback there is the word question alone. `initials()` holds every token of the role at once, so it can and does (next bullet). Measured cost of the wide reading IN CASE REPAIR, so nobody re-derives it: `replace(family='de la')` gives `de la` today and a parse gives `De La`; widening case repair's fallback would make them agree and would thereby REVERSE the replace/revise boundary recorded two bullets above, with `revise()` as its documented crossing. Do not widen THAT one without reopening the decision; `initials()` is a different view with a different reach and reverses nothing. Verified after the fix: `replace(family='velasquez y garcia').capitalized(force=True)` is `Velasquez y Garcia`, `.initials()` is `v. g.`, `replace(family='de la').capitalized(force=True)` is `de la`, and the parsed and `revise()` paths are unchanged. + +- 2026-08-29 (#458 review) — the SAME FALLBACK in `initials()`, fixing a defect older than #458 and only found beside it. On master `replace(family='velasquez y garcia').initials()` gave `v. y. g.` where the parse gives `v. g.`, so the two views were inconsistently wrong before the #458 commit and consistently wrong after it; both now ask the same question of the same token. The v2 core is the whole reach: `HumanName.initials_list()` computes from the field STRINGS with its own lexicon (`_facade._initials_lists`, v1 parity) and already gave `['j', 'v', 'g']` at 1.4.0, 2.1.0 and on this branch. `initials()` takes no lexicon, so the fallback reads `Lexicon.default()` — the same default-config fallback `ParsedName.capitalized()` documents for an omitted argument — and only where a spanless token is present, so a parsed name pays a scan and no lookup. BOTH of R3's questions are answered there, not just the conjunction one — reviewed and widened on Derek's ruling, the narrow scope having been argued from case repair's limits and wrongly carried across. `replace(family='de la vega').initials()` gave `j. d. l. v.` where 1.4.0, 2.1.0, the facade and the parse of the same name all give `j. v.`, and `van der berg` gave `j. v. d. b.` for `j. b.`. The all-particle test is `_types._remarked`'s own — every word of the part carries `particle` — with the vocabulary standing in for the tags a spliced token never got, so the fallback keeps answering as the pipeline would; that also makes it a provable no-op on a role the parse classified whole, since there the two agree by construction. Note `_remarked` is not the bug and needs no change: it recomputes the mark from TAGS after every edit, which is right for a record of what the parse decided and simply silent about text nobody classified. SCOPED PER ROLE, not per name: a role the parse classified whole is decided by its tags however the other roles were built. Per-NAME scoping passes every other test in the suite — the recomputation agreeing with the mark wherever the parse set one — so it is pinned by a hand-built name in tests/v2/test_render.py; a role holding parsed and spliced tokens at once is not reachable through `replace()`, which replaces a role whole (measured), and each token there answers with its best evidence. Verified against the facade and the parse on six spliced families, all three agreeing after the change and the core disagreeing on two of them before it. - 2026-08-29 — WHY THE BOUNDARY WENT UNNOTICED UNTIL #407, which is where a future reader should look for it. For an ALL-PARTICLE part the other three tag-driven views give the same answer through `replace()` and `revise()` alike: measured over `de la`, `van der`, `do`, `de` and `van de la`, all five agree on `family_particles=''`, on a `family_base` holding the whole part, and on initials from every word. They converge because an UNTAGGED part and a MARKED all-particle part reach the same place by different routes — untagged, no word is recognized as a particle; marked, none is ACTING as one — and all three views only ask which words are particles. Case repair is the one view that asks a second question, since it must also decide whether to lowercase, so it is where the two routes first come apart. The mirror case confirms the reading: on a MIXED part the convergence is the other way round — `de la vega` and `van der berg` diverge in all three views between `replace()` and `revise()` (`replace()` reports particles `''` and base `'de la vega'` where `revise()` reports `'de la'` and `'vega'`) and AGREE on case repair, R4's all-particle clause not reaching them. So before #407 the distinction was invisible on exactly the shape the clause is about, and visible only on shapes the clause does not govern. diff --git a/docs/design/rules.md b/docs/design/rules.md index a0dbdeb4..253d17a5 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -1119,6 +1119,16 @@ R3. Rationale: initials abbreviate the person's name words; titles, vocabulary, where "Juan de y" is that part and initials "J. d." while the "y" still contributes nothing, and it is pinned in tests/v2/test_render.py. + Accepted: this rule reads a part the parser read, and where a + field was set as raw text after the parse it reads that text as + the parser would have — the words of the part are all in hand + here, so the every-word-is-particle-vocabulary test is answerable + without them. A family set to "de la vega" therefore initials just + its "V.", and one set to "de la" initials both its words, matching + the same names parsed. Stated without an example line because + every line here names an input string, and this shape needs a + field edited after the parse. Case repair is the view that cannot + do this, and R4 says why. history: decisions.md#R2 · interacts: R2 · implemented: nameparser/_render.py, nameparser/_facade.py R4. Rationale: case repair is a display concern, applied only on @@ -1154,15 +1164,23 @@ R4. Rationale: case repair is a display concern, applied only on repaired exactly this way before the clause existed. Stated without an example line because every line here names an input string, and this shape needs a field edited after the parse. - Accepted, the same boundary read the other way: the conjunction - carve-out reaches a word the parse read as a conjunction, and - nothing else. A word of that vocabulary standing inside a longer - written word is not one, and neither is one spliced into a field - after the parse — a family set that way to "de y" repairs to "de - Y" where the parsed name keeps the "y" as written. The two - directions are one fact: repair honors the reading the parse - recorded and takes none of its own. revise() classifies the value - here too, so it agrees with the parse in both directions. + Accepted, and the reason the boundary is drawn per question + rather than per field: the conjunction carve-out reaches a word + the parse read as a conjunction, and where the parse read nothing + at all it reaches what the vocabulary says. A word of that + vocabulary standing inside a longer written word is not a + conjunction, because the parse read that word as one ordinary + name word — but a field spliced in as raw text was read by + nobody, so repair asks the vocabulary and a family set to "de y" + keeps its "y" lowercase. Whether a word is the conjunction or an + initial is a property of the word, which a vocabulary can answer; + whether a particle is acting as a particle is a property of the + whole part, and repair reads one word at a time, so that half + falls through to particle treatment and the "de la" boundary above + stands. Initials are the contrast worth knowing, and R3 states it: + that view holds every word of the part at once, so it answers the + part question for a spliced field too. revise() classifies the + value and crosses both. history: decisions.md#R4 · interacts: R2, R3, R5 · implemented: nameparser/_render.py R5. Rationale: mixed case is evidence that the writer cased the name diff --git a/docs/release_log.rst b/docs/release_log.rst index 571a8873..3622af6e 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -67,7 +67,9 @@ Release Log - Fix case repair lowercasing the words of a family name made only of particle words, where every other view already reads them as ordinary name words: ``HumanName("ANH DO").capitalize()`` gives ``Anh Do`` where it gave ``Anh do``, and ``"anh van do"`` gives ``Anh Van Do`` where it gave ``Anh van do``. A particle earns its name by joining forward to the word it modifies, so a part whose every word is particle vocabulary leaves none of them anything to join; the fix above already made those words anchor ``family_base`` and contribute initials, and case repair now agrees with them rather than reading the same word two ways. The test is the whole part, not a particle standing alone, which is why the two-word family in ``"anh van do"`` moves along with the one-word family in ``"ANH DO"`` -- the same Vietnamese surname, and a standing-alone test would have read it one way behind a given name and another way alone. This DIFFERS FROM 1.4.0 deliberately and does not restore it: 1.4.0 returned ``Anh do``, lowercasing on vocabulary membership alone. The accepted cost is that a degenerate family which is nothing but particles capitalizes too, so ``"juan van der"`` gives ``Juan Van Der`` where 1.4.0 gave ``Juan van der``. A conjunction is untouched by any of this, so ``"der, y van"`` gives ``y Van Der`` -- the family capitalizing while the conjunction keeps the lowercase it always had; and where the particles DO join a name word nothing changes, ``"juan de la vega"`` still giving ``Juan de la Vega``. Case repair is not one of the seven role fields the differential harness compares, so no gate run can see this change either way and none of its counts move; the ``rules.md#R4`` examples and the v1 capitalization tests are what pin it (closes #407) - - Change case repair to read the parser's own ``conjunction`` tag instead of re-deciding, from the word's spelling, whether a word is a conjunction or an initial. The parse answers that question already -- ``"Scott E. Werner"`` reads ``E.`` as an initial rather than the Italian conjunction -- and the other views honor the answer; case repair asked again, with a shape test applied to each word of a token's text rather than to the token. Two spellings of one name disagreed because of it: ``"juan e-f smith"`` capitalized to ``Juan e-F Smith`` while ``"JUAN E-F SMITH"`` gave ``Juan E-F Smith``; both give ``Juan E-F Smith`` now, ``e-f`` being a middle name and no conjunction of the parse's reading. A conjunction written as a word of its own is untouched, and so is the one-letter carve-out where it applies -- ``"juan y garcia"`` still repairs to ``Juan y Garcia``, ``"JUAN Y GARCIA"`` still to ``Juan Y Garcia``. One consequence for callers who edit fields: a value ``replace()`` splices in carries no reading for repair to honor, so a family set to ``de y`` now repairs to ``de Y``, where ``Parser.revise()`` classifies the value and keeps ``de y`` as the parsed name does. Case repair is not one of the seven role fields the differential harness compares, so no gate run can see this change either way and none of its counts move; measured directly instead, no name of the 1094-name differential corpus moves under ``capitalized()`` or ``capitalized(force=True)``, its uppercased and lowercased spellings included (closes #458) + - Change case repair to read the parser's own ``conjunction`` tag instead of re-deciding, from the word's spelling, whether a word is a conjunction or an initial. The parse answers that question already -- ``"Scott E. Werner"`` reads ``E.`` as an initial rather than the Italian conjunction -- and the other views honor the answer; case repair asked again, with a shape test applied to each word of a token's text rather than to the token. Two spellings of one name disagreed because of it: ``"juan e-f smith"`` capitalized to ``Juan e-F Smith`` while ``"JUAN E-F SMITH"`` gave ``Juan E-F Smith``; both give ``Juan E-F Smith`` now, ``e-f`` being a middle name and no conjunction of the parse's reading. A conjunction written as a word of its own is untouched, and so is the one-letter carve-out where it applies -- ``"juan y garcia"`` still repairs to ``Juan y Garcia``, ``"JUAN Y GARCIA"`` still to ``Juan Y Garcia``. A field assigned after the parse is unaffected: its text was never classified, so there is no reading to honor and repair asks the vocabulary, which is what every earlier version did everywhere -- ``h.last = "velasquez y garcia"`` still repairs to ``Velasquez y Garcia`` and ``h.middle = "e."`` to ``E.``. Case repair is not one of the seven role fields the differential harness compares, so no gate run can see this change either way and none of its counts move; measured directly instead, no name of the 1094-name differential corpus moves under ``capitalized()`` or ``capitalized(force=True)``, its uppercased and lowercased spellings included (closes #458) + + - Fix ``ParsedName.initials()`` reading a field assigned after the parse as though every word in it were a name word, so particles and conjunctions inside one contributed initials that the same words parsed do not. ``parse("john smith").replace(family="velasquez y garcia").initials()`` gave ``j. v. y. g.`` and ``replace(family="de la vega")`` gave ``j. d. l. v.``, where parsing those names gives ``j. v. g.`` and ``j. v.``. Text spliced into a field is never classified, so the view has no reading to honor; it now asks the vocabulary, which answers as the parser would -- an assigned middle initial stays an initial rather than becoming the Italian conjunction (``replace(middle="E.")`` still gives ``j. E. s.``), an uppercase ``Y`` in an assigned surname still contributes, and a field that is nothing but particle words still initials all of them, since none of them is doing a particle's work there (``replace(family="de la")`` gives ``j. d. l.``, as parsing it does). This is the 2.0 API only: ``HumanName.initials()`` and ``initials_list()`` compute from the field strings with their own vocabulary and already gave every one of these answers. Shipped since 2.0.0, and found while reviewing the case-repair change above (#458) - Fix ``initials()`` initialing a conjunction that stands inside a family name made only of particle words, where every other reading of that name treats it as the conjunction it is. A part whose every word is particle vocabulary has none of them acting as a particle, so those words initial like ordinary name words (#402 above) -- but a conjunction is not one of them in any part, and the readmission covered it too. One token was then read two ways: with ``y`` configured as a particle as well as the conjunction it already is, ``"Anh y Van"`` capitalized to ``Anh y Van`` and initialed to ``A. y. V.``; it initials to ``A. V.`` now, and ``"Juan de y"`` to ``J. d.`` -- the ``de`` an ordinary name word there, the ``y`` still contributing nothing. Reaching this needs a ``Lexicon`` a caller configured with a word in both ``particles`` and ``conjunctions``: the shipped sets are disjoint, in the default vocabulary and in every locale pack, so no name parsed with the shipped vocabulary changes at all -- none of the 1094 differential corpus names moves, and the gate reports the same 229 / 194 / 102 intentional diffs with nothing unexplained at the 1.4.0, 2.0.0 and 2.1.0 baselines. Filed so the rule and the implementation would not stay silently in disagreement, ``rules.md#R3`` having said this since it was written (closes #461) diff --git a/nameparser/_render.py b/nameparser/_render.py index 030d2f4e..947f1f51 100644 --- a/nameparser/_render.py +++ b/nameparser/_render.py @@ -42,6 +42,39 @@ #: Not STABLE_TAGS -- that also contains "initial", which must contribute. _SKIP_TAGS = frozenset({"particle", "conjunction"}) +# Ported verbatim from v1 (nameparser/config/regexes.py "initial", minus +# the empty alternative) -- layering forbids importing the pipeline here; +# keep in sync with _pipeline/_vocab.py by hand. +# Its one reader is _reads_as_conjunction below, and that reader only +# ever sees text the parse never classified: for anything the +# parser DID see, the tag is the answer and this pattern is not asked. +# So the two copies no longer decide the same question about the same +# token -- _vocab's says what the parse decided, this one says what it +# WOULD have decided about text spliced in afterwards -- which is why +# they must keep answering alike, and why test_regex_sync pins the +# patterns against each other and against config. +# Deliberately NOT composed with _vocab's repertoire test (#320): +# layering forbids the import. The divergence is reachable only for a +# caller-added CJK conjunction spliced into a field, since no shipped +# vocabulary carries one; there it costs nothing in case repair (CJK is +# caseless, so lower() and capitalize() return the same string) and +# would let such a word initial where a parsed one would not. +_INITIAL = re.compile(r"^(\w\.|[A-Z])$") + + +def _reads_as_conjunction(word: str, lex: Lexicon) -> bool: + """v1's is_conjunction, asked only of text the parse never saw. + + A token with a span was classified, so its tags are the answer and + this is not consulted. A token without one was spliced into a field + by replace() and carries no reading, so the views fall back to the + vocabulary -- which gives the answer the parser would have given, + the initial carve-out included ('E.' assigned to middle is an + initial, not the Italian conjunction). + """ + return bool(_normalize(word) in lex.conjunctions + and not _INITIAL.fullmatch(word)) + def _collapse(rendered: str) -> str: """The #254 collapse: empty fields substitute '' and every artifact @@ -92,32 +125,91 @@ def initials(name: ParsedName, spec: str, delimiter: str, separator: str) -> str initial in middle/family (given-name tokens always contribute), and the unjoined mark readmits the particles of an all-particle part but never a conjunction standing in one; tags come from the - pipeline -- hand-built untagged tokens all contribute. Valid spec - keys: given, middle, family.""" + pipeline. A token with no span was never classified -- replace() + splices one in -- so its role's words are read from the default + vocabulary instead, all-particle test included. Valid spec keys: + given, middle, family.""" if not isinstance(delimiter, str): raise TypeError(f"delimiter must be a str, got {delimiter!r}") if not isinstance(separator, str): raise TypeError(f"separator must be a str, got {separator!r}") + # Only pay for the vocabulary where there is unclassified text to + # ask about; Lexicon.default() is cached, the scan is not. + unparsed_lex = (Lexicon.default() + if any(t.span is None for t in name.tokens) else None) values: dict[str, str] = {} for key in _INITIALS_KEYS: role = Role(key) tokens = name.tokens_for(role) if role is not Role.GIVEN: - # rules.md#R3: "A CONJUNCTION never initials, so a base - # that is one contributes nothing even then" -- "even - # then" being the all-particle part the mark names, so the - # override is narrowed to the tag the mark is about (#461) + # per ROLE, not per name: a role the parse classified whole + # is decided by its tags however the other roles were built + lex = (unparsed_lex if unparsed_lex is not None + and any(t.span is None for t in tokens) else None) + unjoined = lex is not None and _all_particles(tokens, lex) tokens = tuple(t for t in tokens - if not (_SKIP_TAGS & t.tags) - or (UNJOINED_TAG in t.tags - and "conjunction" not in t.tags)) + if _initials_from(t, lex, unjoined)) values[key] = separator.join( t.text[0] + delimiter for t in tokens) return _format_spec(spec, values, "initials", _INITIALS_KEYS) +def _is_particle_word(token: Token, lex: Lexicon) -> bool: + """Particle vocabulary, by the tag where the parse left one and by + the vocabulary where it did not.""" + return ("particle" in token.tags + or (token.span is None + and _normalize(token.text) in lex.particles)) + + +def _all_particles(tokens: tuple[Token, ...], lex: Lexicon) -> bool: + """_types._remarked's own test -- every word of the part carries + "particle" -- with the vocabulary standing in for the tags a + spliced token never got. + + _remarked recomputes the unjoined mark from TAGS after every edit, + so a spliced part is never marked however particle-shaped it is. + That is right for the mark (which says what the parse decided) and + wrong for the view, which then reads an all-particle part as if it + were something else. Asking the vocabulary here answers as the + parse would have. A role holding parsed and spliced tokens at once + -- not reachable through replace(), which replaces a role whole, + but constructible by hand -- gets each token's best evidence, which + is again what _remarked would have computed. + """ + return bool(tokens) and all(_is_particle_word(t, lex) for t in tokens) + + +def _initials_from(token: Token, lex: Lexicon | None, + unjoined: bool) -> bool: + """Whether a middle/family token contributes an initial (R3). + + `lex` is None for a role the parse classified whole, where the tags + ARE the answer; it is the fallback vocabulary for a role holding + text the parse never saw, where the same two questions are answered + from tags where there are any and from the vocabulary where there + are none. The two paths agree wherever both apply: `unjoined` is + _remarked's own test, so a fully parsed role recomputes the mark it + already carries. + """ + if lex is None: + # rules.md#R3: "A CONJUNCTION never initials, so a base that is + # one contributes nothing even then" -- "even then" being the + # all-particle part the mark names, so the mark readmits the + # tag it is about and not the other one (#461) + if _SKIP_TAGS & token.tags: + return UNJOINED_TAG in token.tags and "conjunction" not in token.tags + return True + if ("conjunction" in token.tags + or (token.span is None and _reads_as_conjunction(token.text, lex))): + return False + if _is_particle_word(token, lex): + return unjoined + return True + + def _cap_word(word: str, role: Role, tags: frozenset[str], - lex: Lexicon) -> str: + lex: Lexicon, *, parsed: bool) -> str: # v1 cap_word order: particle/conjunction rule first, then the # exceptions map, then Mac/Mc, then str.capitalize normalized = _normalize(word) @@ -153,15 +245,15 @@ def _cap_word(word: str, role: Role, tags: frozenset[str], # stages test tags"). Asking again was not even the same question: # the copy of the initial pattern that stood here was the SHAPE # half alone, and it re-decided per WORD of a token's text, so - # 'juan e-f smith' capitalized to 'Juan e-F Smith'. What moves the - # other way is a token the parse never saw: one hand-built, or - # spliced in by replace(), carries no decision to honor and is - # repaired as an ordinary name word -- rules.md#R4's Accepted - # boundary, which the particle conjunct above draws in the opposite - # direction because it still asks the vocabulary itself. + # 'juan e-f smith' capitalized to 'Juan e-F Smith'. A token the + # parse never saw has no decision to honor, so the vocabulary + # answers for it instead -- _reads_as_conjunction above, which is + # v1's predicate over v1's own input class, keeping every assigned + # field exactly as 1.4.0 repaired it. if ((normalized in lex.particles and role in (Role.MIDDLE, Role.FAMILY) and UNJOINED_TAG not in tags) - or "conjunction" in tags): + or "conjunction" in tags + or (not parsed and _reads_as_conjunction(word, lex))): return word.lower() # v1 cap_word tries the edge-stripped form, then the period-free # form ('Ph.D.' -> 'ph.d' -> 'phd' hits the exceptions map) @@ -177,10 +269,14 @@ def _cap_word(word: str, role: Role, tags: frozenset[str], def _cap_text(text: str, role: Role, tags: frozenset[str], - lex: Lexicon) -> str: + lex: Lexicon, *, parsed: bool) -> str: # word-by-word within the token text: hyphenated names capitalize - # both sides ("macdole-eisenhower" -> "MacDole-Eisenhower") - return _WORD.sub(lambda m: _cap_word(m.group(0), role, tags, lex), text) + # both sides ("macdole-eisenhower" -> "MacDole-Eisenhower"). The + # per-word walk is also why an UNPARSED token gets the vocabulary + # asked per word: the parse would have made one token per word of + # that text, so this is the granularity its answer would have had. + return _WORD.sub( + lambda m: _cap_word(m.group(0), role, tags, lex, parsed=parsed), text) # rules.md#R4: "case repair returns a repaired copy and never mutates @@ -196,13 +292,15 @@ def capitalized(name: ParsedName, lexicon: Lexicon | None, *, word is particle vocabulary is repaired as ordinary name words, and the mark saying so comes from the pipeline, as does the reading that a word is a conjunction rather than an initial. A - hand-built token, or one replace() splices in, carries no tags: - it is repaired as a plain particle, and as no conjunction at all. - A family set that way to 'de la' stays 'de la' where the same - words parsed give 'De La', while one set to 'de y' gives 'de Y' - where the parse would keep the conjunction lowercase. - Parser.revise() is the edit that classifies the value, and gives - 'De La' (rules.md#R4's Accepted boundary). + token replace() splices in has no span and no tags, so it was + never read: the vocabulary answers the per-word conjunction + question for it, and the per-part particle question -- which the + vocabulary cannot answer, the part being what a spliced field + lost -- falls through to plain particle treatment. A family set + that way to 'de la' stays 'de la' where the same words parsed give + 'De La'; one set to 'de y' keeps the 'y' lowercase, as the parse + does and as 1.4.0 did. Parser.revise() is the edit that classifies + the value, and gives 'De La' (rules.md#R4's Accepted boundary). Idempotent: without force, a capitalized result is mixed-case and the gate returns it unchanged; with force, every _cap_word rule is a fixpoint on its own output.""" @@ -217,7 +315,9 @@ def capitalized(name: ParsedName, lexicon: Lexicon | None, *, if not force and joined not in (joined.upper(), joined.lower()): return name new_tokens = tuple( - Token(_cap_text(t.text, t.role, t.tags, lex), t.span, t.role, t.tags) + Token(_cap_text(t.text, t.role, t.tags, lex, + parsed=t.span is not None), + t.span, t.role, t.tags) for t in name.tokens) # equal tokens (possible only for synthetic span=None duplicates) # collapse to one mapping entry -- benign: the rebuilt ambiguity diff --git a/tests/test_capitalization.py b/tests/test_capitalization.py index 3ddbff4b..1855c763 100644 --- a/tests/test_capitalization.py +++ b/tests/test_capitalization.py @@ -265,6 +265,31 @@ def test_a_one_letter_conjunction_is_case_sensitive_to_repair(self) -> None: # 'Y' is initial-shaped, so the conjunction rule declines it self.m(str(uppered), 'Juan Y Garcia', uppered) + # The v1 parity this rests on, at the surface v1 users have. An + # ASSIGNED field is spliced in as raw text and never classified, so + # there is no tag to read and repair asks the vocabulary -- which is + # what v1 always did, everywhere. Measured on the released 1.4.0 + # wheel and on 2.1.0, all four: 'John Velasquez y Garcia', + # 'John Smith y Jones', 'John E. Smith', 'John Smith-y'. Reading the + # tag alone (no fallback) capitalizes every one of these conjunctions + # -- the regression #458's review caught before it shipped. + def test_an_assigned_field_keeps_v1_conjunction_repair(self) -> None: + for field, value, want in ( + ('last', 'velasquez y garcia', 'John Velasquez y Garcia'), + ('last', 'smith y jones', 'John Smith y Jones'), + # the initial carve-out: an assigned middle initial is + # an initial, not the Italian conjunction + ('middle', 'e.', 'John E. Smith'), + # v1 asks per WORD of the assigned text, so the + # conjunction inside a hyphenated word IS lowered here + # -- the opposite of the parsed reading pinned below, + # and the difference is that one carries a reading + ('last', 'smith-y', 'John Smith-y')): + hn = HumanName('john smith') + setattr(hn, field, value) + hn.capitalize(force=True) + self.m(str(hn), want, hn) + # The other half of #458, and the half that MOVED: the decision is # the whole token's, so a conjunction word inside a longer token is # not one. `e` is the Italian conjunction and `e-f` is a middle diff --git a/tests/v2/test_regex_sync.py b/tests/v2/test_regex_sync.py index 038905e0..cc42bfa2 100644 --- a/tests/v2/test_regex_sync.py +++ b/tests/v2/test_regex_sync.py @@ -72,19 +72,24 @@ def test_patronymic_patterns_match_config() -> None: assert copy.flags == source.flags, key -def test_initial_copy_matches_config_minus_the_empty_alternative() -> None: - # _vocab._INITIAL is v1's "initial" pattern minus its trailing "?" - # (documented at the definition: its callers always fullmatch a +def test_initial_copies_agree_with_each_other_and_config() -> None: + # _vocab._INITIAL and _render._INITIAL are both v1's "initial" + # pattern minus its trailing "?" (documented at _render.py's + # _INITIAL definition: the two call sites always fullmatch a # non-empty token, so the empty-string alternative is dropped on - # purpose). Assert the exact, documented relationship to the config - # source, so a future edit to either side that breaks the - # relationship fails loudly here instead of silently drifting. - # There was a second copy, in _render, and this test also asserted - # the two agreed. It went with #458: case repair stopped asking - # whether a word is initial-shaped -- classify records that, and a - # view reads the tag -- leaving the constant with no reader. A copy - # kept alive for its own sync assertion pins nothing about - # behavior, so it was deleted rather than commented. + # purpose). Assert both the internal-copy agreement and the exact, + # documented relationship to the config source, so a future edit to + # either side that breaks the relationship fails loudly here + # instead of silently drifting. + # The two copies stopped answering the same question in #458 and + # the agreement matters MORE for it, not less: _vocab's asks what + # the parse decided about a token, _render's what it would have + # decided about text spliced into a field afterwards, and the + # second is only right while it answers as the first would. This + # test is what says so. (#458 briefly deleted _render's copy, when + # nothing read it; the unparsed-text fallback gave it a reader + # again -- see _render._reads_as_conjunction.) + assert _vocab._INITIAL.pattern == _render._INITIAL.pattern source = _config.REGEXES["initial"] # config's pattern is the pipeline copy with an extra "?" spliced in # just before the trailing "$", making the whole group optional. @@ -121,7 +126,8 @@ def test_initial_copy_matches_config_minus_the_empty_alternative() -> None: ("_vocab", "_PERIOD_NOT_AT_END"): "period_not_at_end", # Deliberately NOT a straight copy -- pinned by the dedicated tests # above, which assert the documented RELATIONSHIP instead: - ("_vocab", "_INITIAL"): None, # config's pattern minus one "?" + ("_render", "_INITIAL"): None, # config's pattern minus one "?" + ("_vocab", "_INITIAL"): None, # same ("_tokenize", "_BIDI"): None, # re_bidi, not a REGEXES key # Mirrors _pipeline._state.COMMA_CHARS, not nameparser.config ("_render", "_COMMA_CHAR"): None, diff --git a/tests/v2/test_render.py b/tests/v2/test_render.py index b2010ba6..160ea7cc 100644 --- a/tests/v2/test_render.py +++ b/tests/v2/test_render.py @@ -310,6 +310,99 @@ def test_capitalized_lowers_the_words_the_parse_tagged_conjunction() -> None: assert hyphenated.capitalized(force=True).middle == "E-F" +def test_views_fall_back_to_the_vocabulary_for_text_never_parsed() -> None: + """A token with no span was never classified, so there is no + decision to honor and the views ask the vocabulary instead -- + getting the answer the parser would have given, v1's initial + carve-out included. Both views, on one parse: `replace()` splices + raw text into a field, and reading it two ways is what #458's + review found. + + The two views ask it of different scopes, and that is the whole of + the asymmetry. Case repair walks one word at a time and never sees + the part, so it answers the per-WORD question (conjunction or + initial?) and leaves the per-PART one (is this particle acting as + a particle?) to fall through to particle treatment -- widening it + there would reverse rules.md#R4's Accepted boundary, whose crossing + is revise(), which the `de la` capitalization assertions pin. + initials() DOES hold the part -- every token of the role is in hand + -- so it answers both, by _types._remarked's own test with the + vocabulary standing in for the tags a spliced token never got. + """ + p = Parser() + base = p.parse("john smith") + + spliced = base.replace(family="velasquez y garcia") + assert [t.span for t in spliced.tokens[1:]] == [None, None, None] + assert spliced.capitalized(force=True).family == "Velasquez y Garcia" + assert spliced.initials() == "j. v. g." + # the carve-out rides along: an assigned middle initial is an + # initial, not the Italian conjunction, in either view + assert base.replace(middle="e.").capitalized(force=True).middle == "E." + assert base.replace(middle="E.").initials() == "j. E. s." + assert base.replace(family="velasquez Y garcia").initials() == "j. v. Y. g." + + # the per-part question, which initials() can answer: a spliced + # family with a name word in it has its particles skipped, and one + # that is nothing but particles has them all contribute (R2/R3). + # Each is what the facade and the parse both give. + assert base.replace(family="de la vega").initials() == "j. v." + assert base.replace(family="van der berg").initials() == "j. b." + assert base.replace(family="de la").initials() == "j. d. l." + assert p.parse("john de la").initials() == "j. d. l." + assert base.replace(family="smith").initials() == "j. s." + + # unchanged: the boundary a spliced particle keeps in CASE REPAIR, + # which cannot see the part + assert base.replace(family="de la").capitalized(force=True).family == "de la" + assert (base.replace(family="de la vega") + .capitalized(force=True).family == "de la Vega") + # ... and revise(), which classifies, still crosses it + assert p.revise(base, family="de la").capitalized(force=True).family == "De La" + assert (p.revise(base, family="velasquez y garcia") + .capitalized(force=True).family == "Velasquez y Garcia") + + # unchanged: a token the parse DID see is decided by its tags, so + # a hyphenated word it read as one ordinary name word stays one + assert p.parse("juan e-f smith").capitalized(force=True).middle == "E-F" + + # A role holding both is not reachable through replace(), which + # replaces a role whole (measured), but is constructible: each + # token answers with its best evidence, which is what _remarked + # would have computed had the spliced half been classified. Here + # the parsed 'de' is particle-TAGGED and the spliced 'la' is only + # particle vocabulary, so the part is all particles and both + # contribute; swap in a name word and neither does. + mixed = _pn("john de la", [ + Token("john", Span(0, 4), Role.GIVEN), + Token("de", Span(5, 7), Role.FAMILY, frozenset({"particle"})), + Token("la", None, Role.FAMILY), + ]) + assert mixed.initials() == "j. d. l." + with_name_word = _pn("john de la", [ + Token("john", Span(0, 4), Role.GIVEN), + Token("de", Span(5, 7), Role.FAMILY, frozenset({"particle"})), + Token("vega", None, Role.FAMILY), + ]) + assert with_name_word.initials() == "j. v." + + # Scoped per ROLE, not per name: a role the parse classified whole + # keeps its tags as the answer however the other roles were built. + # The family here is all particle vocabulary and carries no mark, + # which is the parse saying those words are doing a particle's work + # -- a spliced middle beside it does not reopen that. Hand-built + # because every edit path recomputes the mark (_types._remarked) + # and would agree with the vocabulary by construction; per-NAME + # scoping passes every other test in the suite and fails here. + other_role_spliced = _pn("john van der", [ + Token("john", Span(0, 4), Role.GIVEN), + Token("q", None, Role.MIDDLE), + Token("van", Span(5, 8), Role.FAMILY, frozenset({"particle"})), + Token("der", Span(9, 12), Role.FAMILY, frozenset({"particle"})), + ]) + assert other_role_spliced.initials() == "j. q." + + def test_capitalized_rebuilds_ambiguity_tokens() -> None: tok = Token("van", Span(0, 3), Role.GIVEN, frozenset({"particle"})) pn = ParsedName( From 62c5c592fb392619dc08d9b8be2f5bc734346213 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sat, 29 Aug 2026 21:15:28 -0700 Subject: [PATCH 04/14] docs(design): RENDER-HONORS-THE-PARSE, the contract these three fixes share The parse decides it; the render views honor those decisions and never re-evaluate them. #458 and #461 are the two directions that breaks in -- re-deriving the answer from the text against a view's own copy of a pipeline predicate, and honoring the record then overriding it -- and #408, filed and open, is a third instance that should cite the entry rather than argue it again from scratch. The Known-limit clause carries the boundary the third commit established: a token the parse never saw carries no decision to honor, so a view falls back to the vocabulary, with how far a view may fall back differing per VIEW rather than per question. Within a view the line is drawn per question, as rules.md#R4's Accepted clause says; what differs between views is which questions they can answer at all, since `initials()` holds every token of the role and `_cap_word` walks one word of one token. Two limits are recorded rather than papered over, both measured while writing this and both falsifying a flatter first draft. The fallback's agreement with the pipeline is held by HAND: test_regex_sync pins the two `_INITIAL` copies to each other and to config, while the #320 repertoire half is deliberately not carried across, so a caller-added conjunction in an initialless script initials from a spliced field where the same word parsed does not. And `_cap_word`'s PARTICLE conjunct still keys on the lexicon handed to the view rather than on the tag -- `parse('juan smith vega')` repairs to `Juan Smith vega` under `Lexicon.default().add(particles={'vega'})` -- which decisions.md#R4 records as NOT DONE because reading the tag there moves a boundary rules.md#R4 states in prose. `_cap_word`'s comment cites the new entry twice, verbatim, for the two claims it was making in free prose: the honoring claim (which had been citing VOCAB-TAGS, whose quoted excerpt is about STAGES, and a view is not a stage) and the unparsed-token fallback. VOCAB-TAGS keeps its "a view is" trigger and now points across, so the two entries do not answer the same question without precedence. Co-Authored-By: Claude Opus 5 --- docs/design/mechanisms.md | 6 +++++- nameparser/_render.py | 18 +++++++++++------- 2 files changed, 16 insertions(+), 8 deletions(-) diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 4e333d9c..1ca2e35d 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -21,7 +21,7 @@ Problem shape. A rule wants words to RENDER in a different order than they sit i Problem shape. A later stage needs to know what the vocabulary knew about a word. Contract statement. classify tags every token with what the vocabulary knows about it, and later stages test tags — they never re-look a word up. How it works. One lookup site means one answer: a stage that re-derived vocabulary facts could disagree with the stage before it. Stable tags ("particle", "conjunction", "initial") are API; "vocab:"-namespaced ones are not. -Lives in. nameparser/_pipeline/_classify.py (producer); consumers throughout _group/_assign/_post_rules, and nameparser/_render.py, which is not a stage but reads the same tags to decide what a view shows (#458 moved case repair's conjunction test onto the tag; initials() had read them since 2.0). Reach for it when. A stage is about to import Lexicon to ask about a word classify already saw — or a view is, which is the harder one to notice, since a view legitimately holds a Lexicon for the questions classify never answered. +Lives in. nameparser/_pipeline/_classify.py (producer); consumers throughout _group/_assign/_post_rules, and nameparser/_render.py, which is not a stage but reads the same tags to decide what a view shows (#458 moved case repair's conjunction test onto the tag; initials() had read them since 2.0). Reach for it when. A stage is about to import Lexicon to ask about a word classify already saw — or a view is, which is the harder one to notice, since a view legitimately holds a Lexicon for the questions classify never answered; the view side is RENDER-HONORS-THE-PARSE, which owns it and covers the decisions this entry does not produce. ## PIECES — joining structure survives assignment @@ -57,6 +57,10 @@ Problem shape. "Which stage does X?" — asked before attributing behavior in pr Problem shape. Two stages need the same answer about the same input, and the one that does not own the decision is about to test for it. Contract statement. Where two sites ask the same question, exactly one predicate answers it and every other site calls that one — never a condition written to match it. The predicate belongs to the QUESTION, not to whichever stage decides: it may sit in a leaf both stages import, and for the leading-title test it must, since the deciding stage is assign and group cannot import assign. How it works. A hand-written mirror agrees with its original only until one of them moves, and the drift is invisible in both directions: each site keeps passing its own tests while they disagree about an input neither covers. Five instances, every one found as a defect before it was found as a pattern — #319 lifted the wholly-suffix predicate into the vocabulary layer "so the comma decision and the honorific peel's segment test cannot drift apart"; #401/#421 lifted the trailing-numeral fork out of assign so the bound-given reserve stopped carrying a copy, its hand-written mirror having been falsified in review more than once — the lesson recorded there being that what must be mirrored is assign's WALK, not merely its condition; #425 replaced that reserve's hand re-derivation of the trailing peel with one function over the view the join would leave; #424 moved assign's leading-title test down because group's own `title()` does not see H2's unlisted abbreviations, so `Xyz. van Johnson` chained where `Dr. van Johnson` did not; #429 moved the no-name-segment test down because group asked by segment INDEX where assign asks by CONTENT. The destination follows the LAYER, not the topic: a predicate over token text goes to `_vocab`, one over pieces and tags to `_pieces`. Both are leaves the stages sit on. The piece layer got its own module only in #439 — until then those predicates collected in `_group`, not because grouping owned them but because `_assign` imports `_group` and cannot be imported back, so group was the one place both stages could reach; five had accumulated across four PRs before the module existed. Stage order is this mechanism's limit, and it forecloses the alternative: where the reader comes AFTER the decider, record the answer on the state instead — `ParseState.order` is that shape, "Recorded rather than recomputed downstream, because the two can differ" — which is unavailable whenever the EARLIER stage is the one asking. (The concrete assign→group import that forced the `_group` collection is gone since #439; what remains is the ordering it was a symptom of, and tests/v2/test_layering.py is where the leaf's contract is now written down.) The cost is a second evaluation of the same predicate, measured for #429 at 1.2–2.2% of a family-comma parse and 0% of every other; recording that number was the right answer there over plumbing a state field the two sites would not otherwise share. Lives in. nameparser/_pipeline/_vocab.py over text (is_wholly_suffix; is_trailing_numeral_suffix — the #401/#421 instance, whose only caller since #439 is the shared peel rather than a stage; and maiden_marker_run, the #434 instance and the clearest two-stage case, called by classify over token texts and by extract over a clause's whitespace words, with group reading the tags classify recorded because it runs later) and nameparser/_pipeline/_pieces.py over pieces: is_suffix_piece, is_leading_title, leading_titles, peel_walk, peel_trailing and segment_suffix_reading are called by both stages — the last of those is #430's instance, where THREE readers share one answer, the render join being the third — while is_title_piece and trailing_start are called by group alone — `trailing_start` being the one to know, since it answers where the trailing run begins and is what P2's chain and M2's walk stop at. tests/v2/test_layering.py holds each module's contract, and a piece predicate growing a dependency on a STAGE shows up there as a widened entry. Reach for it when. You are about to write a condition that mirrors, matches or "does what X does" — or you find a comment saying one does. Grep for the other site's predicate and call it instead. +## RENDER-HONORS-THE-PARSE — the parse decides it, the views honor it + +Problem shape. A render view needs a fact the parse already settled — whether a word is the conjunction or an initial, whether a particle is acting as one, which word renders first. Contract statement. The parse decides it; the render views honor those decisions and never re-evaluate them. Two directions break that, and each has been found here as a defect: a view RE-DERIVES the answer from the text, keeping its own copy of a pipeline predicate — that one shipped through 2.0 and 2.1 — or a view honors the record and then OVERRIDES it, readmitting what the decision excluded, which was caught before release. How it works. Re-deriving fails because the two copies stop being the same question long before anyone notices they are two: `_cap_word` re-ran the conjunction-versus-initial decision from the word's spelling against a hand-maintained copy of the pipeline's `_INITIAL` pattern while classify had already answered it and recorded it on the token, and `_classify.py` asks `is_initial()`, the shape test ANDed with a script-repertoire test since #320, where `_render.py` asked the bare pattern, and asked it per WORD of a token's text rather than per token, so `juan e-f smith` repaired to `Juan e-F Smith` (#458). Overriding fails more directly: `initials()` honored the `conjunction` tag through `_SKIP_TAGS` and then readmitted it with an `or UNJOINED_TAG in t.tags`, overriding a decision it had not taken, where rules.md#R3 excludes a conjunction "even then" — which left the two views disagreeing about one token under a caller's vocabulary that puts a word in both sets: `Anh y Van` repaired to `Anh y Van` and honored the carve-out, while the same parse initialed `A. y. V.` and ignored it (#461). Only the FORCED call witnesses the repair half, R5's gate refusing a mixed-case name before any of this is consulted. This is the CONSUMER-side rule over the producer-side entries — VOCAB-TAGS records what the vocabulary knew, MARK-DONT-STRIP what a stage decided about it, FOLDED_TAG what order to render in — and a view reads what they recorded, whichever kind it is. It sits where ONE-PREDICATE-PER-QUESTION's stated limit leaves off: where two live sites need one answer they share a predicate, but a render view always comes AFTER the decider, so the answer is recorded rather than shared — on the TOKEN, which is the views' equivalent of that entry's `ParseState.order`, no view being able to see a ParseState at all. Known limit, and the half most easily got wrong next: a token the parse never saw carries no decision to honor, so a view falls back to the vocabulary — the tell is `span is None`, which `ParsedName.replace()` builds when it splices raw text into a field, and NOT untaggedness, since an ordinary parsed name word carries no tags either. Within a view that fallback is drawn per QUESTION (rules.md#R4's Accepted clause); which questions a view can answer at all differs per VIEW and not per question, because the views hold different amounts of the name. Whether a word is the conjunction or an initial is a property of the word, and both views answer it; whether a part is wholly particles is a property of the whole PART, which `initials()` answers because it holds every token of the role at once (scoped per role — rules.md#R3's Accepted clause) and `_cap_word` cannot, walking one word of one token, so repair leaves that half to plain particle treatment and rules.md#R4's Accepted boundary records the consequence — a spliced field is not repaired as a parsed one is, with `Parser.revise()` as the supported crossing. A fallback is right only while it answers as the pipeline would, and THAT is held by hand rather than mechanically: test_regex_sync pins the two `_INITIAL` copies to each other and to config, while the repertoire half of the pipeline's predicate (#320) is deliberately not carried across, layering forbidding the import — so a caller-added conjunction in an initialless script initials from a spliced field where the same word parsed does not. A second limit, recorded rather than closed: `_cap_word`'s PARTICLE conjunct still keys on the lexicon handed to the view rather than on the `particle` tag, so a repair run with a lexicon other than the parse's re-decides a word the parse already read — a name parsed under the default vocabulary, where `parse('juan smith vega')` reads `vega` as the family, repairs to `Juan Smith vega` when `capitalized()` is handed `Lexicon.default().add(particles={'vega'})` instead — the divergence needs the two lexicons to differ, and repairing under the parse's own lexicon gives `Juan Smith Vega`. Making it read the tag moves a boundary rules.md#R4 states in prose, so it is a separate decision and not a cleanup (decisions.md#R4, "NOT DONE"). Lives in. nameparser/_render.py (`capitalized`/`_cap_word`, `initials`/`_initials_from`, `_reads_as_conjunction`) and nameparser/_types.py (`_text_for`), reading what nameparser/_pipeline/ recorded — the mark those views read is recomputed producer-side in `_remarked`, which is deliberately silent about text nobody classified and is right to be. Reach for it when. A view is about to consult a Lexicon, a regex or an exception list about a word the parse already saw — or a view and a field disagree about the same parse. #408 is filed as that second shape, and was open on 2026-08-29: `initials()` walks tokens in written order where the family field applies FOLDED_TAG's ordering, so `parse("der, y van")` gave family `van der` and initials `y. d. v.` that day, one view not yet honoring the fold. It belongs here rather than argued again from scratch. + ## CLAUSE-CONTENT-OVERRULES-DELIMITER — content wins Problem shape. A bracketed clause should be treated as something other than what its delimiter pair says. Contract statement. extract may inspect a clause's content against the lexicon and, when it matches, mask only the two delimiter spans so the inner content rejoins the main token stream for ordinary downstream parsing. How it works. "Andrew Perkins (MBA)" is not a nickname (rule S1): the parens are masked away and MBA is classified by the normal machinery — reusing the downstream path, so the delimited and bare forms cannot drift. Lives in. nameparser/_pipeline/_extract.py (_suffix_shaped and the inner-span branch). Reach for it when. About to add a second code path that duplicates what the bare form already does — #335's fix is this shape (a _maiden_marked sibling predicate). diff --git a/nameparser/_render.py b/nameparser/_render.py index 947f1f51..39dc1957 100644 --- a/nameparser/_render.py +++ b/nameparser/_render.py @@ -240,15 +240,19 @@ def _cap_word(word: str, role: Role, tags: frozenset[str], # the conjunction-versus-initial decision once, over the whole # token -- v1's is_conjunction excludes initials, so 'E.' in # 'Scott E. Werner' is an initial and is never tagged (pinned live - # 2026-07-17) -- and a view honors that decision rather than taking - # it again from the spelling (mechanisms.md#VOCAB-TAGS: "later - # stages test tags"). Asking again was not even the same question: + # 2026-07-17) -- and a view honors that decision rather than + # taking it again from the spelling + # (mechanisms.md#RENDER-HONORS-THE-PARSE: "the render views honor + # those decisions and never re-evaluate them"), the tags being + # classify's record of it (mechanisms.md#VOCAB-TAGS: "later stages + # test tags"). Asking again was not even the same question: # the copy of the initial pattern that stood here was the SHAPE # half alone, and it re-decided per WORD of a token's text, so - # 'juan e-f smith' capitalized to 'Juan e-F Smith'. A token the - # parse never saw has no decision to honor, so the vocabulary - # answers for it instead -- _reads_as_conjunction above, which is - # v1's predicate over v1's own input class, keeping every assigned + # 'juan e-f smith' capitalized to 'Juan e-F Smith'. + # mechanisms.md#RENDER-HONORS-THE-PARSE: "a token the parse never + # saw carries no decision to honor, so a view falls back to the + # vocabulary" -- _reads_as_conjunction above, which is v1's + # predicate over v1's own input class, keeping every assigned # field exactly as 1.4.0 repaired it. if ((normalized in lex.particles and role in (Role.MIDDLE, Role.FAMILY) and UNJOINED_TAG not in tags) From f9c246bb3348a5569d45fe97f30679dad9a563bc Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sat, 29 Aug 2026 21:23:31 -0700 Subject: [PATCH 05/14] docs: initials falls back for a spliced field; the tag-less views cannot usage.rst's replace() passage documented and doctested the behavior this branch changes: it said the particles 'start contributing initials' and asserted 'J. d. l. V. S.'. Since the fallback commit initials() answers as a parse would, so the doctest asserts 'J. V. S.' and the prose says which views degrade and which do not. The distinction is structural, not an oversight, and mechanisms.md's RENDER-HONORS-THE-PARSE known-limit clause now carries it: a view can fall back only if it HOLDS a vocabulary. initials() and capitalized() are handed one; family_base and family_particles are properties on ParsedName, whose fields are original/tokens/ambiguities and nothing else. A spliced field still empties the particles view and leaves the base the whole field, with revise() the crossing there too. Caught by CI, not locally: uv run pytest covers --doctest-modules over nameparser/ but not the .rst files, which CI runs through sphinx-build -b doctest. Co-Authored-By: Claude Opus 5 --- docs/design/mechanisms.md | 2 +- docs/usage.rst | 16 +++++++++++++--- 2 files changed, 14 insertions(+), 4 deletions(-) diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 1ca2e35d..575e28d2 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -59,7 +59,7 @@ Problem shape. Two stages need the same answer about the same input, and the one ## RENDER-HONORS-THE-PARSE — the parse decides it, the views honor it -Problem shape. A render view needs a fact the parse already settled — whether a word is the conjunction or an initial, whether a particle is acting as one, which word renders first. Contract statement. The parse decides it; the render views honor those decisions and never re-evaluate them. Two directions break that, and each has been found here as a defect: a view RE-DERIVES the answer from the text, keeping its own copy of a pipeline predicate — that one shipped through 2.0 and 2.1 — or a view honors the record and then OVERRIDES it, readmitting what the decision excluded, which was caught before release. How it works. Re-deriving fails because the two copies stop being the same question long before anyone notices they are two: `_cap_word` re-ran the conjunction-versus-initial decision from the word's spelling against a hand-maintained copy of the pipeline's `_INITIAL` pattern while classify had already answered it and recorded it on the token, and `_classify.py` asks `is_initial()`, the shape test ANDed with a script-repertoire test since #320, where `_render.py` asked the bare pattern, and asked it per WORD of a token's text rather than per token, so `juan e-f smith` repaired to `Juan e-F Smith` (#458). Overriding fails more directly: `initials()` honored the `conjunction` tag through `_SKIP_TAGS` and then readmitted it with an `or UNJOINED_TAG in t.tags`, overriding a decision it had not taken, where rules.md#R3 excludes a conjunction "even then" — which left the two views disagreeing about one token under a caller's vocabulary that puts a word in both sets: `Anh y Van` repaired to `Anh y Van` and honored the carve-out, while the same parse initialed `A. y. V.` and ignored it (#461). Only the FORCED call witnesses the repair half, R5's gate refusing a mixed-case name before any of this is consulted. This is the CONSUMER-side rule over the producer-side entries — VOCAB-TAGS records what the vocabulary knew, MARK-DONT-STRIP what a stage decided about it, FOLDED_TAG what order to render in — and a view reads what they recorded, whichever kind it is. It sits where ONE-PREDICATE-PER-QUESTION's stated limit leaves off: where two live sites need one answer they share a predicate, but a render view always comes AFTER the decider, so the answer is recorded rather than shared — on the TOKEN, which is the views' equivalent of that entry's `ParseState.order`, no view being able to see a ParseState at all. Known limit, and the half most easily got wrong next: a token the parse never saw carries no decision to honor, so a view falls back to the vocabulary — the tell is `span is None`, which `ParsedName.replace()` builds when it splices raw text into a field, and NOT untaggedness, since an ordinary parsed name word carries no tags either. Within a view that fallback is drawn per QUESTION (rules.md#R4's Accepted clause); which questions a view can answer at all differs per VIEW and not per question, because the views hold different amounts of the name. Whether a word is the conjunction or an initial is a property of the word, and both views answer it; whether a part is wholly particles is a property of the whole PART, which `initials()` answers because it holds every token of the role at once (scoped per role — rules.md#R3's Accepted clause) and `_cap_word` cannot, walking one word of one token, so repair leaves that half to plain particle treatment and rules.md#R4's Accepted boundary records the consequence — a spliced field is not repaired as a parsed one is, with `Parser.revise()` as the supported crossing. A fallback is right only while it answers as the pipeline would, and THAT is held by hand rather than mechanically: test_regex_sync pins the two `_INITIAL` copies to each other and to config, while the repertoire half of the pipeline's predicate (#320) is deliberately not carried across, layering forbidding the import — so a caller-added conjunction in an initialless script initials from a spliced field where the same word parsed does not. A second limit, recorded rather than closed: `_cap_word`'s PARTICLE conjunct still keys on the lexicon handed to the view rather than on the `particle` tag, so a repair run with a lexicon other than the parse's re-decides a word the parse already read — a name parsed under the default vocabulary, where `parse('juan smith vega')` reads `vega` as the family, repairs to `Juan Smith vega` when `capitalized()` is handed `Lexicon.default().add(particles={'vega'})` instead — the divergence needs the two lexicons to differ, and repairing under the parse's own lexicon gives `Juan Smith Vega`. Making it read the tag moves a boundary rules.md#R4 states in prose, so it is a separate decision and not a cleanup (decisions.md#R4, "NOT DONE"). Lives in. nameparser/_render.py (`capitalized`/`_cap_word`, `initials`/`_initials_from`, `_reads_as_conjunction`) and nameparser/_types.py (`_text_for`), reading what nameparser/_pipeline/ recorded — the mark those views read is recomputed producer-side in `_remarked`, which is deliberately silent about text nobody classified and is right to be. Reach for it when. A view is about to consult a Lexicon, a regex or an exception list about a word the parse already saw — or a view and a field disagree about the same parse. #408 is filed as that second shape, and was open on 2026-08-29: `initials()` walks tokens in written order where the family field applies FOLDED_TAG's ordering, so `parse("der, y van")` gave family `van der` and initials `y. d. v.` that day, one view not yet honoring the fold. It belongs here rather than argued again from scratch. +Problem shape. A render view needs a fact the parse already settled — whether a word is the conjunction or an initial, whether a particle is acting as one, which word renders first. Contract statement. The parse decides it; the render views honor those decisions and never re-evaluate them. Two directions break that, and each has been found here as a defect: a view RE-DERIVES the answer from the text, keeping its own copy of a pipeline predicate — that one shipped through 2.0 and 2.1 — or a view honors the record and then OVERRIDES it, readmitting what the decision excluded, which was caught before release. How it works. Re-deriving fails because the two copies stop being the same question long before anyone notices they are two: `_cap_word` re-ran the conjunction-versus-initial decision from the word's spelling against a hand-maintained copy of the pipeline's `_INITIAL` pattern while classify had already answered it and recorded it on the token, and `_classify.py` asks `is_initial()`, the shape test ANDed with a script-repertoire test since #320, where `_render.py` asked the bare pattern, and asked it per WORD of a token's text rather than per token, so `juan e-f smith` repaired to `Juan e-F Smith` (#458). Overriding fails more directly: `initials()` honored the `conjunction` tag through `_SKIP_TAGS` and then readmitted it with an `or UNJOINED_TAG in t.tags`, overriding a decision it had not taken, where rules.md#R3 excludes a conjunction "even then" — which left the two views disagreeing about one token under a caller's vocabulary that puts a word in both sets: `Anh y Van` repaired to `Anh y Van` and honored the carve-out, while the same parse initialed `A. y. V.` and ignored it (#461). Only the FORCED call witnesses the repair half, R5's gate refusing a mixed-case name before any of this is consulted. This is the CONSUMER-side rule over the producer-side entries — VOCAB-TAGS records what the vocabulary knew, MARK-DONT-STRIP what a stage decided about it, FOLDED_TAG what order to render in — and a view reads what they recorded, whichever kind it is. It sits where ONE-PREDICATE-PER-QUESTION's stated limit leaves off: where two live sites need one answer they share a predicate, but a render view always comes AFTER the decider, so the answer is recorded rather than shared — on the TOKEN, which is the views' equivalent of that entry's `ParseState.order`, no view being able to see a ParseState at all. Known limit, and the half most easily got wrong next: a token the parse never saw carries no decision to honor, so a view falls back to the vocabulary — the tell is `span is None`, which `ParsedName.replace()` builds when it splices raw text into a field, and NOT untaggedness, since an ordinary parsed name word carries no tags either. A view can only fall back if it HOLDS a vocabulary, which is why the fallback reaches `initials()` and `capitalized()` and not `family_base` or `family_particles`: those are properties on ParsedName, whose fields are original/tokens/ambiguities and nothing else, so a spliced field empties the particles view and leaves the base the whole field, with `Parser.revise()` the crossing there too (docs/usage.rst says so where it documents the degradation). The two that can fall back reach for `Lexicon.default()`, not the parse's lexicon, which ParsedName does not keep either. Within a view that fallback is drawn per QUESTION (rules.md#R4's Accepted clause); which questions a view can answer at all differs per VIEW and not per question, because the views hold different amounts of the name. Whether a word is the conjunction or an initial is a property of the word, and both views answer it; whether a part is wholly particles is a property of the whole PART, which `initials()` answers because it holds every token of the role at once (scoped per role — rules.md#R3's Accepted clause) and `_cap_word` cannot, walking one word of one token, so repair leaves that half to plain particle treatment and rules.md#R4's Accepted boundary records the consequence — a spliced field is not repaired as a parsed one is, with `Parser.revise()` as the supported crossing. A fallback is right only while it answers as the pipeline would, and THAT is held by hand rather than mechanically: test_regex_sync pins the two `_INITIAL` copies to each other and to config, while the repertoire half of the pipeline's predicate (#320) is deliberately not carried across, layering forbidding the import — so a caller-added conjunction in an initialless script initials from a spliced field where the same word parsed does not. A second limit, recorded rather than closed: `_cap_word`'s PARTICLE conjunct still keys on the lexicon handed to the view rather than on the `particle` tag, so a repair run with a lexicon other than the parse's re-decides a word the parse already read — a name parsed under the default vocabulary, where `parse('juan smith vega')` reads `vega` as the family, repairs to `Juan Smith vega` when `capitalized()` is handed `Lexicon.default().add(particles={'vega'})` instead — the divergence needs the two lexicons to differ, and repairing under the parse's own lexicon gives `Juan Smith Vega`. Making it read the tag moves a boundary rules.md#R4 states in prose, so it is a separate decision and not a cleanup (decisions.md#R4, "NOT DONE"). Lives in. nameparser/_render.py (`capitalized`/`_cap_word`, `initials`/`_initials_from`, `_reads_as_conjunction`) and nameparser/_types.py (`_text_for`), reading what nameparser/_pipeline/ recorded — the mark those views read is recomputed producer-side in `_remarked`, which is deliberately silent about text nobody classified and is right to be. Reach for it when. A view is about to consult a Lexicon, a regex or an exception list about a word the parse already saw — or a view and a field disagree about the same parse. #408 is filed as that second shape, and was open on 2026-08-29: `initials()` walks tokens in written order where the family field applies FOLDED_TAG's ordering, so `parse("der, y van")` gave family `van der` and initials `y. d. v.` that day, one view not yet honoring the fold. It belongs here rather than argued again from scratch. ## CLAUSE-CONTENT-OVERRULES-DELIMITER — content wins diff --git a/docs/usage.rst b/docs/usage.rst index 13017133..a831f74f 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -887,9 +887,17 @@ clears the flag, while correcting an unrelated field keeps it. ``replace()`` splits values on whitespace into plain, untagged tokens — the vocabulary knowledge a parse would have about the new -text is not there. The views that depend on tags degrade: the parser +text is not there. The views that read those tags degrade: the parser no longer knows ``de la`` are particles, so ``family_particles`` -empties and the particles start contributing initials. +empties and ``family_base`` takes the whole field. + +A view that is handed a vocabulary can fall back to it, since a token +the parse never saw carries no decision to honor — +:meth:`~nameparser.ParsedName.initials` and +:meth:`~nameparser.ParsedName.capitalized` do that, and answer as a +parse would. ``family_particles`` and ``family_base`` are properties +on the parsed name, which holds no vocabulary of its own, so they have +nothing to fall back to. .. doctest:: @@ -898,8 +906,10 @@ empties and the particles start contributing initials. >>> replaced = name.replace(family="de la Vega Smith") >>> replaced.family_particles '' + >>> replaced.family_base + 'de la Vega Smith' >>> replaced.initials() - 'J. d. l. V. S.' + 'J. V. S.' :meth:`Parser.revise() ` is the same operation with each value classified by the parser's vocabulary, so From 1c56ec3ddfcff1b7c6c23646d2b1fd76fd4d8374 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sat, 29 Aug 2026 22:04:19 -0700 Subject: [PATCH 06/14] revert(render,docs): back out #461, a conjunction in an all-particle part #461 narrowed the unjoined mark's readmission to the `particle` tag, so that a conjunction standing inside an all-particle part contributed no initial, on the authority of rules.md#R3's "A CONJUNCTION never initials, so a base that is one contributes nothing even then". Backed out here, before the branch merges. The mark is a statement about a whole PART -- none of its words is doing the work its tag names -- and #461 honored it for some of the part's words while keeping one of them out. Under `Lexicon.default().add(particles={'y'})`, `parse("Juan de y")` is such a part: `family_base` is `de y`, both words carry the mark, and #461 initialed `J. d.`, admitting the `de` as the name word the mark makes it and refusing the `y`. R2's reasoning does not split that way. A particle with nothing left to join is not acting as a particle, which is why the mark exists; a conjunction inside the same part has nothing left to join either, at the same moment and for the same reason, so it is a name word of that part and initials with the rest. `J. d. y.` is restored. That is scoped to a part the mark has ALREADY turned into name words, and not to any conjunction that joins nothing. Outside such a part the skip stands: `Juan Velasquez y Garcia` initials `J. V. G.` over base `Velasquez y Garcia`, `Juan y Garcia` gives `J. G.`, and `Juan de y` under the default vocabulary still gives `J.` -- none of them touched here or by #461. So what is wrong is R3's "even then" clause, which carries the carve-out into the one part where the joining has stopped, rather than the code that did not implement it. The clause is left standing and the question goes back to the issue; the paragraph #461 added under R3 is removed, and decisions.md records the attempt, what it broke and where the question went. mechanisms.md#RENDER-HONORS-THE-PARSE keeps the OVERRIDING direction -- it is a real hazard and the entry needs both directions -- with its instance now reading as open, the way that entry already handles #408. No shipped name is affected in either direction: `particles` and `conjunctions` are disjoint in the default vocabulary and in every locale pack, so 0 of the 1094 corpus names move and the gate holds at 229 / 194 / 102 intentional diffs, unexplained 0, at 1.4.0 / 2.0.0 / 2.1.0. #461's test is split rather than deleted. Case repair's ungated conjunction conjunct is unaffected by any of this and rests on R4's own sentence, so it keeps its half as test_repair_keeps_a_conjunction_lowercase_in_a_particle_part -- gating that conjunct passed the entire suite until #461's test existed, which is the part of #461 worth keeping. The initials half is re-asserted at today's value as test_initials_readmits_a_conjunction_in_a_particle_part, doing what a `deviates:` marker would do if one could hang here: no marker can, because markers hang on rules.md example lines and every line there parses with the default vocabulary, which cannot reach this shape. So settling #461 will fail the suite until that pin moves with it, and the values the three documents quote in prose cannot go stale unnoticed. Found while writing the reversal, recorded in decisions.md and not fixed here: R3's "A CONJUNCTION never initials" is unqualified, and a conjunction in the GIVEN group has always initialed -- `John and Jane Smith` gives `J. a. J. S.`, `Duke of Edinburgh` gives `D. o. E.` -- 25 of the 1094 corpus names, reachable from the default vocabulary and so markable, unlike this one. Co-Authored-By: Claude Opus 5 --- docs/design/decisions.md | 2 +- docs/design/mechanisms.md | 2 +- docs/design/rules.md | 9 ----- docs/release_log.rst | 2 - nameparser/_render.py | 56 ++++++++++++++++----------- tests/v2/test_render.py | 81 +++++++++++++++++++++++++++------------ 6 files changed, 91 insertions(+), 61 deletions(-) diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 647e7b0b..ec9ab974 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -716,7 +716,7 @@ Declined: "Juan van der" stays "J." (no borne name, no base, and initials of a bare particle run would be nonsense). The general lesson, worth more than this instance: a deviates: marker gets written on the rule whose STATEMENT changed, but a rule can change another rule's OUTPUT without touching its statement, and nothing looks for that — the runner asserts per example line, so an unmarked downstream rule stays green precisely because its own examples avoid the affected input. When adding a marker, walk the changed rule's `interacts:` targets and ask whether any of THEIR examples move. -- 2026-08-29 #461 — FIXED: the unjoined mark readmitted BOTH tags to `initials()`, and it is a statement about one of them. R3 says a conjunction never initials "even then" — even inside the all-particle part R2 makes ordinary name words — and the override readmitted a conjunction along with the particles it is about. The reach is a caller's vocabulary alone: `particles` and `conjunctions` are disjoint in the defaults and in every locale pack, so 0 of the 1094 corpus names move, and the gate's counts hold at 229 / 194 / 102 with 0 unexplained. Under `Lexicon.default().add(particles={'y'})` three corpus names do move, all in the fixed direction: `Juan de y` from `J. d. y.` to `J. d.`, `der, y van` from `d. y. v.` to `d. v.`, and `johnny y` from `j. y.` to `j.` — the last being R3's "a base that is one contributes nothing" read literally. What makes this worth fixing over a shape nothing shipped can reach is that the two views disagreed about ONE TOKEN: `Anh y Van` capitalized as `Anh y Van`, honoring the carve-out, and initialed as `A. y. V.`, ignoring it. Case repair's conjunction conjunct was deliberately ungated on the mark for exactly this reason (rules.md#R4, and the CONJUNCTION CARVE-OUT bullet under it) — that decision was recorded, argued and unpinned, and gating the conjunct passed the whole suite until this fix's test went in. It now asserts both views on the same parse, so the pair moves together or fails together. R3 gains prose rather than an example line, for the reason the R4 carve-out gains none: the doc runner parses with the default vocabulary, over which the shape is unreachable. +- 2026-08-29 #461 — TRIED AND BACKED OUT; the question is back on the issue. The unjoined mark readmits the words of an all-particle part to `initials()`, and #461 narrowed that readmission to the `particle` tag alone, on the authority of R3's "A CONJUNCTION never initials, so a base that is one contributes nothing even then". It was reverted before merge, in the same PR. What the narrowing got wrong is best said in terms of the MARK rather than of the fields. The mark is a statement about a whole PART — none of its words is doing the work its tag names — and #461 honored that statement for some of the part's words while keeping one of them out. Under `Lexicon.default().add(particles={'y'})`, `parse("Juan de y")` is such a part: `family_base` is `de y`, both words carry the mark, and #461 initialed `J. d.`, admitting the `de` as the name word the mark makes it and refusing the `y`. R2's reasoning does not split that way. A particle with nothing left to join is not acting as a particle, which is the whole reason the mark exists; a conjunction inside that same part has nothing left to join either, for the same reason and at the same moment, so it is a name word of the part and initials with the rest. Scope that carefully, because its general form is far wider than the claim and would be wrong: this is about a part the mark has ALREADY turned into name words, not about any conjunction that happens to join nothing. Outside such a part the skip stands whatever the conjunction is or is not joining, and none of those moved or may move — `parse("Juan y Garcia")` initials `J. G.` with the `y` a middle name word by P3, `parse("Juan Velasquez y Garcia")` initials `J. V. G.` over base `Velasquez y Garcia`, `parse("Jon Dough and")` initials `J. D.` over base `Dough and`. Base and initials differ in every one of those, legitimately, and R3's own `"Juan de y" → initials="J."` line is a fourth, its `family_base` being `y` under the default vocabulary. So "the base holds a word the initials do not" is NOT the criterion and was not the finding; the mark is. What is therefore in question is R3's "even then" clause, which carries the carve-out into the one part where the joining has stopped — not the code that failed to implement it. The clause is left standing and #461 now asks whether it belongs there; nothing else in R3 moved, and the paragraph #461 added under it is gone. R4's conjunction sentence rests on that clause by name ("the carve-out R3 states for initials") and is untouched here, so whoever settles #461 settles R4's cross-reference with it. Note what the backout cannot record: a `deviates:` marker hangs on an example LINE and this shape has none to hang on — `particles` and `conjunctions` are disjoint in the default vocabulary and in every locale pack, so no input string the doc runner parses reaches it. rules.md itself therefore carries no trace, and this bullet with mechanisms.md#RENDER-HONORS-THE-PARSE is where the gap is written down. It is not the only gap under R3's conjunction sentence, and the other is older, wider and markable: "A CONJUNCTION never initials" is unqualified, while a conjunction in the GIVEN group has always initialed — `parse("John and Jane Smith")` gives `J. a. J. S.` and `parse("Duke of Edinburgh")` `D. o. E.`, 25 of the 1094 corpus names, all reachable from the default vocabulary. Recorded here because it was found here; it is not #461's to fix. Neither direction is visible to the gate: 0 of the 1094 corpus names move either way, and the counts hold at 229 / 194 / 102 with 0 unexplained at 1.4.0 / 2.0.0 / 2.1.0. Both views of the contested token are pinned rather than left to prose — `test_repair_keeps_a_conjunction_lowercase_in_a_particle_part` holds R4's ungated conjunct, gating which had passed the entire suite until #461's test existed, and `test_initials_readmits_a_conjunction_in_a_particle_part` holds today's initials answer the way a `deviates:` marker would, so re-deciding #461 fails the suite until that pin moves with it. Declined: diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 575e28d2..7ff5a84c 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -59,7 +59,7 @@ Problem shape. Two stages need the same answer about the same input, and the one ## RENDER-HONORS-THE-PARSE — the parse decides it, the views honor it -Problem shape. A render view needs a fact the parse already settled — whether a word is the conjunction or an initial, whether a particle is acting as one, which word renders first. Contract statement. The parse decides it; the render views honor those decisions and never re-evaluate them. Two directions break that, and each has been found here as a defect: a view RE-DERIVES the answer from the text, keeping its own copy of a pipeline predicate — that one shipped through 2.0 and 2.1 — or a view honors the record and then OVERRIDES it, readmitting what the decision excluded, which was caught before release. How it works. Re-deriving fails because the two copies stop being the same question long before anyone notices they are two: `_cap_word` re-ran the conjunction-versus-initial decision from the word's spelling against a hand-maintained copy of the pipeline's `_INITIAL` pattern while classify had already answered it and recorded it on the token, and `_classify.py` asks `is_initial()`, the shape test ANDed with a script-repertoire test since #320, where `_render.py` asked the bare pattern, and asked it per WORD of a token's text rather than per token, so `juan e-f smith` repaired to `Juan e-F Smith` (#458). Overriding fails more directly: `initials()` honored the `conjunction` tag through `_SKIP_TAGS` and then readmitted it with an `or UNJOINED_TAG in t.tags`, overriding a decision it had not taken, where rules.md#R3 excludes a conjunction "even then" — which left the two views disagreeing about one token under a caller's vocabulary that puts a word in both sets: `Anh y Van` repaired to `Anh y Van` and honored the carve-out, while the same parse initialed `A. y. V.` and ignored it (#461). Only the FORCED call witnesses the repair half, R5's gate refusing a mixed-case name before any of this is consulted. This is the CONSUMER-side rule over the producer-side entries — VOCAB-TAGS records what the vocabulary knew, MARK-DONT-STRIP what a stage decided about it, FOLDED_TAG what order to render in — and a view reads what they recorded, whichever kind it is. It sits where ONE-PREDICATE-PER-QUESTION's stated limit leaves off: where two live sites need one answer they share a predicate, but a render view always comes AFTER the decider, so the answer is recorded rather than shared — on the TOKEN, which is the views' equivalent of that entry's `ParseState.order`, no view being able to see a ParseState at all. Known limit, and the half most easily got wrong next: a token the parse never saw carries no decision to honor, so a view falls back to the vocabulary — the tell is `span is None`, which `ParsedName.replace()` builds when it splices raw text into a field, and NOT untaggedness, since an ordinary parsed name word carries no tags either. A view can only fall back if it HOLDS a vocabulary, which is why the fallback reaches `initials()` and `capitalized()` and not `family_base` or `family_particles`: those are properties on ParsedName, whose fields are original/tokens/ambiguities and nothing else, so a spliced field empties the particles view and leaves the base the whole field, with `Parser.revise()` the crossing there too (docs/usage.rst says so where it documents the degradation). The two that can fall back reach for `Lexicon.default()`, not the parse's lexicon, which ParsedName does not keep either. Within a view that fallback is drawn per QUESTION (rules.md#R4's Accepted clause); which questions a view can answer at all differs per VIEW and not per question, because the views hold different amounts of the name. Whether a word is the conjunction or an initial is a property of the word, and both views answer it; whether a part is wholly particles is a property of the whole PART, which `initials()` answers because it holds every token of the role at once (scoped per role — rules.md#R3's Accepted clause) and `_cap_word` cannot, walking one word of one token, so repair leaves that half to plain particle treatment and rules.md#R4's Accepted boundary records the consequence — a spliced field is not repaired as a parsed one is, with `Parser.revise()` as the supported crossing. A fallback is right only while it answers as the pipeline would, and THAT is held by hand rather than mechanically: test_regex_sync pins the two `_INITIAL` copies to each other and to config, while the repertoire half of the pipeline's predicate (#320) is deliberately not carried across, layering forbidding the import — so a caller-added conjunction in an initialless script initials from a spliced field where the same word parsed does not. A second limit, recorded rather than closed: `_cap_word`'s PARTICLE conjunct still keys on the lexicon handed to the view rather than on the `particle` tag, so a repair run with a lexicon other than the parse's re-decides a word the parse already read — a name parsed under the default vocabulary, where `parse('juan smith vega')` reads `vega` as the family, repairs to `Juan Smith vega` when `capitalized()` is handed `Lexicon.default().add(particles={'vega'})` instead — the divergence needs the two lexicons to differ, and repairing under the parse's own lexicon gives `Juan Smith Vega`. Making it read the tag moves a boundary rules.md#R4 states in prose, so it is a separate decision and not a cleanup (decisions.md#R4, "NOT DONE"). Lives in. nameparser/_render.py (`capitalized`/`_cap_word`, `initials`/`_initials_from`, `_reads_as_conjunction`) and nameparser/_types.py (`_text_for`), reading what nameparser/_pipeline/ recorded — the mark those views read is recomputed producer-side in `_remarked`, which is deliberately silent about text nobody classified and is right to be. Reach for it when. A view is about to consult a Lexicon, a regex or an exception list about a word the parse already saw — or a view and a field disagree about the same parse. #408 is filed as that second shape, and was open on 2026-08-29: `initials()` walks tokens in written order where the family field applies FOLDED_TAG's ordering, so `parse("der, y van")` gave family `van der` and initials `y. d. v.` that day, one view not yet honoring the fold. It belongs here rather than argued again from scratch. +Problem shape. A render view needs a fact the parse already settled — whether a word is the conjunction or an initial, whether a particle is acting as one, which word renders first. Contract statement. The parse decides it; the render views honor those decisions and never re-evaluate them. Two directions break that, and each has been found here as a defect: a view RE-DERIVES the answer from the text, keeping its own copy of a pipeline predicate — that one shipped through 2.0 and 2.1 — or a view honors the record and then OVERRIDES it, readmitting what the decision excluded, which is filed and open here. How it works. Re-deriving fails because the two copies stop being the same question long before anyone notices they are two: `_cap_word` re-ran the conjunction-versus-initial decision from the word's spelling against a hand-maintained copy of the pipeline's `_INITIAL` pattern while classify had already answered it and recorded it on the token, and `_classify.py` asks `is_initial()`, the shape test ANDed with a script-repertoire test since #320, where `_render.py` asked the bare pattern, and asked it per WORD of a token's text rather than per token, so `juan e-f smith` repaired to `Juan e-F Smith` (#458). Overriding fails more directly, and this entry's instance of it is OPEN rather than closed: `initials()` honors the `conjunction` tag through `_SKIP_TAGS` and then readmits the token whenever it also carries UNJOINED_TAG, the mark of an all-particle part, where rules.md#R3 excludes a conjunction "even then" — so under a caller's vocabulary that puts a word in both sets the view readmits what the rule excluded, and `parse("Anh y Van")` initialed `A. y. V.` and `parse("Juan de y")` `J. d. y.` on 2026-08-29. #461 is where that stands, and it is worth reading before re-fixing it: the narrowing was written, measured and BACKED OUT in the same PR, because honoring R3 there cost `initials()` its agreement with `family_base`, which reads that same `de y` as the base — this entry's other shape, arrived at from the render side, and the reason the question moved from the code to R3's clause (decisions.md carries the argument). What to carry away is the DIRECTION and not its verdict here: a view that honors a record and then readmits what the record excluded is overriding a decision it never took, whichever way this one settles. Case repair reads that same token and does NOT readmit it — `capitalized(force=True)` on `Anh y Van` gives `Anh y Van`, R4 stating the carve-out in its own right — so the two views disagree today about that token exactly as they did before #461 and the backout restores that disagreement knowingly; only the FORCED call witnesses the repair half, R5's gate refusing a mixed-case name before any of this is consulted. This is the CONSUMER-side rule over the producer-side entries — VOCAB-TAGS records what the vocabulary knew, MARK-DONT-STRIP what a stage decided about it, FOLDED_TAG what order to render in — and a view reads what they recorded, whichever kind it is. It sits where ONE-PREDICATE-PER-QUESTION's stated limit leaves off: where two live sites need one answer they share a predicate, but a render view always comes AFTER the decider, so the answer is recorded rather than shared — on the TOKEN, which is the views' equivalent of that entry's `ParseState.order`, no view being able to see a ParseState at all. Known limit, and the half most easily got wrong next: a token the parse never saw carries no decision to honor, so a view falls back to the vocabulary — the tell is `span is None`, which `ParsedName.replace()` builds when it splices raw text into a field, and NOT untaggedness, since an ordinary parsed name word carries no tags either. A view can only fall back if it HOLDS a vocabulary, which is why the fallback reaches `initials()` and `capitalized()` and not `family_base` or `family_particles`: those are properties on ParsedName, whose fields are original/tokens/ambiguities and nothing else, so a spliced field empties the particles view and leaves the base the whole field, with `Parser.revise()` the crossing there too (docs/usage.rst says so where it documents the degradation). The two that can fall back reach for `Lexicon.default()`, not the parse's lexicon, which ParsedName does not keep either. Within a view that fallback is drawn per QUESTION (rules.md#R4's Accepted clause); which questions a view can answer at all differs per VIEW and not per question, because the views hold different amounts of the name. Whether a word is the conjunction or an initial is a property of the word, and both views answer it; whether a part is wholly particles is a property of the whole PART, which `initials()` answers because it holds every token of the role at once (scoped per role — rules.md#R3's Accepted clause) and `_cap_word` cannot, walking one word of one token, so repair leaves that half to plain particle treatment and rules.md#R4's Accepted boundary records the consequence — a spliced field is not repaired as a parsed one is, with `Parser.revise()` as the supported crossing. A fallback is right only while it answers as the pipeline would, and THAT is held by hand rather than mechanically: test_regex_sync pins the two `_INITIAL` copies to each other and to config, while the repertoire half of the pipeline's predicate (#320) is deliberately not carried across, layering forbidding the import — so a caller-added conjunction in an initialless script initials from a spliced field where the same word parsed does not. A second limit, recorded rather than closed: `_cap_word`'s PARTICLE conjunct still keys on the lexicon handed to the view rather than on the `particle` tag, so a repair run with a lexicon other than the parse's re-decides a word the parse already read — a name parsed under the default vocabulary, where `parse('juan smith vega')` reads `vega` as the family, repairs to `Juan Smith vega` when `capitalized()` is handed `Lexicon.default().add(particles={'vega'})` instead — the divergence needs the two lexicons to differ, and repairing under the parse's own lexicon gives `Juan Smith Vega`. Making it read the tag moves a boundary rules.md#R4 states in prose, so it is a separate decision and not a cleanup (decisions.md#R4, "NOT DONE"). Lives in. nameparser/_render.py (`capitalized`/`_cap_word`, `initials`/`_initials_from`, `_reads_as_conjunction`) and nameparser/_types.py (`_text_for`), reading what nameparser/_pipeline/ recorded — the mark those views read is recomputed producer-side in `_remarked`, which is deliberately silent about text nobody classified and is right to be. Reach for it when. A view is about to consult a Lexicon, a regex or an exception list about a word the parse already saw — or a view and a field disagree about the same parse. #408 is filed as that second shape, and was open on 2026-08-29: `initials()` walks tokens in written order where the family field applies FOLDED_TAG's ordering, so `parse("der, y van")` gave family `van der` and initials `y. d. v.` that day, one view not yet honoring the fold. It belongs here rather than argued again from scratch. ## CLAUSE-CONTENT-OVERRULES-DELIMITER — content wins diff --git a/docs/design/rules.md b/docs/design/rules.md index 253d17a5..21f572c4 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -1110,15 +1110,6 @@ R3. Rationale: initials abbreviate the person's name words; titles, rather than nothing: they are the base (R2), so they initial. "Juan van der" → initials="J. v. d." "Juan de y" → initials="J." - The conjunction half of that carries no example line, and the - reason is a limit of this document rather than a choice: every - line here names an input string parsed with the default - vocabulary, over which particles and conjunctions are disjoint — - in the defaults and in every locale pack — so no string can put a - conjunction inside an all-particle part. It takes a caller's own - vocabulary, where "Juan de y" is that part and initials "J. d." - while the "y" still contributes nothing, and it is pinned in - tests/v2/test_render.py. Accepted: this rule reads a part the parser read, and where a field was set as raw text after the parse it reads that text as the parser would have — the words of the part are all in hand diff --git a/docs/release_log.rst b/docs/release_log.rst index 3622af6e..4fa78128 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -71,8 +71,6 @@ Release Log - Fix ``ParsedName.initials()`` reading a field assigned after the parse as though every word in it were a name word, so particles and conjunctions inside one contributed initials that the same words parsed do not. ``parse("john smith").replace(family="velasquez y garcia").initials()`` gave ``j. v. y. g.`` and ``replace(family="de la vega")`` gave ``j. d. l. v.``, where parsing those names gives ``j. v. g.`` and ``j. v.``. Text spliced into a field is never classified, so the view has no reading to honor; it now asks the vocabulary, which answers as the parser would -- an assigned middle initial stays an initial rather than becoming the Italian conjunction (``replace(middle="E.")`` still gives ``j. E. s.``), an uppercase ``Y`` in an assigned surname still contributes, and a field that is nothing but particle words still initials all of them, since none of them is doing a particle's work there (``replace(family="de la")`` gives ``j. d. l.``, as parsing it does). This is the 2.0 API only: ``HumanName.initials()`` and ``initials_list()`` compute from the field strings with their own vocabulary and already gave every one of these answers. Shipped since 2.0.0, and found while reviewing the case-repair change above (#458) - - Fix ``initials()`` initialing a conjunction that stands inside a family name made only of particle words, where every other reading of that name treats it as the conjunction it is. A part whose every word is particle vocabulary has none of them acting as a particle, so those words initial like ordinary name words (#402 above) -- but a conjunction is not one of them in any part, and the readmission covered it too. One token was then read two ways: with ``y`` configured as a particle as well as the conjunction it already is, ``"Anh y Van"`` capitalized to ``Anh y Van`` and initialed to ``A. y. V.``; it initials to ``A. V.`` now, and ``"Juan de y"`` to ``J. d.`` -- the ``de`` an ordinary name word there, the ``y`` still contributing nothing. Reaching this needs a ``Lexicon`` a caller configured with a word in both ``particles`` and ``conjunctions``: the shipped sets are disjoint, in the default vocabulary and in every locale pack, so no name parsed with the shipped vocabulary changes at all -- none of the 1094 differential corpus names moves, and the gate reports the same 229 / 194 / 102 intentional diffs with nothing unexplained at the 1.4.0, 2.0.0 and 2.1.0 baselines. Filed so the rule and the implementation would not stay silently in disagreement, ``rules.md#R3`` having said this since it was written (closes #461) - - Fix a tussenvoegsel attached to the family name after a comma deciding a genuinely uncertain reading and reporting nothing. ``"Van Johnson"`` reports a ``PARTICLE_OR_GIVEN`` ambiguity -- ``Van`` is a Dutch particle and a Vietnamese given name, and the parser has to pick one -- while ``"Nguyen, Thi Van"`` picked the same word the same way, silently, and lost the given name doing it. The attachment now reports the fork it decides, in the kind that names the reading it declined. A particle that could be an ordinary name reports ``PARTICLE_OR_GIVEN``: ``"Nguyen, Thi Van"``, ``"Berg, Jan van der"`` and ``"Vega, Juan de la"`` each gain one, the ``detail`` naming the ambiguous word. A particle the parser had already read as a post-nominal reports ``SUFFIX_OR_NAME`` instead, because the credential reading is what the attachment overrode: ``"Berg, Jan vd"`` gains one, ``vd`` being read as *van der* rather than as the Volunteer Decoration. Which kind you get follows the reading that was overridden rather than the word's vocabulary, so ``"Berg, Jan do"`` reports ``PARTICLE_OR_GIVEN`` even though ``do`` is a postnominal too -- it was already being read as a name word, so no credential reading was overridden. A particle where nothing was overridden reports nothing at all: ``"Jong, Piet de"`` and ``"Jong, Anke de"`` are unchanged, ``de`` being no name in any reading and no postnominal either, and so is ``"Berg, Jan de vd"``, whose run was read as name words whole. Worth knowing before you filter on this: ``"Beethoven, Ludwig van"`` -- the textbook Dutch listing, read exactly right -- now carries a report too. It is the same string shape over the same vocabulary as ``"Nguyen, Thi Van"``, and nothing in the input separates them, so a report on one is a report on both. This adds the report and nothing else: every field these names parse to is exactly what the tussenvoegsel fix below already gave them, and ``ambiguities`` is the only value that grows. Seven differential corpus names gain a kind against the 2.0.0 and 2.1.0 baselines, seven of the nine the tussenvoegsel fix below already moved -- five of ``fix(#379)``'s seven and both of ``fix(#380)``'s two, now carrying ``_ambiguities`` in their diff as well; the two left out are the pair named unchanged above, ``Jong, Piet de`` and ``Jong, Anke de`` -- and none against 1.4.0, which had no ambiguity reporting at all (closes #405) - Fix a tussenvoegsel after a family comma being parsed as a middle name. Dutch and Belgian alphabetized listings move the particle behind the given name -- ``"Beethoven, Ludwig van"`` is how ``"Ludwig van Beethoven"`` is filed -- and the trailing particle run was read as a middle name rather than as part of the surname: ``"Beethoven, Ludwig van"`` gave middle ``van``, last ``Beethoven``, and ``"Berg, Jan van der"`` gave middle ``van der``. The run now attaches to the family the comma has already named and renders before it, so those read family ``van Beethoven`` and ``van der Berg`` with the given name unchanged. The derived views move with the parse, so ``family_particles`` is ``van`` and ``family_base`` is ``Beethoven`` where they were empty and ``Beethoven`` before. `#130 `_ asked for the split and got it in 1.3.0 as ``last_base``/``last_prefixes``; 2.0 renamed them ``family_base``/``family_particles``. What was wrong until now was the values they reported for this listing. Both halves of the particle vocabulary attach -- never-given ``de`` and may-be-given ``van`` alike -- because after a comma the family is already named and the particle has no other role to take. Two guards bound it. A name whose only given word is the particle keeps it, so ``"Nguyen, Van"`` still reads given ``Van``: the attachment needs a given word to spare. And where the word is BOTH particle and suffix vocabulary the attachment outranks the post-nominal reading, so ``"Berg, Jan vd"`` reads family ``vd Berg`` where 1.4.0 and 2.1 alike gave suffix ``vd`` -- a trailing abbreviation after a family comma is the tussenvoegsel far more often than the decoration it collides with, and the same shape sweeps in ``mc``, which 2.1 also read as a suffix. ``do`` is in ``SUFFIX_ACRONYMS_AMBIGUOUS`` and 2.1 already read a trailing one as a name word, so it attaches by the plain rule rather than by the override (closes #379, closes #380). Names without the comma are untouched: ``"Ludwig van Beethoven"`` already read family ``van Beethoven`` and is byte-identical. One of the 751 differential corpus names moves, ``"Vega, Juan de la"``, at the 1.4.0, 2.0.0 and 2.1.0 baselines alike diff --git a/nameparser/_render.py b/nameparser/_render.py index 39dc1957..1f4c9a82 100644 --- a/nameparser/_render.py +++ b/nameparser/_render.py @@ -35,10 +35,10 @@ #: Tags whose tokens contribute no initial outside the given group -- #: unless the token also carries UNJOINED_TAG, i.e. the whole part is #: particles, in which case they are the part's only words and do -#: contribute (rules.md#R3, #404). The mark readmits the PARTICLE tag -#: alone: it says those particles are not acting as particles here and -#: nothing about a conjunction, which rules.md#R3 excludes "even then" -#: (#461). +#: contribute (rules.md#R3, #404). The mark readmits a token carrying +#: EITHER tag: a conjunction with nothing to join is not acting as a +#: conjunction any more than a particle with nothing to join is acting +#: as a particle, so it is a name word of the part like the rest. #: Not STABLE_TAGS -- that also contains "initial", which must contribute. _SKIP_TAGS = frozenset({"particle", "conjunction"}) @@ -123,8 +123,8 @@ def initials(name: ParsedName, spec: str, delimiter: str, separator: str) -> str delimiter follows each initial, separator sits between initials within a group. Tokens tagged particle/conjunction contribute no initial in middle/family (given-name tokens always contribute), - and the unjoined mark readmits the particles of an all-particle - part but never a conjunction standing in one; tags come from the + and the unjoined mark readmits the words of an all-particle part + whichever of those tags they carry; tags come from the pipeline. A token with no span was never classified -- replace() splices one in -- so its role's words are read from the default vocabulary instead, all-particle test included. Valid spec keys: @@ -193,17 +193,20 @@ def _initials_from(token: Token, lex: Lexicon | None, already carries. """ if lex is None: - # rules.md#R3: "A CONJUNCTION never initials, so a base that is - # one contributes nothing even then" -- "even then" being the - # all-particle part the mark names, so the mark readmits the - # tag it is about and not the other one (#461) + # the mark readmits a skip-tagged token whichever of the two + # tags it carries: inside an all-particle part none of the + # words is doing the work its tag names -- a conjunction with + # nothing to join no more than a particle with nothing to join + # (R2) -- so they are the part's name words and initial. if _SKIP_TAGS & token.tags: - return UNJOINED_TAG in token.tags and "conjunction" not in token.tags + return UNJOINED_TAG in token.tags return True + # the same two questions, answered from the vocabulary where the + # token carries no tags, and reaching the same answer: both skip + # words are readmitted by the mark and by nothing else if ("conjunction" in token.tags - or (token.span is None and _reads_as_conjunction(token.text, lex))): - return False - if _is_particle_word(token, lex): + or (token.span is None and _reads_as_conjunction(token.text, lex)) + or _is_particle_word(token, lex)): return unjoined return True @@ -217,11 +220,10 @@ def _cap_word(word: str, role: Role, tags: frozenset[str], # repaired as ordinary name words, since none of them is doing a # particle's work there" -- UNJOINED_TAG is that mark (#407). # Only the PARTICLE conjunct is gated on it, and that is the rule - # rather than an omission: rules.md#R4 carries the carve-out R3 - # already states for initials -- "A CONJUNCTION never initials, so - # a base that is one contributes nothing even then" -- so a - # conjunction keeps conjunction treatment even inside a part the - # mark has turned into ordinary name words. + # rather than an omission -- rules.md#R4: "A CONJUNCTION keeps its + # lowercase even inside such a part, being no name word in any + # part" -- so a conjunction keeps conjunction treatment even + # inside a part the mark has turned into ordinary name words. # No SHIPPED name witnesses the difference: `particles` and # `conjunctions` are disjoint in the default vocabulary and in # every locale pack, so no shipped conjunction can sit in an @@ -232,10 +234,18 @@ def _cap_word(word: str, role: Role, tags: frozenset[str], # under `Lexicon.default().add(particles={'y'})`, `anh y van` has # an all-particle family whose `y` carries both tags and the mark, # and gives 'Anh y Van'; gating this conjunct too would give - # 'Anh Y Van'. That is pinned, on the same parse as the initials() - # half it matches, by test_initials_and_repair_agree_on_a_ - # conjunction_in_a_particle_part (#461) -- until which gating it - # passed the whole suite. + # 'Anh Y Van'. That is pinned by test_repair_keeps_a_conjunction_ + # lowercase_in_a_particle_part -- until which gating it passed the + # whole suite. + # initials() does NOT match this carve-out, and the mismatch is + # recorded rather than fixed: #461 made it match and was backed + # out, the mark being a statement about a whole PART that #461 + # honored for some of the part's words and not for one of them, + # so what is in question is R3's "even then" clause rather than + # the code (decisions.md, under R2). Under that same lexicon + # `Anh y Van` repairs to 'Anh y Van' and initials 'A. y. V.', + # pinned by + # test_initials_readmits_a_conjunction_in_a_particle_part. # That conjunct reads the TAG, not the word (#458). classify takes # the conjunction-versus-initial decision once, over the whole # token -- v1's is_conjunction excludes initials, so 'E.' in diff --git a/tests/v2/test_render.py b/tests/v2/test_render.py index 160ea7cc..f3af5c22 100644 --- a/tests/v2/test_render.py +++ b/tests/v2/test_render.py @@ -1,6 +1,6 @@ import pytest -from nameparser import Parser +from nameparser import Parser, parse from nameparser._lexicon import Lexicon from nameparser._render import _collapse, render from nameparser._types import (UNJOINED_TAG, Ambiguity, AmbiguityKind, @@ -126,24 +126,22 @@ def test_initials_skips_tagged_particles_outside_given() -> None: assert pn.initials("{family}") == "S." -def test_initials_and_repair_agree_on_a_conjunction_in_a_particle_part() -> None: - """#461, and the one shape that needs a caller's own Lexicon. +def test_repair_keeps_a_conjunction_lowercase_in_a_particle_part() -> None: + """rules.md#R4's conjunction carve-out, which no shipped name reaches. - The unjoined mark readmits the words of an all-particle part - because none of them is acting as a particle there -- but it says - nothing about a conjunction, which rules.md#R3 excludes "even - then". Reaching that needs a word in `particles` AND + The unjoined mark makes the words of an all-particle part + ordinary name words, since none of them is acting as a particle + there -- but repair's conjunction conjunct is deliberately UNGATED + on that mark, R4 keeping a conjunction lowercase "even inside such + a part". Witnessing it needs a word in `particles` AND `conjunctions`; the shipped sets are disjoint, in the default vocabulary and in every locale pack, so no input string can - witness it and rules.md can carry no example line for it. - - This is also where the two views can be shown to agree, which is - the whole point of the carve-out: before #461, `Anh y Van` - capitalized as `Anh y Van` and initialed as `A. y. V.` -- one - token, two views, opposite readings. The `capitalized` assertions - here are what pins case repair's ungated conjunction conjunct - (rules.md#R4); gating it on the mark passed the entire suite until - this test existed, since no shipped name reaches it either. + witness it and rules.md can carry no example line for it. Gating + the conjunct on the mark passed the entire suite until this test + existed. + + Only the FORCED call gets here: R5's gate returns a mixed-case + name before any of this is consulted. """ # mechanisms.md#VOCABULARY-OVERLAP-AS-PRECONDITION: "assert the # intersection as a precondition" -- built here rather than found, @@ -158,17 +156,50 @@ def test_initials_and_repair_agree_on_a_conjunction_in_a_particle_part() -> None van = p.parse("Anh y Van") tags = [t.tags for t in van.tokens if t.text == "y"][0] assert {"conjunction", "particle", UNJOINED_TAG} <= tags, sorted(tags) - assert van.initials() == "A. V." assert van.capitalized(lex, force=True).family == "y Van" - # R3's own example line, under a lexicon that makes `de y` the - # all-particle part the default vocabulary cannot: `de` initials - # as the ordinary name word R2 makes it, `y` still does not. - assert p.parse("Juan de y").initials() == "J. d." - # and a base that IS the conjunction contributes nothing at all - assert p.parse("johnny y").initials() == "j." - # unchanged where the part is particles alone - assert p.parse("Juan van der").initials() == "J. v. d." + +def test_initials_readmits_a_conjunction_in_a_particle_part() -> None: + """Today's answer on an OPEN question (#461), pinned as such. + + rules.md#R3 says a conjunction never initials "even then" -- even + inside the all-particle part R2 turns into ordinary name words -- + and `initials()` does not do that: the mark readmits the part's + words whichever skip tag they carry. #461 made the code match the + clause and was backed out, the clause rather than the code being + what is now in question (decisions.md, under R2). + + So this pins what a `deviates:` marker would pin if one could + hang here -- TODAY's output, strictly, so that settling #461 + fails the suite until this moves with it. It cannot be a marker: + markers hang on rules.md example lines and every line there names + an input string parsed with the DEFAULT vocabulary, over which + `particles` and `conjunctions` are disjoint and no string reaches + this shape. It is also what keeps the values quoted in prose by + decisions.md, mechanisms.md#RENDER-HONORS-THE-PARSE and + `_render.py` from going stale unnoticed. + """ + assert "y" in Lexicon.default().conjunctions, ( + "'y' left the default conjunctions; this test builds the " + "particle/conjunction overlap it needs from the other side and " + "no longer has one. Pick another shipped conjunction.") + p = Parser(lexicon=Lexicon.default().add(particles={"y"})) + + # the part the mark has turned into name words: every word of it + # initials, the conjunction included, and the base agrees + de_y = p.parse("Juan de y") + assert de_y.family_base == "de y" + assert de_y.initials() == "J. d. y." + assert p.parse("Anh y Van").initials() == "A. y. V." + assert p.parse("johnny y").initials() == "j. y." + + # and OUTSIDE such a part the skip stands, joining or not -- + # these are what the readmission must not reach + assert p.parse("Juan Velasquez y Garcia").initials() == "J. V. G." + assert p.parse("Juan y Garcia").initials() == "J. G." + # including under the default vocabulary, where 'y' is no particle + # and the family is therefore not all-particle + assert parse("Juan de y").initials() == "J." def test_initials_custom_delimiter_and_separator() -> None: From 25844a1104775a34bcbba1b2d93f60d6d67c5c6c Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sat, 29 Aug 2026 23:31:14 -0700 Subject: [PATCH 07/14] revert(render,docs): initials has no vocabulary, so it stops guessing one The #458-review fallback made `initials()` read a spliced field the way the parser would have. It had to reach for `Lexicon.default()` to do it, because `initials()` takes no lexicon -- and guessing the default vocabulary for a name parsed under a custom one is worse than answering from tags alone. Measured: under `Lexicon.default().add(particles={'y'})`, `parse("Juan de y").initials()` is `J. d. y.` and `parse("Juan Perez").replace(family="de y").initials()` was `J. d. y.` before the fallback and `J.` after it -- a whole field's initials gone, because the default vocabulary reads `de y` as particles-plus-conjunction where the caller's reads it as an all-particle part. `_cap_word`'s fallback STAYS: it is handed the caller's lexicon, so it guesses nothing, and it is what keeps `h.last = "velasquez y garcia"` repairing as v1 does. The defect the fallback aimed at is real and goes back to being unfixed: `replace(family='de la vega').initials()` is `j. d. l. v.` where the facade and a parse both give `j. v.`. That is a 2.0-core defect and it wants the `Parser.initials` crossing that does not exist -- rules.md#R3's Accepted clause and decisions.md now say so, and an issue is drafted for the crossing. Removes `_all_particles`, `_is_particle_word`, `_initials_from` and the `unparsed_lex` scan; `_INITIAL` and `_reads_as_conjunction` stay, case repair being their reader. Co-Authored-By: Claude Opus 5 --- docs/design/decisions.md | 4 +- docs/design/rules.md | 32 ++++++----- docs/release_log.rst | 2 - docs/usage.rst | 24 +++++--- nameparser/_render.py | 82 ++++----------------------- tests/v2/test_render.py | 118 +++++++++++++++------------------------ 6 files changed, 90 insertions(+), 172 deletions(-) diff --git a/docs/design/decisions.md b/docs/design/decisions.md index ec9ab974..564ff2b6 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -737,9 +737,9 @@ Declined: - 2026-08-29 #458 — DONE: case repair's conjunction test reads the `conjunction` TAG instead of re-deriving the word's class at render time, so a view honors the decision the parse recorded (mechanisms.md#VOCAB-TAGS) rather than taking it again. The two copies of the question were not the same question, which is the reason this was worth doing over a corpus that cannot see it: `_classify.py` asks `is_initial(token.text)` — the shape test ANDed with a repertoire test since #320 — while `_render.py` asked a bare pattern, shape only, over each WORD of a token's text rather than the token. It also retires the hand-sync obligation the module carried in a comment ("keep in sync with `nameparser/_pipeline/_vocab.py` by hand"): a tag read has nothing to keep in sync, and `_render._INITIAL` is deleted, its last reader gone. MEASURED, re-derived on this branch rather than taken from the issue: 0 of the 1094 corpus names move under `capitalized()` or `capitalized(force=True)`, and 0 move with each name also re-run uppercased and lowercased (6565 name/variant/lexicon rows, the second lexicon being `Lexicon.default().add(particles={'y'})`); the R5 force-versus-case counts recorded above hold at 62/16 through the facade and 63 through the core. Movement EXISTS outside that population, and the issue's "no output moves" is a claim about the corpus only: the old predicate re-decided per word, so `juan e-f smith` capitalized to `Juan e-F Smith` — the Italian conjunction inside a hyphenated middle name — and now gives `Juan E-F Smith`, agreeing with the uppercase spelling it always gave. An earlier draft of this change moved untagged tokens the same way — a family `replace()` splices in as `de y` repairing to `de Y` — and that was a REGRESSION, caught in review before it shipped and fixed in the bullet below; read this bullet with that one. NOT DONE, and the parenthetical in the boundary bullet above should be read as still open: the PARTICLE conjunct still keys on lexicon membership, so `replace()` does not agree there. Making it read the `particle` tag is a second change with its own blast radius — it moves a boundary rules.md#R4 states in prose — and #458 as filed does not ask for it. -- 2026-08-29 (#458 review) — THE FALLBACK FOR TEXT THE PARSE NEVER SAW, and why it is scoped to one of the two questions `_cap_word` asks. Reading the tag and nothing else regressed every ASSIGNED field holding a conjunction, because a value `replace()` splices in is never classified and so carries no tag to read. Measured on the released 1.4.0 and 2.1.0 wheels, all agreeing and all broken by the tag-only reading: `h.last = 'velasquez y garcia'` then `capitalize(force=True)` gives `John Velasquez y Garcia`, `h.middle = 'e.'` gives `John E. Smith`, `h.last = 'smith-y'` gives `John Smith-y`, `h.first = 'y'` gives `y Smith`. That is the canonical Spanish surname and the shape `y` is in the vocabulary FOR, so an Accepted clause was not available: it had to be fixed. RULE: a synthetic token — `span is None`, which `ParsedName.replace()` builds and every parsed token has — was never read, so there is no decision to honor and the view falls back to the vocabulary. Note what the discriminator is NOT: an ordinary parsed name word carries no tags either (`velasquez` parses to `tags=[]`), so "untagged" would have swept in the whole parsed corpus. The span is the tell. SCOPED TO THE PER-WORD QUESTION, and this is the durable half. *Is this word a conjunction rather than an initial* is a property of the word and its spelling: an unread token has no ambiguity the parse would have resolved differently, so the vocabulary gives the answer the parser would have given, v1's initial carve-out included — which is why `_render._INITIAL` came back and why test_regex_sync's two-copy assertion matters MORE now, the fallback being right only while it answers as `_vocab` would. *Is this particle acting as a particle here* is a property of the whole PART, and the two views hold different amounts of it — which is why the scope differs per VIEW and not per question, a distinction the first draft of this bullet got wrong. Case repair walks one word of one token at a time and never sees the part, so it cannot answer it and does not ask: the fallback there is the word question alone. `initials()` holds every token of the role at once, so it can and does (next bullet). Measured cost of the wide reading IN CASE REPAIR, so nobody re-derives it: `replace(family='de la')` gives `de la` today and a parse gives `De La`; widening case repair's fallback would make them agree and would thereby REVERSE the replace/revise boundary recorded two bullets above, with `revise()` as its documented crossing. Do not widen THAT one without reopening the decision; `initials()` is a different view with a different reach and reverses nothing. Verified after the fix: `replace(family='velasquez y garcia').capitalized(force=True)` is `Velasquez y Garcia`, `.initials()` is `v. g.`, `replace(family='de la').capitalized(force=True)` is `de la`, and the parsed and `revise()` paths are unchanged. +- 2026-08-29 (#458 review) — THE FALLBACK FOR TEXT THE PARSE NEVER SAW, and why it is scoped to one of the two questions `_cap_word` asks. Reading the tag and nothing else regressed every ASSIGNED field holding a conjunction, because a value `replace()` splices in is never classified and so carries no tag to read. Measured on the released 1.4.0 and 2.1.0 wheels, all agreeing and all broken by the tag-only reading: `h.last = 'velasquez y garcia'` then `capitalize(force=True)` gives `John Velasquez y Garcia`, `h.middle = 'e.'` gives `John E. Smith`, `h.last = 'smith-y'` gives `John Smith-y`, `h.first = 'y'` gives `y Smith`. That is the canonical Spanish surname and the shape `y` is in the vocabulary FOR, so an Accepted clause was not available: it had to be fixed. RULE: a synthetic token — `span is None`, which `ParsedName.replace()` builds and every parsed token has — was never read, so there is no decision to honor and the view falls back to the vocabulary. Note what the discriminator is NOT: an ordinary parsed name word carries no tags either (`velasquez` parses to `tags=[]`), so "untagged" would have swept in the whole parsed corpus. The span is the tell. SCOPED TO THE PER-WORD QUESTION, and this is the durable half. *Is this word a conjunction rather than an initial* is a property of the word and its spelling: an unread token has no ambiguity the parse would have resolved differently, so the vocabulary gives the answer the parser would have given, v1's initial carve-out included — which is why `_render._INITIAL` came back and why test_regex_sync's two-copy assertion matters MORE now, the fallback being right only while it answers as `_vocab` would. *Is this particle acting as a particle here* is a property of the whole PART, and the two views hold different amounts of it — which is why the scope differs per VIEW and not per question, a distinction the first draft of this bullet got wrong. Case repair walks one word of one token at a time and never sees the part, so it cannot answer it and does not ask: the fallback there is the word question alone. `initials()` holds every token of the role at once and so COULD answer it, but is handed no vocabulary to answer it from, which is what the next bullet turns on. Measured cost of the wide reading IN CASE REPAIR, so nobody re-derives it: `replace(family='de la')` gives `de la` today and a parse gives `De La`; widening case repair's fallback would make them agree and would thereby REVERSE the replace/revise boundary recorded two bullets above, with `revise()` as its documented crossing. Do not widen it without reopening the decision. Verified after the fix: `replace(family='velasquez y garcia').capitalized(force=True)` is `Velasquez y Garcia`, `replace(family='de la').capitalized(force=True)` is `de la`, and the parsed and `revise()` paths are unchanged. -- 2026-08-29 (#458 review) — the SAME FALLBACK in `initials()`, fixing a defect older than #458 and only found beside it. On master `replace(family='velasquez y garcia').initials()` gave `v. y. g.` where the parse gives `v. g.`, so the two views were inconsistently wrong before the #458 commit and consistently wrong after it; both now ask the same question of the same token. The v2 core is the whole reach: `HumanName.initials_list()` computes from the field STRINGS with its own lexicon (`_facade._initials_lists`, v1 parity) and already gave `['j', 'v', 'g']` at 1.4.0, 2.1.0 and on this branch. `initials()` takes no lexicon, so the fallback reads `Lexicon.default()` — the same default-config fallback `ParsedName.capitalized()` documents for an omitted argument — and only where a spanless token is present, so a parsed name pays a scan and no lookup. BOTH of R3's questions are answered there, not just the conjunction one — reviewed and widened on Derek's ruling, the narrow scope having been argued from case repair's limits and wrongly carried across. `replace(family='de la vega').initials()` gave `j. d. l. v.` where 1.4.0, 2.1.0, the facade and the parse of the same name all give `j. v.`, and `van der berg` gave `j. v. d. b.` for `j. b.`. The all-particle test is `_types._remarked`'s own — every word of the part carries `particle` — with the vocabulary standing in for the tags a spliced token never got, so the fallback keeps answering as the pipeline would; that also makes it a provable no-op on a role the parse classified whole, since there the two agree by construction. Note `_remarked` is not the bug and needs no change: it recomputes the mark from TAGS after every edit, which is right for a record of what the parse decided and simply silent about text nobody classified. SCOPED PER ROLE, not per name: a role the parse classified whole is decided by its tags however the other roles were built. Per-NAME scoping passes every other test in the suite — the recomputation agreeing with the mark wherever the parse set one — so it is pinned by a hand-built name in tests/v2/test_render.py; a role holding parsed and spliced tokens at once is not reachable through `replace()`, which replaces a role whole (measured), and each token there answers with its best evidence. Verified against the facade and the parse on six spliced families, all three agreeing after the change and the core disagreeing on two of them before it. +- 2026-08-29 (#458 review, then the review of PR #463) — the SAME FALLBACK in `initials()`: WRITTEN, MEASURED AND DROPPED before merge. The defect it aimed at is real and older than #458 — `replace(family='velasquez y garcia').initials()` gives `j. v. y. g.` where the parse, the facade and 1.4.0 all give `j. v. g.`, and `replace(family='de la vega')` gives `j. d. l. v.` for `j. v.` — and it is unfixed again. What killed the fix is the parameter list: `initials()` takes no lexicon, so the fallback had to reach for `Lexicon.default()` and GUESS. `capitalized(lexicon=...)` is handed the caller's vocabulary and so guesses nothing, which is why its fallback stays. Measured cost of the guess, and the reason a merely-imperfect fallback was not good enough: under `Parser(lexicon=Lexicon.default().add(particles={'y'}))`, `parse('Juan de y').initials()` is `J. d. y.` and `parse('Juan Perez').replace(family='de y').initials()` was `J. d. y.` before the fallback and `J.` after it — an entire field's initials gone, because the default vocabulary reads `de y` as particles-plus-conjunction where the caller's reads it as an all-particle part. Guessing the wrong vocabulary for a name parsed under a custom one is worse than answering from tags alone, and `initials()` cannot be told better. The missing crossing is the whole of it: `Parser` carries `capitalized`, `matches` and `revise` as its custom-lexicon crossings and no `initials`, so there is no supported way to hand this view the parse's vocabulary. An issue is filed for that; when it lands, the fallback becomes answerable rather than a guess and this decision is worth reopening. ACCEPTED until then, and stated so a reader does not re-derive it as a bug: `replace(family='de la vega').initials()` is `j. d. l. v.` where the facade and a parse both give `j. v.`. The reach is the v2 core only — `HumanName.initials_list()` computes from the field STRINGS with its own lexicon (`_facade._initials_lists`, v1 parity) and gives `['j', 'v', 'g']` at 1.4.0, 2.1.0 and here — which is also why the facade cannot be used as evidence that the core is fine. Two things learned in the attempt are worth keeping though the code is gone. `_types._remarked` is not the bug and needs no change: it recomputes the mark from TAGS after every edit, which is right for a record of what the parse decided and simply silent about text nobody classified. And any future fallback must be scoped per ROLE, not per name — a role the parse classified whole is decided by its tags however the other roles were built — a distinction that passes every other test in the suite and needs a hand-built name to witness. - 2026-08-29 — WHY THE BOUNDARY WENT UNNOTICED UNTIL #407, which is where a future reader should look for it. For an ALL-PARTICLE part the other three tag-driven views give the same answer through `replace()` and `revise()` alike: measured over `de la`, `van der`, `do`, `de` and `van de la`, all five agree on `family_particles=''`, on a `family_base` holding the whole part, and on initials from every word. They converge because an UNTAGGED part and a MARKED all-particle part reach the same place by different routes — untagged, no word is recognized as a particle; marked, none is ACTING as one — and all three views only ask which words are particles. Case repair is the one view that asks a second question, since it must also decide whether to lowercase, so it is where the two routes first come apart. The mirror case confirms the reading: on a MIXED part the convergence is the other way round — `de la vega` and `van der berg` diverge in all three views between `replace()` and `revise()` (`replace()` reports particles `''` and base `'de la vega'` where `revise()` reports `'de la'` and `'vega'`) and AGREE on case repair, R4's all-particle clause not reaching them. So before #407 the distinction was invisible on exactly the shape the clause is about, and visible only on shapes the clause does not govern. diff --git a/docs/design/rules.md b/docs/design/rules.md index 21f572c4..c58f8260 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -1110,17 +1110,19 @@ R3. Rationale: initials abbreviate the person's name words; titles, rather than nothing: they are the base (R2), so they initial. "Juan van der" → initials="J. v. d." "Juan de y" → initials="J." - Accepted: this rule reads a part the parser read, and where a - field was set as raw text after the parse it reads that text as - the parser would have — the words of the part are all in hand - here, so the every-word-is-particle-vocabulary test is answerable - without them. A family set to "de la vega" therefore initials just - its "V.", and one set to "de la" initials both its words, matching - the same names parsed. Stated without an example line because - every line here names an input string, and this shape needs a - field edited after the parse. Case repair is the view that cannot - do this, and R4 says why. - history: decisions.md#R2 · interacts: R2 · implemented: nameparser/_render.py, nameparser/_facade.py + Accepted: this rule reads a part the parser read. A field set as + raw text after the parse carries no reading, and this view is + handed no vocabulary to supply one — it takes a format spec and + two separators and nothing else — so every word of such a field + initials, particles and conjunctions alike: a family set that way + to "de la vega" gives "j. d. l. v." where parsing the same + name gives "j. v.". Case repair IS handed a vocabulary, so it falls + back for the one question a word can answer on its own, and R4 + says which. Revising the field through the parser classifies it + and matches the parse in this view. Stated without an example + line because every line here names an input string, and this + shape needs a field edited after the parse. + history: decisions.md#R2 · interacts: R2, R4 · implemented: nameparser/_render.py, nameparser/_facade.py R4. Rationale: case repair is a display concern, applied only on request and never destructively. @@ -1169,9 +1171,11 @@ R4. Rationale: case repair is a display concern, applied only on whole part, and repair reads one word at a time, so that half falls through to particle treatment and the "de la" boundary above stands. Initials are the contrast worth knowing, and R3 states it: - that view holds every word of the part at once, so it answers the - part question for a spliced field too. revise() classifies the - value and crosses both. + that view is handed no vocabulary at all, so it falls back on + neither question and a spliced field's every word initials. + revise() classifies the value and crosses the part question; the + word question it does not, a middle revised to "e-f" repairing + to "e-F" where the parsed name gives "E-F". history: decisions.md#R4 · interacts: R2, R3, R5 · implemented: nameparser/_render.py R5. Rationale: mixed case is evidence that the writer cased the name diff --git a/docs/release_log.rst b/docs/release_log.rst index 4fa78128..49dd11d6 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -69,8 +69,6 @@ Release Log - Change case repair to read the parser's own ``conjunction`` tag instead of re-deciding, from the word's spelling, whether a word is a conjunction or an initial. The parse answers that question already -- ``"Scott E. Werner"`` reads ``E.`` as an initial rather than the Italian conjunction -- and the other views honor the answer; case repair asked again, with a shape test applied to each word of a token's text rather than to the token. Two spellings of one name disagreed because of it: ``"juan e-f smith"`` capitalized to ``Juan e-F Smith`` while ``"JUAN E-F SMITH"`` gave ``Juan E-F Smith``; both give ``Juan E-F Smith`` now, ``e-f`` being a middle name and no conjunction of the parse's reading. A conjunction written as a word of its own is untouched, and so is the one-letter carve-out where it applies -- ``"juan y garcia"`` still repairs to ``Juan y Garcia``, ``"JUAN Y GARCIA"`` still to ``Juan Y Garcia``. A field assigned after the parse is unaffected: its text was never classified, so there is no reading to honor and repair asks the vocabulary, which is what every earlier version did everywhere -- ``h.last = "velasquez y garcia"`` still repairs to ``Velasquez y Garcia`` and ``h.middle = "e."`` to ``E.``. Case repair is not one of the seven role fields the differential harness compares, so no gate run can see this change either way and none of its counts move; measured directly instead, no name of the 1094-name differential corpus moves under ``capitalized()`` or ``capitalized(force=True)``, its uppercased and lowercased spellings included (closes #458) - - Fix ``ParsedName.initials()`` reading a field assigned after the parse as though every word in it were a name word, so particles and conjunctions inside one contributed initials that the same words parsed do not. ``parse("john smith").replace(family="velasquez y garcia").initials()`` gave ``j. v. y. g.`` and ``replace(family="de la vega")`` gave ``j. d. l. v.``, where parsing those names gives ``j. v. g.`` and ``j. v.``. Text spliced into a field is never classified, so the view has no reading to honor; it now asks the vocabulary, which answers as the parser would -- an assigned middle initial stays an initial rather than becoming the Italian conjunction (``replace(middle="E.")`` still gives ``j. E. s.``), an uppercase ``Y`` in an assigned surname still contributes, and a field that is nothing but particle words still initials all of them, since none of them is doing a particle's work there (``replace(family="de la")`` gives ``j. d. l.``, as parsing it does). This is the 2.0 API only: ``HumanName.initials()`` and ``initials_list()`` compute from the field strings with their own vocabulary and already gave every one of these answers. Shipped since 2.0.0, and found while reviewing the case-repair change above (#458) - - Fix a tussenvoegsel attached to the family name after a comma deciding a genuinely uncertain reading and reporting nothing. ``"Van Johnson"`` reports a ``PARTICLE_OR_GIVEN`` ambiguity -- ``Van`` is a Dutch particle and a Vietnamese given name, and the parser has to pick one -- while ``"Nguyen, Thi Van"`` picked the same word the same way, silently, and lost the given name doing it. The attachment now reports the fork it decides, in the kind that names the reading it declined. A particle that could be an ordinary name reports ``PARTICLE_OR_GIVEN``: ``"Nguyen, Thi Van"``, ``"Berg, Jan van der"`` and ``"Vega, Juan de la"`` each gain one, the ``detail`` naming the ambiguous word. A particle the parser had already read as a post-nominal reports ``SUFFIX_OR_NAME`` instead, because the credential reading is what the attachment overrode: ``"Berg, Jan vd"`` gains one, ``vd`` being read as *van der* rather than as the Volunteer Decoration. Which kind you get follows the reading that was overridden rather than the word's vocabulary, so ``"Berg, Jan do"`` reports ``PARTICLE_OR_GIVEN`` even though ``do`` is a postnominal too -- it was already being read as a name word, so no credential reading was overridden. A particle where nothing was overridden reports nothing at all: ``"Jong, Piet de"`` and ``"Jong, Anke de"`` are unchanged, ``de`` being no name in any reading and no postnominal either, and so is ``"Berg, Jan de vd"``, whose run was read as name words whole. Worth knowing before you filter on this: ``"Beethoven, Ludwig van"`` -- the textbook Dutch listing, read exactly right -- now carries a report too. It is the same string shape over the same vocabulary as ``"Nguyen, Thi Van"``, and nothing in the input separates them, so a report on one is a report on both. This adds the report and nothing else: every field these names parse to is exactly what the tussenvoegsel fix below already gave them, and ``ambiguities`` is the only value that grows. Seven differential corpus names gain a kind against the 2.0.0 and 2.1.0 baselines, seven of the nine the tussenvoegsel fix below already moved -- five of ``fix(#379)``'s seven and both of ``fix(#380)``'s two, now carrying ``_ambiguities`` in their diff as well; the two left out are the pair named unchanged above, ``Jong, Piet de`` and ``Jong, Anke de`` -- and none against 1.4.0, which had no ambiguity reporting at all (closes #405) - Fix a tussenvoegsel after a family comma being parsed as a middle name. Dutch and Belgian alphabetized listings move the particle behind the given name -- ``"Beethoven, Ludwig van"`` is how ``"Ludwig van Beethoven"`` is filed -- and the trailing particle run was read as a middle name rather than as part of the surname: ``"Beethoven, Ludwig van"`` gave middle ``van``, last ``Beethoven``, and ``"Berg, Jan van der"`` gave middle ``van der``. The run now attaches to the family the comma has already named and renders before it, so those read family ``van Beethoven`` and ``van der Berg`` with the given name unchanged. The derived views move with the parse, so ``family_particles`` is ``van`` and ``family_base`` is ``Beethoven`` where they were empty and ``Beethoven`` before. `#130 `_ asked for the split and got it in 1.3.0 as ``last_base``/``last_prefixes``; 2.0 renamed them ``family_base``/``family_particles``. What was wrong until now was the values they reported for this listing. Both halves of the particle vocabulary attach -- never-given ``de`` and may-be-given ``van`` alike -- because after a comma the family is already named and the particle has no other role to take. Two guards bound it. A name whose only given word is the particle keeps it, so ``"Nguyen, Van"`` still reads given ``Van``: the attachment needs a given word to spare. And where the word is BOTH particle and suffix vocabulary the attachment outranks the post-nominal reading, so ``"Berg, Jan vd"`` reads family ``vd Berg`` where 1.4.0 and 2.1 alike gave suffix ``vd`` -- a trailing abbreviation after a family comma is the tussenvoegsel far more often than the decoration it collides with, and the same shape sweeps in ``mc``, which 2.1 also read as a suffix. ``do`` is in ``SUFFIX_ACRONYMS_AMBIGUOUS`` and 2.1 already read a trailing one as a name word, so it attaches by the plain rule rather than by the override (closes #379, closes #380). Names without the comma are untouched: ``"Ludwig van Beethoven"`` already read family ``van Beethoven`` and is byte-identical. One of the 751 differential corpus names moves, ``"Vega, Juan de la"``, at the 1.4.0, 2.0.0 and 2.1.0 baselines alike diff --git a/docs/usage.rst b/docs/usage.rst index a831f74f..c09ca979 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -891,13 +891,19 @@ text is not there. The views that read those tags degrade: the parser no longer knows ``de la`` are particles, so ``family_particles`` empties and ``family_base`` takes the whole field. -A view that is handed a vocabulary can fall back to it, since a token -the parse never saw carries no decision to honor — -:meth:`~nameparser.ParsedName.initials` and -:meth:`~nameparser.ParsedName.capitalized` do that, and answer as a -parse would. ``family_particles`` and ``family_base`` are properties -on the parsed name, which holds no vocabulary of its own, so they have -nothing to fall back to. +A token the parse never saw carries no decision to honor, so a view +that is *handed* a vocabulary can fall back to it — +:meth:`~nameparser.ParsedName.capitalized` is the one that is, and it +falls back for one question only: whether a word is a conjunction or +an initial, which a word answers on its own. Whether a particle is +acting as a particle is a fact about the whole part, and there is no +reading on any word of a spliced field to derive it from, so a family +set to ``de la`` stays lowercase where the same words parsed are +repaired to ``De La``. :meth:`~nameparser.ParsedName.initials` takes +no vocabulary at all, so it falls back on neither question and every +word of a spliced field contributes an initial. ``family_particles`` +and ``family_base`` are properties on the parsed name, which holds no +vocabulary of its own either. .. doctest:: @@ -909,7 +915,9 @@ nothing to fall back to. >>> replaced.family_base 'de la Vega Smith' >>> replaced.initials() - 'J. V. S.' + 'J. d. l. V. S.' + >>> name.replace(family="de la").capitalized(force=True).family + 'de la' :meth:`Parser.revise() ` is the same operation with each value classified by the parser's vocabulary, so diff --git a/nameparser/_render.py b/nameparser/_render.py index 1f4c9a82..953585ad 100644 --- a/nameparser/_render.py +++ b/nameparser/_render.py @@ -124,93 +124,31 @@ def initials(name: ParsedName, spec: str, delimiter: str, separator: str) -> str within a group. Tokens tagged particle/conjunction contribute no initial in middle/family (given-name tokens always contribute), and the unjoined mark readmits the words of an all-particle part - whichever of those tags they carry; tags come from the - pipeline. A token with no span was never classified -- replace() - splices one in -- so its role's words are read from the default - vocabulary instead, all-particle test included. Valid spec keys: - given, middle, family.""" + whichever of those tags they carry; tags come from the pipeline -- + hand-built untagged tokens all contribute, and so do the words of + a field spliced in by replace(), which the parse never read. + This view takes NO lexicon, so it has none to fall back to for + that text: `replace(family='de la vega')` initials all four words + where the same name parsed gives 'j. v.' (rules.md#R3's Accepted + clause, and decisions.md under R3 for why the fallback was tried + and dropped). Valid spec keys: given, middle, family.""" if not isinstance(delimiter, str): raise TypeError(f"delimiter must be a str, got {delimiter!r}") if not isinstance(separator, str): raise TypeError(f"separator must be a str, got {separator!r}") - # Only pay for the vocabulary where there is unclassified text to - # ask about; Lexicon.default() is cached, the scan is not. - unparsed_lex = (Lexicon.default() - if any(t.span is None for t in name.tokens) else None) values: dict[str, str] = {} for key in _INITIALS_KEYS: role = Role(key) tokens = name.tokens_for(role) if role is not Role.GIVEN: - # per ROLE, not per name: a role the parse classified whole - # is decided by its tags however the other roles were built - lex = (unparsed_lex if unparsed_lex is not None - and any(t.span is None for t in tokens) else None) - unjoined = lex is not None and _all_particles(tokens, lex) tokens = tuple(t for t in tokens - if _initials_from(t, lex, unjoined)) + if not (_SKIP_TAGS & t.tags) + or UNJOINED_TAG in t.tags) values[key] = separator.join( t.text[0] + delimiter for t in tokens) return _format_spec(spec, values, "initials", _INITIALS_KEYS) -def _is_particle_word(token: Token, lex: Lexicon) -> bool: - """Particle vocabulary, by the tag where the parse left one and by - the vocabulary where it did not.""" - return ("particle" in token.tags - or (token.span is None - and _normalize(token.text) in lex.particles)) - - -def _all_particles(tokens: tuple[Token, ...], lex: Lexicon) -> bool: - """_types._remarked's own test -- every word of the part carries - "particle" -- with the vocabulary standing in for the tags a - spliced token never got. - - _remarked recomputes the unjoined mark from TAGS after every edit, - so a spliced part is never marked however particle-shaped it is. - That is right for the mark (which says what the parse decided) and - wrong for the view, which then reads an all-particle part as if it - were something else. Asking the vocabulary here answers as the - parse would have. A role holding parsed and spliced tokens at once - -- not reachable through replace(), which replaces a role whole, - but constructible by hand -- gets each token's best evidence, which - is again what _remarked would have computed. - """ - return bool(tokens) and all(_is_particle_word(t, lex) for t in tokens) - - -def _initials_from(token: Token, lex: Lexicon | None, - unjoined: bool) -> bool: - """Whether a middle/family token contributes an initial (R3). - - `lex` is None for a role the parse classified whole, where the tags - ARE the answer; it is the fallback vocabulary for a role holding - text the parse never saw, where the same two questions are answered - from tags where there are any and from the vocabulary where there - are none. The two paths agree wherever both apply: `unjoined` is - _remarked's own test, so a fully parsed role recomputes the mark it - already carries. - """ - if lex is None: - # the mark readmits a skip-tagged token whichever of the two - # tags it carries: inside an all-particle part none of the - # words is doing the work its tag names -- a conjunction with - # nothing to join no more than a particle with nothing to join - # (R2) -- so they are the part's name words and initial. - if _SKIP_TAGS & token.tags: - return UNJOINED_TAG in token.tags - return True - # the same two questions, answered from the vocabulary where the - # token carries no tags, and reaching the same answer: both skip - # words are readmitted by the mark and by nothing else - if ("conjunction" in token.tags - or (token.span is None and _reads_as_conjunction(token.text, lex)) - or _is_particle_word(token, lex)): - return unjoined - return True - - def _cap_word(word: str, role: Role, tags: frozenset[str], lex: Lexicon, *, parsed: bool) -> str: # v1 cap_word order: particle/conjunction rule first, then the diff --git a/tests/v2/test_render.py b/tests/v2/test_render.py index f3af5c22..3a131834 100644 --- a/tests/v2/test_render.py +++ b/tests/v2/test_render.py @@ -1,6 +1,6 @@ import pytest -from nameparser import Parser, parse +from nameparser import HumanName, Parser, parse from nameparser._lexicon import Lexicon from nameparser._render import _collapse, render from nameparser._types import (UNJOINED_TAG, Ambiguity, AmbiguityKind, @@ -341,24 +341,14 @@ def test_capitalized_lowers_the_words_the_parse_tagged_conjunction() -> None: assert hyphenated.capitalized(force=True).middle == "E-F" -def test_views_fall_back_to_the_vocabulary_for_text_never_parsed() -> None: - """A token with no span was never classified, so there is no - decision to honor and the views ask the vocabulary instead -- - getting the answer the parser would have given, v1's initial - carve-out included. Both views, on one parse: `replace()` splices - raw text into a field, and reading it two ways is what #458's - review found. - - The two views ask it of different scopes, and that is the whole of - the asymmetry. Case repair walks one word at a time and never sees - the part, so it answers the per-WORD question (conjunction or - initial?) and leaves the per-PART one (is this particle acting as - a particle?) to fall through to particle treatment -- widening it - there would reverse rules.md#R4's Accepted boundary, whose crossing - is revise(), which the `de la` capitalization assertions pin. - initials() DOES hold the part -- every token of the role is in hand - -- so it answers both, by _types._remarked's own test with the - vocabulary standing in for the tags a spliced token never got. +def test_case_repair_falls_back_for_text_the_parse_never_read() -> None: + """A token the parse never read carries no decision to honor, so + case repair -- which is handed a lexicon -- asks the vocabulary + instead, getting the answer the parser would have given, v1's + initial carve-out included. + + ONE view falls back. `initials()` takes no lexicon, so it has none + to ask; the sibling test below pins what that costs. """ p = Parser() base = p.parse("john smith") @@ -366,72 +356,52 @@ def test_views_fall_back_to_the_vocabulary_for_text_never_parsed() -> None: spliced = base.replace(family="velasquez y garcia") assert [t.span for t in spliced.tokens[1:]] == [None, None, None] assert spliced.capitalized(force=True).family == "Velasquez y Garcia" - assert spliced.initials() == "j. v. g." # the carve-out rides along: an assigned middle initial is an - # initial, not the Italian conjunction, in either view + # initial, not the Italian conjunction assert base.replace(middle="e.").capitalized(force=True).middle == "E." - assert base.replace(middle="E.").initials() == "j. E. s." - assert base.replace(family="velasquez Y garcia").initials() == "j. v. Y. g." - - # the per-part question, which initials() can answer: a spliced - # family with a name word in it has its particles skipped, and one - # that is nothing but particles has them all contribute (R2/R3). - # Each is what the facade and the parse both give. - assert base.replace(family="de la vega").initials() == "j. v." - assert base.replace(family="van der berg").initials() == "j. b." - assert base.replace(family="de la").initials() == "j. d. l." - assert p.parse("john de la").initials() == "j. d. l." - assert base.replace(family="smith").initials() == "j. s." - - # unchanged: the boundary a spliced particle keeps in CASE REPAIR, - # which cannot see the part + + # the per-part particle question is NOT asked: re-deriving it needs + # a reading on every word of the part and these words have none, so + # it falls through to plain particle treatment assert base.replace(family="de la").capitalized(force=True).family == "de la" assert (base.replace(family="de la vega") .capitalized(force=True).family == "de la Vega") - # ... and revise(), which classifies, still crosses it + # ... and revise(), which classifies, crosses it assert p.revise(base, family="de la").capitalized(force=True).family == "De La" assert (p.revise(base, family="velasquez y garcia") .capitalized(force=True).family == "Velasquez y Garcia") - # unchanged: a token the parse DID see is decided by its tags, so - # a hyphenated word it read as one ordinary name word stays one + # a token the parse DID see is decided by its tags, so a hyphenated + # word it read as one ordinary name word stays one assert p.parse("juan e-f smith").capitalized(force=True).middle == "E-F" - # A role holding both is not reachable through replace(), which - # replaces a role whole (measured), but is constructible: each - # token answers with its best evidence, which is what _remarked - # would have computed had the spliced half been classified. Here - # the parsed 'de' is particle-TAGGED and the spliced 'la' is only - # particle vocabulary, so the part is all particles and both - # contribute; swap in a name word and neither does. - mixed = _pn("john de la", [ - Token("john", Span(0, 4), Role.GIVEN), - Token("de", Span(5, 7), Role.FAMILY, frozenset({"particle"})), - Token("la", None, Role.FAMILY), - ]) - assert mixed.initials() == "j. d. l." - with_name_word = _pn("john de la", [ - Token("john", Span(0, 4), Role.GIVEN), - Token("de", Span(5, 7), Role.FAMILY, frozenset({"particle"})), - Token("vega", None, Role.FAMILY), - ]) - assert with_name_word.initials() == "j. v." - - # Scoped per ROLE, not per name: a role the parse classified whole - # keeps its tags as the answer however the other roles were built. - # The family here is all particle vocabulary and carries no mark, - # which is the parse saying those words are doing a particle's work - # -- a spliced middle beside it does not reopen that. Hand-built - # because every edit path recomputes the mark (_types._remarked) - # and would agree with the vocabulary by construction; per-NAME - # scoping passes every other test in the suite and fails here. - other_role_spliced = _pn("john van der", [ - Token("john", Span(0, 4), Role.GIVEN), - Token("q", None, Role.MIDDLE), - Token("van", Span(5, 8), Role.FAMILY, frozenset({"particle"})), - Token("der", Span(9, 12), Role.FAMILY, frozenset({"particle"})), - ]) - assert other_role_spliced.initials() == "j. q." + +def test_initials_has_no_lexicon_so_a_spliced_field_is_all_name_words()\ + -> None: + """The accepted cost of `initials()` taking no lexicon: a field + spliced in as raw text has no reading, and this view has nothing to + read one from, so every word of it initials. + + That disagrees with the facade and with the same name parsed, and + it is a 2.0-core defect rather than a decision -- see the issue + filed for giving `Parser` an `initials` crossing. A fallback to + `Lexicon.default()` was written and dropped: it guesses a + vocabulary, and under a caller's own the guess erases a whole + field (`Lexicon.default().add(particles={'y'})`, family 'de y'). + """ + p = Parser() + base = p.parse("john smith") + + assert base.replace(family="de la vega").initials() == "j. d. l. v." + assert p.parse("john de la vega").initials() == "j. v." + assert HumanName("john de la vega").initials() == "j. v." + + # the vocabulary a fallback would have had to guess, and the field + # it erased when it guessed wrong + y_lex = Lexicon.default().add(particles={"y"}) + py = Parser(lexicon=y_lex) + assert py.parse("Juan de y").initials() == "J. d. y." + assert py.parse("Juan Perez").replace(family="de y").initials() == "J. d. y." def test_capitalized_rebuilds_ambiguity_tokens() -> None: From 2a9cd70ca70f606e2c6f4b603ea3012a9dd982b7 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sat, 29 Aug 2026 23:31:52 -0700 Subject: [PATCH 08/14] fix(types,render,facade,docs): the mark, not the span, says a token was never read `span is None` means SYNTHETIC, which is a wider set than unclassified. `Parser.revise()` builds span-less tokens too, from a full sub-parse whose tags it keeps on purpose, and its docstring promises the tag-driven views "behave as if the text had been parsed" -- so keying case repair's fallback on the span overrode exactly the tags revise() exists to preserve. Measured against 82a7bd1: revise(name, middle="e-f").capitalized(force=True) base 'e-F', parse 'e-F' -- agreed span discriminator: 'e-F' where the parse now gives 'E-F' Parser(lexicon=Lexicon()): parse("john de la vega").initials() j. d. l. v. revise(family="de la vega").initials() base j. d. l. v., span discriminator j. v. UNCLASSIFIED_TAG replaces it, single-sourced in _types.py beside UNJOINED_TAG and FOLDED_TAG, and stamped by the two producers of genuinely unclassified text: ParsedName.replace(), and the facade's v1 pickle load, which rebuilds a name from the *_list strings and no tags. Without the second, a restored `juan ortega y gasset` would repair to `Juan Ortega Y Gasset` -- neither 1.4.0's answer nor the same object's before pickling. Pinned by a new test. A HAND-BUILT span-less token is not marked, so it takes the tag path like every other token in the library. That moves one reading against 2.1: an untagged token whose text is conjunction vocabulary now capitalizes. The particle conjunct is untouched -- it keys on lexicon membership, not on the mark -- so a hand-built `de la Vega` is unchanged. `_cap_word` reads the mark out of the tags it was already given, so the `parsed=` keyword goes away rather than changing meaning. Co-Authored-By: Claude Opus 5 --- docs/design/decisions.md | 2 +- docs/design/rules.md | 9 +++-- docs/release_log.rst | 2 +- nameparser/_facade.py | 18 +++++++-- nameparser/_render.py | 78 +++++++++++++++++++++--------------- nameparser/_types.py | 31 +++++++++++--- tests/test_capitalization.py | 21 ++++++++++ tests/v2/test_render.py | 74 ++++++++++++++++++++++++++++++---- 8 files changed, 179 insertions(+), 56 deletions(-) diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 564ff2b6..09da71da 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -737,7 +737,7 @@ Declined: - 2026-08-29 #458 — DONE: case repair's conjunction test reads the `conjunction` TAG instead of re-deriving the word's class at render time, so a view honors the decision the parse recorded (mechanisms.md#VOCAB-TAGS) rather than taking it again. The two copies of the question were not the same question, which is the reason this was worth doing over a corpus that cannot see it: `_classify.py` asks `is_initial(token.text)` — the shape test ANDed with a repertoire test since #320 — while `_render.py` asked a bare pattern, shape only, over each WORD of a token's text rather than the token. It also retires the hand-sync obligation the module carried in a comment ("keep in sync with `nameparser/_pipeline/_vocab.py` by hand"): a tag read has nothing to keep in sync, and `_render._INITIAL` is deleted, its last reader gone. MEASURED, re-derived on this branch rather than taken from the issue: 0 of the 1094 corpus names move under `capitalized()` or `capitalized(force=True)`, and 0 move with each name also re-run uppercased and lowercased (6565 name/variant/lexicon rows, the second lexicon being `Lexicon.default().add(particles={'y'})`); the R5 force-versus-case counts recorded above hold at 62/16 through the facade and 63 through the core. Movement EXISTS outside that population, and the issue's "no output moves" is a claim about the corpus only: the old predicate re-decided per word, so `juan e-f smith` capitalized to `Juan e-F Smith` — the Italian conjunction inside a hyphenated middle name — and now gives `Juan E-F Smith`, agreeing with the uppercase spelling it always gave. An earlier draft of this change moved untagged tokens the same way — a family `replace()` splices in as `de y` repairing to `de Y` — and that was a REGRESSION, caught in review before it shipped and fixed in the bullet below; read this bullet with that one. NOT DONE, and the parenthetical in the boundary bullet above should be read as still open: the PARTICLE conjunct still keys on lexicon membership, so `replace()` does not agree there. Making it read the `particle` tag is a second change with its own blast radius — it moves a boundary rules.md#R4 states in prose — and #458 as filed does not ask for it. -- 2026-08-29 (#458 review) — THE FALLBACK FOR TEXT THE PARSE NEVER SAW, and why it is scoped to one of the two questions `_cap_word` asks. Reading the tag and nothing else regressed every ASSIGNED field holding a conjunction, because a value `replace()` splices in is never classified and so carries no tag to read. Measured on the released 1.4.0 and 2.1.0 wheels, all agreeing and all broken by the tag-only reading: `h.last = 'velasquez y garcia'` then `capitalize(force=True)` gives `John Velasquez y Garcia`, `h.middle = 'e.'` gives `John E. Smith`, `h.last = 'smith-y'` gives `John Smith-y`, `h.first = 'y'` gives `y Smith`. That is the canonical Spanish surname and the shape `y` is in the vocabulary FOR, so an Accepted clause was not available: it had to be fixed. RULE: a synthetic token — `span is None`, which `ParsedName.replace()` builds and every parsed token has — was never read, so there is no decision to honor and the view falls back to the vocabulary. Note what the discriminator is NOT: an ordinary parsed name word carries no tags either (`velasquez` parses to `tags=[]`), so "untagged" would have swept in the whole parsed corpus. The span is the tell. SCOPED TO THE PER-WORD QUESTION, and this is the durable half. *Is this word a conjunction rather than an initial* is a property of the word and its spelling: an unread token has no ambiguity the parse would have resolved differently, so the vocabulary gives the answer the parser would have given, v1's initial carve-out included — which is why `_render._INITIAL` came back and why test_regex_sync's two-copy assertion matters MORE now, the fallback being right only while it answers as `_vocab` would. *Is this particle acting as a particle here* is a property of the whole PART, and the two views hold different amounts of it — which is why the scope differs per VIEW and not per question, a distinction the first draft of this bullet got wrong. Case repair walks one word of one token at a time and never sees the part, so it cannot answer it and does not ask: the fallback there is the word question alone. `initials()` holds every token of the role at once and so COULD answer it, but is handed no vocabulary to answer it from, which is what the next bullet turns on. Measured cost of the wide reading IN CASE REPAIR, so nobody re-derives it: `replace(family='de la')` gives `de la` today and a parse gives `De La`; widening case repair's fallback would make them agree and would thereby REVERSE the replace/revise boundary recorded two bullets above, with `revise()` as its documented crossing. Do not widen it without reopening the decision. Verified after the fix: `replace(family='velasquez y garcia').capitalized(force=True)` is `Velasquez y Garcia`, `replace(family='de la').capitalized(force=True)` is `de la`, and the parsed and `revise()` paths are unchanged. +- 2026-08-29 (#458 review) — THE FALLBACK FOR TEXT THE PARSE NEVER SAW, and why it is scoped to one of the two questions `_cap_word` asks. Reading the tag and nothing else regressed every ASSIGNED field holding a conjunction, because a value `replace()` splices in is never classified and so carries no tag to read. Measured on the released 1.4.0 and 2.1.0 wheels, all agreeing and all broken by the tag-only reading: `h.last = 'velasquez y garcia'` then `capitalize(force=True)` gives `John Velasquez y Garcia`, `h.middle = 'e.'` gives `John E. Smith`, `h.last = 'smith-y'` gives `John Smith-y`, `h.first = 'y'` gives `y Smith`. That is the canonical Spanish surname and the shape `y` is in the vocabulary FOR, so an Accepted clause was not available: it had to be fixed. RULE: a token carrying `UNCLASSIFIED_TAG` was never read, so there is no decision to honor and the view falls back to the vocabulary. Note what the discriminator is NOT, in two directions, the second of which took a second review round to find. It is not UNTAGGEDNESS: an ordinary parsed name word carries no tags either (`velasquez` parses to `tags=[]`), so that would have swept in the whole parsed corpus. And it is not the absent SPAN, which is what the first fix used. Span-less means SYNTHETIC, a wider set than unclassified: `Parser.revise()` builds span-less tokens too, from a full sub-parse whose tags it keeps deliberately, and its docstring promises the tag-driven views "behave as if the text had been parsed" — so keying the fallback on the span overrode exactly the tags `revise()` exists to preserve. Measured on the branch that did it: `revise(name, middle='e-f').capitalized(force=True)` gave `e-F` where `parse('john e-f smith')` gave `E-F`, and under `Parser(lexicon=Lexicon())` a revised family `de la vega` initialed `j. v.` where the parse initialed `j. d. l. v.`. So the mark is stamped by the two producers of genuinely unclassified text — `ParsedName.replace()`, and the facade's v1 pickle load, which rebuilds a name from `*_list` strings and no tags — and a HAND-BUILT span-less token is not marked, so it takes the tag path like everything else in the library. That is the tag-driven semantics every other view already has, and it is a change from the span reading in exactly one place: a hand-built span-less token whose text is CONJUNCTION vocabulary now capitalizes rather than lowercasing (measured: a hand-built family `velasquez y garcia` repairs to `Velasquez Y Garcia` untagged, `Velasquez y Garcia` with either the `conjunction` tag or the mark). The PARTICLE conjunct is unaffected either way — it keys on lexicon membership rather than on the mark, which is the open item this bullet's neighbours record — so a hand-built `de la Vega` is unchanged. SCOPED TO THE PER-WORD QUESTION, and this is the durable half. *Is this word a conjunction rather than an initial* is a property of the word and its spelling: an unread token has no ambiguity the parse would have resolved differently, so the vocabulary gives the answer the parser would have given, v1's initial carve-out included — which is why `_render._INITIAL` came back and why test_regex_sync's two-copy assertion matters MORE now, the fallback being right only while it answers as `_vocab` would. *Is this particle acting as a particle here* is a property of the whole PART, and the two views hold different amounts of it — which is why the scope differs per VIEW and not per question, a distinction the first draft of this bullet got wrong. Case repair walks one word of one token at a time and never sees the part, so it cannot answer it and does not ask: the fallback there is the word question alone. `initials()` holds every token of the role at once and so COULD answer it, but is handed no vocabulary to answer it from, which is what the next bullet turns on. Measured cost of the wide reading IN CASE REPAIR, so nobody re-derives it: `replace(family='de la')` gives `de la` today and a parse gives `De La`; widening case repair's fallback would make them agree and would thereby REVERSE the replace/revise boundary recorded two bullets above, with `revise()` as its documented crossing. Do not widen it without reopening the decision. Verified after the fix: `replace(family='velasquez y garcia').capitalized(force=True)` is `Velasquez y Garcia`, `replace(family='de la').capitalized(force=True)` is `de la`, and the parsed and `revise()` paths are unchanged. - 2026-08-29 (#458 review, then the review of PR #463) — the SAME FALLBACK in `initials()`: WRITTEN, MEASURED AND DROPPED before merge. The defect it aimed at is real and older than #458 — `replace(family='velasquez y garcia').initials()` gives `j. v. y. g.` where the parse, the facade and 1.4.0 all give `j. v. g.`, and `replace(family='de la vega')` gives `j. d. l. v.` for `j. v.` — and it is unfixed again. What killed the fix is the parameter list: `initials()` takes no lexicon, so the fallback had to reach for `Lexicon.default()` and GUESS. `capitalized(lexicon=...)` is handed the caller's vocabulary and so guesses nothing, which is why its fallback stays. Measured cost of the guess, and the reason a merely-imperfect fallback was not good enough: under `Parser(lexicon=Lexicon.default().add(particles={'y'}))`, `parse('Juan de y').initials()` is `J. d. y.` and `parse('Juan Perez').replace(family='de y').initials()` was `J. d. y.` before the fallback and `J.` after it — an entire field's initials gone, because the default vocabulary reads `de y` as particles-plus-conjunction where the caller's reads it as an all-particle part. Guessing the wrong vocabulary for a name parsed under a custom one is worse than answering from tags alone, and `initials()` cannot be told better. The missing crossing is the whole of it: `Parser` carries `capitalized`, `matches` and `revise` as its custom-lexicon crossings and no `initials`, so there is no supported way to hand this view the parse's vocabulary. An issue is filed for that; when it lands, the fallback becomes answerable rather than a guess and this decision is worth reopening. ACCEPTED until then, and stated so a reader does not re-derive it as a bug: `replace(family='de la vega').initials()` is `j. d. l. v.` where the facade and a parse both give `j. v.`. The reach is the v2 core only — `HumanName.initials_list()` computes from the field STRINGS with its own lexicon (`_facade._initials_lists`, v1 parity) and gives `['j', 'v', 'g']` at 1.4.0, 2.1.0 and here — which is also why the facade cannot be used as evidence that the core is fine. Two things learned in the attempt are worth keeping though the code is gone. `_types._remarked` is not the bug and needs no change: it recomputes the mark from TAGS after every edit, which is right for a record of what the parse decided and simply silent about text nobody classified. And any future fallback must be scoped per ROLE, not per name — a role the parse classified whole is decided by its tags however the other roles were built — a distinction that passes every other test in the suite and needs a hand-built name to witness. diff --git a/docs/design/rules.md b/docs/design/rules.md index c58f8260..671c35c8 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -1119,7 +1119,7 @@ R3. Rationale: initials abbreviate the person's name words; titles, name gives "j. v.". Case repair IS handed a vocabulary, so it falls back for the one question a word can answer on its own, and R4 says which. Revising the field through the parser classifies it - and matches the parse in this view. Stated without an example + and matches the parse in both views. Stated without an example line because every line here names an input string, and this shape needs a field edited after the parse. history: decisions.md#R2 · interacts: R2, R4 · implemented: nameparser/_render.py, nameparser/_facade.py @@ -1173,9 +1173,10 @@ R4. Rationale: case repair is a display concern, applied only on stands. Initials are the contrast worth knowing, and R3 states it: that view is handed no vocabulary at all, so it falls back on neither question and a spliced field's every word initials. - revise() classifies the value and crosses the part question; the - word question it does not, a middle revised to "e-f" repairing - to "e-F" where the parsed name gives "E-F". + revise() classifies the + value and crosses both questions, in both views: a middle revised + to "e-f" repairs to "E-F" as the parsed name does, where splicing + the same text in gives "e-F". history: decisions.md#R4 · interacts: R2, R3, R5 · implemented: nameparser/_render.py R5. Rationale: mixed case is evidence that the writer cased the name diff --git a/docs/release_log.rst b/docs/release_log.rst index 49dd11d6..5dc3fa15 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -67,7 +67,7 @@ Release Log - Fix case repair lowercasing the words of a family name made only of particle words, where every other view already reads them as ordinary name words: ``HumanName("ANH DO").capitalize()`` gives ``Anh Do`` where it gave ``Anh do``, and ``"anh van do"`` gives ``Anh Van Do`` where it gave ``Anh van do``. A particle earns its name by joining forward to the word it modifies, so a part whose every word is particle vocabulary leaves none of them anything to join; the fix above already made those words anchor ``family_base`` and contribute initials, and case repair now agrees with them rather than reading the same word two ways. The test is the whole part, not a particle standing alone, which is why the two-word family in ``"anh van do"`` moves along with the one-word family in ``"ANH DO"`` -- the same Vietnamese surname, and a standing-alone test would have read it one way behind a given name and another way alone. This DIFFERS FROM 1.4.0 deliberately and does not restore it: 1.4.0 returned ``Anh do``, lowercasing on vocabulary membership alone. The accepted cost is that a degenerate family which is nothing but particles capitalizes too, so ``"juan van der"`` gives ``Juan Van Der`` where 1.4.0 gave ``Juan van der``. A conjunction is untouched by any of this, so ``"der, y van"`` gives ``y Van Der`` -- the family capitalizing while the conjunction keeps the lowercase it always had; and where the particles DO join a name word nothing changes, ``"juan de la vega"`` still giving ``Juan de la Vega``. Case repair is not one of the seven role fields the differential harness compares, so no gate run can see this change either way and none of its counts move; the ``rules.md#R4`` examples and the v1 capitalization tests are what pin it (closes #407) - - Change case repair to read the parser's own ``conjunction`` tag instead of re-deciding, from the word's spelling, whether a word is a conjunction or an initial. The parse answers that question already -- ``"Scott E. Werner"`` reads ``E.`` as an initial rather than the Italian conjunction -- and the other views honor the answer; case repair asked again, with a shape test applied to each word of a token's text rather than to the token. Two spellings of one name disagreed because of it: ``"juan e-f smith"`` capitalized to ``Juan e-F Smith`` while ``"JUAN E-F SMITH"`` gave ``Juan E-F Smith``; both give ``Juan E-F Smith`` now, ``e-f`` being a middle name and no conjunction of the parse's reading. A conjunction written as a word of its own is untouched, and so is the one-letter carve-out where it applies -- ``"juan y garcia"`` still repairs to ``Juan y Garcia``, ``"JUAN Y GARCIA"`` still to ``Juan Y Garcia``. A field assigned after the parse is unaffected: its text was never classified, so there is no reading to honor and repair asks the vocabulary, which is what every earlier version did everywhere -- ``h.last = "velasquez y garcia"`` still repairs to ``Velasquez y Garcia`` and ``h.middle = "e."`` to ``E.``. Case repair is not one of the seven role fields the differential harness compares, so no gate run can see this change either way and none of its counts move; measured directly instead, no name of the 1094-name differential corpus moves under ``capitalized()`` or ``capitalized(force=True)``, its uppercased and lowercased spellings included (closes #458) + - Change case repair to read the parser's own ``conjunction`` tag instead of re-deciding, from the word's spelling, whether a word is a conjunction or an initial. The parse answers that question already -- ``"Scott E. Werner"`` reads ``E.`` as an initial rather than the Italian conjunction -- and the other views honor the answer; case repair asked again, with a shape test applied to each word of a token's text rather than to the token. Two spellings of one name disagreed because of it: ``"juan e-f smith"`` capitalized to ``Juan e-F Smith`` while ``"JUAN E-F SMITH"`` gave ``Juan E-F Smith``; both give ``Juan E-F Smith`` now, ``e-f`` being a middle name and no conjunction of the parse's reading. A conjunction written as a word of its own is untouched, and so is the one-letter carve-out where it applies -- ``"juan y garcia"`` still repairs to ``Juan y Garcia``, ``"JUAN Y GARCIA"`` still to ``Juan Y Garcia``. A field assigned after the parse is unaffected: its text was never classified, so there is no reading to honor and repair asks the vocabulary, which is what every earlier version did everywhere -- ``h.last = "velasquez y garcia"`` still repairs to ``Velasquez y Garcia`` and ``h.middle = "e."`` to ``E.``. What decides which path a token takes is a mark the assignment leaves, not the absence of a span: a value revised through ``Parser.revise()`` is classified by a sub-parse and keeps its tags, so it repairs as the parse does. One reading does change for hand-built ``Token``\ s in the 2.0 API: an untagged token whose text is conjunction vocabulary is now an ordinary name word and capitalizes, where 2.1 lowercased it -- tags are what the views read, and a hand-built token that carries none is a token with nothing to declare. Case repair is not one of the seven role fields the differential harness compares, so no gate run can see this change either way and none of its counts move; measured directly instead, no name of the 1094-name differential corpus moves under ``capitalized()`` or ``capitalized(force=True)``, its uppercased and lowercased spellings included (closes #458) - Fix a tussenvoegsel attached to the family name after a comma deciding a genuinely uncertain reading and reporting nothing. ``"Van Johnson"`` reports a ``PARTICLE_OR_GIVEN`` ambiguity -- ``Van`` is a Dutch particle and a Vietnamese given name, and the parser has to pick one -- while ``"Nguyen, Thi Van"`` picked the same word the same way, silently, and lost the given name doing it. The attachment now reports the fork it decides, in the kind that names the reading it declined. A particle that could be an ordinary name reports ``PARTICLE_OR_GIVEN``: ``"Nguyen, Thi Van"``, ``"Berg, Jan van der"`` and ``"Vega, Juan de la"`` each gain one, the ``detail`` naming the ambiguous word. A particle the parser had already read as a post-nominal reports ``SUFFIX_OR_NAME`` instead, because the credential reading is what the attachment overrode: ``"Berg, Jan vd"`` gains one, ``vd`` being read as *van der* rather than as the Volunteer Decoration. Which kind you get follows the reading that was overridden rather than the word's vocabulary, so ``"Berg, Jan do"`` reports ``PARTICLE_OR_GIVEN`` even though ``do`` is a postnominal too -- it was already being read as a name word, so no credential reading was overridden. A particle where nothing was overridden reports nothing at all: ``"Jong, Piet de"`` and ``"Jong, Anke de"`` are unchanged, ``de`` being no name in any reading and no postnominal either, and so is ``"Berg, Jan de vd"``, whose run was read as name words whole. Worth knowing before you filter on this: ``"Beethoven, Ludwig van"`` -- the textbook Dutch listing, read exactly right -- now carries a report too. It is the same string shape over the same vocabulary as ``"Nguyen, Thi Van"``, and nothing in the input separates them, so a report on one is a report on both. This adds the report and nothing else: every field these names parse to is exactly what the tussenvoegsel fix below already gave them, and ``ambiguities`` is the only value that grows. Seven differential corpus names gain a kind against the 2.0.0 and 2.1.0 baselines, seven of the nine the tussenvoegsel fix below already moved -- five of ``fix(#379)``'s seven and both of ``fix(#380)``'s two, now carrying ``_ambiguities`` in their diff as well; the two left out are the pair named unchanged above, ``Jong, Piet de`` and ``Jong, Anke de`` -- and none against 1.4.0, which had no ambiguity reporting at all (closes #405) diff --git a/nameparser/_facade.py b/nameparser/_facade.py index 099e6a00..051d44e9 100644 --- a/nameparser/_facade.py +++ b/nameparser/_facade.py @@ -32,7 +32,8 @@ from nameparser._config_shim import CONSTANTS, Constants, _cached_parser from nameparser._lexicon import _normalize from nameparser._parser import Parser -from nameparser._types import FOLDED_TAG, ParsedName, Role, Token +from nameparser._types import (FOLDED_TAG, UNCLASSIFIED_TAG, ParsedName, + Role, Token) _V2_FIELD = {"first": "given", "last": "family"} # v1 name -> v2 name _V1_SPELLING = {v2: v1 for v1, v2 in _V2_FIELD.items()} @@ -746,8 +747,17 @@ def __setstate__(self, state: dict[str, Any]) -> None: f"nameparser" ) for position, word in enumerate(entry.split()): - tokens.append(Token( - word, None, role, - frozenset({"joined"}) if position else frozenset())) + # UNCLASSIFIED_TAG for the same reason replace() + # stamps it: a pickle carries the *_list STRINGS + # and no tags, so nothing here was read by a parse + # and case repair must ask the vocabulary rather + # than read an absent conjunction tag. Without it a + # restored "juan ortega y gasset" repairs to + # "Ortega Y Gasset", which is neither v1's answer + # nor the same name's unpickled one. + tags = {UNCLASSIFIED_TAG} + if position: + tags.add("joined") + tokens.append(Token(word, None, role, frozenset(tags))) self._parsed = ParsedName( original=str(state.get("original", "")), tokens=tuple(tokens)) diff --git a/nameparser/_render.py b/nameparser/_render.py index 953585ad..08a4748a 100644 --- a/nameparser/_render.py +++ b/nameparser/_render.py @@ -15,8 +15,8 @@ import re from nameparser._lexicon import Lexicon, _normalize -from nameparser._types import (UNJOINED_TAG, Ambiguity, ParsedName, Role, - Token) +from nameparser._types import (UNCLASSIFIED_TAG, UNJOINED_TAG, Ambiguity, + ParsedName, Role, Token) _SPACES = re.compile(r"\s+") _SPACE_BEFORE_COMMA = re.compile(r"\s+,") @@ -56,21 +56,22 @@ # Deliberately NOT composed with _vocab's repertoire test (#320): # layering forbids the import. The divergence is reachable only for a # caller-added CJK conjunction spliced into a field, since no shipped -# vocabulary carries one; there it costs nothing in case repair (CJK is -# caseless, so lower() and capitalize() return the same string) and -# would let such a word initial where a parsed one would not. +# vocabulary carries one, and it costs nothing there: CJK is caseless, +# so the carve-out's lower() and the fall-through's capitalize() return +# the same string, and case repair is now this pattern's only reader. _INITIAL = re.compile(r"^(\w\.|[A-Z])$") def _reads_as_conjunction(word: str, lex: Lexicon) -> bool: """v1's is_conjunction, asked only of text the parse never saw. - A token with a span was classified, so its tags are the answer and - this is not consulted. A token without one was spliced into a field - by replace() and carries no reading, so the views fall back to the - vocabulary -- which gives the answer the parser would have given, - the initial carve-out included ('E.' assigned to middle is an - initial, not the Italian conjunction). + A token the parse classified carries its reading in its tags and + this is not consulted. A token carrying UNCLASSIFIED_TAG was + spliced into a field as raw text -- by replace(), or by the + facade's v1 pickle load -- and carries no reading, so case repair + falls back to the vocabulary, which gives the answer the parser + would have given, the initial carve-out included ('E.' assigned to + middle is an initial, not the Italian conjunction). """ return bool(_normalize(word) in lex.conjunctions and not _INITIAL.fullmatch(word)) @@ -150,7 +151,7 @@ def initials(name: ParsedName, spec: str, delimiter: str, separator: str) -> str def _cap_word(word: str, role: Role, tags: frozenset[str], - lex: Lexicon, *, parsed: bool) -> str: + lex: Lexicon) -> str: # v1 cap_word order: particle/conjunction rule first, then the # exceptions map, then Mac/Mc, then str.capitalize normalized = _normalize(word) @@ -200,12 +201,24 @@ def _cap_word(word: str, role: Role, tags: frozenset[str], # mechanisms.md#RENDER-HONORS-THE-PARSE: "a token the parse never # saw carries no decision to honor, so a view falls back to the # vocabulary" -- _reads_as_conjunction above, which is v1's - # predicate over v1's own input class, keeping every assigned - # field exactly as 1.4.0 repaired it. + # predicate applied over TODAY's vocabulary rather than 1.4.0's. + # That is the honest claim and it is narrower than parity: the two + # vocabularies differ, so an assigned field can repair differently + # from 1.4.0 without this predicate differing at all. Measured on + # the released wheel: `h.last = "хосе и мария сантос"` gives + # 'Хосе И Мария Сантос' on 1.4.0 and 'Хосе и Мария Сантос' here, + # the Cyrillic `и` being a 2.x conjunction and not a 1.4.0 one; + # `h.last = "de la vega"` gives 'de la Vega' there and here. + # The mark, not the SPAN, is what says the text was never read: + # Parser.revise() also builds span-less tokens, from a sub-parse + # whose tags it keeps on purpose, and keying this on `span is None` + # overrode them -- `revise(middle='e-f')` repaired to 'e-F' where + # the same words parsed gave 'E-F' (#463 review). if ((normalized in lex.particles and role in (Role.MIDDLE, Role.FAMILY) and UNJOINED_TAG not in tags) or "conjunction" in tags - or (not parsed and _reads_as_conjunction(word, lex))): + or (UNCLASSIFIED_TAG in tags + and _reads_as_conjunction(word, lex))): return word.lower() # v1 cap_word tries the edge-stripped form, then the period-free # form ('Ph.D.' -> 'ph.d' -> 'phd' hits the exceptions map) @@ -221,14 +234,14 @@ def _cap_word(word: str, role: Role, tags: frozenset[str], def _cap_text(text: str, role: Role, tags: frozenset[str], - lex: Lexicon, *, parsed: bool) -> str: + lex: Lexicon) -> str: # word-by-word within the token text: hyphenated names capitalize # both sides ("macdole-eisenhower" -> "MacDole-Eisenhower"). The - # per-word walk is also why an UNPARSED token gets the vocabulary - # asked per word: the parse would have made one token per word of - # that text, so this is the granularity its answer would have had. - return _WORD.sub( - lambda m: _cap_word(m.group(0), role, tags, lex, parsed=parsed), text) + # per-word walk is also why an UNCLASSIFIED token gets the + # vocabulary asked per word: the parse would have made one token + # per word of that text, so this is the granularity its answer + # would have had. + return _WORD.sub(lambda m: _cap_word(m.group(0), role, tags, lex), text) # rules.md#R4: "case repair returns a repaired copy and never mutates @@ -244,15 +257,16 @@ def capitalized(name: ParsedName, lexicon: Lexicon | None, *, word is particle vocabulary is repaired as ordinary name words, and the mark saying so comes from the pipeline, as does the reading that a word is a conjunction rather than an initial. A - token replace() splices in has no span and no tags, so it was - never read: the vocabulary answers the per-word conjunction - question for it, and the per-part particle question -- which the - vocabulary cannot answer, the part being what a spliced field - lost -- falls through to plain particle treatment. A family set - that way to 'de la' stays 'de la' where the same words parsed give - 'De La'; one set to 'de y' keeps the 'y' lowercase, as the parse - does and as 1.4.0 did. Parser.revise() is the edit that classifies - the value, and gives 'De La' (rules.md#R4's Accepted boundary). + token carrying UNCLASSIFIED_TAG -- replace() splices those in, and + so does the facade's v1 pickle load -- was never read: the + vocabulary answers the per-word conjunction question for it, and + the per-part particle question is left to plain particle treatment, + since re-deriving the part answer needs a tag on every word of the + part and these have none. A family set that way to 'de la' stays + 'de la' where the same words parsed give 'De La'; one set to + 'de y' keeps the 'y' lowercase, as the parse does and as 1.4.0 + did. Parser.revise() is the edit that classifies the value, and + gives 'De La' (rules.md#R4's Accepted boundary). Idempotent: without force, a capitalized result is mixed-case and the gate returns it unchanged; with force, every _cap_word rule is a fixpoint on its own output.""" @@ -267,9 +281,7 @@ def capitalized(name: ParsedName, lexicon: Lexicon | None, *, if not force and joined not in (joined.upper(), joined.lower()): return name new_tokens = tuple( - Token(_cap_text(t.text, t.role, t.tags, lex, - parsed=t.span is not None), - t.span, t.role, t.tags) + Token(_cap_text(t.text, t.role, t.tags, lex), t.span, t.role, t.tags) for t in name.tokens) # equal tokens (possible only for synthetic span=None duplicates) # collapse to one mapping entry -- benign: the rebuilt ambiguity diff --git a/nameparser/_types.py b/nameparser/_types.py index 975d1583..d194b6f6 100644 --- a/nameparser/_types.py +++ b/nameparser/_types.py @@ -115,6 +115,22 @@ def __add__(self, other: object) -> NoReturn: # type: ignore[override] #: drift. FOLDED_TAG = "vocab:folded-middle" +#: Raw text spliced into a field after the parse, which no parse ever +#: read: `ParsedName.replace()` stamps it, and so does the facade's +#: v1 pickle load, which rebuilds a name from `*_list` strings alone. +#: A view that HOLDS a vocabulary falls back to it for a token carrying +#: this -- there is no decision to honor -- and reads every other token +#: by its tags (mechanisms.md#RENDER-HONORS-THE-PARSE). +#: The absence of a SPAN is not that signal and was tried as one: a +#: span-less token is SYNTHETIC, which `Parser.revise()` also builds, +#: from a full sub-parse whose tags it deliberately keeps. Keying the +#: fallback on the span overrode exactly those tags. Absence of TAGS is +#: not the signal either -- an ordinary parsed name word carries none. +UNCLASSIFIED_TAG = "vocab:unclassified" + +#: The one-element tag set its two producers stamp, built once. +_UNCLASSIFIED = frozenset({UNCLASSIFIED_TAG}) + _E = TypeVar("_E", bound=Enum) @@ -737,14 +753,19 @@ def replace(self, **fields: str) -> ParsedName: empty value clears the field. original is unchanged (provenance). Ambiguities referencing replaced tokens are dropped. - Replacement tokens carry NO tags, so tag-driven views degrade: - family_particles empties, particles regain their initials, and - a multi-word suffix is comma-joined. Parser.revise() is the - tag-preserving alternative. + Replacement tokens carry no STABLE tag, so tag-driven views + degrade: family_particles empties, particles regain their + initials, and a multi-word suffix is comma-joined. + Parser.revise() is the tag-preserving alternative. + They do carry UNCLASSIFIED_TAG, which says the text was never + read rather than that it was read and found plain: a view + holding a vocabulary falls back to it for these and to the tags + for everything else. """ replaced = _validated_field_strings(fields) synthetic = { - role: tuple(Token(word, None, role) for word in value.split()) + role: tuple(Token(word, None, role, _UNCLASSIFIED) + for word in value.split()) for role, value in replaced.items() } return self._with_field_tokens(synthetic) diff --git a/tests/test_capitalization.py b/tests/test_capitalization.py index 1855c763..44de2c55 100644 --- a/tests/test_capitalization.py +++ b/tests/test_capitalization.py @@ -1,3 +1,7 @@ +# pickle here round-trips an object this test just built; the library's +# own pickle support is what is under test, and no foreign data is read. +import pickle + import pytest from nameparser import HumanName @@ -305,3 +309,20 @@ def test_a_conjunction_inside_a_longer_token_is_a_name_word(self) -> None: uppered = HumanName('JUAN E-F SMITH') uppered.capitalize() self.m(str(uppered), 'Juan E-F Smith', uppered) + + # The third producer of never-classified text, and the one that is + # not an assignment: __getstate__ pickles the *_list STRINGS and + # nothing else (mechanisms.md#FACADE-CONTRACT -- components come + # back exactly as pickled, never re-parsed), so a load rebuilds + # tokens with no tags on them. They are marked UNCLASSIFIED_TAG for + # the same reason an assigned field is: reading the absent + # conjunction tag as "not a conjunction" would repair a restored + # 'juan ortega y gasset' to 'Juan Ortega Y Gasset', which is + # neither 1.4.0's answer nor the same object's before pickling. + def test_a_restored_pickle_keeps_v1_conjunction_repair(self) -> None: + for text, want in (('juan ortega y gasset', 'Juan Ortega y Gasset'), + ('john de la vega', 'John de la Vega'), + ('juan y garcia', 'Juan y Garcia')): + restored = pickle.loads(pickle.dumps(HumanName(text))) + restored.capitalize(force=True) + self.m(str(restored), want, restored) diff --git a/tests/v2/test_render.py b/tests/v2/test_render.py index 3a131834..87af670f 100644 --- a/tests/v2/test_render.py +++ b/tests/v2/test_render.py @@ -1,10 +1,10 @@ import pytest -from nameparser import HumanName, Parser, parse +from nameparser import HumanName, Parser, Policy, parse from nameparser._lexicon import Lexicon from nameparser._render import _collapse, render -from nameparser._types import (UNJOINED_TAG, Ambiguity, AmbiguityKind, - ParsedName, Role, Span, Token) +from nameparser._types import (UNCLASSIFIED_TAG, UNJOINED_TAG, Ambiguity, + AmbiguityKind, ParsedName, Role, Span, Token) def test_collapse_is_the_254_algorithm() -> None: @@ -342,10 +342,10 @@ def test_capitalized_lowers_the_words_the_parse_tagged_conjunction() -> None: def test_case_repair_falls_back_for_text_the_parse_never_read() -> None: - """A token the parse never read carries no decision to honor, so - case repair -- which is handed a lexicon -- asks the vocabulary - instead, getting the answer the parser would have given, v1's - initial carve-out included. + """A token carrying UNCLASSIFIED_TAG holds raw text no parse read, + so there is no decision to honor and case repair -- which is handed + a lexicon -- asks the vocabulary instead, getting the answer the + parser would have given, v1's initial carve-out included. ONE view falls back. `initials()` takes no lexicon, so it has none to ask; the sibling test below pins what that costs. @@ -354,7 +354,7 @@ def test_case_repair_falls_back_for_text_the_parse_never_read() -> None: base = p.parse("john smith") spliced = base.replace(family="velasquez y garcia") - assert [t.span for t in spliced.tokens[1:]] == [None, None, None] + assert all(UNCLASSIFIED_TAG in t.tags for t in spliced.tokens[1:]) assert spliced.capitalized(force=True).family == "Velasquez y Garcia" # the carve-out rides along: an assigned middle initial is an # initial, not the Italian conjunction @@ -376,6 +376,64 @@ def test_case_repair_falls_back_for_text_the_parse_never_read() -> None: assert p.parse("juan e-f smith").capitalized(force=True).middle == "E-F" +def test_the_mark_and_not_the_span_says_a_token_was_never_read() -> None: + """`span is None` means SYNTHETIC, which is a wider set than + unclassified, and keying the fallback on it was a measured + regression (#463 review). + + `Parser.revise()` builds span-less tokens too, from a full + sub-parse whose tags it keeps on purpose, and its docstring + promises the tag-driven views "behave as if the text had been + parsed". Under the span discriminator the fallback overrode exactly + those tags: `revise(middle='e-f')` repaired to 'e-F' where the same + words parsed gave 'E-F'. A HAND-BUILT span-less token is not marked + either, so it takes the tag path like every other token in the + library -- tag-driven semantics, not span-driven. + """ + p = Parser() + base = p.parse("john smith") + + revised = p.revise(base, middle="e-f") + assert [t.span for t in revised.tokens if t.role is Role.MIDDLE] == [None] + assert UNCLASSIFIED_TAG not in revised.tokens_for(Role.MIDDLE)[0].tags + # both sides of the pair the span discriminator split + assert p.parse("john e-f smith").capitalized(force=True).middle == "E-F" + assert revised.capitalized(force=True).middle == "E-F" + + # the initials half of the same regression, which needs a lexicon + # holding no particles at all to witness -- under the default one + # 'de la vega' is particle vocabulary and the parse skips it too. + # segment_scripts off because a from-scratch Lexicon covers no + # script the default policy activates, and the warning is an error + # under this suite's filters. + empty = Parser(lexicon=Lexicon(), + policy=Policy(segment_scripts=frozenset())) + assert (empty.parse("john de la vega").initials() + == empty.revise(empty.parse("john smith"), + family="de la vega").initials() + == "j. d. l. v.") + + # hand-built, span-less, untagged: classified by default + handbuilt = _pn("john de la vega", [ + Token("john", None, Role.GIVEN), + Token("de", None, Role.FAMILY), + Token("la", None, Role.FAMILY), + Token("vega", None, Role.FAMILY), + ]) + assert handbuilt.initials() == "j. d. l. v." + assert handbuilt.capitalized(force=True).family == "de la Vega" + # ... and the same tokens MARKED take the fallback, which is the + # only thing that moves them + marked = _pn("john de la vega", [ + Token("john", None, Role.GIVEN), + Token("de", None, Role.FAMILY), + Token("la", None, Role.FAMILY), + Token("y", None, Role.FAMILY, frozenset({UNCLASSIFIED_TAG})), + Token("vega", None, Role.FAMILY), + ]) + assert marked.capitalized(force=True).family == "de la y Vega" + + def test_initials_has_no_lexicon_so_a_spliced_field_is_all_name_words()\ -> None: """The accepted cost of `initials()` taking no lexicon: a field From c90a6d667e369db8e2e5228bac2d5c518d8bb60c Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sat, 29 Aug 2026 23:32:42 -0700 Subject: [PATCH 09/14] docs: the false claims a five-agent review of #463 found, re-measured MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Each was checked against the released wheels or re-derived on this branch before being rewritten. release_log.rst -- "HumanName.initials() and initials_list() already gave every one of these answers" was false: `h.middle = "E."` then `h.initials()` gives `j. s.`, not `j. E. s.` (#462). The bullet it stood in described the initials fallback and went with it in the first commit of this series; what remains is the #458 bullet, whose 1.4.0 parity claim is now scoped -- v1's predicate over TODAY's vocabulary, which is narrower than parity, because `h.last = "хосе и мария сантос"` gives `Хосе И Мария Сантос` on the 1.4.0 wheel and `Хосе и Мария Сантос` here, the Cyrillic `и` being a 2.x conjunction and not a 1.4.0 one. _render.py carried the same claim and carries the scoped one now. decisions.md, three re-derivations: - `_render._INITIAL` "is deleted, its last reader gone" -- it was deleted and restored in the same PR; case repair's fallback reads it, and the hand-sync obligation is narrower rather than retired. - "6565 name/variant/lexicon rows" does not derive. It is 6564: 1094 x 3 spellings x 2 lexicons, 13128 capitalized calls. 0 rows move between 82a7bd1 and here. - the R5 force-versus-case counts. Both counts move with the surface, not just the upper one: core 63/18, facade 62/16. The facade drops `Jane van der Berg née y Jones` in the upper direction and `John van der J. V` / `abdul V Smith` in the lower. mechanisms.md -- "the two that can fall back reach for Lexicon.default(), not the parse's lexicon" is false for `capitalized()`, which reads the lexicon it is handed; after the first commit here only one view falls back at all. "_cap_word cannot [answer the part question], walking one word of one token" is false as stated -- it is handed the whole token's tags and gates on UNJOINED_TAG; what it cannot do is RE-DERIVE the answer where no word of the part carries a tag. Its two `parse(...)` values are true only under `Lexicon.default().add(particles={'y'})` and are now scoped there: under the default vocabulary `Anh y Van` initials `A. V.` and `Juan de y` initials `J.`, the second being a test-pinned rules.md#R3 example line. Three documents disagreed on whether R4's conjunction carve-out stands on its own or rests on R3's clause. R4's own text says it rests on R3 ("the carve-out R3 states for initials"), so mechanisms.md's "in its own right" is the one that moves. Co-Authored-By: Claude Opus 5 --- docs/design/decisions.md | 6 +++--- docs/design/mechanisms.md | 2 +- docs/design/rules.md | 13 +++++++------ docs/release_log.rst | 2 +- 4 files changed, 12 insertions(+), 11 deletions(-) diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 09da71da..82c05ef8 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -735,9 +735,9 @@ Declined: - 2026-08-29 — the REPLACE/REVISE BOUNDARY, raised by the review of this branch as a gap needing a systematic fix and corrected by Derek to what it actually is: the documented boundary, with a supported path across it. `_cap_word` keys the particle TEST on lexicon membership but the REPAIR on the unjoined mark, and the mark is a parse product — so `parse('de la').capitalized(force=True)` is `'De La'` while `base.replace(family='de la').capitalized(force=True)` is `'de la'`, `replace()` splicing raw text into a field without classifying it. `Parser.revise()` is the method that classifies it, and its docstring already promised exactly this: each value gets a full sub-parse "so the stable tags survive and the tag-driven views ... behave as if the text had been parsed". Measured: `Parser().revise(base, family='de la').capitalized(force=True)` is `'De La'`. So there is nothing to fix — rules.md#R4 carries it as an Accepted boundary naming `revise()`, not as a deviation, and `revise()`'s enumeration of the tag-driven views gains `capitalized()`, which #407 made the fourth. (#458 would additionally make `replace()` agree, by keying the test on the `particle` tag rather than re-deriving membership from the word — the same shape it already names for `conjunction`/`initial`. That is a convenience, not the resolution.) It is a limit and not a regression either way: 2.1.0 gives `'de la'` on the spliced path too, so only the parsed path moved. -- 2026-08-29 #458 — DONE: case repair's conjunction test reads the `conjunction` TAG instead of re-deriving the word's class at render time, so a view honors the decision the parse recorded (mechanisms.md#VOCAB-TAGS) rather than taking it again. The two copies of the question were not the same question, which is the reason this was worth doing over a corpus that cannot see it: `_classify.py` asks `is_initial(token.text)` — the shape test ANDed with a repertoire test since #320 — while `_render.py` asked a bare pattern, shape only, over each WORD of a token's text rather than the token. It also retires the hand-sync obligation the module carried in a comment ("keep in sync with `nameparser/_pipeline/_vocab.py` by hand"): a tag read has nothing to keep in sync, and `_render._INITIAL` is deleted, its last reader gone. MEASURED, re-derived on this branch rather than taken from the issue: 0 of the 1094 corpus names move under `capitalized()` or `capitalized(force=True)`, and 0 move with each name also re-run uppercased and lowercased (6565 name/variant/lexicon rows, the second lexicon being `Lexicon.default().add(particles={'y'})`); the R5 force-versus-case counts recorded above hold at 62/16 through the facade and 63 through the core. Movement EXISTS outside that population, and the issue's "no output moves" is a claim about the corpus only: the old predicate re-decided per word, so `juan e-f smith` capitalized to `Juan e-F Smith` — the Italian conjunction inside a hyphenated middle name — and now gives `Juan E-F Smith`, agreeing with the uppercase spelling it always gave. An earlier draft of this change moved untagged tokens the same way — a family `replace()` splices in as `de y` repairing to `de Y` — and that was a REGRESSION, caught in review before it shipped and fixed in the bullet below; read this bullet with that one. NOT DONE, and the parenthetical in the boundary bullet above should be read as still open: the PARTICLE conjunct still keys on lexicon membership, so `replace()` does not agree there. Making it read the `particle` tag is a second change with its own blast radius — it moves a boundary rules.md#R4 states in prose — and #458 as filed does not ask for it. +- 2026-08-29 #458 — DONE: case repair's conjunction test reads the `conjunction` TAG instead of re-deriving the word's class at render time, so a view honors the decision the parse recorded (mechanisms.md#VOCAB-TAGS) rather than taking it again. The two copies of the question were not the same question, which is the reason this was worth doing over a corpus that cannot see it: `_classify.py` asks `is_initial(token.text)` — the shape test ANDed with a repertoire test since #320 — while `_render.py` asked a bare pattern, shape only, over each WORD of a token's text rather than the token. It does NOT retire the hand-sync obligation the module carried in a comment ("keep in sync with `nameparser/_pipeline/_vocab.py` by hand"), and an earlier draft of this bullet said it did: the tag read has nothing to keep in sync, but `_render._INITIAL` was deleted and then restored in the same PR, the fallback in the bullet below giving it a reader again. The obligation is narrower and sharper than before — the two copies no longer answer the same question about the same token, so what has to hold is that the fallback answers as `_vocab` would — and test_regex_sync is what holds it. MEASURED, re-derived on this branch rather than taken from the issue: 0 of the 1094 corpus names move under `capitalized()` or `capitalized(force=True)`, and 0 move with each name also re-run uppercased and lowercased (6564 name/variant/lexicon rows = 1094 x 3 spellings x 2 lexicons, the second lexicon being `Lexicon.default().add(particles={'y'})`, and 13128 `capitalized` calls over them); the R5 force-versus-case counts recorded above hold — 62 upper / 16 lower through the facade, 63 upper / 18 lower through the core. Movement EXISTS outside that population, and the issue's "no output moves" is a claim about the corpus only: the old predicate re-decided per word, so `juan e-f smith` capitalized to `Juan e-F Smith` — the Italian conjunction inside a hyphenated middle name — and now gives `Juan E-F Smith`, agreeing with the uppercase spelling it always gave. An earlier draft of this change moved untagged tokens the same way — a family `replace()` splices in as `de y` repairing to `de Y` — and that was a REGRESSION, caught in review before it shipped and fixed in the bullet below; read this bullet with that one. NOT DONE, and the parenthetical in the boundary bullet above should be read as still open: the PARTICLE conjunct still keys on lexicon membership, so `replace()` does not agree there. Making it read the `particle` tag is a second change with its own blast radius — it moves a boundary rules.md#R4 states in prose — and #458 as filed does not ask for it. -- 2026-08-29 (#458 review) — THE FALLBACK FOR TEXT THE PARSE NEVER SAW, and why it is scoped to one of the two questions `_cap_word` asks. Reading the tag and nothing else regressed every ASSIGNED field holding a conjunction, because a value `replace()` splices in is never classified and so carries no tag to read. Measured on the released 1.4.0 and 2.1.0 wheels, all agreeing and all broken by the tag-only reading: `h.last = 'velasquez y garcia'` then `capitalize(force=True)` gives `John Velasquez y Garcia`, `h.middle = 'e.'` gives `John E. Smith`, `h.last = 'smith-y'` gives `John Smith-y`, `h.first = 'y'` gives `y Smith`. That is the canonical Spanish surname and the shape `y` is in the vocabulary FOR, so an Accepted clause was not available: it had to be fixed. RULE: a token carrying `UNCLASSIFIED_TAG` was never read, so there is no decision to honor and the view falls back to the vocabulary. Note what the discriminator is NOT, in two directions, the second of which took a second review round to find. It is not UNTAGGEDNESS: an ordinary parsed name word carries no tags either (`velasquez` parses to `tags=[]`), so that would have swept in the whole parsed corpus. And it is not the absent SPAN, which is what the first fix used. Span-less means SYNTHETIC, a wider set than unclassified: `Parser.revise()` builds span-less tokens too, from a full sub-parse whose tags it keeps deliberately, and its docstring promises the tag-driven views "behave as if the text had been parsed" — so keying the fallback on the span overrode exactly the tags `revise()` exists to preserve. Measured on the branch that did it: `revise(name, middle='e-f').capitalized(force=True)` gave `e-F` where `parse('john e-f smith')` gave `E-F`, and under `Parser(lexicon=Lexicon())` a revised family `de la vega` initialed `j. v.` where the parse initialed `j. d. l. v.`. So the mark is stamped by the two producers of genuinely unclassified text — `ParsedName.replace()`, and the facade's v1 pickle load, which rebuilds a name from `*_list` strings and no tags — and a HAND-BUILT span-less token is not marked, so it takes the tag path like everything else in the library. That is the tag-driven semantics every other view already has, and it is a change from the span reading in exactly one place: a hand-built span-less token whose text is CONJUNCTION vocabulary now capitalizes rather than lowercasing (measured: a hand-built family `velasquez y garcia` repairs to `Velasquez Y Garcia` untagged, `Velasquez y Garcia` with either the `conjunction` tag or the mark). The PARTICLE conjunct is unaffected either way — it keys on lexicon membership rather than on the mark, which is the open item this bullet's neighbours record — so a hand-built `de la Vega` is unchanged. SCOPED TO THE PER-WORD QUESTION, and this is the durable half. *Is this word a conjunction rather than an initial* is a property of the word and its spelling: an unread token has no ambiguity the parse would have resolved differently, so the vocabulary gives the answer the parser would have given, v1's initial carve-out included — which is why `_render._INITIAL` came back and why test_regex_sync's two-copy assertion matters MORE now, the fallback being right only while it answers as `_vocab` would. *Is this particle acting as a particle here* is a property of the whole PART, and the two views hold different amounts of it — which is why the scope differs per VIEW and not per question, a distinction the first draft of this bullet got wrong. Case repair walks one word of one token at a time and never sees the part, so it cannot answer it and does not ask: the fallback there is the word question alone. `initials()` holds every token of the role at once and so COULD answer it, but is handed no vocabulary to answer it from, which is what the next bullet turns on. Measured cost of the wide reading IN CASE REPAIR, so nobody re-derives it: `replace(family='de la')` gives `de la` today and a parse gives `De La`; widening case repair's fallback would make them agree and would thereby REVERSE the replace/revise boundary recorded two bullets above, with `revise()` as its documented crossing. Do not widen it without reopening the decision. Verified after the fix: `replace(family='velasquez y garcia').capitalized(force=True)` is `Velasquez y Garcia`, `replace(family='de la').capitalized(force=True)` is `de la`, and the parsed and `revise()` paths are unchanged. +- 2026-08-29 (#458 review) — THE FALLBACK FOR TEXT THE PARSE NEVER SAW, and why it is scoped to one of the two questions `_cap_word` asks. Reading the tag and nothing else regressed every ASSIGNED field holding a conjunction, because a value `replace()` splices in is never classified and so carries no tag to read. Measured on the released 1.4.0 and 2.1.0 wheels, all agreeing and all broken by the tag-only reading: `h.last = 'velasquez y garcia'` then `capitalize(force=True)` gives `John Velasquez y Garcia`, `h.middle = 'e.'` gives `John E. Smith`, `h.last = 'smith-y'` gives `John Smith-y`, `h.first = 'y'` gives `y Smith`. That is the canonical Spanish surname and the shape `y` is in the vocabulary FOR, so an Accepted clause was not available: it had to be fixed. RULE: a token carrying `UNCLASSIFIED_TAG` was never read, so there is no decision to honor and the view falls back to the vocabulary. Note what the discriminator is NOT, in two directions, the second of which took a second review round to find. It is not UNTAGGEDNESS: an ordinary parsed name word carries no tags either (`velasquez` parses to `tags=[]`), so that would have swept in the whole parsed corpus. And it is not the absent SPAN, which is what the first fix used. Span-less means SYNTHETIC, a wider set than unclassified: `Parser.revise()` builds span-less tokens too, from a full sub-parse whose tags it keeps deliberately, and its docstring promises the tag-driven views "behave as if the text had been parsed" — so keying the fallback on the span overrode exactly the tags `revise()` exists to preserve. Measured on the branch that did it: `revise(name, middle='e-f').capitalized(force=True)` gave `e-F` where `parse('john e-f smith')` gave `E-F`, and under `Parser(lexicon=Lexicon())` a revised family `de la vega` initialed `j. v.` where the parse initialed `j. d. l. v.`. So the mark is stamped by the two producers of genuinely unclassified text — `ParsedName.replace()`, and the facade's v1 pickle load, which rebuilds a name from `*_list` strings and no tags — and a HAND-BUILT span-less token is not marked, so it takes the tag path like everything else in the library. That is the tag-driven semantics every other view already has, and it is a change from the span reading in exactly one place: a hand-built span-less token whose text is CONJUNCTION vocabulary now capitalizes rather than lowercasing (measured: a hand-built family `velasquez y garcia` repairs to `Velasquez Y Garcia` untagged, `Velasquez y Garcia` with either the `conjunction` tag or the mark). The PARTICLE conjunct is unaffected either way — it keys on lexicon membership rather than on the mark, which is the open item this bullet's neighbours record — so a hand-built `de la Vega` is unchanged. SCOPED TO THE PER-WORD QUESTION, and this is the durable half. *Is this word a conjunction rather than an initial* is a property of the word and its spelling: an unread token has no ambiguity the parse would have resolved differently, so the vocabulary gives the answer the parser would have given, v1's initial carve-out included — which is why `_render._INITIAL` came back and why test_regex_sync's two-copy assertion matters MORE now, the fallback being right only while it answers as `_vocab` would. *Is this particle acting as a particle here* is a property of the whole PART, and answering it needs a reading on every word of the part, which is exactly what a spliced field lacks. `_cap_word` is handed the whole token's tags — it is not blind to the part because it walks one word, and an earlier draft of this bullet said it was — but with no tag on any word of the part there is nothing to re-derive the part answer FROM, so the fallback here is the word question alone and the particle half falls through to plain particle treatment. Measured cost of the wide reading IN CASE REPAIR, so nobody re-derives it: `replace(family='de la')` gives `de la` today and a parse gives `De La`; widening case repair's fallback would make them agree and would thereby REVERSE the replace/revise boundary recorded two bullets above, with `revise()` as its documented crossing. Do not widen it without reopening the decision. Verified after the fix: `replace(family='velasquez y garcia').capitalized(force=True)` is `Velasquez y Garcia`, `replace(family='de la').capitalized(force=True)` is `de la`, and the parsed and `revise()` paths are unchanged. - 2026-08-29 (#458 review, then the review of PR #463) — the SAME FALLBACK in `initials()`: WRITTEN, MEASURED AND DROPPED before merge. The defect it aimed at is real and older than #458 — `replace(family='velasquez y garcia').initials()` gives `j. v. y. g.` where the parse, the facade and 1.4.0 all give `j. v. g.`, and `replace(family='de la vega')` gives `j. d. l. v.` for `j. v.` — and it is unfixed again. What killed the fix is the parameter list: `initials()` takes no lexicon, so the fallback had to reach for `Lexicon.default()` and GUESS. `capitalized(lexicon=...)` is handed the caller's vocabulary and so guesses nothing, which is why its fallback stays. Measured cost of the guess, and the reason a merely-imperfect fallback was not good enough: under `Parser(lexicon=Lexicon.default().add(particles={'y'}))`, `parse('Juan de y').initials()` is `J. d. y.` and `parse('Juan Perez').replace(family='de y').initials()` was `J. d. y.` before the fallback and `J.` after it — an entire field's initials gone, because the default vocabulary reads `de y` as particles-plus-conjunction where the caller's reads it as an all-particle part. Guessing the wrong vocabulary for a name parsed under a custom one is worse than answering from tags alone, and `initials()` cannot be told better. The missing crossing is the whole of it: `Parser` carries `capitalized`, `matches` and `revise` as its custom-lexicon crossings and no `initials`, so there is no supported way to hand this view the parse's vocabulary. An issue is filed for that; when it lands, the fallback becomes answerable rather than a guess and this decision is worth reopening. ACCEPTED until then, and stated so a reader does not re-derive it as a bug: `replace(family='de la vega').initials()` is `j. d. l. v.` where the facade and a parse both give `j. v.`. The reach is the v2 core only — `HumanName.initials_list()` computes from the field STRINGS with its own lexicon (`_facade._initials_lists`, v1 parity) and gives `['j', 'v', 'g']` at 1.4.0, 2.1.0 and here — which is also why the facade cannot be used as evidence that the core is fine. Two things learned in the attempt are worth keeping though the code is gone. `_types._remarked` is not the bug and needs no change: it recomputes the mark from TAGS after every edit, which is right for a record of what the parse decided and simply silent about text nobody classified. And any future fallback must be scoped per ROLE, not per name — a role the parse classified whole is decided by its tags however the other roles were built — a distinction that passes every other test in the suite and needs a hand-built name to witness. @@ -750,7 +750,7 @@ Declined: - 2026-08-29 — WHY R5'S STATEMENT WAS REWORDED RATHER THAN GIVEN A THIRD EXAMPLE, recorded at length because the wrong turn was taken twice and the reasoning is reusable. R5 first said a mixed-case name is returned untouched "whether or not its casing is correct". That phrasing reads as a disjunction with two branches, and invites the question of which example witnesses the already-correct one. Nothing can: for a mixed-case name that repair would not alter, kept and repaired are the SAME STRING by construction, so no example line distinguishes a parser with the gate from one without it. The unwitnessable branch is a property of the claim, not a gap in our choice of names — and the bullet above had already said so ("an already-correct input cannot witness a rule whose subject is what gets kept REGARDLESS of correctness") before two review rounds pushed past it. Two rows were tried and both withdrawn. `"Juan McDonald" → capitalized="Juan McDonald"` was INERT (mechanisms.md's inert-measurement class): it passes with R5's gate deleted, which is the whole failure shape that class names. `"Vincent Van Gogh" → capitalized="Vincent Van Gogh"` did discriminate — measured by comparing the shipped call against gate-off behavior, which the forced call reproduces exactly, since deleting the gate is what `force` already does: shipped `'Vincent Van Gogh'`, gate-off `'Vincent van Gogh'`, the particle rule lowercasing `Van` the moment repair runs — but it was REDUNDANT, not complementary. It asserts the same proposition as the `Shirley Maclaine` row above it: repair would change this name, and the gate keeps it anyway. The two differ only in which repair rule would have fired, and which rule repair applies is R4's subject, not R5's. And the frame it was chosen under was itself wrong, which is the more useful half of the lesson (reframed 2026-08-29 on Derek's correction, while R4 was being reworked). The row was picked to be bearer-correct — a name whose casing the bearer would endorse — and that frame drags the document into per-name arguments about whose spelling is right: this one needed `Vincent van Gogh` to be the correct spelling, which P6's own examples contradict, and the tussenvoegsel convention behind them is amended later on this branch anyway. CORRECTNESS DOES NOT ENTER INTO IT. Mixed case is the writer making an explicit choice, and repair defers to that choice rather than judging it — a name kept is not a name endorsed. Read that way the withdrawn row needed no claim about Dutch orthography at all, and the reason it needed one is exactly that the frame was wrong. The fix was to stop claiming the unwitnessable branch: the statement now says a mixed-case name is kept and that whether its casing is right does not enter into the decision, which `Shirley Maclaine` — casing wrong, kept — witnesses whole. GENERAL LESSON, and the reason this is long: when an example cannot be found for half a rule, suspect the STATEMENT before suspecting the example set. A phrasing that promises more branches than the behavior has will absorb inert examples indefinitely, each one looking like progress. Caution for the commit that reworks R4 and will choose its own mixed-case rows: avoid a name whose family base is wholly particle vocabulary (`Anh Van Do`, base `Van Do`), because the #407 work changes how those capitalize and would neuter such a row silently; `gogh` and `vega` are in neither `particles` nor `particles_ambiguous`, `van` and `do` are in both. - 2026-08-29 — the override has TWO routes, and the rule states neither, by design. Per-call is the obvious one. The second is a v1 Constants attribute the facade still honors, `force_mixed_case_capitalization` (nameparser/_facade.py resolves it when no per-call value is given; docs/release_log.rst records it as "still honored through the facade"). Measured today: with that attribute set True, `HumanName('Shirley Maclaine')` then a bare `.capitalize()` — no argument at all — LEAVES `'Shirley MacLaine'` (the v1 call mutates in place and returns None, so the name is read back with `str()`). An earlier draft of R5 said "only an explicit request to repair regardless overrides that", which that measurement falsifies; the statement now says repair was asked for anyway, without saying by what route (it read `repair regardless of how the name is cased was asked for` until 2026-08-29, when that phrasing turned out to have a second reading -- see the override bullet below), because rules.md is implementation-free by its own preamble. - 2026-08-29 — `capitalized_forced` is a test-side pseudo-field, not a parser field. The route the EXAMPLES use is the per-call argument to `capitalized()`, and an argument is not a policy, locale or extras gate — the only three things rules.md's grammar admits in an example's annotation slot — so it cannot ride that slot, and the doc runner grew a resolver branch instead. (The facade attribute above is a second route to the same behavior, not a second thing to assert; the core takes the argument only.) The R5 block's example lines are asserted by the suite like any others. They do enlarge one file: `corpus_rules.jsonl` is generated from this document's examples, and `Shirley Maclaine` had not been among them, so regenerating adds it (235 lines, from 234). Whether that moves the DIFFERENTIAL population is a separate question with a per-name answer, and the general answer is that a rules.md example CAN move it: the harness dedupes across corpus files, so a new example name costs a population slot exactly when no other corpus already carries that same string. `Shirley Maclaine` was already in `corpus.jsonl`, so it costs nothing and the population holds at 1090 — measured, not assumed. The row withdrawn above is the counter-case, and worth keeping for it: `Vincent Van Gogh` appears in no other corpus (`corpus.jsonl` and `corpus_issues.jsonl` carry `Vincent van Gogh`, a DIFFERENT string that does not dedupe against it), and while it was in the block the population read 1091. Neither name needed a ledger rule, which is a measurement rather than a consequence of the above: the gate exits 0 with 0 unexplained at all three baselines either way. -- 2026-08-29 — WHAT THE OVERRIDE DOES AND DOES NOT PROMISE, from Derek's framing of R5 and then measured, because the framing implies a property that is ALMOST true and the gap is the useful part. The framing first, and it supersedes the correctness talk elsewhere in this entry: mixed case is the writer making an explicit choice, and repair defers to that choice instead of judging it. Nothing is being called correct or incorrect — a name kept is a name whose writer said something about it, and a name repaired is one whose writer did not. The property that seems to follow is that asking for repair REGARDLESS should ignore the given casing entirely, so one name repairs to one string however it was written. MEASURED over the 1094 corpus names and it does NOT hold. On the v2 core — `parse(n).capitalized(force=True)` against `parse(n.upper()).capitalized()`, rendering all seven roles — the two differ for 63 names, and the lowercase direction for 16. Naming the surface matters here: through `HumanName` and `str()`, which is what tests/test_capitalization.py uses, the upper count is 62, because the facade's default render spec omits the maiden name and so cannot see `Jane van der Berg née y Jones` (the whole difference, and it is a conjunction inside the maiden name). Two things about those numbers are the opposite of what one would guess. UPPERCASE IS THE WORSE DIRECTION, not the clean one. And the misses are not merely the pipeline's case-sensitivity leaking in: of the 63, only 25 move a role at all, and the other 38 parse byte-identically and diverge inside the repair (25 and 37 through the facade, the missing name being one of the byte-identical ones). The mechanism is v1's initial carve-out — a conjunction is not lowercased where it is written initial-shaped, and initial-shaped means one CAPITAL letter — so uppercasing turns every one-letter conjunction into an initial (`Velasquez y Garcia, Dr. Juan Q.` forced keeps `y`; the same name uppercased then repaired gives `Y`, with a byte-identical partition either side -- the comma form is the one to cite here, the space-written `Dr. Juan Q. Velasquez y Garcia` being a member of the 25 whose roles DO move), and lowercasing turns a middle initial `E` into the Italian conjunction. 1.4.0 does the same (`JUAN Y GARCIA` capitalizes to `Juan Y Garcia`), so this is inherited, and it is recorded here rather than fixed here. Recompute both directions by running the two forms over the four corpus files deduped and diffing. +- 2026-08-29 — WHAT THE OVERRIDE DOES AND DOES NOT PROMISE, from Derek's framing of R5 and then measured, because the framing implies a property that is ALMOST true and the gap is the useful part. The framing first, and it supersedes the correctness talk elsewhere in this entry: mixed case is the writer making an explicit choice, and repair defers to that choice instead of judging it. Nothing is being called correct or incorrect — a name kept is a name whose writer said something about it, and a name repaired is one whose writer did not. The property that seems to follow is that asking for repair REGARDLESS should ignore the given casing entirely, so one name repairs to one string however it was written. MEASURED over the 1094 corpus names and it does NOT hold. On the v2 core — `parse(n).capitalized(force=True)` against `parse(n.upper()).capitalized()`, rendering all seven roles — the two differ for 63 names, and the lowercase direction for 18. Naming the surface matters here, and BOTH counts move with the surface, which an earlier draft of this bullet got half right: through `HumanName` and `str()`, which is what tests/test_capitalization.py uses, the counts are 62 and 16, because the facade's default render spec omits the maiden name and so cannot see `Jane van der Berg née y Jones` (the whole of the upper difference, and it is a conjunction inside the maiden name); the two the facade drops in the lower direction are `John van der J. V` and `abdul V Smith`. Re-derived 2026-08-29 on this branch, and named per name so the next reader can check the arithmetic: core 63/18, facade 62/16. Two things about those numbers are the opposite of what one would guess. UPPERCASE IS THE WORSE DIRECTION, not the clean one. And the misses are not merely the pipeline's case-sensitivity leaking in: of the 63, only 25 move a role at all, and the other 38 parse byte-identically and diverge inside the repair (25 and 37 through the facade, the missing name being one of the byte-identical ones). The mechanism is v1's initial carve-out — a conjunction is not lowercased where it is written initial-shaped, and initial-shaped means one CAPITAL letter — so uppercasing turns every one-letter conjunction into an initial (`Velasquez y Garcia, Dr. Juan Q.` forced keeps `y`; the same name uppercased then repaired gives `Y`, with a byte-identical partition either side -- the comma form is the one to cite here, the space-written `Dr. Juan Q. Velasquez y Garcia` being a member of the 25 whose roles DO move), and lowercasing turns a middle initial `E` into the Italian conjunction. 1.4.0 does the same (`JUAN Y GARCIA` capitalizes to `Juan Y Garcia`), so this is inherited, and it is recorded here rather than fixed here. Recompute both directions by running the two forms over the four corpus files deduped and diffing. - 2026-08-29 — and therefore NOT stated in rules.md, which is a deliberate choice rather than an oversight. The document's examples are keyed on input STRINGS, so any statement of the property invites exactly the test that falsifies it — re-case the input, expect the same output — and the counterexamples are already in the corpora. The property is true of the repair given a parse, and rules.md speaks input-to-output; a rule stating it would be over-broad in the one direction a reader would check. What R5's statement says is enough for the promise that IS kept: a mixed-case name is kept unless repair was asked for anyway. That clause was REWORDED for this, and the reword is the whole point rather than a tidy-up. It read `unless repair regardless of how the name is cased was asked for`, which carries two readings -- the intended one, that the request overrides the keeping, and a second one, that the repair disregards the input's casing, which is this property in nearly this bullet's own words. A reader taking the second reading would run the re-casing test predicted above, land on `Velasquez y Garcia, Dr. Juan Q.` (in the corpus today), and conclude the RULE is wrong when only the phrasing was. Nine words, and they asserted the thing the paragraph exists to deny. The property is pinned in tests/test_capitalization.py instead, over names carrying no single-letter word whose class case decides, with `juan y garcia` beside it as the recorded exception. R5's example block gains `"SHIRLEY MACLAINE" → capitalized="Shirley MacLaine"` from this work, and it earns its place on its own ground rather than as half of a convergence pair: it is the only row in the block that fails when the gate is narrowed to lowercase-only, every other row passing that mutation. Measured three ways — gate deleted (passes, so it does not witness the gate's existence), gate narrowed to accept only all-lowercase (FAILS, and alone in the block), Mac/Mc convention deleted (fails, with the other two rows). Until it was added, R5 stated that repair acts on a name written entirely in one case and witnessed only the lowercase half of it. That lowercase half is still `"juan mcdonald"`, which is byte for byte an R4 row as well, and the duplication is deliberate rather than an editing slip: the two rules make different claims about the same line — R4 that the repair honors the Mac/Mc convention, R5 that an all-lowercase name is acted on at all — and dropping it from R5 would leave the gate's lowercase half unwitnessed inside the rule that states the gate. Five other rows already sit under two rules apiece for the same reason (P5/P6 twice, P5/O5, N3/M4, W1/W3). - 2026-08-29 — DEBT this extraction leaves, named so the next commit inherits an obligation rather than a rediscovery. Pulling the gate out into R5 leaves R4 carrying ONE falsehood and ONE ambiguity — different defects wanting different repairs, and `interacts: R5` carries neither, the field being advisory. FALSE: R4 promises repair "vocabulary exceptions (McDonald) included", but `str(parse('Juan Mcdonald').capitalized())` is `'Juan Mcdonald'` — the gate refuses before any vocabulary is consulted, and only `str(parse('Juan Mcdonald').capitalized(force=True))`, `'Juan McDonald'`, reaches the exception. R4 needs its promise scoped to names the gate admits. AMBIGUOUS, not false: R4's "an already-correct name comes back unchanged" means correct by the repair's own conventions, i.e. idempotence, and under that meaning it is true; a reader hears correct as the bearer writes it, and under THAT meaning `str(parse('bell hooks').capitalized())` — `'Bell Hooks'` — looks like a counterexample. It is not one, because `bell hooks` is not already-correct in R4's sense. What R4 owes is a disambiguation of "correct", NOT a narrowing to spare deliberately single-cased names: that would be new behavior, and R5's own rationale declines it on the ground that single case leaves the repair nothing to read. Also for that commit, and inert as things stand: R4's boundary row `"Juan McDonald" → capitalized="Juan McDonald"` passes with R5's gate deleted, exactly like the R5 row that was withdrawn above; rewriting it to `capitalized_forced=` makes it discriminate for R4's own subject but still witnesses nothing about the already-correct question. This commit adds R5 and touches R4 only on its pointer line, leaving both defects as found rather than half-fixed by a commit whose subject is something else. diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 7ff5a84c..77286130 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -59,7 +59,7 @@ Problem shape. Two stages need the same answer about the same input, and the one ## RENDER-HONORS-THE-PARSE — the parse decides it, the views honor it -Problem shape. A render view needs a fact the parse already settled — whether a word is the conjunction or an initial, whether a particle is acting as one, which word renders first. Contract statement. The parse decides it; the render views honor those decisions and never re-evaluate them. Two directions break that, and each has been found here as a defect: a view RE-DERIVES the answer from the text, keeping its own copy of a pipeline predicate — that one shipped through 2.0 and 2.1 — or a view honors the record and then OVERRIDES it, readmitting what the decision excluded, which is filed and open here. How it works. Re-deriving fails because the two copies stop being the same question long before anyone notices they are two: `_cap_word` re-ran the conjunction-versus-initial decision from the word's spelling against a hand-maintained copy of the pipeline's `_INITIAL` pattern while classify had already answered it and recorded it on the token, and `_classify.py` asks `is_initial()`, the shape test ANDed with a script-repertoire test since #320, where `_render.py` asked the bare pattern, and asked it per WORD of a token's text rather than per token, so `juan e-f smith` repaired to `Juan e-F Smith` (#458). Overriding fails more directly, and this entry's instance of it is OPEN rather than closed: `initials()` honors the `conjunction` tag through `_SKIP_TAGS` and then readmits the token whenever it also carries UNJOINED_TAG, the mark of an all-particle part, where rules.md#R3 excludes a conjunction "even then" — so under a caller's vocabulary that puts a word in both sets the view readmits what the rule excluded, and `parse("Anh y Van")` initialed `A. y. V.` and `parse("Juan de y")` `J. d. y.` on 2026-08-29. #461 is where that stands, and it is worth reading before re-fixing it: the narrowing was written, measured and BACKED OUT in the same PR, because honoring R3 there cost `initials()` its agreement with `family_base`, which reads that same `de y` as the base — this entry's other shape, arrived at from the render side, and the reason the question moved from the code to R3's clause (decisions.md carries the argument). What to carry away is the DIRECTION and not its verdict here: a view that honors a record and then readmits what the record excluded is overriding a decision it never took, whichever way this one settles. Case repair reads that same token and does NOT readmit it — `capitalized(force=True)` on `Anh y Van` gives `Anh y Van`, R4 stating the carve-out in its own right — so the two views disagree today about that token exactly as they did before #461 and the backout restores that disagreement knowingly; only the FORCED call witnesses the repair half, R5's gate refusing a mixed-case name before any of this is consulted. This is the CONSUMER-side rule over the producer-side entries — VOCAB-TAGS records what the vocabulary knew, MARK-DONT-STRIP what a stage decided about it, FOLDED_TAG what order to render in — and a view reads what they recorded, whichever kind it is. It sits where ONE-PREDICATE-PER-QUESTION's stated limit leaves off: where two live sites need one answer they share a predicate, but a render view always comes AFTER the decider, so the answer is recorded rather than shared — on the TOKEN, which is the views' equivalent of that entry's `ParseState.order`, no view being able to see a ParseState at all. Known limit, and the half most easily got wrong next: a token the parse never saw carries no decision to honor, so a view falls back to the vocabulary — the tell is `span is None`, which `ParsedName.replace()` builds when it splices raw text into a field, and NOT untaggedness, since an ordinary parsed name word carries no tags either. A view can only fall back if it HOLDS a vocabulary, which is why the fallback reaches `initials()` and `capitalized()` and not `family_base` or `family_particles`: those are properties on ParsedName, whose fields are original/tokens/ambiguities and nothing else, so a spliced field empties the particles view and leaves the base the whole field, with `Parser.revise()` the crossing there too (docs/usage.rst says so where it documents the degradation). The two that can fall back reach for `Lexicon.default()`, not the parse's lexicon, which ParsedName does not keep either. Within a view that fallback is drawn per QUESTION (rules.md#R4's Accepted clause); which questions a view can answer at all differs per VIEW and not per question, because the views hold different amounts of the name. Whether a word is the conjunction or an initial is a property of the word, and both views answer it; whether a part is wholly particles is a property of the whole PART, which `initials()` answers because it holds every token of the role at once (scoped per role — rules.md#R3's Accepted clause) and `_cap_word` cannot, walking one word of one token, so repair leaves that half to plain particle treatment and rules.md#R4's Accepted boundary records the consequence — a spliced field is not repaired as a parsed one is, with `Parser.revise()` as the supported crossing. A fallback is right only while it answers as the pipeline would, and THAT is held by hand rather than mechanically: test_regex_sync pins the two `_INITIAL` copies to each other and to config, while the repertoire half of the pipeline's predicate (#320) is deliberately not carried across, layering forbidding the import — so a caller-added conjunction in an initialless script initials from a spliced field where the same word parsed does not. A second limit, recorded rather than closed: `_cap_word`'s PARTICLE conjunct still keys on the lexicon handed to the view rather than on the `particle` tag, so a repair run with a lexicon other than the parse's re-decides a word the parse already read — a name parsed under the default vocabulary, where `parse('juan smith vega')` reads `vega` as the family, repairs to `Juan Smith vega` when `capitalized()` is handed `Lexicon.default().add(particles={'vega'})` instead — the divergence needs the two lexicons to differ, and repairing under the parse's own lexicon gives `Juan Smith Vega`. Making it read the tag moves a boundary rules.md#R4 states in prose, so it is a separate decision and not a cleanup (decisions.md#R4, "NOT DONE"). Lives in. nameparser/_render.py (`capitalized`/`_cap_word`, `initials`/`_initials_from`, `_reads_as_conjunction`) and nameparser/_types.py (`_text_for`), reading what nameparser/_pipeline/ recorded — the mark those views read is recomputed producer-side in `_remarked`, which is deliberately silent about text nobody classified and is right to be. Reach for it when. A view is about to consult a Lexicon, a regex or an exception list about a word the parse already saw — or a view and a field disagree about the same parse. #408 is filed as that second shape, and was open on 2026-08-29: `initials()` walks tokens in written order where the family field applies FOLDED_TAG's ordering, so `parse("der, y van")` gave family `van der` and initials `y. d. v.` that day, one view not yet honoring the fold. It belongs here rather than argued again from scratch. +Problem shape. A render view needs a fact the parse already settled — whether a word is the conjunction or an initial, whether a particle is acting as one, which word renders first. Contract statement. The parse decides it; the render views honor those decisions and never re-evaluate them. Two directions break that, and each has been found here as a defect: a view RE-DERIVES the answer from the text, keeping its own copy of a pipeline predicate — that one shipped through 2.0 and 2.1 — or a view honors the record and then OVERRIDES it, readmitting what the decision excluded, which is filed and open here. How it works. Re-deriving fails because the two copies stop being the same question long before anyone notices they are two: `_cap_word` re-ran the conjunction-versus-initial decision from the word's spelling against a hand-maintained copy of the pipeline's `_INITIAL` pattern while classify had already answered it and recorded it on the token, and `_classify.py` asks `is_initial()`, the shape test ANDed with a script-repertoire test since #320, where `_render.py` asked the bare pattern, and asked it per WORD of a token's text rather than per token, so `juan e-f smith` repaired to `Juan e-F Smith` (#458). Overriding fails more directly, and this entry's instance of it is OPEN rather than closed: `initials()` honors the `conjunction` tag through `_SKIP_TAGS` and then readmits the token whenever it also carries UNJOINED_TAG, the mark of an all-particle part, where rules.md#R3 excludes a conjunction "even then" — so under a caller's vocabulary that puts a word in both sets the view readmits what the rule excluded, and under `Lexicon.default().add(particles={'y'})` — the lexicon those two readings need, and they hold under no other — `parse("Anh y Van")` initialed `A. y. V.` and `parse("Juan de y")` `J. d. y.` on 2026-08-29. Read those two values scoped to that lexicon or they are simply false: under the DEFAULT vocabulary the same strings give `A. V.` and `J.`, and the second is a rules.md#R3 example line the doc runner asserts. #461 is where that stands, and it is worth reading before re-fixing it: the narrowing was written, measured and BACKED OUT in the same PR, because honoring R3 there cost `initials()` its agreement with `family_base`, which reads that same `de y` as the base — this entry's other shape, arrived at from the render side, and the reason the question moved from the code to R3's clause (decisions.md carries the argument). What to carry away is the DIRECTION and not its verdict here: a view that honors a record and then readmits what the record excluded is overriding a decision it never took, whichever way this one settles. Case repair reads that same token and does NOT readmit it — `capitalized(force=True)` on `Anh y Van` gives `Anh y Van` under that same lexicon, R4 carrying the carve-out in its own words but ON R3's authority — its text reads "being no name word in any part — the carve-out R3 states for initials", so the two rules stand or fall together and a change to R3's clause reaches R4 — so the two views disagree today about that token exactly as they did before #461 and the backout restores that disagreement knowingly; only the FORCED call witnesses the repair half, R5's gate refusing a mixed-case name before any of this is consulted. This is the CONSUMER-side rule over the producer-side entries — VOCAB-TAGS records what the vocabulary knew, MARK-DONT-STRIP what a stage decided about it, FOLDED_TAG what order to render in — and a view reads what they recorded, whichever kind it is. It sits where ONE-PREDICATE-PER-QUESTION's stated limit leaves off: where two live sites need one answer they share a predicate, but a render view always comes AFTER the decider, so the answer is recorded rather than shared — on the TOKEN, which is the views' equivalent of that entry's `ParseState.order`, no view being able to see a ParseState at all. Known limit, and the half most easily got wrong next: a token the parse never saw carries no decision to honor, so a view falls back to the vocabulary — the tell is `UNCLASSIFIED_TAG`, which `ParsedName.replace()` stamps when it splices raw text into a field and the facade's v1 pickle load stamps when it rebuilds a name from `*_list` strings. It is NOT untaggedness, since an ordinary parsed name word carries no tags either; and it is NOT `span is None`, which was tried and is wrong in the other direction — span-less means SYNTHETIC, and `Parser.revise()` builds span-less tokens from a full sub-parse whose tags it keeps on purpose, so the span reading overrode exactly the tags `revise()` exists to preserve (`revise(middle='e-f')` repaired to `e-F` where the parse gave `E-F`). A hand-built span-less token is unmarked and therefore classified, which is the same tag-driven default every other view applies. A view can only fall back if it is HANDED a vocabulary, and exactly one is: `capitalized(lexicon=...)`. `family_base` and `family_particles` are properties on ParsedName, whose fields are original/tokens/ambiguities and nothing else, so a spliced field empties the particles view and leaves the base the whole field, with `Parser.revise()` the crossing there too (docs/usage.rst says so where it documents the degradation). `initials()` is the near miss and the instructive one: it is a METHOD, so it looks like it could ask, but its signature is `(spec, delimiter, separator)` and carries no lexicon — a fallback there was written and dropped because it had to GUESS `Lexicon.default()`, and the guess erased a whole field under a caller's own vocabulary (decisions.md, under R3). `capitalized()` guesses nothing: it reads the lexicon it was handed, and only defaults to `Lexicon.default()` when the caller passes none, which is the documented meaning of omitting the argument rather than a fallback. Within a view that fallback is drawn per QUESTION (rules.md#R4's Accepted clause); which questions a view can answer at all differs per VIEW and not per question, because the views hold different amounts of the name. Whether a word is the conjunction or an initial is a property of the word, and both views answer it; whether a part is wholly particles is a property of the whole PART, which the pipeline answers once and records as UNJOINED_TAG. What `_cap_word` cannot do is RE-DERIVE that answer where no word of the part carries a tag — it is handed the whole token's tags and gates on the mark, so it is not blind to the part, it simply has no evidence to reconstruct one from — so repair leaves that half to plain particle treatment and rules.md#R4's Accepted boundary records the consequence — a spliced field is not repaired as a parsed one is, with `Parser.revise()` as the supported crossing. A fallback is right only while it answers as the pipeline would, and THAT is held by hand rather than mechanically: test_regex_sync pins the two `_INITIAL` copies to each other and to config, while the repertoire half of the pipeline's predicate (#320) is deliberately not carried across, layering forbidding the import — so a caller-added conjunction in an initialless script initials from a spliced field where the same word parsed does not. A second limit, recorded rather than closed: `_cap_word`'s PARTICLE conjunct still keys on the lexicon handed to the view rather than on the `particle` tag, so a repair run with a lexicon other than the parse's re-decides a word the parse already read — a name parsed under the default vocabulary, where `parse('juan smith vega')` reads `vega` as the family, repairs to `Juan Smith vega` when `capitalized()` is handed `Lexicon.default().add(particles={'vega'})` instead — the divergence needs the two lexicons to differ, and repairing under the parse's own lexicon gives `Juan Smith Vega`. Making it read the tag moves a boundary rules.md#R4 states in prose, so it is a separate decision and not a cleanup (decisions.md#R4, "NOT DONE"). Lives in. nameparser/_render.py (`capitalized`/`_cap_word`, `_reads_as_conjunction`, and `initials`, which honors tags and never falls back) and nameparser/_types.py (`_text_for`, `UNCLASSIFIED_TAG` and its two producers), reading what nameparser/_pipeline/ recorded — the mark those views read is recomputed producer-side in `_remarked`, which is deliberately silent about text nobody classified and is right to be. Reach for it when. A view is about to consult a Lexicon, a regex or an exception list about a word the parse already saw — or a view and a field disagree about the same parse. #408 is filed as that second shape, and was open on 2026-08-29: `initials()` walks tokens in written order where the family field applies FOLDED_TAG's ordering, so `parse("der, y van")` gave family `van der` and initials `y. d. v.` that day, one view not yet honoring the fold. It belongs here rather than argued again from scratch. ## CLAUSE-CONTENT-OVERRULES-DELIMITER — content wins diff --git a/docs/design/rules.md b/docs/design/rules.md index 671c35c8..f42565d5 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -1168,12 +1168,13 @@ R4. Rationale: case repair is a display concern, applied only on keeps its "y" lowercase. Whether a word is the conjunction or an initial is a property of the word, which a vocabulary can answer; whether a particle is acting as a particle is a property of the - whole part, and repair reads one word at a time, so that half - falls through to particle treatment and the "de la" boundary above - stands. Initials are the contrast worth knowing, and R3 states it: - that view is handed no vocabulary at all, so it falls back on - neither question and a spliced field's every word initials. - revise() classifies the + whole part, which the parse settles and records; re-deriving it + needs a reading on every word of the part, and a spliced field + has none on any, so that half falls through to particle treatment + and the "de la" boundary above stands. Initials are the contrast + worth knowing, and R3 states it: that view is handed no + vocabulary at all, so it falls back on neither question and a + spliced field's every word initials. revise() classifies the value and crosses both questions, in both views: a middle revised to "e-f" repairs to "E-F" as the parsed name does, where splicing the same text in gives "e-F". diff --git a/docs/release_log.rst b/docs/release_log.rst index 5dc3fa15..e85b8301 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -67,7 +67,7 @@ Release Log - Fix case repair lowercasing the words of a family name made only of particle words, where every other view already reads them as ordinary name words: ``HumanName("ANH DO").capitalize()`` gives ``Anh Do`` where it gave ``Anh do``, and ``"anh van do"`` gives ``Anh Van Do`` where it gave ``Anh van do``. A particle earns its name by joining forward to the word it modifies, so a part whose every word is particle vocabulary leaves none of them anything to join; the fix above already made those words anchor ``family_base`` and contribute initials, and case repair now agrees with them rather than reading the same word two ways. The test is the whole part, not a particle standing alone, which is why the two-word family in ``"anh van do"`` moves along with the one-word family in ``"ANH DO"`` -- the same Vietnamese surname, and a standing-alone test would have read it one way behind a given name and another way alone. This DIFFERS FROM 1.4.0 deliberately and does not restore it: 1.4.0 returned ``Anh do``, lowercasing on vocabulary membership alone. The accepted cost is that a degenerate family which is nothing but particles capitalizes too, so ``"juan van der"`` gives ``Juan Van Der`` where 1.4.0 gave ``Juan van der``. A conjunction is untouched by any of this, so ``"der, y van"`` gives ``y Van Der`` -- the family capitalizing while the conjunction keeps the lowercase it always had; and where the particles DO join a name word nothing changes, ``"juan de la vega"`` still giving ``Juan de la Vega``. Case repair is not one of the seven role fields the differential harness compares, so no gate run can see this change either way and none of its counts move; the ``rules.md#R4`` examples and the v1 capitalization tests are what pin it (closes #407) - - Change case repair to read the parser's own ``conjunction`` tag instead of re-deciding, from the word's spelling, whether a word is a conjunction or an initial. The parse answers that question already -- ``"Scott E. Werner"`` reads ``E.`` as an initial rather than the Italian conjunction -- and the other views honor the answer; case repair asked again, with a shape test applied to each word of a token's text rather than to the token. Two spellings of one name disagreed because of it: ``"juan e-f smith"`` capitalized to ``Juan e-F Smith`` while ``"JUAN E-F SMITH"`` gave ``Juan E-F Smith``; both give ``Juan E-F Smith`` now, ``e-f`` being a middle name and no conjunction of the parse's reading. A conjunction written as a word of its own is untouched, and so is the one-letter carve-out where it applies -- ``"juan y garcia"`` still repairs to ``Juan y Garcia``, ``"JUAN Y GARCIA"`` still to ``Juan Y Garcia``. A field assigned after the parse is unaffected: its text was never classified, so there is no reading to honor and repair asks the vocabulary, which is what every earlier version did everywhere -- ``h.last = "velasquez y garcia"`` still repairs to ``Velasquez y Garcia`` and ``h.middle = "e."`` to ``E.``. What decides which path a token takes is a mark the assignment leaves, not the absence of a span: a value revised through ``Parser.revise()`` is classified by a sub-parse and keeps its tags, so it repairs as the parse does. One reading does change for hand-built ``Token``\ s in the 2.0 API: an untagged token whose text is conjunction vocabulary is now an ordinary name word and capitalizes, where 2.1 lowercased it -- tags are what the views read, and a hand-built token that carries none is a token with nothing to declare. Case repair is not one of the seven role fields the differential harness compares, so no gate run can see this change either way and none of its counts move; measured directly instead, no name of the 1094-name differential corpus moves under ``capitalized()`` or ``capitalized(force=True)``, its uppercased and lowercased spellings included (closes #458) + - Change case repair to read the parser's own ``conjunction`` tag instead of re-deciding, from the word's spelling, whether a word is a conjunction or an initial. The parse answers that question already -- ``"Scott E. Werner"`` reads ``E.`` as an initial rather than the Italian conjunction -- and the other views honor the answer; case repair asked again, with a shape test applied to each word of a token's text rather than to the token. Two spellings of one name disagreed because of it: ``"juan e-f smith"`` capitalized to ``Juan e-F Smith`` while ``"JUAN E-F SMITH"`` gave ``Juan E-F Smith``; both give ``Juan E-F Smith`` now, ``e-f`` being a middle name and no conjunction of the parse's reading. A conjunction written as a word of its own is untouched, and so is the one-letter carve-out where it applies -- ``"juan y garcia"`` still repairs to ``Juan y Garcia``, ``"JUAN Y GARCIA"`` still to ``Juan Y Garcia``. A field assigned after the parse is unaffected: its text was never classified, so there is no reading to honor and repair asks the vocabulary, applying v1's own predicate the way every earlier version applied it everywhere -- ``h.last = "velasquez y garcia"`` still repairs to ``Velasquez y Garcia`` and ``h.middle = "e."`` to ``E.``. That is the predicate over TODAY's vocabulary, which is narrower than parity with 1.4.0 and the difference is real: ``h.last = "хосе и мария сантос"`` gives ``Хосе И Мария Сантос`` on 1.4.0 and ``Хосе и Мария Сантос`` here, because the Cyrillic ``и`` is a 2.x conjunction and was not a 1.4.0 one. What decides which path a token takes is a mark the assignment leaves, not the absence of a span: a value revised through ``Parser.revise()`` is classified by a sub-parse and keeps its tags, so it repairs as the parse does. One reading does change for hand-built ``Token``\ s in the 2.0 API: an untagged token whose text is conjunction vocabulary is now an ordinary name word and capitalizes, where 2.1 lowercased it -- tags are what the views read, and a hand-built token that carries none is a token with nothing to declare. Case repair is not one of the seven role fields the differential harness compares, so no gate run can see this change either way and none of its counts move; measured directly instead, no name of the 1094-name differential corpus moves under ``capitalized()`` or ``capitalized(force=True)``, its uppercased and lowercased spellings included -- 6564 name/spelling/lexicon rows and 13128 calls (closes #458) - Fix a tussenvoegsel attached to the family name after a comma deciding a genuinely uncertain reading and reporting nothing. ``"Van Johnson"`` reports a ``PARTICLE_OR_GIVEN`` ambiguity -- ``Van`` is a Dutch particle and a Vietnamese given name, and the parser has to pick one -- while ``"Nguyen, Thi Van"`` picked the same word the same way, silently, and lost the given name doing it. The attachment now reports the fork it decides, in the kind that names the reading it declined. A particle that could be an ordinary name reports ``PARTICLE_OR_GIVEN``: ``"Nguyen, Thi Van"``, ``"Berg, Jan van der"`` and ``"Vega, Juan de la"`` each gain one, the ``detail`` naming the ambiguous word. A particle the parser had already read as a post-nominal reports ``SUFFIX_OR_NAME`` instead, because the credential reading is what the attachment overrode: ``"Berg, Jan vd"`` gains one, ``vd`` being read as *van der* rather than as the Volunteer Decoration. Which kind you get follows the reading that was overridden rather than the word's vocabulary, so ``"Berg, Jan do"`` reports ``PARTICLE_OR_GIVEN`` even though ``do`` is a postnominal too -- it was already being read as a name word, so no credential reading was overridden. A particle where nothing was overridden reports nothing at all: ``"Jong, Piet de"`` and ``"Jong, Anke de"`` are unchanged, ``de`` being no name in any reading and no postnominal either, and so is ``"Berg, Jan de vd"``, whose run was read as name words whole. Worth knowing before you filter on this: ``"Beethoven, Ludwig van"`` -- the textbook Dutch listing, read exactly right -- now carries a report too. It is the same string shape over the same vocabulary as ``"Nguyen, Thi Van"``, and nothing in the input separates them, so a report on one is a report on both. This adds the report and nothing else: every field these names parse to is exactly what the tussenvoegsel fix below already gave them, and ``ambiguities`` is the only value that grows. Seven differential corpus names gain a kind against the 2.0.0 and 2.1.0 baselines, seven of the nine the tussenvoegsel fix below already moved -- five of ``fix(#379)``'s seven and both of ``fix(#380)``'s two, now carrying ``_ambiguities`` in their diff as well; the two left out are the pair named unchanged above, ``Jong, Piet de`` and ``Jong, Anke de`` -- and none against 1.4.0, which had no ambiguity reporting at all (closes #405) From e39d8a342b84a129d84a11238b1c74f94cfe5dca Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sat, 29 Aug 2026 23:34:49 -0700 Subject: [PATCH 10/14] test(capitalization): pin the one name a pickle round trip changes #458 moved the conjunction-versus-initial decision into the parse, and the facade's pickle carries the *_list strings and no tags, so a restored name is repaired the way 1.4.0 repaired everything -- per word. `juan e-f smith` capitalizes to `Juan E-F Smith` directly and `Juan e-F Smith` after a round trip. That is the pickle contract (strings only, never a re-parse) meeting the tag read; pinned so it is not rediscovered as a defect in either. Co-Authored-By: Claude Opus 5 --- tests/test_capitalization.py | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/tests/test_capitalization.py b/tests/test_capitalization.py index 44de2c55..22d93d5e 100644 --- a/tests/test_capitalization.py +++ b/tests/test_capitalization.py @@ -326,3 +326,19 @@ def test_a_restored_pickle_keeps_v1_conjunction_repair(self) -> None: restored = pickle.loads(pickle.dumps(HumanName(text))) restored.capitalize(force=True) self.m(str(restored), want, restored) + + # The ONE name a pickle round trip does change, pinned so it is not + # rediscovered as a bug. #458 moved the conjunction-versus-initial + # decision into the parse, and a pickle carries no tags, so the + # restored name is repaired the way 1.4.0 repaired everything -- + # per word of the text, giving the Italian conjunction inside a + # hyphenated middle name. It is the pickle contract (strings only, + # never a re-parse) meeting the tag read, not a defect in either. + # 1.4.0 gave 'Juan e-F Smith' both ways. + def test_a_pickle_round_trip_loses_the_e_f_reading(self) -> None: + direct = HumanName('juan e-f smith') + direct.capitalize(force=True) + self.m(str(direct), 'Juan E-F Smith', direct) + restored = pickle.loads(pickle.dumps(HumanName('juan e-f smith'))) + restored.capitalize(force=True) + self.m(str(restored), 'Juan e-F Smith', restored) From cba0e9a2739b609bcd170a57136a01a0befc7ee6 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sat, 29 Aug 2026 23:36:22 -0700 Subject: [PATCH 11/14] docs(render,types): two docstring precisions from a self-review 'a view that HOLDS a vocabulary' -> 'is HANDED one', and name which view that is; 'initials all four words' -> 'every word of that field', the family being three words and the fourth initial the given name's. Co-Authored-By: Claude Opus 5 --- nameparser/_render.py | 5 +++-- nameparser/_types.py | 8 +++++--- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/nameparser/_render.py b/nameparser/_render.py index 08a4748a..d26eb76c 100644 --- a/nameparser/_render.py +++ b/nameparser/_render.py @@ -129,8 +129,9 @@ def initials(name: ParsedName, spec: str, delimiter: str, separator: str) -> str hand-built untagged tokens all contribute, and so do the words of a field spliced in by replace(), which the parse never read. This view takes NO lexicon, so it has none to fall back to for - that text: `replace(family='de la vega')` initials all four words - where the same name parsed gives 'j. v.' (rules.md#R3's Accepted + that text: `replace(family='de la vega')` initials every word of + that field where the same name parsed gives 'j. v.' + (rules.md#R3's Accepted clause, and decisions.md under R3 for why the fallback was tried and dropped). Valid spec keys: given, middle, family.""" if not isinstance(delimiter, str): diff --git a/nameparser/_types.py b/nameparser/_types.py index d194b6f6..745ff06d 100644 --- a/nameparser/_types.py +++ b/nameparser/_types.py @@ -118,9 +118,11 @@ def __add__(self, other: object) -> NoReturn: # type: ignore[override] #: Raw text spliced into a field after the parse, which no parse ever #: read: `ParsedName.replace()` stamps it, and so does the facade's #: v1 pickle load, which rebuilds a name from `*_list` strings alone. -#: A view that HOLDS a vocabulary falls back to it for a token carrying -#: this -- there is no decision to honor -- and reads every other token -#: by its tags (mechanisms.md#RENDER-HONORS-THE-PARSE). +#: A view that is HANDED a vocabulary falls back to it for a token +#: carrying this -- there is no decision to honor -- and reads every +#: other token by its tags (mechanisms.md#RENDER-HONORS-THE-PARSE). +#: `capitalized(lexicon=...)` is the one such view; `initials()` takes +#: no lexicon and so honors tags alone, marked or not. #: The absence of a SPAN is not that signal and was tried as one: a #: span-less token is SYNTHETIC, which `Parser.revise()` also builds, #: from a full sub-parse whose tags it deliberately keeps. Keying the From 20be4a24097c95a3025b95167becc526e68445ad Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sat, 29 Aug 2026 23:37:12 -0700 Subject: [PATCH 12/14] docs(mechanisms): the repertoire divergence is now unwitnessable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The clause said a caller-added conjunction in an initialless script initials from a spliced field where the same word parsed does not. `initials()` no longer falls back, so case repair is the divergence's only reader, and there the two paths differ by lower() versus capitalize() over a caseless script. Measured: _INITIAL matches '씨.', _vocab.is_initial does not, and '씨.'.lower() == '씨.'.capitalize(). Co-Authored-By: Claude Opus 5 --- docs/design/mechanisms.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 77286130..93ff4eda 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -59,7 +59,7 @@ Problem shape. Two stages need the same answer about the same input, and the one ## RENDER-HONORS-THE-PARSE — the parse decides it, the views honor it -Problem shape. A render view needs a fact the parse already settled — whether a word is the conjunction or an initial, whether a particle is acting as one, which word renders first. Contract statement. The parse decides it; the render views honor those decisions and never re-evaluate them. Two directions break that, and each has been found here as a defect: a view RE-DERIVES the answer from the text, keeping its own copy of a pipeline predicate — that one shipped through 2.0 and 2.1 — or a view honors the record and then OVERRIDES it, readmitting what the decision excluded, which is filed and open here. How it works. Re-deriving fails because the two copies stop being the same question long before anyone notices they are two: `_cap_word` re-ran the conjunction-versus-initial decision from the word's spelling against a hand-maintained copy of the pipeline's `_INITIAL` pattern while classify had already answered it and recorded it on the token, and `_classify.py` asks `is_initial()`, the shape test ANDed with a script-repertoire test since #320, where `_render.py` asked the bare pattern, and asked it per WORD of a token's text rather than per token, so `juan e-f smith` repaired to `Juan e-F Smith` (#458). Overriding fails more directly, and this entry's instance of it is OPEN rather than closed: `initials()` honors the `conjunction` tag through `_SKIP_TAGS` and then readmits the token whenever it also carries UNJOINED_TAG, the mark of an all-particle part, where rules.md#R3 excludes a conjunction "even then" — so under a caller's vocabulary that puts a word in both sets the view readmits what the rule excluded, and under `Lexicon.default().add(particles={'y'})` — the lexicon those two readings need, and they hold under no other — `parse("Anh y Van")` initialed `A. y. V.` and `parse("Juan de y")` `J. d. y.` on 2026-08-29. Read those two values scoped to that lexicon or they are simply false: under the DEFAULT vocabulary the same strings give `A. V.` and `J.`, and the second is a rules.md#R3 example line the doc runner asserts. #461 is where that stands, and it is worth reading before re-fixing it: the narrowing was written, measured and BACKED OUT in the same PR, because honoring R3 there cost `initials()` its agreement with `family_base`, which reads that same `de y` as the base — this entry's other shape, arrived at from the render side, and the reason the question moved from the code to R3's clause (decisions.md carries the argument). What to carry away is the DIRECTION and not its verdict here: a view that honors a record and then readmits what the record excluded is overriding a decision it never took, whichever way this one settles. Case repair reads that same token and does NOT readmit it — `capitalized(force=True)` on `Anh y Van` gives `Anh y Van` under that same lexicon, R4 carrying the carve-out in its own words but ON R3's authority — its text reads "being no name word in any part — the carve-out R3 states for initials", so the two rules stand or fall together and a change to R3's clause reaches R4 — so the two views disagree today about that token exactly as they did before #461 and the backout restores that disagreement knowingly; only the FORCED call witnesses the repair half, R5's gate refusing a mixed-case name before any of this is consulted. This is the CONSUMER-side rule over the producer-side entries — VOCAB-TAGS records what the vocabulary knew, MARK-DONT-STRIP what a stage decided about it, FOLDED_TAG what order to render in — and a view reads what they recorded, whichever kind it is. It sits where ONE-PREDICATE-PER-QUESTION's stated limit leaves off: where two live sites need one answer they share a predicate, but a render view always comes AFTER the decider, so the answer is recorded rather than shared — on the TOKEN, which is the views' equivalent of that entry's `ParseState.order`, no view being able to see a ParseState at all. Known limit, and the half most easily got wrong next: a token the parse never saw carries no decision to honor, so a view falls back to the vocabulary — the tell is `UNCLASSIFIED_TAG`, which `ParsedName.replace()` stamps when it splices raw text into a field and the facade's v1 pickle load stamps when it rebuilds a name from `*_list` strings. It is NOT untaggedness, since an ordinary parsed name word carries no tags either; and it is NOT `span is None`, which was tried and is wrong in the other direction — span-less means SYNTHETIC, and `Parser.revise()` builds span-less tokens from a full sub-parse whose tags it keeps on purpose, so the span reading overrode exactly the tags `revise()` exists to preserve (`revise(middle='e-f')` repaired to `e-F` where the parse gave `E-F`). A hand-built span-less token is unmarked and therefore classified, which is the same tag-driven default every other view applies. A view can only fall back if it is HANDED a vocabulary, and exactly one is: `capitalized(lexicon=...)`. `family_base` and `family_particles` are properties on ParsedName, whose fields are original/tokens/ambiguities and nothing else, so a spliced field empties the particles view and leaves the base the whole field, with `Parser.revise()` the crossing there too (docs/usage.rst says so where it documents the degradation). `initials()` is the near miss and the instructive one: it is a METHOD, so it looks like it could ask, but its signature is `(spec, delimiter, separator)` and carries no lexicon — a fallback there was written and dropped because it had to GUESS `Lexicon.default()`, and the guess erased a whole field under a caller's own vocabulary (decisions.md, under R3). `capitalized()` guesses nothing: it reads the lexicon it was handed, and only defaults to `Lexicon.default()` when the caller passes none, which is the documented meaning of omitting the argument rather than a fallback. Within a view that fallback is drawn per QUESTION (rules.md#R4's Accepted clause); which questions a view can answer at all differs per VIEW and not per question, because the views hold different amounts of the name. Whether a word is the conjunction or an initial is a property of the word, and both views answer it; whether a part is wholly particles is a property of the whole PART, which the pipeline answers once and records as UNJOINED_TAG. What `_cap_word` cannot do is RE-DERIVE that answer where no word of the part carries a tag — it is handed the whole token's tags and gates on the mark, so it is not blind to the part, it simply has no evidence to reconstruct one from — so repair leaves that half to plain particle treatment and rules.md#R4's Accepted boundary records the consequence — a spliced field is not repaired as a parsed one is, with `Parser.revise()` as the supported crossing. A fallback is right only while it answers as the pipeline would, and THAT is held by hand rather than mechanically: test_regex_sync pins the two `_INITIAL` copies to each other and to config, while the repertoire half of the pipeline's predicate (#320) is deliberately not carried across, layering forbidding the import — so a caller-added conjunction in an initialless script initials from a spliced field where the same word parsed does not. A second limit, recorded rather than closed: `_cap_word`'s PARTICLE conjunct still keys on the lexicon handed to the view rather than on the `particle` tag, so a repair run with a lexicon other than the parse's re-decides a word the parse already read — a name parsed under the default vocabulary, where `parse('juan smith vega')` reads `vega` as the family, repairs to `Juan Smith vega` when `capitalized()` is handed `Lexicon.default().add(particles={'vega'})` instead — the divergence needs the two lexicons to differ, and repairing under the parse's own lexicon gives `Juan Smith Vega`. Making it read the tag moves a boundary rules.md#R4 states in prose, so it is a separate decision and not a cleanup (decisions.md#R4, "NOT DONE"). Lives in. nameparser/_render.py (`capitalized`/`_cap_word`, `_reads_as_conjunction`, and `initials`, which honors tags and never falls back) and nameparser/_types.py (`_text_for`, `UNCLASSIFIED_TAG` and its two producers), reading what nameparser/_pipeline/ recorded — the mark those views read is recomputed producer-side in `_remarked`, which is deliberately silent about text nobody classified and is right to be. Reach for it when. A view is about to consult a Lexicon, a regex or an exception list about a word the parse already saw — or a view and a field disagree about the same parse. #408 is filed as that second shape, and was open on 2026-08-29: `initials()` walks tokens in written order where the family field applies FOLDED_TAG's ordering, so `parse("der, y van")` gave family `van der` and initials `y. d. v.` that day, one view not yet honoring the fold. It belongs here rather than argued again from scratch. +Problem shape. A render view needs a fact the parse already settled — whether a word is the conjunction or an initial, whether a particle is acting as one, which word renders first. Contract statement. The parse decides it; the render views honor those decisions and never re-evaluate them. Two directions break that, and each has been found here as a defect: a view RE-DERIVES the answer from the text, keeping its own copy of a pipeline predicate — that one shipped through 2.0 and 2.1 — or a view honors the record and then OVERRIDES it, readmitting what the decision excluded, which is filed and open here. How it works. Re-deriving fails because the two copies stop being the same question long before anyone notices they are two: `_cap_word` re-ran the conjunction-versus-initial decision from the word's spelling against a hand-maintained copy of the pipeline's `_INITIAL` pattern while classify had already answered it and recorded it on the token, and `_classify.py` asks `is_initial()`, the shape test ANDed with a script-repertoire test since #320, where `_render.py` asked the bare pattern, and asked it per WORD of a token's text rather than per token, so `juan e-f smith` repaired to `Juan e-F Smith` (#458). Overriding fails more directly, and this entry's instance of it is OPEN rather than closed: `initials()` honors the `conjunction` tag through `_SKIP_TAGS` and then readmits the token whenever it also carries UNJOINED_TAG, the mark of an all-particle part, where rules.md#R3 excludes a conjunction "even then" — so under a caller's vocabulary that puts a word in both sets the view readmits what the rule excluded, and under `Lexicon.default().add(particles={'y'})` — the lexicon those two readings need, and they hold under no other — `parse("Anh y Van")` initialed `A. y. V.` and `parse("Juan de y")` `J. d. y.` on 2026-08-29. Read those two values scoped to that lexicon or they are simply false: under the DEFAULT vocabulary the same strings give `A. V.` and `J.`, and the second is a rules.md#R3 example line the doc runner asserts. #461 is where that stands, and it is worth reading before re-fixing it: the narrowing was written, measured and BACKED OUT in the same PR, because honoring R3 there cost `initials()` its agreement with `family_base`, which reads that same `de y` as the base — this entry's other shape, arrived at from the render side, and the reason the question moved from the code to R3's clause (decisions.md carries the argument). What to carry away is the DIRECTION and not its verdict here: a view that honors a record and then readmits what the record excluded is overriding a decision it never took, whichever way this one settles. Case repair reads that same token and does NOT readmit it — `capitalized(force=True)` on `Anh y Van` gives `Anh y Van` under that same lexicon, R4 carrying the carve-out in its own words but ON R3's authority — its text reads "being no name word in any part — the carve-out R3 states for initials", so the two rules stand or fall together and a change to R3's clause reaches R4 — so the two views disagree today about that token exactly as they did before #461 and the backout restores that disagreement knowingly; only the FORCED call witnesses the repair half, R5's gate refusing a mixed-case name before any of this is consulted. This is the CONSUMER-side rule over the producer-side entries — VOCAB-TAGS records what the vocabulary knew, MARK-DONT-STRIP what a stage decided about it, FOLDED_TAG what order to render in — and a view reads what they recorded, whichever kind it is. It sits where ONE-PREDICATE-PER-QUESTION's stated limit leaves off: where two live sites need one answer they share a predicate, but a render view always comes AFTER the decider, so the answer is recorded rather than shared — on the TOKEN, which is the views' equivalent of that entry's `ParseState.order`, no view being able to see a ParseState at all. Known limit, and the half most easily got wrong next: a token the parse never saw carries no decision to honor, so a view falls back to the vocabulary — the tell is `UNCLASSIFIED_TAG`, which `ParsedName.replace()` stamps when it splices raw text into a field and the facade's v1 pickle load stamps when it rebuilds a name from `*_list` strings. It is NOT untaggedness, since an ordinary parsed name word carries no tags either; and it is NOT `span is None`, which was tried and is wrong in the other direction — span-less means SYNTHETIC, and `Parser.revise()` builds span-less tokens from a full sub-parse whose tags it keeps on purpose, so the span reading overrode exactly the tags `revise()` exists to preserve (`revise(middle='e-f')` repaired to `e-F` where the parse gave `E-F`). A hand-built span-less token is unmarked and therefore classified, which is the same tag-driven default every other view applies. A view can only fall back if it is HANDED a vocabulary, and exactly one is: `capitalized(lexicon=...)`. `family_base` and `family_particles` are properties on ParsedName, whose fields are original/tokens/ambiguities and nothing else, so a spliced field empties the particles view and leaves the base the whole field, with `Parser.revise()` the crossing there too (docs/usage.rst says so where it documents the degradation). `initials()` is the near miss and the instructive one: it is a METHOD, so it looks like it could ask, but its signature is `(spec, delimiter, separator)` and carries no lexicon — a fallback there was written and dropped because it had to GUESS `Lexicon.default()`, and the guess erased a whole field under a caller's own vocabulary (decisions.md, under R3). `capitalized()` guesses nothing: it reads the lexicon it was handed, and only defaults to `Lexicon.default()` when the caller passes none, which is the documented meaning of omitting the argument rather than a fallback. Within a view that fallback is drawn per QUESTION (rules.md#R4's Accepted clause); which questions a view can answer at all differs per VIEW and not per question, because the views hold different amounts of the name. Whether a word is the conjunction or an initial is a property of the word, and both views answer it; whether a part is wholly particles is a property of the whole PART, which the pipeline answers once and records as UNJOINED_TAG. What `_cap_word` cannot do is RE-DERIVE that answer where no word of the part carries a tag — it is handed the whole token's tags and gates on the mark, so it is not blind to the part, it simply has no evidence to reconstruct one from — so repair leaves that half to plain particle treatment and rules.md#R4's Accepted boundary records the consequence — a spliced field is not repaired as a parsed one is, with `Parser.revise()` as the supported crossing. A fallback is right only while it answers as the pipeline would, and THAT is held by hand rather than mechanically: test_regex_sync pins the two `_INITIAL` copies to each other and to config, while the repertoire half of the pipeline's predicate (#320) is deliberately not carried across, layering forbidding the import. What that divergence can reach is now nothing observable: it needs a caller-added conjunction written initial-SHAPED in a script that has no initials (`씨.`), and case repair is the fallback's only reader, so the two paths differ by `lower()` versus `capitalize()` over a caseless script — the same string either way. `initials()` used to be the reader that could witness it, and no longer falls back at all. A second limit, recorded rather than closed: `_cap_word`'s PARTICLE conjunct still keys on the lexicon handed to the view rather than on the `particle` tag, so a repair run with a lexicon other than the parse's re-decides a word the parse already read — a name parsed under the default vocabulary, where `parse('juan smith vega')` reads `vega` as the family, repairs to `Juan Smith vega` when `capitalized()` is handed `Lexicon.default().add(particles={'vega'})` instead — the divergence needs the two lexicons to differ, and repairing under the parse's own lexicon gives `Juan Smith Vega`. Making it read the tag moves a boundary rules.md#R4 states in prose, so it is a separate decision and not a cleanup (decisions.md#R4, "NOT DONE"). Lives in. nameparser/_render.py (`capitalized`/`_cap_word`, `_reads_as_conjunction`, and `initials`, which honors tags and never falls back) and nameparser/_types.py (`_text_for`, `UNCLASSIFIED_TAG` and its two producers), reading what nameparser/_pipeline/ recorded — the mark those views read is recomputed producer-side in `_remarked`, which is deliberately silent about text nobody classified and is right to be. Reach for it when. A view is about to consult a Lexicon, a regex or an exception list about a word the parse already saw — or a view and a field disagree about the same parse. #408 is filed as that second shape, and was open on 2026-08-29: `initials()` walks tokens in written order where the family field applies FOLDED_TAG's ordering, so `parse("der, y van")` gave family `van der` and initials `y. d. v.` that day, one view not yet honoring the fold. It belongs here rather than argued again from scratch. ## CLAUSE-CONTENT-OVERRULES-DELIMITER — content wins From a8a61a3afaa959ef8e98fe2755281b9427bfc652 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sat, 29 Aug 2026 23:40:10 -0700 Subject: [PATCH 13/14] docs(mechanisms): two more sentences the fallback removal falsified "both views answer it" -- only case repair does now; and "the views hold different amounts of the name" was never the reason, they hold different amounts of VOCABULARY. Also names which conjunct \`_cap_word\` gates on UNJOINED_TAG, "the mark" now being ambiguous between two. Co-Authored-By: Claude Opus 5 --- docs/design/mechanisms.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 93ff4eda..80e79503 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -59,7 +59,7 @@ Problem shape. Two stages need the same answer about the same input, and the one ## RENDER-HONORS-THE-PARSE — the parse decides it, the views honor it -Problem shape. A render view needs a fact the parse already settled — whether a word is the conjunction or an initial, whether a particle is acting as one, which word renders first. Contract statement. The parse decides it; the render views honor those decisions and never re-evaluate them. Two directions break that, and each has been found here as a defect: a view RE-DERIVES the answer from the text, keeping its own copy of a pipeline predicate — that one shipped through 2.0 and 2.1 — or a view honors the record and then OVERRIDES it, readmitting what the decision excluded, which is filed and open here. How it works. Re-deriving fails because the two copies stop being the same question long before anyone notices they are two: `_cap_word` re-ran the conjunction-versus-initial decision from the word's spelling against a hand-maintained copy of the pipeline's `_INITIAL` pattern while classify had already answered it and recorded it on the token, and `_classify.py` asks `is_initial()`, the shape test ANDed with a script-repertoire test since #320, where `_render.py` asked the bare pattern, and asked it per WORD of a token's text rather than per token, so `juan e-f smith` repaired to `Juan e-F Smith` (#458). Overriding fails more directly, and this entry's instance of it is OPEN rather than closed: `initials()` honors the `conjunction` tag through `_SKIP_TAGS` and then readmits the token whenever it also carries UNJOINED_TAG, the mark of an all-particle part, where rules.md#R3 excludes a conjunction "even then" — so under a caller's vocabulary that puts a word in both sets the view readmits what the rule excluded, and under `Lexicon.default().add(particles={'y'})` — the lexicon those two readings need, and they hold under no other — `parse("Anh y Van")` initialed `A. y. V.` and `parse("Juan de y")` `J. d. y.` on 2026-08-29. Read those two values scoped to that lexicon or they are simply false: under the DEFAULT vocabulary the same strings give `A. V.` and `J.`, and the second is a rules.md#R3 example line the doc runner asserts. #461 is where that stands, and it is worth reading before re-fixing it: the narrowing was written, measured and BACKED OUT in the same PR, because honoring R3 there cost `initials()` its agreement with `family_base`, which reads that same `de y` as the base — this entry's other shape, arrived at from the render side, and the reason the question moved from the code to R3's clause (decisions.md carries the argument). What to carry away is the DIRECTION and not its verdict here: a view that honors a record and then readmits what the record excluded is overriding a decision it never took, whichever way this one settles. Case repair reads that same token and does NOT readmit it — `capitalized(force=True)` on `Anh y Van` gives `Anh y Van` under that same lexicon, R4 carrying the carve-out in its own words but ON R3's authority — its text reads "being no name word in any part — the carve-out R3 states for initials", so the two rules stand or fall together and a change to R3's clause reaches R4 — so the two views disagree today about that token exactly as they did before #461 and the backout restores that disagreement knowingly; only the FORCED call witnesses the repair half, R5's gate refusing a mixed-case name before any of this is consulted. This is the CONSUMER-side rule over the producer-side entries — VOCAB-TAGS records what the vocabulary knew, MARK-DONT-STRIP what a stage decided about it, FOLDED_TAG what order to render in — and a view reads what they recorded, whichever kind it is. It sits where ONE-PREDICATE-PER-QUESTION's stated limit leaves off: where two live sites need one answer they share a predicate, but a render view always comes AFTER the decider, so the answer is recorded rather than shared — on the TOKEN, which is the views' equivalent of that entry's `ParseState.order`, no view being able to see a ParseState at all. Known limit, and the half most easily got wrong next: a token the parse never saw carries no decision to honor, so a view falls back to the vocabulary — the tell is `UNCLASSIFIED_TAG`, which `ParsedName.replace()` stamps when it splices raw text into a field and the facade's v1 pickle load stamps when it rebuilds a name from `*_list` strings. It is NOT untaggedness, since an ordinary parsed name word carries no tags either; and it is NOT `span is None`, which was tried and is wrong in the other direction — span-less means SYNTHETIC, and `Parser.revise()` builds span-less tokens from a full sub-parse whose tags it keeps on purpose, so the span reading overrode exactly the tags `revise()` exists to preserve (`revise(middle='e-f')` repaired to `e-F` where the parse gave `E-F`). A hand-built span-less token is unmarked and therefore classified, which is the same tag-driven default every other view applies. A view can only fall back if it is HANDED a vocabulary, and exactly one is: `capitalized(lexicon=...)`. `family_base` and `family_particles` are properties on ParsedName, whose fields are original/tokens/ambiguities and nothing else, so a spliced field empties the particles view and leaves the base the whole field, with `Parser.revise()` the crossing there too (docs/usage.rst says so where it documents the degradation). `initials()` is the near miss and the instructive one: it is a METHOD, so it looks like it could ask, but its signature is `(spec, delimiter, separator)` and carries no lexicon — a fallback there was written and dropped because it had to GUESS `Lexicon.default()`, and the guess erased a whole field under a caller's own vocabulary (decisions.md, under R3). `capitalized()` guesses nothing: it reads the lexicon it was handed, and only defaults to `Lexicon.default()` when the caller passes none, which is the documented meaning of omitting the argument rather than a fallback. Within a view that fallback is drawn per QUESTION (rules.md#R4's Accepted clause); which questions a view can answer at all differs per VIEW and not per question, because the views hold different amounts of the name. Whether a word is the conjunction or an initial is a property of the word, and both views answer it; whether a part is wholly particles is a property of the whole PART, which the pipeline answers once and records as UNJOINED_TAG. What `_cap_word` cannot do is RE-DERIVE that answer where no word of the part carries a tag — it is handed the whole token's tags and gates on the mark, so it is not blind to the part, it simply has no evidence to reconstruct one from — so repair leaves that half to plain particle treatment and rules.md#R4's Accepted boundary records the consequence — a spliced field is not repaired as a parsed one is, with `Parser.revise()` as the supported crossing. A fallback is right only while it answers as the pipeline would, and THAT is held by hand rather than mechanically: test_regex_sync pins the two `_INITIAL` copies to each other and to config, while the repertoire half of the pipeline's predicate (#320) is deliberately not carried across, layering forbidding the import. What that divergence can reach is now nothing observable: it needs a caller-added conjunction written initial-SHAPED in a script that has no initials (`씨.`), and case repair is the fallback's only reader, so the two paths differ by `lower()` versus `capitalize()` over a caseless script — the same string either way. `initials()` used to be the reader that could witness it, and no longer falls back at all. A second limit, recorded rather than closed: `_cap_word`'s PARTICLE conjunct still keys on the lexicon handed to the view rather than on the `particle` tag, so a repair run with a lexicon other than the parse's re-decides a word the parse already read — a name parsed under the default vocabulary, where `parse('juan smith vega')` reads `vega` as the family, repairs to `Juan Smith vega` when `capitalized()` is handed `Lexicon.default().add(particles={'vega'})` instead — the divergence needs the two lexicons to differ, and repairing under the parse's own lexicon gives `Juan Smith Vega`. Making it read the tag moves a boundary rules.md#R4 states in prose, so it is a separate decision and not a cleanup (decisions.md#R4, "NOT DONE"). Lives in. nameparser/_render.py (`capitalized`/`_cap_word`, `_reads_as_conjunction`, and `initials`, which honors tags and never falls back) and nameparser/_types.py (`_text_for`, `UNCLASSIFIED_TAG` and its two producers), reading what nameparser/_pipeline/ recorded — the mark those views read is recomputed producer-side in `_remarked`, which is deliberately silent about text nobody classified and is right to be. Reach for it when. A view is about to consult a Lexicon, a regex or an exception list about a word the parse already saw — or a view and a field disagree about the same parse. #408 is filed as that second shape, and was open on 2026-08-29: `initials()` walks tokens in written order where the family field applies FOLDED_TAG's ordering, so `parse("der, y van")` gave family `van der` and initials `y. d. v.` that day, one view not yet honoring the fold. It belongs here rather than argued again from scratch. +Problem shape. A render view needs a fact the parse already settled — whether a word is the conjunction or an initial, whether a particle is acting as one, which word renders first. Contract statement. The parse decides it; the render views honor those decisions and never re-evaluate them. Two directions break that, and each has been found here as a defect: a view RE-DERIVES the answer from the text, keeping its own copy of a pipeline predicate — that one shipped through 2.0 and 2.1 — or a view honors the record and then OVERRIDES it, readmitting what the decision excluded, which is filed and open here. How it works. Re-deriving fails because the two copies stop being the same question long before anyone notices they are two: `_cap_word` re-ran the conjunction-versus-initial decision from the word's spelling against a hand-maintained copy of the pipeline's `_INITIAL` pattern while classify had already answered it and recorded it on the token, and `_classify.py` asks `is_initial()`, the shape test ANDed with a script-repertoire test since #320, where `_render.py` asked the bare pattern, and asked it per WORD of a token's text rather than per token, so `juan e-f smith` repaired to `Juan e-F Smith` (#458). Overriding fails more directly, and this entry's instance of it is OPEN rather than closed: `initials()` honors the `conjunction` tag through `_SKIP_TAGS` and then readmits the token whenever it also carries UNJOINED_TAG, the mark of an all-particle part, where rules.md#R3 excludes a conjunction "even then" — so under a caller's vocabulary that puts a word in both sets the view readmits what the rule excluded, and under `Lexicon.default().add(particles={'y'})` — the lexicon those two readings need, and they hold under no other — `parse("Anh y Van")` initialed `A. y. V.` and `parse("Juan de y")` `J. d. y.` on 2026-08-29. Read those two values scoped to that lexicon or they are simply false: under the DEFAULT vocabulary the same strings give `A. V.` and `J.`, and the second is a rules.md#R3 example line the doc runner asserts. #461 is where that stands, and it is worth reading before re-fixing it: the narrowing was written, measured and BACKED OUT in the same PR, because honoring R3 there cost `initials()` its agreement with `family_base`, which reads that same `de y` as the base — this entry's other shape, arrived at from the render side, and the reason the question moved from the code to R3's clause (decisions.md carries the argument). What to carry away is the DIRECTION and not its verdict here: a view that honors a record and then readmits what the record excluded is overriding a decision it never took, whichever way this one settles. Case repair reads that same token and does NOT readmit it — `capitalized(force=True)` on `Anh y Van` gives `Anh y Van` under that same lexicon, R4 carrying the carve-out in its own words but ON R3's authority — its text reads "being no name word in any part — the carve-out R3 states for initials", so the two rules stand or fall together and a change to R3's clause reaches R4 — so the two views disagree today about that token exactly as they did before #461 and the backout restores that disagreement knowingly; only the FORCED call witnesses the repair half, R5's gate refusing a mixed-case name before any of this is consulted. This is the CONSUMER-side rule over the producer-side entries — VOCAB-TAGS records what the vocabulary knew, MARK-DONT-STRIP what a stage decided about it, FOLDED_TAG what order to render in — and a view reads what they recorded, whichever kind it is. It sits where ONE-PREDICATE-PER-QUESTION's stated limit leaves off: where two live sites need one answer they share a predicate, but a render view always comes AFTER the decider, so the answer is recorded rather than shared — on the TOKEN, which is the views' equivalent of that entry's `ParseState.order`, no view being able to see a ParseState at all. Known limit, and the half most easily got wrong next: a token the parse never saw carries no decision to honor, so a view falls back to the vocabulary — the tell is `UNCLASSIFIED_TAG`, which `ParsedName.replace()` stamps when it splices raw text into a field and the facade's v1 pickle load stamps when it rebuilds a name from `*_list` strings. It is NOT untaggedness, since an ordinary parsed name word carries no tags either; and it is NOT `span is None`, which was tried and is wrong in the other direction — span-less means SYNTHETIC, and `Parser.revise()` builds span-less tokens from a full sub-parse whose tags it keeps on purpose, so the span reading overrode exactly the tags `revise()` exists to preserve (`revise(middle='e-f')` repaired to `e-F` where the parse gave `E-F`). A hand-built span-less token is unmarked and therefore classified, which is the same tag-driven default every other view applies. A view can only fall back if it is HANDED a vocabulary, and exactly one is: `capitalized(lexicon=...)`. `family_base` and `family_particles` are properties on ParsedName, whose fields are original/tokens/ambiguities and nothing else, so a spliced field empties the particles view and leaves the base the whole field, with `Parser.revise()` the crossing there too (docs/usage.rst says so where it documents the degradation). `initials()` is the near miss and the instructive one: it is a METHOD, so it looks like it could ask, but its signature is `(spec, delimiter, separator)` and carries no lexicon — a fallback there was written and dropped because it had to GUESS `Lexicon.default()`, and the guess erased a whole field under a caller's own vocabulary (decisions.md, under R3). `capitalized()` guesses nothing: it reads the lexicon it was handed, and only defaults to `Lexicon.default()` when the caller passes none, which is the documented meaning of omitting the argument rather than a fallback. Within the one view that can fall back, the fallback is drawn per QUESTION (rules.md#R4's Accepted clause). Whether a word is the conjunction or an initial is a property of the word, which a vocabulary answers alone, so case repair asks it. Whether a part is wholly particles is a property of the whole PART, which the pipeline answers once and records as UNJOINED_TAG. What `_cap_word` cannot do is RE-DERIVE that answer where no word of the part carries a tag — it is handed the whole token's tags and gates the particle conjunct on UNJOINED_TAG, so it is not blind to the part, it simply has no evidence to reconstruct one from — so repair leaves that half to plain particle treatment and rules.md#R4's Accepted boundary records the consequence — a spliced field is not repaired as a parsed one is, with `Parser.revise()` as the supported crossing. A fallback is right only while it answers as the pipeline would, and THAT is held by hand rather than mechanically: test_regex_sync pins the two `_INITIAL` copies to each other and to config, while the repertoire half of the pipeline's predicate (#320) is deliberately not carried across, layering forbidding the import. What that divergence can reach is now nothing observable: it needs a caller-added conjunction written initial-SHAPED in a script that has no initials (`씨.`), and case repair is the fallback's only reader, so the two paths differ by `lower()` versus `capitalize()` over a caseless script — the same string either way. `initials()` used to be the reader that could witness it, and no longer falls back at all. A second limit, recorded rather than closed: `_cap_word`'s PARTICLE conjunct still keys on the lexicon handed to the view rather than on the `particle` tag, so a repair run with a lexicon other than the parse's re-decides a word the parse already read — a name parsed under the default vocabulary, where `parse('juan smith vega')` reads `vega` as the family, repairs to `Juan Smith vega` when `capitalized()` is handed `Lexicon.default().add(particles={'vega'})` instead — the divergence needs the two lexicons to differ, and repairing under the parse's own lexicon gives `Juan Smith Vega`. Making it read the tag moves a boundary rules.md#R4 states in prose, so it is a separate decision and not a cleanup (decisions.md#R4, "NOT DONE"). Lives in. nameparser/_render.py (`capitalized`/`_cap_word`, `_reads_as_conjunction`, and `initials`, which honors tags and never falls back) and nameparser/_types.py (`_text_for`, `UNCLASSIFIED_TAG` and its two producers), reading what nameparser/_pipeline/ recorded — the mark those views read is recomputed producer-side in `_remarked`, which is deliberately silent about text nobody classified and is right to be. Reach for it when. A view is about to consult a Lexicon, a regex or an exception list about a word the parse already saw — or a view and a field disagree about the same parse. #408 is filed as that second shape, and was open on 2026-08-29: `initials()` walks tokens in written order where the family field applies FOLDED_TAG's ordering, so `parse("der, y van")` gave family `van der` and initials `y. d. v.` that day, one view not yet honoring the fold. It belongs here rather than argued again from scratch. ## CLAUSE-CONTENT-OVERRULES-DELIMITER — content wins From 122e75d2bfc12e0ac5f7e1d18021262ed1179352 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sat, 29 Aug 2026 23:59:41 -0700 Subject: [PATCH 14/14] docs: the seven findings a design-docs review of #463 raised All seven verified against HEAD before editing; all seven still held. 1. decisions.md#R4's dropped-fallback bullet said "An issue is filed for that" -- now true, and named: #464. The same claim in test_initials_has_no_lexicon_so_a_spliced_field_is_all_name_words gets the number too. 2. mechanisms.md and _render.py both pointed at "decisions.md under R3" for why the initials() fallback was dropped. decisions.md has no ### R3; the bullet lives under ### R4. Both now cite decisions.md#R4, which the anchor test can see. (The prose form escaped test_doc_internal_anchors_resolve because its regex wants a '#'; widening it wants its own review and is not done here.) 3. mechanisms.md#RENDER-HONORS-THE-PARSE's "Lives in." named UNCLASSIFIED_TAG's "two producers" and listed only _types.py. _facade.py's v1 pickle load is the second, and it is the site test_a_restored_pickle_keeps_v1_conjunction_repair protects. 4. decisions.md#R5 explained the whole facade/core count gap by the render spec omitting the maiden name. Re-measured: that is the UPPER difference only. The two lower-direction names have an empty maiden field in both directions -- str() CONCATENATES adjacent roles, so their given/middle and family/suffix boundary moves are invisible in the joined string. Counts reproduce: core 63/18, facade 62/16. 5. "(`_facade._initials_lists`, v1 parity)" is contradicted by open #462. Measured with the fields held fixed: HumanName('Scott E. Werner') partitions to Scott/E./Werner at 1.4.0, 2.1.0 and here, byte-identical, and initials() on those fields is 'S. E. W.' at 1.4.0 against 'S. W.' since -- a break INSIDE the facade, not upstream of it, reproducible by assigning the three fields by hand. The parity label is dropped; the bullet's own per-name measurements stand unchanged. 6. rules.md#R3's "A CONJUNCTION never initials" was flatly false over 25 default-reachable corpus names and only decisions.md said so. R3 now scopes the carve-out to the middle and base family words and declares the given group unsettled, in prose and with NO deviates marker: a marker states the INTENDED value and there is none to state -- R3 counts name words while P3 makes a connective and its neighbours ONE name word, so "John and Jane Smith" has four candidate answers (J. a. J. S. today, J. J. S., J. S., J a J. S. at 1.4.0) and no entry picks one. decisions.md#R2 records the rejection and its reasoning. 7. mechanisms.md said R3 and R4 "stand or fall together". Measured, they already come apart on those same 25 names: parse("john and jane smith").capitalized() keeps `and` lowercase while .initials() gives 'j. a. j. s.'. The TEXTUAL dependency is real and is kept; the behavioral claim is gone. Also, from the review of this commit: the #464 bullet's 1.4.0 values are ASSIGNED-form values (h = HumanName('juan smith'); h.last = ...), and a first pass at this commit re-measured them with the PARSED form and "corrected" two true claims. The constructions agree from 2.0 on and disagree at 1.4.0, whose parse joins the conjunction run into one last_list element while assignment re-splits it -- same `last` string, different initials. Both claims are restored, each value now names the construction it came from, and the trap is recorded in the bullet. No example line added, so corpus_rules.jsonl holds at 241 and the gate holds at 1094 / 229 / 194 / 102 with unexplained: 0. Co-Authored-By: Claude Opus 5 --- docs/design/decisions.md | 6 +++--- docs/design/mechanisms.md | 2 +- docs/design/rules.md | 27 +++++++++++++++++++++++++-- nameparser/_render.py | 5 +++-- tests/v2/test_render.py | 4 ++-- 5 files changed, 34 insertions(+), 10 deletions(-) diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 82c05ef8..8776d743 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -716,7 +716,7 @@ Declined: "Juan van der" stays "J." (no borne name, no base, and initials of a bare particle run would be nonsense). The general lesson, worth more than this instance: a deviates: marker gets written on the rule whose STATEMENT changed, but a rule can change another rule's OUTPUT without touching its statement, and nothing looks for that — the runner asserts per example line, so an unmarked downstream rule stays green precisely because its own examples avoid the affected input. When adding a marker, walk the changed rule's `interacts:` targets and ask whether any of THEIR examples move. -- 2026-08-29 #461 — TRIED AND BACKED OUT; the question is back on the issue. The unjoined mark readmits the words of an all-particle part to `initials()`, and #461 narrowed that readmission to the `particle` tag alone, on the authority of R3's "A CONJUNCTION never initials, so a base that is one contributes nothing even then". It was reverted before merge, in the same PR. What the narrowing got wrong is best said in terms of the MARK rather than of the fields. The mark is a statement about a whole PART — none of its words is doing the work its tag names — and #461 honored that statement for some of the part's words while keeping one of them out. Under `Lexicon.default().add(particles={'y'})`, `parse("Juan de y")` is such a part: `family_base` is `de y`, both words carry the mark, and #461 initialed `J. d.`, admitting the `de` as the name word the mark makes it and refusing the `y`. R2's reasoning does not split that way. A particle with nothing left to join is not acting as a particle, which is the whole reason the mark exists; a conjunction inside that same part has nothing left to join either, for the same reason and at the same moment, so it is a name word of the part and initials with the rest. Scope that carefully, because its general form is far wider than the claim and would be wrong: this is about a part the mark has ALREADY turned into name words, not about any conjunction that happens to join nothing. Outside such a part the skip stands whatever the conjunction is or is not joining, and none of those moved or may move — `parse("Juan y Garcia")` initials `J. G.` with the `y` a middle name word by P3, `parse("Juan Velasquez y Garcia")` initials `J. V. G.` over base `Velasquez y Garcia`, `parse("Jon Dough and")` initials `J. D.` over base `Dough and`. Base and initials differ in every one of those, legitimately, and R3's own `"Juan de y" → initials="J."` line is a fourth, its `family_base` being `y` under the default vocabulary. So "the base holds a word the initials do not" is NOT the criterion and was not the finding; the mark is. What is therefore in question is R3's "even then" clause, which carries the carve-out into the one part where the joining has stopped — not the code that failed to implement it. The clause is left standing and #461 now asks whether it belongs there; nothing else in R3 moved, and the paragraph #461 added under it is gone. R4's conjunction sentence rests on that clause by name ("the carve-out R3 states for initials") and is untouched here, so whoever settles #461 settles R4's cross-reference with it. Note what the backout cannot record: a `deviates:` marker hangs on an example LINE and this shape has none to hang on — `particles` and `conjunctions` are disjoint in the default vocabulary and in every locale pack, so no input string the doc runner parses reaches it. rules.md itself therefore carries no trace, and this bullet with mechanisms.md#RENDER-HONORS-THE-PARSE is where the gap is written down. It is not the only gap under R3's conjunction sentence, and the other is older, wider and markable: "A CONJUNCTION never initials" is unqualified, while a conjunction in the GIVEN group has always initialed — `parse("John and Jane Smith")` gives `J. a. J. S.` and `parse("Duke of Edinburgh")` `D. o. E.`, 25 of the 1094 corpus names, all reachable from the default vocabulary. Recorded here because it was found here; it is not #461's to fix. Neither direction is visible to the gate: 0 of the 1094 corpus names move either way, and the counts hold at 229 / 194 / 102 with 0 unexplained at 1.4.0 / 2.0.0 / 2.1.0. Both views of the contested token are pinned rather than left to prose — `test_repair_keeps_a_conjunction_lowercase_in_a_particle_part` holds R4's ungated conjunct, gating which had passed the entire suite until #461's test existed, and `test_initials_readmits_a_conjunction_in_a_particle_part` holds today's initials answer the way a `deviates:` marker would, so re-deciding #461 fails the suite until that pin moves with it. +- 2026-08-29 #461 — TRIED AND BACKED OUT; the question is back on the issue. The unjoined mark readmits the words of an all-particle part to `initials()`, and #461 narrowed that readmission to the `particle` tag alone, on the authority of R3's "A CONJUNCTION never initials, so a base that is one contributes nothing even then". It was reverted before merge, in the same PR. What the narrowing got wrong is best said in terms of the MARK rather than of the fields. The mark is a statement about a whole PART — none of its words is doing the work its tag names — and #461 honored that statement for some of the part's words while keeping one of them out. Under `Lexicon.default().add(particles={'y'})`, `parse("Juan de y")` is such a part: `family_base` is `de y`, both words carry the mark, and #461 initialed `J. d.`, admitting the `de` as the name word the mark makes it and refusing the `y`. R2's reasoning does not split that way. A particle with nothing left to join is not acting as a particle, which is the whole reason the mark exists; a conjunction inside that same part has nothing left to join either, for the same reason and at the same moment, so it is a name word of the part and initials with the rest. Scope that carefully, because its general form is far wider than the claim and would be wrong: this is about a part the mark has ALREADY turned into name words, not about any conjunction that happens to join nothing. Outside such a part the skip stands whatever the conjunction is or is not joining, and none of those moved or may move — `parse("Juan y Garcia")` initials `J. G.` with the `y` a middle name word by P3, `parse("Juan Velasquez y Garcia")` initials `J. V. G.` over base `Velasquez y Garcia`, `parse("Jon Dough and")` initials `J. D.` over base `Dough and`. Base and initials differ in every one of those, legitimately, and R3's own `"Juan de y" → initials="J."` line is a fourth, its `family_base` being `y` under the default vocabulary. So "the base holds a word the initials do not" is NOT the criterion and was not the finding; the mark is. What is therefore in question is R3's "even then" clause, which carries the carve-out into the one part where the joining has stopped — not the code that failed to implement it. The clause is left standing and #461 now asks whether it belongs there; nothing else in R3 moved, and the paragraph #461 added under it is gone. R4's conjunction sentence rests on that clause by name ("the carve-out R3 states for initials") and is untouched here, so whoever settles #461 settles R4's cross-reference with it. Note what the backout cannot record: a `deviates:` marker hangs on an example LINE and this shape has none to hang on — `particles` and `conjunctions` are disjoint in the default vocabulary and in every locale pack, so no input string the doc runner parses reaches it. rules.md itself therefore carries no trace, and this bullet with mechanisms.md#RENDER-HONORS-THE-PARSE is where the gap is written down. It is not the only gap under R3's conjunction sentence, and the other is older, wider and markable: "A CONJUNCTION never initials" is unqualified, while a conjunction in the GIVEN group has always initialed — `parse("John and Jane Smith")` gives `J. a. J. S.` and `parse("Duke of Edinburgh")` `D. o. E.`, 25 of the 1094 corpus names, all reachable from the default vocabulary. Recorded here because it was found here; it is not #461's to fix. RESOLVED into the normative document on 2026-08-29, in the review of PR #463: rules.md#R3 now scopes its carve-out to the middle and base family words and declares the given group unsettled, in prose and WITHOUT a `deviates:` marker. The marker was considered and rejected on its own definition — it states the INTENDED value, and there is none to state. R3 counts name words while P3 makes a connective and its neighbours ONE name word, so `John and Jane Smith` has four candidate answers and no entry anywhere picks one: `J. a. J. S.` today, `J. J. S.` reading the carve-out as written, `J. S.` reading P3's join as a single name word, and `J a J. S.` at 1.4.0 (measured on the released wheel, which joins the run rather than skipping the conjunction). Marking it would put an invented value in a normative document and hold the parser to it, since the runner asserts the today-value strictly. The marker's `#N` slot has no owner either — the sentence above disclaims #461 for this gap, and filing its own issue is a maintainer's call rather than a docs commit's. What "markable" meant two sentences up is the MECHANICAL property that an input string exists to hang a marker on, unlike the #461 shape where none does; it was never a claim that the intended value is known. Neither direction is visible to the gate: 0 of the 1094 corpus names move either way, and the counts hold at 229 / 194 / 102 with 0 unexplained at 1.4.0 / 2.0.0 / 2.1.0. Both views of the contested token are pinned rather than left to prose — `test_repair_keeps_a_conjunction_lowercase_in_a_particle_part` holds R4's ungated conjunct, gating which had passed the entire suite until #461's test existed, and `test_initials_readmits_a_conjunction_in_a_particle_part` holds today's initials answer the way a `deviates:` marker would, so re-deciding #461 fails the suite until that pin moves with it. Declined: @@ -739,7 +739,7 @@ Declined: - 2026-08-29 (#458 review) — THE FALLBACK FOR TEXT THE PARSE NEVER SAW, and why it is scoped to one of the two questions `_cap_word` asks. Reading the tag and nothing else regressed every ASSIGNED field holding a conjunction, because a value `replace()` splices in is never classified and so carries no tag to read. Measured on the released 1.4.0 and 2.1.0 wheels, all agreeing and all broken by the tag-only reading: `h.last = 'velasquez y garcia'` then `capitalize(force=True)` gives `John Velasquez y Garcia`, `h.middle = 'e.'` gives `John E. Smith`, `h.last = 'smith-y'` gives `John Smith-y`, `h.first = 'y'` gives `y Smith`. That is the canonical Spanish surname and the shape `y` is in the vocabulary FOR, so an Accepted clause was not available: it had to be fixed. RULE: a token carrying `UNCLASSIFIED_TAG` was never read, so there is no decision to honor and the view falls back to the vocabulary. Note what the discriminator is NOT, in two directions, the second of which took a second review round to find. It is not UNTAGGEDNESS: an ordinary parsed name word carries no tags either (`velasquez` parses to `tags=[]`), so that would have swept in the whole parsed corpus. And it is not the absent SPAN, which is what the first fix used. Span-less means SYNTHETIC, a wider set than unclassified: `Parser.revise()` builds span-less tokens too, from a full sub-parse whose tags it keeps deliberately, and its docstring promises the tag-driven views "behave as if the text had been parsed" — so keying the fallback on the span overrode exactly the tags `revise()` exists to preserve. Measured on the branch that did it: `revise(name, middle='e-f').capitalized(force=True)` gave `e-F` where `parse('john e-f smith')` gave `E-F`, and under `Parser(lexicon=Lexicon())` a revised family `de la vega` initialed `j. v.` where the parse initialed `j. d. l. v.`. So the mark is stamped by the two producers of genuinely unclassified text — `ParsedName.replace()`, and the facade's v1 pickle load, which rebuilds a name from `*_list` strings and no tags — and a HAND-BUILT span-less token is not marked, so it takes the tag path like everything else in the library. That is the tag-driven semantics every other view already has, and it is a change from the span reading in exactly one place: a hand-built span-less token whose text is CONJUNCTION vocabulary now capitalizes rather than lowercasing (measured: a hand-built family `velasquez y garcia` repairs to `Velasquez Y Garcia` untagged, `Velasquez y Garcia` with either the `conjunction` tag or the mark). The PARTICLE conjunct is unaffected either way — it keys on lexicon membership rather than on the mark, which is the open item this bullet's neighbours record — so a hand-built `de la Vega` is unchanged. SCOPED TO THE PER-WORD QUESTION, and this is the durable half. *Is this word a conjunction rather than an initial* is a property of the word and its spelling: an unread token has no ambiguity the parse would have resolved differently, so the vocabulary gives the answer the parser would have given, v1's initial carve-out included — which is why `_render._INITIAL` came back and why test_regex_sync's two-copy assertion matters MORE now, the fallback being right only while it answers as `_vocab` would. *Is this particle acting as a particle here* is a property of the whole PART, and answering it needs a reading on every word of the part, which is exactly what a spliced field lacks. `_cap_word` is handed the whole token's tags — it is not blind to the part because it walks one word, and an earlier draft of this bullet said it was — but with no tag on any word of the part there is nothing to re-derive the part answer FROM, so the fallback here is the word question alone and the particle half falls through to plain particle treatment. Measured cost of the wide reading IN CASE REPAIR, so nobody re-derives it: `replace(family='de la')` gives `de la` today and a parse gives `De La`; widening case repair's fallback would make them agree and would thereby REVERSE the replace/revise boundary recorded two bullets above, with `revise()` as its documented crossing. Do not widen it without reopening the decision. Verified after the fix: `replace(family='velasquez y garcia').capitalized(force=True)` is `Velasquez y Garcia`, `replace(family='de la').capitalized(force=True)` is `de la`, and the parsed and `revise()` paths are unchanged. -- 2026-08-29 (#458 review, then the review of PR #463) — the SAME FALLBACK in `initials()`: WRITTEN, MEASURED AND DROPPED before merge. The defect it aimed at is real and older than #458 — `replace(family='velasquez y garcia').initials()` gives `j. v. y. g.` where the parse, the facade and 1.4.0 all give `j. v. g.`, and `replace(family='de la vega')` gives `j. d. l. v.` for `j. v.` — and it is unfixed again. What killed the fix is the parameter list: `initials()` takes no lexicon, so the fallback had to reach for `Lexicon.default()` and GUESS. `capitalized(lexicon=...)` is handed the caller's vocabulary and so guesses nothing, which is why its fallback stays. Measured cost of the guess, and the reason a merely-imperfect fallback was not good enough: under `Parser(lexicon=Lexicon.default().add(particles={'y'}))`, `parse('Juan de y').initials()` is `J. d. y.` and `parse('Juan Perez').replace(family='de y').initials()` was `J. d. y.` before the fallback and `J.` after it — an entire field's initials gone, because the default vocabulary reads `de y` as particles-plus-conjunction where the caller's reads it as an all-particle part. Guessing the wrong vocabulary for a name parsed under a custom one is worse than answering from tags alone, and `initials()` cannot be told better. The missing crossing is the whole of it: `Parser` carries `capitalized`, `matches` and `revise` as its custom-lexicon crossings and no `initials`, so there is no supported way to hand this view the parse's vocabulary. An issue is filed for that; when it lands, the fallback becomes answerable rather than a guess and this decision is worth reopening. ACCEPTED until then, and stated so a reader does not re-derive it as a bug: `replace(family='de la vega').initials()` is `j. d. l. v.` where the facade and a parse both give `j. v.`. The reach is the v2 core only — `HumanName.initials_list()` computes from the field STRINGS with its own lexicon (`_facade._initials_lists`, v1 parity) and gives `['j', 'v', 'g']` at 1.4.0, 2.1.0 and here — which is also why the facade cannot be used as evidence that the core is fine. Two things learned in the attempt are worth keeping though the code is gone. `_types._remarked` is not the bug and needs no change: it recomputes the mark from TAGS after every edit, which is right for a record of what the parse decided and simply silent about text nobody classified. And any future fallback must be scoped per ROLE, not per name — a role the parse classified whole is decided by its tags however the other roles were built — a distinction that passes every other test in the suite and needs a hand-built name to witness. +- 2026-08-29 (#458 review, then the review of PR #463) — the SAME FALLBACK in `initials()`: WRITTEN, MEASURED AND DROPPED before merge. The defect it aimed at is real and older than #458 — `replace(family='velasquez y garcia').initials()` gives `j. v. y. g.` where the parse, the facade and 1.4.0 all give `j. v. g.` (the facade and 1.4.0 measured through the TWIN of that splice — `h = HumanName('juan smith')` then `h.last = 'velasquez y garcia'`, the assigned construction this bullet is about, keeping `first`), and `replace(family='de la vega')` gives `j. d. l. v.` for `j. v.` — and it is unfixed again. What killed the fix is the parameter list: `initials()` takes no lexicon, so the fallback had to reach for `Lexicon.default()` and GUESS. `capitalized(lexicon=...)` is handed the caller's vocabulary and so guesses nothing, which is why its fallback stays. Measured cost of the guess, and the reason a merely-imperfect fallback was not good enough: under `Parser(lexicon=Lexicon.default().add(particles={'y'}))`, `parse('Juan de y').initials()` is `J. d. y.` and `parse('Juan Perez').replace(family='de y').initials()` was `J. d. y.` before the fallback and `J.` after it — an entire field's initials gone, because the default vocabulary reads `de y` as particles-plus-conjunction where the caller's reads it as an all-particle part. Guessing the wrong vocabulary for a name parsed under a custom one is worse than answering from tags alone, and `initials()` cannot be told better. The missing crossing is the whole of it: `Parser` carries `capitalized`, `matches` and `revise` as its custom-lexicon crossings and no `initials`, so there is no supported way to hand this view the parse's vocabulary. #464 is filed for that; when it lands, the fallback becomes answerable rather than a guess and this decision is worth reopening. ACCEPTED until then, and stated so a reader does not re-derive it as a bug: `replace(family='de la vega').initials()` is `j. d. l. v.` where the facade and a parse both give `j. v.`. The reach is the v2 core only — `HumanName.initials_list()` computes from the field STRINGS with its own lexicon (`_facade._initials_lists`) and gives `['j', 'v', 'g']` at 1.4.0, 2.1.0 and here, under that same assignment. That is why the facade cannot be used as evidence that the core is fine. Do NOT read `_initials_lists` as a v1 parity REFERENCE, though, which is how an earlier wording of this bullet labelled it: #462 is open on a parity break in that exact path, and it is a break inside the facade rather than upstream of it. `HumanName('Scott E. Werner')` partitions to `first='Scott'`, `middle='E.'`, `last='Werner'` at 1.4.0, at 2.1.0 and here — byte-identical fields — and `initials()` on them is `S. E. W.` at 1.4.0 and `S. W.` at 2.1.0 and here, the facade reading the middle `E` as the conjunction; assigning those three fields by hand reproduces each version's answer, so no parse difference is involved. The parity label was therefore doing work its own measurement cannot support, and only the per-name measurements are load-bearing here. MEASURE THIS BULLET WITH THE ASSIGNED CONSTRUCTION, and name the construction beside each value: the review of PR #463 measured `HumanName('juan velasquez y garcia')` — the PARSED form — against sentences about the assigned one, and 'corrected' two true claims into false ones. The two constructions agree from 2.0 on and DISAGREE at 1.4.0, where the parsed form gives `['j', 'v g']` and `j. v g.`. The `last` string is byte-identical either way; `initials_list()` reads `last_list`, which the 1.4.0 parse leaves as `['velasquez y garcia']` — the conjunction run joined into one element — where assignment re-splits it to `['velasquez', 'y', 'garcia']`. So `computes from the field STRINGS` is exact for 2.x and true of the ASSIGNED form only at 1.4.0. Two things learned in the attempt are worth keeping though the code is gone. `_types._remarked` is not the bug and needs no change: it recomputes the mark from TAGS after every edit, which is right for a record of what the parse decided and simply silent about text nobody classified. And any future fallback must be scoped per ROLE, not per name — a role the parse classified whole is decided by its tags however the other roles were built — a distinction that passes every other test in the suite and needs a hand-built name to witness. - 2026-08-29 — WHY THE BOUNDARY WENT UNNOTICED UNTIL #407, which is where a future reader should look for it. For an ALL-PARTICLE part the other three tag-driven views give the same answer through `replace()` and `revise()` alike: measured over `de la`, `van der`, `do`, `de` and `van de la`, all five agree on `family_particles=''`, on a `family_base` holding the whole part, and on initials from every word. They converge because an UNTAGGED part and a MARKED all-particle part reach the same place by different routes — untagged, no word is recognized as a particle; marked, none is ACTING as one — and all three views only ask which words are particles. Case repair is the one view that asks a second question, since it must also decide whether to lowercase, so it is where the two routes first come apart. The mirror case confirms the reading: on a MIXED part the convergence is the other way round — `de la vega` and `van der berg` diverge in all three views between `replace()` and `revise()` (`replace()` reports particles `''` and base `'de la vega'` where `revise()` reports `'de la'` and `'vega'`) and AGREE on case repair, R4's all-particle clause not reaching them. So before #407 the distinction was invisible on exactly the shape the clause is about, and visible only on shapes the clause does not govern. @@ -750,7 +750,7 @@ Declined: - 2026-08-29 — WHY R5'S STATEMENT WAS REWORDED RATHER THAN GIVEN A THIRD EXAMPLE, recorded at length because the wrong turn was taken twice and the reasoning is reusable. R5 first said a mixed-case name is returned untouched "whether or not its casing is correct". That phrasing reads as a disjunction with two branches, and invites the question of which example witnesses the already-correct one. Nothing can: for a mixed-case name that repair would not alter, kept and repaired are the SAME STRING by construction, so no example line distinguishes a parser with the gate from one without it. The unwitnessable branch is a property of the claim, not a gap in our choice of names — and the bullet above had already said so ("an already-correct input cannot witness a rule whose subject is what gets kept REGARDLESS of correctness") before two review rounds pushed past it. Two rows were tried and both withdrawn. `"Juan McDonald" → capitalized="Juan McDonald"` was INERT (mechanisms.md's inert-measurement class): it passes with R5's gate deleted, which is the whole failure shape that class names. `"Vincent Van Gogh" → capitalized="Vincent Van Gogh"` did discriminate — measured by comparing the shipped call against gate-off behavior, which the forced call reproduces exactly, since deleting the gate is what `force` already does: shipped `'Vincent Van Gogh'`, gate-off `'Vincent van Gogh'`, the particle rule lowercasing `Van` the moment repair runs — but it was REDUNDANT, not complementary. It asserts the same proposition as the `Shirley Maclaine` row above it: repair would change this name, and the gate keeps it anyway. The two differ only in which repair rule would have fired, and which rule repair applies is R4's subject, not R5's. And the frame it was chosen under was itself wrong, which is the more useful half of the lesson (reframed 2026-08-29 on Derek's correction, while R4 was being reworked). The row was picked to be bearer-correct — a name whose casing the bearer would endorse — and that frame drags the document into per-name arguments about whose spelling is right: this one needed `Vincent van Gogh` to be the correct spelling, which P6's own examples contradict, and the tussenvoegsel convention behind them is amended later on this branch anyway. CORRECTNESS DOES NOT ENTER INTO IT. Mixed case is the writer making an explicit choice, and repair defers to that choice rather than judging it — a name kept is not a name endorsed. Read that way the withdrawn row needed no claim about Dutch orthography at all, and the reason it needed one is exactly that the frame was wrong. The fix was to stop claiming the unwitnessable branch: the statement now says a mixed-case name is kept and that whether its casing is right does not enter into the decision, which `Shirley Maclaine` — casing wrong, kept — witnesses whole. GENERAL LESSON, and the reason this is long: when an example cannot be found for half a rule, suspect the STATEMENT before suspecting the example set. A phrasing that promises more branches than the behavior has will absorb inert examples indefinitely, each one looking like progress. Caution for the commit that reworks R4 and will choose its own mixed-case rows: avoid a name whose family base is wholly particle vocabulary (`Anh Van Do`, base `Van Do`), because the #407 work changes how those capitalize and would neuter such a row silently; `gogh` and `vega` are in neither `particles` nor `particles_ambiguous`, `van` and `do` are in both. - 2026-08-29 — the override has TWO routes, and the rule states neither, by design. Per-call is the obvious one. The second is a v1 Constants attribute the facade still honors, `force_mixed_case_capitalization` (nameparser/_facade.py resolves it when no per-call value is given; docs/release_log.rst records it as "still honored through the facade"). Measured today: with that attribute set True, `HumanName('Shirley Maclaine')` then a bare `.capitalize()` — no argument at all — LEAVES `'Shirley MacLaine'` (the v1 call mutates in place and returns None, so the name is read back with `str()`). An earlier draft of R5 said "only an explicit request to repair regardless overrides that", which that measurement falsifies; the statement now says repair was asked for anyway, without saying by what route (it read `repair regardless of how the name is cased was asked for` until 2026-08-29, when that phrasing turned out to have a second reading -- see the override bullet below), because rules.md is implementation-free by its own preamble. - 2026-08-29 — `capitalized_forced` is a test-side pseudo-field, not a parser field. The route the EXAMPLES use is the per-call argument to `capitalized()`, and an argument is not a policy, locale or extras gate — the only three things rules.md's grammar admits in an example's annotation slot — so it cannot ride that slot, and the doc runner grew a resolver branch instead. (The facade attribute above is a second route to the same behavior, not a second thing to assert; the core takes the argument only.) The R5 block's example lines are asserted by the suite like any others. They do enlarge one file: `corpus_rules.jsonl` is generated from this document's examples, and `Shirley Maclaine` had not been among them, so regenerating adds it (235 lines, from 234). Whether that moves the DIFFERENTIAL population is a separate question with a per-name answer, and the general answer is that a rules.md example CAN move it: the harness dedupes across corpus files, so a new example name costs a population slot exactly when no other corpus already carries that same string. `Shirley Maclaine` was already in `corpus.jsonl`, so it costs nothing and the population holds at 1090 — measured, not assumed. The row withdrawn above is the counter-case, and worth keeping for it: `Vincent Van Gogh` appears in no other corpus (`corpus.jsonl` and `corpus_issues.jsonl` carry `Vincent van Gogh`, a DIFFERENT string that does not dedupe against it), and while it was in the block the population read 1091. Neither name needed a ledger rule, which is a measurement rather than a consequence of the above: the gate exits 0 with 0 unexplained at all three baselines either way. -- 2026-08-29 — WHAT THE OVERRIDE DOES AND DOES NOT PROMISE, from Derek's framing of R5 and then measured, because the framing implies a property that is ALMOST true and the gap is the useful part. The framing first, and it supersedes the correctness talk elsewhere in this entry: mixed case is the writer making an explicit choice, and repair defers to that choice instead of judging it. Nothing is being called correct or incorrect — a name kept is a name whose writer said something about it, and a name repaired is one whose writer did not. The property that seems to follow is that asking for repair REGARDLESS should ignore the given casing entirely, so one name repairs to one string however it was written. MEASURED over the 1094 corpus names and it does NOT hold. On the v2 core — `parse(n).capitalized(force=True)` against `parse(n.upper()).capitalized()`, rendering all seven roles — the two differ for 63 names, and the lowercase direction for 18. Naming the surface matters here, and BOTH counts move with the surface, which an earlier draft of this bullet got half right: through `HumanName` and `str()`, which is what tests/test_capitalization.py uses, the counts are 62 and 16, because the facade's default render spec omits the maiden name and so cannot see `Jane van der Berg née y Jones` (the whole of the upper difference, and it is a conjunction inside the maiden name); the two the facade drops in the lower direction are `John van der J. V` and `abdul V Smith`. Re-derived 2026-08-29 on this branch, and named per name so the next reader can check the arithmetic: core 63/18, facade 62/16. Two things about those numbers are the opposite of what one would guess. UPPERCASE IS THE WORSE DIRECTION, not the clean one. And the misses are not merely the pipeline's case-sensitivity leaking in: of the 63, only 25 move a role at all, and the other 38 parse byte-identically and diverge inside the repair (25 and 37 through the facade, the missing name being one of the byte-identical ones). The mechanism is v1's initial carve-out — a conjunction is not lowercased where it is written initial-shaped, and initial-shaped means one CAPITAL letter — so uppercasing turns every one-letter conjunction into an initial (`Velasquez y Garcia, Dr. Juan Q.` forced keeps `y`; the same name uppercased then repaired gives `Y`, with a byte-identical partition either side -- the comma form is the one to cite here, the space-written `Dr. Juan Q. Velasquez y Garcia` being a member of the 25 whose roles DO move), and lowercasing turns a middle initial `E` into the Italian conjunction. 1.4.0 does the same (`JUAN Y GARCIA` capitalizes to `Juan Y Garcia`), so this is inherited, and it is recorded here rather than fixed here. Recompute both directions by running the two forms over the four corpus files deduped and diffing. +- 2026-08-29 — WHAT THE OVERRIDE DOES AND DOES NOT PROMISE, from Derek's framing of R5 and then measured, because the framing implies a property that is ALMOST true and the gap is the useful part. The framing first, and it supersedes the correctness talk elsewhere in this entry: mixed case is the writer making an explicit choice, and repair defers to that choice instead of judging it. Nothing is being called correct or incorrect — a name kept is a name whose writer said something about it, and a name repaired is one whose writer did not. The property that seems to follow is that asking for repair REGARDLESS should ignore the given casing entirely, so one name repairs to one string however it was written. MEASURED over the 1094 corpus names and it does NOT hold. On the v2 core — `parse(n).capitalized(force=True)` against `parse(n.upper()).capitalized()`, rendering all seven roles — the two differ for 63 names, and the lowercase direction for 18. Naming the surface matters here, and BOTH counts move with the surface, which an earlier draft of this bullet got half right: through `HumanName` and `str()`, which is what tests/test_capitalization.py uses, the counts are 62 and 16 — and the two directions are dropped for DIFFERENT reasons, which an earlier draft of this bullet ran together under the first. UPPER: the facade's default render spec omits the maiden name, so `str()` cannot see `Jane van der Berg née y Jones` (the whole of the upper difference, and it is a conjunction inside the maiden name — `maiden` is `y Jones` forced against `Y Jones` uppercased and every other role is byte-identical). LOWER: the two the facade drops are `John van der J. V` and `abdul V Smith`, and the maiden field is EMPTY in both directions for both, so the omitted role explains neither. What hides them is that `str()` CONCATENATES adjacent roles, so a token that crosses a role BOUNDARY and moves nothing else is invisible in the joined string: `John van der J. V` is family `van der J. V` forced against family `van der J.` plus suffix `V` lowercased, and `abdul V Smith` is given `Abdul V` forced against given `Abdul` plus middle `V` lowercased — same seven roles rendered, same string joined. Re-derived 2026-08-29 on this branch, and named per name so the next reader can check the arithmetic: core 63/18, facade 62/16. Two things about those numbers are the opposite of what one would guess. UPPERCASE IS THE WORSE DIRECTION, not the clean one. And the misses are not merely the pipeline's case-sensitivity leaking in: of the 63, only 25 move a role at all, and the other 38 parse byte-identically and diverge inside the repair (25 and 37 through the facade, the missing name being one of the byte-identical ones). The mechanism is v1's initial carve-out — a conjunction is not lowercased where it is written initial-shaped, and initial-shaped means one CAPITAL letter — so uppercasing turns every one-letter conjunction into an initial (`Velasquez y Garcia, Dr. Juan Q.` forced keeps `y`; the same name uppercased then repaired gives `Y`, with a byte-identical partition either side -- the comma form is the one to cite here, the space-written `Dr. Juan Q. Velasquez y Garcia` being a member of the 25 whose roles DO move), and lowercasing turns a middle initial `E` into the Italian conjunction. 1.4.0 does the same (`JUAN Y GARCIA` capitalizes to `Juan Y Garcia`), so this is inherited, and it is recorded here rather than fixed here. Recompute both directions by running the two forms over the four corpus files deduped and diffing. - 2026-08-29 — and therefore NOT stated in rules.md, which is a deliberate choice rather than an oversight. The document's examples are keyed on input STRINGS, so any statement of the property invites exactly the test that falsifies it — re-case the input, expect the same output — and the counterexamples are already in the corpora. The property is true of the repair given a parse, and rules.md speaks input-to-output; a rule stating it would be over-broad in the one direction a reader would check. What R5's statement says is enough for the promise that IS kept: a mixed-case name is kept unless repair was asked for anyway. That clause was REWORDED for this, and the reword is the whole point rather than a tidy-up. It read `unless repair regardless of how the name is cased was asked for`, which carries two readings -- the intended one, that the request overrides the keeping, and a second one, that the repair disregards the input's casing, which is this property in nearly this bullet's own words. A reader taking the second reading would run the re-casing test predicted above, land on `Velasquez y Garcia, Dr. Juan Q.` (in the corpus today), and conclude the RULE is wrong when only the phrasing was. Nine words, and they asserted the thing the paragraph exists to deny. The property is pinned in tests/test_capitalization.py instead, over names carrying no single-letter word whose class case decides, with `juan y garcia` beside it as the recorded exception. R5's example block gains `"SHIRLEY MACLAINE" → capitalized="Shirley MacLaine"` from this work, and it earns its place on its own ground rather than as half of a convergence pair: it is the only row in the block that fails when the gate is narrowed to lowercase-only, every other row passing that mutation. Measured three ways — gate deleted (passes, so it does not witness the gate's existence), gate narrowed to accept only all-lowercase (FAILS, and alone in the block), Mac/Mc convention deleted (fails, with the other two rows). Until it was added, R5 stated that repair acts on a name written entirely in one case and witnessed only the lowercase half of it. That lowercase half is still `"juan mcdonald"`, which is byte for byte an R4 row as well, and the duplication is deliberate rather than an editing slip: the two rules make different claims about the same line — R4 that the repair honors the Mac/Mc convention, R5 that an all-lowercase name is acted on at all — and dropping it from R5 would leave the gate's lowercase half unwitnessed inside the rule that states the gate. Five other rows already sit under two rules apiece for the same reason (P5/P6 twice, P5/O5, N3/M4, W1/W3). - 2026-08-29 — DEBT this extraction leaves, named so the next commit inherits an obligation rather than a rediscovery. Pulling the gate out into R5 leaves R4 carrying ONE falsehood and ONE ambiguity — different defects wanting different repairs, and `interacts: R5` carries neither, the field being advisory. FALSE: R4 promises repair "vocabulary exceptions (McDonald) included", but `str(parse('Juan Mcdonald').capitalized())` is `'Juan Mcdonald'` — the gate refuses before any vocabulary is consulted, and only `str(parse('Juan Mcdonald').capitalized(force=True))`, `'Juan McDonald'`, reaches the exception. R4 needs its promise scoped to names the gate admits. AMBIGUOUS, not false: R4's "an already-correct name comes back unchanged" means correct by the repair's own conventions, i.e. idempotence, and under that meaning it is true; a reader hears correct as the bearer writes it, and under THAT meaning `str(parse('bell hooks').capitalized())` — `'Bell Hooks'` — looks like a counterexample. It is not one, because `bell hooks` is not already-correct in R4's sense. What R4 owes is a disambiguation of "correct", NOT a narrowing to spare deliberately single-cased names: that would be new behavior, and R5's own rationale declines it on the ground that single case leaves the repair nothing to read. Also for that commit, and inert as things stand: R4's boundary row `"Juan McDonald" → capitalized="Juan McDonald"` passes with R5's gate deleted, exactly like the R5 row that was withdrawn above; rewriting it to `capitalized_forced=` makes it discriminate for R4's own subject but still witnesses nothing about the already-correct question. This commit adds R5 and touches R4 only on its pointer line, leaving both defects as found rather than half-fixed by a commit whose subject is something else. diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 80e79503..4e82b735 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -59,7 +59,7 @@ Problem shape. Two stages need the same answer about the same input, and the one ## RENDER-HONORS-THE-PARSE — the parse decides it, the views honor it -Problem shape. A render view needs a fact the parse already settled — whether a word is the conjunction or an initial, whether a particle is acting as one, which word renders first. Contract statement. The parse decides it; the render views honor those decisions and never re-evaluate them. Two directions break that, and each has been found here as a defect: a view RE-DERIVES the answer from the text, keeping its own copy of a pipeline predicate — that one shipped through 2.0 and 2.1 — or a view honors the record and then OVERRIDES it, readmitting what the decision excluded, which is filed and open here. How it works. Re-deriving fails because the two copies stop being the same question long before anyone notices they are two: `_cap_word` re-ran the conjunction-versus-initial decision from the word's spelling against a hand-maintained copy of the pipeline's `_INITIAL` pattern while classify had already answered it and recorded it on the token, and `_classify.py` asks `is_initial()`, the shape test ANDed with a script-repertoire test since #320, where `_render.py` asked the bare pattern, and asked it per WORD of a token's text rather than per token, so `juan e-f smith` repaired to `Juan e-F Smith` (#458). Overriding fails more directly, and this entry's instance of it is OPEN rather than closed: `initials()` honors the `conjunction` tag through `_SKIP_TAGS` and then readmits the token whenever it also carries UNJOINED_TAG, the mark of an all-particle part, where rules.md#R3 excludes a conjunction "even then" — so under a caller's vocabulary that puts a word in both sets the view readmits what the rule excluded, and under `Lexicon.default().add(particles={'y'})` — the lexicon those two readings need, and they hold under no other — `parse("Anh y Van")` initialed `A. y. V.` and `parse("Juan de y")` `J. d. y.` on 2026-08-29. Read those two values scoped to that lexicon or they are simply false: under the DEFAULT vocabulary the same strings give `A. V.` and `J.`, and the second is a rules.md#R3 example line the doc runner asserts. #461 is where that stands, and it is worth reading before re-fixing it: the narrowing was written, measured and BACKED OUT in the same PR, because honoring R3 there cost `initials()` its agreement with `family_base`, which reads that same `de y` as the base — this entry's other shape, arrived at from the render side, and the reason the question moved from the code to R3's clause (decisions.md carries the argument). What to carry away is the DIRECTION and not its verdict here: a view that honors a record and then readmits what the record excluded is overriding a decision it never took, whichever way this one settles. Case repair reads that same token and does NOT readmit it — `capitalized(force=True)` on `Anh y Van` gives `Anh y Van` under that same lexicon, R4 carrying the carve-out in its own words but ON R3's authority — its text reads "being no name word in any part — the carve-out R3 states for initials", so the two rules stand or fall together and a change to R3's clause reaches R4 — so the two views disagree today about that token exactly as they did before #461 and the backout restores that disagreement knowingly; only the FORCED call witnesses the repair half, R5's gate refusing a mixed-case name before any of this is consulted. This is the CONSUMER-side rule over the producer-side entries — VOCAB-TAGS records what the vocabulary knew, MARK-DONT-STRIP what a stage decided about it, FOLDED_TAG what order to render in — and a view reads what they recorded, whichever kind it is. It sits where ONE-PREDICATE-PER-QUESTION's stated limit leaves off: where two live sites need one answer they share a predicate, but a render view always comes AFTER the decider, so the answer is recorded rather than shared — on the TOKEN, which is the views' equivalent of that entry's `ParseState.order`, no view being able to see a ParseState at all. Known limit, and the half most easily got wrong next: a token the parse never saw carries no decision to honor, so a view falls back to the vocabulary — the tell is `UNCLASSIFIED_TAG`, which `ParsedName.replace()` stamps when it splices raw text into a field and the facade's v1 pickle load stamps when it rebuilds a name from `*_list` strings. It is NOT untaggedness, since an ordinary parsed name word carries no tags either; and it is NOT `span is None`, which was tried and is wrong in the other direction — span-less means SYNTHETIC, and `Parser.revise()` builds span-less tokens from a full sub-parse whose tags it keeps on purpose, so the span reading overrode exactly the tags `revise()` exists to preserve (`revise(middle='e-f')` repaired to `e-F` where the parse gave `E-F`). A hand-built span-less token is unmarked and therefore classified, which is the same tag-driven default every other view applies. A view can only fall back if it is HANDED a vocabulary, and exactly one is: `capitalized(lexicon=...)`. `family_base` and `family_particles` are properties on ParsedName, whose fields are original/tokens/ambiguities and nothing else, so a spliced field empties the particles view and leaves the base the whole field, with `Parser.revise()` the crossing there too (docs/usage.rst says so where it documents the degradation). `initials()` is the near miss and the instructive one: it is a METHOD, so it looks like it could ask, but its signature is `(spec, delimiter, separator)` and carries no lexicon — a fallback there was written and dropped because it had to GUESS `Lexicon.default()`, and the guess erased a whole field under a caller's own vocabulary (decisions.md, under R3). `capitalized()` guesses nothing: it reads the lexicon it was handed, and only defaults to `Lexicon.default()` when the caller passes none, which is the documented meaning of omitting the argument rather than a fallback. Within the one view that can fall back, the fallback is drawn per QUESTION (rules.md#R4's Accepted clause). Whether a word is the conjunction or an initial is a property of the word, which a vocabulary answers alone, so case repair asks it. Whether a part is wholly particles is a property of the whole PART, which the pipeline answers once and records as UNJOINED_TAG. What `_cap_word` cannot do is RE-DERIVE that answer where no word of the part carries a tag — it is handed the whole token's tags and gates the particle conjunct on UNJOINED_TAG, so it is not blind to the part, it simply has no evidence to reconstruct one from — so repair leaves that half to plain particle treatment and rules.md#R4's Accepted boundary records the consequence — a spliced field is not repaired as a parsed one is, with `Parser.revise()` as the supported crossing. A fallback is right only while it answers as the pipeline would, and THAT is held by hand rather than mechanically: test_regex_sync pins the two `_INITIAL` copies to each other and to config, while the repertoire half of the pipeline's predicate (#320) is deliberately not carried across, layering forbidding the import. What that divergence can reach is now nothing observable: it needs a caller-added conjunction written initial-SHAPED in a script that has no initials (`씨.`), and case repair is the fallback's only reader, so the two paths differ by `lower()` versus `capitalize()` over a caseless script — the same string either way. `initials()` used to be the reader that could witness it, and no longer falls back at all. A second limit, recorded rather than closed: `_cap_word`'s PARTICLE conjunct still keys on the lexicon handed to the view rather than on the `particle` tag, so a repair run with a lexicon other than the parse's re-decides a word the parse already read — a name parsed under the default vocabulary, where `parse('juan smith vega')` reads `vega` as the family, repairs to `Juan Smith vega` when `capitalized()` is handed `Lexicon.default().add(particles={'vega'})` instead — the divergence needs the two lexicons to differ, and repairing under the parse's own lexicon gives `Juan Smith Vega`. Making it read the tag moves a boundary rules.md#R4 states in prose, so it is a separate decision and not a cleanup (decisions.md#R4, "NOT DONE"). Lives in. nameparser/_render.py (`capitalized`/`_cap_word`, `_reads_as_conjunction`, and `initials`, which honors tags and never falls back) and nameparser/_types.py (`_text_for`, `UNCLASSIFIED_TAG` and its two producers), reading what nameparser/_pipeline/ recorded — the mark those views read is recomputed producer-side in `_remarked`, which is deliberately silent about text nobody classified and is right to be. Reach for it when. A view is about to consult a Lexicon, a regex or an exception list about a word the parse already saw — or a view and a field disagree about the same parse. #408 is filed as that second shape, and was open on 2026-08-29: `initials()` walks tokens in written order where the family field applies FOLDED_TAG's ordering, so `parse("der, y van")` gave family `van der` and initials `y. d. v.` that day, one view not yet honoring the fold. It belongs here rather than argued again from scratch. +Problem shape. A render view needs a fact the parse already settled — whether a word is the conjunction or an initial, whether a particle is acting as one, which word renders first. Contract statement. The parse decides it; the render views honor those decisions and never re-evaluate them. Two directions break that, and each has been found here as a defect: a view RE-DERIVES the answer from the text, keeping its own copy of a pipeline predicate — that one shipped through 2.0 and 2.1 — or a view honors the record and then OVERRIDES it, readmitting what the decision excluded, which is filed and open here. How it works. Re-deriving fails because the two copies stop being the same question long before anyone notices they are two: `_cap_word` re-ran the conjunction-versus-initial decision from the word's spelling against a hand-maintained copy of the pipeline's `_INITIAL` pattern while classify had already answered it and recorded it on the token, and `_classify.py` asks `is_initial()`, the shape test ANDed with a script-repertoire test since #320, where `_render.py` asked the bare pattern, and asked it per WORD of a token's text rather than per token, so `juan e-f smith` repaired to `Juan e-F Smith` (#458). Overriding fails more directly, and this entry's instance of it is OPEN rather than closed: `initials()` honors the `conjunction` tag through `_SKIP_TAGS` and then readmits the token whenever it also carries UNJOINED_TAG, the mark of an all-particle part, where rules.md#R3 excludes a conjunction "even then" — so under a caller's vocabulary that puts a word in both sets the view readmits what the rule excluded, and under `Lexicon.default().add(particles={'y'})` — the lexicon those two readings need, and they hold under no other — `parse("Anh y Van")` initialed `A. y. V.` and `parse("Juan de y")` `J. d. y.` on 2026-08-29. Read those two values scoped to that lexicon or they are simply false: under the DEFAULT vocabulary the same strings give `A. V.` and `J.`, and the second is a rules.md#R3 example line the doc runner asserts. #461 is where that stands, and it is worth reading before re-fixing it: the narrowing was written, measured and BACKED OUT in the same PR, because honoring R3 there cost `initials()` its agreement with `family_base`, which reads that same `de y` as the base — this entry's other shape, arrived at from the render side, and the reason the question moved from the code to R3's clause (decisions.md carries the argument). What to carry away is the DIRECTION and not its verdict here: a view that honors a record and then readmits what the record excluded is overriding a decision it never took, whichever way this one settles. Case repair reads that same token and does NOT readmit it — `capitalized(force=True)` on `Anh y Van` gives `Anh y Van` under that same lexicon, R4 carrying the carve-out in its own words but ON R3's authority — its text reads "being no name word in any part — the carve-out R3 states for initials", so a change to R3's clause reaches R4's TEXT. What does NOT follow, though an earlier wording of this sentence asserted it, is that the two stand or fall together in BEHAVIOR: they have already come apart, over the 25 corpus names carrying a conjunction in the GIVEN group — `parse("john and jane smith").capitalized()` keeps `and` lowercase, so R4's carve-out holds there, while `.initials()` gives `j. a. j. s.`, so R3's does not (decisions.md#R2 carries that population, and rules.md#R3 now says so in its own words). The dependency is textual, and only textual — so the two views disagree today about that token exactly as they did before #461 and the backout restores that disagreement knowingly; only the FORCED call witnesses the repair half, R5's gate refusing a mixed-case name before any of this is consulted. This is the CONSUMER-side rule over the producer-side entries — VOCAB-TAGS records what the vocabulary knew, MARK-DONT-STRIP what a stage decided about it, FOLDED_TAG what order to render in — and a view reads what they recorded, whichever kind it is. It sits where ONE-PREDICATE-PER-QUESTION's stated limit leaves off: where two live sites need one answer they share a predicate, but a render view always comes AFTER the decider, so the answer is recorded rather than shared — on the TOKEN, which is the views' equivalent of that entry's `ParseState.order`, no view being able to see a ParseState at all. Known limit, and the half most easily got wrong next: a token the parse never saw carries no decision to honor, so a view falls back to the vocabulary — the tell is `UNCLASSIFIED_TAG`, which `ParsedName.replace()` stamps when it splices raw text into a field and the facade's v1 pickle load stamps when it rebuilds a name from `*_list` strings. It is NOT untaggedness, since an ordinary parsed name word carries no tags either; and it is NOT `span is None`, which was tried and is wrong in the other direction — span-less means SYNTHETIC, and `Parser.revise()` builds span-less tokens from a full sub-parse whose tags it keeps on purpose, so the span reading overrode exactly the tags `revise()` exists to preserve (`revise(middle='e-f')` repaired to `e-F` where the parse gave `E-F`). A hand-built span-less token is unmarked and therefore classified, which is the same tag-driven default every other view applies. A view can only fall back if it is HANDED a vocabulary, and exactly one is: `capitalized(lexicon=...)`. `family_base` and `family_particles` are properties on ParsedName, whose fields are original/tokens/ambiguities and nothing else, so a spliced field empties the particles view and leaves the base the whole field, with `Parser.revise()` the crossing there too (docs/usage.rst says so where it documents the degradation). `initials()` is the near miss and the instructive one: it is a METHOD, so it looks like it could ask, but its signature is `(spec, delimiter, separator)` and carries no lexicon — a fallback there was written and dropped because it had to GUESS `Lexicon.default()`, and the guess erased a whole field under a caller's own vocabulary (decisions.md#R4). `capitalized()` guesses nothing: it reads the lexicon it was handed, and only defaults to `Lexicon.default()` when the caller passes none, which is the documented meaning of omitting the argument rather than a fallback. Within the one view that can fall back, the fallback is drawn per QUESTION (rules.md#R4's Accepted clause). Whether a word is the conjunction or an initial is a property of the word, which a vocabulary answers alone, so case repair asks it. Whether a part is wholly particles is a property of the whole PART, which the pipeline answers once and records as UNJOINED_TAG. What `_cap_word` cannot do is RE-DERIVE that answer where no word of the part carries a tag — it is handed the whole token's tags and gates the particle conjunct on UNJOINED_TAG, so it is not blind to the part, it simply has no evidence to reconstruct one from — so repair leaves that half to plain particle treatment and rules.md#R4's Accepted boundary records the consequence — a spliced field is not repaired as a parsed one is, with `Parser.revise()` as the supported crossing. A fallback is right only while it answers as the pipeline would, and THAT is held by hand rather than mechanically: test_regex_sync pins the two `_INITIAL` copies to each other and to config, while the repertoire half of the pipeline's predicate (#320) is deliberately not carried across, layering forbidding the import. What that divergence can reach is now nothing observable: it needs a caller-added conjunction written initial-SHAPED in a script that has no initials (`씨.`), and case repair is the fallback's only reader, so the two paths differ by `lower()` versus `capitalize()` over a caseless script — the same string either way. `initials()` used to be the reader that could witness it, and no longer falls back at all. A second limit, recorded rather than closed: `_cap_word`'s PARTICLE conjunct still keys on the lexicon handed to the view rather than on the `particle` tag, so a repair run with a lexicon other than the parse's re-decides a word the parse already read — a name parsed under the default vocabulary, where `parse('juan smith vega')` reads `vega` as the family, repairs to `Juan Smith vega` when `capitalized()` is handed `Lexicon.default().add(particles={'vega'})` instead — the divergence needs the two lexicons to differ, and repairing under the parse's own lexicon gives `Juan Smith Vega`. Making it read the tag moves a boundary rules.md#R4 states in prose, so it is a separate decision and not a cleanup (decisions.md#R4, "NOT DONE"). Lives in. nameparser/_render.py (`capitalized`/`_cap_word`, `_reads_as_conjunction`, and `initials`, which honors tags and never falls back) and nameparser/_types.py (`_text_for` and `UNCLASSIFIED_TAG`, with the `ParsedName.replace()` producer beside it) and nameparser/_facade.py (the v1 pickle load, the SECOND producer of that mark — it is named in this list because a change that follows the list into `_types.py` alone leaves it behind, which is the site test_a_restored_pickle_keeps_v1_conjunction_repair exists to protect), reading what nameparser/_pipeline/ recorded — the mark those views read is recomputed producer-side in `_remarked`, which is deliberately silent about text nobody classified and is right to be. Reach for it when. A view is about to consult a Lexicon, a regex or an exception list about a word the parse already saw — or a view and a field disagree about the same parse. #408 is filed as that second shape, and was open on 2026-08-29: `initials()` walks tokens in written order where the family field applies FOLDED_TAG's ordering, so `parse("der, y van")` gave family `van der` and initials `y. d. v.` that day, one view not yet honoring the fold. It belongs here rather than argued again from scratch. ## CLAUSE-CONTENT-OVERRULES-DELIMITER — content wins diff --git a/docs/design/rules.md b/docs/design/rules.md index f42565d5..27277858 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -1101,7 +1101,17 @@ R3. Rationale: initials abbreviate the person's name words; titles, contribute nothing — except the particles of a part whose every word is one, which are not acting as particles there (R2) and initial like any other name word. A CONJUNCTION never initials, - so a base that is one contributes nothing even then. + so a base that is one contributes nothing even then. That + carve-out is stated for the middle and base family words; the + GIVEN group is not settled here. A conjunction written among + given names does initial today, and this document does not yet + say whether it should — because two of its own rules answer + differently and neither answer has been taken: this rule counts + name words, while P3 makes a connective and its neighbours ONE + name word, so a joined given group owes one initial under P3 and + one per joined name word under the carve-out. Until that is + decided the given group's answer is pinned-but-undocumented + rather than specified, and no line below asserts it. "Dr. Juan Q. Xavier de la Vega III" → initials="J. Q. X. V." "Anh Do" → initials="A. D." "Nguyen, Van Le" → initials="V. L. N." @@ -1122,7 +1132,20 @@ R3. Rationale: initials abbreviate the person's name words; titles, and matches the parse in both views. Stated without an example line because every line here names an input string, and this shape needs a field edited after the parse. - history: decisions.md#R2 · interacts: R2, R4 · implemented: nameparser/_render.py, nameparser/_facade.py + Accepted: the unsettled given-group answer above is neither rare + nor hypothetical — 25 of the corpus names carry a conjunction + among the given names, every one of them reachable from the + default vocabulary, and it has initialed since 1.4.0. It carries + no marked deviation, for the reason that mechanism exists: a + marker states the INTENDED value, and one name, "John and Jane + Smith", has four candidates. Today gives "J. a. J. S."; the + carve-out read as written gives "J. J. S."; P3's one-name-word + join gives just "J. S."; and 1.4.0 gave "J a J. S.". Marking it + would put an invented value in a normative document and hold + the parser to it. #461 asks the neighbouring question about the + all-particle base and does not own this one; decisions.md#R2 + carries the population and the measurements. + history: decisions.md#R2 · interacts: P3, R2, R4 · implemented: nameparser/_render.py, nameparser/_facade.py R4. Rationale: case repair is a display concern, applied only on request and never destructively. diff --git a/nameparser/_render.py b/nameparser/_render.py index d26eb76c..d3634669 100644 --- a/nameparser/_render.py +++ b/nameparser/_render.py @@ -132,8 +132,9 @@ def initials(name: ParsedName, spec: str, delimiter: str, separator: str) -> str that text: `replace(family='de la vega')` initials every word of that field where the same name parsed gives 'j. v.' (rules.md#R3's Accepted - clause, and decisions.md under R3 for why the fallback was tried - and dropped). Valid spec keys: given, middle, family.""" + clause, and decisions.md#R4 for why the fallback was tried + and dropped -- #464 is the crossing that would make it + answerable). Valid spec keys: given, middle, family.""" if not isinstance(delimiter, str): raise TypeError(f"delimiter must be a str, got {delimiter!r}") if not isinstance(separator, str): diff --git a/tests/v2/test_render.py b/tests/v2/test_render.py index 87af670f..32f312e0 100644 --- a/tests/v2/test_render.py +++ b/tests/v2/test_render.py @@ -441,8 +441,8 @@ def test_initials_has_no_lexicon_so_a_spliced_field_is_all_name_words()\ read one from, so every word of it initials. That disagrees with the facade and with the same name parsed, and - it is a 2.0-core defect rather than a decision -- see the issue - filed for giving `Parser` an `initials` crossing. A fallback to + it is a 2.0-core defect rather than a decision -- see #464, filed + for giving `Parser` an `initials` crossing. A fallback to `Lexicon.default()` was written and dropped: it guesses a vocabulary, and under a caller's own the guess erases a whole field (`Lexicon.default().add(particles={'y'})`, family 'de y').