Merge pull request #463 from derek73/fix/458-461-render-honors-the-parse

Render views honor the parser's decisions instead of re-deciding (#458)

Author: derek73

docs/design/decisions.md

@@ -1101,7 +1101,17 @@ R3. Rationale: initials abbreviate the person's name words; titles,
     contribute nothing — except the particles of a part whose every
     word is one, which are not acting as particles there (R2) and
     initial like any other name word. A CONJUNCTION never initials,
-    so a base that is one contributes nothing even then.
+    so a base that is one contributes nothing even then. That
+    carve-out is stated for the middle and base family words; the
+    GIVEN group is not settled here. A conjunction written among
+    given names does initial today, and this document does not yet
+    say whether it should — because two of its own rules answer
+    differently and neither answer has been taken: this rule counts
+    name words, while P3 makes a connective and its neighbours ONE
+    name word, so a joined given group owes one initial under P3 and
+    one per joined name word under the carve-out. Until that is
+    decided the given group's answer is pinned-but-undocumented
+    rather than specified, and no line below asserts it.
       "Dr. Juan Q. Xavier de la Vega III"  →  initials="J. Q. X. V."
       "Anh Do"                    →  initials="A. D."
       "Nguyen, Van Le"            →  initials="V. L. N."
@@ -1110,7 +1120,32 @@ R3. Rationale: initials abbreviate the person's name words; titles,
     rather than nothing: they are the base (R2), so they initial.
       "Juan van der"              →  initials="J. v. d."
       "Juan de y"                 →  initials="J."
-    history: decisions.md#R2 · interacts: R2 · implemented: nameparser/_render.py, nameparser/_facade.py
+    Accepted: this rule reads a part the parser read. A field set as
+    raw text after the parse carries no reading, and this view is
+    handed no vocabulary to supply one — it takes a format spec and
+    two separators and nothing else — so every word of such a field
+    initials, particles and conjunctions alike: a family set that way
+    to "de la vega" gives "j. d. l. v." where parsing the same
+    name gives "j. v.". Case repair IS handed a vocabulary, so it falls
+    back for the one question a word can answer on its own, and R4
+    says which. Revising the field through the parser classifies it
+    and matches the parse in both views. Stated without an example
+    line because every line here names an input string, and this
+    shape needs a field edited after the parse.
+    Accepted: the unsettled given-group answer above is neither rare
+    nor hypothetical — 25 of the corpus names carry a conjunction
+    among the given names, every one of them reachable from the
+    default vocabulary, and it has initialed since 1.4.0. It carries
+    no marked deviation, for the reason that mechanism exists: a
+    marker states the INTENDED value, and one name, "John and Jane
+    Smith", has four candidates. Today gives "J. a. J. S."; the
+    carve-out read as written gives "J. J. S."; P3's one-name-word
+    join gives just "J. S."; and 1.4.0 gave "J a J. S.". Marking it
+    would put an invented value in a normative document and hold
+    the parser to it. #461 asks the neighbouring question about the
+    all-particle base and does not own this one; decisions.md#R2
+    carries the population and the measurements.
+    history: decisions.md#R2 · interacts: P3, R2, R4 · implemented: nameparser/_render.py, nameparser/_facade.py
 
 R4. Rationale: case repair is a display concern, applied only on
     request and never destructively.
@@ -1145,6 +1180,27 @@ R4. Rationale: case repair is a display concern, applied only on
     repaired exactly this way before the clause existed. Stated
     without an example line because every line here names an input
     string, and this shape needs a field edited after the parse.
+    Accepted, and the reason the boundary is drawn per question
+    rather than per field: the conjunction carve-out reaches a word
+    the parse read as a conjunction, and where the parse read nothing
+    at all it reaches what the vocabulary says. A word of that
+    vocabulary standing inside a longer written word is not a
+    conjunction, because the parse read that word as one ordinary
+    name word — but a field spliced in as raw text was read by
+    nobody, so repair asks the vocabulary and a family set to "de y"
+    keeps its "y" lowercase. Whether a word is the conjunction or an
+    initial is a property of the word, which a vocabulary can answer;
+    whether a particle is acting as a particle is a property of the
+    whole part, which the parse settles and records; re-deriving it
+    needs a reading on every word of the part, and a spliced field
+    has none on any, so that half falls through to particle treatment
+    and the "de la" boundary above stands. Initials are the contrast
+    worth knowing, and R3 states it: that view is handed no
+    vocabulary at all, so it falls back on neither question and a
+    spliced field's every word initials. revise() classifies the
+    value and crosses both questions, in both views: a middle revised
+    to "e-f" repairs to "E-F" as the parsed name does, where splicing
+    the same text in gives "e-F".
     history: decisions.md#R4 · interacts: R2, R3, R5 · implemented: nameparser/_render.py
 
 R5. Rationale: mixed case is evidence that the writer cased the name

docs/design/mechanisms.md

@@ -67,6 +67,8 @@ Release Log
 
     - Fix case repair lowercasing the words of a family name made only of particle words, where every other view already reads them as ordinary name words: ``HumanName("ANH DO").capitalize()`` gives ``Anh Do`` where it gave ``Anh do``, and ``"anh van do"`` gives ``Anh Van Do`` where it gave ``Anh van do``. A particle earns its name by joining forward to the word it modifies, so a part whose every word is particle vocabulary leaves none of them anything to join; the fix above already made those words anchor ``family_base`` and contribute initials, and case repair now agrees with them rather than reading the same word two ways. The test is the whole part, not a particle standing alone, which is why the two-word family in ``"anh van do"`` moves along with the one-word family in ``"ANH DO"`` -- the same Vietnamese surname, and a standing-alone test would have read it one way behind a given name and another way alone. This DIFFERS FROM 1.4.0 deliberately and does not restore it: 1.4.0 returned ``Anh do``, lowercasing on vocabulary membership alone. The accepted cost is that a degenerate family which is nothing but particles capitalizes too, so ``"juan van der"`` gives ``Juan Van Der`` where 1.4.0 gave ``Juan van der``. A conjunction is untouched by any of this, so ``"der, y van"`` gives ``y Van Der`` -- the family capitalizing while the conjunction keeps the lowercase it always had; and where the particles DO join a name word nothing changes, ``"juan de la vega"`` still giving ``Juan de la Vega``. Case repair is not one of the seven role fields the differential harness compares, so no gate run can see this change either way and none of its counts move; the ``rules.md#R4`` examples and the v1 capitalization tests are what pin it (closes #407)
 
+    - Change case repair to read the parser's own ``conjunction`` tag instead of re-deciding, from the word's spelling, whether a word is a conjunction or an initial. The parse answers that question already -- ``"Scott E. Werner"`` reads ``E.`` as an initial rather than the Italian conjunction -- and the other views honor the answer; case repair asked again, with a shape test applied to each word of a token's text rather than to the token. Two spellings of one name disagreed because of it: ``"juan e-f smith"`` capitalized to ``Juan e-F Smith`` while ``"JUAN E-F SMITH"`` gave ``Juan E-F Smith``; both give ``Juan E-F Smith`` now, ``e-f`` being a middle name and no conjunction of the parse's reading. A conjunction written as a word of its own is untouched, and so is the one-letter carve-out where it applies -- ``"juan y garcia"`` still repairs to ``Juan y Garcia``, ``"JUAN Y GARCIA"`` still to ``Juan Y Garcia``. A field assigned after the parse is unaffected: its text was never classified, so there is no reading to honor and repair asks the vocabulary, applying v1's own predicate the way every earlier version applied it everywhere -- ``h.last = "velasquez y garcia"`` still repairs to ``Velasquez y Garcia`` and ``h.middle = "e."`` to ``E.``. That is the predicate over TODAY's vocabulary, which is narrower than parity with 1.4.0 and the difference is real: ``h.last = "хосе и мария сантос"`` gives ``Хосе И Мария Сантос`` on 1.4.0 and ``Хосе и Мария Сантос`` here, because the Cyrillic ``и`` is a 2.x conjunction and was not a 1.4.0 one. What decides which path a token takes is a mark the assignment leaves, not the absence of a span: a value revised through ``Parser.revise()`` is classified by a sub-parse and keeps its tags, so it repairs as the parse does. One reading does change for hand-built ``Token``\ s in the 2.0 API: an untagged token whose text is conjunction vocabulary is now an ordinary name word and capitalizes, where 2.1 lowercased it -- tags are what the views read, and a hand-built token that carries none is a token with nothing to declare. Case repair is not one of the seven role fields the differential harness compares, so no gate run can see this change either way and none of its counts move; measured directly instead, no name of the 1094-name differential corpus moves under ``capitalized()`` or ``capitalized(force=True)``, its uppercased and lowercased spellings included -- 6564 name/spelling/lexicon rows and 13128 calls (closes #458)
+
     - Fix a tussenvoegsel attached to the family name after a comma deciding a genuinely uncertain reading and reporting nothing. ``"Van Johnson"`` reports a ``PARTICLE_OR_GIVEN`` ambiguity -- ``Van`` is a Dutch particle and a Vietnamese given name, and the parser has to pick one -- while ``"Nguyen, Thi Van"`` picked the same word the same way, silently, and lost the given name doing it. The attachment now reports the fork it decides, in the kind that names the reading it declined. A particle that could be an ordinary name reports ``PARTICLE_OR_GIVEN``: ``"Nguyen, Thi Van"``, ``"Berg, Jan van der"`` and ``"Vega, Juan de la"`` each gain one, the ``detail`` naming the ambiguous word. A particle the parser had already read as a post-nominal reports ``SUFFIX_OR_NAME`` instead, because the credential reading is what the attachment overrode: ``"Berg, Jan vd"`` gains one, ``vd`` being read as *van der* rather than as the Volunteer Decoration. Which kind you get follows the reading that was overridden rather than the word's vocabulary, so ``"Berg, Jan do"`` reports ``PARTICLE_OR_GIVEN`` even though ``do`` is a postnominal too -- it was already being read as a name word, so no credential reading was overridden. A particle where nothing was overridden reports nothing at all: ``"Jong, Piet de"`` and ``"Jong, Anke de"`` are unchanged, ``de`` being no name in any reading and no postnominal either, and so is ``"Berg, Jan de vd"``, whose run was read as name words whole. Worth knowing before you filter on this: ``"Beethoven, Ludwig van"`` -- the textbook Dutch listing, read exactly right -- now carries a report too. It is the same string shape over the same vocabulary as ``"Nguyen, Thi Van"``, and nothing in the input separates them, so a report on one is a report on both. This adds the report and nothing else: every field these names parse to is exactly what the tussenvoegsel fix below already gave them, and ``ambiguities`` is the only value that grows. Seven differential corpus names gain a kind against the 2.0.0 and 2.1.0 baselines, seven of the nine the tussenvoegsel fix below already moved -- five of ``fix(#379)``'s seven and both of ``fix(#380)``'s two, now carrying ``_ambiguities`` in their diff as well; the two left out are the pair named unchanged above, ``Jong, Piet de`` and ``Jong, Anke de`` -- and none against 1.4.0, which had no ambiguity reporting at all (closes #405)
 
     - Fix a tussenvoegsel after a family comma being parsed as a middle name. Dutch and Belgian alphabetized listings move the particle behind the given name -- ``"Beethoven, Ludwig van"`` is how ``"Ludwig van Beethoven"`` is filed -- and the trailing particle run was read as a middle name rather than as part of the surname: ``"Beethoven, Ludwig van"`` gave middle ``van``, last ``Beethoven``, and ``"Berg, Jan van der"`` gave middle ``van der``. The run now attaches to the family the comma has already named and renders before it, so those read family ``van Beethoven`` and ``van der Berg`` with the given name unchanged. The derived views move with the parse, so ``family_particles`` is ``van`` and ``family_base`` is ``Beethoven`` where they were empty and ``Beethoven`` before. `#130 <https://github.com/derek73/python-nameparser/issues/130>`_ asked for the split and got it in 1.3.0 as ``last_base``/``last_prefixes``; 2.0 renamed them ``family_base``/``family_particles``. What was wrong until now was the values they reported for this listing. Both halves of the particle vocabulary attach -- never-given ``de`` and may-be-given ``van`` alike -- because after a comma the family is already named and the particle has no other role to take. Two guards bound it. A name whose only given word is the particle keeps it, so ``"Nguyen, Van"`` still reads given ``Van``: the attachment needs a given word to spare. And where the word is BOTH particle and suffix vocabulary the attachment outranks the post-nominal reading, so ``"Berg, Jan vd"`` reads family ``vd Berg`` where 1.4.0 and 2.1 alike gave suffix ``vd`` -- a trailing abbreviation after a family comma is the tussenvoegsel far more often than the decoration it collides with, and the same shape sweeps in ``mc``, which 2.1 also read as a suffix. ``do`` is in ``SUFFIX_ACRONYMS_AMBIGUOUS`` and 2.1 already read a trailing one as a name word, so it attaches by the plain rule rather than by the override (closes #379, closes #380). Names without the comma are untouched: ``"Ludwig van Beethoven"`` already read family ``van Beethoven`` and is byte-identical. One of the 751 differential corpus names moves, ``"Vega, Juan de la"``, at the 1.4.0, 2.0.0 and 2.1.0 baselines alike

docs/design/rules.md

@@ -887,9 +887,23 @@ clears the flag, while correcting an unrelated field keeps it.
 
 ``replace()`` splits values on whitespace into plain, untagged
 tokens — the vocabulary knowledge a parse would have about the new
-text is not there. The views that depend on tags degrade: the parser
+text is not there. The views that read those tags degrade: the parser
 no longer knows ``de la`` are particles, so ``family_particles``
-empties and the particles start contributing initials.
+empties and ``family_base`` takes the whole field.
+
+A token the parse never saw carries no decision to honor, so a view
+that is *handed* a vocabulary can fall back to it —
+:meth:`~nameparser.ParsedName.capitalized` is the one that is, and it
+falls back for one question only: whether a word is a conjunction or
+an initial, which a word answers on its own. Whether a particle is
+acting as a particle is a fact about the whole part, and there is no
+reading on any word of a spliced field to derive it from, so a family
+set to ``de la`` stays lowercase where the same words parsed are
+repaired to ``De La``. :meth:`~nameparser.ParsedName.initials` takes
+no vocabulary at all, so it falls back on neither question and every
+word of a spliced field contributes an initial. ``family_particles``
+and ``family_base`` are properties on the parsed name, which holds no
+vocabulary of its own either.
 
 .. doctest::
 
@@ -898,8 +912,12 @@ empties and the particles start contributing initials.
     >>> replaced = name.replace(family="de la Vega Smith")
     >>> replaced.family_particles
     ''
+    >>> replaced.family_base
+    'de la Vega Smith'
     >>> replaced.initials()
     'J. d. l. V. S.'
+    >>> name.replace(family="de la").capitalized(force=True).family
+    'de la'
 
 :meth:`Parser.revise() <nameparser.Parser.revise>` is the same
 operation with each value classified by the parser's vocabulary, so

docs/release_log.rst

@@ -32,7 +32,8 @@
 from nameparser._config_shim import CONSTANTS, Constants, _cached_parser
 from nameparser._lexicon import _normalize
 from nameparser._parser import Parser
-from nameparser._types import FOLDED_TAG, ParsedName, Role, Token
+from nameparser._types import (FOLDED_TAG, UNCLASSIFIED_TAG, ParsedName,
+                               Role, Token)
 
 _V2_FIELD = {"first": "given", "last": "family"}  # v1 name -> v2 name
 _V1_SPELLING = {v2: v1 for v1, v2 in _V2_FIELD.items()}
@@ -746,8 +747,17 @@ def __setstate__(self, state: dict[str, Any]) -> None:
                         f"nameparser"
                     )
                 for position, word in enumerate(entry.split()):
-                    tokens.append(Token(
-                        word, None, role,
-                        frozenset({"joined"}) if position else frozenset()))
+                    # UNCLASSIFIED_TAG for the same reason replace()
+                    # stamps it: a pickle carries the *_list STRINGS
+                    # and no tags, so nothing here was read by a parse
+                    # and case repair must ask the vocabulary rather
+                    # than read an absent conjunction tag. Without it a
+                    # restored "juan ortega y gasset" repairs to
+                    # "Ortega Y Gasset", which is neither v1's answer
+                    # nor the same name's unpickled one.
+                    tags = {UNCLASSIFIED_TAG}
+                    if position:
+                        tags.add("joined")
+                    tokens.append(Token(word, None, role, frozenset(tags)))
         self._parsed = ParsedName(
             original=str(state.get("original", "")), tokens=tuple(tokens))