diff --git a/docs/design/decisions.md b/docs/design/decisions.md index c794ae8a..692f56f2 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -118,6 +118,21 @@ Open: [#380](https://github.com/derek73/python-nameparser/issues/380) covers "Be Measured over the 1094-name differential corpus: P6's attachment fires on 9 names, of which 5 take the ambiguous arm (`Beethoven, Ludwig van`, `Berg, Jan van der`, `Nguyen, Thi Van`, `Vega, Juan de la`, `der, y van`), 2 the suffix arm (`Berg, Jan vd`, `Berg, abdul vd`) and 2 stay silent (`Jong, Anke de`, `Jong, Piet de`). Seven names gain an ambiguity kind and no role field moves that was not already moving. Read that seven as evidence about the corpus rather than about the blast radius, the caution the 2026-08-16 entry above already gives for this rule: 304 of the 1094 names carry a comma and only 13 end in a particle, 11 of those being rows the #379/#380 arc added itself (this change added four corpus names and not one of them carries a comma, so the 11 is unchanged by it). The reach is a SHAPE — every family-comma listing ending in an ambiguous particle, which is to say ordinary Dutch, Flemish and unaccented Vietnamese listings — and a caller holding such records sees a report on all of them, not on seven. The LEDGER shape that follows from that is not the one the plan predicted, and the difference is worth recording because it will recur for any report added to a rule whose fields already move. A new `fields = ["_ambiguities"]` rule explains none of the seven: `classify` matches on a rule's `fields` being a SUPERSET of the whole diff, and against 2.0.0 and 2.1.0 these names diff in `middle`/`family` (or `family`/`suffix`) from #379 and #380 as well. So the repair is to WIDEN those two existing rules to `{middle, family, _ambiguities}` and `{family, suffix, _ambiguities}` in both ledgers, not to add a rule. Each widened rule still passes the #452 equality check, since the union over the names it explains is exactly what it declares — the two silent `Jong` names contribute `{middle, family}` and the five ambiguous ones contribute `_ambiguities` on top. 1.4.0 needs nothing at all, having no ambiguity surface to diff. Gate before and after: byte-identical output at all three baselines apart from the provenance header — 1094 names, 229/194/102 intentional, 0 unexplained, and no per-heading count moved. The seven names were already in the diff; only their field sets grew. The most visible consequence, and it is not the Vietnamese one: `Beethoven, Ludwig van` — the textbook-correct Dutch listing, read exactly right — now carries a report. That is honest rather than regrettable. The parser cannot separate it from `Nguyen, Thi Van`; the two are the same string shape over the same vocabulary, and a report on one is a report on the other. A caller who wants only the doubtful cases has nothing here to filter on, because there is nothing to filter on in the input either. +- 2026-08-30 #365 (the no-comma site) — a declared family-first order names the family, so the attachment fires without a comma too. This DECIDES the parenthesis the 2026-08-16 Scope bullet above left open ("arguably a missing comma under a declared family-first order") and narrows that bullet to the default order, which is the only place nothing names the family. The same Dutch listing is written both ways and now reads alike: `Jong, Anke de` and `Jong Anke de` under a family-first order both give family 'de Jong'. + CREDIT WHERE THE DESIGN CAME FROM, since a first draft of this entry claimed it for the PR. The issue proposed adding MIDDLE as a third P1 site and its revision comment proposed dropping the `FAMILY_FIRST_GIVEN_LAST` fold, and those two ARE both worse — the first loses `Garcia` from `Mesnil Garcia de` because a particle trailed it, the second reports `de` AS the given name, which P1 exists to prevent. But the same comment records a third reading and it is the one that shipped, mechanism included: family-first as an implied family comma, with the note that today's fold "folds while preserving input order" and that `FOLDED_TAG` in `_types._text_for` "already performs exactly that reorder". What this PR added is the guards, not the idea. + NEVER-GIVEN ONLY, where the comma path takes ambiguous particles too, and P6's own Accepted clause is what decides it: the comma-less `FAMILY_FIRST_GIVEN_LAST` form is the ONE format that reads the unaccented Vietnamese spelling correctly, and an ambiguous-inclusive attachment takes that format too — measured, `Nguyen Thi Van` under that order would give family 'Van Nguyen', given 'Thi', where P6's example pins given 'Van'. The words-to-spare guard does not save it (a given word does remain), so the vocabulary line is the only one that holds. The cost is the Dutch flagship word itself: `van` is ambiguous, so `Beethoven Ludwig van` keeps middle 'van' where `Beethoven, Ludwig van` gives family 'van Beethoven'. The two forms agree for the never-given half of the tussenvoegsel set and not for the rest, and the rule says so. + THREE CONDITIONS, EACH A DEFECT BEFORE IT WAS A CONDITION. All three were found by the review round, none by the corpus, and none by the suite. + (i) A NAME WORD MUST REMAIN BESIDES THE FAMILY — P6's own words-to-spare test, in the shape a positional read needs it. Without it the no-comma site was WIDER than the comma site it claims to narrow: `Jong de` attached and gave family 'de Jong' where `Jong, de` declines and gives 'Jong de', so two writings of one one-given-word listing disagreed about word order. It would also have half-decided the no-given-word case that the Open note below deliberately leaves open. With it, the two forms agree row for row, and `Mesnil de` stays P1's fold — which is why P1's example line for it is unchanged by this PR. + (ii) THE FAMILY MUST HAVE A BASE OF ITS OWN, not merely a name word somewhere ahead. The piece test passed on `van Berg Jan de` under `FAMILY_FIRST` and the name lost its given name outright — 'Berg' on master, empty after, family 'de van Berg Jan' — because the leftover is partitioned into UNITS and `_units` chains a leading particle over every word after it, so `[van][Berg][Jan]` collapses to ONE unit and `_name_positions` gives it FAMILY. 288 of a 45,648-parse battery lost a given name that way. P1 masks it for every never-given opening (its lead site fires afterward and re-runs its own narrower reach), which is why every fixture written for this rule — all opening with `de` — missed it. + (iii) NO SUFFIX WORD MAY BE STRANDED AT THE END. assign leaves a trailing `Jr.` in a name position when a particle follows it; take the particle away and the leftover is not the same name one word shorter, because the re-layout does not re-run assign's trailing peel. `Berg Jan Jr. de` reported given 'Jr.' under `FAMILY_FIRST_GIVEN_LAST`, and every battery name where the two family-first orders agreed before and disagreed after was this shape — the rule re-opening the divergence it exists to close. The rule DECLINES rather than re-running the peel: mirroring assign by hand is the failure this project has on record from #424/#425, and a boundary is cheaper than a second copy of its walk. + NO STRUCTURE TEST, which was the fourth defect and the opposite kind — the rule missing an instance of its own bug. `state.order is not None` already excludes every comma that named a family, since assign records no order there; testing `structure is not FAMILY_COMMA` as well excluded ONLY the comma that fixed nothing (segment 1 holding no name word, assign reading segment 0 positionally, #296), where #365's symptom survived verbatim: `Mesnil Garcia de, Dr.` stranded `de` as a middle under `FAMILY_FIRST` and swallowed the given slot under `FAMILY_FIRST_GIVEN_LAST`. Segment 0 is this site's for the same reason, which is `_leading_name_piece`'s rule read at `order is not None`. + THE TWO SITES ARE MUTUALLY EXCLUSIVE, and that retired a guard. P1's lead site fires only on a leading LONE never-given particle — a family piece with no base, exactly what condition (ii) rejects — so an exclusion was added to P1's redistribution, for tokens this site had already placed, and is now unreachable: 0 differences over 61,854 generated parses with it removed. It is deleted rather than kept defensively, and `tests/v2/pipeline/test_post_rules.py` pins the exclusivity with the name that needed it (`de la Vega de`). Recorded because the shape it prevented is real and would return if condition (ii) ever loosened. + REACH: 12 of 6600 parses move — the shipped 1100-name corpus × three `name_order` values × `middle_as_family` off and on — over four names (`Jong Anke de`, `Mesnil Garcia de`, `Berg Jan de Jr.`, `Mesnil Garcia Carlos de`), none under the default order. TWO OF THE FOUR ARE NAMES THIS PR'S OWN EXAMPLE LINES ADD, which is why the corpus is 1100 and not master's 1094: `corpus_rules.jsonl` regenerates from rules.md, so a rule that adds examples adds corpus names and then counts them. State the corpus with the count or the two drift apart — a first draft of this entry said "14 of 6564 ... 1094 corpus names" and by the time it shipped both numbers named a corpus the tree no longer had. + Recompute by parsing the four `tools/differential/corpus*.jsonl` files of THIS tree, deduped, under each of the six policies, and comparing the seven role fields against a checkout of the parent commit reading the SAME corpus files. The comparator is the pre-change code, never a construction of this rule's own. The count is a dated snapshot, measured 2026-08-30: nothing in the repository holds the parent tree. + THE DIFFERENTIAL GATE CANNOT SEE ANY OF IT, for a reason different from the render views' — `tools/differential/compare.py` parses every corpus name as `HumanName(name)` and sweeps no policy at all, so no non-default `name_order` behavior has ever been compared across versions. Gate output is byte-identical before and after at all three baselines apart from the corpus names the new example lines add, none of which diffs at any baseline. The issue's own line "Needs a differential pass: it changes parse output for shipped policies" is true about the change and wrong about the instrument. + The issue's headline number does not measure this shape. "464 inputs where the two orders differ in field content" is what permuting the given and middle positions DOES; measured on the pre-change tree over master's corpus it was 247 of 1094, and this change reduces it, so the number cannot be quoted in the present tense at all. The shape itself — a lone never-given particle standing alone in a non-family role beside a real family — was 5 rows over that corpus × 3 orders before the change, three of them the comma case `Jong, van der` that P6's words-to-spare guard declines on purpose. + + ### M2 — the maiden-marker rule - 2026-07-03 (maiden-bucket design; #274 filed 2026-07-07, landed in the v2 core, PR #288) — the marker takes everything after it up to a trailing suffix, greedily: "née Jones Smith" is a two-word maiden name, matching how the marker is actually used in running text. The marker itself is dropped as structural, like a delimiter character. diff --git a/docs/design/rules.md b/docs/design/rules.md index 02347166..d8e66d34 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -123,7 +123,15 @@ P1. Rationale: a never-given particle standing alone cannot be declaring that order asserts that what follows the family is not more surname. Opening the name under the default order, or standing in the given position under any order, it takes the rest - of the name: nothing there marks where the surname ends. One name + of the name: nothing there marks where the surname ends. One + case of that second position is P6's rather than this rule's, + and only one: a particle ENDING the name under a declared + family-first order, where a family with a base and a name word + besides it both remain. P6 attaches it to that family and writes + it before the base. Everything else in the given position is + still this rule's fold, including the name that has nothing left + once the particle is taken — which is why "Mesnil de" reports + family "Mesnil de" here and not P6's "de Mesnil". One name word means one UNIT — a particle chain (P2), a conjunction join (P3) or a bound given-name pair (P5) is taken whole or not at all. A title does not move the opening position (P4), but a @@ -365,7 +373,9 @@ P6. Rationale: a particle ending the name has nothing to link after a comma there is no signal that separates the two readings. Dutch and Flemish names are listed exactly this way ("Beethoven, Ludwig van"), the tussenvoegsel trailing - the given name but belonging to the surname. + the given name but belonging to the surname. The comma is not + the only thing that names the family: a declared family-first + order says the same, and the same listing is written both ways. Where a family comma has already named the family, a particle ending the name attaches to that family name and is written before it — provided at least one given word remains, so that a @@ -388,6 +398,42 @@ P6. Rationale: a particle ending the name has nothing to link "Beethoven, Ludwig van" → family_base="Beethoven" "Beethoven, Ludwig van" → family_particles="van" "Nguyen, Van" → given="Van" · boundary + Without a comma, a declared family-first order has named the + family in the same way and the attachment fires there too. The + words-to-spare test is the same one, in the shape a positional + read needs it: a name word must remain ahead of the run BESIDES + the family it attaches to, so the one-given-word listing keeps + the reading it has either way — "Jong, de" and "Jong de" under a + family-first order both report family "Jong de", which is P1's + fold and not this rule. Two conditions are the comma's alone. + Only a NEVER-GIVEN particle attaches, an ambiguous one keeping + whatever reading its position gives it (P1). And the suffix + reading stands: a trailing abbreviation is a post-nominal until + a comma makes the tussenvoegsel commoner, so the run is found + among the pieces that hold a name, a trailing piece holding none + being walked past as it is there. The family the run attaches to + must have a base of its own — a word no particle vocabulary + claims (R2) — since a family that is all particles is not a + family written beside anything. + The pieces left over are laid out in the declared order for + their own number. At two of them that is family-then-given in + either family-first order, which is why the orders agree on the + shape this rule was filed for; at three they still differ, the + declared order being what decides given from middle. + "Jong Anke de" family-first → family="de Jong" + "Jong Anke de" family-first → given="Anke" + "Mesnil Garcia de" family-first → family="de Mesnil" + "Mesnil Garcia de" family-first → given="Garcia" + "Mesnil Garcia de" family-first-given-last → family="de Mesnil" + "Mesnil Garcia de" family-first-given-last → given="Garcia" + "Berg Jan de Jr." family-first → family="de Berg" + "Berg Jan de Jr." family-first → suffix="Jr." + "Mesnil Garcia Carlos de" family-first → given="Garcia" + "Mesnil Garcia Carlos de" family-first-given-last → given="Carlos" + "Berg Jan vd" family-first → suffix="vd" + "van der" family-first → family="van der" · boundary + "Mesnil de" family-first → family="Mesnil de" · boundary + "van Berg Jan de" family-first → given="Berg" · boundary Accepted: an ambiguous particle attaches on the same terms as a never-given one, so a Vietnamese name written in this listing loses its given name — but only in the UNACCENTED @@ -417,12 +463,22 @@ P6. Rationale: a particle ending the name has nothing to link "Nguyen Thi Van" family-first-given-last → family="Nguyen" "Nguyen Thi Van" family-first-given-last → given="Van" "Nguyen Thi Van" family-first → middle="Van" - Accepted: without a family comma the name's written shape is not - settled — "Jong Anke de" may be a misformatted listing, and a - bare "Jong de" may be a given name beside a particle — so the - attachment is scoped to the comma form, and the comma-less - shapes keep their positional reading. + Accepted: under the DEFAULT order a comma-less name's written + shape is not settled — "Jong Anke de" may be a misformatted + listing, and a bare "Jong de" may be a given name beside a + particle — so nothing there names the family and the positional + reading stands. What the attachment needs is not the comma but + what the comma says; a declared family-first order says it, and + the default order says nothing. "Jong Anke de" → family="de" + Accepted: the never-given restriction cuts across the Dutch + tussenvoegsel set, so the two formats agree for the never-given + `de` and not for the ambiguous `van`. The Accepted clause above + is what decides it: the comma-less family-first-given-last form + is the ONE format that reads the unaccented Vietnamese spelling + correctly, and an attachment taking ambiguous particles without + a comma would take that format too. + "Beethoven Ludwig van" family-first → middle="van" Accepted: the precedence over S2 is stated for the shape, so it sweeps in every word that is both particle and suffix vocabulary — today vd, do and mc. Only vd's reading was @@ -440,6 +496,9 @@ P6. Rationale: a particle ending the name has nothing to link it as its pair first (P5), so the attachment never sees it — unless the particle is of the unambiguous suffix vocabulary too (vd, mc), which the join declines and the attachment then takes. + That last exception is the COMMA site's: without a comma the + suffix reading stands, so a family-first `Berg abdul vd` reports + suffix `vd` where `Berg, abdul vd` reports family `vd Berg`. "Berg, abdul van" → given="abdul van" "Berg, abdul vd" → family="vd Berg" history: decisions.md#P6 · interacts: A1, C1, P1, S2, P5 · implemented: nameparser/_pipeline/_post_rules.py diff --git a/docs/release_log.rst b/docs/release_log.rst index 7a22ec8d..7fb62fb9 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -16,6 +16,8 @@ Release Log the shapes and the corpus names it moves, and the gate output is the source of record. The unusual part is the family-first orders, which change too: seven names from the family-first fix, + four more from the trailing-particle fix below (#365), which is + family-first only and moves nothing under the default order, plus the names #367's title fix moves, which it moves in every order. The other title fix below, #410, is default-order only -- under either family-first order the word behind the title is @@ -61,7 +63,7 @@ Release Log - Fix a space-separated credential run after a family comma rendering with a comma the name never had: ``"Smith, MD PhD"`` gives suffix ``MD PhD`` where it gave ``MD, PhD``, and ``"Smith, CBE MC"``, ``"Smith, BSc MBA"``, ``"Smith, PhD Jr."`` and ``"Smith, Dr. MD PhD"`` the same. The roles are unchanged; only the rendered string carried the extra comma. One comma segment is one suffix entry, and grouping decided which segments those were by their position while assignment decided the same segment by its content, so the two disagreed about the segment a family comma leaves after it. The reading follows assignment now, which is where the segment's roles were already decided. This reaches any family comma whose following segment holds no name word, not only a one-word family: ``"John Smith, Jr. III"`` gives suffix ``Jr. III``, which is also what 1.4.0 gave. The full-name comma form was never affected -- ``"John Smith, MD PhD"`` has given suffix ``MD PhD`` since 1.4.0 -- and this brings the rest into line with it. A run written with commas keeps them (``"Smith, MD, PhD"`` is still suffix ``MD, PhD``), a comma between two entries survives (``"Smith Jr., Mr. Jr."`` is still suffix ``Jr., Jr.``), and a name word in the segment still makes it the given-and-suffix reading (``"Smith, John Jr."``). A name written without a comma is unaffected and still renders its run comma-joined, so re-parsing ``str()`` output does not reproduce the run (closes #429) - Fix a one-character suffix word after a comma being read by the wrong neighbour: ``"Smith, PSM I"`` gives suffix ``PSM I`` where it gave given ``PSM`` and suffix ``I``, and ``"Smith, John V."`` gives middle ``V.`` where it gave suffix ``V.``. Inside a comma part a suffix word short enough to be mistaken for an initial -- ``I``, ``V`` and ``2`` in the shipped vocabulary -- is read by what stands before it. Behind a credential it is describing that credential -- ``PSM I`` is Professional Scrum Master level I -- so the run continues through it whether or not a period is written, since an initial in that position is not a name shape anyone writes; ``"Smith, MD I"`` and ``"Smith, Jr. I"`` reached the same wrong answer by a different route and move with it. Behind a name the period decides, marking an abbreviation and so a middle initial, exactly as ``"Smith, John B."`` always read. A numeral written bare after a name is still the generation it looks like (``"Smith, John V"`` is suffix ``V``), and a name with no comma is untouched (closes #430, closes #432) - - Fix a name opening with a particle that is *never* a given name being split at the particle under a family-first name order -- ``Policy(name_order=FAMILY_FIRST)`` and ``Policy(name_order=FAMILY_FIRST_GIVEN_LAST)`` alike, and identically: ``"de Mesnil"`` read as family ``de``, given ``Mesnil``, and ``"de la Vega"`` as family ``de``, given ``la Vega``. Each is now the whole surname, as it has always been in the default order. The rule enforcing it asked for the particle by the ``GIVEN`` role, which under a family-first order belongs to the token *after* the particle, so the test read the wrong word and declined. It now also asks by position -- the piece that opens the name -- so both shapes of the same rule are caught: where such a particle stands alone as a piece, either opening the name or in the given position, the name is left with no given name at all, the given and the middles folding into the family. Standing *alone* is the whole of it, and the rule claims nothing wider: ``"Juan de la Vega"`` under ``FAMILY_FIRST`` still reports given ``de la Vega``, because there the particle chained onto the words after it rather than standing alone, and a bare ``"de"`` with nothing to fold into is still reported as the given name. The decision behind the fix: a word that can never be a given name leaves ``name_order`` nothing to decide, so declaring family-first is not a reason to make ``de`` a surname on its own. A leading particle that *may* be a given name is genuinely order-dependent and is untouched -- ``"van Gogh"`` still reads as family ``van``, given ``Gogh`` under both family-first orders. This is also what gives ``Lexicon.particles_ambiguous`` an effect outside the default order: taking a word out of it now changes the parsed fields under a family-first order, where before it moved only the ambiguity report. Seven of the 751 differential corpus names move, the same seven under each family-first order; default-order output is byte-identical over all 751, at the 1.4.0, 2.0.0 and 2.1.0 differential baselines alike (closes #359) + - Fix a name opening with a particle that is *never* a given name being split at the particle under a family-first name order -- ``Policy(name_order=FAMILY_FIRST)`` and ``Policy(name_order=FAMILY_FIRST_GIVEN_LAST)`` alike, and identically: ``"de Mesnil"`` read as family ``de``, given ``Mesnil``, and ``"de la Vega"`` as family ``de``, given ``la Vega``. Each is now the whole surname, as it has always been in the default order. The rule enforcing it asked for the particle by the ``GIVEN`` role, which under a family-first order belongs to the token *after* the particle, so the test read the wrong word and declined. It now also asks by position -- the piece that opens the name -- so both shapes of the same rule are caught: where such a particle stands alone as a piece, either opening the name or in the given position, the given and the middles fold into the family. (The trailing half of that -- a particle standing alone in the given position with nothing after it -- is narrowed by the #365 fix below, which attaches it to the family instead wherever a name word remains beside that family, leaving this fold the names that have nothing to spare.) Standing *alone* is the whole of it, and the rule claims nothing wider: ``"Juan de la Vega"`` under ``FAMILY_FIRST`` still reports given ``de la Vega``, because there the particle chained onto the words after it rather than standing alone, and a bare ``"de"`` with nothing to fold into is still reported as the given name. The decision behind the fix: a word that can never be a given name leaves ``name_order`` nothing to decide, so declaring family-first is not a reason to make ``de`` a surname on its own. A leading particle that *may* be a given name is genuinely order-dependent and is untouched -- ``"van Gogh"`` still reads as family ``van``, given ``Gogh`` under both family-first orders. This is also what gives ``Lexicon.particles_ambiguous`` an effect outside the default order: taking a word out of it now changes the parsed fields under a family-first order, where before it moved only the ambiguity report. Seven of the 751 differential corpus names move, the same seven under each family-first order; default-order output is byte-identical over all 751, at the 1.4.0, 2.0.0 and 2.1.0 differential baselines alike (closes #359) - Fix a family name made only of particle words reporting no base on the 2.0 API, so the surname vanished from ``family_base`` and, on both APIs, from the initials: ``parse("Anh Do")`` gave family ``Do`` with ``family_base`` ``''`` and initials ``A.``, and under ``Policy(name_order=FAMILY_FIRST)`` ``"Del Toro"`` gave family ``Del`` the same way. The empty base was a 2.0 regression rather than a longstanding bug -- 1.4.0's own guard kept ``HumanName("Anh Do").last_base`` at ``Do``, and the facade has kept it right throughout; what 1.4.0 and 2.1 shared was the missing initial. A particle earns its name by joining forward to the word it modifies, so a particle standing alone in a name part is not doing a particle's work there and reads as an ordinary name word: it anchors the base, leaves the particles view, and contributes an initial. ``"Anh Do"`` is now base ``Do``, initials ``A. D.``; ``"Juan van der"`` is base ``van der``, initials ``J. v. d.``; ``"Nguyen, Van Le"`` initials ``V. L. N.`` where the middle name used to be dropped. Position decides this, not vocabulary -- whether the word is borne as a surname somewhere does not enter into it, which reverses the reasoning ``rules.md#R2`` carried before. The invariant it exists to hold: a non-empty family always has a non-empty base, because a particle needs a base to attach to. Where the particles DO join a name word nothing changes -- ``"Juan de la Vega"`` keeps base ``Vega``, particles ``de la`` and initials ``J. V.``. The parse fields themselves do not move: ``title``, ``given``, ``middle``, ``family``, ``suffix``, ``nickname`` and ``maiden`` are byte-identical over all 751 differential corpus names in all three name orders, and only the derived views and the initials change -- which is why the differential harness, which compares those seven roles, reports no diffs for this at any baseline. One consequence to know about: where the whole family is particles AND a tussenvoegsel was attached to it (#379), this change exposed a pre-existing ordering gap between ``initials()`` and the family field of the same name -- fixed in the #408 entry below, so the two agree as shipped (closes #385, closes #402) @@ -69,6 +71,7 @@ Release Log - Change case repair to read the parser's own ``conjunction`` tag instead of re-deciding, from the word's spelling, whether a word is a conjunction or an initial. The parse answers that question already -- ``"Scott E. Werner"`` reads ``E.`` as an initial rather than the Italian conjunction -- and the other views honor the answer; case repair asked again, with a shape test applied to each word of a token's text rather than to the token. Two spellings of one name disagreed because of it: ``"juan e-f smith"`` capitalized to ``Juan e-F Smith`` while ``"JUAN E-F SMITH"`` gave ``Juan E-F Smith``; both give ``Juan E-F Smith`` now, ``e-f`` being a middle name and no conjunction of the parse's reading. A conjunction written as a word of its own is untouched, and so is the one-letter carve-out where it applies -- ``"juan y garcia"`` still repairs to ``Juan y Garcia``, ``"JUAN Y GARCIA"`` still to ``Juan Y Garcia``. A field assigned after the parse is unaffected: its text was never classified, so there is no reading to honor and repair asks the vocabulary, applying v1's own predicate the way every earlier version applied it everywhere -- ``h.last = "velasquez y garcia"`` still repairs to ``Velasquez y Garcia`` and ``h.middle = "e."`` to ``E.``. That is the predicate over TODAY's vocabulary, which is narrower than parity with 1.4.0 and the difference is real: ``h.last = "хосе и мария сантос"`` gives ``Хосе И Мария Сантос`` on 1.4.0 and ``Хосе и Мария Сантос`` here, because the Cyrillic ``и`` is a 2.x conjunction and was not a 1.4.0 one. What decides which path a token takes is a mark the assignment leaves, not the absence of a span: a value revised through ``Parser.revise()`` is classified by a sub-parse and keeps its tags, so it repairs as the parse does. One reading does change for hand-built ``Token``\ s in the 2.0 API: an untagged token whose text is conjunction vocabulary is now an ordinary name word and capitalizes, where 2.1 lowercased it -- tags are what the views read, and a hand-built token that carries none is a token with nothing to declare. Case repair is not one of the seven role fields the differential harness compares, so no gate run can see this change either way and none of its counts move; measured directly instead, no name of the 1094-name differential corpus moves under ``capitalized()`` or ``capitalized(force=True)``, its uppercased and lowercased spellings included -- 6564 name/spelling/lexicon rows and 13128 calls (closes #458) + - Fix a trailing surname particle being stranded as a standalone middle or given name under a declared family-first order, where the same listing with a comma reads it as part of the surname. A particle ending the name has nothing to link forward to and cannot be a name by itself, so after a family comma it attaches to the family the comma named and is written before it -- ``parse("Jong, Anke de")`` gives family ``de Jong``. A family-first order names the family the same way, and the attachment now fires there too: ``Parser(policy=Policy(name_order=FAMILY_FIRST)).parse("Jong Anke de")`` gave family ``Jong`` with ``de`` left as a middle name and now gives family ``de Jong``, given ``Anke``. The two family-first orders had disagreed about the same input, because they permute which role the trailing particle falls into and only one of those roles was inspected: ``"Mesnil Garcia de"`` read family ``Mesnil``, given ``Garcia``, middle ``de`` under ``FAMILY_FIRST`` and family ``Mesnil Garcia de`` with no given name at all under ``FAMILY_FIRST_GIVEN_LAST``. Both now give family ``de Mesnil``, given ``Garcia``. Three conditions keep the rule to that shape, and each is the comma form's behavior read across: a name word must remain BESIDES the family, so a one-given-word listing keeps the reading it has either way (``"Jong, de"`` and ``"Jong de"`` both report family ``Jong de``); the family the particle attaches to must have a base of its own, so a name that is nothing but particles is untouched (``"van der"`` stays family ``van der``); and a post-nominal is not promoted into a name to make room (``"Berg Jan Jr. de"`` is left alone, while ``"Berg Jan de Jr."`` attaches and keeps suffix ``Jr.``). Only a NEVER-GIVEN particle attaches without a comma; an ambiguous one keeps whatever reading its position gives it, which is what leaves the unaccented Vietnamese listing alone -- ``"Nguyen Thi Van"`` under ``FAMILY_FIRST_GIVEN_LAST`` still reads given ``Van``, and that form is the one format the parser reads correctly. The cost of that line is the Dutch flagship word: ``van`` is ambiguous vocabulary, so ``"Beethoven Ludwig van"`` keeps middle ``van`` where ``"Beethoven, Ludwig van"`` gives family ``van Beethoven``. Nothing moves under the DEFAULT name order. Measured, 12 of 6600 parses move -- this release's 1100-name corpus under three ``name_order`` values with ``middle_as_family`` off and on -- over four names, two of which are names this fix's own rules.md examples added to the corpus. Those counts cannot come from the gate: ``tools/differential/compare.py`` parses every corpus name with the default policy and sweeps no policy at all, so no non-default ``name_order`` behavior has ever been compared across versions, and the gate output here is unchanged apart from the corpus names the new examples add, none of which diffs at any baseline. What moves is recomputed by the recipe in the ``P6`` entry of ``docs/design/decisions.md``, which compares the seven role fields against a checkout of the parent commit reading the same corpus files; the count is a dated snapshot, measured 2026-08-30, since nothing in the repository holds that tree. The ``rules.md#P6`` example lines and ``tests/v2/pipeline/test_post_rules.py`` are what pin the behavior (closes #365) - Fix ``initials()`` reading a name in a different order than the fields of the same name. Two rules fold words into the family name and render them before the rest of it -- ``Policy(middle_as_family=True)``, which sends every middle word to the family, and the tussenvoegsel attachment after a family comma -- and both do it by marking the words rather than moving them, since a parsed word keeps the position it was written at. The ``family`` field reads that mark and ``initials()`` did not, so one parse gave two orders: ``parse("der, y van")`` gave family ``van der`` and initials ``y. d. v.``, and now gives ``y. v. d.``. This RESTORES v1: ``middle_name_as_last`` is v1's spelling of the same option, so most of what moves has a 1.4.0 answer to be measured against, and measured over the 1094-name differential corpus at the default name order, 71 names move under that option, 54 of them back to exactly what 1.4.0 returns and not one of them away from it -- ``"Doe, Dr. John A."`` gives ``J. A. D.`` again where 2.0 through 2.2 gave ``J. D. A.``, and ``"Brundridge, Contessa A"`` gives ``C. A. B.`` where they gave ``C. B. A.``. Of the 17 that match 1.4.0 neither before nor after, 14 now agree with it on the ORDER and differ only in how v1 grouped initials -- 1.4.0 gives one initial per element of its own ``last_list``, so a conjunction-joined surname yields ``V G.`` where the 2.x view, one initial per word, yields ``V. G.``; one more is ``"der, y van"``, whose family is nothing but particles, where 1.4.0 contributes no initial at all and 2.x contributes its words, a difference this release does not touch; and the remaining two parse differently from 1.4.0 for reasons that predate this fix. Without the option, one corpus name moves, the ``"der, y van"`` above -- and it is one of one, since it is the only name in the corpus whose family holds two contributing words with a folded one behind the other, the shape an order change can be seen in at all. ``HumanName.initials()`` was already right and is unchanged, measured: no name of the 1094-name corpus moves through the facade, with the option or without it. It reads ``first_list``/``middle_list``/``last_list``, which prepend the folded words as v1 did, so it is the 2.0 API's ``ParsedName.initials()`` that was out of step -- with the field beside it, with the facade, and with 1.4.0 at once. ``initials()`` is not one of the seven role fields the differential harness compares, so no gate run can see this change: run at all three baselines before and after, the output is identical to the byte -- 1094 corpus names, 229 / 194 / 102 intentional diffs and ``unexplained: 0`` at 1.4.0 / 2.0.0 / 2.1.0, with every per-heading count unchanged. So where the counts in this bullet come from has to be said, the gate's classified summary not being able to supply them and the ``rules.md#R3`` example line witnessing the order without counting anything: what MOVES is recomputed by the recipe in the ``R3`` entry of ``docs/design/decisions.md``, which compares this view against the pre-change rendering over these same four corpora and reproduces the 71 and the one-of-one; the 1.4.0 comparisons and the facade sweep are dated snapshots rather than re-derivable ones, measured 2026-08-30 against the released 1.4.0 wheel and against the pre-change tree, which nothing in the repository re-runs. The ``rules.md#R3`` example line and ``tests/v2/test_render.py`` are what pin the behavior (closes #408) - Fix a tussenvoegsel attached to the family name after a comma deciding a genuinely uncertain reading and reporting nothing. ``"Van Johnson"`` reports a ``PARTICLE_OR_GIVEN`` ambiguity -- ``Van`` is a Dutch particle and a Vietnamese given name, and the parser has to pick one -- while ``"Nguyen, Thi Van"`` picked the same word the same way, silently, and lost the given name doing it. The attachment now reports the fork it decides, in the kind that names the reading it declined. A particle that could be an ordinary name reports ``PARTICLE_OR_GIVEN``: ``"Nguyen, Thi Van"``, ``"Berg, Jan van der"`` and ``"Vega, Juan de la"`` each gain one, the ``detail`` naming the ambiguous word. A particle the parser had already read as a post-nominal reports ``SUFFIX_OR_NAME`` instead, because the credential reading is what the attachment overrode: ``"Berg, Jan vd"`` gains one, ``vd`` being read as *van der* rather than as the Volunteer Decoration. Which kind you get follows the reading that was overridden rather than the word's vocabulary, so ``"Berg, Jan do"`` reports ``PARTICLE_OR_GIVEN`` even though ``do`` is a postnominal too -- it was already being read as a name word, so no credential reading was overridden. A particle where nothing was overridden reports nothing at all: ``"Jong, Piet de"`` and ``"Jong, Anke de"`` are unchanged, ``de`` being no name in any reading and no postnominal either, and so is ``"Berg, Jan de vd"``, whose run was read as name words whole. Worth knowing before you filter on this: ``"Beethoven, Ludwig van"`` -- the textbook Dutch listing, read exactly right -- now carries a report too. It is the same string shape over the same vocabulary as ``"Nguyen, Thi Van"``, and nothing in the input separates them, so a report on one is a report on both. This adds the report and nothing else: every field these names parse to is exactly what the tussenvoegsel fix below already gave them, and ``ambiguities`` is the only value that grows. Seven differential corpus names gain a kind against the 2.0.0 and 2.1.0 baselines, seven of the nine the tussenvoegsel fix below already moved -- five of ``fix(#379)``'s seven and both of ``fix(#380)``'s two, now carrying ``_ambiguities`` in their diff as well; the two left out are the pair named unchanged above, ``Jong, Piet de`` and ``Jong, Anke de`` -- and none against 1.4.0, which had no ambiguity reporting at all (closes #405) diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 1869dae8..767008af 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -2,15 +2,18 @@ Consumes: tokens (roles assigned), plus pieces and structure -- the particle fold reads the opening piece of segment 0, or of segment 1 -under a family comma (#359). structure was always read here, for the -rotation gate. +under a family comma (#359), and P6's no-comma site reads the +TRAILING pieces of segment 0 (#365). structure was always read here, +for the rotation gate. Produces: tokens with roles adjusted by the post rules, plus the ambiguity P6's attachment reports for the fork it decides (#405). Reads: Policy.patronymic_rules, Policy.middle_as_family; Lexicon.given_name_titles. -Implements rules H1, M4, P1, O1, O2 and O3 of docs/design/rules.md; -each is cited at its code below, and H1/P1/O1/O2's history lives in +Implements rules H1, M4, P1, P6, O1, O2, O3 and R2 of +docs/design/rules.md; each is cited at its code below. P6 is here at +TWO sites, one on each side of P1 -- see the comment at the first for +why neither can move to the other. H1/P1/O1/O2/P6's history lives in docs/design/decisions.md. """ from __future__ import annotations @@ -171,11 +174,27 @@ def _fold_reach(tokens: list[WorkToken], name_idx: list[int]) -> int: return i + len(_units(tokens, name_idx[i:])[0]) +def _has_base(piece: tuple[int, ...], tokens: list[WorkToken]) -> bool: + """Whether a PIECE holds a word no particle vocabulary claims -- + the base a particle can attach to. Callers pass a piece that + already holds a name role, which is why the role is not re-asked + here. No citation: R2 states "a particle needs a base to attach + to" in an Accepted clause, and the excerpt test can only verify + the part of a rule ahead of its first example.""" + return any("particle" not in tokens[i].tags for i in piece) + + +def _is_never_given_particle(token: WorkToken) -> bool: + """Particle vocabulary that no culture uses as a given name -- the + discriminator P1 and P6's no-comma site both key on, read off the + tags classify recorded rather than re-asked of the lexicon.""" + return ("particle" in token.tags + and "vocab:particle-ambiguous" not in token.tags) + + def _is_lone_never_given_particle(site: tuple[int, ...], tokens: list[WorkToken]) -> bool: - return (len(site) == 1 - and "particle" in tokens[site[0]].tags - and "vocab:particle-ambiguous" not in tokens[site[0]].tags) + return len(site) == 1 and _is_never_given_particle(tokens[site[0]]) def post_rules(state: ParseState) -> ParseState: @@ -245,6 +264,122 @@ def post_rules(state: ParseState) -> ParseState: middles = _idx(tokens, Role.MIDDLE) families = _idx(tokens, Role.FAMILY) + # rules.md#P6: "a particle ending the name attaches to that + # family name and is written before it" -- P6's other + # precondition. A family comma names the family; so does a + # declared family-first order, and the same Dutch listing is + # written both ways: "Jong, Anke de" and "Jong Anke de" under + # FAMILY_FIRST both give family "de Jong" (#365). + # + # NEVER-GIVEN vocabulary only, where the comma path takes + # ambiguous particles too. P6 rests that on the comma leaving "no + # signal that separates the two readings"; without one the + # positional read is the signal, and P1 says what it decides -- + # "an ambiguous particle keeps whatever reading its position + # gives it". That is what leaves the Vietnamese listing alone: + # "Nguyen Thi Van" under family-first-given-last keeps given + # "Van", P6's own example, where "Nguyen, Thi Van" loses it. The + # cost is the Dutch flagship word: `van` is ambiguous vocabulary, + # so "Beethoven Ludwig van" keeps middle "van" where the comma + # form gives family "van Beethoven". The two forms agree for the + # never-given half of the tussenvoegsel set and not for the rest. + # + # The suffix override is the comma path's alone for the same + # reason -- a trailing `vd` or `mc` is a post-nominal until a + # comma makes the tussenvoegsel commoner -- so the run is found + # among the pieces that hold a NAME role, and a piece holding + # none is walked past ("Berg Jan de Jr."). Not quite as the comma + # path walks: its loop refuses to step over an all-particle piece, + # which is how a suffix-roled `vd` stays part of the run there. + # + # THREE THINGS MUST REMAIN, and each was a defect in review + # before it was a condition (decisions.md#P6, 2026-08-30): + # a name word BESIDES the family, which is P6's own words-to-spare + # test in the shape a positional read needs -- without it + # "Jong de" attaches where "Jong, de" declines, and the two + # writings of one listing disagree about word order; a BASE in the + # piece that will be the family, since a family that is all + # particles is not a family written beside anything -- without it + # "van Berg Jan de" loses its given name outright, the leftover + # collapsing to a single unit; and no SUFFIX WORD stranded at the + # end, since assign left one in a name position only because the + # particle followed it, and the re-layout does not re-run assign's + # trailing peel -- without it "Berg Jan Jr. de" reports given + # "Jr." under FAMILY_FIRST_GIVEN_LAST. + # + # Placed BEFORE P1, which is why P6 lands at two code sites. P1's + # given-position site takes the same run under a family-first + # order and takes it with the other reading -- the whole given + # slot into the family, in written order, leaving no given name + # ("Mesnil Garcia de" -> family "Mesnil Garcia de"). Attaching + # first leaves it the runs this site declines, which is what + # "Mesnil de" and "van der" still reach it for. The COMMA path + # cannot move up here with it: it reads the roles P1 settles, for + # the reason recorded at that site below. + # + # No structure test. A real family comma records no order, so + # `state.order is not None` excludes it already; testing the + # structure as well excludes only the comma that fixed NOTHING -- + # segment 1 holding no name word, assign reading segment 0 + # positionally (#296) -- where #365's symptom survives verbatim + # ("Mesnil Garcia de, Dr."). Segment 0 is this site's for the same + # reason, which is _leading_name_piece's rule read at `order is + # not None`. + # + # What is left is a shorter name of the same order, and unlike + # P1's fold the family is NOT among the pieces already placed, so + # the remaining units take the positions _name_positions gives + # them outright. At TWO of them that is family-then-given in + # either family-first order, which is why the orders agree on + # #365's shape; at three they differ again, and the declared + # order is what decides given from middle. + if (state.order is not None + and state.order[0] is Role.FAMILY + and state.pieces): + seg = state.pieces[0] + end = len(seg) + while end and not any(tokens[i].role in _NAME_ROLES + for i in seg[end - 1]): + end -= 1 + k = end + while k and all(_is_never_given_particle(tokens[i]) + for i in seg[k - 1]): + k -= 1 + # The words-to-spare test is a name word ahead of the run, + # and it must be one no particle vocabulary claims: what P6 + # attaches to is "the family written beside it", and a name + # that is nothing but particles has no family written beside + # anything. Without that second half the rule reorders inside + # an all-particle name -- "van der" under a family-first + # order attaching `der` to `van` and rendering "der van", + # against words R2 reads as ordinary name words. Reordering + # ordinary name words is not a thing any rule does. + # P1's accepted "de" -> given="de" is the same test at one + # piece. + kept_pieces = [piece for piece in seg[:k] + if any(tokens[i].role in _NAME_ROLES for i in piece)] + if (end > k and len(kept_pieces) > 1 + and _has_base(kept_pieces[0], tokens) + and not all("vocab:suffix" in tokens[i].tags + for i in kept_pieces[-1])): + for piece in seg[k:end]: + for i in piece: + tokens[i] = dataclasses.replace( + tokens[i], role=Role.FAMILY, + tags=tokens[i].tags | {FOLDED_TAG}) + kept = [i for piece in seg[:k] for i in piece + if tokens[i].role in _NAME_ROLES] + units = _units(tokens, kept) + for unit, role in zip(units, _name_positions( + state.order, len(units))): + for i in unit: + _retag(tokens, i, role) + # recomputed for H1's reason, stated at H1: a stale index + # list is the bug shape #359 fixed + givens = _idx(tokens, Role.GIVEN) + middles = _idx(tokens, Role.MIDDLE) + families = _idx(tokens, Role.FAMILY) + # rules.md#P1: "a never-given particle standing alone where the # given name would go — or opening the name — marks the name as # surname-only: the particle run and the name words it attaches @@ -281,6 +416,14 @@ def post_rules(state: ParseState) -> ParseState: # entry can put the family first under a given-first # policy, and the roles below have to match the read # assign actually made. + # Minus what P6's site above already attached: the fold + # lays out what is LEFT of the name, and a word another + # rule has placed is not left. Without this the + # redistribution hands the attached particle back to the + # given slot it was taken out of ("de la Vega de" under a + # family-first order). FOLDED_TAG is that site's alone + # here -- O3's fold and P6's comma path both run later in + # this stage. name_idx = sorted(givens + middles + families) cut = _fold_reach(tokens, name_idx) for i in name_idx[:cut]: diff --git a/tests/v2/pipeline/test_post_rules.py b/tests/v2/pipeline/test_post_rules.py index 6f835a99..645d5210 100644 --- a/tests/v2/pipeline/test_post_rules.py +++ b/tests/v2/pipeline/test_post_rules.py @@ -9,7 +9,8 @@ from nameparser._policy import (FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST, GIVEN_FIRST, PatronymicRule, Policy, Script) -from nameparser._types import STABLE_TAGS, AmbiguityKind, Role +from nameparser._types import (FOLDED_TAG, STABLE_TAGS, AmbiguityKind, + Role) # A reduced lexicon, the convention in every pipeline stage module: a # stage test should not move when shipped vocabulary does. What it must @@ -310,14 +311,227 @@ def test_lone_never_given_particle_in_given_position_folds( policy: Policy) -> None: # The opening-position test alone does not carry the rule: under a # family-first order the given position is the TRAILING piece, and - # a lone 'de' landing there has to fold into the family beside it - # or the parse leaves the whole given name as a word the vocabulary - # says is never a given name. Guarded here because a refactor that - # reads the rule as leading-particle-only drops exactly this shape, + # a lone 'de' landing there has to join the family beside it or the + # parse leaves the whole given name as a word the vocabulary says + # is never a given name. Guarded here because a refactor that reads + # the rule as leading-particle-only drops exactly this shape, # silently and under a non-default order (#359 review). + # + # Still P1's fold, and the negative assertion is what says so: + # P6's no-comma site (#365) declines here because nothing would + # remain beside the family, so the run is folded in WRITTEN order + # rather than attached and rendered before the base. `_by_role` + # reads written order and cannot tell those two apart on its own. out = _parsed("Mesnil de", policy) assert _by_role(out, Role.FAMILY) == "Mesnil de" assert not _by_role(out, Role.GIVEN) + assert not _folded(out) + + +def _folded(state: ParseState) -> str: + return " ".join(t.text for t in state.tokens if FOLDED_TAG in t.tags) + + +@pytest.mark.parametrize("policy", _FAMILY_FIRST) +def test_trailing_particle_attaches_under_both_family_first_orders( + policy: Policy) -> None: + # #365: the two orders permute which role the trailing particle + # falls into -- GIVEN under one, MIDDLE under the other -- and only + # one of those was inspected, so `FAMILY_FIRST` stranded 'de' as a + # standalone middle name while `FAMILY_FIRST_GIVEN_LAST` folded the + # whole given slot away. Both now attach the run and lay the rest + # out for its own number, which is what makes them agree. + out = _parsed("Mesnil Garcia de", policy) + assert _by_role(out, Role.FAMILY) == "Mesnil de" + assert _by_role(out, Role.GIVEN) == "Garcia" + assert not _by_role(out, Role.MIDDLE) + assert _folded(out) == "de" + + +@pytest.mark.parametrize("policy", _FAMILY_FIRST) +def test_trailing_particle_needs_a_base_to_attach_to( + policy: Policy) -> None: + # The words-to-spare test is a name word no particle vocabulary + # claims, not merely a name PIECE. With the wider reading this name + # attaches 'der' to 'van' and renders 'der van' -- reordering two + # words R2 reads as ordinary name words, which no rule does. + out = _parsed("van der", policy) + assert _by_role(out, Role.FAMILY) == "van der" + assert not _folded(out) + + +@pytest.mark.parametrize("policy", _FAMILY_FIRST) +def test_trailing_ambiguous_particle_keeps_its_position( + policy: Policy) -> None: + # Never-given vocabulary only, where the comma path takes ambiguous + # particles too. This is the Vietnamese listing P6 records as the + # ONE format that reads correctly: taking 'van' here would give + # family 'van Nguyen' and take that format away. + out = _parsed("Nguyen Thi van", policy) + assert _by_role(out, Role.FAMILY) == "Nguyen" + assert not _folded(out) + + +@pytest.mark.parametrize("policy", _FAMILY_FIRST) +def test_trailing_suffix_particle_keeps_the_post_nominal_reading( + policy: Policy) -> None: + # The S2 precedence is the comma form's alone: a trailing `vd` is a + # post-nominal until a comma makes the tussenvoegsel commoner. The + # same input WITH the comma reads family 'vd Berg' (P6's example), + # so this pins the half of the rule that did not widen. + out = _parsed("Berg Jan vd", policy) + assert _by_role(out, Role.SUFFIX) == "vd" + assert _by_role(out, Role.FAMILY) == "Berg" + assert not _folded(out) + + +def test_p1_given_position_site_still_has_a_shape_to_fire_on() -> None: + # P6's site takes the TRAILING run, which is every corpus firing of + # P1's given-position site that the leading site does not also take + # (6 of 1094 names x 3 orders, measured 2026-08-30). The site is + # kept because it is still reachable, and this is the shape that + # reaches it: a suffix piece breaks the particle chain, leaving a + # lone 'de' in the given slot with a name word still after it. If + # this ever stops firing, P1's statement should lose the clause, + # not keep a site nothing can reach. + out = _parsed("Mesnil de Jr. Garcia", + Policy(name_order=FAMILY_FIRST)) + assert _by_role(out, Role.GIVEN) == "" + assert not _folded(out) + + +def test_the_two_particle_sites_cannot_both_fire() -> None: + # P1's lead site fires only on a leading LONE never-given + # particle, which is a family piece with no base -- exactly what + # P6's no-comma site requires there. So the two are mutually + # exclusive by construction (measured over 61,854 generated + # parses, 2026-08-30), and P1's redistribution needs no exclusion + # for tokens the attachment has already placed. + # + # Guarded because an earlier draft DID need one: with a looser + # base test the attachment fired here, and P1's lead branch then + # handed the particle straight back to the given slot it had been + # taken out of. The `given` this leaves is P1's own gap and not + # this rule's -- P1 says a never-given particle never stands as + # the given name, and its fold produces one here -- so it is + # asserted as it is rather than as it should be. + out = _parsed("de la Vega de", Policy(name_order=FAMILY_FIRST)) + assert _by_role(out, Role.FAMILY) == "de la Vega" + assert _by_role(out, Role.GIVEN) == "de" + assert not _folded(out) + + +@pytest.mark.parametrize("policy", _FAMILY_FIRST) +def test_the_run_needs_a_name_word_besides_the_family( + policy: Policy) -> None: + # P6's words-to-spare test, in the shape a positional read needs + # it. The comma form declines "Jong, de" for the same reason, so + # without this the two writings of a one-given-word listing + # disagree about word order -- 'Jong de' with the comma and + # 'de Jong' without it. + out = _parsed("Mesnil de", policy) + assert not _folded(out) + + +def test_the_family_the_run_attaches_to_must_have_a_base() -> None: + # A name word must remain AND the family must be the piece that + # holds one. Testing only "a name word remains somewhere ahead" + # passes here and loses the given name: the leftover is one UNIT, + # the leading particle chaining over every word after it, so the + # whole name becomes the family and `Berg` stops being a given + # name at all. + # + # FAMILY_FIRST only, deliberately. Under FAMILY_FIRST_GIVEN_LAST + # the trailing particle lands in the given slot and P1's fold + # takes the whole name anyway, so that order has no given name to + # lose and cannot witness the defect. + out = _parsed("van Berg Jan de", Policy(name_order=FAMILY_FIRST)) + assert _by_role(out, Role.GIVEN) == "Berg" + assert not _folded(out) + + +@pytest.mark.parametrize("policy", _FAMILY_FIRST) +def test_a_title_does_not_supply_the_base(policy: Policy) -> None: + # The base test reads a NAME role as well as the vocabulary. On + # the vocabulary alone a title ahead of an all-particle family + # passes it, and 'van der' is reordered to 'der van' -- two words + # R2 reads as ordinary name words. + out = _parsed("Mr. van der", policy) + assert _by_role(out, Role.FAMILY) == "van der" + assert not _folded(out) + + +@pytest.mark.parametrize("policy", _FAMILY_FIRST) +def test_the_run_is_found_past_a_trailing_post_nominal( + policy: Policy) -> None: + # The walk over trailing pieces that hold no name, in the + # direction that FIRES. Pinned separately from the `vd` case, + # which pins it only where the rule declines: a site that refused + # to fire once the walk had moved would pass that one and restore + # both of #365's symptoms here. + out = _parsed("Berg Jan de Jr.", policy) + assert _by_role(out, Role.GIVEN) == "Jan" + assert _by_role(out, Role.SUFFIX) == "Jr." + assert _folded(out) == "de" + + +@pytest.mark.parametrize("policy", _FAMILY_FIRST) +def test_the_run_does_not_strand_a_suffix_word_in_a_name_role( + policy: Policy) -> None: + # assign left 'Jr.' in a name position only because the particle + # followed it. Take the particle away and the leftover is not the + # same name one word shorter, so the rule declines rather than + # promoting a post-nominal into a name: the re-layout does not + # re-run assign's trailing peel, and under + # FAMILY_FIRST_GIVEN_LAST 'Jr.' would take the given slot. + out = _parsed("Berg Jan Jr. de", policy) + assert not _folded(out) + assert _by_role(out, Role.GIVEN) != "Jr." + + +def test_a_comma_that_fixed_nothing_still_reaches_the_site() -> None: + # structure is FAMILY_COMMA but the comma named no family -- + # segment 1 holds no name word, so assign read segment 0 + # positionally and recorded the order (#296). A guard on the + # STRUCTURE excludes exactly this shape and nothing else, since + # `state.order` is None wherever a comma really did name the + # family. #365's symptom survived here in the first draft. + out = _parsed("Mesnil Garcia de, Dr.", + Policy(name_order=FAMILY_FIRST)) + assert _by_role(out, Role.GIVEN) == "Garcia" + assert not _by_role(out, Role.MIDDLE) + assert _folded(out) == "de" + + +def test_the_leftover_keeps_the_units_other_rules_built() -> None: + # The leftover is partitioned into UNITS, not words and not + # pieces: a conjunction join and a bound given-name pair each + # count as one name word wherever another rule counts them (P3, + # P5). Per-word the conjunction becomes the given name; by piece + # the bound pair splits, P5's join never having built one (it + # joins only where the bound word is the first non-title piece). + ff = Policy(name_order=FAMILY_FIRST) + joined = _parsed("Garcia y Santos Juan de", ff) + assert _by_role(joined, Role.GIVEN) == "Juan" + bound = _parsed("Rahman abdul Juan de", ff) + assert _by_role(bound, Role.GIVEN) == "abdul Juan" + + +def test_three_leftover_pieces_still_read_by_the_declared_order() -> None: + # The two family-first orders agree on #365's shape because + # _name_positions gives family-then-given at TWO leftover pieces + # in both. At three they diverge again, which is the declared + # order doing its job -- and this is the only fixture where the + # order argument to the re-layout is observable at all. + ff = _parsed("Mesnil Garcia Carlos de", + Policy(name_order=FAMILY_FIRST)) + gl = _parsed("Mesnil Garcia Carlos de", + Policy(name_order=FAMILY_FIRST_GIVEN_LAST)) + assert _by_role(ff, Role.GIVEN) == "Garcia" + assert _by_role(ff, Role.MIDDLE) == "Carlos" + assert _by_role(gl, Role.GIVEN) == "Carlos" + assert _by_role(gl, Role.MIDDLE) == "Garcia" + assert _folded(ff) == _folded(gl) == "de" def test_lone_never_given_particle_needs_no_repair_by_default() -> None: diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index ebad1264..d7849f3e 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -20,7 +20,10 @@ "Anna z (domu) Nowak" "Anna z Nowak" "Asst. Vice Chancellor John Smith" +"Beethoven Ludwig van" "Beethoven, Ludwig van" +"Berg Jan de Jr." +"Berg Jan vd" "Berg, Jan van der" "Berg, Jan vd" "Berg, abd née Jones" @@ -119,6 +122,8 @@ "Marquess of Bath" "Mary Beth Smith" "Mc Donald" +"Mesnil Garcia Carlos de" +"Mesnil Garcia de" "Mesnil de" "Morse, Det. Insp. Jane" "Mr. Jack Jill" @@ -205,7 +210,9 @@ "juan mcdonald" "mohamad ali smith" "née Jones" +"van Berg Jan de" "van Gogh" +"van der" "van der Berg, abdul née Jones" "Иван Петрович Абрамович" "Сидоров Иван Петрович"