diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 5729111c..8776d743 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -716,6 +716,8 @@ Declined: "Juan van der" stays "J." (no borne name, no base, and initials of a bare particle run would be nonsense). The general lesson, worth more than this instance: a deviates: marker gets written on the rule whose STATEMENT changed, but a rule can change another rule's OUTPUT without touching its statement, and nothing looks for that — the runner asserts per example line, so an unmarked downstream rule stays green precisely because its own examples avoid the affected input. When adding a marker, walk the changed rule's `interacts:` targets and ask whether any of THEIR examples move. +- 2026-08-29 #461 — TRIED AND BACKED OUT; the question is back on the issue. The unjoined mark readmits the words of an all-particle part to `initials()`, and #461 narrowed that readmission to the `particle` tag alone, on the authority of R3's "A CONJUNCTION never initials, so a base that is one contributes nothing even then". It was reverted before merge, in the same PR. What the narrowing got wrong is best said in terms of the MARK rather than of the fields. The mark is a statement about a whole PART — none of its words is doing the work its tag names — and #461 honored that statement for some of the part's words while keeping one of them out. Under `Lexicon.default().add(particles={'y'})`, `parse("Juan de y")` is such a part: `family_base` is `de y`, both words carry the mark, and #461 initialed `J. d.`, admitting the `de` as the name word the mark makes it and refusing the `y`. R2's reasoning does not split that way. A particle with nothing left to join is not acting as a particle, which is the whole reason the mark exists; a conjunction inside that same part has nothing left to join either, for the same reason and at the same moment, so it is a name word of the part and initials with the rest. Scope that carefully, because its general form is far wider than the claim and would be wrong: this is about a part the mark has ALREADY turned into name words, not about any conjunction that happens to join nothing. Outside such a part the skip stands whatever the conjunction is or is not joining, and none of those moved or may move — `parse("Juan y Garcia")` initials `J. G.` with the `y` a middle name word by P3, `parse("Juan Velasquez y Garcia")` initials `J. V. G.` over base `Velasquez y Garcia`, `parse("Jon Dough and")` initials `J. D.` over base `Dough and`. Base and initials differ in every one of those, legitimately, and R3's own `"Juan de y" → initials="J."` line is a fourth, its `family_base` being `y` under the default vocabulary. So "the base holds a word the initials do not" is NOT the criterion and was not the finding; the mark is. What is therefore in question is R3's "even then" clause, which carries the carve-out into the one part where the joining has stopped — not the code that failed to implement it. The clause is left standing and #461 now asks whether it belongs there; nothing else in R3 moved, and the paragraph #461 added under it is gone. R4's conjunction sentence rests on that clause by name ("the carve-out R3 states for initials") and is untouched here, so whoever settles #461 settles R4's cross-reference with it. Note what the backout cannot record: a `deviates:` marker hangs on an example LINE and this shape has none to hang on — `particles` and `conjunctions` are disjoint in the default vocabulary and in every locale pack, so no input string the doc runner parses reaches it. rules.md itself therefore carries no trace, and this bullet with mechanisms.md#RENDER-HONORS-THE-PARSE is where the gap is written down. It is not the only gap under R3's conjunction sentence, and the other is older, wider and markable: "A CONJUNCTION never initials" is unqualified, while a conjunction in the GIVEN group has always initialed — `parse("John and Jane Smith")` gives `J. a. J. S.` and `parse("Duke of Edinburgh")` `D. o. E.`, 25 of the 1094 corpus names, all reachable from the default vocabulary. Recorded here because it was found here; it is not #461's to fix. RESOLVED into the normative document on 2026-08-29, in the review of PR #463: rules.md#R3 now scopes its carve-out to the middle and base family words and declares the given group unsettled, in prose and WITHOUT a `deviates:` marker. The marker was considered and rejected on its own definition — it states the INTENDED value, and there is none to state. R3 counts name words while P3 makes a connective and its neighbours ONE name word, so `John and Jane Smith` has four candidate answers and no entry anywhere picks one: `J. a. J. S.` today, `J. J. S.` reading the carve-out as written, `J. S.` reading P3's join as a single name word, and `J a J. S.` at 1.4.0 (measured on the released wheel, which joins the run rather than skipping the conjunction). Marking it would put an invented value in a normative document and hold the parser to it, since the runner asserts the today-value strictly. The marker's `#N` slot has no owner either — the sentence above disclaims #461 for this gap, and filing its own issue is a maintainer's call rather than a docs commit's. What "markable" meant two sentences up is the MECHANICAL property that an input string exists to hang a marker on, unlike the #461 shape where none does; it was never a claim that the intended value is known. Neither direction is visible to the gate: 0 of the 1094 corpus names move either way, and the counts hold at 229 / 194 / 102 with 0 unexplained at 1.4.0 / 2.0.0 / 2.1.0. Both views of the contested token are pinned rather than left to prose — `test_repair_keeps_a_conjunction_lowercase_in_a_particle_part` holds R4's ungated conjunct, gating which had passed the entire suite until #461's test existed, and `test_initials_readmits_a_conjunction_in_a_particle_part` holds today's initials answer the way a `deviates:` marker would, so re-deciding #461 fails the suite until that pin moves with it. + Declined: - 2026-08-18 — the GROUPING half of #404: a particle run that joins nothing does not chain, so "Jong van der" would split into middle 'van' plus family 'der'. Measured and rejected, though NOT for the reason first recorded here. The first draft said the split makes the family "no longer all-particle so the base fix stops firing" — false, and `der` and `la` are both shipped particles, so a family of either IS all-particle and the rule fires on it ("Juan Smith der" gives base 'der'). What the split actually costs is the SCOPE of the base and a stray particle relocated: grouping can decline to merge but cannot keep the words apart, because roles re-assemble them and two adjacent same-role pieces are one part at the field level. "Juan Smith van der" becomes middle 'Smith van', family 'der' — a base of 'der' rather than 'van der', and a middle name nobody wrote. Keeping the run whole in one part is what gives the base its full extent. The split reading needs the leftover distribution to know these are separate units, which is mechanisms.md#UNIT-PARTITION's problem. @@ -733,6 +735,12 @@ Declined: - 2026-08-29 — the REPLACE/REVISE BOUNDARY, raised by the review of this branch as a gap needing a systematic fix and corrected by Derek to what it actually is: the documented boundary, with a supported path across it. `_cap_word` keys the particle TEST on lexicon membership but the REPAIR on the unjoined mark, and the mark is a parse product — so `parse('de la').capitalized(force=True)` is `'De La'` while `base.replace(family='de la').capitalized(force=True)` is `'de la'`, `replace()` splicing raw text into a field without classifying it. `Parser.revise()` is the method that classifies it, and its docstring already promised exactly this: each value gets a full sub-parse "so the stable tags survive and the tag-driven views ... behave as if the text had been parsed". Measured: `Parser().revise(base, family='de la').capitalized(force=True)` is `'De La'`. So there is nothing to fix — rules.md#R4 carries it as an Accepted boundary naming `revise()`, not as a deviation, and `revise()`'s enumeration of the tag-driven views gains `capitalized()`, which #407 made the fourth. (#458 would additionally make `replace()` agree, by keying the test on the `particle` tag rather than re-deriving membership from the word — the same shape it already names for `conjunction`/`initial`. That is a convenience, not the resolution.) It is a limit and not a regression either way: 2.1.0 gives `'de la'` on the spliced path too, so only the parsed path moved. +- 2026-08-29 #458 — DONE: case repair's conjunction test reads the `conjunction` TAG instead of re-deriving the word's class at render time, so a view honors the decision the parse recorded (mechanisms.md#VOCAB-TAGS) rather than taking it again. The two copies of the question were not the same question, which is the reason this was worth doing over a corpus that cannot see it: `_classify.py` asks `is_initial(token.text)` — the shape test ANDed with a repertoire test since #320 — while `_render.py` asked a bare pattern, shape only, over each WORD of a token's text rather than the token. It does NOT retire the hand-sync obligation the module carried in a comment ("keep in sync with `nameparser/_pipeline/_vocab.py` by hand"), and an earlier draft of this bullet said it did: the tag read has nothing to keep in sync, but `_render._INITIAL` was deleted and then restored in the same PR, the fallback in the bullet below giving it a reader again. The obligation is narrower and sharper than before — the two copies no longer answer the same question about the same token, so what has to hold is that the fallback answers as `_vocab` would — and test_regex_sync is what holds it. MEASURED, re-derived on this branch rather than taken from the issue: 0 of the 1094 corpus names move under `capitalized()` or `capitalized(force=True)`, and 0 move with each name also re-run uppercased and lowercased (6564 name/variant/lexicon rows = 1094 x 3 spellings x 2 lexicons, the second lexicon being `Lexicon.default().add(particles={'y'})`, and 13128 `capitalized` calls over them); the R5 force-versus-case counts recorded above hold — 62 upper / 16 lower through the facade, 63 upper / 18 lower through the core. Movement EXISTS outside that population, and the issue's "no output moves" is a claim about the corpus only: the old predicate re-decided per word, so `juan e-f smith` capitalized to `Juan e-F Smith` — the Italian conjunction inside a hyphenated middle name — and now gives `Juan E-F Smith`, agreeing with the uppercase spelling it always gave. An earlier draft of this change moved untagged tokens the same way — a family `replace()` splices in as `de y` repairing to `de Y` — and that was a REGRESSION, caught in review before it shipped and fixed in the bullet below; read this bullet with that one. NOT DONE, and the parenthetical in the boundary bullet above should be read as still open: the PARTICLE conjunct still keys on lexicon membership, so `replace()` does not agree there. Making it read the `particle` tag is a second change with its own blast radius — it moves a boundary rules.md#R4 states in prose — and #458 as filed does not ask for it. + +- 2026-08-29 (#458 review) — THE FALLBACK FOR TEXT THE PARSE NEVER SAW, and why it is scoped to one of the two questions `_cap_word` asks. Reading the tag and nothing else regressed every ASSIGNED field holding a conjunction, because a value `replace()` splices in is never classified and so carries no tag to read. Measured on the released 1.4.0 and 2.1.0 wheels, all agreeing and all broken by the tag-only reading: `h.last = 'velasquez y garcia'` then `capitalize(force=True)` gives `John Velasquez y Garcia`, `h.middle = 'e.'` gives `John E. Smith`, `h.last = 'smith-y'` gives `John Smith-y`, `h.first = 'y'` gives `y Smith`. That is the canonical Spanish surname and the shape `y` is in the vocabulary FOR, so an Accepted clause was not available: it had to be fixed. RULE: a token carrying `UNCLASSIFIED_TAG` was never read, so there is no decision to honor and the view falls back to the vocabulary. Note what the discriminator is NOT, in two directions, the second of which took a second review round to find. It is not UNTAGGEDNESS: an ordinary parsed name word carries no tags either (`velasquez` parses to `tags=[]`), so that would have swept in the whole parsed corpus. And it is not the absent SPAN, which is what the first fix used. Span-less means SYNTHETIC, a wider set than unclassified: `Parser.revise()` builds span-less tokens too, from a full sub-parse whose tags it keeps deliberately, and its docstring promises the tag-driven views "behave as if the text had been parsed" — so keying the fallback on the span overrode exactly the tags `revise()` exists to preserve. Measured on the branch that did it: `revise(name, middle='e-f').capitalized(force=True)` gave `e-F` where `parse('john e-f smith')` gave `E-F`, and under `Parser(lexicon=Lexicon())` a revised family `de la vega` initialed `j. v.` where the parse initialed `j. d. l. v.`. So the mark is stamped by the two producers of genuinely unclassified text — `ParsedName.replace()`, and the facade's v1 pickle load, which rebuilds a name from `*_list` strings and no tags — and a HAND-BUILT span-less token is not marked, so it takes the tag path like everything else in the library. That is the tag-driven semantics every other view already has, and it is a change from the span reading in exactly one place: a hand-built span-less token whose text is CONJUNCTION vocabulary now capitalizes rather than lowercasing (measured: a hand-built family `velasquez y garcia` repairs to `Velasquez Y Garcia` untagged, `Velasquez y Garcia` with either the `conjunction` tag or the mark). The PARTICLE conjunct is unaffected either way — it keys on lexicon membership rather than on the mark, which is the open item this bullet's neighbours record — so a hand-built `de la Vega` is unchanged. SCOPED TO THE PER-WORD QUESTION, and this is the durable half. *Is this word a conjunction rather than an initial* is a property of the word and its spelling: an unread token has no ambiguity the parse would have resolved differently, so the vocabulary gives the answer the parser would have given, v1's initial carve-out included — which is why `_render._INITIAL` came back and why test_regex_sync's two-copy assertion matters MORE now, the fallback being right only while it answers as `_vocab` would. *Is this particle acting as a particle here* is a property of the whole PART, and answering it needs a reading on every word of the part, which is exactly what a spliced field lacks. `_cap_word` is handed the whole token's tags — it is not blind to the part because it walks one word, and an earlier draft of this bullet said it was — but with no tag on any word of the part there is nothing to re-derive the part answer FROM, so the fallback here is the word question alone and the particle half falls through to plain particle treatment. Measured cost of the wide reading IN CASE REPAIR, so nobody re-derives it: `replace(family='de la')` gives `de la` today and a parse gives `De La`; widening case repair's fallback would make them agree and would thereby REVERSE the replace/revise boundary recorded two bullets above, with `revise()` as its documented crossing. Do not widen it without reopening the decision. Verified after the fix: `replace(family='velasquez y garcia').capitalized(force=True)` is `Velasquez y Garcia`, `replace(family='de la').capitalized(force=True)` is `de la`, and the parsed and `revise()` paths are unchanged. + +- 2026-08-29 (#458 review, then the review of PR #463) — the SAME FALLBACK in `initials()`: WRITTEN, MEASURED AND DROPPED before merge. The defect it aimed at is real and older than #458 — `replace(family='velasquez y garcia').initials()` gives `j. v. y. g.` where the parse, the facade and 1.4.0 all give `j. v. g.` (the facade and 1.4.0 measured through the TWIN of that splice — `h = HumanName('juan smith')` then `h.last = 'velasquez y garcia'`, the assigned construction this bullet is about, keeping `first`), and `replace(family='de la vega')` gives `j. d. l. v.` for `j. v.` — and it is unfixed again. What killed the fix is the parameter list: `initials()` takes no lexicon, so the fallback had to reach for `Lexicon.default()` and GUESS. `capitalized(lexicon=...)` is handed the caller's vocabulary and so guesses nothing, which is why its fallback stays. Measured cost of the guess, and the reason a merely-imperfect fallback was not good enough: under `Parser(lexicon=Lexicon.default().add(particles={'y'}))`, `parse('Juan de y').initials()` is `J. d. y.` and `parse('Juan Perez').replace(family='de y').initials()` was `J. d. y.` before the fallback and `J.` after it — an entire field's initials gone, because the default vocabulary reads `de y` as particles-plus-conjunction where the caller's reads it as an all-particle part. Guessing the wrong vocabulary for a name parsed under a custom one is worse than answering from tags alone, and `initials()` cannot be told better. The missing crossing is the whole of it: `Parser` carries `capitalized`, `matches` and `revise` as its custom-lexicon crossings and no `initials`, so there is no supported way to hand this view the parse's vocabulary. #464 is filed for that; when it lands, the fallback becomes answerable rather than a guess and this decision is worth reopening. ACCEPTED until then, and stated so a reader does not re-derive it as a bug: `replace(family='de la vega').initials()` is `j. d. l. v.` where the facade and a parse both give `j. v.`. The reach is the v2 core only — `HumanName.initials_list()` computes from the field STRINGS with its own lexicon (`_facade._initials_lists`) and gives `['j', 'v', 'g']` at 1.4.0, 2.1.0 and here, under that same assignment. That is why the facade cannot be used as evidence that the core is fine. Do NOT read `_initials_lists` as a v1 parity REFERENCE, though, which is how an earlier wording of this bullet labelled it: #462 is open on a parity break in that exact path, and it is a break inside the facade rather than upstream of it. `HumanName('Scott E. Werner')` partitions to `first='Scott'`, `middle='E.'`, `last='Werner'` at 1.4.0, at 2.1.0 and here — byte-identical fields — and `initials()` on them is `S. E. W.` at 1.4.0 and `S. W.` at 2.1.0 and here, the facade reading the middle `E` as the conjunction; assigning those three fields by hand reproduces each version's answer, so no parse difference is involved. The parity label was therefore doing work its own measurement cannot support, and only the per-name measurements are load-bearing here. MEASURE THIS BULLET WITH THE ASSIGNED CONSTRUCTION, and name the construction beside each value: the review of PR #463 measured `HumanName('juan velasquez y garcia')` — the PARSED form — against sentences about the assigned one, and 'corrected' two true claims into false ones. The two constructions agree from 2.0 on and DISAGREE at 1.4.0, where the parsed form gives `['j', 'v g']` and `j. v g.`. The `last` string is byte-identical either way; `initials_list()` reads `last_list`, which the 1.4.0 parse leaves as `['velasquez y garcia']` — the conjunction run joined into one element — where assignment re-splits it to `['velasquez', 'y', 'garcia']`. So `computes from the field STRINGS` is exact for 2.x and true of the ASSIGNED form only at 1.4.0. Two things learned in the attempt are worth keeping though the code is gone. `_types._remarked` is not the bug and needs no change: it recomputes the mark from TAGS after every edit, which is right for a record of what the parse decided and simply silent about text nobody classified. And any future fallback must be scoped per ROLE, not per name — a role the parse classified whole is decided by its tags however the other roles were built — a distinction that passes every other test in the suite and needs a hand-built name to witness. + - 2026-08-29 — WHY THE BOUNDARY WENT UNNOTICED UNTIL #407, which is where a future reader should look for it. For an ALL-PARTICLE part the other three tag-driven views give the same answer through `replace()` and `revise()` alike: measured over `de la`, `van der`, `do`, `de` and `van de la`, all five agree on `family_particles=''`, on a `family_base` holding the whole part, and on initials from every word. They converge because an UNTAGGED part and a MARKED all-particle part reach the same place by different routes — untagged, no word is recognized as a particle; marked, none is ACTING as one — and all three views only ask which words are particles. Case repair is the one view that asks a second question, since it must also decide whether to lowercase, so it is where the two routes first come apart. The mirror case confirms the reading: on a MIXED part the convergence is the other way round — `de la vega` and `van der berg` diverge in all three views between `replace()` and `revise()` (`replace()` reports particles `''` and base `'de la vega'` where `revise()` reports `'de la'` and `'vega'`) and AGREE on case repair, R4's all-particle clause not reaching them. So before #407 the distinction was invisible on exactly the shape the clause is about, and visible only on shapes the clause does not govern. ### R5 — the case-repair gate @@ -742,7 +750,7 @@ Declined: - 2026-08-29 — WHY R5'S STATEMENT WAS REWORDED RATHER THAN GIVEN A THIRD EXAMPLE, recorded at length because the wrong turn was taken twice and the reasoning is reusable. R5 first said a mixed-case name is returned untouched "whether or not its casing is correct". That phrasing reads as a disjunction with two branches, and invites the question of which example witnesses the already-correct one. Nothing can: for a mixed-case name that repair would not alter, kept and repaired are the SAME STRING by construction, so no example line distinguishes a parser with the gate from one without it. The unwitnessable branch is a property of the claim, not a gap in our choice of names — and the bullet above had already said so ("an already-correct input cannot witness a rule whose subject is what gets kept REGARDLESS of correctness") before two review rounds pushed past it. Two rows were tried and both withdrawn. `"Juan McDonald" → capitalized="Juan McDonald"` was INERT (mechanisms.md's inert-measurement class): it passes with R5's gate deleted, which is the whole failure shape that class names. `"Vincent Van Gogh" → capitalized="Vincent Van Gogh"` did discriminate — measured by comparing the shipped call against gate-off behavior, which the forced call reproduces exactly, since deleting the gate is what `force` already does: shipped `'Vincent Van Gogh'`, gate-off `'Vincent van Gogh'`, the particle rule lowercasing `Van` the moment repair runs — but it was REDUNDANT, not complementary. It asserts the same proposition as the `Shirley Maclaine` row above it: repair would change this name, and the gate keeps it anyway. The two differ only in which repair rule would have fired, and which rule repair applies is R4's subject, not R5's. And the frame it was chosen under was itself wrong, which is the more useful half of the lesson (reframed 2026-08-29 on Derek's correction, while R4 was being reworked). The row was picked to be bearer-correct — a name whose casing the bearer would endorse — and that frame drags the document into per-name arguments about whose spelling is right: this one needed `Vincent van Gogh` to be the correct spelling, which P6's own examples contradict, and the tussenvoegsel convention behind them is amended later on this branch anyway. CORRECTNESS DOES NOT ENTER INTO IT. Mixed case is the writer making an explicit choice, and repair defers to that choice rather than judging it — a name kept is not a name endorsed. Read that way the withdrawn row needed no claim about Dutch orthography at all, and the reason it needed one is exactly that the frame was wrong. The fix was to stop claiming the unwitnessable branch: the statement now says a mixed-case name is kept and that whether its casing is right does not enter into the decision, which `Shirley Maclaine` — casing wrong, kept — witnesses whole. GENERAL LESSON, and the reason this is long: when an example cannot be found for half a rule, suspect the STATEMENT before suspecting the example set. A phrasing that promises more branches than the behavior has will absorb inert examples indefinitely, each one looking like progress. Caution for the commit that reworks R4 and will choose its own mixed-case rows: avoid a name whose family base is wholly particle vocabulary (`Anh Van Do`, base `Van Do`), because the #407 work changes how those capitalize and would neuter such a row silently; `gogh` and `vega` are in neither `particles` nor `particles_ambiguous`, `van` and `do` are in both. - 2026-08-29 — the override has TWO routes, and the rule states neither, by design. Per-call is the obvious one. The second is a v1 Constants attribute the facade still honors, `force_mixed_case_capitalization` (nameparser/_facade.py resolves it when no per-call value is given; docs/release_log.rst records it as "still honored through the facade"). Measured today: with that attribute set True, `HumanName('Shirley Maclaine')` then a bare `.capitalize()` — no argument at all — LEAVES `'Shirley MacLaine'` (the v1 call mutates in place and returns None, so the name is read back with `str()`). An earlier draft of R5 said "only an explicit request to repair regardless overrides that", which that measurement falsifies; the statement now says repair was asked for anyway, without saying by what route (it read `repair regardless of how the name is cased was asked for` until 2026-08-29, when that phrasing turned out to have a second reading -- see the override bullet below), because rules.md is implementation-free by its own preamble. - 2026-08-29 — `capitalized_forced` is a test-side pseudo-field, not a parser field. The route the EXAMPLES use is the per-call argument to `capitalized()`, and an argument is not a policy, locale or extras gate — the only three things rules.md's grammar admits in an example's annotation slot — so it cannot ride that slot, and the doc runner grew a resolver branch instead. (The facade attribute above is a second route to the same behavior, not a second thing to assert; the core takes the argument only.) The R5 block's example lines are asserted by the suite like any others. They do enlarge one file: `corpus_rules.jsonl` is generated from this document's examples, and `Shirley Maclaine` had not been among them, so regenerating adds it (235 lines, from 234). Whether that moves the DIFFERENTIAL population is a separate question with a per-name answer, and the general answer is that a rules.md example CAN move it: the harness dedupes across corpus files, so a new example name costs a population slot exactly when no other corpus already carries that same string. `Shirley Maclaine` was already in `corpus.jsonl`, so it costs nothing and the population holds at 1090 — measured, not assumed. The row withdrawn above is the counter-case, and worth keeping for it: `Vincent Van Gogh` appears in no other corpus (`corpus.jsonl` and `corpus_issues.jsonl` carry `Vincent van Gogh`, a DIFFERENT string that does not dedupe against it), and while it was in the block the population read 1091. Neither name needed a ledger rule, which is a measurement rather than a consequence of the above: the gate exits 0 with 0 unexplained at all three baselines either way. -- 2026-08-29 — WHAT THE OVERRIDE DOES AND DOES NOT PROMISE, from Derek's framing of R5 and then measured, because the framing implies a property that is ALMOST true and the gap is the useful part. The framing first, and it supersedes the correctness talk elsewhere in this entry: mixed case is the writer making an explicit choice, and repair defers to that choice instead of judging it. Nothing is being called correct or incorrect — a name kept is a name whose writer said something about it, and a name repaired is one whose writer did not. The property that seems to follow is that asking for repair REGARDLESS should ignore the given casing entirely, so one name repairs to one string however it was written. MEASURED over the 1094 corpus names and it does NOT hold. On the v2 core — `parse(n).capitalized(force=True)` against `parse(n.upper()).capitalized()`, rendering all seven roles — the two differ for 63 names, and the lowercase direction for 16. Naming the surface matters here: through `HumanName` and `str()`, which is what tests/test_capitalization.py uses, the upper count is 62, because the facade's default render spec omits the maiden name and so cannot see `Jane van der Berg née y Jones` (the whole difference, and it is a conjunction inside the maiden name). Two things about those numbers are the opposite of what one would guess. UPPERCASE IS THE WORSE DIRECTION, not the clean one. And the misses are not merely the pipeline's case-sensitivity leaking in: of the 63, only 25 move a role at all, and the other 38 parse byte-identically and diverge inside the repair (25 and 37 through the facade, the missing name being one of the byte-identical ones). The mechanism is v1's initial carve-out — a conjunction is not lowercased where it is written initial-shaped, and initial-shaped means one CAPITAL letter — so uppercasing turns every one-letter conjunction into an initial (`Velasquez y Garcia, Dr. Juan Q.` forced keeps `y`; the same name uppercased then repaired gives `Y`, with a byte-identical partition either side -- the comma form is the one to cite here, the space-written `Dr. Juan Q. Velasquez y Garcia` being a member of the 25 whose roles DO move), and lowercasing turns a middle initial `E` into the Italian conjunction. 1.4.0 does the same (`JUAN Y GARCIA` capitalizes to `Juan Y Garcia`), so this is inherited, and it is recorded here rather than fixed here. Recompute both directions by running the two forms over the four corpus files deduped and diffing. +- 2026-08-29 — WHAT THE OVERRIDE DOES AND DOES NOT PROMISE, from Derek's framing of R5 and then measured, because the framing implies a property that is ALMOST true and the gap is the useful part. The framing first, and it supersedes the correctness talk elsewhere in this entry: mixed case is the writer making an explicit choice, and repair defers to that choice instead of judging it. Nothing is being called correct or incorrect — a name kept is a name whose writer said something about it, and a name repaired is one whose writer did not. The property that seems to follow is that asking for repair REGARDLESS should ignore the given casing entirely, so one name repairs to one string however it was written. MEASURED over the 1094 corpus names and it does NOT hold. On the v2 core — `parse(n).capitalized(force=True)` against `parse(n.upper()).capitalized()`, rendering all seven roles — the two differ for 63 names, and the lowercase direction for 18. Naming the surface matters here, and BOTH counts move with the surface, which an earlier draft of this bullet got half right: through `HumanName` and `str()`, which is what tests/test_capitalization.py uses, the counts are 62 and 16 — and the two directions are dropped for DIFFERENT reasons, which an earlier draft of this bullet ran together under the first. UPPER: the facade's default render spec omits the maiden name, so `str()` cannot see `Jane van der Berg née y Jones` (the whole of the upper difference, and it is a conjunction inside the maiden name — `maiden` is `y Jones` forced against `Y Jones` uppercased and every other role is byte-identical). LOWER: the two the facade drops are `John van der J. V` and `abdul V Smith`, and the maiden field is EMPTY in both directions for both, so the omitted role explains neither. What hides them is that `str()` CONCATENATES adjacent roles, so a token that crosses a role BOUNDARY and moves nothing else is invisible in the joined string: `John van der J. V` is family `van der J. V` forced against family `van der J.` plus suffix `V` lowercased, and `abdul V Smith` is given `Abdul V` forced against given `Abdul` plus middle `V` lowercased — same seven roles rendered, same string joined. Re-derived 2026-08-29 on this branch, and named per name so the next reader can check the arithmetic: core 63/18, facade 62/16. Two things about those numbers are the opposite of what one would guess. UPPERCASE IS THE WORSE DIRECTION, not the clean one. And the misses are not merely the pipeline's case-sensitivity leaking in: of the 63, only 25 move a role at all, and the other 38 parse byte-identically and diverge inside the repair (25 and 37 through the facade, the missing name being one of the byte-identical ones). The mechanism is v1's initial carve-out — a conjunction is not lowercased where it is written initial-shaped, and initial-shaped means one CAPITAL letter — so uppercasing turns every one-letter conjunction into an initial (`Velasquez y Garcia, Dr. Juan Q.` forced keeps `y`; the same name uppercased then repaired gives `Y`, with a byte-identical partition either side -- the comma form is the one to cite here, the space-written `Dr. Juan Q. Velasquez y Garcia` being a member of the 25 whose roles DO move), and lowercasing turns a middle initial `E` into the Italian conjunction. 1.4.0 does the same (`JUAN Y GARCIA` capitalizes to `Juan Y Garcia`), so this is inherited, and it is recorded here rather than fixed here. Recompute both directions by running the two forms over the four corpus files deduped and diffing. - 2026-08-29 — and therefore NOT stated in rules.md, which is a deliberate choice rather than an oversight. The document's examples are keyed on input STRINGS, so any statement of the property invites exactly the test that falsifies it — re-case the input, expect the same output — and the counterexamples are already in the corpora. The property is true of the repair given a parse, and rules.md speaks input-to-output; a rule stating it would be over-broad in the one direction a reader would check. What R5's statement says is enough for the promise that IS kept: a mixed-case name is kept unless repair was asked for anyway. That clause was REWORDED for this, and the reword is the whole point rather than a tidy-up. It read `unless repair regardless of how the name is cased was asked for`, which carries two readings -- the intended one, that the request overrides the keeping, and a second one, that the repair disregards the input's casing, which is this property in nearly this bullet's own words. A reader taking the second reading would run the re-casing test predicted above, land on `Velasquez y Garcia, Dr. Juan Q.` (in the corpus today), and conclude the RULE is wrong when only the phrasing was. Nine words, and they asserted the thing the paragraph exists to deny. The property is pinned in tests/test_capitalization.py instead, over names carrying no single-letter word whose class case decides, with `juan y garcia` beside it as the recorded exception. R5's example block gains `"SHIRLEY MACLAINE" → capitalized="Shirley MacLaine"` from this work, and it earns its place on its own ground rather than as half of a convergence pair: it is the only row in the block that fails when the gate is narrowed to lowercase-only, every other row passing that mutation. Measured three ways — gate deleted (passes, so it does not witness the gate's existence), gate narrowed to accept only all-lowercase (FAILS, and alone in the block), Mac/Mc convention deleted (fails, with the other two rows). Until it was added, R5 stated that repair acts on a name written entirely in one case and witnessed only the lowercase half of it. That lowercase half is still `"juan mcdonald"`, which is byte for byte an R4 row as well, and the duplication is deliberate rather than an editing slip: the two rules make different claims about the same line — R4 that the repair honors the Mac/Mc convention, R5 that an all-lowercase name is acted on at all — and dropping it from R5 would leave the gate's lowercase half unwitnessed inside the rule that states the gate. Five other rows already sit under two rules apiece for the same reason (P5/P6 twice, P5/O5, N3/M4, W1/W3). - 2026-08-29 — DEBT this extraction leaves, named so the next commit inherits an obligation rather than a rediscovery. Pulling the gate out into R5 leaves R4 carrying ONE falsehood and ONE ambiguity — different defects wanting different repairs, and `interacts: R5` carries neither, the field being advisory. FALSE: R4 promises repair "vocabulary exceptions (McDonald) included", but `str(parse('Juan Mcdonald').capitalized())` is `'Juan Mcdonald'` — the gate refuses before any vocabulary is consulted, and only `str(parse('Juan Mcdonald').capitalized(force=True))`, `'Juan McDonald'`, reaches the exception. R4 needs its promise scoped to names the gate admits. AMBIGUOUS, not false: R4's "an already-correct name comes back unchanged" means correct by the repair's own conventions, i.e. idempotence, and under that meaning it is true; a reader hears correct as the bearer writes it, and under THAT meaning `str(parse('bell hooks').capitalized())` — `'Bell Hooks'` — looks like a counterexample. It is not one, because `bell hooks` is not already-correct in R4's sense. What R4 owes is a disambiguation of "correct", NOT a narrowing to spare deliberately single-cased names: that would be new behavior, and R5's own rationale declines it on the ground that single case leaves the repair nothing to read. Also for that commit, and inert as things stand: R4's boundary row `"Juan McDonald" → capitalized="Juan McDonald"` passes with R5's gate deleted, exactly like the R5 row that was withdrawn above; rewriting it to `capitalized_forced=` makes it discriminate for R4's own subject but still witnesses nothing about the already-correct question. This commit adds R5 and touches R4 only on its pointer line, leaving both defects as found rather than half-fixed by a commit whose subject is something else. diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 03484464..4e82b735 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -21,7 +21,7 @@ Problem shape. A rule wants words to RENDER in a different order than they sit i Problem shape. A later stage needs to know what the vocabulary knew about a word. Contract statement. classify tags every token with what the vocabulary knows about it, and later stages test tags — they never re-look a word up. How it works. One lookup site means one answer: a stage that re-derived vocabulary facts could disagree with the stage before it. Stable tags ("particle", "conjunction", "initial") are API; "vocab:"-namespaced ones are not. -Lives in. nameparser/_pipeline/_classify.py (producer); consumers throughout _group/_assign/_post_rules. Reach for it when. A stage is about to import Lexicon to ask about a word classify already saw. +Lives in. nameparser/_pipeline/_classify.py (producer); consumers throughout _group/_assign/_post_rules, and nameparser/_render.py, which is not a stage but reads the same tags to decide what a view shows (#458 moved case repair's conjunction test onto the tag; initials() had read them since 2.0). Reach for it when. A stage is about to import Lexicon to ask about a word classify already saw — or a view is, which is the harder one to notice, since a view legitimately holds a Lexicon for the questions classify never answered; the view side is RENDER-HONORS-THE-PARSE, which owns it and covers the decisions this entry does not produce. ## PIECES — joining structure survives assignment @@ -57,6 +57,10 @@ Problem shape. "Which stage does X?" — asked before attributing behavior in pr Problem shape. Two stages need the same answer about the same input, and the one that does not own the decision is about to test for it. Contract statement. Where two sites ask the same question, exactly one predicate answers it and every other site calls that one — never a condition written to match it. The predicate belongs to the QUESTION, not to whichever stage decides: it may sit in a leaf both stages import, and for the leading-title test it must, since the deciding stage is assign and group cannot import assign. How it works. A hand-written mirror agrees with its original only until one of them moves, and the drift is invisible in both directions: each site keeps passing its own tests while they disagree about an input neither covers. Five instances, every one found as a defect before it was found as a pattern — #319 lifted the wholly-suffix predicate into the vocabulary layer "so the comma decision and the honorific peel's segment test cannot drift apart"; #401/#421 lifted the trailing-numeral fork out of assign so the bound-given reserve stopped carrying a copy, its hand-written mirror having been falsified in review more than once — the lesson recorded there being that what must be mirrored is assign's WALK, not merely its condition; #425 replaced that reserve's hand re-derivation of the trailing peel with one function over the view the join would leave; #424 moved assign's leading-title test down because group's own `title()` does not see H2's unlisted abbreviations, so `Xyz. van Johnson` chained where `Dr. van Johnson` did not; #429 moved the no-name-segment test down because group asked by segment INDEX where assign asks by CONTENT. The destination follows the LAYER, not the topic: a predicate over token text goes to `_vocab`, one over pieces and tags to `_pieces`. Both are leaves the stages sit on. The piece layer got its own module only in #439 — until then those predicates collected in `_group`, not because grouping owned them but because `_assign` imports `_group` and cannot be imported back, so group was the one place both stages could reach; five had accumulated across four PRs before the module existed. Stage order is this mechanism's limit, and it forecloses the alternative: where the reader comes AFTER the decider, record the answer on the state instead — `ParseState.order` is that shape, "Recorded rather than recomputed downstream, because the two can differ" — which is unavailable whenever the EARLIER stage is the one asking. (The concrete assign→group import that forced the `_group` collection is gone since #439; what remains is the ordering it was a symptom of, and tests/v2/test_layering.py is where the leaf's contract is now written down.) The cost is a second evaluation of the same predicate, measured for #429 at 1.2–2.2% of a family-comma parse and 0% of every other; recording that number was the right answer there over plumbing a state field the two sites would not otherwise share. Lives in. nameparser/_pipeline/_vocab.py over text (is_wholly_suffix; is_trailing_numeral_suffix — the #401/#421 instance, whose only caller since #439 is the shared peel rather than a stage; and maiden_marker_run, the #434 instance and the clearest two-stage case, called by classify over token texts and by extract over a clause's whitespace words, with group reading the tags classify recorded because it runs later) and nameparser/_pipeline/_pieces.py over pieces: is_suffix_piece, is_leading_title, leading_titles, peel_walk, peel_trailing and segment_suffix_reading are called by both stages — the last of those is #430's instance, where THREE readers share one answer, the render join being the third — while is_title_piece and trailing_start are called by group alone — `trailing_start` being the one to know, since it answers where the trailing run begins and is what P2's chain and M2's walk stop at. tests/v2/test_layering.py holds each module's contract, and a piece predicate growing a dependency on a STAGE shows up there as a widened entry. Reach for it when. You are about to write a condition that mirrors, matches or "does what X does" — or you find a comment saying one does. Grep for the other site's predicate and call it instead. +## RENDER-HONORS-THE-PARSE — the parse decides it, the views honor it + +Problem shape. A render view needs a fact the parse already settled — whether a word is the conjunction or an initial, whether a particle is acting as one, which word renders first. Contract statement. The parse decides it; the render views honor those decisions and never re-evaluate them. Two directions break that, and each has been found here as a defect: a view RE-DERIVES the answer from the text, keeping its own copy of a pipeline predicate — that one shipped through 2.0 and 2.1 — or a view honors the record and then OVERRIDES it, readmitting what the decision excluded, which is filed and open here. How it works. Re-deriving fails because the two copies stop being the same question long before anyone notices they are two: `_cap_word` re-ran the conjunction-versus-initial decision from the word's spelling against a hand-maintained copy of the pipeline's `_INITIAL` pattern while classify had already answered it and recorded it on the token, and `_classify.py` asks `is_initial()`, the shape test ANDed with a script-repertoire test since #320, where `_render.py` asked the bare pattern, and asked it per WORD of a token's text rather than per token, so `juan e-f smith` repaired to `Juan e-F Smith` (#458). Overriding fails more directly, and this entry's instance of it is OPEN rather than closed: `initials()` honors the `conjunction` tag through `_SKIP_TAGS` and then readmits the token whenever it also carries UNJOINED_TAG, the mark of an all-particle part, where rules.md#R3 excludes a conjunction "even then" — so under a caller's vocabulary that puts a word in both sets the view readmits what the rule excluded, and under `Lexicon.default().add(particles={'y'})` — the lexicon those two readings need, and they hold under no other — `parse("Anh y Van")` initialed `A. y. V.` and `parse("Juan de y")` `J. d. y.` on 2026-08-29. Read those two values scoped to that lexicon or they are simply false: under the DEFAULT vocabulary the same strings give `A. V.` and `J.`, and the second is a rules.md#R3 example line the doc runner asserts. #461 is where that stands, and it is worth reading before re-fixing it: the narrowing was written, measured and BACKED OUT in the same PR, because honoring R3 there cost `initials()` its agreement with `family_base`, which reads that same `de y` as the base — this entry's other shape, arrived at from the render side, and the reason the question moved from the code to R3's clause (decisions.md carries the argument). What to carry away is the DIRECTION and not its verdict here: a view that honors a record and then readmits what the record excluded is overriding a decision it never took, whichever way this one settles. Case repair reads that same token and does NOT readmit it — `capitalized(force=True)` on `Anh y Van` gives `Anh y Van` under that same lexicon, R4 carrying the carve-out in its own words but ON R3's authority — its text reads "being no name word in any part — the carve-out R3 states for initials", so a change to R3's clause reaches R4's TEXT. What does NOT follow, though an earlier wording of this sentence asserted it, is that the two stand or fall together in BEHAVIOR: they have already come apart, over the 25 corpus names carrying a conjunction in the GIVEN group — `parse("john and jane smith").capitalized()` keeps `and` lowercase, so R4's carve-out holds there, while `.initials()` gives `j. a. j. s.`, so R3's does not (decisions.md#R2 carries that population, and rules.md#R3 now says so in its own words). The dependency is textual, and only textual — so the two views disagree today about that token exactly as they did before #461 and the backout restores that disagreement knowingly; only the FORCED call witnesses the repair half, R5's gate refusing a mixed-case name before any of this is consulted. This is the CONSUMER-side rule over the producer-side entries — VOCAB-TAGS records what the vocabulary knew, MARK-DONT-STRIP what a stage decided about it, FOLDED_TAG what order to render in — and a view reads what they recorded, whichever kind it is. It sits where ONE-PREDICATE-PER-QUESTION's stated limit leaves off: where two live sites need one answer they share a predicate, but a render view always comes AFTER the decider, so the answer is recorded rather than shared — on the TOKEN, which is the views' equivalent of that entry's `ParseState.order`, no view being able to see a ParseState at all. Known limit, and the half most easily got wrong next: a token the parse never saw carries no decision to honor, so a view falls back to the vocabulary — the tell is `UNCLASSIFIED_TAG`, which `ParsedName.replace()` stamps when it splices raw text into a field and the facade's v1 pickle load stamps when it rebuilds a name from `*_list` strings. It is NOT untaggedness, since an ordinary parsed name word carries no tags either; and it is NOT `span is None`, which was tried and is wrong in the other direction — span-less means SYNTHETIC, and `Parser.revise()` builds span-less tokens from a full sub-parse whose tags it keeps on purpose, so the span reading overrode exactly the tags `revise()` exists to preserve (`revise(middle='e-f')` repaired to `e-F` where the parse gave `E-F`). A hand-built span-less token is unmarked and therefore classified, which is the same tag-driven default every other view applies. A view can only fall back if it is HANDED a vocabulary, and exactly one is: `capitalized(lexicon=...)`. `family_base` and `family_particles` are properties on ParsedName, whose fields are original/tokens/ambiguities and nothing else, so a spliced field empties the particles view and leaves the base the whole field, with `Parser.revise()` the crossing there too (docs/usage.rst says so where it documents the degradation). `initials()` is the near miss and the instructive one: it is a METHOD, so it looks like it could ask, but its signature is `(spec, delimiter, separator)` and carries no lexicon — a fallback there was written and dropped because it had to GUESS `Lexicon.default()`, and the guess erased a whole field under a caller's own vocabulary (decisions.md#R4). `capitalized()` guesses nothing: it reads the lexicon it was handed, and only defaults to `Lexicon.default()` when the caller passes none, which is the documented meaning of omitting the argument rather than a fallback. Within the one view that can fall back, the fallback is drawn per QUESTION (rules.md#R4's Accepted clause). Whether a word is the conjunction or an initial is a property of the word, which a vocabulary answers alone, so case repair asks it. Whether a part is wholly particles is a property of the whole PART, which the pipeline answers once and records as UNJOINED_TAG. What `_cap_word` cannot do is RE-DERIVE that answer where no word of the part carries a tag — it is handed the whole token's tags and gates the particle conjunct on UNJOINED_TAG, so it is not blind to the part, it simply has no evidence to reconstruct one from — so repair leaves that half to plain particle treatment and rules.md#R4's Accepted boundary records the consequence — a spliced field is not repaired as a parsed one is, with `Parser.revise()` as the supported crossing. A fallback is right only while it answers as the pipeline would, and THAT is held by hand rather than mechanically: test_regex_sync pins the two `_INITIAL` copies to each other and to config, while the repertoire half of the pipeline's predicate (#320) is deliberately not carried across, layering forbidding the import. What that divergence can reach is now nothing observable: it needs a caller-added conjunction written initial-SHAPED in a script that has no initials (`씨.`), and case repair is the fallback's only reader, so the two paths differ by `lower()` versus `capitalize()` over a caseless script — the same string either way. `initials()` used to be the reader that could witness it, and no longer falls back at all. A second limit, recorded rather than closed: `_cap_word`'s PARTICLE conjunct still keys on the lexicon handed to the view rather than on the `particle` tag, so a repair run with a lexicon other than the parse's re-decides a word the parse already read — a name parsed under the default vocabulary, where `parse('juan smith vega')` reads `vega` as the family, repairs to `Juan Smith vega` when `capitalized()` is handed `Lexicon.default().add(particles={'vega'})` instead — the divergence needs the two lexicons to differ, and repairing under the parse's own lexicon gives `Juan Smith Vega`. Making it read the tag moves a boundary rules.md#R4 states in prose, so it is a separate decision and not a cleanup (decisions.md#R4, "NOT DONE"). Lives in. nameparser/_render.py (`capitalized`/`_cap_word`, `_reads_as_conjunction`, and `initials`, which honors tags and never falls back) and nameparser/_types.py (`_text_for` and `UNCLASSIFIED_TAG`, with the `ParsedName.replace()` producer beside it) and nameparser/_facade.py (the v1 pickle load, the SECOND producer of that mark — it is named in this list because a change that follows the list into `_types.py` alone leaves it behind, which is the site test_a_restored_pickle_keeps_v1_conjunction_repair exists to protect), reading what nameparser/_pipeline/ recorded — the mark those views read is recomputed producer-side in `_remarked`, which is deliberately silent about text nobody classified and is right to be. Reach for it when. A view is about to consult a Lexicon, a regex or an exception list about a word the parse already saw — or a view and a field disagree about the same parse. #408 is filed as that second shape, and was open on 2026-08-29: `initials()` walks tokens in written order where the family field applies FOLDED_TAG's ordering, so `parse("der, y van")` gave family `van der` and initials `y. d. v.` that day, one view not yet honoring the fold. It belongs here rather than argued again from scratch. + ## CLAUSE-CONTENT-OVERRULES-DELIMITER — content wins Problem shape. A bracketed clause should be treated as something other than what its delimiter pair says. Contract statement. extract may inspect a clause's content against the lexicon and, when it matches, mask only the two delimiter spans so the inner content rejoins the main token stream for ordinary downstream parsing. How it works. "Andrew Perkins (MBA)" is not a nickname (rule S1): the parens are masked away and MBA is classified by the normal machinery — reusing the downstream path, so the delimited and bare forms cannot drift. Lives in. nameparser/_pipeline/_extract.py (_suffix_shaped and the inner-span branch). Reach for it when. About to add a second code path that duplicates what the bare form already does — #335's fix is this shape (a _maiden_marked sibling predicate). diff --git a/docs/design/rules.md b/docs/design/rules.md index f0b0dcc1..27277858 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -1101,7 +1101,17 @@ R3. Rationale: initials abbreviate the person's name words; titles, contribute nothing — except the particles of a part whose every word is one, which are not acting as particles there (R2) and initial like any other name word. A CONJUNCTION never initials, - so a base that is one contributes nothing even then. + so a base that is one contributes nothing even then. That + carve-out is stated for the middle and base family words; the + GIVEN group is not settled here. A conjunction written among + given names does initial today, and this document does not yet + say whether it should — because two of its own rules answer + differently and neither answer has been taken: this rule counts + name words, while P3 makes a connective and its neighbours ONE + name word, so a joined given group owes one initial under P3 and + one per joined name word under the carve-out. Until that is + decided the given group's answer is pinned-but-undocumented + rather than specified, and no line below asserts it. "Dr. Juan Q. Xavier de la Vega III" → initials="J. Q. X. V." "Anh Do" → initials="A. D." "Nguyen, Van Le" → initials="V. L. N." @@ -1110,7 +1120,32 @@ R3. Rationale: initials abbreviate the person's name words; titles, rather than nothing: they are the base (R2), so they initial. "Juan van der" → initials="J. v. d." "Juan de y" → initials="J." - history: decisions.md#R2 · interacts: R2 · implemented: nameparser/_render.py, nameparser/_facade.py + Accepted: this rule reads a part the parser read. A field set as + raw text after the parse carries no reading, and this view is + handed no vocabulary to supply one — it takes a format spec and + two separators and nothing else — so every word of such a field + initials, particles and conjunctions alike: a family set that way + to "de la vega" gives "j. d. l. v." where parsing the same + name gives "j. v.". Case repair IS handed a vocabulary, so it falls + back for the one question a word can answer on its own, and R4 + says which. Revising the field through the parser classifies it + and matches the parse in both views. Stated without an example + line because every line here names an input string, and this + shape needs a field edited after the parse. + Accepted: the unsettled given-group answer above is neither rare + nor hypothetical — 25 of the corpus names carry a conjunction + among the given names, every one of them reachable from the + default vocabulary, and it has initialed since 1.4.0. It carries + no marked deviation, for the reason that mechanism exists: a + marker states the INTENDED value, and one name, "John and Jane + Smith", has four candidates. Today gives "J. a. J. S."; the + carve-out read as written gives "J. J. S."; P3's one-name-word + join gives just "J. S."; and 1.4.0 gave "J a J. S.". Marking it + would put an invented value in a normative document and hold + the parser to it. #461 asks the neighbouring question about the + all-particle base and does not own this one; decisions.md#R2 + carries the population and the measurements. + history: decisions.md#R2 · interacts: P3, R2, R4 · implemented: nameparser/_render.py, nameparser/_facade.py R4. Rationale: case repair is a display concern, applied only on request and never destructively. @@ -1145,6 +1180,27 @@ R4. Rationale: case repair is a display concern, applied only on repaired exactly this way before the clause existed. Stated without an example line because every line here names an input string, and this shape needs a field edited after the parse. + Accepted, and the reason the boundary is drawn per question + rather than per field: the conjunction carve-out reaches a word + the parse read as a conjunction, and where the parse read nothing + at all it reaches what the vocabulary says. A word of that + vocabulary standing inside a longer written word is not a + conjunction, because the parse read that word as one ordinary + name word — but a field spliced in as raw text was read by + nobody, so repair asks the vocabulary and a family set to "de y" + keeps its "y" lowercase. Whether a word is the conjunction or an + initial is a property of the word, which a vocabulary can answer; + whether a particle is acting as a particle is a property of the + whole part, which the parse settles and records; re-deriving it + needs a reading on every word of the part, and a spliced field + has none on any, so that half falls through to particle treatment + and the "de la" boundary above stands. Initials are the contrast + worth knowing, and R3 states it: that view is handed no + vocabulary at all, so it falls back on neither question and a + spliced field's every word initials. revise() classifies the + value and crosses both questions, in both views: a middle revised + to "e-f" repairs to "E-F" as the parsed name does, where splicing + the same text in gives "e-F". history: decisions.md#R4 · interacts: R2, R3, R5 · implemented: nameparser/_render.py R5. Rationale: mixed case is evidence that the writer cased the name diff --git a/docs/release_log.rst b/docs/release_log.rst index a6b7dd6f..e85b8301 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -67,6 +67,8 @@ Release Log - Fix case repair lowercasing the words of a family name made only of particle words, where every other view already reads them as ordinary name words: ``HumanName("ANH DO").capitalize()`` gives ``Anh Do`` where it gave ``Anh do``, and ``"anh van do"`` gives ``Anh Van Do`` where it gave ``Anh van do``. A particle earns its name by joining forward to the word it modifies, so a part whose every word is particle vocabulary leaves none of them anything to join; the fix above already made those words anchor ``family_base`` and contribute initials, and case repair now agrees with them rather than reading the same word two ways. The test is the whole part, not a particle standing alone, which is why the two-word family in ``"anh van do"`` moves along with the one-word family in ``"ANH DO"`` -- the same Vietnamese surname, and a standing-alone test would have read it one way behind a given name and another way alone. This DIFFERS FROM 1.4.0 deliberately and does not restore it: 1.4.0 returned ``Anh do``, lowercasing on vocabulary membership alone. The accepted cost is that a degenerate family which is nothing but particles capitalizes too, so ``"juan van der"`` gives ``Juan Van Der`` where 1.4.0 gave ``Juan van der``. A conjunction is untouched by any of this, so ``"der, y van"`` gives ``y Van Der`` -- the family capitalizing while the conjunction keeps the lowercase it always had; and where the particles DO join a name word nothing changes, ``"juan de la vega"`` still giving ``Juan de la Vega``. Case repair is not one of the seven role fields the differential harness compares, so no gate run can see this change either way and none of its counts move; the ``rules.md#R4`` examples and the v1 capitalization tests are what pin it (closes #407) + - Change case repair to read the parser's own ``conjunction`` tag instead of re-deciding, from the word's spelling, whether a word is a conjunction or an initial. The parse answers that question already -- ``"Scott E. Werner"`` reads ``E.`` as an initial rather than the Italian conjunction -- and the other views honor the answer; case repair asked again, with a shape test applied to each word of a token's text rather than to the token. Two spellings of one name disagreed because of it: ``"juan e-f smith"`` capitalized to ``Juan e-F Smith`` while ``"JUAN E-F SMITH"`` gave ``Juan E-F Smith``; both give ``Juan E-F Smith`` now, ``e-f`` being a middle name and no conjunction of the parse's reading. A conjunction written as a word of its own is untouched, and so is the one-letter carve-out where it applies -- ``"juan y garcia"`` still repairs to ``Juan y Garcia``, ``"JUAN Y GARCIA"`` still to ``Juan Y Garcia``. A field assigned after the parse is unaffected: its text was never classified, so there is no reading to honor and repair asks the vocabulary, applying v1's own predicate the way every earlier version applied it everywhere -- ``h.last = "velasquez y garcia"`` still repairs to ``Velasquez y Garcia`` and ``h.middle = "e."`` to ``E.``. That is the predicate over TODAY's vocabulary, which is narrower than parity with 1.4.0 and the difference is real: ``h.last = "хосе и мария сантос"`` gives ``Хосе И Мария Сантос`` on 1.4.0 and ``Хосе и Мария Сантос`` here, because the Cyrillic ``и`` is a 2.x conjunction and was not a 1.4.0 one. What decides which path a token takes is a mark the assignment leaves, not the absence of a span: a value revised through ``Parser.revise()`` is classified by a sub-parse and keeps its tags, so it repairs as the parse does. One reading does change for hand-built ``Token``\ s in the 2.0 API: an untagged token whose text is conjunction vocabulary is now an ordinary name word and capitalizes, where 2.1 lowercased it -- tags are what the views read, and a hand-built token that carries none is a token with nothing to declare. Case repair is not one of the seven role fields the differential harness compares, so no gate run can see this change either way and none of its counts move; measured directly instead, no name of the 1094-name differential corpus moves under ``capitalized()`` or ``capitalized(force=True)``, its uppercased and lowercased spellings included -- 6564 name/spelling/lexicon rows and 13128 calls (closes #458) + - Fix a tussenvoegsel attached to the family name after a comma deciding a genuinely uncertain reading and reporting nothing. ``"Van Johnson"`` reports a ``PARTICLE_OR_GIVEN`` ambiguity -- ``Van`` is a Dutch particle and a Vietnamese given name, and the parser has to pick one -- while ``"Nguyen, Thi Van"`` picked the same word the same way, silently, and lost the given name doing it. The attachment now reports the fork it decides, in the kind that names the reading it declined. A particle that could be an ordinary name reports ``PARTICLE_OR_GIVEN``: ``"Nguyen, Thi Van"``, ``"Berg, Jan van der"`` and ``"Vega, Juan de la"`` each gain one, the ``detail`` naming the ambiguous word. A particle the parser had already read as a post-nominal reports ``SUFFIX_OR_NAME`` instead, because the credential reading is what the attachment overrode: ``"Berg, Jan vd"`` gains one, ``vd`` being read as *van der* rather than as the Volunteer Decoration. Which kind you get follows the reading that was overridden rather than the word's vocabulary, so ``"Berg, Jan do"`` reports ``PARTICLE_OR_GIVEN`` even though ``do`` is a postnominal too -- it was already being read as a name word, so no credential reading was overridden. A particle where nothing was overridden reports nothing at all: ``"Jong, Piet de"`` and ``"Jong, Anke de"`` are unchanged, ``de`` being no name in any reading and no postnominal either, and so is ``"Berg, Jan de vd"``, whose run was read as name words whole. Worth knowing before you filter on this: ``"Beethoven, Ludwig van"`` -- the textbook Dutch listing, read exactly right -- now carries a report too. It is the same string shape over the same vocabulary as ``"Nguyen, Thi Van"``, and nothing in the input separates them, so a report on one is a report on both. This adds the report and nothing else: every field these names parse to is exactly what the tussenvoegsel fix below already gave them, and ``ambiguities`` is the only value that grows. Seven differential corpus names gain a kind against the 2.0.0 and 2.1.0 baselines, seven of the nine the tussenvoegsel fix below already moved -- five of ``fix(#379)``'s seven and both of ``fix(#380)``'s two, now carrying ``_ambiguities`` in their diff as well; the two left out are the pair named unchanged above, ``Jong, Piet de`` and ``Jong, Anke de`` -- and none against 1.4.0, which had no ambiguity reporting at all (closes #405) - Fix a tussenvoegsel after a family comma being parsed as a middle name. Dutch and Belgian alphabetized listings move the particle behind the given name -- ``"Beethoven, Ludwig van"`` is how ``"Ludwig van Beethoven"`` is filed -- and the trailing particle run was read as a middle name rather than as part of the surname: ``"Beethoven, Ludwig van"`` gave middle ``van``, last ``Beethoven``, and ``"Berg, Jan van der"`` gave middle ``van der``. The run now attaches to the family the comma has already named and renders before it, so those read family ``van Beethoven`` and ``van der Berg`` with the given name unchanged. The derived views move with the parse, so ``family_particles`` is ``van`` and ``family_base`` is ``Beethoven`` where they were empty and ``Beethoven`` before. `#130 `_ asked for the split and got it in 1.3.0 as ``last_base``/``last_prefixes``; 2.0 renamed them ``family_base``/``family_particles``. What was wrong until now was the values they reported for this listing. Both halves of the particle vocabulary attach -- never-given ``de`` and may-be-given ``van`` alike -- because after a comma the family is already named and the particle has no other role to take. Two guards bound it. A name whose only given word is the particle keeps it, so ``"Nguyen, Van"`` still reads given ``Van``: the attachment needs a given word to spare. And where the word is BOTH particle and suffix vocabulary the attachment outranks the post-nominal reading, so ``"Berg, Jan vd"`` reads family ``vd Berg`` where 1.4.0 and 2.1 alike gave suffix ``vd`` -- a trailing abbreviation after a family comma is the tussenvoegsel far more often than the decoration it collides with, and the same shape sweeps in ``mc``, which 2.1 also read as a suffix. ``do`` is in ``SUFFIX_ACRONYMS_AMBIGUOUS`` and 2.1 already read a trailing one as a name word, so it attaches by the plain rule rather than by the override (closes #379, closes #380). Names without the comma are untouched: ``"Ludwig van Beethoven"`` already read family ``van Beethoven`` and is byte-identical. One of the 751 differential corpus names moves, ``"Vega, Juan de la"``, at the 1.4.0, 2.0.0 and 2.1.0 baselines alike diff --git a/docs/usage.rst b/docs/usage.rst index 13017133..c09ca979 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -887,9 +887,23 @@ clears the flag, while correcting an unrelated field keeps it. ``replace()`` splits values on whitespace into plain, untagged tokens — the vocabulary knowledge a parse would have about the new -text is not there. The views that depend on tags degrade: the parser +text is not there. The views that read those tags degrade: the parser no longer knows ``de la`` are particles, so ``family_particles`` -empties and the particles start contributing initials. +empties and ``family_base`` takes the whole field. + +A token the parse never saw carries no decision to honor, so a view +that is *handed* a vocabulary can fall back to it — +:meth:`~nameparser.ParsedName.capitalized` is the one that is, and it +falls back for one question only: whether a word is a conjunction or +an initial, which a word answers on its own. Whether a particle is +acting as a particle is a fact about the whole part, and there is no +reading on any word of a spliced field to derive it from, so a family +set to ``de la`` stays lowercase where the same words parsed are +repaired to ``De La``. :meth:`~nameparser.ParsedName.initials` takes +no vocabulary at all, so it falls back on neither question and every +word of a spliced field contributes an initial. ``family_particles`` +and ``family_base`` are properties on the parsed name, which holds no +vocabulary of its own either. .. doctest:: @@ -898,8 +912,12 @@ empties and the particles start contributing initials. >>> replaced = name.replace(family="de la Vega Smith") >>> replaced.family_particles '' + >>> replaced.family_base + 'de la Vega Smith' >>> replaced.initials() 'J. d. l. V. S.' + >>> name.replace(family="de la").capitalized(force=True).family + 'de la' :meth:`Parser.revise() ` is the same operation with each value classified by the parser's vocabulary, so diff --git a/nameparser/_facade.py b/nameparser/_facade.py index 099e6a00..051d44e9 100644 --- a/nameparser/_facade.py +++ b/nameparser/_facade.py @@ -32,7 +32,8 @@ from nameparser._config_shim import CONSTANTS, Constants, _cached_parser from nameparser._lexicon import _normalize from nameparser._parser import Parser -from nameparser._types import FOLDED_TAG, ParsedName, Role, Token +from nameparser._types import (FOLDED_TAG, UNCLASSIFIED_TAG, ParsedName, + Role, Token) _V2_FIELD = {"first": "given", "last": "family"} # v1 name -> v2 name _V1_SPELLING = {v2: v1 for v1, v2 in _V2_FIELD.items()} @@ -746,8 +747,17 @@ def __setstate__(self, state: dict[str, Any]) -> None: f"nameparser" ) for position, word in enumerate(entry.split()): - tokens.append(Token( - word, None, role, - frozenset({"joined"}) if position else frozenset())) + # UNCLASSIFIED_TAG for the same reason replace() + # stamps it: a pickle carries the *_list STRINGS + # and no tags, so nothing here was read by a parse + # and case repair must ask the vocabulary rather + # than read an absent conjunction tag. Without it a + # restored "juan ortega y gasset" repairs to + # "Ortega Y Gasset", which is neither v1's answer + # nor the same name's unpickled one. + tags = {UNCLASSIFIED_TAG} + if position: + tags.add("joined") + tokens.append(Token(word, None, role, frozenset(tags))) self._parsed = ParsedName( original=str(state.get("original", "")), tokens=tuple(tokens)) diff --git a/nameparser/_render.py b/nameparser/_render.py index 4502329d..d3634669 100644 --- a/nameparser/_render.py +++ b/nameparser/_render.py @@ -15,8 +15,8 @@ import re from nameparser._lexicon import Lexicon, _normalize -from nameparser._types import (UNJOINED_TAG, Ambiguity, ParsedName, Role, - Token) +from nameparser._types import (UNCLASSIFIED_TAG, UNJOINED_TAG, Ambiguity, + ParsedName, Role, Token) _SPACES = re.compile(r"\s+") _SPACE_BEFORE_COMMA = re.compile(r"\s+,") @@ -35,28 +35,48 @@ #: Tags whose tokens contribute no initial outside the given group -- #: unless the token also carries UNJOINED_TAG, i.e. the whole part is #: particles, in which case they are the part's only words and do -#: contribute (rules.md#R3, #404). +#: contribute (rules.md#R3, #404). The mark readmits a token carrying +#: EITHER tag: a conjunction with nothing to join is not acting as a +#: conjunction any more than a particle with nothing to join is acting +#: as a particle, so it is a name word of the part like the rest. #: Not STABLE_TAGS -- that also contains "initial", which must contribute. _SKIP_TAGS = frozenset({"particle", "conjunction"}) -# Ported verbatim from v1 (nameparser/config/regexes.py "initial", -# minus the empty alternative) -- layering forbids importing the -# pipeline here; keep in sync with _pipeline/_vocab.py by hand. -# Deliberately NOT composed with that module's repertoire test (#320): -# layering forbids the import, and nothing here needs it. The only use -# is v1's conjunction carve-out in _cap_word below, which this pattern -# can only reach once `normalized in lex.conjunctions` already holds -- -# and no CJK token reaches that, the shipped vocabulary carrying no CJK -# conjunction or particle in the default lexicon or in any locale pack. -# That is a property of the shipped DATA, not an invariant -- conjunctions -# is public, configurable API -- but the divergence stays harmless if a -# user adds one: CJK is caseless, so the carve-out's word.lower() and the -# fall-through's word.capitalize() return the same string either way. -# So the two copies keep identical PATTERNS and divergent PREDICATES; -# test_regex_sync pins the patterns, which is the promise being kept. +# Ported verbatim from v1 (nameparser/config/regexes.py "initial", minus +# the empty alternative) -- layering forbids importing the pipeline here; +# keep in sync with _pipeline/_vocab.py by hand. +# Its one reader is _reads_as_conjunction below, and that reader only +# ever sees text the parse never classified: for anything the +# parser DID see, the tag is the answer and this pattern is not asked. +# So the two copies no longer decide the same question about the same +# token -- _vocab's says what the parse decided, this one says what it +# WOULD have decided about text spliced in afterwards -- which is why +# they must keep answering alike, and why test_regex_sync pins the +# patterns against each other and against config. +# Deliberately NOT composed with _vocab's repertoire test (#320): +# layering forbids the import. The divergence is reachable only for a +# caller-added CJK conjunction spliced into a field, since no shipped +# vocabulary carries one, and it costs nothing there: CJK is caseless, +# so the carve-out's lower() and the fall-through's capitalize() return +# the same string, and case repair is now this pattern's only reader. _INITIAL = re.compile(r"^(\w\.|[A-Z])$") +def _reads_as_conjunction(word: str, lex: Lexicon) -> bool: + """v1's is_conjunction, asked only of text the parse never saw. + + A token the parse classified carries its reading in its tags and + this is not consulted. A token carrying UNCLASSIFIED_TAG was + spliced into a field as raw text -- by replace(), or by the + facade's v1 pickle load -- and carries no reading, so case repair + falls back to the vocabulary, which gives the answer the parser + would have given, the initial carve-out included ('E.' assigned to + middle is an initial, not the Italian conjunction). + """ + return bool(_normalize(word) in lex.conjunctions + and not _INITIAL.fullmatch(word)) + + def _collapse(rendered: str) -> str: """The #254 collapse: empty fields substitute '' and every artifact of that is removed -- dangling empty-nickname wrappers, space runs, @@ -103,9 +123,18 @@ def initials(name: ParsedName, spec: str, delimiter: str, separator: str) -> str """First letter of each contributing token per group, v1 semantics: delimiter follows each initial, separator sits between initials within a group. Tokens tagged particle/conjunction contribute no - initial in middle/family (given-name tokens always contribute); - tags come from the pipeline -- hand-built untagged tokens all - contribute. Valid spec keys: given, middle, family.""" + initial in middle/family (given-name tokens always contribute), + and the unjoined mark readmits the words of an all-particle part + whichever of those tags they carry; tags come from the pipeline -- + hand-built untagged tokens all contribute, and so do the words of + a field spliced in by replace(), which the parse never read. + This view takes NO lexicon, so it has none to fall back to for + that text: `replace(family='de la vega')` initials every word of + that field where the same name parsed gives 'j. v.' + (rules.md#R3's Accepted + clause, and decisions.md#R4 for why the fallback was tried + and dropped -- #464 is the crossing that would make it + answerable). Valid spec keys: given, middle, family.""" if not isinstance(delimiter, str): raise TypeError(f"delimiter must be a str, got {delimiter!r}") if not isinstance(separator, str): @@ -132,11 +161,10 @@ def _cap_word(word: str, role: Role, tags: frozenset[str], # repaired as ordinary name words, since none of them is doing a # particle's work there" -- UNJOINED_TAG is that mark (#407). # Only the PARTICLE conjunct is gated on it, and that is the rule - # rather than an omission: rules.md#R4 carries the carve-out R3 - # already states for initials -- "A CONJUNCTION never initials, so - # a base that is one contributes nothing even then" -- so a - # conjunction keeps conjunction treatment even inside a part the - # mark has turned into ordinary name words. + # rather than an omission -- rules.md#R4: "A CONJUNCTION keeps its + # lowercase even inside such a part, being no name word in any + # part" -- so a conjunction keeps conjunction treatment even + # inside a part the mark has turned into ordinary name words. # No SHIPPED name witnesses the difference: `particles` and # `conjunctions` are disjoint in the default vocabulary and in # every locale pack, so no shipped conjunction can sit in an @@ -147,13 +175,52 @@ def _cap_word(word: str, role: Role, tags: frozenset[str], # under `Lexicon.default().add(particles={'y'})`, `anh y van` has # an all-particle family whose `y` carries both tags and the mark, # and gives 'Anh y Van'; gating this conjunct too would give - # 'Anh Y Van'. - # v1's is_conjunction excludes initials: 'E.' in 'Scott E. Werner' - # is an initial, not the conjunction 'e' (pinned live 2026-07-17) + # 'Anh Y Van'. That is pinned by test_repair_keeps_a_conjunction_ + # lowercase_in_a_particle_part -- until which gating it passed the + # whole suite. + # initials() does NOT match this carve-out, and the mismatch is + # recorded rather than fixed: #461 made it match and was backed + # out, the mark being a statement about a whole PART that #461 + # honored for some of the part's words and not for one of them, + # so what is in question is R3's "even then" clause rather than + # the code (decisions.md, under R2). Under that same lexicon + # `Anh y Van` repairs to 'Anh y Van' and initials 'A. y. V.', + # pinned by + # test_initials_readmits_a_conjunction_in_a_particle_part. + # That conjunct reads the TAG, not the word (#458). classify takes + # the conjunction-versus-initial decision once, over the whole + # token -- v1's is_conjunction excludes initials, so 'E.' in + # 'Scott E. Werner' is an initial and is never tagged (pinned live + # 2026-07-17) -- and a view honors that decision rather than + # taking it again from the spelling + # (mechanisms.md#RENDER-HONORS-THE-PARSE: "the render views honor + # those decisions and never re-evaluate them"), the tags being + # classify's record of it (mechanisms.md#VOCAB-TAGS: "later stages + # test tags"). Asking again was not even the same question: + # the copy of the initial pattern that stood here was the SHAPE + # half alone, and it re-decided per WORD of a token's text, so + # 'juan e-f smith' capitalized to 'Juan e-F Smith'. + # mechanisms.md#RENDER-HONORS-THE-PARSE: "a token the parse never + # saw carries no decision to honor, so a view falls back to the + # vocabulary" -- _reads_as_conjunction above, which is v1's + # predicate applied over TODAY's vocabulary rather than 1.4.0's. + # That is the honest claim and it is narrower than parity: the two + # vocabularies differ, so an assigned field can repair differently + # from 1.4.0 without this predicate differing at all. Measured on + # the released wheel: `h.last = "хосе и мария сантос"` gives + # 'Хосе И Мария Сантос' on 1.4.0 and 'Хосе и Мария Сантос' here, + # the Cyrillic `и` being a 2.x conjunction and not a 1.4.0 one; + # `h.last = "de la vega"` gives 'de la Vega' there and here. + # The mark, not the SPAN, is what says the text was never read: + # Parser.revise() also builds span-less tokens, from a sub-parse + # whose tags it keeps on purpose, and keying this on `span is None` + # overrode them -- `revise(middle='e-f')` repaired to 'e-F' where + # the same words parsed gave 'E-F' (#463 review). if ((normalized in lex.particles and role in (Role.MIDDLE, Role.FAMILY) and UNJOINED_TAG not in tags) - or (normalized in lex.conjunctions - and not _INITIAL.fullmatch(word))): + or "conjunction" in tags + or (UNCLASSIFIED_TAG in tags + and _reads_as_conjunction(word, lex))): return word.lower() # v1 cap_word tries the edge-stripped form, then the period-free # form ('Ph.D.' -> 'ph.d' -> 'phd' hits the exceptions map) @@ -171,7 +238,11 @@ def _cap_word(word: str, role: Role, tags: frozenset[str], def _cap_text(text: str, role: Role, tags: frozenset[str], lex: Lexicon) -> str: # word-by-word within the token text: hyphenated names capitalize - # both sides ("macdole-eisenhower" -> "MacDole-Eisenhower") + # both sides ("macdole-eisenhower" -> "MacDole-Eisenhower"). The + # per-word walk is also why an UNCLASSIFIED token gets the + # vocabulary asked per word: the parse would have made one token + # per word of that text, so this is the granularity its answer + # would have had. return _WORD.sub(lambda m: _cap_word(m.group(0), role, tags, lex), text) @@ -186,12 +257,18 @@ def capitalized(name: ParsedName, lexicon: Lexicon | None, *, formatting and the #254 collapse). The repair reads token TAGS as well as texts: a part whose every word is particle vocabulary is repaired as ordinary name words, - and the mark saying so comes from the pipeline. A hand-built - token, or one replace() splices in, carries no tags and is - repaired as a plain particle instead: a family set that way to - 'de la' stays 'de la' where the same words parsed give 'De La'. - Parser.revise() is the edit that classifies the value, and gives - 'De La' (rules.md#R4's Accepted boundary). + and the mark saying so comes from the pipeline, as does the + reading that a word is a conjunction rather than an initial. A + token carrying UNCLASSIFIED_TAG -- replace() splices those in, and + so does the facade's v1 pickle load -- was never read: the + vocabulary answers the per-word conjunction question for it, and + the per-part particle question is left to plain particle treatment, + since re-deriving the part answer needs a tag on every word of the + part and these have none. A family set that way to 'de la' stays + 'de la' where the same words parsed give 'De La'; one set to + 'de y' keeps the 'y' lowercase, as the parse does and as 1.4.0 + did. Parser.revise() is the edit that classifies the value, and + gives 'De La' (rules.md#R4's Accepted boundary). Idempotent: without force, a capitalized result is mixed-case and the gate returns it unchanged; with force, every _cap_word rule is a fixpoint on its own output.""" diff --git a/nameparser/_types.py b/nameparser/_types.py index 975d1583..745ff06d 100644 --- a/nameparser/_types.py +++ b/nameparser/_types.py @@ -115,6 +115,24 @@ def __add__(self, other: object) -> NoReturn: # type: ignore[override] #: drift. FOLDED_TAG = "vocab:folded-middle" +#: Raw text spliced into a field after the parse, which no parse ever +#: read: `ParsedName.replace()` stamps it, and so does the facade's +#: v1 pickle load, which rebuilds a name from `*_list` strings alone. +#: A view that is HANDED a vocabulary falls back to it for a token +#: carrying this -- there is no decision to honor -- and reads every +#: other token by its tags (mechanisms.md#RENDER-HONORS-THE-PARSE). +#: `capitalized(lexicon=...)` is the one such view; `initials()` takes +#: no lexicon and so honors tags alone, marked or not. +#: The absence of a SPAN is not that signal and was tried as one: a +#: span-less token is SYNTHETIC, which `Parser.revise()` also builds, +#: from a full sub-parse whose tags it deliberately keeps. Keying the +#: fallback on the span overrode exactly those tags. Absence of TAGS is +#: not the signal either -- an ordinary parsed name word carries none. +UNCLASSIFIED_TAG = "vocab:unclassified" + +#: The one-element tag set its two producers stamp, built once. +_UNCLASSIFIED = frozenset({UNCLASSIFIED_TAG}) + _E = TypeVar("_E", bound=Enum) @@ -737,14 +755,19 @@ def replace(self, **fields: str) -> ParsedName: empty value clears the field. original is unchanged (provenance). Ambiguities referencing replaced tokens are dropped. - Replacement tokens carry NO tags, so tag-driven views degrade: - family_particles empties, particles regain their initials, and - a multi-word suffix is comma-joined. Parser.revise() is the - tag-preserving alternative. + Replacement tokens carry no STABLE tag, so tag-driven views + degrade: family_particles empties, particles regain their + initials, and a multi-word suffix is comma-joined. + Parser.revise() is the tag-preserving alternative. + They do carry UNCLASSIFIED_TAG, which says the text was never + read rather than that it was read and found plain: a view + holding a vocabulary falls back to it for these and to the tags + for everything else. """ replaced = _validated_field_strings(fields) synthetic = { - role: tuple(Token(word, None, role) for word in value.split()) + role: tuple(Token(word, None, role, _UNCLASSIFIED) + for word in value.split()) for role, value in replaced.items() } return self._with_field_tokens(synthetic) diff --git a/tests/test_capitalization.py b/tests/test_capitalization.py index abbe5262..22d93d5e 100644 --- a/tests/test_capitalization.py +++ b/tests/test_capitalization.py @@ -1,3 +1,7 @@ +# pickle here round-trips an object this test just built; the library's +# own pickle support is what is under test, and no foreign data is read. +import pickle + import pytest from nameparser import HumanName @@ -227,10 +231,12 @@ def test_capitalize_all_particle_family_beside_a_conjunction(self) -> None: # the four corpus files deduped and diffing, on whichever surface # you name. # - # The mechanism is v1's initial carve-out, which _cap_word - # documents a few lines from the predicate: a conjunction is not - # lowercased where it is written initial-shaped, and - # initial-shaped means one CAPITAL letter. Uppercase a name and + # The mechanism is v1's initial carve-out, taken in the PARSE + # since #458 and read off the tag by the repair: a word of the + # conjunction vocabulary is not tagged one where it is written + # initial-shaped, and initial-shaped means one CAPITAL letter + # (nameparser/_pipeline/_classify.py, and the tests beside it -- + # the repair no longer asks). Uppercase a name and # every one-letter conjunction becomes an initial; lowercase one # and a middle initial `E` becomes the Italian conjunction. So # the property is pinned over names carrying no single-letter @@ -262,3 +268,77 @@ def test_a_one_letter_conjunction_is_case_sensitive_to_repair(self) -> None: uppered.capitalize() # 'Y' is initial-shaped, so the conjunction rule declines it self.m(str(uppered), 'Juan Y Garcia', uppered) + + # The v1 parity this rests on, at the surface v1 users have. An + # ASSIGNED field is spliced in as raw text and never classified, so + # there is no tag to read and repair asks the vocabulary -- which is + # what v1 always did, everywhere. Measured on the released 1.4.0 + # wheel and on 2.1.0, all four: 'John Velasquez y Garcia', + # 'John Smith y Jones', 'John E. Smith', 'John Smith-y'. Reading the + # tag alone (no fallback) capitalizes every one of these conjunctions + # -- the regression #458's review caught before it shipped. + def test_an_assigned_field_keeps_v1_conjunction_repair(self) -> None: + for field, value, want in ( + ('last', 'velasquez y garcia', 'John Velasquez y Garcia'), + ('last', 'smith y jones', 'John Smith y Jones'), + # the initial carve-out: an assigned middle initial is + # an initial, not the Italian conjunction + ('middle', 'e.', 'John E. Smith'), + # v1 asks per WORD of the assigned text, so the + # conjunction inside a hyphenated word IS lowered here + # -- the opposite of the parsed reading pinned below, + # and the difference is that one carries a reading + ('last', 'smith-y', 'John Smith-y')): + hn = HumanName('john smith') + setattr(hn, field, value) + hn.capitalize(force=True) + self.m(str(hn), want, hn) + + # The other half of #458, and the half that MOVED: the decision is + # the whole token's, so a conjunction word inside a longer token is + # not one. `e` is the Italian conjunction and `e-f` is a middle + # name; before #458 the repair re-ran the test over each word of a + # token's text and gave 'Juan e-F Smith'. The uppercase spelling is + # here to show the old answer was not even self-consistent: it read + # `E` as initial-shaped and capitalized, so the same name repaired + # to two different strings depending on how it was written. + def test_a_conjunction_inside_a_longer_token_is_a_name_word(self) -> None: + lowered = HumanName('juan e-f smith') + lowered.capitalize(force=True) + self.m(str(lowered), 'Juan E-F Smith', lowered) + uppered = HumanName('JUAN E-F SMITH') + uppered.capitalize() + self.m(str(uppered), 'Juan E-F Smith', uppered) + + # The third producer of never-classified text, and the one that is + # not an assignment: __getstate__ pickles the *_list STRINGS and + # nothing else (mechanisms.md#FACADE-CONTRACT -- components come + # back exactly as pickled, never re-parsed), so a load rebuilds + # tokens with no tags on them. They are marked UNCLASSIFIED_TAG for + # the same reason an assigned field is: reading the absent + # conjunction tag as "not a conjunction" would repair a restored + # 'juan ortega y gasset' to 'Juan Ortega Y Gasset', which is + # neither 1.4.0's answer nor the same object's before pickling. + def test_a_restored_pickle_keeps_v1_conjunction_repair(self) -> None: + for text, want in (('juan ortega y gasset', 'Juan Ortega y Gasset'), + ('john de la vega', 'John de la Vega'), + ('juan y garcia', 'Juan y Garcia')): + restored = pickle.loads(pickle.dumps(HumanName(text))) + restored.capitalize(force=True) + self.m(str(restored), want, restored) + + # The ONE name a pickle round trip does change, pinned so it is not + # rediscovered as a bug. #458 moved the conjunction-versus-initial + # decision into the parse, and a pickle carries no tags, so the + # restored name is repaired the way 1.4.0 repaired everything -- + # per word of the text, giving the Italian conjunction inside a + # hyphenated middle name. It is the pickle contract (strings only, + # never a re-parse) meeting the tag read, not a defect in either. + # 1.4.0 gave 'Juan e-F Smith' both ways. + def test_a_pickle_round_trip_loses_the_e_f_reading(self) -> None: + direct = HumanName('juan e-f smith') + direct.capitalize(force=True) + self.m(str(direct), 'Juan E-F Smith', direct) + restored = pickle.loads(pickle.dumps(HumanName('juan e-f smith'))) + restored.capitalize(force=True) + self.m(str(restored), 'Juan e-F Smith', restored) diff --git a/tests/v2/test_regex_sync.py b/tests/v2/test_regex_sync.py index 61cd4a24..cc42bfa2 100644 --- a/tests/v2/test_regex_sync.py +++ b/tests/v2/test_regex_sync.py @@ -81,6 +81,14 @@ def test_initial_copies_agree_with_each_other_and_config() -> None: # documented relationship to the config source, so a future edit to # either side that breaks the relationship fails loudly here # instead of silently drifting. + # The two copies stopped answering the same question in #458 and + # the agreement matters MORE for it, not less: _vocab's asks what + # the parse decided about a token, _render's what it would have + # decided about text spliced into a field afterwards, and the + # second is only right while it answers as the first would. This + # test is what says so. (#458 briefly deleted _render's copy, when + # nothing read it; the unparsed-text fallback gave it a reader + # again -- see _render._reads_as_conjunction.) assert _vocab._INITIAL.pattern == _render._INITIAL.pattern source = _config.REGEXES["initial"] # config's pattern is the pipeline copy with an extra "?" spliced in diff --git a/tests/v2/test_render.py b/tests/v2/test_render.py index 224df8cd..32f312e0 100644 --- a/tests/v2/test_render.py +++ b/tests/v2/test_render.py @@ -1,8 +1,10 @@ import pytest +from nameparser import HumanName, Parser, Policy, parse from nameparser._lexicon import Lexicon from nameparser._render import _collapse, render -from nameparser._types import Ambiguity, AmbiguityKind, ParsedName, Role, Span, Token +from nameparser._types import (UNCLASSIFIED_TAG, UNJOINED_TAG, Ambiguity, + AmbiguityKind, ParsedName, Role, Span, Token) def test_collapse_is_the_254_algorithm() -> None: @@ -124,6 +126,82 @@ def test_initials_skips_tagged_particles_outside_given() -> None: assert pn.initials("{family}") == "S." +def test_repair_keeps_a_conjunction_lowercase_in_a_particle_part() -> None: + """rules.md#R4's conjunction carve-out, which no shipped name reaches. + + The unjoined mark makes the words of an all-particle part + ordinary name words, since none of them is acting as a particle + there -- but repair's conjunction conjunct is deliberately UNGATED + on that mark, R4 keeping a conjunction lowercase "even inside such + a part". Witnessing it needs a word in `particles` AND + `conjunctions`; the shipped sets are disjoint, in the default + vocabulary and in every locale pack, so no input string can + witness it and rules.md can carry no example line for it. Gating + the conjunct on the mark passed the entire suite until this test + existed. + + Only the FORCED call gets here: R5's gate returns a mixed-case + name before any of this is consulted. + """ + # mechanisms.md#VOCABULARY-OVERLAP-AS-PRECONDITION: "assert the + # intersection as a precondition" -- built here rather than found, + # so what has to hold is the half not being added. + assert "y" in Lexicon.default().conjunctions, ( + "'y' left the default conjunctions; this test builds the " + "particle/conjunction overlap it needs from the other side and " + "no longer has one. Pick another shipped conjunction.") + lex = Lexicon.default().add(particles={"y"}) + p = Parser(lexicon=lex) + + van = p.parse("Anh y Van") + tags = [t.tags for t in van.tokens if t.text == "y"][0] + assert {"conjunction", "particle", UNJOINED_TAG} <= tags, sorted(tags) + assert van.capitalized(lex, force=True).family == "y Van" + + +def test_initials_readmits_a_conjunction_in_a_particle_part() -> None: + """Today's answer on an OPEN question (#461), pinned as such. + + rules.md#R3 says a conjunction never initials "even then" -- even + inside the all-particle part R2 turns into ordinary name words -- + and `initials()` does not do that: the mark readmits the part's + words whichever skip tag they carry. #461 made the code match the + clause and was backed out, the clause rather than the code being + what is now in question (decisions.md, under R2). + + So this pins what a `deviates:` marker would pin if one could + hang here -- TODAY's output, strictly, so that settling #461 + fails the suite until this moves with it. It cannot be a marker: + markers hang on rules.md example lines and every line there names + an input string parsed with the DEFAULT vocabulary, over which + `particles` and `conjunctions` are disjoint and no string reaches + this shape. It is also what keeps the values quoted in prose by + decisions.md, mechanisms.md#RENDER-HONORS-THE-PARSE and + `_render.py` from going stale unnoticed. + """ + assert "y" in Lexicon.default().conjunctions, ( + "'y' left the default conjunctions; this test builds the " + "particle/conjunction overlap it needs from the other side and " + "no longer has one. Pick another shipped conjunction.") + p = Parser(lexicon=Lexicon.default().add(particles={"y"})) + + # the part the mark has turned into name words: every word of it + # initials, the conjunction included, and the base agrees + de_y = p.parse("Juan de y") + assert de_y.family_base == "de y" + assert de_y.initials() == "J. d. y." + assert p.parse("Anh y Van").initials() == "A. y. V." + assert p.parse("johnny y").initials() == "j. y." + + # and OUTSIDE such a part the skip stands, joining or not -- + # these are what the readmission must not reach + assert p.parse("Juan Velasquez y Garcia").initials() == "J. V. G." + assert p.parse("Juan y Garcia").initials() == "J. G." + # including under the default vocabulary, where 'y' is no particle + # and the family is therefore not all-particle + assert parse("Juan de y").initials() == "J." + + def test_initials_custom_delimiter_and_separator() -> None: assert _bobdole().initials(delimiter="", separator="") == "B A D" @@ -211,11 +289,12 @@ def test_capitalized_with_explicit_lexicon() -> None: assert out.suffix == "Phd" -def test_capitalized_lowers_conjunctions_but_not_initial_shapes() -> None: - # v1's is_conjunction excludes initial-shaped words: a lowercase - # 'y' lowers, but an uppercase 'Y' looks like an initial and - # capitalizes ('JOSE ORTEGA Y GASSET' -> 'Jose Ortega Y Gasset', - # pinned live against v1.4 2026-07-17) +def test_capitalized_lowers_the_words_the_parse_tagged_conjunction() -> None: + # #458: whether a word is the conjunction or an initial is + # classify's decision, recorded as the tag; repair honors the tag + # and never asks the word. Both halves of that are asserted here, + # since a repair that lowered every conjunction-vocabulary word + # would pass the first alone. # 01234567890123456789 pn = _pn("juan ortega y gasset", [ Token("juan", Span(0, 4), Role.GIVEN), @@ -225,13 +304,162 @@ def test_capitalized_lowers_conjunctions_but_not_initial_shapes() -> None: ]) out = pn.capitalized(force=True) assert out.family == "Ortega y Gasset" + # v1's is_conjunction excludes initial-shaped words, so classify + # withholds the tag from an uppercase 'Y' and repair capitalizes it + # ('JOSE ORTEGA Y GASSET' -> 'Jose Ortega Y Gasset', pinned live + # against v1.4 2026-07-17 and pinned end to end in + # tests/test_capitalization.py). These tokens are what a parse of + # the uppercase name builds. upper = _pn("JUAN ORTEGA Y GASSET", [ Token("JUAN", Span(0, 4), Role.GIVEN), Token("ORTEGA", Span(5, 11), Role.FAMILY), - Token("Y", Span(12, 13), Role.FAMILY, frozenset({"conjunction"})), + Token("Y", Span(12, 13), Role.FAMILY), Token("GASSET", Span(14, 20), Role.FAMILY), ]) assert upper.capitalized(force=True).family == "Ortega Y Gasset" + # The tag decides even against the shape: a token tagged + # conjunction lowers however it is written. Nothing shipped builds + # this token -- that is the point, since the old predicate could + # not have honored it. + tagged = _pn("JUAN ORTEGA Y GASSET", [ + Token("JUAN", Span(0, 4), Role.GIVEN), + Token("ORTEGA", Span(5, 11), Role.FAMILY), + Token("Y", Span(12, 13), Role.FAMILY, frozenset({"conjunction"})), + Token("GASSET", Span(14, 20), Role.FAMILY), + ]) + assert tagged.capitalized(force=True).family == "Ortega y Gasset" + # ... and an untagged word of the conjunction vocabulary is an + # ordinary name word. The reachable shape is a token whose text is + # more than one word, since the repair walks a token's words while + # the tag is the whole token's: 'juan e-f smith' capitalized to + # 'Juan e-F Smith' while the old predicate re-decided per word. + hyphenated = _pn("juan e-f smith", [ + Token("juan", Span(0, 4), Role.GIVEN), + Token("e-f", Span(5, 8), Role.MIDDLE), + Token("smith", Span(9, 14), Role.FAMILY), + ]) + assert hyphenated.capitalized(force=True).middle == "E-F" + + +def test_case_repair_falls_back_for_text_the_parse_never_read() -> None: + """A token carrying UNCLASSIFIED_TAG holds raw text no parse read, + so there is no decision to honor and case repair -- which is handed + a lexicon -- asks the vocabulary instead, getting the answer the + parser would have given, v1's initial carve-out included. + + ONE view falls back. `initials()` takes no lexicon, so it has none + to ask; the sibling test below pins what that costs. + """ + p = Parser() + base = p.parse("john smith") + + spliced = base.replace(family="velasquez y garcia") + assert all(UNCLASSIFIED_TAG in t.tags for t in spliced.tokens[1:]) + assert spliced.capitalized(force=True).family == "Velasquez y Garcia" + # the carve-out rides along: an assigned middle initial is an + # initial, not the Italian conjunction + assert base.replace(middle="e.").capitalized(force=True).middle == "E." + + # the per-part particle question is NOT asked: re-deriving it needs + # a reading on every word of the part and these words have none, so + # it falls through to plain particle treatment + assert base.replace(family="de la").capitalized(force=True).family == "de la" + assert (base.replace(family="de la vega") + .capitalized(force=True).family == "de la Vega") + # ... and revise(), which classifies, crosses it + assert p.revise(base, family="de la").capitalized(force=True).family == "De La" + assert (p.revise(base, family="velasquez y garcia") + .capitalized(force=True).family == "Velasquez y Garcia") + + # a token the parse DID see is decided by its tags, so a hyphenated + # word it read as one ordinary name word stays one + assert p.parse("juan e-f smith").capitalized(force=True).middle == "E-F" + + +def test_the_mark_and_not_the_span_says_a_token_was_never_read() -> None: + """`span is None` means SYNTHETIC, which is a wider set than + unclassified, and keying the fallback on it was a measured + regression (#463 review). + + `Parser.revise()` builds span-less tokens too, from a full + sub-parse whose tags it keeps on purpose, and its docstring + promises the tag-driven views "behave as if the text had been + parsed". Under the span discriminator the fallback overrode exactly + those tags: `revise(middle='e-f')` repaired to 'e-F' where the same + words parsed gave 'E-F'. A HAND-BUILT span-less token is not marked + either, so it takes the tag path like every other token in the + library -- tag-driven semantics, not span-driven. + """ + p = Parser() + base = p.parse("john smith") + + revised = p.revise(base, middle="e-f") + assert [t.span for t in revised.tokens if t.role is Role.MIDDLE] == [None] + assert UNCLASSIFIED_TAG not in revised.tokens_for(Role.MIDDLE)[0].tags + # both sides of the pair the span discriminator split + assert p.parse("john e-f smith").capitalized(force=True).middle == "E-F" + assert revised.capitalized(force=True).middle == "E-F" + + # the initials half of the same regression, which needs a lexicon + # holding no particles at all to witness -- under the default one + # 'de la vega' is particle vocabulary and the parse skips it too. + # segment_scripts off because a from-scratch Lexicon covers no + # script the default policy activates, and the warning is an error + # under this suite's filters. + empty = Parser(lexicon=Lexicon(), + policy=Policy(segment_scripts=frozenset())) + assert (empty.parse("john de la vega").initials() + == empty.revise(empty.parse("john smith"), + family="de la vega").initials() + == "j. d. l. v.") + + # hand-built, span-less, untagged: classified by default + handbuilt = _pn("john de la vega", [ + Token("john", None, Role.GIVEN), + Token("de", None, Role.FAMILY), + Token("la", None, Role.FAMILY), + Token("vega", None, Role.FAMILY), + ]) + assert handbuilt.initials() == "j. d. l. v." + assert handbuilt.capitalized(force=True).family == "de la Vega" + # ... and the same tokens MARKED take the fallback, which is the + # only thing that moves them + marked = _pn("john de la vega", [ + Token("john", None, Role.GIVEN), + Token("de", None, Role.FAMILY), + Token("la", None, Role.FAMILY), + Token("y", None, Role.FAMILY, frozenset({UNCLASSIFIED_TAG})), + Token("vega", None, Role.FAMILY), + ]) + assert marked.capitalized(force=True).family == "de la y Vega" + + +def test_initials_has_no_lexicon_so_a_spliced_field_is_all_name_words()\ + -> None: + """The accepted cost of `initials()` taking no lexicon: a field + spliced in as raw text has no reading, and this view has nothing to + read one from, so every word of it initials. + + That disagrees with the facade and with the same name parsed, and + it is a 2.0-core defect rather than a decision -- see #464, filed + for giving `Parser` an `initials` crossing. A fallback to + `Lexicon.default()` was written and dropped: it guesses a + vocabulary, and under a caller's own the guess erases a whole + field (`Lexicon.default().add(particles={'y'})`, family 'de y'). + """ + p = Parser() + base = p.parse("john smith") + + assert base.replace(family="de la vega").initials() == "j. d. l. v." + assert p.parse("john de la vega").initials() == "j. v." + assert HumanName("john de la vega").initials() == "j. v." + + # the vocabulary a fallback would have had to guess, and the field + # it erased when it guessed wrong + y_lex = Lexicon.default().add(particles={"y"}) + py = Parser(lexicon=y_lex) + assert py.parse("Juan de y").initials() == "J. d. y." + assert py.parse("Juan Perez").replace(family="de y").initials() == "J. d. y." def test_capitalized_rebuilds_ambiguity_tokens() -> None: