fix(phi-scan): scan the tracked corpus, reconcile it against git, and add the HL7 v2 structured pass (PHI-SCAN-WALK-ROOT-SCOPE) - #35
Merged
Conversation
…plus an HL7 v2 structured pass (PHI-SCAN-WALK-ROOT-SCOPE) Both enumerating routes covered `test/fixtures/` + `src/` only: 31 of 102 tracked files. 71 were read by NEITHER route, 27 of those under `test/`, and 8 of those carried inline HL7 `PID|` literals with names, DOBs and MRNs. THE SHARPEST HALF IS THIS REPO'S OWN: `test/fixtures/` HAS NEVER EXISTED ON ANY COMMIT (`git log --all` is empty for it). The walk's `existsSync` guard returned on its first line for that root on every run this scanner has ever made, and every one of those runs printed "OK: no hits" and exited 0. A count does not detect that (it counts the roots that DID exist) and neither does an existence check (an EMPTIED root opens nothing), so the remedy is reconciliation: the walk now covers every tracked directory plus repo-root files, and `reconcileWithGit` refuses (exit 2), naming every tracked path the walk did not open. A non-directory root refuses before the walk too, because `existsSync` FOLLOWS: a dangling root read false and printed clean over an off-disk corpus, and a symlinked root was followed. Exit code for a regular-file root is 2 HERE, derived from this script's own contract (`ENOTDIR` into `walk`'s catch), not ported from a sibling. WIDENING IS TWO-SIDED. Enumerating buys the SSN/email floor and nothing else, and measured over the 8 `PID|` files the floor finds ZERO: no dashed SSN, no email. So an HL7 v2 structured pass ships with it, in addition to the floor and never instead: PID/NK1/GT1/IN1 by field and component, checking names, DOB, MRN/member-id/SSN, address and phone against the allow-list. It finds segment literals INLINE, because this package ships no standalone `.hl7` file at all: every message in its corpus is a `.ts` string literal. THE GRID, PROVED RATHER THAN ARGUED: 169 base-vs-head cells over 15 paths, 4 payloads and all 3 routes. 37 base `1` cells still `1`, 74 cells `0 -> 1`, and ONE `1 -> 0`. THAT ONE CELL, DISCLOSED: `phi-scan package.json` exited 1 on the npm publisher mailbox in its `author` field and now exits 0. Declared with the new `EMAIL` tag, which takes a path AND an address, so the same address elsewhere and any other address there both still report. An allow-list entry is still ROUTE-BLIND and that reach is pinned from both directions by tests. 69 files hand-read. Nothing patient-identifying. The fixture placeholders are NAMED in `scripts/phi-allow-list.txt` rather than scrubbed: `Jane Q. Public`, `Jane Doe`, the suite mnemonics, `MRN1`/`MRN2`/`MRN12345`, an SSN-shaped `999887777` in area 999 (never issued), placeholder streets, and two numbers in the reserved 555 range. The one org-traceable string is the publisher mailbox. The scanner's own test file is now inside the corpus, so its violator payloads are assembled from parts at runtime. Allow-listing them would blind the floor globally and route-blind; exempting the file would leave the largest violator-bearing file unscanned. Also: the walk's regular-`*.md` skip is removed (purely additive, since the `<path>` route always read those bytes, so the two routes disagreed); `.git` is a regular FILE in a submodule and is skipped by literal name; the all-route exemption list is two literal paths (the vendored gzip tarballs, whose bytes are not the text they carry) and never reaches a blocking route. Four residuals stay disclosed, not closed, including that the reconciliation compares path SETS and not the bytes git carries at them, and that the enumerate-then-read race's precondition has now happened (`.changeset/` is in scope; it exits 2, not clean).
…IN1-17 and the published figures (PHI-SCAN-WALK-ROOT-SCOPE) Pass 1 REFUTED with three INTRODUCED findings. Two of them were reachable silent misses reported CLEAN, so they are FIXED rather than disclosed; the third was a wrong spec clause. Every figure is re-derived after the changeset existed, which is what made the first set off by one. 1. A NAME COMPONENT OUTSIDE ASCII reported clean. `looksLikeNameToken` was `[A-Za-z]`, so an unaccented name hit while the same name with its accent exited 0, as did every name in a non-Latin script: a gate blind to exactly the names least likely to be synthetic. It is a Unicode letter class now, which still excludes digits so a coded value stays out. Purely additive. 2. A WHOLE MESSAGE IN ONE LITERAL WITH ESCAPED SEPARATORS was never located, because the character before `PID|` is the letter `r` of the escape and the boundary class rejected it. Measured before: an ADT carrying a name, a DOB, an MRN, an address and two phones scanned clean at exit 0, while the same message one segment per array element produced 8 hits. That is the other way a `.ts` file carries a message here, and the shape `parseHL7(raw)` consumes. The escaped separator is now a boundary AND a terminator, and the terminator half is load-bearing: without it the message is read as ONE segment and the next-of-kin's relationship code is reported as the patient's PID-11 address. 3. IN1-17 WAS MAPPED AS A TELEPHONE FIELD. HL7 v2.5.1 defines it as Insured's Relationship To Patient, so a SNOMED relationship code was reported as a phone number and the remedy it steered a developer toward was a global `PHONE` clearance of that digit string. IN1 carries no insured telephone at all (IN1-7 is the payer's), so IN1 is now absent from `PHONE_FIELDS` deliberately. Every field number now cites its v2.5.1 chapter and clause, because an uncited table is what produced this one. 4. THE BANNER'S "STILL NOT DETECTED" LIST WAS ASSERTED EXHAUSTIVE AND WAS NOT, and two other files defer to it as the authoritative statement of limits. It now also names the non-default component separator, the one-character name component, the unmapped PID fields, and binary or compressed targets. 5. THE PUBLISHED FIGURES WERE OFF BY ONE, self-referentially: writing the changeset added a tracked file the changeset then counted. Both denominators are now stated. Base `daf75c3`: 102 tracked / 31 opened / 71 in neither / 27 of those under `test/`. Head: 103 tracked / 101 opened / 2 in neither / 0 under `test/`. 70 newly opened, of which 69 existed at base and were hand- read. The removed `*.md` walk skip dropped 14 files at base, not 16. Also: `refuseNonDirectoryRoots` swallowed every `lstat` error as "absent" and now excuses only `ENOENT`; the reconciliation's remedy text steered toward widening the exemption list for a deleted-but-tracked file and now names that cause first; and the untracked-outside-every-root residual the refuter named is disclosed AND pinned by a test that asserts the gap, so closing it later reds that case rather than letting the disclosure outlive the defect. THE GRID WAS RE-RUN AGAINST THE CORRECTED HEAD: 37 base `1` cells still `1`, 74 cells `0 -> 1`, and the same single disclosed `1 -> 0` on the publisher mailbox. The two recogniser widenings added detections and subtracted none. The new escaped-separator case is the one test whose payload is itself a live segment literal, so every value in it is assembled at runtime. The gate caught it as a literal first, which is the evidence the widening works.
… last two figures (PHI-SCAN-WALK-ROOT-SCOPE) Pass 2 REFUTED. Every finding was about the accuracy of a claim rather than the behaviour of the gate, and the major one is the same shape twice. 1. THE EXHAUSTIVE NEGATIVE LIST WAS MEASURED INCOMPLETE A SECOND TIME, AFTER BEING EXTENDED IN ANSWER TO THE FIRST PASS. Seven PHI-bearing v2.5.1 fields reported clean while the list called itself the authoritative statement of this gate's limits: NK1-26, NK1-31, NK1-32, NK1-37, GT1-2, GT1-4 and IN1-49. Two of those matter beyond the count. NK1-37 is a social security number field, and an undashed one there is invisible to the floor as well, so both passes miss it at once. IN1-49 is the member id, while three files told a reader the pass covers "member id". A LIST OF THAT SHAPE CANNOT BE KEPT TRUE: every clause of every segment of the standard would have to appear on it, and two passes proved it. So the claim is inverted. The banner now enumerates EXACTLY the fields that are read and says anything not named is not checked. That is checkable against the code; the other shape never was. `phi-scan-overrides.md` and `ci.yml`, which defer to the banner, drop their unqualified "member id" wording and point at the named set instead. NEVER RESTORE THE NEGATIVE SHAPE. The table is NOT grown in answer to this. Growing it is what produced the IN1-17 defect: more uncited field numbers from the same memory. 2. PROVENANCE, WHOSE ABSENCE WAS THE ROOT CAUSE AND IS NOW SAID PLAINLY. The field numbers are asserted from HL7 v2.5.1 and were cross-corroborated IN-REPO ONLY, against `src/messages/related-person.ts` and the vendored `@cosyte/hl7` type surface. They were NOT checked against a published copy of the standard, and one of them was wrong on the way here. That is why the table is deliberately narrow and why widening it means citing a source. 3. A FOURTH RECOGNISER LIMIT, DISCLOSED RATHER THAN GUESSED AT: a literal backslash followed by `r` or `n` inside a field value ends the segment early, because the escaped separator is also the terminator. Measured, a Windows path in PID-11 truncates there and PID-13/14 go unread. It can only SHORTEN a segment, never renumber one, and the field it cuts in still reports. Not decidable from static text. 4. THE `*.md` SKIP FIGURE WAS WRONG IN BOTH DIRECTIONS AND IS NOW ZERO. Under the OLD roots the skip dropped no files at all: `src/` holds no markdown and `test/fixtures/` never existed, so it was dead code. Under the widened roots it would drop 15, which is the only reason removing it matters. 16 and 14 were both counts of something else. 5. THE GRID TOTALS NOW FACTOR AND THE PUBLISHED FIGURE DID NOT. It is 14 paths x 4 payloads x 3 routes = 168 cells: 37 base `1` still `1`, 74 `0 -> 1`, 1 `1 -> 0`, and 56 unchanged zeros, which sums to 168. "169 over 15 paths" factored to 180 and left 57 cells unaccounted for. It is recorded as a MEASUREMENT, not as a fixture: nothing pins the base half, because a head-only test structurally cannot, and saying otherwise was the overclaim. The new suite case asserts the coverage boundary from BOTH sides in one run: eight fields inside the named set must report, and eight outside it must be clean, so each zero is a declared gap rather than a detector failure. A field added to the table without being added to the banner reds there. The grid was re-run against this head: 37 / 74 / 1 / 56, unchanged, and the same single disclosed `1 -> 0` on the publisher mailbox.
…pin all 28 named fields (PHI-SCAN-WALK-ROOT-SCOPE) Pass 3 NOT REFUTED, with four minor claim-accuracy corrections to land without another round (the ADR 0016 cap is spent) and one stale citation. All four were in the flattering direction, inside the commit whose purpose was to fix claim-accuracy defects, and one of them was the same figure wrong a third time. 1. THE `*.md` SKIP FIGURE IS 11, NOT 15, AND THAT IS THE FOURTH VALUE PUBLISHED FOR IT. Measured by restoring the skip byte for byte and running the scanner: 11 tracked files are opened by removing it. The other four tracked markdown files sit at the repository ROOT and are enumerated by `walkTopLevel`, a function that never carried the skip, so they were never the skip's to drop. 16, 14 and 15 each counted something else (tracked markdown at head, at base, at head again). Count what the SKIP drops, not what matches `*.md`. 2. THE PROVENANCE SENTENCE OVERCLAIMED ITS OWN CORROBORATION, so it is now field by field. 13 of the 28 numbers are corroborated in-repo (PID-3/5/7/11/13/14, NK1-2/4/5/6/16, IN1-16/36). 15 are corroborated by nothing here at all, and that is where the residual risk sits: the WHOLE GT1 row, which no in-repo source mentions even once, plus PID-6/9/19/20, NK1-30/33 and IN1-18/19. Saying "cross-corroborated in-repo" without naming which concealed exactly the numbers most likely to be the next IN1-17. 3. THE TRUNCATION DISCLOSURE CARRIED A FALSE SUB-CLAUSE. "The field it cuts in still reports" holds only when the surviving prefix clears a recogniser floor. Measured: a cut inside a family name silences that field AND everything after it, leaving only the fields before the cut. The "can only shorten, never renumber" half is true and stays. A second probe pins the corrected half. 4. THE NEW CASE'S DURABILITY CLAIM WAS FALSE, so the claim is made TRUE rather than narrowed. It named eight fields and said a field added to the table without being added to the banner would red there; adding two left the suite green. The NARROWING direction was worse: 15 of the 28 named fields fired in no test at all, so the code dropping a field while the banner kept promising it also shipped green. The case now enumerates ALL 28 positives beside the ten declared gaps, in one run, so both directions red. 5. A stale citation: the field-table header cited PV1 Chapter 3 beside segments PV1 does not appear in, while the banner says PV1 is not read at all. No behaviour changed in this commit. The gate reads exactly what the banner says it reads, in both directions, verified mechanically: named-not-read is empty and read-not-named is empty, 28 each.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this closes
PHI-SCAN-WALK-ROOT-SCOPE, thetransformleg. Both of the PHI gate's enumerating routes coveredtest/fixtures/andsrc/only, which was 31 of 102 tracked files. 71 were read by NEITHER route, 27 of them undertest/, and 8 of those carried inline HL7PID|literals with names, DOBs and MRNs.The sharpest half is this repository's own, not a sibling's:
test/fixtures/HAS NEVER EXISTED ON ANY COMMIT.git log --all -- 'test/fixtures*'is empty. The walk'sexistsSyncguard returned on its first line for that root on every run this scanner has ever made, and every one of those runs printedOK: no hitsand exited 0.The two halves, each in addition to the other
Enumeration. The walk now covers every tracked directory plus the repo-root regular files, and
reconcileWithGitcompares what it actually OPENED againstgit ls-files, refusing (exit 2) and naming every tracked path it did not open. A count cannot detect the failure this closes (it counts the roots that DID exist) and neither can an existence check (an EMPTIED root opens nothing). A declared root that is not a directory refuses before the walk:existsSyncFOLLOWS, so a dangling root read false and printed clean over an off-disk corpus, and a symlinked root was followed.Recognition. Enumerating more files buys the SSN/email floor and nothing else, and measured over the 8
PID|files the floor finds ZERO: no dashed SSN, no email. So an HL7 v2 structured pass ships with it, on all three routes, over a named set of PID / NK1 / GT1 / IN1 fields. It finds segment literals inline, because this package ships no standalone.hl7file at all: every message in its corpus is a.tsstring literal.Figures, both denominators
test/daf75c3Both remaining files are declared literal-path exemptions (the vendored gzip tarballs). 70 files newly opened, of which 69 existed at base and were hand-read. A regular-file walk root exits 2 here, derived from this script's own contract, not ported.
The grid
14 paths x 4 payloads x 3 routes = 168 cells, each run against the base scanner and the head scanner: 37 base
1still1, 740 -> 1, 56 unchanged zeros, and ONE1 -> 0.That one cell, disclosed rather than found later:
pnpm phi-scan package.jsonexited 1 on the npm publisher mailbox in itsauthorfield and now exits 0. It is declared with anEMAILallow-list entry taking a path AND an address, the narrowest instrument here: the same address in any other file still reports, and any other address in that file still reports. An allow-list entry is still route-blind and that reach is pinned from both directions by tests. The alternatives were a bigger hole (exempting the whole file) or an unusable gate (every commit touchingpackage.jsonblocked).What was found in the corpus
Nothing patient-identifying, in any of the 69 hand-read files. The fixture placeholders are named in
scripts/phi-allow-list.txtrather than scrubbed, because scrubbing destroys the evidence the audit happened:Jane Q. Public,Jane Doe, the suite mnemonics,MRN1/MRN2/MRN12345, an SSN-shaped999887777in area 999 which the SSA has never issued, placeholder streets, and two numbers in the reserved 555 range. The one org-traceable string is the publisher mailbox above, which is public and non-patient.The gate
conformance-refuter, three passes (the ADR 0016 cap), each on a frozen sha.9c6bb95): the limits banner was asserted exhaustive and missed two reachable silent misses;IN1-17was mapped as a telephone field when v2.5.1 defines it as Insured's Relationship To Patient; three figures were off by one. Both silent misses were fixed rather than disclosed (a name component outside ASCII, and a whole message in one literal with escaped\rseparators, each reported CLEAN before).689931e): the extended negative list was measured incomplete again, seven more fields. A list of that shape cannot be kept true, so the claim was inverted: the banner now enumerates exactly which fields are read and says anything not named is not checked. Provenance recorded, grid totals corrected to factor.e55adef), with four minor corrections landed here without another round, as that pass directed: the*.mdskip figure is 11; the provenance sentence named which 13 of 28 fields are corroborated in-repo and which 15 are not (the whole GT1 row); the truncation disclosure's "the field it cuts in still reports" was false and is corrected; and the coverage case now enumerates all 28 named fields rather than eight, so both the widening and the narrowing direction red.Residuals, disclosed and not closed
The reconciliation compares path sets, not the bytes git carries at them (no repo in this ecosystem has closed that; widening makes it narrower, since a decoy must now mirror 101 names rather than 7). It is vacuous on an empty index. The
--stagedroute's prefix scope still misses a scan root's own path staged as a non-regular entry. An empty enumeration still reports clean. Untracked content under an undeclared top-level directory is invisible to both enumerating routes, and that gap is pinned by a test that asserts it, so closing it later reds rather than letting the disclosure outlive the defect. The enumerate-then-read race's precondition has now happened:.changeset/is in scope and a release step creates and deletes files there; it exits 2, not clean.Verify
scripts/verify.sh transformgreen.ran:typecheck, lint, format:check, phi-scan, check:no-emdash, check:no-internal-refs, test:coverage, build, attw, pluspnpm audit --prodand the licence allowlist. 26 test files, 427 tests.check:agent-notesis a gate this ladder does not know, so it was run by hand and is green.