fix(summary): keep the summary when one cross-section reading fails

CHANGELOG.md

@@ -4,6 +4,10 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n
 
 ## Unreleased
 
+- The Rust `og` binary now writes the whole summary.json: `og summary collect` without `--only` reads all 48 sections over one walk of the sample folder, then runs the post pass. The post pass joins sections: the repeat readings with the traits and CPIC states, Locityper's primary calls with HLA, pypgx and Cyrius, and CNVpytor's chrY filter with the declared sex. It also builds the CYP2D6 block and writes `not_run`, `not_assessed`, `recheck` and `new_since`. Without `--out` it writes `summary.json` in the sample folder, as the Python did; `--only` still needs `--out`. Nothing calls it yet: the reports still read the summary that `bin/collect_summary.py` writes. The whole file is the same as the Python's on 8 of 9 new parity cases, compared as JSON with key order and only `generated_utc` masked; the ninth holds the deliberate fix below. On a synthetic folder with every section it takes 0.01 to 0.03 s and 15 MB, where the Python took 0.15 to 0.21 s and 42 to 44 MB. With a 1,000,000-record VCF it takes 0.44 to 0.62 s and 17 to 20 MB, where the Python took 1.5 s and 42 to 44 MB. What differs from the Python:
+  - The post pass keeps the summary. The Python ran it outside its per-section error handling, so one row of an unexpected shape there lost summary.json and both reports. The Rust parts read with defaults and have no such error to raise; a part that returns an error makes only its own section unreadable, with a note that names the file.
+  - A section reader that stops on an error the Python did not catch keeps the summary too. A telomere table with an infinite read length stopped the Python with an OverflowError and no summary.json; og marks only the telomere section unreadable.
+
 - Steps 43 (archaic ancestry) and 44 (ancient similarity) now run in the Rust `og` binary: `og archaic table`, `strict`, `genotypes` and `report`, and `og ancient select`, `lift`, `person-geno`, `align` and `report`. They replace `bin/archaic_report.py` and `bin/ancient_report.py`, which are gone, in the step scripts, the Nextflow modules, `setup.sh --archaic` and `setup.sh --ancient-panel`. og is static, so each part still runs in the image it ran in. `og ancient lift` reads UCSC's chain file with a new chain reader in `og-intervals`. The outputs are the same, byte for byte (the gzip genotype files by content), on the 49 parity cases, except where a line below says otherwise. No option, output file or format changes. On the real archaic table, `og archaic table` merges chromosome 1 (4.1 million sites) in 18 s and 190 MB where the Python took 76 s and 1.25 GB; bcftools and Picard still take most of each chromosome's setup. On a synthetic input of real size, step 44's panel column over 1,150,106 sites and 819 people takes 2.5 s and 342 MB instead of 4.2 s and 524 MB. What changes on purpose:
   - The archaic table's setup stamp now holds a format id (`ARCHAIC_TABLE_FORMAT` in `setup.sh`) instead of the sha256 of the Python's table code. A table the last Python built has the same bytes (chromosomes 1 to 6 and 21 of the real table, rebuilt from the same public files, are the same after decompression), so its stamp is taken as current once and rewritten: no install downloads the 150 GB of archaic genomes again for this change.
   - `og archaic genotypes` streams the sorted VCF and table together with one duplicate-position window, and writes mask runs as they close. On a synthetic chromosome with 3 million table rows and 1 million VCF records, peak memory falls from the initial port's 166 MB to 4 MB (3.79 s before, 3.91 s after). A position that goes backwards within a VCF chromosome or in the table stops with exit 2, also when a skipped record (an indel, ALT `.`) of another chromosome sits between the two. Duplicate calls, clashes, masked sites, mask runs and span counts keep their meaning.

docs/rust.md

@@ -21,7 +21,7 @@ rust/
   crates/og-qc/         og qc sex-loss: mosaic loss of Y or X for step 33
   crates/og-setup/      what setup.sh runs once: the STRchive catalog, zips, the genetic map
   crates/og-steps/      the step helpers: constraint-join, prs format, step 40's consumer export; og family (step 38); og pgx outside-calls|hla-concordance|pypgx-summary (steps 08, 32, 36); og paraphase (step 35b)
-  crates/og-summary/    the step tables of steps 20, 25 and 33: og mito, og prs table, og qc sample; og summary collect --only a|b|c, QC, clinical and the remaining summary sections (no caller yet); og roh summary (step 11)
+  crates/og-summary/    the step tables of steps 20, 25 and 33: og mito, og prs table, og qc sample; og summary collect, the whole summary.json with its post pass, or one group with --only a|b|c (no caller yet); og roh summary (step 11)
   crates/og-pedigree/   the pedigree rules: families files, members, kinship, PARs
   crates/og-chip/       step 49, clonal haematopoiesis: og chip prepare, report, check-table
   crates/og-cn/         copy number from depth (steps 32, 35, 41, 46): og cn bed, call, intervals, check-bam, marker-regions, marker-read

rust/README.md

@@ -10,7 +10,7 @@ OwnGenome's own tools, ported from `bin/*.py` one at a time into one static bina
 | `crates/og-bam` | BAM reading as pysam does it (fetch, `reference_end`, `cigarstring`, `get_blocks`), on htslib linked statically, and the tools that read BAMs: the reads at a variant (`reads`: one CIGAR walk per read, SA pieces, filters per tool), `og cdna check`, `og cdna scan`, `og comphet phase`, `og bam contigs`, the CYP2D6 depth check and the RFC1 motif count |
 | `crates/og-pedigree` | the pedigree rules of step 38: families files, members, kinship, declared relationships, PARs |
 | `crates/og-qc` | `og qc sex-loss`: step 33's mosaic loss of Y or X, with CPython's medians, `fmean` and exact `variance` |
-| `crates/og-summary` | the step tables: `og qc sample` (step 33), `og prs table` (step 25), `og mito annotate`, `og mito deletions`, `og mito copies` (step 20); the sections of summary.json, `og summary collect --only a` (QC), `--only b` (ClinVar, carrier list, PGx, CPSR, the clinical filter with ACMG SF and two hits, slivar) and `--only c` (the remaining summary sections), beside the Python collector, which still writes summary.json; `og roh summary` (step 11) |
+| `crates/og-summary` | the step tables: `og qc sample` (step 33), `og prs table` (step 25), `og mito annotate`, `og mito deletions`, `og mito copies` (step 20); the sections of summary.json, `og summary collect --only a` (QC), `--only b` (ClinVar, carrier list, PGx, CPSR, the clinical filter with ACMG SF and two hits, slivar) and `--only c` (the remaining summary sections), and without `--only` the whole summary.json with its post pass (the readings that join sections, the CYP2D6 block, `not_run`, `not_assessed`, `recheck`, `new_since`), beside the Python collector, which still writes summary.json; `og roh summary` (step 11) |
 | `crates/og-chip` | step 49, clonal haematopoiesis: `og chip prepare`, `og chip report`, `og chip check-table` |
 | `crates/og-cn` | copy number from depth, steps 32, 35, 41 and 46: `og cn bed`, `og cn call`, `og cn intervals`, `og cn check-bam`, `og cn marker-regions`, `og cn marker-read` |
 | `crates/og-bloodtype` | step 41, blood groups: `og bloodtype report`, `og bloodtype check-markers`, `og bloodtype systems` |

rust/crates/og-summary/Cargo.toml

@@ -15,6 +15,7 @@ og-steps.workspace = true
 og-ancestry.workspace = true
 og-chip.workspace = true
 og-cn.workspace = true
+og-runlog.workspace = true
 indexmap.workspace = true
 memchr.workspace = true