diff --git a/CHANGELOG.md b/CHANGELOG.md index dd16e9c1..0bf731b5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,10 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n ## Unreleased +- The Rust `og` binary now writes the whole summary.json: `og summary collect` without `--only` reads all 48 sections over one walk of the sample folder, then runs the post pass. The post pass joins sections: the repeat readings with the traits and CPIC states, Locityper's primary calls with HLA, pypgx and Cyrius, and CNVpytor's chrY filter with the declared sex. It also builds the CYP2D6 block and writes `not_run`, `not_assessed`, `recheck` and `new_since`. Without `--out` it writes `summary.json` in the sample folder, as the Python did; `--only` still needs `--out`. Nothing calls it yet: the reports still read the summary that `bin/collect_summary.py` writes. The whole file is the same as the Python's on 8 of 9 new parity cases, compared as JSON with key order and only `generated_utc` masked; the ninth holds the deliberate fix below. On a synthetic folder with every section it takes 0.01 to 0.03 s and 15 MB, where the Python took 0.15 to 0.21 s and 42 to 44 MB. With a 1,000,000-record VCF it takes 0.44 to 0.62 s and 17 to 20 MB, where the Python took 1.5 s and 42 to 44 MB. What differs from the Python: + - The post pass keeps the summary. The Python ran it outside its per-section error handling, so one row of an unexpected shape there lost summary.json and both reports. The Rust parts read with defaults and have no such error to raise; a part that returns an error makes only its own section unreadable, with a note that names the file. + - A section reader that stops on an error the Python did not catch keeps the summary too. A telomere table with an infinite read length stopped the Python with an OverflowError and no summary.json; og marks only the telomere section unreadable. + - Steps 43 (archaic ancestry) and 44 (ancient similarity) now run in the Rust `og` binary: `og archaic table`, `strict`, `genotypes` and `report`, and `og ancient select`, `lift`, `person-geno`, `align` and `report`. They replace `bin/archaic_report.py` and `bin/ancient_report.py`, which are gone, in the step scripts, the Nextflow modules, `setup.sh --archaic` and `setup.sh --ancient-panel`. og is static, so each part still runs in the image it ran in. `og ancient lift` reads UCSC's chain file with a new chain reader in `og-intervals`. The outputs are the same, byte for byte (the gzip genotype files by content), on the 49 parity cases, except where a line below says otherwise. No option, output file or format changes. On the real archaic table, `og archaic table` merges chromosome 1 (4.1 million sites) in 18 s and 190 MB where the Python took 76 s and 1.25 GB; bcftools and Picard still take most of each chromosome's setup. On a synthetic input of real size, step 44's panel column over 1,150,106 sites and 819 people takes 2.5 s and 342 MB instead of 4.2 s and 524 MB. What changes on purpose: - The archaic table's setup stamp now holds a format id (`ARCHAIC_TABLE_FORMAT` in `setup.sh`) instead of the sha256 of the Python's table code. A table the last Python built has the same bytes (chromosomes 1 to 6 and 21 of the real table, rebuilt from the same public files, are the same after decompression), so its stamp is taken as current once and rewritten: no install downloads the 150 GB of archaic genomes again for this change. - `og archaic genotypes` streams the sorted VCF and table together with one duplicate-position window, and writes mask runs as they close. On a synthetic chromosome with 3 million table rows and 1 million VCF records, peak memory falls from the initial port's 166 MB to 4 MB (3.79 s before, 3.91 s after). A position that goes backwards within a VCF chromosome or in the table stops with exit 2, also when a skipped record (an indel, ALT `.`) of another chromosome sits between the two. Duplicate calls, clashes, masked sites, mask runs and span counts keep their meaning. diff --git a/docs/rust.md b/docs/rust.md index 32341ab3..5f079320 100644 --- a/docs/rust.md +++ b/docs/rust.md @@ -21,7 +21,7 @@ rust/ crates/og-qc/ og qc sex-loss: mosaic loss of Y or X for step 33 crates/og-setup/ what setup.sh runs once: the STRchive catalog, zips, the genetic map crates/og-steps/ the step helpers: constraint-join, prs format, step 40's consumer export; og family (step 38); og pgx outside-calls|hla-concordance|pypgx-summary (steps 08, 32, 36); og paraphase (step 35b) - crates/og-summary/ the step tables of steps 20, 25 and 33: og mito, og prs table, og qc sample; og summary collect --only a|b|c, QC, clinical and the remaining summary sections (no caller yet); og roh summary (step 11) + crates/og-summary/ the step tables of steps 20, 25 and 33: og mito, og prs table, og qc sample; og summary collect, the whole summary.json with its post pass, or one group with --only a|b|c (no caller yet); og roh summary (step 11) crates/og-pedigree/ the pedigree rules: families files, members, kinship, PARs crates/og-chip/ step 49, clonal haematopoiesis: og chip prepare, report, check-table crates/og-cn/ copy number from depth (steps 32, 35, 41, 46): og cn bed, call, intervals, check-bam, marker-regions, marker-read diff --git a/rust/Cargo.lock b/rust/Cargo.lock index 9eb4d76a..0dbef91f 100644 --- a/rust/Cargo.lock +++ b/rust/Cargo.lock @@ -875,6 +875,7 @@ dependencies = [ "og-clinvar", "og-cn", "og-io", + "og-runlog", "og-steps", "pycompat", ] diff --git a/rust/README.md b/rust/README.md index 0e58bfca..c69f31c2 100644 --- a/rust/README.md +++ b/rust/README.md @@ -10,7 +10,7 @@ OwnGenome's own tools, ported from `bin/*.py` one at a time into one static bina | `crates/og-bam` | BAM reading as pysam does it (fetch, `reference_end`, `cigarstring`, `get_blocks`), on htslib linked statically, and the tools that read BAMs: the reads at a variant (`reads`: one CIGAR walk per read, SA pieces, filters per tool), `og cdna check`, `og cdna scan`, `og comphet phase`, `og bam contigs`, the CYP2D6 depth check and the RFC1 motif count | | `crates/og-pedigree` | the pedigree rules of step 38: families files, members, kinship, declared relationships, PARs | | `crates/og-qc` | `og qc sex-loss`: step 33's mosaic loss of Y or X, with CPython's medians, `fmean` and exact `variance` | -| `crates/og-summary` | the step tables: `og qc sample` (step 33), `og prs table` (step 25), `og mito annotate`, `og mito deletions`, `og mito copies` (step 20); the sections of summary.json, `og summary collect --only a` (QC), `--only b` (ClinVar, carrier list, PGx, CPSR, the clinical filter with ACMG SF and two hits, slivar) and `--only c` (the remaining summary sections), beside the Python collector, which still writes summary.json; `og roh summary` (step 11) | +| `crates/og-summary` | the step tables: `og qc sample` (step 33), `og prs table` (step 25), `og mito annotate`, `og mito deletions`, `og mito copies` (step 20); the sections of summary.json, `og summary collect --only a` (QC), `--only b` (ClinVar, carrier list, PGx, CPSR, the clinical filter with ACMG SF and two hits, slivar) and `--only c` (the remaining summary sections), and without `--only` the whole summary.json with its post pass (the readings that join sections, the CYP2D6 block, `not_run`, `not_assessed`, `recheck`, `new_since`), beside the Python collector, which still writes summary.json; `og roh summary` (step 11) | | `crates/og-chip` | step 49, clonal haematopoiesis: `og chip prepare`, `og chip report`, `og chip check-table` | | `crates/og-cn` | copy number from depth, steps 32, 35, 41 and 46: `og cn bed`, `og cn call`, `og cn intervals`, `og cn check-bam`, `og cn marker-regions`, `og cn marker-read` | | `crates/og-bloodtype` | step 41, blood groups: `og bloodtype report`, `og bloodtype check-markers`, `og bloodtype systems` | diff --git a/rust/crates/og-summary/Cargo.toml b/rust/crates/og-summary/Cargo.toml index 325b1f47..dedbfb69 100644 --- a/rust/crates/og-summary/Cargo.toml +++ b/rust/crates/og-summary/Cargo.toml @@ -15,6 +15,7 @@ og-steps.workspace = true og-ancestry.workspace = true og-chip.workspace = true og-cn.workspace = true +og-runlog.workspace = true indexmap.workspace = true memchr.workspace = true diff --git a/rust/crates/og-summary/src/collect.rs b/rust/crates/og-summary/src/collect.rs index e351fd30..4bbc9529 100644 --- a/rust/crates/og-summary/src/collect.rs +++ b/rust/crates/og-summary/src/collect.rs @@ -14,6 +14,10 @@ //! ancestry, structural variants, repeats, telomeres, runs of //! homozygosity, the haplogroups, the opt-in step tables, chrM, traits with //! the APOL1 G2 deletion, and the couple screen ([`group_c`]). +//! Without `--only` it reads all three groups and runs the post pass +//! ([`post`]): the readings that join sections, the steps not run, what no +//! step assessed, the last database re-check and what changed since the +//! previous run. That is the whole summary.json. //! //! How it reads a folder: one walk of the folder ([`scan::FolderIndex`]) //! instead of a `glob` per candidate path; each input read once (the @@ -32,6 +36,7 @@ pub mod group_b; pub mod group_c; pub mod model; pub mod pgx; +pub mod post; pub mod scan; pub mod sections; pub mod text; @@ -46,18 +51,17 @@ use sections::{Callability, Coverage, Data, Mosaic, ReadCount, SampleQc, SexChec use std::io::Write; use std::path::Path; -const USAGE: &str = - "usage: og summary collect --sample SAMPLE --sample-dir SAMPLE_DIR --only {a,b,c} - --out OUT [--declared-sex DECLARED_SEX] [--assets ASSETS] +const USAGE: &str = "usage: og summary collect --sample SAMPLE --sample-dir SAMPLE_DIR [--out OUT] + [--declared-sex DECLARED_SEX] [--assets ASSETS] [--only {a,b,c}] "; const OPTS: &[Opt] = &[ Opt("--sample", Kind::Value, true), Opt("--sample-dir", Kind::Value, true), Opt("--declared-sex", Kind::Value, false), - Opt("--out", Kind::Value, true), + Opt("--out", Kind::Value, false), Opt("--assets", Kind::Value, false), - Opt("--only", Kind::Choice(&["a", "b", "c"]), true), + Opt("--only", Kind::Choice(&["a", "b", "c"]), false), ]; /// What one collect reads. @@ -70,7 +74,7 @@ pub struct Input { pub declared_sex: Option, /// The repository's `assets/` folder. pub assets: String, - /// The group of sections to read: `a`, `b` or `c`. + /// The group of sections to read: `a`, `b`, `c`, or all of them. pub group: Group, /// The folder of the two tables group c reads from `bin/`. pub tables: String, @@ -82,6 +86,8 @@ pub enum Group { A, B, C, + /// Every section and the post pass: the whole summary.json. + All, } /// `og summary collect`: exit 0 when summary.json is written (whatever @@ -102,28 +108,39 @@ pub fn cli(argv: &[String], out: &mut impl Write, err: &mut impl Write) -> u8 { declared_sex: a.get("--declared-sex").map(str::to_owned), assets: a.get("--assets").map_or_else(default_assets, str::to_owned), tables: group_c::util::default_tables(), - group: match a.text("--only") { - "b" => Group::B, - "c" => Group::C, - _ => Group::A, + group: match a.get("--only") { + Some("a") => Group::A, + Some("b") => Group::B, + Some("c") => Group::C, + _ => Group::All, }, }; // A part of the sections must never replace the sample's complete - // summary.json, so there is no default path while `--only` exists. - let target = a.text("--out").to_owned(); - let read = match a.text("--only") { - "b" => group_b::collect, - _ => collect, - }; - let summary = match read(&input) { - Ok(s) => s, - Err(e) => return crate::finish(Err(e), err), + // summary.json, so a part is written only where `--out` says. + let target = match a.get("--out") { + Some(t) => t.to_owned(), + None if input.group == Group::All => crate::join(&input.sample_dir, "summary.json"), + None => { + let _ = write!(err, "{USAGE}"); + let _ = writeln!( + err, + "og summary collect: error: --only needs --out: a part of the sections never replaces summary.json" + ); + return 2; + } }; // The sections' own data is freed before the text is made, and the // tree before the file is written: a large ClinVar list is in memory in // one form at a time. - let value = summary.to_json(); - drop(summary); + let read = match input.group { + Group::All => post::collect(&input), + Group::B => group_b::collect(&input).map(|s| s.to_json()), + Group::A | Group::C => collect(&input).map(|s| s.to_json()), + }; + let value = match read { + Ok(v) => v, + Err(e) => return crate::finish(Err(e), err), + }; let body = match dumps( &value, &Dump { @@ -161,47 +178,83 @@ fn default_assets() -> String { .to_string() } -/// Reads the folder into the summary of group a. An error is a run status -/// or a manifest that is there and cannot be read: without it a result of -/// an earlier run would be called current, so no summary is written. +/// What every group reads first: one walk of the folder, the run status +/// and the manifest. +pub struct Folder { + pub index: FolderIndex, + pub run: Option, + pub manifest: Option, +} + +impl Folder { + /// Walks the folder and reads its run status and manifest. An error is + /// a run status or a manifest that is there and cannot be read: without + /// it a result of an earlier run would be called current, so no summary + /// is written. + pub fn open(input: &Input) -> Result { + let dir = Path::new(&input.sample_dir); + let index = FolderIndex::walk(dir); + let run = + read_optional(&dir.join("logs").join("run_status.tsv"))?.map(|t| RunStatus::parse(&t)); + let manifest = read_optional(&dir.join("run_manifest.tsv"))?.map(|t| Manifest::parse(&t)); + Ok(Folder { + index, + run, + manifest, + }) + } +} + +/// Reads the folder into the summary of group a or group c. pub fn collect(input: &Input) -> Result { - let dir = Path::new(&input.sample_dir); - let index = FolderIndex::walk(dir); - let run = - read_optional(&dir.join("logs").join("run_status.tsv"))?.map(|t| RunStatus::parse(&t)); - let manifest = read_optional(&dir.join("run_manifest.tsv"))?.map(|t| Manifest::parse(&t)); - let s = pycompat::mtime::escape(&input.sample); - let find = |pats: &[String]| index.first(pats).map(str::to_owned); - let file = |rel: String| -> (String, String) { (crate::join(&input.sample_dir, &rel), rel) }; - if input.group == Group::C { + let folder = Folder::open(input)?; + let (sections, declared_sex) = if input.group == Group::C { // The declared sex the chrY filter of CNVpytor reads comes from // step 33's table when the run status and manifest have none. - let qc = find(&[format!("qc/{s}_sample_qc.tsv")]).map(file); - let qc_declared = qc.and_then(|(p, _)| { + let s = pycompat::mtime::escape(&input.sample); + let qc = folder + .index + .first(&[format!("qc/{s}_sample_qc.tsv")]) + .map(|rel| crate::join(&input.sample_dir, rel)); + let qc_declared = qc.and_then(|p| { let rows = sections::read_tsv(&p).ok()?; SampleQc::read(&p, &rows) .ok()? .declared_sex() .map(str::to_owned) }); - let declared_sex = declared( + let declared_sex = declared(input, &folder, qc_declared.as_deref()); + let mut sections = read_c(input, &folder); + post_c( input, - run.as_ref(), - manifest.as_ref(), - qc_declared.as_deref(), + &folder.index, + &mut sections, + &group_c::Others::default(), + declared_sex.as_deref(), ); - let sections = read_group_c(input, &index, run.as_ref(), declared_sex.as_deref()); - return Ok(Summary { - sample: input.sample.clone(), - generated_utc: time::utc_minute(now()).unwrap_or_default(), - run, - manifest, - declared_sex, - sections, - cyp2d6: None, - }); - } + (sections, declared_sex) + } else { + let sections = read_a(input, &folder); + let declared_sex = declared(input, &folder, qc_declared(§ions)); + (sections, declared_sex) + }; + Ok(Summary { + sample: input.sample.clone(), + generated_utc: time::utc_minute(now()).unwrap_or_default(), + run: folder.run, + manifest: folder.manifest, + declared_sex, + sections, + cyp2d6: None, + }) +} +/// Reads the sections of group a, in [`GROUP_A`] order: the sample VCF on +/// its own thread while the other six are read. +pub fn read_a(input: &Input, folder: &Folder) -> Vec
{ + let s = pycompat::mtime::escape(&input.sample); + let find = |pats: &[String]| folder.index.first(pats).map(str::to_owned); + let file = |rel: String| -> (String, String) { (crate::join(&input.sample_dir, &rel), rel) }; let variants_at = find(&[format!("vcf/{s}.vcf.gz")]); let (variants, rest) = std::thread::scope(|scope| { let v = scope.spawn(|| { @@ -233,29 +286,23 @@ pub fn collect(input: &Input) -> Result { read_count, variants, ]; - let sections: Vec
= GROUP_A + GROUP_A .iter() .zip(reads) - .map(|(spec, r): (&Spec, _)| Section::new(*spec, r, run.as_ref())) - .collect(); + .map(|(spec, r): (&Spec, _)| Section::new(*spec, r, folder.run.as_ref())) + .collect() +} - let qc_declared = sections +/// The declared sex step 33's table records, from its section (read when +/// the section is ok or stale). +pub fn qc_declared(sections: &[Section]) -> Option<&str> { + sections .iter() .find(|s| s.spec.key == "sample_qc") .and_then(|s| match &s.data { Some(Data::SampleQc(q)) => q.declared_sex(), _ => None, - }); - let declared_sex = declared(input, run.as_ref(), manifest.as_ref(), qc_declared); - Ok(Summary { - sample: input.sample.clone(), - generated_utc: time::utc_minute(now()).unwrap_or_default(), - run, - manifest, - declared_sex, - sections, - cyp2d6: None, - }) + }) } fn now() -> f64 { @@ -266,16 +313,14 @@ fn now() -> f64 { /// The declared sex: the command line's, the run status's, the /// manifest's, then step 33's; the first that is not empty. -fn declared( - input: &Input, - run: Option<&RunStatus>, - manifest: Option<&Manifest>, - qc: Option<&str>, -) -> Option { +pub fn declared(input: &Input, folder: &Folder, qc: Option<&str>) -> Option { [ input.declared_sex.as_deref(), - run.and_then(|r| r.declared_sex.as_deref()), - manifest.and_then(|m| m.value("run", "declared_sex")), + folder.run.as_ref().and_then(|r| r.declared_sex.as_deref()), + folder + .manifest + .as_ref() + .and_then(|m| m.value("run", "declared_sex")), qc, ] .into_iter() @@ -284,39 +329,56 @@ fn declared( .map(str::to_owned) } -/// Reads the sections of group c, then runs the parts of the post pass -/// that change them. -fn read_group_c( +/// Reads the sections of group c, in [`group_c::GROUP_C`] order. +pub fn read_c(input: &Input, folder: &Folder) -> Vec
{ + let c = ctx_c(input, &folder.index); + group_c::read_all(&c) + .into_iter() + .zip(group_c::GROUP_C.iter()) + .map(|(r, spec)| { + let r = r.and_then(|found| match found { + None => Ok(None), + Some((f, v)) => with_mtime(&f.path, f.rel, Data::Json(v)), + }); + Section::new(*spec, r, folder.run.as_ref()) + }) + .collect() +} + +/// The parts of the post pass that change group c's sections: the repeat +/// readings, Locityper's primary calls and CNVpytor's chrY filter. A part +/// that fails makes its own section unreadable, with a note that names the +/// file and the reading; the other sections and the summary stay. +pub fn post_c( input: &Input, index: &FolderIndex, - run: Option<&RunStatus>, + sections: &mut [Section], + others: &group_c::Others, declared_sex: Option<&str>, -) -> Vec
{ - let c = group_c::Ctx { +) { + let c = ctx_c(input, index); + let failed = { + let mut view: Vec<(Spec, model::State, Option<&mut Data>)> = sections + .iter_mut() + .map(|s| (s.spec, s.state, s.data.as_mut())) + .collect(); + group_c::post(&c, &mut view, others, declared_sex) + }; + for (key, why) in failed { + if let Some(s) = sections.iter_mut().find(|s| s.spec.key == key) { + s.make_unreadable(&why); + } + } +} + +fn ctx_c<'a>(input: &'a Input, index: &'a FolderIndex) -> group_c::Ctx<'a> { + group_c::Ctx { sample: &input.sample, dir: &input.sample_dir, assets: &input.assets, tables: &input.tables, index, - }; - let reads = group_c::read_all(&c); - let mut sections: Vec
= group_c::GROUP_C - .iter() - .zip(reads) - .map(|(spec, r)| { - let r = r.and_then(|found| match found { - None => Ok(None), - Some((f, v)) => with_mtime(&f.path, f.rel, Data::Json(v)), - }); - Section::new(*spec, r, run) - }) - .collect(); - let mut view: Vec<(Spec, model::State, Option<&mut Data>)> = sections - .iter_mut() - .map(|s| (s.spec, s.state, s.data.as_mut())) - .collect(); - group_c::post(&c, &mut view, &group_c::Others::default(), declared_sex); - sections + } } /// What one reader gave: the file (relative), its mtime and its data. diff --git a/rust/crates/og-summary/src/collect/group_b.rs b/rust/crates/og-summary/src/collect/group_b.rs index cb0ba8a1..394178cf 100644 --- a/rust/crates/og-summary/src/collect/group_b.rs +++ b/rust/crates/og-summary/src/collect/group_b.rs @@ -14,13 +14,12 @@ use super::carrier::Carrier; use super::clinical::{Clinical, Slivar}; use super::clinvar::{ClinVar, Floor, HitsVcf}; use super::cpsr::{self, Cpsr}; -use super::model::{Manifest, RunStatus, Section, Spec, Summary}; +use super::model::{Section, Spec, Summary}; use super::pgx::{Cpic, Cyp2d6, Cyrius, PharmCat, Pypgx}; use super::scan::FolderIndex; use super::sections::{Data, SampleQc}; use super::two_hits::{self, DeletionInputs}; -use super::{Input, Read, read_at, read_optional, with_mtime}; -use std::path::Path; +use super::{Folder, Input, Read, read_at, with_mtime}; /// The sections `og summary collect --only b` reads, in summary.json's order. pub const GROUP_B: [Spec; 9] = [ @@ -74,12 +73,42 @@ pub const GROUP_B: [Spec; 9] = [ /// Reads the folder into the summary of group b. An error is a run status /// or a manifest that is there and cannot be read. pub fn collect(input: &Input) -> Result { - let dir = Path::new(&input.sample_dir); + let folder = Folder::open(input)?; + let sections = read(input, &folder); + let cyp2d6 = cyp2d6(input, &folder.index, §ions); + let s = pycompat::mtime::escape(&input.sample); + // The run block is group a's: the declared sex from the command line, + // the run status, the manifest or step 33's table, the first that has one. + let qc_declared = folder + .index + .first(&[format!("qc/{s}_sample_qc.tsv")]) + .and_then(|rel| { + let p = crate::join(&input.sample_dir, rel); + let rows = super::sections::read_tsv(&p).ok()?; + SampleQc::read(&p, &rows) + .ok()? + .declared_sex() + .map(str::to_owned) + }); + let declared_sex = super::declared(input, &folder, qc_declared.as_deref()); + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map_or(0.0, |d| d.as_secs_f64()); + Ok(Summary { + sample: input.sample.clone(), + generated_utc: super::time::utc_minute(now).unwrap_or_default(), + run: folder.run, + manifest: folder.manifest, + declared_sex, + sections, + cyp2d6, + }) +} + +/// Reads the sections of group b, in [`GROUP_B`] order. +pub fn read(input: &Input, folder: &Folder) -> Vec
{ let d = input.sample_dir.as_str(); - let index = FolderIndex::walk(dir); - let run = - read_optional(&dir.join("logs").join("run_status.tsv"))?.map(|t| RunStatus::parse(&t)); - let manifest = read_optional(&dir.join("run_manifest.tsv"))?.map(|t| Manifest::parse(&t)); + let index = &folder.index; let s = pycompat::mtime::escape(&input.sample); let raw = input.sample.as_str(); let file = |rel: &str| -> (String, String) { (crate::join(d, rel), rel.to_owned()) }; @@ -122,7 +151,7 @@ pub fn collect(input: &Input) -> Result { )?; with_mtime(&p, rel, Data::Clinical(data)) }); - let rest = read_rest(input, &s, &index, &find, &hits, &deletions); + let rest = read_rest(input, &s, index, &find, &hits, &deletions); let join = |h: std::thread::ScopedJoinHandle<'_, Read>| { h.join() .unwrap_or_else(|_| Err("internal error: the section reader stopped".into())) @@ -133,55 +162,26 @@ pub fn collect(input: &Input) -> Result { let reads = [ clinvar, carrier, pharmcat, cpic, pypgx, cyrius, cpsr, clinical, slivar, ]; - let sections: Vec
= GROUP_B + GROUP_B .iter() .zip(reads) - .map(|(spec, r)| Section::new(*spec, r, run.as_ref())) - .collect(); + .map(|(spec, r)| Section::new(*spec, r, folder.run.as_ref())) + .collect() +} + +/// The `cyp2d6` block: CPIC, pypgx and Cyrius side by side, with step 36's +/// row. `None` when `sections` lack one of the three. +pub fn cyp2d6(input: &Input, index: &FolderIndex, sections: &[Section]) -> Option { + let s = pycompat::mtime::escape(&input.sample); + let d = input.sample_dir.as_str(); let sec = |k: &str| sections.iter().find(|x| x.spec.key == k); - let cyp2d6 = match (sec("cpic"), sec("pypgx"), sec("cyrius")) { - (Some(c), Some(p), Some(y)) => Some(Cyp2d6::new( - c, - p, - y, - one(format!("pgx_consensus/{s}_pgx_consensus.tsv")), - d, - )), + let consensus = index + .first(&[format!("pgx_consensus/{s}_pgx_consensus.tsv")]) + .map(|rel| (crate::join(d, rel), rel.to_owned())); + match (sec("cpic"), sec("pypgx"), sec("cyrius")) { + (Some(c), Some(p), Some(y)) => Some(Cyp2d6::new(c, p, y, consensus, d)), _ => None, - }; - - // The run block is group a's: the declared sex from the command line, - // the run status, the manifest or step 33's table, the first that has one. - let qc_declared = one(format!("qc/{s}_sample_qc.tsv")).and_then(|(p, _)| { - let rows = super::sections::read_tsv(&p).ok()?; - SampleQc::read(&p, &rows) - .ok()? - .declared_sex() - .map(str::to_owned) - }); - let declared_sex = [ - input.declared_sex.clone(), - run.as_ref().and_then(|r| r.declared_sex.clone()), - manifest - .as_ref() - .and_then(|m| m.value("run", "declared_sex").map(str::to_owned)), - qc_declared, - ] - .into_iter() - .flatten() - .find(|v| !v.is_empty()); - let now = std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .map_or(0.0, |d| d.as_secs_f64()); - Ok(Summary { - sample: input.sample.clone(), - generated_utc: super::time::utc_minute(now).unwrap_or_default(), - run, - manifest, - declared_sex, - sections, - cyp2d6, - }) + } } /// Finds a section's file: the full path and the path in the folder. diff --git a/rust/crates/og-summary/src/collect/group_c.rs b/rust/crates/og-summary/src/collect/group_c.rs index 15ab2778..d9c8c950 100644 --- a/rust/crates/og-summary/src/collect/group_c.rs +++ b/rust/crates/og-summary/src/collect/group_c.rs @@ -9,7 +9,8 @@ //! readings with the traits and CPIC states, Locityper's primary calls, and //! the chrY filter of CNVpytor's count for a declared female. What they //! need from group b (the CPIC state, pypgx and Cyrius) comes in -//! [`Others`]; `--only c` reads none of it, so it is empty there. +//! [`Others`]; `--only c` reads none of it, so it is empty there. A part +//! that fails makes only its own section unreadable. mod chrm; mod cn; @@ -236,13 +237,17 @@ pub fn read_all(c: &Ctx<'_>) -> Vec { /// The post pass over the read sections (`sections` in [`GROUP_C`] /// order, each with its state and data): the repeat readings with the /// traits and CPIC states, each Locityper locus's primary call, and the -/// chrY filter of CNVpytor's count. +/// chrY filter of CNVpytor's count. Each part runs on its own: one that +/// returns an error is returned as its section's key and why, and the +/// others still run. The parts read with defaults, so a row of an +/// unexpected shape is no error. A panic is not caught: the release binary +/// aborts on one (`panic = "abort"`). pub fn post( c: &Ctx<'_>, sections: &mut [(Spec, State, Option<&mut Data>)], others: &Others, declared_sex: Option<&str>, -) { +) -> Vec<(&'static str, String)> { let state = |key: &str, secs: &[(Spec, State, Option<&mut Data>)]| { secs.iter() .find(|(s, ..)| s.key == key) @@ -257,20 +262,39 @@ pub fn post( .and_then(|x| x.2.as_deref()) .and_then(json) .cloned(); + let mut failed = Vec::new(); for (spec, st, data) in sections.iter_mut() { let live = matches!(st, State::Ok | State::Stale); let Some(Data::Json(v)) = data.as_deref_mut() else { continue; }; - match spec.key { - "expansions" if live => repeats::post(c, v, traits_state, cpic_state), - "locityper" if live => { - tables::locityper_post(v, hla_state, hla_data.as_ref(), others); + let part = || -> Result<(), String> { + match spec.key { + "expansions" if live => repeats::post(c, v, traits_state, cpic_state) + .map_err(|e| format!("the repeat readings could not be made: {e}")), + "locityper" if live => { + tables::locityper_post(v, hla_state, hla_data.as_ref(), others); + Ok(()) + } + "cnvpytor" => { + sv::cnvpytor_sex_filter(v, declared_sex); + Ok(()) + } + _ => Ok(()), } - "cnvpytor" => sv::cnvpytor_sex_filter(v, declared_sex), - _ => {} - } + }; + failed.extend(contain(spec.key, part)); } + failed +} + +/// Runs one part of the post pass: `None` when it went through, else its +/// section's key and the error it returned. +pub fn contain( + key: &'static str, + part: impl FnOnce() -> Result<(), String>, +) -> Option<(&'static str, String)> { + part().err().map(|e| (key, e)) } fn json(d: &Data) -> Option<&Value> { diff --git a/rust/crates/og-summary/src/collect/group_c/repeats.rs b/rust/crates/og-summary/src/collect/group_c/repeats.rs index 9d9610fd..3081543d 100644 --- a/rust/crates/og-summary/src/collect/group_c/repeats.rs +++ b/rust/crates/og-summary/src/collect/group_c/repeats.rs @@ -910,9 +910,10 @@ fn readings( } /// The post pass: the readings again, with the traits and CPIC states, over -/// the loci as the section lists them. -pub fn post(c: &Ctx<'_>, v: &mut Value, traits: State, cpic: State) { - let Value::Object(m) = v else { return }; +/// the loci as the section lists them. An error is a reading that could +/// not be made; the section's data is then left as it was. +pub fn post(c: &Ctx<'_>, v: &mut Value, traits: State, cpic: State) -> Result<(), String> { + let Value::Object(m) = v else { return Ok(()) }; let loci: Vec<(String, Obj)> = match m.get("loci") { Some(Value::Array(xs)) => xs .iter() @@ -924,10 +925,10 @@ pub fn post(c: &Ctx<'_>, v: &mut Value, traits: State, cpic: State) { _ => None, }) .collect(), - _ => return, + _ => return Ok(()), }; let by = by_locus(loci.iter().map(|(k, o)| (k, o))); - if let Ok(r) = readings(c, &by, traits, cpic) { - m.insert("readings".into(), r); - } + let r = readings(c, &by, traits, cpic)?; + m.insert("readings".into(), r); + Ok(()) } diff --git a/rust/crates/og-summary/src/collect/model.rs b/rust/crates/og-summary/src/collect/model.rs index c6d6e22a..2d6e7c99 100644 --- a/rust/crates/og-summary/src/collect/model.rs +++ b/rust/crates/og-summary/src/collect/model.rs @@ -178,6 +178,20 @@ impl Section { s } + /// Marks a section whose reading failed after its file was read (a + /// part of the post pass): `unreadable`, as a reader's error leaves it, + /// with a note that names the file the section came from. + pub fn make_unreadable(&mut self, why: &str) { + self.note = Some(match self.source.as_deref() { + Some(src) => format!("{}: {why}", crate::basename(src)), + None => why.to_owned(), + }); + self.state = State::Unreadable; + self.source = None; + self.file_date = None; + self.data = None; + } + pub fn to_json(&self) -> Value { let mut m = IndexMap::new(); m.insert("title".into(), Value::Str(self.spec.title.into())); diff --git a/rust/crates/og-summary/src/collect/post.rs b/rust/crates/og-summary/src/collect/post.rs new file mode 100644 index 00000000..6065d540 --- /dev/null +++ b/rust/crates/og-summary/src/collect/post.rs @@ -0,0 +1,341 @@ +//! The whole summary.json: every section of groups a, b and c, read over +//! one walk of the folder, then the post pass. +//! +//! The post pass is what joins sections after each is read: +//! +//! - the repeat readings with the traits and CPIC states, Locityper's +//! primary calls with HLA, pypgx and Cyrius, and CNVpytor's chrY filter +//! with the declared sex ([`super::post_c`]); +//! - the `cyp2d6` block over CPIC, pypgx and Cyrius; +//! - `not_run`: the steps whose section has no file; +//! - `not_assessed`: what no step looked at, with the lines that depend on +//! the polygenic scores, HBA1/HBA2, mobile elements, VNtyper and MoChA; +//! - `recheck`: the last database re-check of the sample; +//! - `new_since`: what changed since the previous run kept in `history/`. +//! +//! Each part that changes a section runs on its own: when it returns an +//! error, only its section becomes `unreadable`, with a note naming the +//! file, and the summary is still written. The parts read with defaults, so +//! a row of an unexpected shape is no error; a panic is not caught, as the +//! release binary aborts on one. The Python collector ran this pass outside +//! its per-section error handling, so one failure there lost summary.json +//! and both reports. A previous run that cannot be read leaves `new_since` +//! out, as before; the comparison itself has no error to hide. + +use super::group_c::Others; +use super::model::{Section, State, Summary}; +use super::sections::Data; +use super::{Folder, Input, declared, group_b, post_c, qc_declared, read_a, read_c, time}; +use og_runlog::{delta, files}; +use pycompat::json::Value; +use pycompat::value::truthy; +use std::path::Path; + +/// summary.json's sections, in its order. +pub const ORDER: [&str; 48] = [ + "coverage", + "callability", + "sex_check", + "sample_qc", + "mosaic_sex_loss", + "read_count", + "variants", + "clinvar", + "couple", + "carrier", + "pharmcat", + "cpic", + "pypgx", + "cyrius", + "parascopy", + "paralogs", + "hla", + "prs", + "ancestry_fine", + "manta", + "delly", + "cnvpytor", + "sv_consensus", + "annotsv", + "expansions", + "telomere", + "roh", + "haplogroup", + "y_haplogroup", + "y_tree", + "y_str", + "consumer_export", + "bloodtype", + "lpa", + "vntyper", + "locityper", + "mito", + "traits", + "archaic", + "ancient", + "viral", + "exomiser", + "chip", + "mei", + "mosaic", + "cpsr", + "clinical", + "slivar", +]; + +/// Sections that are no step of their own, so never in `not_run`: the read +/// count is written by alignment, which a run from a BAM or a VCF never +/// does; the couple screen and the carrier list come with other steps. +const NOT_A_STEP: [&str; 3] = ["read_count", "couple", "carrier"]; + +/// What no step assesses. +const NOT_ASSESSED: [&str; 5] = [ + "Mosaic and low-fraction small variants (only chrM heteroplasmy is reported, and with step 49 the CHIP genes)", + "Methylation and imprinting", + "Repeat expansions outside the ExpansionHunter catalog", + "Phase: two variants in one gene are compound-het candidates, not confirmed", + "Variants of uncertain significance (not interpreted)", +]; + +/// Added unless step 25 ran with an ancestry reference panel. +const PRS_NOT_ADJUSTED: &str = + "Polygenic score percentiles: no ancestry reference panel was used, so the scores are raw sums"; + +/// Added unless step 35's depth count (short reads) or step 35b +/// (Paraphase, PacBio HiFi, both genes) made a call. +const HBA_NOT_ASSESSED: &str = "HBA1/HBA2 copy number (alpha-thalassaemia deletions): neither step 35's read-depth count \ +(short reads) nor step 35b (Paraphase, PacBio HiFi) made a call"; + +/// Added when step 50 (opt-in, MEGAnE) did not run. A failed step 50 is +/// listed under not run, and an unreadable table under its section. +const MEI_NOT_ASSESSED: &str = "Mobile element insertions (Alu, L1 and SVA, the FKTN, TAF1 and BBS1 SVA insertions among them): \ +not looked for (step 50, MEGAnE, is opt-in)"; + +/// Added unless step 52 (VNtyper) gave a positive, possible or negative +/// reading: no other step sees a frameshift inside the MUC1 VNTR. +const VNTYPER_NOT_ASSESSED: &str = "MUC1 VNTR frameshifts (ADTKD-MUC1): step 52 (VNtyper, opt-in) gave no reading; \ +no standard caller sees them"; + +/// Added unless step 51 (MoChA) assessed the sample. +const MCA_NOT_ASSESSED: &str = "Autosomal mosaic chromosomal alterations (large gains, losses and copy-neutral LOH in a share \ +of the blood cells): step 51 (MoChA) did not assess this sample"; + +/// Reads the whole summary of a sample folder. The three groups are read +/// at once over one walk of the folder. An error is a run status or a +/// manifest that is there and cannot be read. +pub fn collect(input: &Input) -> Result { + let folder = Folder::open(input)?; + let (a, b, mut c) = std::thread::scope(|scope| { + let a = scope.spawn(|| read_a(input, &folder)); + let b = scope.spawn(|| group_b::read(input, &folder)); + let c = read_c(input, &folder); + let stopped = |specs: &[super::model::Spec]| -> Vec
{ + specs + .iter() + .map(|s| { + Section::new( + *s, + Err("internal error: the section reader stopped".into()), + folder.run.as_ref(), + ) + }) + .collect() + }; + let a = a.join().unwrap_or_else(|_| stopped(&super::model::GROUP_A)); + let b = b.join().unwrap_or_else(|_| stopped(&group_b::GROUP_B)); + (a, b, c) + }); + let declared_sex = declared(input, &folder, qc_declared(&a)); + post_c( + input, + &folder.index, + &mut c, + &others(&b), + declared_sex.as_deref(), + ); + let cyp2d6 = group_b::cyp2d6(input, &folder.index, &b); + let sections = in_order(a.into_iter().chain(b).chain(c).collect()); + let not_run = not_run(§ions); + let not_assessed = not_assessed(§ions); + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map_or(0.0, |d| d.as_secs_f64()); + let summary = Summary { + sample: input.sample.clone(), + generated_utc: time::utc_minute(now).unwrap_or_default(), + run: folder.run, + manifest: folder.manifest, + declared_sex, + sections, + cyp2d6, + }; + let mut v = summary.to_json(); + drop(summary); + let Value::Object(m) = &mut v else { + return Err("internal error: the summary is not an object".into()); + }; + m.insert( + "not_run".into(), + Value::Array(not_run.into_iter().map(Value::Str).collect()), + ); + m.insert( + "not_assessed".into(), + Value::Array( + not_assessed + .into_iter() + .map(|x| Value::Str(x.to_owned())) + .collect(), + ), + ); + let dir = Path::new(&input.sample_dir); + if let Some(r) = delta::read_recheck(&dir.join("recheck").join("state.json")) { + m.insert("recheck".into(), Value::Object(r)); + } + if let Some(n) = new_since(dir, &v) + && let Value::Object(m) = &mut v + { + m.insert("new_since".into(), n); + } + Ok(v) +} + +/// What the post pass of group c needs from group b: the CPIC state, +/// pypgx's CYP2D6 call and Cyrius's genotype and Filter. +fn others(b: &[Section]) -> Others { + let sec = |k: &str| b.iter().find(|s| s.spec.key == k); + let live = |s: &Section| matches!(s.state, State::Ok | State::Stale); + Others { + cpic: sec("cpic").map(|s| s.state), + traits: None, + pypgx: sec("pypgx") + .filter(|s| live(s)) + .and_then(|s| match &s.data { + Some(Data::Pypgx(p)) => Some((s.state, p.cyp2d6.clone())), + _ => None, + }), + cyrius: sec("cyrius") + .filter(|s| live(s)) + .and_then(|s| match &s.data { + Some(Data::Cyrius(c)) => { + Some((s.state, Some(c.genotype.clone()), c.filter.clone())) + } + _ => None, + }), + } +} + +/// The sections in summary.json's order ([`ORDER`]). +fn in_order(mut sections: Vec
) -> Vec
{ + sections.sort_by_key(|s| ORDER.iter().position(|k| *k == s.spec.key)); + sections +} + +/// The steps whose section has no file, with " (step N failed)" for a step +/// that failed in the last run. +fn not_run(sections: &[Section]) -> Vec { + sections + .iter() + .filter(|s| matches!(s.state, State::Missing | State::Failed)) + .filter(|s| !NOT_A_STEP.contains(&s.spec.key)) + .map(|s| { + if s.state == State::Failed { + format!("{} (step {} failed)", s.spec.title, s.spec.step) + } else { + s.spec.title.to_owned() + } + }) + .collect() +} + +/// What no step assessed for this sample. +fn not_assessed(sections: &[Section]) -> Vec<&'static str> { + let sec = |k: &str| sections.iter().find(|s| s.spec.key == k); + let state = |k: &str| sec(k).map_or(State::Missing, |s| s.state); + // A section's data is there when it is ok or stale; else it reads {}. + let field = |k: &str, key: &str| -> Option<&Value> { + match sec(k).and_then(|s| s.data.as_ref()) { + Some(Data::Json(Value::Object(m))) => m.get(key), + _ => None, + } + }; + let mut out = NOT_ASSESSED.to_vec(); + if !field("prs", "adjusted").is_some_and(truthy) { + out.push(PRS_NOT_ADJUSTED); + } + // Step 35b on a region subset (--paraphase_regions) may leave hba out. + let hba_genes: Vec<&str> = match field("paralogs", "genes") { + Some(Value::Array(genes)) => genes + .iter() + .filter(|g| is(at(Some(g), "region"), "hba") && is(at(Some(g), "status"), "called")) + .filter_map(|g| match at(Some(g), "gene") { + Some(Value::Str(x)) => Some(x.as_str()), + _ => None, + }) + .collect(), + _ => Vec::new(), + }; + let paraphase_hba = ["HBA1", "HBA2"].iter().all(|g| hba_genes.contains(g)); + if !(paraphase_hba || is(at(field("parascopy", "hba"), "status"), "called")) { + out.push(HBA_NOT_ASSESSED); + } + if state("mei") == State::Missing { + out.push(MEI_NOT_ASSESSED); + } + let vntyper_read = state("vntyper") == State::Ok + && ["positive", "possible", "negative"] + .iter() + .any(|w| is(field("vntyper", "status"), w)); + if !vntyper_read { + out.push(VNTYPER_NOT_ASSESSED); + } + if !is(field("mosaic", "status"), "assessed") { + out.push(MCA_NOT_ASSESSED); + } + out +} + +/// True when `v` is the string `word`. +fn is(v: Option<&Value>, word: &str) -> bool { + matches!(v, Some(Value::Str(x)) if x == word) +} + +/// `v[key]` when `v` is an object that has it. +fn at<'a>(v: Option<&'a Value>, key: &str) -> Option<&'a Value> { + match v { + Some(Value::Object(m)) => m.get(key), + _ => None, + } +} + +/// `{previous_utc, sections}`: what changed since the newest run kept in +/// `history/` that holds a summary.json (generate-report.sh keeps one per +/// run), or `None` without one. A previous run whose summary or manifest +/// cannot be read, or whose summary is not shaped as a summary, counts as +/// none. +fn new_since(dir: &Path, cur: &Value) -> Option { + let prev_dir = delta::newest_history(dir)?; + let bytes = std::fs::read(prev_dir.join("summary.json")).ok()?; + let prev = pycompat::json::loads(std::str::from_utf8(&bytes).ok()?).ok()?; + let prev_manifest = files::read_manifest(Some(&prev_dir.join("run_manifest.tsv"))).ok()?; + let manifest = files::read_manifest(Some(&dir.join("run_manifest.tsv"))).ok()?; + let changes = delta::compare(&prev, &prev_manifest, cur, &manifest).ok()?; + let previous = match &prev { + Value::Object(m) => m.get("generated_utc").cloned().unwrap_or(Value::Null), + _ => Value::Null, + }; + let mut out = indexmap::IndexMap::new(); + out.insert("previous_utc".to_owned(), previous); + out.insert( + "sections".to_owned(), + Value::Array( + delta::changed_sections(&changes) + .into_iter() + .map(Value::Str) + .collect(), + ), + ); + Some(Value::Object(out)) +} + +#[cfg(test)] +mod tests; diff --git a/rust/crates/og-summary/src/collect/post/tests.rs b/rust/crates/og-summary/src/collect/post/tests.rs new file mode 100644 index 00000000..34d18cc6 --- /dev/null +++ b/rust/crates/og-summary/src/collect/post/tests.rs @@ -0,0 +1,348 @@ +use super::super::group_c::{self, GROUP_C}; +use super::super::model::{GROUP_A, Spec}; +use super::super::{Group, Input, group_b}; +use super::*; +use crate::testutil::{Dir, run}; +use pycompat::json::{Value, loads}; + +fn input(d: &Dir) -> Input { + Input { + sample: "S".into(), + sample_dir: d.path("S"), + declared_sex: None, + assets: d.path("assets"), + group: Group::All, + tables: d.path("tables"), + } +} + +fn summary(d: &Dir) -> Value { + std::fs::create_dir_all(d.path("S")).unwrap(); + collect(&input(d)).unwrap() +} + +fn at<'a>(v: &'a Value, path: &[&str]) -> &'a Value { + path.iter().fold(v, |v, k| match v { + Value::Object(m) => m.get(*k).unwrap_or_else(|| panic!("no {k} in {path:?}")), + Value::Array(a) => &a[k.parse::().unwrap()], + _ => panic!("{path:?}"), + }) +} + +fn has(v: &Value, key: &str) -> bool { + matches!(v, Value::Object(m) if m.contains_key(key)) +} + +fn texts(v: &Value) -> Vec { + match v { + Value::Array(a) => a + .iter() + .map(|x| match x { + Value::Str(s) => s.clone(), + _ => panic!("{x:?}"), + }) + .collect(), + _ => panic!("{v:?}"), + } +} + +fn word(v: &Value) -> &str { + match v { + Value::Str(s) => s, + _ => panic!("{v:?}"), + } +} + +#[test] +fn the_order_holds_every_section_once() { + let mut keys: Vec<&str> = GROUP_A + .iter() + .chain(group_b::GROUP_B.iter()) + .chain(GROUP_C.iter()) + .map(|s: &Spec| s.key) + .collect(); + assert_eq!(keys.len(), ORDER.len()); + keys.sort_unstable(); + let mut order = ORDER.to_vec(); + order.sort_unstable(); + assert_eq!(keys, order); +} + +#[test] +fn an_empty_folder_lists_every_step_and_every_line() { + let d = Dir::new("post-empty"); + let v = summary(&d); + let Value::Object(m) = &v else { panic!() }; + let keys: Vec<&str> = m.keys().map(String::as_str).collect(); + assert_eq!( + keys, + [ + "schema_version", + "sample", + "generated_utc", + "run", + "databases", + "manifest", + "sections", + "cyp2d6", + "not_run", + "not_assessed" + ] + ); + let Value::Object(sections) = at(&v, &["sections"]) else { + panic!() + }; + assert_eq!( + sections.keys().map(String::as_str).collect::>(), + ORDER + ); + let not_run = texts(at(&v, &["not_run"])); + assert_eq!(not_run.len(), 45); + assert_eq!(not_run[0], "Coverage (mosdepth)"); + assert!(!not_run.contains(&"Reads kept by alignment".to_owned())); + assert!(!not_run.contains(&"Carrier list".to_owned())); + assert!(!not_run.contains(&"Couple carrier screen".to_owned())); + let lines = texts(at(&v, &["not_assessed"])); + let mut expected: Vec<&str> = NOT_ASSESSED.to_vec(); + expected.extend([ + PRS_NOT_ADJUSTED, + HBA_NOT_ASSESSED, + MEI_NOT_ASSESSED, + VNTYPER_NOT_ASSESSED, + MCA_NOT_ASSESSED, + ]); + assert_eq!(lines, expected); +} + +#[test] +fn a_failed_step_says_so_under_not_run() { + let d = Dir::new("post-failed"); + d.put( + "S/logs/run_status.tsv", + "meta\tstarted_epoch\t1577836800\nstep\t06\tfailed (exit 1)\nstep\t02\tfailed (exit 1)\n", + ); + let v = summary(&d); + let not_run = texts(at(&v, &["not_run"])); + assert!(not_run.contains(&"ClinVar screen (step 06 failed)".to_owned())); + // The read count is no step of its own, failed or not; the carrier + // list shares step 06 with ClinVar. + assert!(!not_run.iter().any(|x| x.starts_with("Reads kept"))); + assert!(!not_run.iter().any(|x| x.starts_with("Carrier list"))); + assert_eq!(word(at(&v, &["sections", "carrier", "state"])), "failed"); +} + +#[test] +fn step_35s_hba_count_assesses_hba() { + let d = Dir::new("post-assessed"); + d.put( + "S/paralogs/S_hba_copy_number.tsv", + "metric\tvalue\nstatus\tcalled\ncopies\t4\nreading\tfour\n", + ); + let v = summary(&d); + assert_eq!(word(at(&v, &["sections", "parascopy", "state"])), "ok"); + let lines = texts(at(&v, &["not_assessed"])); + assert!(!lines.contains(&HBA_NOT_ASSESSED.to_owned())); + assert!(lines.contains(&VNTYPER_NOT_ASSESSED.to_owned())); +} + +#[test] +fn a_stale_vntyper_reading_is_not_assessed() { + // Only an ok section counts for VNtyper; any other state leaves the line. + let mk = |state: State, status: &str| { + let mut s = Section::new( + GROUP_C + .iter() + .copied() + .find(|s| s.key == "vntyper") + .unwrap(), + Ok(None), + None, + ); + s.state = state; + s.data = Some(Data::Json( + loads(&format!(r#"{{"status": "{status}"}}"#)).unwrap(), + )); + s + }; + let lines = not_assessed(&[mk(State::Ok, "positive")]); + assert!(!lines.contains(&VNTYPER_NOT_ASSESSED)); + let lines = not_assessed(&[mk(State::Stale, "positive")]); + assert!(lines.contains(&VNTYPER_NOT_ASSESSED)); + let lines = not_assessed(&[mk(State::Ok, "failed")]); + assert!(lines.contains(&VNTYPER_NOT_ASSESSED)); +} + +#[test] +fn paraphase_needs_both_hba_genes_called() { + let paralogs = |genes: &str| { + let mut s = Section::new( + GROUP_C + .iter() + .copied() + .find(|s| s.key == "paralogs") + .unwrap(), + Ok(None), + None, + ); + s.state = State::Ok; + s.data = Some(Data::Json( + loads(&format!(r#"{{"genes": {genes}}}"#)).unwrap(), + )); + s + }; + let both = r#"[{"region": "hba", "gene": "HBA1", "status": "called"}, + {"region": "hba", "gene": "HBA2", "status": "called"}]"#; + let one = r#"[{"region": "hba", "gene": "HBA1", "status": "called"}, + {"region": "hba", "gene": "HBA2", "status": "no call"}]"#; + // A row without a region, the shape that stopped the Python collector. + let odd = r#"[{"gene": "HBA1", "status": "called"}, 7]"#; + assert!(!not_assessed(&[paralogs(both)]).contains(&HBA_NOT_ASSESSED)); + assert!(not_assessed(&[paralogs(one)]).contains(&HBA_NOT_ASSESSED)); + assert!(not_assessed(&[paralogs(odd)]).contains(&HBA_NOT_ASSESSED)); +} + +#[test] +fn a_part_of_the_post_pass_that_fails_stays_in_its_section() { + // An error comes back as the section's key and why. + assert_eq!(group_c::contain("cnvpytor", || Ok(())), None); + assert_eq!( + group_c::contain("expansions", || Err("no table".into())), + Some(("expansions", "no table".to_owned())) + ); + // The section is then unreadable, with no data, and names its file. + let mut s = Section::new( + GROUP_C + .iter() + .copied() + .find(|s| s.key == "locityper") + .unwrap(), + Ok(Some(( + "locityper/S_locityper.tsv".into(), + 1_600_000_000.0, + Data::Json(Value::Object(indexmap::IndexMap::new())), + ))), + None, + ); + s.make_unreadable("no table"); + let j = s.to_json(); + assert_eq!(word(at(&j, &["state"])), "unreadable"); + assert_eq!(word(at(&j, &["note"])), "S_locityper.tsv: no table"); + assert_eq!(at(&j, &["source"]), &Value::Null); + assert_eq!(at(&j, &["data"]), &Value::Object(indexmap::IndexMap::new())); +} + +#[test] +fn an_infinite_telomere_read_length_keeps_the_summary() { + // The Python collector stopped on this table with an OverflowError and + // wrote no summary.json; here only the telomere section is unreadable. + let d = Dir::new("post-inf"); + d.put( + "S/telomere/S/S/S_summary.tsv", + "PID\tsample\ttotal_reads\tread_lengths\tintratel_reads\ttel_content\nS\ttumor\t1000\tinf\t40\t1.5\n", + ); + d.put( + "S/mosdepth/S.mosdepth.summary.txt", + "chrom\tlength\tbases\tmean\tmin\tmax\ntotal\t100\t3000\t30\t0\t60\n", + ); + let v = summary(&d); + assert_eq!( + word(at(&v, &["sections", "telomere", "state"])), + "unreadable" + ); + assert_eq!(word(at(&v, &["sections", "coverage", "state"])), "ok"); +} + +#[test] +fn a_declared_female_drops_cnvpytor_chry_calls_across_groups() { + // The declared sex comes from step 33's table (group a) and changes + // CNVpytor's count (group c). + let d = Dir::new("post-sex"); + d.put( + "S/qc/S_sample_qc.tsv", + "key\tvalue\nsample\tS\ndeclared_sex\tfemale\ninferred_sex\tfemale\nsex_check\tok\n", + ); + d.put( + "S/cnvpytor/S_cnvs.txt", + "deletion\tchrY:1-200000\t200000\t0.4\t0.00001\t1\t1\t1\t0.00001\t0.00001\t0.1\n\ +deletion\tchr2:1-200000\t200000\t0.4\t0.00001\t1\t1\t1\t0.00001\t0.00001\t0.1\n", + ); + let v = summary(&d); + assert_eq!(word(at(&v, &["run", "declared_sex"])), "female"); + let cnv = at(&v, &["sections", "cnvpytor"]); + assert_eq!(word(at(cnv, &["state"])), "ok"); + assert_eq!(word(at(cnv, &["data", "declared_sex"])), "female"); + assert!(has(at(cnv, &["data", "dropped"]), "chry_female")); +} + +const PREV: &str = r#"{"schema_version": 2, "generated_utc": "2020-01-01 00:00 UTC", + "sections": {"coverage": {"state": "ok", "data": {}}, "clinvar": {"state": "missing", "data": {}}}}"#; + +#[test] +fn new_since_compares_with_the_newest_kept_run() { + let d = Dir::new("post-history"); + d.put("S/history/20200101T000000/summary.json", PREV); + // A newer folder without a summary.json is a half-copied one: passed over. + d.put("S/history/20990101T000000/run_manifest.tsv", "run\tx\ty\n"); + let v = summary(&d); + assert_eq!( + at(&v, &["new_since", "previous_utc"]), + &Value::Str("2020-01-01 00:00 UTC".into()) + ); + assert!(texts(at(&v, &["new_since", "sections"])).contains(&"coverage".to_owned())); +} + +#[test] +fn a_previous_run_that_cannot_be_read_leaves_new_since_out() { + for body in ["{not json", "[]", r#"{"sections": 7}"#] { + let d = Dir::new("post-history-bad"); + d.put("S/history/20200101T000000/summary.json", body); + let v = summary(&d); + assert!(!has(&v, "new_since"), "{body}"); + } +} + +#[test] +fn the_recheck_block_comes_from_its_state_file() { + let d = Dir::new("post-recheck"); + d.put( + "S/recheck/state.json", + r#"{"checked_utc": "2026-01-02 03:04:05 UTC", "items": [{"item": "clinvar", "class": "newer", "x": 1}, 5], + "affected_steps": ["06"]}"#, + ); + let v = summary(&d); + assert_eq!( + word(at(&v, &["recheck", "checked_utc"])), + "2026-01-02 03:04:05 UTC" + ); + assert_eq!(word(at(&v, &["recheck", "items", "0", "item"])), "clinvar"); + // A check older than the run's manifest is of an earlier run. + d.put( + "S/run_manifest.tsv", + "run\twritten_utc\t2027-01-01 00:00:00 UTC\n", + ); + assert!(!has(&summary(&d), "recheck")); +} + +#[test] +fn without_only_the_command_writes_summary_json_in_the_folder() { + let d = Dir::new("post-cli"); + std::fs::create_dir_all(d.path("S")).unwrap(); + let dir = d.path("S"); + let assets = d.path("assets"); + let (code, out, err) = run( + crate::collect::cli, + &["--sample", "S", "--sample-dir", &dir, "--assets", &assets], + ); + assert_eq!((code, err.as_str()), (0, "")); + let target = format!("{dir}/summary.json"); + assert_eq!(out, format!("Summary written: {target}\n")); + let v = loads(&d.read("S/summary.json")).unwrap(); + assert!(has(&v, "not_assessed")); + // A part of the sections never takes summary.json's place. + let (code, _, err) = run( + crate::collect::cli, + &["--sample", "S", "--sample-dir", &dir, "--only", "a"], + ); + assert_eq!(code, 2); + assert!(err.contains("--only needs --out"), "{err}"); +} diff --git a/rust/crates/og-summary/src/collect/tests.rs b/rust/crates/og-summary/src/collect/tests.rs index 75031f74..4f622c77 100644 --- a/rust/crates/og-summary/src/collect/tests.rs +++ b/rust/crates/og-summary/src/collect/tests.rs @@ -448,22 +448,22 @@ fn the_command_line() { let text = d.read("out.json"); assert!(text.starts_with("{\n \"schema_version\": 2,\n \"sample\": \"S\",\n")); assert!(text.ends_with("}\n")); - // --only is required until every section is ported. + // Without --only it writes every section and the post pass. let (code, _, err) = run( cli, &["--sample", "S", "--sample-dir", &d.path("S"), "--out", &out], ); - assert_eq!(code, 2); - assert!(err.contains("required: --only"), "{err}"); - // So is --out: a part of the sections must not land on the sample's - // complete summary.json. + assert_eq!((code, err.as_str()), (0, "")); + assert!(d.read("out.json").contains("\n \"not_assessed\": [\n")); + // --only needs --out: a part of the sections must not land on the + // sample's complete summary.json. std::fs::write(d.path("S/summary.json"), "complete\n").unwrap(); let (code, _, err) = run( cli, &["--sample", "S", "--sample-dir", &d.path("S"), "--only", "a"], ); assert_eq!(code, 2); - assert!(err.contains("required: --out"), "{err}"); + assert!(err.contains("--only needs --out"), "{err}"); assert_eq!(d.read("S/summary.json"), "complete\n"); let (code, _, err) = run( cli, diff --git a/rust/crates/og-summary/src/lib.rs b/rust/crates/og-summary/src/lib.rs index ac404041..4aa53eec 100644 --- a/rust/crates/og-summary/src/lib.rs +++ b/rust/crates/og-summary/src/lib.rs @@ -7,8 +7,8 @@ //! table from pgsc_calc's output (`prs/_prs_summary.tsv`, //! `ancestry/_ancestry.tsv`). //! - [`mito`]: `og mito annotate|deletions|copies`, step 20's chrM tables. -//! - [`collect`]: `og summary collect`, summary.json, ported one group of -//! sections at a time beside the Python collector (`--only a`). +//! - [`collect`]: `og summary collect`, the whole summary.json, or one +//! group with `--only a|b|c`, beside the Python collector (no caller yet). //! //! Each tool exits 0 when it wrote its table, 1 with `ERROR: ` //! when an input cannot be read or the table cannot be written, and 2 with diff --git a/rust/og/src/main.rs b/rust/og/src/main.rs index a9be16ab..7603ed02 100644 --- a/rust/og/src/main.rs +++ b/rust/og/src/main.rs @@ -50,7 +50,7 @@ const USAGE: &str = "usage: og [args] og prs format [args] og prs table [args] og mito annotate|deletions|copies [args] - og summary collect --only {a,b,c} [args] + og summary collect [--only {a,b,c}] [args] og roh summary --sample SAMPLE --roh ROH [--out OUT] og chip prepare|report|check-table [args] og cn bed|call|intervals|check-bam|marker-regions|marker-read [args] diff --git a/tests/parity/cases.tsv b/tests/parity/cases.tsv index 40f24ffe..7fbfa608 100644 --- a/tests/parity/cases.tsv +++ b/tests/parity/cases.tsv @@ -870,6 +870,15 @@ summary-collect cb-notbi gen:summary-clinical.sh,summary-collect/oracle.sh,summa summary-collect cb-notgz gen:summary-clinical.sh,summary-collect/oracle.sh,summary-collect/cb_vep.vcf.gz,summary-collect/cb_vep.vcf.gz.tbi --sample S --sample-dir cb-notgz --assets assets --out cb-notgz.json --only b cb-notgz.json json:paths=schema_version,sample,run,databases,manifest,sections.clinvar,sections.carrier,sections.pharmcat,sections.cpic,sections.pypgx,sections.cyrius,sections.cpsr,sections.clinical,sections.slivar,cyp2d6 PYTHON_IMAGE summary-collect cb-empty gen:summary-clinical.sh,summary-collect/oracle.sh,summary-collect/cb_vep.vcf.gz,summary-collect/cb_vep.vcf.gz.tbi --sample S --sample-dir cb-empty --assets assets --out cb-empty.json --only b cb-empty.json json:paths=schema_version,sample,run,databases,manifest,sections.clinvar,sections.carrier,sections.pharmcat,sections.cpic,sections.pypgx,sections.cyrius,sections.cpsr,sections.clinical,sections.slivar,cyp2d6 PYTHON_IMAGE summary-collect cb-no-assets gen:summary-clinical.sh,summary-collect/oracle.sh,summary-collect/cb_vep.vcf.gz,summary-collect/cb_vep.vcf.gz.tbi --sample S --sample-dir cb-full --assets nowhere --out cb-no-assets.json --only b cb-no-assets.json json:paths=schema_version,sample,run,databases,manifest,sections.clinvar,sections.carrier,sections.pharmcat,sections.cpic,sections.pypgx,sections.cyrius,sections.cpsr,sections.clinical,sections.slivar,cyp2d6 PYTHON_IMAGE +summary-collect all gen:summary-collect-full.sh,summary-collect/oracle.sh,summary-collect/cb_vep.vcf.gz,summary-collect/cb_vep.vcf.gz.tbi --sample S --sample-dir all --assets assets --out all.json all.json json:mask=generated_utc PYTHON_IMAGE +summary-collect all-nf gen:summary-collect-full.sh,summary-collect/oracle.sh,summary-collect/cb_vep.vcf.gz,summary-collect/cb_vep.vcf.gz.tbi --sample S --sample-dir nf --assets assets --out all-nf.json all-nf.json json:mask=generated_utc PYTHON_IMAGE +summary-collect all-bad gen:summary-collect-full.sh,summary-collect/oracle.sh,summary-collect/cb_vep.vcf.gz,summary-collect/cb_vep.vcf.gz.tbi --sample S --sample-dir bad --assets assets --out all-bad.json all-bad.json json:mask=generated_utc PYTHON_IMAGE +summary-collect all-failed gen:summary-collect-full.sh,summary-collect/oracle.sh,summary-collect/cb_vep.vcf.gz,summary-collect/cb_vep.vcf.gz.tbi --sample S --sample-dir failed --assets assets --out all-failed.json all-failed.json json:mask=generated_utc PYTHON_IMAGE +summary-collect all-empty gen:summary-collect-full.sh,summary-collect/oracle.sh,summary-collect/cb_vep.vcf.gz,summary-collect/cb_vep.vcf.gz.tbi --sample S --sample-dir empty --assets assets --out all-empty.json all-empty.json json:mask=generated_utc PYTHON_IMAGE +summary-collect all-history gen:summary-collect-full.sh,summary-collect/oracle.sh,summary-collect/cb_vep.vcf.gz,summary-collect/cb_vep.vcf.gz.tbi --sample S --sample-dir history --assets assets --out all-history.json all-history.json json:mask=generated_utc PYTHON_IMAGE +summary-collect all-history-bad gen:summary-collect-full.sh,summary-collect/oracle.sh,summary-collect/cb_vep.vcf.gz,summary-collect/cb_vep.vcf.gz.tbi --sample S --sample-dir history-bad --assets assets --out all-history-bad.json all-history-bad.json json:mask=generated_utc PYTHON_IMAGE +summary-collect all-inf gen:summary-collect-full.sh,summary-collect/oracle.sh,summary-collect/cb_vep.vcf.gz,summary-collect/cb_vep.vcf.gz.tbi --sample S --sample-dir inf --assets assets --out all-inf.json all-inf.json json:mask=generated_utc PYTHON_IMAGE +summary-collect all-default-out gen:summary-collect-full.sh,summary-collect/oracle.sh,summary-collect/cb_vep.vcf.gz,summary-collect/cb_vep.vcf.gz.tbi --sample S --sample-dir history-bad --assets assets history-bad/summary.json json:mask=generated_utc PYTHON_IMAGE y-tree-positions small gen:y-inputs.sh --tree-dir t pos.txt pos.txt bytes PYTHON_IMAGE y-tree-positions big gen:y-inputs.sh --tree-dir ./big/ pos.txt pos.txt bytes PYTHON_IMAGE y-tree-positions odd gen:y-inputs.sh --tree-dir odd pos.txt pos.txt bytes PYTHON_IMAGE diff --git a/tests/parity/gen/summary-collect-full.sh b/tests/parity/gen/summary-collect-full.sh new file mode 100644 index 00000000..accdcdf7 --- /dev/null +++ b/tests/parity/gen/summary-collect-full.sh @@ -0,0 +1,107 @@ +#!/usr/bin/env bash +# summary-collect-full.sh: writes the sample folders of the whole-file +# summary-collect parity cases (og summary collect without --only) in the +# current directory, the same every time. It runs the generators of the +# three groups (summary-collect.sh, summary-clinical.sh and +# summary-collect-c.sh) and joins their folders, so one folder holds every +# section and the post pass reads across the groups: the repeat readings +# with the CPIC state, Locityper's primary calls with pypgx and Cyrius, +# CNVpytor's chrY filter with step 33's declared sex, the cyp2d6 block, +# not_run and not_assessed. On top: a run with failed steps, an empty +# folder, previous runs kept in history/ (one half-copied, one whose +# summary is no JSON) with a database re-check, and a telomere table whose +# infinite read length stopped the Python collector (a golden: og keeps +# the summary and marks that one section unreadable). +# py/ holds the modules the oracle imports at the oracle's commit, with the +# real run_delta.py for recheck and new_since. The case stages +# cb_vep.vcf.gz and its index beside the folders. +# scripts/ci/oracle-diff.sh runs it. +set -euo pipefail +export LC_ALL=C TZ=UTC +here=$(cd "$(dirname "$0")" && pwd) +root=$(cd "${here}/../../.." && pwd) +ORACLE_SHA=98aa9252ff77825a8a797d388b449931fa9a234e + +put() { mkdir -p "$(dirname "$1")"; printf '%b' "$2" > "$1"; at "$NEW" "$1"; } +at() { touch -t "$1" "${@:2}"; } # [[CC]YY]MMDDhhmm[.SS], in UTC +NEW=202103040506.07 # after the run of the three groups' folders + +for g in a b c; do mkdir -p "parts/$g"; done +(cd parts/a && bash "${here}/summary-collect.sh") +(cd parts/b && bash "${here}/summary-clinical.sh") +(cd parts/c && bash "${here}/summary-collect-c.sh") + +# merge DEST SRC...: the folders' files, the later folder's file winning; +# times and symlinks kept. +merge() { + local dest=$1 src + shift + mkdir -p "$dest" + for src in "$@"; do cp -pPR "parts/${src}/." "$dest"/; done +} + +mkdir -p assets py pymods +for g in a b c; do cp -pR "parts/${g}/assets/." assets/; done +cp -pR parts/b/pymods/. pymods/ +cp -pR parts/c/py/. py/ +git -C "$root" cat-file -e "${ORACLE_SHA}^{commit}" 2> /dev/null \ + || git -C "$root" fetch -q --depth=1 origin "$ORACLE_SHA" +git -C "$root" show "${ORACLE_SHA}:bin/run_delta.py" > py/run_delta.py + +# all: every section of the three groups, the bash names. +merge all a/full b/cb-full c/full +# nf: the Nextflow names, steps failed and from an earlier run. +merge nf a/nf b/cb-nf c/nf +# bad: an unreadable file in sections of each group. +merge bad a/bad-a b/cb-bad c/bad1 +# failed: steps that failed and wrote nothing, under not_run. +merge failed a/failed b/cb-failed +mkdir -p empty + +# history: all, with three earlier runs kept. The newest has no summary +# (a half-copied folder) and is passed over; the next is compared: its +# ClinVar, CPIC, pypgx and PRS results and its section states differ, and +# so do the inputs of its manifest. A database re-check newer than the +# run's manifest. +merge history a/full b/cb-full c/full +put history/run_manifest.tsv "run\twritten_utc\t2021-03-04 05:06:07 UTC\nimage\tDEEPVARIANT_IMAGE\tv1.10.0\ndata\tclinvar_file_date\t2021-03-01\n" +put history/history/20200101T000000/summary.json '{"schema_version": 2, "sample": "S", "generated_utc": "2019-12-01 00:00 UTC", "sections": {}}' +put history/history/20210101T000000/summary.json '{"schema_version": 2, "sample": "S", "generated_utc": "2021-01-01 00:00 UTC", + "sections": { + "coverage": {"title": "Coverage (mosdepth)", "state": "missing", "data": {}}, + "clinvar": {"title": "ClinVar screen", "state": "ok", "data": {"hits": [ + {"chrom": "chr1", "pos": "45331556", "ref": "C", "alt": "T", "gene": "MUTYH", "significance": "Likely_pathogenic", "stars": 1, "review_status": "x", "genotype": "1/1"}, + {"chrom": "chr9", "pos": "1", "ref": "A", "alt": "G", "gene": "X", "significance": "Pathogenic", "stars": 2}]}}, + "cpic": {"title": "CPIC", "state": "ok", "data": {"genes": [{"gene": "CYP2D6", "diplotype": "*1/*1", "phenotype": "Normal"}]}}, + "pypgx": {"title": "pypgx", "state": "ok", "data": {"cyp2d6": "*1/*1"}}, + "prs": {"title": "PRS", "state": "ok", "data": {"scores": [{"pgs_id": "PGS000001", "condition": "x", "percentile": 10}]}}, + "mei": {"title": "Mobile element insertions (MEGAnE)", "state": "ok", "data": {}}}, + "cyp2d6": {"consensus": {"result": "*1/*1"}}}\n' +put history/history/20210101T000000/run_manifest.tsv "run\twritten_utc\t2021-01-01 00:00:00 UTC\nimage\tDEEPVARIANT_IMAGE\tv1.9.0\ndata\tclinvar_file_date\t2020-12-01\n" +mkdir -p history/history/20220101T000000 +put history/history/20220101T000000/run_manifest.tsv "run\twritten_utc\t2022-01-01 00:00:00 UTC\n" +put history/recheck/state.json '{"checked_utc": "2021-03-05 00:00:00 UTC", "items": [ + {"item": "clinvar", "label": "ClinVar", "class": "newer-release", "ran_with": "2021-03-01", "upstream": "2021-03-04", "action": "rerun step 06", "extra": 1}, + {"item": "vep", "label": "VEP", "class": "no-newer-release"}, 7], + "affected_steps": ["06"], "rerun": {"command": "run-all.sh"}}\n' + +# history-bad: the only kept run has a summary that is no JSON, so nothing +# is compared; a re-check older than the run's manifest is of an earlier +# run and left out. +mkdir -p history-bad +put history-bad/run_manifest.tsv "run\twritten_utc\t2021-03-04 05:06:07 UTC\n" +put history-bad/history/20210101T000000/summary.json '{"schema_version": 2,' +put history-bad/recheck/state.json '{"checked_utc": "2021-03-01 00:00:00 UTC", "items": []}\n' + +# inf: a telomere table whose read length is infinite, beside the +# coverage, step 33 and repeat sections. The Python collector stopped with +# an uncaught OverflowError and wrote no summary; og writes it with the +# telomere section unreadable (a golden). +mkdir -p inf +for f in mosdepth/S.mosdepth.summary.txt qc/S_sample_qc.tsv logs/run_status.tsv; do + mkdir -p "inf/$(dirname "$f")" && cp -p "parts/a/full/$f" "inf/$f" +done +cp -pR parts/c/full/expansion_hunter inf/ +put inf/telomere/S/S/S_summary.tsv "PID\tsample\ttotal_reads\tread_lengths\tintratel_reads\ttel_content\nS\ttumor\t1000\tinf\t40\t1.5\n" + +rm -rf parts diff --git a/tests/parity/golden/summary-collect/all-inf/all-inf.json b/tests/parity/golden/summary-collect/all-inf/all-inf.json new file mode 100644 index 00000000..99b10ea8 --- /dev/null +++ b/tests/parity/golden/summary-collect/all-inf/all-inf.json @@ -0,0 +1,820 @@ +{ + "schema_version": 2, + "sample": "S", + "generated_utc": "2026-11-13 23:22 UTC", + "run": { + "status_file": true, + "started_utc": "2020-01-01 00:00 UTC", + "declared_sex": "male" + }, + "databases": { + "clinvar_release": null, + "cpsr_clinvar_release": null, + "hla_database": null + }, + "manifest": [], + "sections": { + "coverage": { + "title": "Coverage (mosdepth)", + "step": "16b", + "state": "ok", + "source": "mosdepth/S.mosdepth.summary.txt", + "file_date": "2021-03-04 05:06 UTC", + "note": null, + "data": { + "mean_depth": 31.25 + } + }, + "callability": { + "title": "Gene callability (mosdepth)", + "step": "16b", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "sex_check": { + "title": "Sex check (indexcov)", + "step": "16", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "sample_qc": { + "title": "Sample identity and contamination (somalier, VerifyBamID2)", + "step": "33", + "state": "ok", + "source": "qc/S_sample_qc.tsv", + "file_date": "2021-03-04 05:06 UTC", + "note": null, + "data": { + "sample": "S", + "inferred_sex": "male", + "sex_check": "ok", + "x_depth_ratio": "0.51", + "y_depth_ratio": "0.49", + "x_sites": "400", + "freemix": "0.0010", + "declared_sex": "male", + "mosaic_call": "not_detected", + "mosaic_depth_ratio": "0.98", + "mosaic_depth_ratio_expected": "1", + "mosaic_loss_depth": "0.02", + "mosaic_loss_depth_low": "-0.01", + "mosaic_loss_depth_high": "0.05", + "mosaic_loss_af": "NA", + "mosaic_floor": "0.1", + "mosaic_x_het_sites": "0", + "karyotype": null + } + }, + "mosaic_sex_loss": { + "title": "Mosaic loss of chromosome Y or X", + "step": "33", + "state": "ok", + "source": "qc/S_sample_qc.tsv", + "file_date": "2021-03-04 05:06 UTC", + "note": null, + "data": { + "call": "not_detected", + "depth_ratio": 0.98, + "depth_ratio_expected": 1.0, + "loss_depth": 0.02, + "loss_depth_low": -0.01, + "loss_depth_high": 0.05, + "loss_af": null, + "floor": 0.1, + "x_het_sites": 0, + "loss_af_low": null, + "loss_af_high": null + } + }, + "read_count": { + "title": "Reads kept by alignment", + "step": "02", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "variants": { + "title": "Variant calling", + "step": "03", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "clinvar": { + "title": "ClinVar screen", + "step": "06", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "couple": { + "title": "Couple carrier screen", + "step": "39", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "carrier": { + "title": "Carrier list", + "step": "06", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "pharmcat": { + "title": "PharmCAT", + "step": "07", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "cpic": { + "title": "CPIC drug-gene recommendations", + "step": "27", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "pypgx": { + "title": "pypgx", + "step": "32", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "cyrius": { + "title": "Cyrius (CYP2D6)", + "step": "21", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "parascopy": { + "title": "SMN1/SMN2 copy number (Parascopy, short reads)", + "step": "35", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "paralogs": { + "title": "Paralog genes (Paraphase, PacBio HiFi)", + "step": "35b", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "hla": { + "title": "HLA typing (T1K)", + "step": "08", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "prs": { + "title": "Polygenic risk scores", + "step": "25", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "ancestry_fine": { + "title": "Fine-grained ancestry (ADMIXTURE, RFMix)", + "step": "26b", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "manta": { + "title": "Structural variants (Manta)", + "step": "04", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "delly": { + "title": "Structural variants (Delly)", + "step": "19", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "cnvpytor": { + "title": "Copy number (CNVpytor)", + "step": "18", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "sv_consensus": { + "title": "SV consensus merge", + "step": "22", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "annotsv": { + "title": "Structural variants, class 4 and 5 (AnnotSV)", + "step": "05", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "expansions": { + "title": "Repeat expansions (ExpansionHunter)", + "step": "09", + "state": "ok", + "source": "expansion_hunter/S_eh_stranger.vcf", + "file_date": "2021-03-04 05:06 UTC", + "note": null, + "data": { + "records": 9, + "key_loci": [ + { + "locus": "HTT", + "repeat_count": "17/20" + }, + { + "locus": "FMR1", + "repeat_count": "58" + }, + { + "locus": "C9ORF72", + "repeat_count": "." + }, + { + "locus": "ATXN1", + "repeat_count": "30/37" + }, + { + "locus": "DMPK", + "repeat_count": "not in output" + } + ], + "loci": [ + { + "locus": "HTT", + "repeat_count": "17/20", + "status": "normal,normal", + "normal_max": "26", + "pathologic_min": "40", + "support": "SPANNING/SPANNING", + "adsp": "10/12", + "ci": "17-17/20-20", + "sizing": "sized", + "allele_sizing": "sized/sized", + "zone": "", + "zone_label": "", + "zone_citation": "" + }, + { + "locus": "FMR1", + "repeat_count": "58", + "status": "premutation", + "normal_max": "44", + "pathologic_min": "200", + "support": "SPANNING", + "adsp": "1", + "ci": "55-60", + "sizing": "low_confidence", + "allele_sizing": "low_confidence", + "zone": "", + "zone_label": "", + "zone_citation": "", + "motif": { + "verdict": "interrupted", + "note": "AGG", + "dominant_motif": "CGG", + "dominant_class": ".", + "dominant_share": ".", + "motif_shares": ".", + "repeat_reads": "." + } + }, + { + "locus": "C9ORF72", + "repeat_count": ".", + "status": "", + "normal_max": "", + "pathologic_min": "", + "support": ".", + "adsp": ".", + "ci": ".", + "sizing": "no_call", + "allele_sizing": "no_call", + "zone": "", + "zone_label": "", + "zone_citation": "" + }, + { + "locus": "ATXN1", + "repeat_count": "30/37", + "status": "normal", + "normal_max": "35", + "pathologic_min": "39", + "support": "SPANNING/FLANKING", + "adsp": "4/0", + "ci": "30-30/35-40", + "sizing": "flanking_only", + "allele_sizing": "sized/flanking_only", + "zone": "", + "zone_label": "", + "zone_citation": "" + }, + { + "locus": "CFTR_T", + "repeat_count": "5/7", + "status": "", + "normal_max": "", + "pathologic_min": "", + "support": "SPANNING/SPANNING", + "adsp": "8/9", + "ci": "5-5/7-7", + "sizing": "sized", + "allele_sizing": "sized/sized", + "zone": "", + "zone_label": "", + "zone_citation": "" + }, + { + "locus": "CFTR_TG", + "repeat_count": "11/12", + "status": "", + "normal_max": "", + "pathologic_min": "", + "support": "SPANNING/SPANNING", + "adsp": "8/9", + "ci": "11-11/12-12", + "sizing": "sized", + "allele_sizing": "sized/sized", + "zone": "", + "zone_label": "", + "zone_citation": "" + }, + { + "locus": "EXTRA1", + "repeat_count": "8/9", + "status": "", + "normal_max": "", + "pathologic_min": "", + "support": "SPANNING", + "adsp": "1", + "ci": "8-8/9-9", + "sizing": "low_confidence", + "allele_sizing": "low_confidence/low_confidence", + "zone": "", + "zone_label": "", + "zone_citation": "", + "not_graded": "no published range" + }, + { + "locus": "RFC1", + "repeat_count": "500/600", + "status": "full_mutation", + "normal_max": "", + "pathologic_min": "400", + "support": "INREPEAT/INREPEAT", + "adsp": "0/0", + "ci": "400-700/500-800", + "sizing": "in_repeat", + "allele_sizing": "in_repeat/in_repeat", + "zone": "", + "zone_label": "", + "zone_citation": "", + "rfc1_motif": { + "verdict": "pathogenic", + "note": "AAGGG dominant", + "dominant_motif": "AAGGG", + "dominant_class": "pathogenic", + "dominant_share": "0.9", + "motif_shares": "AAGGG:0.9", + "repeat_reads": "40" + } + }, + { + "locus": "UGT1A1", + "repeat_count": "7/8", + "status": "normal", + "normal_max": "", + "pathologic_min": "", + "support": "SPANNING/SPANNING", + "adsp": "5/6", + "ci": "7-7/8-8", + "sizing": "sized", + "allele_sizing": "sized/sized", + "zone": "", + "zone_label": "", + "zone_citation": "" + } + ], + "stranger": true, + "stranger_outdated": false, + "unsized": [ + "FMR1", + "C9ORF72", + "ATXN1", + "EXTRA1", + "RFC1" + ], + "intermediate": [], + "readings": [ + { + "locus": "CFTR", + "repeat": "intron 9 (legacy IVS8) TG and poly-T tract", + "source": "Deignan et al. 2020, PMID 32404922", + "status": "read", + "genotype": "TG11/TG12 5T/7T", + "pairings": [ + "TG11-5T/TG12-7T", + "TG12-5T/TG11-7T" + ], + "tg": [ + 11, + 12 + ], + "poly_t": [ + 5, + 7 + ], + "five_t": true, + "phase": "not_read", + "r117h": ".", + "reading": "5T: 1 copy with the 5T poly-T tract, TG11 or TG12 on the 5T copy, not told by the counts (TG11-5T/TG12-7T or TG12-5T/TG11-7T). 5T lowers the splicing of exon 10 and is a variable-penetrance allele: with TG12 or TG13 on the same copy it more often causes CBAVD or another CFTR-related disorder when the other copy carries a CF-causing variant. R117H was not read (no step 42 traits row for rs78655421 was called), so whether it sits on the 5T copy is not known." + }, + { + "locus": "UGT1A1", + "repeat": "promoter TA repeat", + "source": "Gammal et al. 2016, PMID 26417955", + "status": "read", + "genotype": "*1/*28", + "ta_units": [ + 7, + 8 + ], + "pharmcat": ".", + "agreement": "not_compared", + "reading": "*1/*28 (TA6/TA7, 7/8 TA units as ExpansionHunter counts them): 1 decreased-function allele (*28 and *37 are decreased function). Not compared with PharmCAT (no unambiguous UGT1A1 call from step 27)." + }, + { + "locus": "SLC6A4", + "repeat": "5-HTTLPR promoter length polymorphism", + "status": "not_tested", + "genotype": ".", + "reading": "not tested by any step. CPIC's 2023 SSRI guideline (Bousman et al. 2023, PMID 37032427) found that the data on SLC6A4 do not support its use in antidepressant prescribing, so a result would carry no recommendation.", + "source": "Bousman et al. 2023, PMID 37032427" + } + ], + "flagged": [ + { + "locus": "FMR1", + "repeat_count": "58", + "status": "premutation", + "sizing": "low_confidence", + "motif": { + "verdict": "interrupted", + "note": "AGG", + "dominant_motif": "CGG", + "dominant_class": ".", + "dominant_share": ".", + "motif_shares": ".", + "repeat_reads": "." + } + }, + { + "locus": "RFC1", + "repeat_count": "500/600", + "status": "full_mutation", + "sizing": "in_repeat", + "rfc1_motif": { + "verdict": "pathogenic", + "note": "AAGGG dominant", + "dominant_motif": "AAGGG", + "dominant_class": "pathogenic", + "dominant_share": "0.9", + "motif_shares": "AAGGG:0.9", + "repeat_reads": "40" + } + } + ], + "no_status": 3, + "not_graded": { + "EXTRA1": "no published range" + } + } + }, + "telomere": { + "title": "Telomere content (TelomereHunter)", + "step": "10", + "state": "unreadable", + "source": null, + "file_date": null, + "note": "OverflowError: cannot round", + "data": {} + }, + "roh": { + "title": "Runs of homozygosity", + "step": "11", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "haplogroup": { + "title": "Mitochondrial haplogroup", + "step": "12", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "y_haplogroup": { + "title": "Y-chromosome haplogroup (Yleaf)", + "step": "37", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "y_tree": { + "title": "Y placement on the current public tree", + "step": "37", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "y_str": { + "title": "Y-STR markers (HipSTR)", + "step": "45", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "consumer_export": { + "title": "Raw-data export (23andMe v5 layout)", + "step": "40", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "bloodtype": { + "title": "Blood groups (predicted)", + "step": "41", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "lpa": { + "title": "Lp(a): LPA KIV-2 repeat count", + "step": "46", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "vntyper": { + "title": "MUC1 VNTR frameshifts (VNtyper)", + "step": "52", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "locityper": { + "title": "Locityper second opinion at hard loci", + "step": "53", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "mito": { + "title": "Mitochondrial variants", + "step": "20", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "traits": { + "title": "Traits panel", + "step": "42", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "archaic": { + "title": "Archaic ancestry (IBDmix)", + "step": "43", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "ancient": { + "title": "Ancient populations (AADR PCA)", + "step": "44", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "viral": { + "title": "Viral DNA in the unmapped reads", + "step": "47", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "exomiser": { + "title": "Symptom-driven ranking (Exomiser)", + "step": "48", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "chip": { + "title": "Clonal haematopoiesis (CHIP)", + "step": "49", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "mei": { + "title": "Mobile element insertions (MEGAnE)", + "step": "50", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "mosaic": { + "title": "Mosaic chromosomal alterations (MoChA)", + "step": "51", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "cpsr": { + "title": "Cancer predisposition (CPSR)", + "step": "17", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "clinical": { + "title": "Clinical variant filter", + "step": "23", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + }, + "slivar": { + "title": "Variant prioritization (slivar)", + "step": "31", + "state": "missing", + "source": null, + "file_date": null, + "note": null, + "data": {} + } + }, + "cyp2d6": { + "calls": { + "PharmCAT": null, + "pypgx": null, + "Cyrius": null + }, + "agree": null, + "consensus": null, + "probable": null + }, + "not_run": [ + "Gene callability (mosdepth)", + "Sex check (indexcov)", + "Variant calling", + "ClinVar screen", + "PharmCAT", + "CPIC drug-gene recommendations", + "pypgx", + "Cyrius (CYP2D6)", + "SMN1/SMN2 copy number (Parascopy, short reads)", + "Paralog genes (Paraphase, PacBio HiFi)", + "HLA typing (T1K)", + "Polygenic risk scores", + "Fine-grained ancestry (ADMIXTURE, RFMix)", + "Structural variants (Manta)", + "Structural variants (Delly)", + "Copy number (CNVpytor)", + "SV consensus merge", + "Structural variants, class 4 and 5 (AnnotSV)", + "Runs of homozygosity", + "Mitochondrial haplogroup", + "Y-chromosome haplogroup (Yleaf)", + "Y placement on the current public tree", + "Y-STR markers (HipSTR)", + "Raw-data export (23andMe v5 layout)", + "Blood groups (predicted)", + "Lp(a): LPA KIV-2 repeat count", + "MUC1 VNTR frameshifts (VNtyper)", + "Locityper second opinion at hard loci", + "Mitochondrial variants", + "Traits panel", + "Archaic ancestry (IBDmix)", + "Ancient populations (AADR PCA)", + "Viral DNA in the unmapped reads", + "Symptom-driven ranking (Exomiser)", + "Clonal haematopoiesis (CHIP)", + "Mobile element insertions (MEGAnE)", + "Mosaic chromosomal alterations (MoChA)", + "Cancer predisposition (CPSR)", + "Clinical variant filter", + "Variant prioritization (slivar)" + ], + "not_assessed": [ + "Mosaic and low-fraction small variants (only chrM heteroplasmy is reported, and with step 49 the CHIP genes)", + "Methylation and imprinting", + "Repeat expansions outside the ExpansionHunter catalog", + "Phase: two variants in one gene are compound-het candidates, not confirmed", + "Variants of uncertain significance (not interpreted)", + "Polygenic score percentiles: no ancestry reference panel was used, so the scores are raw sums", + "HBA1/HBA2 copy number (alpha-thalassaemia deletions): neither step 35's read-depth count (short reads) nor step 35b (Paraphase, PacBio HiFi) made a call", + "Mobile element insertions (Alu, L1 and SVA, the FKTN, TAF1 and BBS1 SVA insertions among them): not looked for (step 50, MEGAnE, is opt-in)", + "MUC1 VNTR frameshifts (ADTKD-MUC1): step 52 (VNtyper, opt-in) gave no reading; no standard caller sees them", + "Autosomal mosaic chromosomal alterations (large gains, losses and copy-neutral LOH in a share of the blood cells): step 51 (MoChA) did not assess this sample" + ] +} diff --git a/tests/parity/golden/summary-collect/all-inf/exit b/tests/parity/golden/summary-collect/all-inf/exit new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/tests/parity/golden/summary-collect/all-inf/exit @@ -0,0 +1 @@ +0 diff --git a/tests/parity/golden/summary-collect/all-inf/stderr b/tests/parity/golden/summary-collect/all-inf/stderr new file mode 100644 index 00000000..e69de29b diff --git a/tests/parity/golden/summary-collect/all-inf/stdout b/tests/parity/golden/summary-collect/all-inf/stdout new file mode 100644 index 00000000..437515d4 --- /dev/null +++ b/tests/parity/golden/summary-collect/all-inf/stdout @@ -0,0 +1 @@ +Summary written: all-inf.json diff --git a/tests/parity/inputs/summary-collect/oracle.sh b/tests/parity/inputs/summary-collect/oracle.sh index 3e33b699..7add40e4 100644 --- a/tests/parity/inputs/summary-collect/oracle.sh +++ b/tests/parity/inputs/summary-collect/oracle.sh @@ -6,8 +6,8 @@ # --only is og's alone (the Python writes every section, and the case # compares the group's subtrees). The modules collect_summary imports are # stubbed, except those the group b and c generators stage in pymods/ -# and py/. No other module is reached for these sample folders, except -# run_delta's two readers of history/ and recheck/, which find nothing here. +# and py/. run_delta is stubbed to find no history/ and no recheck/, unless +# py/ brings the real one (the whole-file cases, summary-collect-full.sh). prog=$1 shift d=$(mktemp -d)