Handle CRAM references, empty thresholds, and zero-target extract inputs

[SuhasSrinivasan]

Review status: Author-reviewed and ready for ONT review.

Fixes #690.
Fixes #691.
Fixes #692.

Summary

• Propagates supplied FASTA references to legacy threshold and extract readers that decode reference-dependent CRAM bodies.
• Rejects automatic threshold requests when the sampled population has no usable modification probabilities, while preserving explicit-threshold, no-filter, and all-modified controls.
• Treats a zero-reference-target mapped phase as an explicit terminal state only for extract consumers that subsequently process unmapped records.
• Delays and protects entropy output creation across threshold/setup failures and makes entropy --no-filtering truly bypass sampling.
• Adds BAM/CRAM parity, missing-reference, empty-population, output-preservation, and indexed all-unmapped regression matrices.

Severity

Severity: High — format-dependent scientific completeness and automatic-threshold correctness

Rationale: #690 can make equivalent CRAM input exit successfully with zero extract rows while BAM emits complete output. #691 can silently turn an undefined automatic threshold into threshold zero, making automatic mode scientifically equivalent to no filtering. #692 is Medium in isolation because it fails explicitly, but it makes valid all-unmapped records unusable solely because an index is present.

Root cause

Reference paths were present at command boundaries but dropped when internal serial/indexed readers, sampling schedules, and unmapped fallbacks created new CRAM readers. Legacy threshold APIs had no reference-aware entry point.

The legacy threshold helper accepted an empty probability map, while modern QualHist paths inferred estimability from parsed-record count rather than actual canonical or modified probability observations.

Finally, generic mapped-coordinate feeders required at least one reference target. Extract invoked those strict constructors before its existing unmapped pass, so an indexed file with zero @SQ targets failed instead of representing an already-finished mapped phase.

Implementation

The ten commits are grouped for review:

Commits	Issue	Purpose
2ad0883, ac08413, 456d543, b836f1b, 8764535	#690	Add compatibility-preserving reference-aware sampling APIs; attach references to CRAM body readers; preserve the legacy duplex call shape; harden BAM/CRAM decode/threshold oracles with reference cache/remote resolution disabled
0d828f1, ed9a6a2, 06cf545, b63f53a	#691	Make entropy no-filter bypass sampling; reject empty legacy/modern automatic populations; protect entropy outputs through bounded setup; preserve valid all-modified samples
0703550	#692	Add opt-in zero-target terminal feeder state only for extract/unmapped fallback consumers while retaining strict constructors elsewhere

Public/internal compatibility wrappers preserve existing callers that have no reference. CRAM detection uses the opened file format rather than filename extension. Reference-independent all-unmapped CRAM remains supported without forcing a FASTA.

Preserved behavior

• Equivalent BAM and CRAM scientific records/rows agree when supplied the same reference.
• Explicit threshold zero and true --no-filtering modes remain valid and do not require an automatic population.
• Samples containing only real modified-state probability observations remain estimable.
• Summary and other coordinate-required consumers retain strict zero-target rejection.
• The existing separate unmapped pass processes each extract record once.

Non-goals

• No fixed-count estimand/sampling-policy change (Decide indexed --num-reads semantics: reference-span-balanced approximate sampling vs exact record-global sampling #654) and no deterministic-sampling or explicit-canonical cap repair (Seeded indexed fractional sampling changes with worker and interval settings #688/Automatic thresholds can use an arbitrary explicit-canonical probability cap #689, PR Make indexed sampling and threshold caps deterministic #705).
• No general worker-error propagation (Probability sampling can hang or return partial histograms after reader and worker failures #677), late decoder-error rollback, or fully transactional writer guarantee.
• Entropy output protection is limited to bounded threshold/setup/preflight failures and paired-output open failures; arbitrary late write/flush failures remain outside scope.
• No relaxation for a nonempty header whose coordinates are removed by motif/region filters; only a genuinely zero-target header may use the terminal state.
• No synthetic reference target, mapped/unmapped eligibility change, or output-schema change.

Behavior before and after

Case	Before	After	Expected oracle
Equivalent BAM/CRAM extract full	BAM 218 rows; CRAM can exit zero with 0 rows	Both emit the same 218-row scientific map	Exact normalized row parity
Equivalent BAM/CRAM extract calls --filter-threshold 0	BAM 109 rows; CRAM can exit zero with 0 rows	Both emit the same 109-row call map	Exact modification/coordinate parity
Automatic sample with no probability observations	Can succeed with threshold zero/empty report	Returns the explicit no-probability error	No scientific rows; setup-protected outputs remain absent or preserve sentinels
Valid all-modified probability sample	Can be mistaken for empty if only canonical bins are considered	Computes a nonzero threshold	Modified histogram is a real population
Entropy --no-filtering	Still enters automatic sampling	Bypasses sampling and uses passthrough caller	Matches explicit threshold-zero scientific output
Indexed all-unmapped BAM/CRAM with zero targets	Fails should be at least 1 contig	Empty mapped phase, then one unmapped pass	Ten unique records; full 218 rows/calls 109 rows where applicable
Missing/wrong CRAM reference detected during setup	Can create/truncate output before the real error	Fails before bounded output mutation	Existing sentinel unchanged; absent output remains absent

Testing

Test environment

• Revision tested: 87645355bdf76111381bb0bbee9f3c89b7864996
• Tree tested and worktree state: af6960103ccf263651384e5c231e9758a1138583; clean after testing
• Toolchain: rustc/cargo 1.90.0
• Platform: macOS 26.6, arm64 Apple Silicon
• Reference binary: installed modkit 0.6.4 and exact upstream parent 5cecc3fb3a9336068d9e3c68d5c08d678153dd2c
• External tools: samtools/htslib 1.23.1 for CRAM/BAM fixture inspection and indexing
• Dependency resolution identity: ignored Cargo.lock SHA-256 9ddcdd1f09bad24c0ecbdb730abcb31bcf5861c6f38e3289d36eae47d50bc1f5, using rust-htslib 0.46.0 and hts-sys 2.2.0; no lockfile or dependency pin enters this patch

Test layer	Exact command, fixture, or matrix	Result and evidence
Core: parent-red #690	Installed 0.6.4 and exact parent on equivalent checked-in BAM/CRAM extract and threshold commands	BAM full/calls emit 218/109 rows while CRAM can emit 0/0 and return success; legacy sampled paths fail despite an available reference
Core: parent-red #691	Installed 0.6.4 fraction-zero pileup-hemi and empty-tag Summary reproductions	Automatic fraction-zero output is byte-identical to no-filter output (341 rows, SHA-256 336479a1351d5594dd46295d65e6e5894620acdeb7208ca9d5973b2a16fe2b0a); Summary reports successful zero-read result
Core: parent-red #692	Installed 0.6.4 serial/indexed all-unmapped BAM/CRAM matrix	Serial succeeds; adding an index makes the same valid records fail before output with should be at least 1 contig
Core: combined focused integration	cargo test -p modkit --test test_cram_reference_consumers --test test_extract_cram_reference --test test_empty_qual_hist_thresholds --test test_empty_threshold_sampling --test test_extract_zero_reference_targets -- --nocapture --test-threads=1	13 passed, 0 failed; covers CRAM consumers, extract CRAM, empty legacy/modern populations, and zero targets
Entropy setup/output lifecycle	cargo test -p modkit --test test_entropy -- --test-threads=1	12 passed, 0 failed; no-filter bypass, threshold failure, symlink/prefix/pair-open protection, stdout, help, and regression controls pass
Core threshold/reference/terminal units	cargo test -p mod_kit empty --lib -- --test-threads=1 plus focused duplex compatibility test	7 empty/reference tests and 1 legacy duplex signature test passed; strict versus opt-in zero terminal, all-modified, and missing-reference setter boundary covered
CRAM parity stress	Isolated call_and_adjust_mods_match_bam_and_cram repeated 20 times with per-test empty REF_PATH/REF_CACHE	20/20 passed. Explicit threshold 0.5 compares all 10 normalized BAM/CRAM MM/ML records; separate automatic CRAM run asserts 10 records plus sampling 100% and estimated pass threshold log evidence
Reader setter red/green	Direct indexed-unmapped threshold unit; temporarily remove only the fallback reference setter	Red path incorrectly succeeds with thresholds [0.0, 0.5292969, 0.0, 0.0]; restored setter fails on the deliberately nonexistent supplied reference as expected
Extract mode/format matrix	Mapped, unmapped, indexed, truly unindexed, automatic, explicit, no-filter, fixed-count, missing-reference, and renamed-CRAM fixtures	BAM/CRAM normalized scientific rows agree; reference-free all-unmapped CRAM remains supported; wrong-reference failures preserve output state
Zero-target eligibility matrix	All-unmapped BAM/CRAM with/without indexes plus strict Summary/coordinate controls	Ten unique records exactly once; 218 full rows and 109 calls rows in applicable controls; strict coordinate consumers still reject zero targets
Core: applicable full workspace gate	TMPDIR=/private/tmp/modkit-full-cram-8764535 cargo test --offline --locked --workspace --all-targets --no-fail-fast -- --test-threads=1	210 passed, 14 ignored, 0 failed on the exact final head
Core: formatting/diff checks	Targeted stable rustfmt --check; git diff --check upstream/master...HEAD; ancestry/name-status audit	Changed files and patch whitespace pass; exactly 10 reviewable commits, 23 changed files, 2,881 insertions and 264 deletions; no #688/#689/#705 production commit imported

Tests not performed

• No large-data performance, memory, or I/O benchmark was run; the changes concern reader setup, terminal scheduling, and bounded error contracts rather than a performance claim.
• No arbitrary late decoder/write/flush failure transaction test was claimed. Output-preservation tests cover only bounded setup/preflight paths documented above.
• No fixed-count sampling-policy or fresh-process canonical-cap determinism test was claimed; PR Make indexed sampling and threshold caps deterministic #705 owns that behavior.

Scientific validation

• Population/eligibility invariant: automatic thresholds require at least one canonical or modified probability observation; record count alone is insufficient, while an all-modified population remains valid.
• Count/category conservation invariant: equivalent BAM/CRAM and serial/indexed extract modes conserve ten input records and exact 218/109 row maps without duplicates.
• Coordinate/strand/interval invariant: zero-target mode produces no synthetic coordinate; strict mapped consumers and filtered-to-zero nonempty headers remain errors.
• Determinism invariant: explicit-threshold BAM/CRAM MM/ML parity is exact; automatic CRAM sampling is exercised independently without depending on inherited Automatic thresholds can use an arbitrary explicit-canonical probability cap #689 cap nondeterminism.
• Independent oracle or specialist review: Rust, Bioinformatics, and supervisor reviews independently traced every reader, empty-population contract, terminal feeder state, and output-preflight boundary on the frozen branch.

Output and compatibility

• User-visible change: reference-dependent CRAM works consistently across affected commands; undefined automatic thresholds fail clearly; indexed all-unmapped extract matches serial behavior.
• Expected output differences: previously empty CRAM extracts gain their complete rows; automatic-empty commands now return nonzero instead of acting like no filtering; zero-target indexed inputs now complete.
• Byte-identical controls: equivalent normalized BAM/CRAM scientific maps and serial/indexed no-header outputs agree in all scoped controls.
• CLI/API/schema compatibility: call-mods and adjust-mods gain/accept --reference/--ref for CRAM; existing internal no-reference wrappers and legacy duplex API remain available; no output schema changes.
• Partial-output or failure semantics: bounded setup/reference/threshold failures occur before protected entropy/extract output mutation where tested; no general runtime transaction guarantee is made.

Reviewer guide

1. Review the commit/issue table above; each commit is intentionally scoped to one reader, threshold, lifecycle, or terminal-state boundary.
2. Review util.rs, reads_sampler, and thresholds.rs for reference-aware compatibility wrappers and propagation.
3. Review sample_probs::QualHist for the probability-observation predicate and indexed-unmapped reference setter.
4. Review extract reader construction and ZeroReferenceTargetMode/feeder constructors for strict versus opt-in behavior.
5. Review entropy writer setup ordering and tests, keeping its protection claim limited to bounded setup/preflight failures.
6. Rerun the 13-test combined focused matrix and the direct indexed-unmapped setter unit as canaries.
7. If PR Make indexed sampling and threshold caps deterministic #705 or Prevent interval endpoint overflow and remove stray adjust-mods stderr output #706 merges first, rebase this PR and rerun the focused/full gates; this branch deliberately excludes Make indexed sampling and threshold caps deterministic #705 and overlaps sample_probs/reads_sampler, while Prevent interval endpoint overflow and remove stray adjust-mods stderr output #706 touches the same interval feeder arithmetic.

Checklist

• The issues contain reproducible observed and expected behavior.
• The change is limited to the linked issues' approved scope.
• The regressions are demonstrably red on the exact parent/released binary.
• All tests actually performed are listed above with their results.
• Unrun or inapplicable tests are disclosed.
• Scientific counts/statistics and output compatibility are explicitly checked.
• Formatting and diff-hygiene checks pass, or unrelated findings are documented.
• No generated data, private sample identifiers, or unrelated changes are included.