Correct DMR site statistics and segmentation completeness

[SuhasSrinivasan]

Review status: Author-reviewed and ready for ONT review.

Summary

This PR consolidates the related DMR correctness fixes across site scoring and the downstream stateful segmenter:

• zero raw or coverage-capped denominators are rejected before NaN or singular beta calculations;
• matched-replicate statistics preserve configured sample identity and are omitted if any configured pair is incomplete;
• absent, zero-coverage, and coverage-filtered observations do not become canonical pseudo-counts, while observed canonical 0/n remains valid;
• Viterbi decoding returns one state for every successfully scored site, including the final site in each nonempty chunk;
• lexical contig order is preserved at batch handoff so a contig cannot be closed and reopened by hash-map iteration.

These changes form one scientific pipeline: accepted per-site populations feed the same ordered HMM segmentation. The ten commits remain five tests/fix pairs for direct review.

Severity

High — scientific statistics, segmentation completeness, and reproducibility.

Affected accepted inputs can emit NaN, abort, cross-pair replicates, impute no-observation samples as canonical, omit one scored site per HMM chunk, or split a contig at an internal batch boundary. These defects change effects, probabilities, representation percentages, segment boundaries, counts, and state continuity.

Bugs and corrected behavior

Case	Previous behavior	Corrected behavior
Raw/capped zero denominator	NaN success or process abort	Recoverable failed site or early configuration error; no scientific NaN row
Three configured replicate pairs	Effects could follow container order and cross-pair samples	Effects follow immutable CLI sample identity
Any incomplete pair	Remaining samples could compact into fabricated pairs	Both optional replicate fields are -
Zero-coverage observation	Could contribute a canonical pseudo-count and 100% representation	Equivalent to absent/filtered; configured sample remains in the representation denominator
One HMM emission	Could produce zero states/regions	Produces one state and one half-open singleton region
Chromosome-20 segmentation	17,271 site rows but segment total 17,270	Site and segment totals both reconcile to 17,271
Cross-contig batch	A contig could leave/re-enter and split one segment	One lexical block per contig; site order within a contig is unchanged

Implementation

• Count and coverage-cap boundaries reject total zero, resolved cap zero, and overflowing replicate-cap multiplication.
• Sample IDs remain attached through organization; replicate construction consumes configured command order.
• Only positive-coverage observations enter pooled/balanced statistics and representation numerators; percentage floors use exact integer arithmetic.
• Viterbi backtrace starts from the final emitted state, traverses emission predecessors only, reverses the complete path, and requires path/position cardinality equality.
• Complete paths convert to half-open regions, including singleton final positions.
• Unique contig tuples are sorted lexically only at the final batch handoff, without changing score order within a contig.

Testing

Tested exact revision: 97bb3f5c374ddbd5c684514bdeaa2f53adea7a11
Tree: d9f773bbee8ba310bc2452a01a2cee2ef883a721
Platform/toolchain: macOS arm64; rustc/cargo 1.90.0; samtools/bgzip/tabix 1.23.1; hts-sys 2.2.0.

• Full serial workspace gate: 208 passed, 14 ignored, 0 failed.
• DMR units: 30 passed.
• HMM units: 3 passed.
• DMR CLI integrations: 13 passed.
• Release-mode exact percentage-floor oracle: 1 passed.
• Parent-red matrices reproduce NaN/status-134 zero paths, zero-cap failures, cross-pairing, incomplete-pair fabrication, zero-row pseudo-count bias, missing final HMM state, chromosome-20 site/segment mismatch, and contig re-entry.
• Positive numerical control remains exact: A=10/10 modified versus B=0/10 retains MAP p-value 0.0000006230948043897833 and effect 1.
• Pairing matrix covers complete/incomplete pairs, command-order identity, 1/1 versus 4/2 threads, and independent one-pair controls.
• Coverage matrix proves zero-coverage, absent, and filtered observations are equivalent while observed canonical 0/10 remains eligible.
• Chromosome-20 output contains 17,271 represented sites with final end 10,804,378 and exact site/segment reconciliation.
• Three-contig output conserves 6 aa, 15 bb, and 6 cc sites in segments aa [0,6), bb [0,15), cc [0,6); interval/thread artifacts are byte-identical.
• Every added BGZF/index/reference fixture passed integrity checks.
• git diff --check, changed-file formatting checks, conflict-marker scan, range-diff, clean-worktree, ancestry, file-scope, and Cargo.lock checks passed.

Bioinformatics review independently verified denominators, sample identity, positive-coverage population semantics, one-state-per-site conservation, half-open segment geometry, chromosome-20 reconciliation, contig non-reentry, and interval/thread invariance. No blocker remained.

Output compatibility

• Affected invalid/statistically incorrect rows and incomplete segments intentionally change.
• Positive numerical controls, CLI options, site/segment schemas, state vocabulary, half-open BED convention, and valid group rows are unchanged.
• Optional matched fields remain available only for complete configured pairs; valid pooled/group rows remain available when matched fields are omitted.
• Complexity remains O(N) for decoding; the final handoff adds O(C log C) comparisons for normally small contig count C.

Reviewer guide

1. Review denominator/cap checks in beta_diff.rs.
2. Review sample identity and positive-coverage organization in tabix.rs and single_site.rs.
3. Review the exhaustive HMM decoder oracle and complete backtrace in hmm.rs.
4. Review path-to-region cardinality and singleton/final-site tests.
5. Review lexical contig handoff and the three-contig fixture.
6. Run cargo test -p mod_kit dmr:: -- --test-threads=1, cargo test -p mod_kit hmm -- --test-threads=1, and cargo test -p modkit --test test_dmr -- --test-threads=1 as focused canaries.

Non-goals

• No change to the MAP estimand, posterior update, quadrature, or automatic-cap percentile population.
• No clamping a zero cap to one and no partial replicate-vector grammar.
• No writer-lifecycle or batching-policy redesign.
• Contig order is lexical, matching the scheduler contract; this does not introduce natural chromosome ordering.