Correct validate truth parsing and observation accounting

[SuhasSrinivasan]

Review status: Author-reviewed and ready for ONT review.

Summary

This PR consolidates validate's related input-truth and observation-accounting corrections:

• every BAM/truth-BED pair receives a fresh complete BAM traversal;
• eligible truth-overlapping observations are counted even when no modification call seeded that position;
• reference skips, deletions, ambiguous aligned bases, molecular/reference strand composition, and edge-filtered fallback sites are classified consistently;
• valid all-No-Call or zero-confident-call reports succeed and render undefined percentages as NA;
• truth-BED open/read/syntax/semantic errors are returned with file and physical-line context before output creation;
• conflicting labels at one (contig, strand, position) fail deterministically, while identical overlaps remain idempotent and opposite strands remain independent.

These changes share the same validate truth population and reporting pipeline. No sampling, scheduling, throughput, or parallelization change is included.

Severity

High — scientific counts, classification, and output integrity.

Affected accepted inputs can silently omit truth sets or observations, change counts by row order, misclassify CIGAR/reference-strand cases, or truncate an existing report before a later input error.

Bugs and corrected behavior

Case	Previous behavior	Corrected behavior
Multiple BAM/truth pairs	Only the first truth BED could receive a full BAM traversal	A new reader is opened for every pair; pair order produces identical complete reports
Eligible unseeded sites	No Call, mismatch, and deletion observations could be omitted	Every biologically eligible truth overlap is classified exactly once
1M3N1M versus 1M3D1M	Reference skips could be counted as deletions	N spans are omitted; D spans remain deletions
Aligned N/IUPAC base	Ambiguity could discard the whole record	The affected site is a local No Call; other sites remain countable
Edge-filtered fallback site	A filtered site could re-enter as No Call/mismatch	Calls and fallback observations use the same forward-query-coordinate predicate
No confident calls	Valid report could exit nonzero or contain NaN	Seven-line report succeeds and undefined metrics are NA
Malformed truth row	Row could be silently dropped and an existing report overwritten	Command fails with path, physical line, and cause before report creation
Conflicting truth labels	Last row won, making counts order-dependent	Both row orders fail with old/new provenance; valid duplicates deduplicate

Implementation

• The BAM reader lifetime moves inside the truth-pair loop.
• Descriptor-derived primary-base/strand metadata defines fallback eligibility before MM positions are expanded.
• Classification uses explicit half-open CIGAR spans and composed alignment/modification strand orientation.
• Ambiguous bases and edge predicates are site-local, with deletion-without-query-position behavior kept explicit.
• Truth parsing uses an enumerated fallible line loop, exact half-open coordinate and +/- validation, documented modification-label grammar, and provenance-aware map insertion.
• Every truth file parses and canonical-base derivation completes before output creation.

Testing

Tested exact revision: e90e6ff8a2cdc5196d35a5364c4c992c6b37da91
Tree: 2c32f401c9b0a2539083299f1456dc6a0452b6a8
Platform/toolchain: macOS arm64; rustc/cargo 1.90.0; rust-htslib 0.46.0 and hts-sys 2.2.0.

• Full serial workspace gate: 211 passed, 14 ignored, 0 failed.
• Validate units: 27 passed.
• Validate CLI integrations: 7 passed.
• Split truth AB/BA fixture: fixed outputs are byte-identical, 326 bytes, SHA-256 1b98e2a75112446f1072cdc6b117bc40d5947b023b285c051ca9560e30b6d0b9.
• Nine-record unseeded fixture: changes from m=3 to the conserved m=3, No Call=6 result; expanded classification matrix covers No Call, mismatch, deletion, reference skip, positive/negative descriptor strands, reverse alignments, empty/N descriptors, and unrelated bases.
• Edge-filter matrix covers ordinary, inverted, asymmetric reverse-query, retained-site, and deletion-without-query-position cases.
• Truth-input matrix covers negative/reversed/zero-width coordinates, malformed rows, exact strands, supported labels/ChEBI precedence, comments/blank rows, identical overlaps, opposite strands, and both conflict orders.
• Five invalid CLI cases preserve the exact pre-existing 9-byte sentinel, SHA-256 b5f7e7d285029324d9b3acae19cc05099271454ac98bfc059a92b0581625cd51.
• Existing valid validate output and ordinary defined percentage formatting remain unchanged.
• git diff --check, direct stable formatting checks on all changed Rust files, clean-worktree, ancestry, test-union, and Cargo.lock checks passed.

The exact combined head was independently reviewed for Rust error propagation, reader lifetimes, population eligibility, strand/CIGAR semantics, deterministic conflict handling, output preservation, and preservation of every source test. No blocker or material nonblocker remained.

Output compatibility

• Expected output changes are limited to omitted/misclassified observations, incomplete multi-truth runs, invalid/conflicting truth inputs, and undefined metrics.
• Valid existing report schema, CLI options, thresholds, balancing, and ordinary numeric formatting are unchanged.
• Invalid truth input now fails before creating or truncating output.

Reviewer guide

1. Review the reader lifetime and complete-truth-pair CLI oracle.
2. Review descriptor eligibility and the unseeded classification tests.
3. Review N versus D, strand composition, ambiguity, and shared edge filtering.
4. Review the enumerated truth parser, label grammar, and provenance-aware conflict insertion.
5. Review output-open ordering and NA formatting.
6. Run cargo test -p mod_kit validate:: -- --test-threads=1 and cargo test -p modkit --test test_validate -- --test-threads=1 as focused canaries.

Non-goals

• No unstranded (.) truth mode or UCSC track/browser directive support.
• No sampling, scheduling, I/O-throughput, or parallelization change.
• No redefinition of deletion-without-query-position policy.
• No validate output-schema redesign.