`entropy` aborts on valid inputs with ten or more modification states

[SuhasSrinivasan]

entropy aborts on valid inputs with ten or more modification states

Summary

modkit entropy encodes canonical and modified states as single text characters. A valid entropy window containing ten distinct modification codes reaches internal state ID 10, attempts to parse the two-character string "10" as one Rust char, and aborts.

Nine modification states succeed, so the failure is an undocumented representation boundary rather than an input-format or coverage error.

Severity

Severity: High — correctness and reliability

Rationale: The command accepts and validates the modBAM, begins processing, then terminates through an internal Rayon panic/abort and leaves an empty output. Multi-modification direct-RNA or synthetic/custom modification alphabets can legitimately exceed nine observed states in one entropy window, and there is no preflight diagnostic or output-level recovery.

User and scientific impact

• Affected result or workflow: modkit entropy on windows whose accepted reads collectively contain ten or more distinct modification codes.
• Direction of error: process abort and empty/partial output rather than an entropy result.
• Likely exposure: data-dependent and uncommon for standard single-modification assays, but realistic for multi-modification direct-RNA, multiplexed research codes, ChEBI-coded states, and validation fixtures.
• Detectability or workaround: split the biological state set to nine or fewer codes, which changes the requested entropy alphabet and is not a scientifically equivalent workaround.

Affected versions and environment

• Released version: modkit 0.6.4.
• Development revision: 5cecc3fb3a9336068d9e3c68d5c08d678153dd2c.
• Verification environment: macOS 26.6, arm64, Apple M2 Ultra.
• Input formats: coordinate-sorted modBAM with BAI and FASTA with FAI.
• Related tool version: samtools 1.23.1.

Steps to reproduce

Create a one-base reference and ten valid one-base records, each carrying a different one-letter modification code.

printf '>chr1\nC\n' > ten.fa
samtools faidx ten.fa

{
  printf '@HD\tVN:1.6\tSO:coordinate\n'
  printf '@SQ\tSN:chr1\tLN:1\n'
  for code in a b c d e f g h i j; do
    printf 'read-%s\t0\tchr1\t1\t60\t1M\t*\t0\t0\tC\t?\tMM:Z:C+%s?,0;\tML:B:C,255\tMN:i:1\tNM:i:0\n' \
      "$code" "$code"
  done
} > ten.sam

samtools view -b -o ten.bam ten.sam
samtools index ten.bam

modkit modbam check-tags ten.bam --head 20 --suppress-progress

modkit entropy \
  --in-bam ten.bam \
  --out-bed ten.bed \
  --ref ten.fa \
  --base C \
  --num-positions 1 \
  --window-size 1 \
  --min-coverage 1 \
  --filter-threshold 0 \
  --threads 1 \
  --io-threads 1 \
  --suppress-progress
echo "exit status: $?"
wc -c ten.bed

For the boundary control, remove the read-j line, rebuild/index nine.bam, and run the same command.

Control or independent oracle

modkit modbam check-tags exits zero, reports ten passing records, and lists C+a? through C+j? as valid tag headers. Every read contributes one fully observed, equiprobable state at the same one-position window.

The Shannon entropy oracle is therefore:

H = -10 * (1/10) * log2(1/10) = log2(10) = 3.321928094...
accepted reads = 10

The matched nine-code control similarly has entropy log2(9) = 3.169925... and accepted-read count 9.

Observed behavior

The ten-record BAM passes tag validation:

no errors
num PASS records: 10 (100.00%)
num records: 10

entropy then aborts with exit status 134 and leaves a zero-byte output:

thread '<unnamed>' panicked at modkit-core/src/entropy/mod.rs:362:63:
called `Result::unwrap()` on an `Err` value: ParseCharError { kind: TooManyChars }
Rayon: detected unexpected panic; aborting

The otherwise identical nine-state control exits zero and reports the expected entropy and read count:

chr1	0	2	3.169925	+	9

The width-two coordinate in this control is a separate entropy interval defect; it does not cause the state-cardinality abort.

Expected behavior

• Ten or more valid modification states are represented atomically and do not panic.
• The ten-code fixture exits zero with entropy approximately 3.321928 and accepted-read count 10.
• Canonical, modified, and filtered/wildcard states cannot collide.
• The same set of modification codes receives the same state mapping regardless of read encounter order, strand, or Rayon thread count.
• Existing inputs with nine or fewer states preserve their established scientific values and output schema.

Root-cause evidence

• The state lookup collects codes in a deterministic BTreeSet, enumerates them from 1, formats each integer as decimal text, and parses that text as one char at entropy/mod.rs:331-365. IDs 1 through 9 are one character; ID 10 is not.
• Encoded read patterns are then represented as String, with canonical 0 and filtered * sentinels at entropy/mod.rs:368-413.
• Wildcard expansion builds regular expressions over those textual states at methylation_entropy.rs:10-139. This makes an arbitrary biological alphabet depend on decimal character width.
• A parent-failing CLI regression reproduces the exact ten-state abort. Focused state tests also establish successful 10/12/17-state cardinality, mixed one-letter/ChEBI code bijection, strand union, wildcard probability mass, and encounter/thread invariance after replacing the representation.

Proposed fix scope

• Introduce a typed integer entropy symbol, reserving one value for canonical and a disjoint sentinel for filtered/wildcard observations.
• Assign checked compact IDs from the existing sorted modification-code union so the mapping remains deterministic.
• Represent each read pattern as a compact symbol slice instead of a String.
• Replace regex-based wildcard matching with direct element-wise symbol matching while preserving the existing concretization union, uniform wildcard mass, duplicate-read weighting, Shannon normalization, and accepted-read count.

Non-goals

• Do not change motif coordinates, interval ownership, thresholding, record eligibility, output schema, or entropy normalization.
• Do not conflate this cardinality repair with final floating-point byte reproducibility. The existing HashMap count reduction can still vary at the last f32 bit across fresh processes; ordered numerical reduction should be a separate follow-up.
• Do not claim arbitrary memory scalability of the complete entropy algorithm; only remove the text/regex cardinality boundary in this repair.

Acceptance criteria

• The parent ten-code fixture fails before the fix and exits zero afterward with accepted-read count 10 and entropy within 1e-5 of log2(10).
• Valid 9-, 10-, 12-, and 17-code windows complete without panic.
• Existing nine-state output remains scientifically and schema compatible.
• Canonical and filtered sentinels are disjoint from every assigned modification symbol.
• One-letter and numeric ChEBI codes retain a one-to-one mapping across positive/negative strand union and encounter-order permutations.
• Wildcard reads distribute exactly one unit of probability mass uniformly over their supported concrete patterns, preserving existing duplicate-read weighting.
• One- and four-thread runs and forward/reverse code encounter orders produce the same output for the exact ten-code fixture.
• Existing entropy integrations, focused representation tests, and the full workspace suite pass.
• Fresh-process last-bit equality is explicitly not an acceptance criterion for this PR; it belongs to the separate ordered-reduction follow-up.

Reproduction artifacts

Artifact	Size	SHA-256	Notes
ten.sam	742 bytes	274be5329f400a84783ac172fb9ffbb26c68032f69e8325bbfca82fd498231a3	Inline ten-code fixture
ten.bam	318 bytes	4748cb0f5bc4f1ae595cf303b4a3ee899b2949a7af83905f8e0626c5135d90b3	samtools 1.23.1 conversion
ten.bam.bai	96 bytes	9b54c7373cf6394990f20441f9269a2e6594ddcb08f0b18f7529bbae14a69653	BAM index
ten.fa	8 bytes	bdaa0b566f868a175805d61a3e04ad59ab51951543111703233aae0841f0f923	Inline one-base reference
ten.fa.fai	13 bytes	930ebf9ac3bd56acfde9b26962ede12cc26cf7584454838addbdaa8ed7120810	FASTA index
nine.sam	672 bytes	7bd6dcc91c76fcc80b0e653eff10b7b12b7b42a1f0416a8e28bb6ffcf487d5be	Nine-code passing control
nine.bam	315 bytes	944db133c6df3f0a6751ceed13b751a173f85dd64bbbe2e9e8a29955d95deb4b	Passing-control BAM

Related work

• Proposed PR: one focused typed-state representation PR rebuilt on current upstream.
• Follow-up: deterministic ordered entropy-count reduction for fresh-process byte reproducibility should remain a separate commit/PR or an explicitly separate issue section.
• Coordinate/window ownership, regional summary statistics, filtering semantics, and runtime lifecycle/output safety are independent review units.

[SuhasSrinivasan]

The confirmed entropy fix is now included in the consolidated, ready-for-review PR #712. Closing this standalone issue in line with the maintainer's request to keep the identified bug and its code change together in the PR.