[haodongchen]

Four changes to dnascope-pangenome that shorten its runtime and fix it on large hosts. Each is its own commit with its own tests, so they can be reviewed or reverted separately.

1. Cap KMC at 128 threads (bug fix)

On hosts with more than 128 logical CPUs, the pipeline failed at k-mer counting:

kmc -t192 ...
Error: Wrong parameter: number of threads for 2nd stage must be in range <1,128>

The cap goes in the three KMC command builders (cmd_kmc, cmd_extract_kmc, _kmc_stdin_cmd). That covers pangenome and hybrid-pangenome, with FASTQ or aligned input. The other commands in those pipelines still use every core.

2. One alignment lane per NUMA node

DNAscope already splits bwa into one job per NUMA node on large hosts. dnascope-pangenome now does the same with FASTQ input, and does it for minimap2 too. It uses DNAscope's rule, split_alignment(): NUMA nodes, halved until each piece has at most 64 CPUs.

• bwa-extract: one job per lane. fqidx extract -F j/N feeds bwa mem -p, and taskset pins bwa and pgutil extract to the lane's CPUs.
• mm2-lift: one job per lane, pinned to the same CPUs. It reads that lane's extracted reads and waits only for its own bwa lane.
• Dedup: both dedup stages read every lane's output.
• Job names: unchanged, so job metrics still report each step once.
• Opting out: a hidden --no_split_alignment keeps the single jobs.
• Unaffected: BAM/CRAM input, hosts with 32 or fewer cores, and the hybrid pipelines.

Needs a decision: fqidx extract rejects -K below 10,000,000, but pangenome's bwa uses -K 2000000. Split runs therefore use -K 10000000 for both fqidx and bwa, so their alignments differ slightly from an unsplit run. Unsplit runs keep 2,000,000. The alternative is to let fqidx extract accept 2,000,000.

3. Give indel2cnv's workers one variant at a time

indel2cnv sorts the candidate variants longest first. It then grouped them into batches of 5 and passed them to starmap with the default chunksize, which is about len(batches) / (4 * workers). So the first chunk held the most expensive variants of the whole run, and they all went to a single worker. That worker kept running long after the others went idle.

Each task is now one variant, with chunksize=1. Results come back in the same order, so the output is unchanged.

4. Run vg haplotypes alongside bwa when both fit in memory

vg-haplotypes needs only the k-mer counts. The current code expects it to overlap bwa when memory allows, but it never did: the job reserved every core, so the scheduler held it until bwa finished, whatever its dependencies said.

Now, if the host has room for vg-haplotypes (100 GB) plus 32 GB per bwa lane, the job reserves no threads, as kmc already does, and runs next to bwa. Otherwise it depends on every bwa lane, as before. This replaces the fixed 70 GB check.

The 100 GB budget covers measured peaks on the HPRC v2.0 and v2.1 graphs: about 60 GB with vg 1.73.0 and 1.78.0, and about 92 GB with a vg build that samples in parallel. The 32 GB per lane leaves headroom over bwa-extract's measured 19 GB peak.

Results

HG002 Illumina 35x, HPRC v2.0, FASTQ to small-variant, SV and CNV calls, on AWS on-demand instances.

	Before	After	Notes
c8a.48xlarge (192 vCPU, 2 NUMA nodes)	52:42	39:16	lanes; bwa-extract 1,152 → 555 s, mm2-lift 539 → 283 s
c8g.48xlarge (192 vCPU, 2 NUMA nodes)	68:09	53:17	lanes; bwa-extract 1,162 → 667 s, mm2-lift 688 → 271 s
c8a.48xlarge, with lanes	30:10	26:24	indel2cnv 156 → 49 s, vg-haplotypes alongside bwa
c7a.16xlarge (64 vCPU, 128 GB)	1:23:50	1:17:55	indel2cnv 420 → 68 s; not enough memory to run vg-haplotypes alongside bwa

The first two rows add lanes to a run that already has the KMC cap, which those hosts need to run at all. In the last two rows, before and after used the same vg build, so only changes 3 and 4 differ.

Accuracy against GIAB v4.2.1 (Aardvark, genotype match):

• Lanes changed total errors from 5,626 to 5,621 on c8a and from 5,623 to 5,619 on c8g. SNV and Indel F1 were unchanged to two decimal places.
• Changes 3 and 4 left the output VCF and CNV VCF bodies byte-identical on both instance types.

Tests

New unit tests:

• test_kmc_threads.py
• test_pangenome_numa_lanes.py, run on a mocked 2-node, 192-CPU host
• test_pangenome_vg_scheduling.py

black, flake8 and mypy pass at every commit. In a clean Python 3.12 environment, the same 18 tests fail on main and on this branch: longread and hybrid DAG and validation tests, plus the dry-run integration tests.

Not run end-to-end:

• 96- and 128-vCPU hosts, which would get 2 lanes
• BAM/CRAM input, where the code path is unchanged apart from the KMC cap
• the hybrid pipelines

🤖 Generated with Claude Code

[DonFreed]

Thanks! Can you merge these changes into dev instead of main?

[haodongchen]

Merge into dev. See #41