Add HLA-HD v1.7.1 module and BAM-input subworkflow - #241
Open
johnoooh wants to merge 34 commits into
Open
Conversation
Module runs HLA-HD for HLA typing from paired-end FASTQ input. Container-only (not available on conda/bioconda). Private container built from JFrog-hosted binary. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Stub test and real test using HLA-region FASTQ from test-datasets hlahd branch. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Composes samtools/view, gatk4/revertsam (optional), samtools/fastq, and hlahd modules into a BAM-to-HLA-typing pipeline. Tests cover both skip_revert_sam paths plus stub test. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Also add the nf-test snapshot file that was missing from prior commits. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…pshots
- Fix output globs: results are at <prefix>/result/, not <prefix>/
- result: ${prefix}/result/${prefix}_final.result.txt
- result_per_locus: ${prefix}/result/${prefix}_*.est.txt
- Switch container URL to dev registry while awaiting next prod release
- Add nextflow.config for subworkflow tests (ext.prefix per process to
avoid GATK4_REVERTSAM input/output name collision)
- Regenerate all snapshots against new HLA class I test data that
produces actual allele calls (A*01:01:01, B*08:01:01, C*07:01:01)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add docker/login-action step to authenticate with mskcc.jfrog.io before running tests. Login is conditional on docker profile and credentials being present, so conda/singularity profiles are unaffected. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Switch from default HLA_gene.split.txt symlink to the explicit 3.50.0 dictionary release file. Refresh module and subworkflow snapshots to account for the additional loci rows (E, G, H, J, K, L, V) emitted by the newer split; class I calls are unchanged.
…onventions
The hlahd_from_bam subworkflow imports three nf-core modules
(samtools/view, gatk4/revertsam, samtools/fastq) that live under
.gitignored modules/nf-core/, so CI checkouts cannot resolve the
includes. Add a tiny bash + yq + git sparse-checkout installer that
reads components: from each subworkflow meta.yml and fetches foreign
components into modules/<org_path>/<name>/ before nf-test runs. No
nf-core/tools dependency, no modules.json.
Also modernize the subworkflow against current nf-core/modules
conventions: SAMTOOLS_VIEW now takes 5 inputs (added bed channel);
samtools/{view,fastq} and gatk4/revertsam emit versions via topic
channels rather than emit: versions, so drop the corresponding
ch_versions.mix() lines. HLAHD itself still uses classic emit, so
its mix line stays.
meta.yml components: upgraded to the dict shape (name + git_remote +
org_path) so the installer can resolve the foreign three; hlahd
stays bare-string for local resolution.
Snapshot regeneration is intentionally deferred -- the new run
produces 1 versions.yml hash per test (HLAHD only) where the old
snap had 3 or 4. Will be updated in a follow-up commit using the
hashes CI reports.
Re-recorded all 3 tests against the modernized subworkflow (5-arg
SAMTOOLS_VIEW; topic-versions for samtools/{view,fastq} and
gatk4/revertsam):
- with revert sam: result.txt md5 unchanged (e51e94f4...) -- HLA
calls byte-identical to prior typing. versions: 4 hashes -> 1
(HLAHD only).
- stub: versions: 4 hashes -> 1.
- skip revert sam: result.txt md5 changed (e51e94f4 -> f2b54c8b),
versions: 3 hashes -> 1. The new md5 is "all Not typed" output --
the previous snap matched with-revert-sam by coincidence and
masked a known issue: when GATK4_REVERTSAM is bypassed, samtools
fastq runs on a coord-sorted BAM and emits singletons, so HLAHD
cannot type. Tracked as a follow-up in the project README; not a
blocker for #241.
Local validation: nf-test 0.9.4, nextflow 25.10.4, docker profile,
public docker.io/orgeraj/hlahd:1.7.1 stand-in (HLAHD binary is
identical to the JFrog image; container URL in modules/msk/hlahd
unchanged).
The hlahd module pointed at omicswf-docker-dev-local, where the 1.7.1 image is no longer available, causing docker shards to fail with manifest unknown despite a successful login. Singularity additionally failed because the registry-login step was gated to profile==docker and apptainer does not consume Docker's auth file regardless. - Point hlahd container at omicswf-docker-prod-local (1.7.1 published) - Export APPTAINER_DOCKER_*/SINGULARITY_DOCKER_* for singularity shards so apptainer can authenticate against mskcc.jfrog.io directly
The dev tag (1.7.1) was rotated out of omicswf-docker-dev-local, so both docker and singularity now return manifest unknown for the dev path. Switching back to prod, where 1.7.1 is published. The prod image's PATH does not include /opt/hlahd/current/bin (likely built from a Dockerfile predating the ENV PATH directive), causing hlahd.sh to fail on bare-name calls to its sibling binaries (pm_extract, stfr, get_diff_fasta, etc.). Prepending the install bin directory to PATH in the script makes the module robust regardless of how the image is built and unblocks CI immediately.
Snapshot regenerated locally against docker.io/orgeraj/hlahd:1.7.1 (deterministic build) using the new HLA-A-only sliced test data. Module container switched to the dev JFrog image, which mirrors the build that produces this snapshot — prod rebuild was producing divergent B/C calls.
…ublishes JFrog dev tag was missing (manifest unknown). docker.io/orgeraj/hlahd:1.7.1 is the deterministic build the snapshot was generated against. Swap back to the JFrog dev/prod image once it is republished.
Setting APPTAINER_DOCKER_USERNAME/PASSWORD globally caused apptainer to send JFrog basic-auth to every docker:// pull, so ghcr.io rejected unrelated images (neoantigen-editing, neoantigen-utils-base, etc.) with 403 across all singularity shards. Replace with a Docker-format auth file at ~/.apptainer/docker-config.json (and ~/.singularity/docker-config.json) keyed to mskcc.jfrog.io only. JFrog pulls still authenticate; pulls from ghcr.io/quay.io/docker.io go anonymous as before.
with-revert path produces test_sample_final.result.txt md5 7ba486d3..., matching the regenerated hlahd module snapshot.
PR #58 in mskcc-omics-workflows/containers fixed the hlahd build but the dev-build workflow is misrouted via shared JFROG_CONTAINER_REPO var, so the fixed image landed in omicswf-docker-prod-local rather than dev-local. Point the module there until the publish workflow is fixed.
2 tasks
Test data is sliced to HLA-A only, so only the A line carries real biological signal. Whole-file md5 of test_sample_final.result.txt was brittle to incidental drift in class II / non-class-I lines whenever the container's bowtie2 dictionary was regenerated (e.g. PR #58 in containers repo). Switch the snapshot assertion to a content match on lines starting with "A\t" only. Also drop the temporary DEBUG_FINAL_RESULT println from the module test now that the diagnosis is in hand. Snapshots regenerated for module and subworkflow against the PR #58 image (mskcc.jfrog.io/omicswf-docker-prod-local/.../hlahd:1.7.1).
skip_revert_sam=true fed SAMTOOLS_VIEW's coordinate-sorted BAM output straight into SAMTOOLS_FASTQ, which needs name-grouped input to pair mates correctly. On a coordinate-sorted BAM it discarded ~99% of reads as singletons, starving HLA-A of coverage and returning "Not typed" in CI (docker|4, singularity|4 on PR #241). The with-revert-sam path only worked because GATK4_REVERTSAM's default queryname sort incidentally collated reads as a side effect. Add SAMTOOLS_COLLATE unconditionally before SAMTOOLS_FASTQ so pairing is correct regardless of branch, instead of relying on that side effect. Verified locally with --profile docker: all 3 subworkflow tests pass, including the previously-failing skip-revert-sam case, against the existing (unchanged) snapshot.
Add docs/ci.md explaining the nf-test GitHub Actions pipeline (change detection, sharding, the nf-test-action composite action, confirm-pass gate), with a focus on the piece that's easy to miss: how a subworkflow's meta.yml `components:` list drives install-components.sh to sparse-checkout nf-core modules at CI time, since modules/nf-core and subworkflows/nf-core are gitignored here. Written while fixing PR #241, where hlahd_from_bam needed a new samtools/collate dependency declared this way.
meta.yml's components: dict entries used a flat shape (- name: ..., git_remote: ..., org_path: ...) that only install-components.sh (our CI script) understood. The actual nf-core subworkflows install tool expects dict entries nested under the component name instead (components_utils.py: get_components_to_install), and crashes with AttributeError: 'str' object has no attribute 'get' on the old shape - verified by running nf-core subworkflows install hlahd_from_bam against this branch in a scratch pipeline. hlahd_from_bam was the first subworkflow in this repo to declare an external (nf-core) component dependency, so this had never been exercised against the real nf-core CLI before. Switch meta.yml to the nested shape nf-core expects, and update install-components.sh to parse it the same way (kept org_path as an MSK-only optional key nf-core silently ignores). Re-verified install-components.sh still fetches all 4 dependencies correctly against the new schema.
price0416
approved these changes
Aug 10, 2026
price0416
left a comment
Collaborator
There was a problem hiding this comment.
Looks solid, passes all tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add HLA-HD v1.7.1 module and BAM-input subworkflow
Summary
modules/msk/hlahd— nf-core-style module for HLA-HD v1.7.1 (high-resolution HLA typing from paired FASTQ)subworkflows/msk/hlahd_from_bam— end-to-end BAM-to-HLA-typing workflowtests/config/test_data.config(data onhlahdbranch of test-datasets repo)docs/ci.mddocumenting the nf-test CI pipeline, written while debugging this PR's CI failure (see below)Module:
modules/msk/hlahdmskcc.jfrog.io/omicswf-docker-prod-local/mskcc-omics-workflows/hlahd:1.7.1[ meta, fastq_1, fastq_2 ]result(final allele calls),result_per_locus(per-gene.est.txtfiles),versionsext.args2(default: 100)Subworkflow:
subworkflows/msk/hlahd_from_bamChains five modules to go from coordinate-sorted BAM to HLA allele calls:
The
skip_revert_samparameter controls whether GATK4 RevertSam runs. Set totruewhen the input BAM has no BQSR applied.SAMTOOLS_COLLATEruns unconditionally regardless ofskip_revert_sam— see "CI fix" below for why.Test data
6:29910247-299136616:31321649-313249896:31236526-31239913~21k reads, ~3.3MB across 4 files (BAM + BAI + paired FASTQ).
Example output (
test_sample_final.result.txt)Class II loci are "Not typed" as expected — only class I regions are included in the test data.
Tests
All 5 nf-test tests pass with deterministic snapshot matching:
Module tests (2):
hlahd - fastq pair - result txt— real HLA-HD run, verifies final result md5hlahd - fastq pair - stub— stub run, verifies versions outputSubworkflow tests (3):
hlahd_from_bam - bam - with revert sam - result— full pipeline with GATK4 RevertSamhlahd_from_bam - bam - skip revert sam - result— pipeline skipping RevertSamhlahd_from_bam - bam - stub— stub runBoth revert/skip-revert paths produce identical final calls.
CI fix:
skip_revert_samdropped ~99% of readsCI was red on
Test | docker | 4/Test | singularity | 4(theskip revert samtest) for a while:SAMTOOLS_VIEW's coordinate-sorted BAMoutput was fed straight into
SAMTOOLS_FASTQwhenskip_revert_sam = true.samtools fastqneeds name-grouped input to pair mates; on acoordinate-sorted BAM it discarded ~99% of reads as singletons (verified: 8443/8501
discarded direct vs. 2183/8501 after
samtools collate). That starvedHLA-A of coverage and it came back "Not typed" — the
with_revert_sampathonly worked because
GATK4_REVERTSAMdefaults to a queryname sort, maskingthe missing step.
Fix: added
SAMTOOLS_COLLATE(nf-core module) unconditionally beforeSAMTOOLS_FASTQ. Declared as a component dependency inhlahd_from_bam'smeta.yml, which CI fetches automatically via.github/scripts/install-components.sh— seedocs/ci.md(added in this PR) for how that mechanism works. All 16 checks passed on
the first version of this fix.
Follow-up:
components:schema didn't match the real nf-core CLIWhile testing "does this subworkflow actually install into a pipeline"
(not just pass our own CI),
nf-core subworkflows install hlahd_from_bamagainst this branch crashed with
AttributeError: 'str' object has no attribute 'get'.hlahd_from_bam'smeta.ymlwas the first subworkflowin the repo to declare an external (nf-core)
components:dependency, andthe shape used (
- name: ..., git_remote: ..., org_path: ...) only matchedwhat our own
install-components.shscript parses — not whatnf-core subworkflows installitself expects(
components_utils.py::get_components_to_install, which wants the dictnested under the component name:
- samtools/collate: {git_remote: ...}).So this passed CI fine but would have silently broken for the first
downstream pipeline to try installing it.
Fixed by switching
meta.ymlto nf-core's real nested shape and updatinginstall-components.shto parse it the same way, so both consumers agreeon one schema. Re-verified both:
install-components.shstill fetches all4 dependencies locally, and
nf-core subworkflows install hlahd_from_bamnow succeeds end-to-end in a scratch pipeline — installs the local
hlahdmodule plus all 4 transitive nf-core modules, correctly recorded in
modules.json. Documented indocs/ci.md.Checklist
hlahdbranch in test-datasets repometa.ymlcomplete for both module and subworkflow