Enhance DICOM processing and add HPC deployment support#15
Merged
Conversation
Enhance DICOM processing with multithreading and improved error handling
Update docstrings, TODOs, and fix KeyErrors in DICOM processing
- start_control.sh: auto-detect container runtime (Docker → Singularity → Apptainer) - Add mri_preprocessing.singularity.def matching existing Dockerfile layers - Exclude .sif images from git
- start_control.sh: detect Docker → Singularity/Apptainer → conda/mamba → error - environment.yml: conda env spec (dcm2niix + Python + test deps) - run_pipeline_conda.sh: standalone conda pipeline runner - code/scripts/install_niftyreg.sh: source build helper for reg_f3d
…partial pipeline runs
… correct typo in Singularity hint
…Lab/MRI_preprocessing into feature/hpc-singularity
Optimize DICOM processing and enhance logging and testing framework
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces major improvements to the MRI preprocessing pipeline, focusing on making it portable and easier to run across different environments (Docker, Singularity, or Conda-only HPC), as well as adding robust environment setup scripts and documentation. The most important changes are grouped below.
Unified and Flexible Pipeline Orchestration
00_preprocess.shto support running specific steps, custom input/output directories, and robust argument parsing. Steps can now be selectively run or skipped, and scan/output paths can be overridden from the command line.Cross-Platform Environment Support
start_control.shas a unified entry point that auto-detects the available runtime (Docker, Singularity, or Conda) and launches the pipeline accordingly, including logic for environment variable setup and runtime-specific instructions. [1] [2]mri_preprocessing.singularity.def) to enable building and running the pipeline on HPC systems without Docker, including all necessary dependencies and NiftyReg installation.AS base) for potential multi-stage builds.Conda-Based Local/HPC Deployment
environment.ymlspecifying all Python and system dependencies for Conda-based installs, making it easy to set up the environment on HPCs without containers.run_pipeline_conda.sh, an interactive script for running the pipeline natively in a Conda environment, including environment checks, dependency verification, and user prompts for data paths.NiftyReg Installation Support
scripts/install_niftyreg.shto automate building and installing NiftyReg from source, including CUDA detection and customizable install prefix, for environments where NiftyReg is not available as a module.These changes significantly improve the portability, usability, and robustness of the MRI preprocessing pipeline across diverse computing environments.