Optimize DICOM processing and enhance logging and testing framework

.github/workflows/docker-test.yml

@@ -0,0 +1,24 @@
+name: Docker Build
+
+on:
+  push:
+    branches: [main, develop, "**"]
+    paths:
+      - "control_system/dockerfile"
+      - "control_system/docker-compose*.yml"
+  pull_request:
+    branches: [main, develop]
+    paths:
+      - "control_system/dockerfile"
+      - "control_system/docker-compose*.yml"
+
+jobs:
+  docker-build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Build Docker image
+        run: |
+          cd control_system
+          docker build -t mri_preprocessing_test --target base .

.github/workflows/tests.yml

@@ -0,0 +1,41 @@
+name: Tests
+
+on:
+  push:
+    branches: [main, develop, "**"]
+  pull_request:
+    branches: [main, develop]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.10", "3.11", "3.12"]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+          pip install -r requirements-dev.txt
+
+      - name: Run unit tests (01_scanDicom)
+        run: |
+          python -m pytest test/test_scanDicom_unit.py -v
+
+      - name: Run full tests (01_scanDicom + 02_parseDicom)
+        run: |
+          python -m pytest test/test_scanDicom_full.py -v
+
+      - name: Run synthetic known-result tests (01 + 02 deterministic verification)
+        run: |
+          python -m pytest test/test_synthetic_known_result.py -v

.gitignore

@@ -19,4 +19,9 @@ tmp/*
 import os.py
 
 reset_02.sh
-*.log
+*.log
+*.json
+.aider*
+
+# Singularity images (large, site-specific)
+*.sif

README.md

@@ -1,63 +1,66 @@
 # MRI Preprocessing Pipeline
 
-A generalized implementation of MRI preprocessing for various ML/AI tasks within the Parra Lab. This project is designed to automate the ingestion, analysis, and processing of raw DICOM MRI data into model-ready inputs.
+A modular pipeline for automated MRI DICOM preprocessing. Converts raw DICOM MRI data into model-ready inputs through a series of numbered processing steps.
 
 ## Table of Contents
 
-- [Overview](#overview)
 - [Key Features](#key-features)
 - [Project Structure](#project-structure)
 - [Installation](#installation)
 - [Usage](#usage)
-  - [Starting the System](#starting-the-system)
-  - [Web Control Interface](#web-control-interface)
-  - [Command Line Interface (CLI)](#command-line-interface-cli)
+  - [Starting the Container](#starting-the-container)
+  - [Direct Container Access](#direct-container-access)
+  - [Running Preprocessing Steps](#running-preprocessing-steps)
 - [Preprocessing Workflow](#preprocessing-workflow)
 - [Testing](#testing)
-- [Contributing](#contributing)
-- [Acknowledgements](#acknowledgements)
-
-## Overview
-
-The MRI Preprocessing Pipeline is a modular system built to handle large datasets of MRI scans. It runs within a Docker container to ensure a consistent environment and supports both an interactive web-based control system and a scriptable command-line interface.
-
-The core functionality resides in `code/preprocessing/`, where a series of Python scripts handle everything from DICOM extraction to NIfTI conversion and spatial alignment.
 
 ## Key Features
 
-- **Automated Scanning**: Recursively scans directories for MRI DICOM files.
-- **Metadata Extraction**: Extracts and standardizes DICOM header information into CSV tables.
-- **Intelligent Parsing**: Identifies scan types (T1, T2, etc.) and orders sequences based on acquisition times.
-- **Modular Design**: Each step of the pipeline is a standalone script, allowing for flexible execution and debugging.
-- **Containerized Environment**: Fully Dockerized setup for easy deployment on Linux and WSL systems.
-- **Web Interface**: (In Development) A Flask-based dashboard to monitor and control the processing status.
+- **Automated DICOM Scanning**: Recursively scans directories for MRI DICOM files and extracts metadata.
+- **Intelligent Parsing**: Identifies scan types, filters artifacts, and orders sequences by acquisition time.
+- **NIfTI Conversion**: Converts DICOM series to NIfTI format using dcm2niix.
+- **Spatial Alignment**: Coregisters scans to a reference volume.
+- **Modular Design**: Each pipeline step is an independent script that can be run manually or in sequence.
+- **Containerized**: Docker image with all dependencies pre-installed (Python, pydicom, nibabel, niftyreg, dcm2niix).
 
 ## Project Structure
 
 ```
 MRI_preprocessing/
 ├── code/
-│   └── preprocessing/       # Core python scripts for data processing
-│       ├── 01_scanDicom.py  # Scans and extracts DICOM metadata
-│       ├── 02_parseDicom.py # Filters and orders scans
-│       ├── ...              # Subsequent processing steps
+│   └── preprocessing/       # Core Python preprocessing scripts
+│       ├── 01_scanDicom.py  # Scan DICOM files and extract metadata
+│       ├── 02_parseDicom.py # Filter and order scans
+│       ├── 03_saveNifti.py  # Convert DICOM to NIfTI
+│       ├── 04_saveRAS.py    # Reorient to RAS
+│       ├── 05_alignScans.py # Coregister scans
+│       ├── 06_genInputs.py  # Generate model inputs
 │       ├── DICOM.py         # DICOM handling utilities
-│       └── toolbox.py       # General helper functions
-├── control_system/          # Docker and Web App configuration
-│   ├── app/                 # Flask web application
-│   └── docker*              # Docker Compose files
-├── data/                    # Data storage (mounted volumes)
+│       ├── toolbox.py       # Shared helper functions
+│       └── 00_preprocess.sh # Run full pipeline
+├── control_system/          # Docker image and compose files
+│   ├── dockerfile           # Container image definition
+│   ├── docker-compose.yml   # Linux compose file
+│   ├── docker-compose-wsl.yml  # WSL compose file
+│   ├── startup.sh           # Container entrypoint
+│   └── README.md            # Container documentation
 ├── test/                    # Unit and integration tests
-├── start_control.sh         # Main entry point script
-└── install.py               # Dependency installation script
+├── docs/                    # Code reviews and improvement recommendations
+├── start_control.sh         # Container startup script
+├── access_preprocessing.sh  # Direct CLI access to container
+├── install.py               # Docker + NVIDIA toolkit installer (Linux)
+├── mount_kirbyPro.sh        # Machine-specific mount script
+├── requirements.txt         # Python runtime dependencies
+└── requirements-dev.txt     # Development/testing dependencies
 ```
 
 ## Installation
 
 ### Prerequisites
-- Linux or Windows Subsystem for Linux (WSL2)
-- Python 3.x
-- Docker & Docker Compose (installed automatically via `install.py` if not present)
+
+- Linux or WSL2
+- Python 3.10+
+- NVIDIA GPU (for preprocessing acceleration)
 
 ### Steps
 
@@ -67,85 +70,84 @@ MRI_preprocessing/
    cd MRI_preprocessing
    ```
 
-2. **Install dependencies and setup Docker:**
+2. **Install Docker and NVIDIA Container Toolkit:**
    ```bash
-   python3 install.py
+   sudo python3 install.py
    ```
-   *Note: This script attempts to install Docker and configure GPU access. If you prefer, you can install Docker manually.*
+   *This installs Docker, configures GPU access, and verifies the setup.*
 
 ## Usage
 
-### Starting the System
-
-The primary way to interact with the pipeline is through the `start_control.sh` script.
+### Starting the Container
 
 ```bash
 bash start_control.sh
 ```
 
-You will be prompted to:
-1.  Enable the webserver component (y/n).
-2.  Provide the path to your raw DICOM data on the host machine.
+You will be prompted for:
+1. The path to your raw DICOM data directory
+2. The path for NIfTI output
 
-The system maps your local data directory to `/FL_system/data/raw/` inside the Docker container.
+The container mounts your host directories into `/FL_system/data/raw/` and `/FL_system/data/nifti/` inside the container.
 
-### Web Control Interface
-If enabled, the web interface is accessible at `http://localhost:5000`. It provides a dashboard to view the status of the preprocessing steps.
-*(Note: The web interface is currently under active development).*
+### Direct Container Access
 
-### Command Line Interface (CLI)
-For batch processing or direct control, you can access the container's shell:
+While the container is running:
 
-**Option 1: Convenience Script**
 ```bash
 bash access_preprocessing.sh
 ```
 
-**Option 2: Direct Docker Exec**
+This opens an interactive shell inside the container. Navigate to `/FL_system/code/preprocessing/` to run preprocessing scripts.
+
+### Running Preprocessing Steps
+
+Each step can be run manually:
+
 ```bash
-docker exec -it control bash
-cd /FL_system/code/preprocessing/
+# Step 1: Scan DICOM files
+python 01_scanDicom.py --scan_dir /FL_system/data/raw --save_dir /FL_system/data
+
+# Step 2: Parse and filter
+python 02_parseDicom.py --save_dir /FL_system/data
+
+# Full pipeline:
+bash /FL_system/code/preprocessing/00_preprocess.sh
 ```
 
 ## Preprocessing Workflow
 
-The pipeline consists of numbered scripts in `code/preprocessing/` that should generally be run in order:
+The pipeline consists of numbered scripts that should generally be run in order:
 
-1.  **01_scanDicom.py**: Scans raw data and builds a `Data_table.csv` of all found DICOM files.
-    *   *Documentation*: See `code/preprocessing/01_scanDicom.py` for detailed usage and arguments.
-2.  **02_parseDicom.py**: Filters relevant scans (e.g., T1) and orders them by time.
-3.  **03_saveNifti.py**: Converts selected DICOM series to NIfTI format.
-4.  **04_saveRAS.py**: Reorients NIfTI files to RAS orientation.
-5.  **05_alignScans.py**: Aligns scans to a reference volume.
-6.  **06_genInputs.py**: Generates final model inputs.
+1. **01_scanDicom.py** — Scans raw DICOM data, extracts metadata, produces `Data_table.csv`
+2. **02_parseDicom.py** — Filters scans (removes T2, DWI, computed images), orders by trigger time, produces `Data_table_timing.csv`
+3. **03_saveNifti.py** — Converts selected DICOM series to NIfTI format using dcm2niix
+4. **04_saveRAS.py** — Reorients NIfTI files to RAS orientation
+5. **05_alignScans.py** — Coregisters all scans to a reference volume
+6. **06_genInputs.py** — Generates numpy inputs for model training
 
-To run a specific step manually inside the container:
-```bash
-python 01_scanDicom.py --scan_dir /FL_system/data/raw --save_dir /FL_system/data
-```
+Intermediate outputs:
+- `/FL_system/data/Data_table.csv` — DICOM metadata table (step 01 output)
+- `/FL_system/data/Data_table_timing.csv` — Filtered and ordered table (step 02 output)
+- `/FL_system/data/nifti/` — NIfTI files (step 03 output)
+- `/FL_system/data/RAS/` — RAS-oriented NIfTI files (step 04 output)
+- `/FL_system/data/coreg/` — Coregistered scans (step 05 output)
+- `/FL_system/data/inputs/` — Final model inputs (step 06 output)
 
 ## Testing
 
-Unit and integration tests are located in the `test/` directory.
-
-To run tests (ensure you have `pytest` installed):
 ```bash
-pytest test/
-```
+# Run all tests
+pytest test/ -v
 
-## Contributing
+# Run unit tests only (fastest)
+pytest test/test_scanDicom_unit.py -v
 
-1.  Fork the repository.
-2.  Create a feature branch (`git checkout -b feature/NewFeature`).
-3.  Commit your changes.
-4.  Push to the branch.
-5.  Open a Pull Request.
+# Run comprehensive tests
+pytest test/test_scanDicom_full.py -v
 
-Please ensure all new code is well-documented and passes existing tests.
-
-## Acknowledgements
-- [Parra Lab](https://www.ccny.cuny.edu/bme/people/lucas-parra)
-- Contributors: [Add names here]
+# Run deterministic known-result tests
+pytest test/test_synthetic_known_result.py -v
+```
 
----
-*For questions or support, please contact nleotta000@citymail.cuny.edu*
+Test coverage for `01_scanDicom.py` is comprehensive (89 tests). See `test/TESTS.md` for the full test suite documentation.

access_preprocessing.sh

@@ -1,7 +1,7 @@
 #!/bin/bash
 
 echo "MRI Preprocessing - Direct CLI Access"
-echo "====================================="
+echo "======================================"
 echo ""
 echo "This script provides direct access to the preprocessing container"
 echo "without starting the webserver component."
@@ -11,7 +11,6 @@ echo ""
 if ! docker ps --format "table {{.Names}}" | grep -q "^control$"; then
     echo "Error: The control container is not running."
     echo "Please start the system first with: bash start_control.sh"
-    echo "And choose 'n' when asked about the webserver component."
     exit 1
 fi