Multimodal deep learning framework integrating multiplex immunofluorescence imaging, spatial transcriptomics, and clinical metadata for prediction of brain tumor-related epilepsy in glioma.
PNN-MultiModal is a comprehensive deep learning framework designed to predict brain tumor-related epilepsy (BTE) in glioma patients by integrating three complementary data modalities:
- Multiplex Immunofluorescence (mIF) Imaging: High-dimensional spatial protein expression data capturing the tumor microenvironment composition
- Spatial Transcriptomics: Gene expression patterns with preserved spatial context within tumor regions
- Clinical Metadata: Patient demographics, tumor characteristics, and clinical outcomes
This framework leverages state-of-the-art deep learning techniques to uncover the complex interactions between immune infiltration, molecular profiles, and clinical manifestations of seizure risk [...]
- Multimodal Integration: Seamlessly combines imaging, genomic, and clinical data
- Spatial Context Preservation: Maintains spatial relationships crucial for understanding tumor microenvironment
- Interpretability: Provides insights into which modalities and features drive epilepsy prediction
- End-to-End Pipeline: From data preprocessing to model evaluation and validation
- Reproducible Research: Comprehensive notebooks and documentation for full transparency
PNN-MultiModal/
├── README.md # This file
├── notebooks/ # Jupyter notebooks for analysis and experiments
│ ├── data_exploration.ipynb
│ ├── preprocessing.ipynb
│ ├── model_training.ipynb
│ └── evaluation.ipynb
├── src/ # Python source code
│ ├── models/ # Deep learning model architectures
│ ├── data/ # Data loading and preprocessing utilities
│ ├── utils/ # Helper functions and utilities
│ └── evaluation/ # Evaluation metrics and visualization
├── data/ # Data directory (not included in repo)
├── results/ # Output directory for model results
└── requirements.txt # Python dependencies
- Deep Learning: PyTorch / TensorFlow
- Data Processing: NumPy, Pandas, SciPy
- Image Analysis: scikit-image, OpenCV
- Spatial Analysis: Scanpy, Squidpy (for spatial transcriptomics)
- Visualization: Matplotlib, Seaborn, Plotly
- Development: Jupyter Notebook, Python 3.8+
- Python 3.8 or higher
- CUDA 11.0+ (for GPU acceleration, optional but recommended)
- Sufficient storage for datasets (varies by data size)
- Clone the repository:
git clone https://github.com/BMIRDS/PNN-MultiModal.git
cd PNN-MultiModal- Create a virtual environment:
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate- Install dependencies:
pip install -r requirements.txt-
Prepare your data in the following format:
- mIF Images: TIFF or PNG format, organized by patient/region
- Spatial Transcriptomics: H5AD format (Anndata) or CSV with spatial coordinates
- Clinical Metadata: CSV with patient IDs and clinical features
-
Place data in the
data/directory following the structure defined indata_preparation.ipynb -
Run preprocessing notebooks to standardize and normalize data
- Start with
notebooks/data_exploration.ipynbto understand your data - Run
notebooks/preprocessing.ipynbto prepare datasets - Execute
notebooks/model_training.ipynbto train models - Analyze results with
notebooks/evaluation.ipynb
from src.models import MultimodalDNN
from src.data import DataLoader
# Load and prepare data
train_loader = DataLoader.load_train_data('data/train')
val_loader = DataLoader.load_val_data('data/val')
# Initialize model
model = MultimodalDNN(
img_channels=5, # Number of protein markers
transcriptomics_dim=2000, # Number of genes
clinical_features=10 # Number of clinical features
)
# Train model
model.train(train_loader, val_loader, epochs=50, learning_rate=0.001)
# Evaluate
results = model.evaluate(test_loader)# Load trained model
model = MultimodalDNN.load('results/best_model.pt')
# Prepare patient data
patient_data = {
'mif_image': img_array,
'transcriptomics': expr_vector,
'clinical': clinical_features
}
# Predict epilepsy risk
prediction = model.predict(patient_data)
print(f"Seizure Risk Score: {prediction['risk_score']:.3f}")
print(f"Feature Importance: {prediction['important_features']}")- Format: Multi-channel TIFF (one channel per protein marker)
- Resolution: 512×512 or higher
- Channels: 3-6 protein markers (e.g., GFAP, Iba1, CD3, NeuN, DAPI)
- Output: Normalized to [0, 1] range
- Format: H5AD (AnnData) or CSV
- Content: Gene expression matrix with spatial coordinates
- Genes: Full transcriptome or pre-selected marker genes (500-5000)
- Output: Log-normalized expression values
- Format: CSV
- Fields: Patient ID, age, gender, tumor grade, tumor location, seizure status, etc.
- Output: Standardized features with appropriate scaling
The framework employs a multimodal neural network architecture:
- Image Encoder: CNN-based encoder for spatial protein patterns
- Transcriptomics Encoder: Graph neural network or transformer for gene expression
- Clinical Encoder: Fully connected layers for tabular clinical data
- Fusion Layer: Multi-head attention mechanism for cross-modal interactions
- Prediction Head: Binary classification for seizure risk or regression for risk score
For detailed architecture descriptions, see src/models/README.md
The model is evaluated using:
- Classification Metrics: Accuracy, Precision, Recall, F1-Score, AUC-ROC
- Cross-Validation: Stratified 5-fold cross-validation
- Feature Importance: SHAP values and attention weights
- Ablation Studies: Analysis of contribution from each modality
- Clinical Validation: Performance on held-out test cohort
Results are saved in results/ directory with visualizations and detailed reports.
If you use this framework in your research, please cite:
@software{pnn_multimodal_2026,
title={PNN-MultiModal: Multimodal Deep Learning for Brain Tumor-Related Epilepsy Prediction},
author={Liu, Wenjun and Sadanandappa, Madhumala and Palisoul, Scott and Zanazzi, George and Hong, Jennifer and Hassanpour, Saeed},
year={2026},
url={https://github.com/BMIRDS/PNN-MultiModal}
}This project is licensed under the GNU General Public License v3.0 - see the LICENSE file for details.
We welcome contributions from the research community! Please:
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request with a clear description of changes
For major changes, please open an issue first to discuss proposed modifications.
For questions, issues, or suggestions, please:
- Open an issue on GitHub Issues
- Contact the BMIRDS lab: Visit BMIRDS
- Check existing documentation in the
docs/directory
Last Updated: 2026
Status: Active Development
Maintainers: BMIRDS Team