Introduction

RareCollab is a Python package for rare diseases powered by Ollama. It integrates multimodal patient data, including DNA, RNA, and phenotype information, to support candidate variant prioritization and diagnostic interpretation.

Our paper is available on arXiv: https://arxiv.org/abs/2602.04058

Before Running RareCollab

Before running RareCollab, please prepare the following files and directories:

1. Please download the RareCollab-data-dependencies from [link].

2. Create a work folder for intermediate files generated by RareCollab.

3. Create an output folder for RareCollab results.

4. Create a reference folder and download all required reference files from here. (If you have RNA data)

5. Create an NCBI API key using your email address from NCBI API Keys. This step is optional, but it can speed up searches using the NCBI and Entrez APIs.

Usage

Installation

Run the code below to install RareCollab:

!pip install git+https://github.com/LiuzLab/RareCollab.git

---

Step 1. Setup

1.1 Check required command-line tools

import RareCollab

RareCollab.Setup.CheckRequiredTools()

Follow the on-screen instructions to install any missing tools into your target environment.

Tip

When you see the message All required command-line tools are available., all external dependencies have been correctly installed and you can proceed to the next step.

1.2 Configure the paths

# Path to the reference data dependencies folder
ref_dir = '/path/RareCollab-data-dependencies-1.0'

# Path to your working folder (will be created if it does not exist)
work_dir = '/path/work'

# Reference genome build: 'hg38' or 'hg19'
ref_ver = 'hg38'

1.3 Prepare the references and container images

This step locates the reference files, builds the required indexes, and sets up the Singularity images that the pipeline will use:

# Locate and validate the reference input files for the chosen genome build
references = RareCollab.Setup.ResolveReferenceInputs(ref_dir=ref_dir, ref_ver=ref_ver)

# Build the FASTA index files required for downstream analysis
fasta_references = RareCollab.Setup.BuildReferenceIndex(ref_dir=ref_dir, ref_ver=ref_ver)

# Pull and prepare the Singularity images for the required tools
singularity_images = RareCollab.Setup.PrepareSingularityImages(ref_dir=ref_dir)

Each call returns a handle (references, fasta_references, singularity_images) that later steps will use, so run all three before continuing.

1.4 Load the samplesheet

# Load the samplesheet and validate its configuration
samplesheet = RareCollab.Setup.LoadSamplesheet(csv_path='/path/samplesheet.csv', fulfill_empty_hpo=False)

The samplesheet must be a CSV file containing exactly the following three columns:

Column	Description
sampleID	A unique identifier for each sample. Must not be duplicated. Letters, digits, hyphens (-), and underscores (_) are safe to use. To avoid potential parsing issues, it is best to avoid other special characters (such as spaces, /, \, *, or #).
vcf_path	The absolute path to the sample's VCF file.
hpo_path	The absolute path to the sample's HPO file.

Each HPO file is a plain-text (.txt) file containing a list of HPO terms, one per line, in the form HP:XXXXXXX. When fulfill_empty_hpo=True, if no HPO file exists at the specified hpo_path, a default HPO file will be created automatically at that location.

Note

A demo samplesheet and a demo HPO file are provided in the demo/ folder of the repository. You can use them as a reference for the expected format.

1.5 Configure the worker settings

# Automatically recommend parallelization settings based on the samplesheet
config = RareCollab.Setup.RecommendWorkerConfig(samplesheet)

Caution

You can adjust the parallelization settings in config if needed, but doing so is at your own risk — overriding the recommended values may lead to excessive resource usage or unstable runs.

---

Step 2. Generate Features

This step turns each sample's VCF into the features used by the downstream analysis. It runs in two stages — first processing the VCFs, then generating the features — and each stage updates samplesheet with the results:

# Stage 1: Process the VCF files (split, normalize, etc.)
samplesheet = RareCollab.Features.ProcessVCF(
    samplesheet,
    max_workers=config['split_workers'],
    work_dir=work_dir,
    references=references,
    fasta_references=fasta_references,
    overwrite=False,
)

# Stage 2: Generate features from the processed VCFs
samplesheet = RareCollab.Features.GenerateFeatures(
    samplesheet,
    work_dir=work_dir,
    references=references,
    fasta_references=fasta_references,
    singularity_images=singularity_images,
    ref_ver=ref_ver,
    config=config,
    overwrite=False,
)

Run both calls in order, since GenerateFeatures depends on the output of ProcessVCF.

Note

With overwrite=False, samples whose results already exist in work_dir are skipped, so you can safely re-run this step to resume an interrupted run. Set overwrite=True to force every sample to be reprocessed from scratch.

Optional: RNA Preprocessing

If RNA data are available, run:

RareCollab.Preprocessing.RNA(
    work_path=work_dir,
    splicing_path=splicing_path,
    expression_path=expression_path,
    ase_path=ase_path,
)

Parameter	Type	Description
work_path	str	Path to the work directory.
splicing_path	str	Path to the output from FRASER2.
expression_path	str	Path to the output from OUTRIDER.
ase_path	str	Path to the output from GATK ASEReadCounter.

---

Step 3. Run RareCollab Agents

3.1 Launch the LLM Server

Before running the downstream LLM-based analysis, you need to start an LLM server and keep it listening, then capture its connection details into llm_config. Run:

# Launch the LLM server (keeps listening for requests)
server = RareCollab.Setup.LaunchLLMServer(
    partition="partition",
    nodelist="node",
    port=12321,
    num_parallel=2,
    model_name="gpt-oss:20b",
)

# Capture the server's connection details into a config object
llm_config = RareCollab.Setup.LLMConfig(
    model_name=server["model_name"],
    ollama_url=server["ollama_url"],
    num_parallel=server["num_parallel"],
    temperature=0.7,
)

Parameters — LaunchLLMServer

Parameter	Type	Description
partition	str	The partition to launch the server on.
nodelist	str	The node (or nodes) to run the server on.
port	int	The port the server listens on.
num_parallel	int	Number of parallel instances. Increase this to run requests in parallel, based on your available GPU capacity.
model_name	str	The LLM model to serve (e.g., gpt-oss:20b).

Parameters — LLMConfig

Parameter	Type	Description
model_name	str	The served model name. Pass through from server["model_name"].
ollama_url	str	The server's URL. Pass through from server["ollama_url"].
num_parallel	int	Number of parallel requests. Pass through from server["num_parallel"].
temperature	float	Sampling temperature. A value of 0.7 is recommended.

Note

When the server starts, LaunchLLMServer prints its SLURM job ID, for example: Submitted SLURM job id: xxxxxxxx. Keep this ID — you'll need it to stop the server later.

To shut the server down when you're done, run:

# Stop the LLM server using the SLURM job ID printed at launch
RareCollab.Setup.StopLLMServer(SLURM_job_id)

3.2 Run the Diagnostic Agents (Serial)

If you have only a single LLM available, run the diagnostic agents serially as shown below. Each agent updates samplesheet and passes it to the next one, so they must be run in order.

We recommend providing an NCBI email and API key — they're used by the database and literature agents. If you don't have them, set both to None.

# NCBI credentials — used by the database and literature agents.
# If you don't have them, set both to None (queries may then be rate-limited).
NCBI_EMAIL = "your_ncbi@email.com"   # or None
NCBI_KEY   = "your-api-key"          # or None

# Run the Mixture-of-Experts (MoE) diagnostic engine
samplesheet = RareCollab.DiagnosticEngine.MoE(
    samplesheet=samplesheet,
    work_dir=work_dir,
    references=references,
)

# Generate the candidate gene/variant list
samplesheet = RareCollab.DiagnosticEngine.Candidates(
    samplesheet=samplesheet,
    work_dir=work_dir,
    config=config,
    overwrite=False,
)

# Database agent: query external databases for each candidate (uses NCBI)
samplesheet = RareCollab.DatabaseAgent.RunAgent(
    samplesheet=samplesheet,
    work_dir=work_dir,
    references=references,
    llm_config=llm_config,
    ncbi_email=NCBI_EMAIL,
    ncbi_api_key=NCBI_KEY,
    config=config,
    overwrite=False,
)

# In-silico agent: run in-silico prediction/analysis on the candidates
samplesheet = RareCollab.InSilicoAgent.RunAgent(
    samplesheet=samplesheet,
    work_dir=work_dir,
    llm_config=llm_config,
    overwrite=False,
)

# Phenotype agent: preprocess the phenotype (HPO) data
samplesheet = RareCollab.PhenotypeAgent.Preprocessing(
    samplesheet=samplesheet,
    work_dir=work_dir,
    references=references,
    overwrite=False,
)

# Phenotype agent: analysis based on HPO terms
samplesheet = RareCollab.PhenotypeAgent.RunAgent_HPO(
    samplesheet=samplesheet,
    work_dir=work_dir,
    llm_config=llm_config,
    overwrite=False,
)

# Phenotype agent: analysis against OMIM
samplesheet = RareCollab.PhenotypeAgent.RunAgent_OMIM(
    samplesheet=samplesheet,
    work_dir=work_dir,
    llm_config=llm_config,
    overwrite=False,
)

# Phenotype agent: analysis from the literature (uses NCBI)
samplesheet = RareCollab.PhenotypeAgent.RunAgent_Literature(
    samplesheet=samplesheet,
    work_dir=work_dir,
    llm_config=llm_config,
    ncbi_email=NCBI_EMAIL,
    ncbi_api_key=NCBI_KEY,
    overwrite=False,
)

Note

As before, overwrite=False lets you safely re-run this block to resume an interrupted run — completed steps are skipped. Set overwrite=True on a given call to force it to recompute.

---

Step 4. Integrate the Results

The final step merges the outputs from all the diagnostic agents into a single integrated result and writes it to output_path:

# Merge all agent outputs into the final integrated result
samplesheet = RareCollab.Integration.Review(
    samplesheet=samplesheet,
    work_dir=work_dir,
    fasta_references=fasta_references,
    output_path='/path/output',
    overwrite=False,
)

After this step completes, your final integrated results are available at output_path.

Output Format

The integrated results are written as CSV files, one per sample (split by identifier / sampleID). Within each CSV, candidate variants are already ranked

The columns in each output CSV are described below.

Variant & Gene Identity

Column	Description
varId	Variant ID, formatted as chromosome + _ + position.
identifier	Sample ID.
geneSymbol	Gene symbol.
HGVSc	Coding-level (cDNA) variant nomenclature in HGVS format.
HGVSc_core	Final retained cDNA HGVS string.
HGVSp	Protein-level variant nomenclature in HGVS format.
transcript_id	Ensembl transcript ID.
geneSymbol_VarId	Combined gene-symbol + variant-ID key.
Chromosome	Chromosome.
Pos	Variant position.
Start	Start coordinate.
End	End coordinate.

Inheritance & Variant Class

Column	Description
dominant	1 = dominant, 0 = not dominant.
recessive	1 = recessive, 0 = not recessive.
zyg	Zygosity flag: 1 = heterozygous site, 0 = otherwise.
cons_frameshift_variant	1 = frameshift variant, 0 = not.
frame_shift	Whether the variant causes a frameshift.
transcript_score	Score for the transcript, computed as a weighted sum based on MANE status and the presence of HGVSp/HGVSc.

Gene Constraint Metrics (gnomAD)

Column	Description
gnomadGenePLI	gnomAD pLI score (probability of loss-of-function intolerance).
gnomadGeneOELof	gnomAD observed/expected ratio for loss-of-function variants.
gnomadGeneOELofUpper	Upper bound of the gnomAD LOEUF (O/E LoF confidence interval).

Diagnostic Engine (Mixture-of-Experts) Scores

The overall score comes from a Mixture-of-Experts (MoE) model. Each expert module produces a score and a rank.

Column	Description
overall_logit	Final-layer logit from the MoE model (interconvertible with overall_prob).
overall_prob	Overall probability (the sigmoid-transformed overall_logit).
score_Database / rank_Database	Score / rank from the Database module.
score_Genetics / rank_Genetics	Score / rank from the Genetics module.
score_InSilico / rank_InSilico	Score / rank from the InSilico module.
score_Overview / rank_Overview	Score / rank from the Overview module.
score_Phenotype / rank_Phenotype	Score / rank from the Phenotype module.
Diagnostic_Engine_Rank	Final rank from the MoE model.

RNA — OUTRIDER (expression outliers)

All Outrider_* columns require the RNA module; they are empty if RNA was not run.

Column	Description
Outrider_pValue	OUTRIDER p-value.
Outrider_padjust	OUTRIDER adjusted p-value.
Outrider_zScore	OUTRIDER z-score (from its fitted distribution).
Outrider_l2f	Log2 fold change.
Outrider_rawcounts	Raw read counts.
Outrider_RawZscore	Z-score computed directly from raw values (differs from Outrider_zScore because OUTRIDER's model z-score is not based on a plain normal distribution).

RNA — FRASER 2.0 (splicing outliers)

All Fraser_* columns come from FRASER 2.0 and require the RNA module; empty if RNA was not run. Names follow FRASER's own terminology.

Column	Description
Fraser_GenePvalue	Gene-level p-value.
Fraser_pvaluesBetaBinomial_jaccard	Beta-binomial p-value for the Jaccard metric.
Fraser_psi5 / Fraser_psi3	PSI values for 5′ / 3′ splice-site usage.
Fraser_rawOtherCounts_psi5 / Fraser_rawOtherCounts_psi3	Raw counts of "other" reads for the psi5 / psi3 metrics.
Fraser_rawCountsJnonsplit	Raw non-split read counts at the junction.
Fraser_jaccard	Jaccard splicing metric.
Fraser_rawOtherCounts_jaccard	Raw "other" counts for the Jaccard metric.
Fraser_delta_jaccard / Fraser_delta_psi5 / Fraser_delta_psi3	Deviation (delta) from expected value for each metric.
Fraser_predictedMeans_jaccard	Model-predicted mean for the Jaccard metric.
Fraser_junction_start / Fraser_junction_end	Splice junction start / end coordinates.

RNA — Allele-Specific Expression (ASE)

ASE_* columns come from the ASE analysis.

Column	Description
ASE_REF / ASE_ALT	Reference / alternate allele.
ASE_REF_COUNT / ASE_ALT_COUNT	Read counts supporting the reference / alternate allele.
ASE_ALT_RATIO	Fraction of reads supporting the alternate allele.
ASE_PVAL	P-value for allelic imbalance.
IS_MAE	Whether the variant shows mono-allelic expression.

RNA — Read Support Counts

All ref_* / alt_* columns are RNA-derived; 0 when the RNA module was not run.

Column	Description
ref_count_max / ref_count_mean / ref_count_min	Max / mean / min reference-allele read counts.
alt_count_max / alt_count_mean / alt_count_min	Max / mean / min alternate-allele read counts.

Candidate Selection & Compound Heterozygosity

Column	Description
is_compound_het_group	Whether the variant belongs to a compound-het group.
evidence_rules	Reason(s) the variant was selected as a disease candidate.
find_compound_het	Whether the variant is part of a compound het.
partner	The paired variant, if this is a compound het.
compound_het_group_id	Group ID for the compound-het group, if applicable.

LLM Agent Judgments

These columns hold conclusions from the LLM agents. An empty value means the candidate did not meet the prerequisites for that judgment (e.g. missing database info). When populated, conclusions may be one of: supporting, opposing, neutral, relevant, not relevant, insufficient information, etc.

Column	Description
Database_Reasoning / Database_Conclusion	Reasoning / conclusion from the Database agent.
Database_Zygosity	Zygosity call from the Database agent.
HPO_Reasoning / HPO_Conclusion	Reasoning / conclusion from the HPO (phenotype) agent.
OMIM_Reasoning / OMIM_Conclusion	Reasoning / conclusion from the OMIM agent.
Literature_Reasoning / Literature_Conclusion	Reasoning / conclusion from the Literature agent.
Insilico_Reasoning / Insilico_Conclusion	Reasoning / conclusion from the in-silico prediction agent.
RNA_GeneLevel_Reasoning / RNA_GeneLevel_Event / RNA_GeneLevel_Conclusion	Gene-level reasoning / event / conclusion from the RNA agent.
RNA_VarLevel_Reasoning / RNA_VarLevel_Event / RNA_VarLevel_Conclusion	Variant-level reasoning / event / conclusion from the RNA agent.

Flags & Tiering

Column	Description
has_repeat_motif	Whether a repeat motif is present (requires RNA; empty otherwise).
misscall_rna_flag	Whether the variant is likely a miscall (requires RNA; empty otherwise).
new_diseasegene_flag	Whether this is a novel disease gene.
omim_flag	Whether the gene is an OMIM gene.
strong_nom_flag	Whether this is a strong candidate.
Tier	Grouping by evidence strength: 1 = most relevant, 2 = intermediate, 3 = least relevant / least evidence.
new_rank	Updated per-gene rank.

Pairwise Comparison Module

All PairWise* / pairwise_* columns come from the pairwise (head-to-head) comparison module.

Column	Description
PairWiseScore	Pairwise comparison score.
PairWiseWins / PairWiseLosses / PairWiseTies	Number of head-to-head wins / losses / ties.
PairWiseEntityRank	Final rank from the pairwise comparison.
PairWiseWithinEntityRank	For a compound het, whether this variant is the higher-ranked (1) or lower-ranked (2) member.
pairwise_entity_id	Entity ID in the comparison module.
pairwise_entity_type	Entity type in the comparison module.