Optimization Solver Benchmark System

A research tool for benchmarking optimization solvers across LP, QP, SOCP, and SDP problems using external problem libraries (DIMACS, SDPLIB) with automated execution and HTML report generation.

Benchmark results are continuously published at: https://napinoco.github.io/optimization-solver-benchmark/

⚠️ Development Status: This system is currently under active development. Some solvers and problem classes have not yet been properly measured, and comprehensive benchmarking coverage across all problem types is still being completed.

Overview

Purpose: Compare optimization solver performance using standardized problem sets and minimal solver configuration for unbiased evaluation.

Problem Libraries:

• DIMACS: SeDuMi .mat format
• SDPLIB: SDPA .dat-s format
• Roughly 120 registered problems; see config/problem_registry.yaml for the authoritative list

Supported Solvers:

• Python (9): SciPy, CVXPY backends (CLARABEL, SCS, ECOS, OSQP, CVXOPT, SDPA, SCIP, HIGHS)
• MATLAB (2): SeDuMi, SDPT3

Quick Start

Installation

git clone --recursive https://github.com/napinoco/optimization-solver-benchmark.git
cd optimization-solver-benchmark
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate
pip install -r requirements.txt

# For MATLAB solvers (optional)
# Install MATLAB R2020a or later

# Setup MATLAB solvers - compiles MEX files for SeDuMi and SDPT3
matlab -batch "setup_matlab_solvers"

Usage

# Validate environment
python main.py --validate

# Run all benchmarks
python main.py --all

# Specific libraries
python main.py --benchmark --library_names DIMACS
python main.py --benchmark --library_names SDPLIB

# Generate reports only
python main.py --report

# Test mode (no database changes)
python main.py --benchmark --dry-run

# Timeout configuration
python main.py --all --timeout 300                    # 5-minute timeout
python main.py --benchmark --timeout 60               # Quick tests with 1-minute limit
python main.py --benchmark --library_names SDPLIB --timeout 1800  # 30-minute timeout for difficult SDP problems

Database Management

Database Restoration

The system maintains results in both SQLite database format and JSON/CSV exports. The database can be restored from JSON exports using the table restoration tool:

# Restore database from default JSON export
python scripts/database/table_restorer.py

# Restore from specific JSON file
python scripts/database/table_restorer.py --input-json /path/to/data.json --output-db database/results.db

# Run restoration tests
python scripts/database/table_restorer.py --test

# Compare restored database with original
python scripts/database/table_restorer.py --compare-with database/original_results.db

Why JSON-based Management?

• Repository Size: Binary database files (results.db ~5MB) are excluded from git tracking to keep repository lightweight
• Version Control: JSON/CSV exports provide meaningful diffs and change tracking
• Data Portability: Published JSON data can recreate the complete database for analysis
• Recovery: Simple restoration process from published benchmark results

Design Philosophy

• Fair baseline benchmarking: solver default parameters for genuine "out of the box" comparison
• Reproducible results: problem libraries pinned via git submodules, solver versions pinned in requirements.txt
• Complete version tracking: every result stored with solver versions, Git commit, and environment details

See basic_design.md for the full design principles.

Project Structure

├── main.py                     # Entry point
├── config/                     # YAML configuration files
├── scripts/
│   ├── benchmark/              # Execution engine
│   ├── solvers/                # Python/MATLAB solver interfaces
│   ├── data_loaders/           # Problem format loaders (MAT/DAT)
│   ├── database/               # SQLite storage
│   └── reporting/              # HTML generation
├── problems/
│   ├── DIMACS/                 # External library (git submodule)
│   └── SDPLIB/                 # External library (git submodule)
├── database/                   # SQLite database files
│   └── results.db              # Benchmark results storage
└── docs/                       # Generated reports and documentation

Architecture

Modular Design:

• Loaders: Convert problem files → standardized ProblemData
• Runners: Execute solvers on ProblemData → SolverResult
• Interfaces: Coordinate problem loading and solver execution

Multi-Language Support: Unified Python/MATLAB integration via JSON data exchange and standardized result format.

Adding Components

New Solvers

Python: Add an entry to scripts/solvers/python/solver_configs.py
MATLAB: Create {solver}_runner.m and add to MATLAB_SOLVER_CONFIGS in matlab_process_interface.py

New Problems

Add external libraries as git submodules in problems/ and extend loaders in scripts/data_loaders/

Documentation

• CLAUDE.md: Development context and instructions
• docs/development/: Technical design and implementation details

License

MIT License - see LICENSE file.

Acknowledgments

• External Problem Libraries: VSDP team for DIMACS and SDPLIB repositories
• MATLAB Solvers: SQLP team for maintaining Git repositories of SeDuMi and SDPT3 solvers
• Open-Source Solvers: SciPy, CVXPY, CLARABEL, SCS, ECOS, OSQP, CVXOPT, SDPA, SCIP, HIGHS communities
• Development Support: Claude Code for coding assistance and development guidance