Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 12 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,17 @@
This repository contains a set benchmarks for evaluating the performance
of different image reconstruction methods implemented in the DeepInverse library.

- **DeepInverse documentation**: https://deepinv.github.io
- **DeepInverse documentation**: https://deepinv.org
- **DeepInverse repository**: https://github.com/deepinv/deepinv

**Leaderboards** are automatically generated and can be found in the [DeepInverse benchmarks documentation](https://deepinv.github.io/deepinv/benchmarks.html).
**Leaderboards** are automatically generated and can be found in the [DeepInverse benchmarks documentation](https://deepinv.org/deepinv/benchmarks.html).

Benchmark results are stored in a HuggingFace repository: https://huggingface.co/datasets/deepinv/benchmarks/tree/main

### Evaluating Your Reconstruction Methods

To evaluate your own reconstruction methods on these benchmarks,
install deepinv_bench:
install `deepinv_bench`:

```bash
pip install git+https://github.com/deepinv/benchmarks.git#egg=deepinv_bench
Expand All @@ -34,27 +34,29 @@ my_solver = dinv.models.RAM() # replace with your reconstruction method
results = run_benchmark(my_solver, "benchmark_name")
```

where `benchmark_name` is the name of the benchmark and `my_solver` is your reconstruction method which receives `(y, physics)` where
where `benchmark_name` is the name of the benchmark and `my_solver` is your out-of-the-box reconstruction method which receives `(y, physics)` where

- `y` is a `torch.Tensor` containing the measurements,
- `physics` is the forward operator, see more details in the [DeepInverse physics documentation](https://deepinv.github.io/deepinv/user_guide/physics/intro.html).
- `physics` is the forward operator, see more details in the [DeepInverse physics documentation](https://deepinv.org/deepinv/user_guide/physics/intro.html).

### Adding New Solvers

To add a new solver to an existing benchmark, open a new pull request on this repository, adding a new `your_solver_name.py` file
in the corresponding benchmark folder. Follow the structure of the existing solver files.
The new solver will be automatically run once the pull request is merged, and the results will be added to the leaderboard.

Also consider adding your solver to the collection of [existing solvers in DeepInverse](https://deepinv.org/user_guide/reconstruction/pretrained-models.html)!

### Adding New Benchmarks

To create a new benchmark, open a new pull request adding a new folder following
the structure given in the existing [benchmark_template](https://github.com/deepinv/benchmarks) folder.

A new benchmark requires:

- A **dataset** from the [DeepInverse datasets](https://deepinv.github.io/deepinv/user_guide/training/datasets.html).
- A **forward operator** from the [DeepInverse operators](https://deepinv.github.io/deepinv/user_guide/physics/physics.html).
- A set of **reconstruction methods** from the [DeepInverse reconstructors](https://deepinv.github.io/deepinv/user_guide/reconstruction/introduction.html) and potentially other custom solvers.
- A set of **metrics** from the [DeepInverse metrics](https://deepinv.github.io/deepinv/user_guide/training/metric.html).
- A **dataset** from the [DeepInverse datasets](https://deepinv.org/deepinv/user_guide/training/datasets.html).
- A **forward operator** from the [DeepInverse operators](https://deepinv.org/deepinv/user_guide/physics/physics.html).
- A set of out-of-the-box **reconstruction methods** from the [DeepInverse reconstructors](https://deepinv.org/deepinv/user_guide/reconstruction/introduction.html) and potentially other custom solvers.
- A set of **metrics** from the [DeepInverse metrics](https://deepinv.org/deepinv/user_guide/training/metric.html).

If you would like to propose a new dataset, metric or forward operator, please [open an issue](https://github.com/deepinv/benchmarks/issues/new/choose).
If you would like to propose a new dataset, metric or forward operator, please [open an issue](https://github.com/deepinv/benchmarks/issues/new/choose).
Loading