Add examples and gower distance; fix OT preds

docs/Project.toml

@@ -1,2 +1,3 @@
 [deps]
 Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
+OptimalTransportDataIntegration = "f00d1d04-589e-4742-a825-5ec24d3c488e"

docs/make.jl

@@ -2,12 +2,12 @@ using Documenter
 using OptimalTransportDataIntegration
 
 makedocs(
-    sitename = "OptimalTransportDataIntegration",
-    authors = "Jeremy Omer, Valérie Garès and Pierre Navaro",
-    format = Documenter.HTML(),
+    sitename = "OptimalTransportDataIntegration.jl",
+    authors = "Jeremy Omer <jeremy.omer@insa-rennes.fr>, Valérie Garès <valerie.gares@inria.fr> and Pierre Navaro <navaro@math.cnrs.fr>",
+    format = Documenter.HTML(; canonical = "https://otrecoding.github.io/OptimalTransportDataIntegration.jl"),
     modules = [OptimalTransportDataIntegration],
     pages = [
-        "Quickstart" => "index.md",
+        "Home" => "index.md",
         "Transport within a data source" => "joint_ot_within_base.md",
         "Transport between data sources" => "joint_ot_between_bases.md",
         "Machine Learning" => "learning.md",

docs/src/index.md

@@ -1,9 +1,197 @@
-# OptimalTransportDataIntegration.jl
+# Optimal Transport for Statistical Matching
 
 ```@meta
 CurrentModule = OptimalTransportDataIntegration
 ```
 
+**Authors**: Valérie Garès, Jeremy Omer and Pierre Navaro.
+
+**Affiliations**: IRMAR (Université de Rennes, INRIA, CNRS, INSA Rennes)
+
+**Objective**: Integrate distinct datasets on the same population, sharing some variables but containing unique ones, to create a unified synthetic dataset.
+
+**Proposed Methods**:
+- Two approaches based on **Optimal Transport (OT)** theory:
+  1. [`JointOTWithinBase`](@ref): Optimal transport of joint distributions of explanatory variables and outcomes **within** each data source.
+  2. [`JointOTBetweenBases`](@ref): Optimal transport of joint distributions **between** data sources, with simultaneous estimation of predictive functions.
+
+## 🔍 Introduction
+
+### Context
+
+- **Data Integration**: Combine heterogeneous data to improve the reliability of statistical analyses.
+- **Statistical Matching (SM)**: Merge datasets with **distinct units** but shared/common variables.
+- **Challenge**: Estimate relationships between variables not jointly observed (e.g., `Y` and `Z` conditional on `X`).
+
+### Formalization
+
+| Dataset       | covariables        | outcomes             |                      |
+|---------------|--------------------|----------------------|----------------------|
+| Dataset ``A`` | ``X^A`` (observed) | ``Y^A`` (observed)   | ``Z^A`` (unobserved) |
+| Dataset ``B`` | ``X^B`` (observed) | ``Y^B`` (unobserved) | ``Z^B`` (observed)   |
+
+**Goal**: Build a synthetic dataset where ``Y`` and ``Z`` are predicted for all units.
+
+---
+
+## 🛠️ Methods
+
+### 1. JointOTWithinBase (JDOT-wi): Optimal Transport **Within** Datasets
+
+- **Assumption**: Conditional distributions ``Y|X`` and ``Z|X`` are identical between ``A`` and ``B``.
+
+- **Algorithm**:
+   1. Estimate joint distributions ``\mu^{(X^A,Y^A)}`` and ``\mu^{(X^A,Z^A)}`` using empirical estimators.
+   2. Solve an optimal transport problem to approximate ``\mu^{(X^A,Y^A,Z^A)}``.
+   3. Predict ``Z^A`` and ``Y^B`` using optimized functions ``f`` and ``g``.
+
+- **Advantages**:
+
+  - Suitable for categorical variables.
+  - Robust if the conditional distribution assumption holds.
+
+- **Limitations**:
+
+  - Requires discretization for continuous variables.
+  - Sensitive to violations of the conditional distribution assumption.
+
+---
+
+### 2. JointOTBetweenBases (JDOT-be): Optimal Transport **Between** Datasets
+
+- **Approach**: Transport the joint distribution ``(X, Y, Z)`` between ``A`` and ``B`` by simultaneously optimizing:
+  - A transport plan ``\gamma``.
+  - Predictive functions ``f`` (for ``Z^A``) and ``g`` (for ``Y^B``).
+
+- **Cost Function**:
+
+```math
+
+c\big((x,y,z),(x',y',z')\big) = d(x,x') + \alpha_1 \mathcal{L}_Y(y,y') + \alpha_2 \mathcal{L}_Z(z,z')
+
+```
+
+**Optimization**: Alternate between:
+1. Updating the transport plan ``\gamma``.
+2. Optimizing functions ``f`` and ``g`` (e.g., neural networks).
+
+**Advantages**:
+- Handles imbalanced data via unbalanced optimal transport.
+- Suitable for mixed data types (continuous/categorical).
+- No strong CIA (Conditional Independence Assumption) required.
+
+**Limitations**:
+- High computational cost.
+- Sensitive to hyperparameters (``\alpha_1, \alpha_2, \rho``).
+
+---
+
+## First example
+
+### Installation
+
+Launch the Julia REPL and enter the Pkg REPL by pressing `]`. To get back to the Julia REPL, press the backspace.
+
+Upon entering the Pkg REPL, you should see the following prompt:
+
+```julia-repl
+(@v1.12) pkg>
+```
+
+To install the package from GitHub, use `add`:
+
+```julia-repl
+(@v1.12) pkg> add https://github.com/otrecoding/OptimalTransportDataIntegration.jl.git
+     Cloning git-repo `https://github.com/otrecoding/OptimalTransportDataIntegration.jl.git`
+    Updating git-repo `https://github.com/otrecoding/OptimalTransportDataIntegration.jl.git`
+   Resolving package versions...
+   Installed PythonOT ─ v0.1.6
+    Updating `~/tmp/Project.toml`
+```
+
+After the package is installed, you need install POT the Python Optimal Transport library
+
+```julia-repl
+julia> using Conda
+ │ Package Conda not found, but a package named Conda is available from a registry.
+ │ Install package?
+ │   (tmp) pkg> add Conda
+ └ (y/n/o) [y]:
+   Resolving package versions...
+    Updating `~/tmp/Project.toml`
+  [8f4d0f93] + Conda v1.10.3
+    Manifest No packages added to or removed from `~/tmp/Manifest.toml`
+
+julia> Conda.add("pot")
+[ Info: Running `conda install -y pot` in root environment
+Retrieving notices: done
+Channels:
+ - conda-forge
+Platform: osx-64
+Collecting package metadata (repodata.json): \
+```
+
+Now the package can be loaded into the Julia session:
+
+```julia-repl
+julia> using OptimalTransportDataIntegration
+
+julia>
+```
+
+### Parameters
+
+Create some defaults paramaters with [`DataParameters`](@ref).
+
+
+```julia-repl
+julia> params = DataParameters()
+```
+
+### Generators
+
+You can generate continuous covariables ``X`` with [`ContinuousDataGenerator`](@ref) or
+discrete covariables ``X`` with [`DiscreteDataGenerator`](@ref).
+
+- Two scenarios:
+  - **`scenario = 1`**: ``\mathbb{P}_{Z^A|X^A=x} = \mathbb{P}_{Z^B|X^B=x}`` (identical conditional distributions).
+  - **`scenario = 2`**: ``\mathbb{P}_{Z^A|X^A=x} = \mathbb{P}_{Z^B|X^B=T(x)}`` (transformation ``T`` between domains).
+
+```julia
+
+rng = ContinuousDataGenerator(params)
+data = generate(rng)
+
+result = otrecod(data, JointOTWithinBase())
+println(" within-r : $(accuracy(result)) ")
+
+result = otrecod(data, JointOTBetweenBases())
+println(" between-r : $(accuracy(result))")
+
+result = otrecod(data, SimpleLearning())
+println(" sl : $(accuracy(result))")
+```
+
+The simulator provided by the package generates a dataframe containing
+the covariates ``X`` from databases ``A`` and ``B`` and the outcomes ``Y`` from
+database ``A`` and ``Z`` from database ``B``. The [`otrecod`](@ref) function can be used
+to obtain a recoded version of variable ``Y`` for database ``B`` and ``Z`` for
+database ``A``. The [`accuracy`](@ref) function can be used to verify the quality
+of the method. It gives you the error on ``Y``, ``Z`` and both outcomes.
+
+Methods available as `otrecod` second argument are:
+
+- [`JointOTWithinBase`](@ref)
+- [`SimpleLearning`](@ref)
+- [`JointOTBetweenBases`](@ref)
+- [`JointOTBetweenBasesDiscreteOrdered`](@ref)
+- [`JointOTBetweenBasesWithPredictors`](@ref)
+- [`JointOTBetweenBasesJDOT`](@ref)
+- [`JointOTDABetweenBasesCovariables`](@ref)
+- [`JointOTDABetweenBasesOutcomes`](@ref)
+- [`JointOTDABetweenBasesOutcomesWithPredictors`](@ref)
+- [`JointOTBetweenBasesWithoutOutcomes`](@ref)
+
 ## Simulation data generation
 
 ```@autodocs

docs/src/joint_ot_between_bases.md

@@ -1,12 +1,149 @@
-# Optimal transport of the joint distribution between data sources
+# Optimal Transport Between Datasets for Statistical Matching
+
+---
+
+## 📌 Overview
+
+[`JointOTBetweenBases`](@ref) (JDOT-be) is an advanced method for
+**statistical matching** that leverages **optimal transport theory**
+to align joint distributions of explanatory variables and outcomes
+**between** datasets. Unlike JDOT-wi, JDOT-be simultaneously optimizes
+a **transport plan** and **predictive functions** to handle missing
+values in a unified synthetic dataset.
+
+---
+
+## 🔍 Context and Motivation
+
+### Problem Statement
+In statistical matching, datasets ``A`` and ``B`` often contain distinct units but share some variables:
+
+| Dataset  | observed | observed | unobserved |
+|---------------|---------------------|---------------------|-----------------------|
+| Dataset ``A`` | ``X^A`` | ``Y^A``  | ``Z^A`` |
+| Dataset ``B`` | ``X^B`` | ``Z^B``  | ``Y^B`` |
+
+**Objective**: Predict ``Z^A`` and ``Y^B`` to create a synthetic dataset where all variables are jointly available.
+
+### Key Idea
+
+JDOT-be **transports the joint distribution** ``(X, Y, Z)`` **between**
+datasets ``A`` and ``B`` while optimizing predictive functions ``f``
+and ``g`` to impute missing values. This approach is particularly
+useful when datasets have **different distributions** or **covariate
+shifts**.
+
+---
+
+## 🛠️ Methodology
+
+### Step 1: Define the Optimal Transport Problem
+
+JDOT-be seeks a coupling ``\gamma`` that minimizes the transportation cost between the joint distributions ``\mu^{(X^A, Y^A, Z^A)}`` and ``\mu^{(X^B, Y^B, Z^B)}``:
+
+```math
+\gamma^\star \in \arg\min_{\gamma \in \Gamma(\mu^{(X^A, Y^A, Z^A)}, \mu^{(X^B, Y^B, Z^B)})} \int c\left((x, y, z), (x', y', z')\right) d\gamma_{(x, y, z), (x', y', z')},
+```
+
+where the **cost function** ``c`` combines:
+- A distance ``d(x, x')`` between explanatory variables ``X^A`` and ``X^B``.
+- Loss functions ``\mathcal{L}_Y(y, y')`` and ``\mathcal{L}_Z(z, z')`` for outcomes ``Y`` and ``Z``:
+```math
+  c\left((x, y, z), (x', y', z')\right) = d(x, x') + \alpha_1 \mathcal{L}_Y(y, y') + \alpha_2 \mathcal{L}_Z(z, z').
+```
+
+``\alpha_1`` and ``\alpha_2`` are hyperparameters balancing the alignment of explanatory variables and outcomes.
+
+---
+
+### Step 2: Handle Unobserved Variables
+
+Since ``Z^A`` and ``Y^B`` are unobserved, JDOT-be replaces them with predictions:
+- ``Z^A`` is predicted using ``f(X^A, Y^A)``.
+- ``Y^B`` is predicted using ``g(X^B, Z^B)``.
+
+The empirical version of the problem becomes:
+
+```math
+\min_{f, g, \gamma} \sum_{i \in A, j \in B} c\left((X_i, Y_i, f(X_i, Y_i)), (X_j, g(X_j, Z_j), Z_j)\right) \gamma_{i,j},
+```
+
+subject to:
+
+```math
+\sum_{i \in A} \gamma_{i,j} = \frac{1}{n}, \quad \forall j \in B,
+```
+
+```math
+\sum_{j \in B} \gamma_{i,j} = \frac{1}{n}, \quad \forall i \in A,
+```
+
+```math
+\gamma_{i,j} \geq 0, \quad \forall i \in A, j \in B.
+```
+
+---
+
+### Step 3: Solve the Optimization Problem
+
+JDOT-be uses a **Block Coordinate Descent (BCD)** algorithm to alternate between:
+1. **Updating the transport plan** ``\gamma`` for fixed ``f`` and ``g``.
+2. **Optimizing the predictive functions** ``f`` and ``g`` for fixed ``\gamma``.
+
+#### Predictive Functions
+
+- ``f: \mathcal{X} \times \mathcal{Y} \to \mathcal{Z}`` predicts ``Z^A`` in dataset ``A``.
+- ``g: \mathcal{X} \times \mathcal{Z} \to \mathcal{Y}`` predicts ``Y^B`` in dataset ``B``.
+
+These functions are typically implemented using **neural networks** or other machine learning models.
+
+---
+
+### Step 4: Unbalanced Optimal Transport (Optional)
+For **imbalanced datasets**, JDOT-be can use **unbalanced optimal transport** to relax marginal constraints. The unbalanced OT problem is defined as:
+
+```math
+\gamma^\star \in \arg\min_{\gamma \geq 0} \sum c\left((x, y, z), (x', y', z')\right) \gamma_{(x, y, z), (x', y', z')} + \rho_1 \text{KL}(\gamma^A \| \mu^{(X^A, Y^A, Z^A)}) + \rho_2 \text{KL}(\gamma^B \| \mu^{(X^B, Y^B, Z^B)}),
+```
+
+where ``\text{KL}`` is the **Kullback-Leibler divergence**, and ``\rho_1, \rho_2`` are regularization parameters.
+
+---
+
+## ✅ Advantages
+
+- **Handles Covariate Shifts**: Effective when datasets have different distributions.
+- **No CIA Assumption**: Unlike traditional methods, JDOT-be does not assume conditional independence between ``Y`` and ``Z`` given ``X``.
+- **Flexibility**: Works with **mixed data types** (continuous and categorical variables).
+- **Theoretical Guarantees**: Provides bounds on prediction error under specific assumptions.
+
+---
+
+## ⚠️ Limitations
+
+- **Computational Cost**: Higher than JDOT-wi due to the joint optimization of ``\gamma``, ``f``, and ``g``.
+- **Hyperparameter Sensitivity**: Performance depends on the choice of ``\alpha_1``, ``\alpha_2``, and ``\rho``.
+- **Complexity**: Requires careful tuning for optimal results.
+
+## 📂 Implementation
+
+### Algorithmic Steps
+
+1. **Input**: Datasets ``A`` and ``B`` with observed variables ``X, Y`` (for ``A``) and ``X, Z`` (for ``B``).
+2. **Initialize**: Set initial predictive functions ``f_0`` and ``g_0``.
+3. **Alternate Optimization**:
+   - Update the transport plan ``\gamma`` for fixed ``f`` and ``g``.
+   - Optimize ``f`` and ``g`` for fixed ``\gamma``.
+4. **Output**: Predicted values ``\hat{Z}^A = f(X^A, Y^A)`` and ``\hat{Y}^B = g(X^B, Z^B)``.
+
 
 ```@autodocs
 Modules = [OptimalTransportDataIntegration]
 Pages = [
 "joint_ot_between_bases_jdot.jl",
 "joint_ot_between_bases_with_predictors.jl",
 "joint_ot_between_bases_without_outcomes.jl",
-"joint_ot_between_bases_category.jl",
+"joint_ot_between_bases_discrete_ordered.jl",
 "joint_ot_between_bases_da_covariables.jl",
 "joint_ot_between_bases_da_outcomes.jl",
 "joint_ot_between_bases_da_outcomes_with_predictors.jl",