Online Conformal Prediction with Untrusted Shift Forecasts

Reproducibility code for the paper "When != Where: Online Conformal Prediction with Untrusted Shift Forecasts." All experiments are CPU-only and run in minutes.

Summary

In the change-point setting, any online conformal predictor whose threshold depends only on past nonconformity scores pays a cumulative coverage-gap (training-conditional regret) of order sqrt((N_cp + 1) T) over a horizon T with N_cp change points. This code studies whether external, untrusted forecasts of the shifts help, and demonstrates a sharp dichotomy:

• forecasts of when shifts occur do not improve the order (the sqrt(.) frontier survives even with exact change times);
• forecasts of where the post-shift quantile lies break it (near-O(N_cp) regret when accurate);

and that the benefit of value forecasts scales with the sharpness of the drift.

Layout

src/
  exp1_dichotomy.py            Synthetic: WHEN vs WHERE; debt vs segment length L.
  exp2_validation_crossover.py Synthetic: + validation cost, CUSUM detector, large L,
                               the fixed-accuracy crossover.
  exp3_real_elec2.py           Real data (ELEC2, mild drift): regime characterization.
  exp4_real_vix.py             Real data (VIX, sharp drift): value-dominance test.
  exp5_selection.py            Check that the unknown-accuracy selector matches the
                               oracle blend up to a constant factor.
  make_figures.py              Regenerate the two paper figures.
data/                          Place ELEC2 and VIX CSVs here (see data/README.md).

Requirements

pip install -r requirements.txt   # numpy, matplotlib

Python 3.9+.

Running

Synthetic experiments need no data:

python src/exp1_dichotomy.py
python src/exp2_validation_crossover.py
python src/exp5_selection.py

Real-data experiments need the CSVs in data/ (see data/README.md):

python src/exp3_real_elec2.py
python src/exp4_real_vix.py

Figures (writes to figures/):

python src/make_figures.py

Metric and honesty notes

On synthetic streams the true distribution is known, so the exact cumulative coverage debt sum_t |P(s_t > q_t) - alpha| is reported. On real data the conditional distribution is unknown, so a realized-coverage proxy is used (post-shift |coverage - alpha| and a windowed global deviation), "shifts" are defined by an offline detector, and scores are model residuals. These caveats are stated in each script and in the paper.

Citation

Kunal Tomar and Vineeta Rathore. When != Where: Online Conformal Prediction with Untrusted Shift Forecasts. (under review).

License

MIT. See LICENSE.