Deep Hedging with Reinforcement Learning

Learn optimal hedging strategies that beat Black-Scholes by 10-30% under real-world market frictions.

This framework trains neural network policies via Monte Carlo Policy Gradients (MCPG) to dynamically hedge European calls and American puts. Unlike classical approaches, the learned strategies adapt to transaction costs, discrete rebalancing, and asymmetric volatility -- producing lower downside risk in realistic scenarios.

            Black-Scholes Delta                    MCPG Learned Policy
          ────────────────────────             ────────────────────────
   Assumes: no costs, continuous           Learns: optimal hedge ratios
   trading, constant volatility            from GJR-GARCH simulations
                  │                                      │
                  ▼                                      ▼
           RSQP ≈ 0.45                            RSQP ≈ 0.32
        (high shortfall risk)                  (30% lower risk)

---

Why This Matters

Traditional delta hedging assumes a frictionless world. In practice, every rebalance costs money and volatility clusters unpredictably. This framework addresses both:

Challenge	Classical Approach	This Framework
Transaction costs	Ignored or Leland approximation	Learned directly from cost-penalized episodes
Volatility dynamics	Constant vol assumption	GJR-GARCH with leverage effect
Discrete rebalancing	Continuous-time theory	Trained on discrete daily steps
Risk measure	Symmetric (MSE)	Asymmetric (RSQP) -- penalizes shortfalls only

---

Quick Start

# 1. Install dependencies
pip install -r requirements.txt

# 2. Train + compare against baselines in one command
python -m src.cli compare --max-updates 200 --save-plots results/ --verbose

# 3. View results
#    -> Prints comparison table to terminal
#    -> Saves plots to results/

Expected terminal output:

MCPG: 100%|██████████| 200/200
=========================================================
Strategy                       RSQP   Mean xi    Std xi
---------------------------------------------------------
Black-Scholes Delta           0.4523    0.0312    0.5891
Leland Delta                  0.4198    0.0287    0.5443
MCPG Policy                   0.3215   -0.0041    0.4127
=========================================================

Saved comparison chart to results/comparison.png
Saved P&L distribution to results/pnl_distribution.png
Saved hedge paths to results/hedge_paths.png

Generated plots:

Strategy Comparison	P&L Distribution	Hedge Ratios
Bar chart showing RSQP per strategy with improvement %	Overlapping histograms of hedging errors	MCPG vs BS delta over time with stock price

Individual Commands

# Train European call hedging policy
python -m src.cli train-european --max-updates 500 --save-path models/euro.pt --verbose

# Train American put hedging policy
python -m src.cli train-american --max-updates 500 --save-path models/amer.pt --verbose

# Quick baseline evaluation
python -m src.cli eval --kappa 0.01

# Compare with a pre-trained policy (skip training)
python -m src.cli compare --load-policy models/euro.pt --save-plots results/

---

How It Works

flowchart LR
    subgraph Market Simulation
        A[GJR-GARCH\nStock Paths] --> B[Hedging\nEnvironment]
        C[AR1 IV\nFeatures] --> B
    end

    subgraph Training Loop
        B -->|state s_t| D[Policy Network\nπ_θ: s → δ]
        D -->|action δ_t| B
        B -->|terminal ξ_T| E[RSQP Loss]
        E -->|∇_θ| D
    end

    subgraph Evaluation
        F[Compare on\n2048 paths] --> G[MCPG vs BS\nvs Leland]
        G --> H[Plots &\nMetrics]
    end

    D -.->|trained policy| F

Loading

Training Pipeline Step by Step

1. Simulate N stock paths using GJR-GARCH (captures volatility clustering + leverage effect)
2. Roll out the neural network policy on each path (observe state, choose hedge ratio)
3. Compute the terminal hedging error: xi_T = payoff - portfolio_value
4. Minimize RSQP(xi_T) via policy gradient (penalizes shortfalls asymmetrically)
5. Validate every 10 updates, early stop on patience

The Hedging Problem

A trader sells a derivative and must hedge using the underlying asset. At each time step the agent observes market state and chooses a hedge ratio:

State s_t (European):                Action a_t:
├── t/T         (time to maturity)   └── δ_{t+1}  (hedge ratio)
├── S_t/S_0     (normalized price)       ∈ [-1.5, 1.5]
├── V_t/V_0     (normalized portfolio)
├── IV_level_t  (implied vol level)
└── IV_slope_t  (implied vol slope)

The portfolio evolves under the self-financing condition with transaction costs:

φ_{t+1} = φ_t · e^{r·Δt} + δ_t · S_t · e^{q·Δt} − κ · S_t · |δ_{t+1} − δ_t|
                                                      └─── transaction costs ───┘

Risk Measure: RSQP

The Root Semi-Quadratic Penalty focuses on downside risk:

RSQP(ξ) = √( E[ ξ² · 1{ξ > 0} ] )

where ξ_T = payoff − V_T  (hedging shortfall)

Unlike symmetric measures (MSE), RSQP only penalizes shortfalls -- situations where the hedging portfolio fails to cover the option payoff. This matches how practitioners actually think about hedging risk.

American Option Pricing

For American puts, the framework uses Dynamic Chebyshev approximation (Glau et al. 2019):

flowchart RL
    T[Maturity T\nPayoff = max K-S, 0] --> T1[t = T-1]
    T1 --> T2[t = T-2]
    T2 --> dots[...]
    dots --> T0[t = 0\nOption Price]

    style T fill:#e8e8e8
    style T0 fill:#55a868,color:white

Loading

At each time step during backward induction:

1. Simulate one-step MC from Chebyshev nodes
2. Compute continuation value via polynomial interpolation
3. Compare with immediate exercise payoff (K - S)+
4. Fit Chebyshev polynomial to max(exercise, continuation)
5. Store exercise boundary S*(t)

The hedging episode terminates when S_t <= S*(t) (early exercise) or at maturity.

---

Python API

Full comparison workflow

from src.experiments import compare_strategies, MarketSetup

# One call: train + evaluate + plot
results = compare_strategies(
    market=MarketSetup(kappa=0.01),  # higher transaction costs
    max_updates=300,
    n_eval_paths=4096,
    save_dir="results/",
)

# Access results programmatically
print(results["rsqp"])
# {'Black-Scholes Delta': 0.5234, 'Leland Delta': 0.4812, 'MCPG Policy': 0.3845}

Train and save a policy

from src.experiments import train_european, MarketSetup

market = MarketSetup(T_steps=63, S0=100, K=100, kappa=0.01)
policy, best_val, log = train_european(market=market, max_updates=300)
print(f"Best RSQP: {best_val:.4f}, Early stopped: {log.stopped_early}")

# Save and reload
policy.save("models/my_policy.pt")

from src.models.policy import PolicyNet
policy = PolicyNet.load("models/my_policy.pt")

Fit GJR-GARCH to your own data

import pandas as pd
import numpy as np
from src.market.gjr_garch import fit_gjr_garch, simulate_paths

# Fit to historical log-returns
returns = pd.Series([...])  # daily log-returns
params = fit_gjr_garch(returns)
print(f"Leverage effect (lambda): {params.lam:.4f}")
print(f"Persistence: {params.nu + params.xi + params.lam/2:.4f}")

# Simulate paths
rng = np.random.default_rng(42)
S, Y = simulate_paths(params, S0=100.0, n_steps=63, n_paths=1000, rng=rng)
# S.shape = (1000, 64), Y.shape = (1000, 63)

Standalone American put pricing

from src.pricing.chebyshev import price_american_put_chebyshev, chebyshev_value_at

result = price_american_put_chebyshev(
    S0=100, K=100, r=0.02, q=0.0, T=0.25, N=63,
    n_nodes=17, n_mc=2000,
    sim_one_step=my_simulator,  # function(s_array, dt, n_mc) -> s_next_array
)
price = chebyshev_value_at(result, t_index=0, S=np.array([100.0]), s_lo=..., s_hi=...)

Custom visualization

from src.viz import plot_training_curve, plot_pnl_distribution, plot_strategy_comparison

# Plot training progress
plot_training_curve(
    log.train_losses, log.val_losses,
    title="My Training Run",
    save_path="training.png",
)

# Compare P&L distributions
plot_pnl_distribution(
    {"BS Delta": xi_bs, "My Policy": xi_policy},
    save_path="pnl.png",
)

# Bar chart comparison
plot_strategy_comparison(
    {"BS Delta": 0.45, "Leland": 0.42, "MCPG": 0.32},
    save_path="comparison.png",
)

---

Project Structure

src/
├── cli.py                   # Unified CLI: train, eval, compare
├── experiments.py           # High-level workflows & strategy comparison
├── viz.py                   # Plotting: training curves, P&L, comparisons
├── algos/
│   └── mcpg.py              # MCPG training loop with early stopping
├── baselines/
│   └── delta.py             # Black-Scholes & Leland delta strategies
├── envs/
│   ├── european_env.py      # European call hedging environment
│   └── american_env.py      # American put hedging environment
├── eval/
│   └── metrics.py           # RSQP risk measure & hedging error
├── market/
│   ├── gjr_garch.py         # GJR-GARCH(1,1) price simulation
│   └── iv_features.py       # AR(1) implied volatility features
├── models/
│   └── policy.py            # PolicyNet with save/load support
└── pricing/
    └── chebyshev.py         # American put pricing via Chebyshev polynomials

---

CLI Reference

compare (recommended)

Train a policy and compare it against BS and Leland baselines on the same paths. Generates a comparison table and optional plots.

python -m src.cli compare \
    --max-updates 300 \
    --kappa 0.01 \
    --n-eval-paths 4096 \
    --save-plots results/ \
    --verbose

Option	Default	Description
--load-policy	None	Skip training, load a pre-trained .pt file
--n-eval-paths	2048	Number of paths for out-of-sample evaluation
--save-plots	None	Directory for comparison plots

train-european

python -m src.cli train-european \
    --max-updates 500 \
    --batch-episodes 1024 \
    --kappa 0.01 \
    --save-path models/euro.pt \
    --verbose

train-american

python -m src.cli train-american \
    --max-updates 500 \
    --kappa 0.005 \
    --cheb-nodes 17 \
    --save-path models/amer.pt \
    --verbose

eval

Quick baseline sanity check (no training).

python -m src.cli eval --kappa 0.005

Common Options

Option	Default	Description
--steps	63	Hedging steps (trading days, ~3 months)
--s0	100	Initial spot price
--k	100	Strike price
--kappa	0.005	Proportional transaction cost
--batch-episodes	512	Episodes per gradient update
--max-updates	200	Maximum training updates
--lr	1e-4	Learning rate
--patience	20	Early stopping patience
--save-path	None	Save trained policy to .pt file
--verbose	off	Enable per-update log output

---

Market Dynamics

GJR-GARCH(1,1)

Stock prices follow a GJR-GARCH process that captures volatility clustering and the leverage effect (bad news increases volatility more than good news):

Y_t = mu + eps_t                                    (log-return)
eps_t = sigma_t * z_t,  z_t ~ N(0,1)                (shock)
sigma_t^2 = nu_0 + (nu + lambda * 1{eps_{t-1}<0}) * eps_{t-1}^2 + xi * sigma_{t-1}^2
                         └── leverage effect ──┘

Default calibration: nu_0=1e-5, nu=0.05, lambda=0.1, xi=0.9 (persistence ~0.975).

Implied Volatility Features

IV level and slope are modeled as AR(1) processes providing forward-looking signals:

IV_level:  phi=0.98, mu=0.20  (persistent, mean-reverting to 20% vol)
IV_slope:  phi=0.95, mu=0.00  (smile skew signal)

---

Expected Results

Strategy	RSQP (kappa=0.005)	RSQP (kappa=0.01)	Improvement vs BS
Black-Scholes Delta	~0.45	~0.52	--
Leland Adjusted	~0.42	~0.48	~7%
MCPG Policy	~0.32	~0.38	~27-30%

The learned policy typically converges within 100-200 updates. The advantage over classical approaches grows as transaction costs increase, because the policy learns to trade off hedge accuracy against rebalancing costs.

---

Tests

python -m pytest test_*.py -v

Test	What it validates
test_chebyshev.py	American put pricing accuracy vs binomial tree
test_delta.py	Black-Scholes delta monotonicity and bounds
test_rsqp.py	RSQP metric correctness (asymmetric penalty)
test_self_financing.py	Self-financing portfolio equation with costs

---

References

• Buehler, H., Gonon, L., Teichmann, J., & Wood, B. (2019). Deep hedging. Quantitative Finance, 19(8), 1271-1291.
• Glosten, L. R., Jagannathan, R., & Runkle, D. E. (1993). On the relation between the expected value and the volatility of the nominal excess return on stocks. The Journal of Finance, 48(5).
• Glau, K., Mahlstedt, M., & Potz, C. (2019). A new approach for American option pricing: The Dynamic Chebyshev method. SIAM Journal on Scientific Computing, 41(1).
• Leland, H. E. (1985). Option pricing and replication with transactions costs. The Journal of Finance, 40(5).