Deterministic-equivalent evaluation can overstate policy quality — add a stage-wise rollout evaluation and a target-violation metric

[DJLee1208]

Summary

Evaluating a trained TS-DDR policy by solving the full-horizon deterministic
equivalent with the policy's targets can silently overstate its quality. The coupled
solve re-optimizes all stages jointly and compensates for targets that are not
followable stage by stage — which is exactly what deployment cannot do. A
stage-wise subproblem rollout evaluation (the package already has the machinery
via the stage-wise / simulate_multistage path) should be the headline metric,
together with a target-violation measure.

Symptom

Same checkpoint (Bolivia ACP example, 96 stages, LSTM 2×128, constant ×1 auto-base
slack penalty), 40 fixed evaluation scenarios, objective excluding the target-slack
penalty term:

evaluation	cost	target-violation share
deterministic equivalent (full horizon)	306,556	0.046
stage-wise rollout (deployment semantics)	324,437 (+5.8%)	0.408

The deterministic-equivalent view reports a policy that looks near-optimal and
target-consistent; executed stage by stage, the same policy is +5.8% more expensive
and violates its own targets ~41% of the time.

Why it matters

Stage-wise execution is the deployment semantics of a target-trajectory policy.
Reporting only deterministic-equivalent numbers can make results look better than the
policy would operate, and hides target-followability failures entirely.

Proposed fix

1. Provide/document an evaluation helper based on the existing stage-wise rollout
   path, and recommend it as the default reported metric in the examples.
2. Report two numbers per evaluation: (a) operational objective excluding the
   target-slack penalty term, and (b) a target-violation share (e.g., the realized
   slack-penalty value normalized by the objective), with the recommendation that
   policy comparisons are only trusted when the violation share is small (≤ ~0.05).
3. Note the interaction with the penalty-annealing issue (#XX): with an annealed
   schedule the two evaluations agree at convergence to ~1e-7 relative — the rollout
   metric is the guard that detects when they don't.

[andrewrosemberg]

Good idea!

Notes on the proposed fix:

1.

To keep the code clean, I would not change the main arguments of the training function (

DecisionRules.jl/src/simulate_multistage.jl

Lines 396 to 404 in a91f1b2

	function train_multistage(model, initial_state, det_equivalent::JuMP.Model,
	state_params_in, state_params_out, uncertainty_sampler;
	num_batches=100, num_train_per_batch=32, optimizer=Flux.Adam(0.01),
	adjust_hyperparameters=(iter, opt_state, num_train_per_batch) -> num_train_per_batch,
	record_loss=(iter, model, loss, tag) -> begin println("tag: $tag, Iter: $iter, Loss: $loss")
	return false
	end,
	get_objective_no_target_deficit=get_objective_no_target_deficit
	)

and

DecisionRules.jl/src/simulate_multistage.jl

Lines 342 to 350 in a91f1b2

	function train_multistage(model, initial_state, subproblems::Vector{JuMP.Model},
	state_params_in, state_params_out, uncertainty_sampler;
	num_batches=100, num_train_per_batch=32, optimizer=Flux.Adam(0.01),
	adjust_hyperparameters=(iter, opt_state, num_train_per_batch) -> num_train_per_batch,
	record_loss=(iter, model, loss, tag) -> begin println("tag: $tag, Iter: $iter, Loss: $loss")
	return false
	end,
	get_objective_no_target_deficit=get_objective_no_target_deficit
	)

)

Instead, I would just change the api around the recording function (

DecisionRules.jl/src/simulate_multistage.jl

Lines 346 to 348 in a91f1b2

	record_loss=(iter, model, loss, tag) -> begin println("tag: $tag, Iter: $iter, Loss: $loss")
	return false
	end,

). Currently, the recording function is called twice with hardcoded values (

DecisionRules.jl/src/simulate_multistage.jl

Lines 370 to 371 in a91f1b2

	record_loss(iter, model, eval_loss, "metrics/loss") && break
	record_loss(iter, model, objective, "metrics/training_loss") && break

).

How about if:

1. we replace the get_objective_no_target_deficit (

   DecisionRules.jl/src/simulate_multistage.jl

   Line 364 in a91f1b2

   eval_loss += get_objective_no_target_deficit(subproblems)

   ) with a sample logger that keeps a cache state (similar to what we use for checkpoints

   DecisionRules.jl/src/utils.jl

   Lines 132 to 141 in a91f1b2

   	function (callback::SaveBest)(iter, model, loss)
   	if loss < callback.best_loss
   	m = model |> cpu
   	@info "best model change" callback.best_loss loss
   	callback.best_loss = loss
   	model_state = normalize_recur_state(Flux.state(m))
   	jldsave(callback.model_path; model_state=model_state)
   	end
   	return false
   	end

   ). This allows for detailed sample-wise logging. Api like: sample_log(s, det_equivalent or subproblems). This is NOT meant to be something we use to log into wandb or disk for evert sample (this would be too costly), just local cache accumulation to later be passed to the batch recording. But having this allows sample-wise debugging: the user can pass a function that can log to disk details of a specific problem in case it gets an error or used to debug.

2. Change the api of the recording function to be called once per batch with the sample cache we stored per sample as an argument beside the usual iter, model alongside it: record(sample_log, iter, model). This allows for regular wandb and disk recording and the user can evaluate the model inside it with a different set of subproblems just like you proposed - having a block inside like:

   DecisionRules.jl/examples/HydroPowerModels/train_dr_hydropowermodels_subproblems.jl

   Lines 106 to 113 in a91f1b2

   	Random.seed!(8788)
   	objective_values = [simulate_multistage(
   	subproblems, state_params_in, state_params_out,
   	initial_state, DecisionRules.sample(uncertainty_samples),
   	models;
   	# ensure_feasibility=(x_out, x_in, uncertainty) -> ensure_feasibility(x_out, x_in, uncertainty, max_volume)
   	) for _ in 1:2]
   	best_obj = mean(objective_values)

   .

2. No comments :) Good idea.