Skip to content

fix(args): reject --kl-coef for GRPO-style estimators - #2359

Open
YeonwooSung wants to merge 1 commit into
THUDM:mainfrom
YeonwooSung:fix/397-reject-grpo-kl-coef
Open

fix(args): reject --kl-coef for GRPO-style estimators#2359
YeonwooSung wants to merge 1 commit into
THUDM:mainfrom
YeonwooSung:fix/397-reject-grpo-kl-coef

Conversation

@YeonwooSung

Copy link
Copy Markdown

Summary

--kl-coef is reward-KL shaping. GRPO-style estimators (grpo, gspo, cispo) do not apply KL to rewards, so a non-zero --kl-coef is a silent no-op today.

This follows the maintainer decision on #399: we should not use KL in the GRPO reward, and should use kl_loss instead. #1247, which applied KL to GRPO rewards, was closed unmerged. This PR does not change get_grpo_returns() or invent reward-KL math.

slime_validate_args now raises a clear ValueError when --kl-coef != 0 is combined with --advantage-estimator {grpo,gspo,cispo}, and points users at --use-kl-loss / --kl-loss-coef.

Test plan

  • python -m pytest tests/test_megatron_argument_validation.py -q --tb=short
    • grpo / gspo / cispo + --kl-coef 0.1 raise ValueError matching kl-coef / kl-loss
    • grpo + --kl-coef 0 still passes
    • existing validate-args tests still pass

Fixes #397

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

if we use GRPO and args.kl_coef is non-zero, is the KL computation incorrect?

1 participant