refactor(diffusion): extract loss_hub for Flow-GRPO and custom loss dispatch#63
Open
niehen6174 wants to merge 7 commits into
Open
refactor(diffusion): extract loss_hub for Flow-GRPO and custom loss dispatch#63niehen6174 wants to merge 7 commits into
niehen6174 wants to merge 7 commits into
Conversation
Extract shared diffusion loss plumbing (context, GRPO advantage helper, flow_grpo_ppo_loss, get_diffusion_loss_function) so actor stays thin and custom objectives can plug in via --custom-loss-function-path.
Move default per-prompt GRPO reward normalization into loss_hub so rollout and training share one implementation; custom post-process still overrides.
…ction-path Expose the diffusion default objective explicitly and wire the custom loss hook that loss_hub dispatch expects, matching miles LLM customization style.
Replace inline Flow-GRPO forward/PPO logic with loss_fn dispatch and keep upstream recompute-old-log-prob support by forwarding write flags to loss_hub.
Cover default flow_grpo_ppo selection, policy_loss alias, custom loss path loading, and grpo_normalize_rewards mean/std behavior.
Use flow_grpo as the primary CLI name to match the Flow-GRPO recipe; keep flow_grpo_ppo and policy_loss as aliases for backward compatibility.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Move Flow-GRPO PPO-clip and GRPO reward normalization out of the actor/rollout into swappable building blocks — same default behavior, ready for NFT/SFT as flag recipes.
actor.py; reward normalization was duplicated inline inrollout.py. There was no working hook to replace the train objective.loss_hub/module +--loss-type flow_grpo/--custom-loss-function-path. Actor only schedules micro-batches and callsloss_fn(ctx, batch).Changes
Roadmap
--custom-reward-post-process-path,--custom-convert-samples-to-train-data-path,--custom-loss-function-path--rollout-function-path, convert, loss,--disable-compute-advantages-and-returnssft_rollout+sft_loss