Support true CFG in Flux ControlNet img2img and inpainting pipelines - #14670
Open
Devam0311 wants to merge 2 commits into
Open
Support true CFG in Flux ControlNet img2img and inpainting pipelines#14670Devam0311 wants to merge 2 commits into
Devam0311 wants to merge 2 commits into
Conversation
FluxControlNetPipeline supports true classifier-free guidance via `true_cfg_scale` and `negative_prompt`, but FluxControlNetImg2ImgPipeline and FluxControlNetInpaintPipeline do not. This blocks de-distilled Flux checkpoints on those paths, since they need real conditional/unconditional guidance rather than the distilled `guidance` embedding. Port the existing implementation from FluxControlNetPipeline, following its conventions: - add `negative_prompt`, `negative_prompt_2`, `true_cfg_scale`, `negative_prompt_embeds` and `negative_pooled_prompt_embeds` to `__call__` - add negative-prompt validation to `check_inputs` - gate on `do_true_cfg = true_cfg_scale > 1 and has_neg_prompt`, so behaviour is unchanged unless a negative prompt is supplied - reuse the ControlNet residuals from the conditional pass for the unconditional pass, as FluxControlNetPipeline does Add three regression tests per pipeline covering the enabled path, the gating when no negative prompt is given, and the precomputed negative-embeddings path. Fixes huggingface#9635 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBYeq5vB4DNZDEaqUsroKR
Author
|
Hi! I implemented the functionality described in #9635 and added regression tests. When you have a chance, I'd appreciate any feedback on the approach. |
76 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
FluxControlNetPipelinesupports true CFG viatrue_cfg_scale+negative_prompt, butFluxControlNetImg2ImgPipelineandFluxControlNetInpaintPipelinedo not.This blocks de-distilled Flux checkpoints (e.g.
nyanko7/flux-dev-de-distill) on the img2img and inpainting ControlNet paths — de-distilled models need real conditional/unconditional guidance rather than the distilledguidanceembedding. Inpainting is the case originally reported in #9635.Solution
Ports the existing true-CFG implementation from
FluxControlNetPipelineto both pipelines, following its conventions:negative_prompt,negative_prompt_2,true_cfg_scale,negative_prompt_embedsandnegative_pooled_prompt_embedsadded to__call__check_inputs, matchingFluxImg2ImgPipelinedo_true_cfg = true_cfg_scale > 1 and has_neg_prompt, so behaviour is unchanged unless a negative prompt (or precomputed negative embeddings) is suppliedFluxControlNetPipelinedoesNote:
negative_prompt/negative_prompt_2/true_cfg_scaleare placed afterprompt_2to match every other Flux pipeline, which shifts positional argument order in__call__. Happy to append them at the end instead if you'd prefer to preserve positional compatibility.Testing
Three regression tests per pipeline in
tests/pipelines/controlnet_flux/:test_*_true_cfg,test_*_true_cfg_requires_negative_prompt(asserts the no-negative-prompt path stays a no-op), andtest_*_true_cfg_with_negative_embeds.utils/check_copies.py,check_dummies.py,check_repo.pyandcheck_inits.pyfail identically on a clean checkout ofmainin my environment (schedulers / bitsandbytes / missing modules), so they appear unrelated to this change.Fixes #9635
Before submitting
self-reviewskill on the diff?flux.mduses[[autodoc]], so the new arguments are picked up from the updated docstrings.Self-review notes
Reviewed against
.ai/references/review-rules.md,pipelines.mdandtesting.md.Blocking issues: none.
Non-blocking / for the reviewer:
__call__(noted above). Consistency with the other Flux pipelines vs. positional back-compat — your call, easy to change.FluxControlNetPipelineand saves one ControlNet forward per step, but it does mean the control signal is not recomputed under the negative prompt.negative_prompt_embedsdeliberately not added to_callback_tensor_inputs— no Flux pipeline does this, so leaving it unchanged keeps them consistent.Intentionally not done: no slow/integration test against a real de-distilled checkpoint. Per
testing.md, integration tests come later after discussion. Verification here was on CPU with tiny dummy components only — I don't have the hardware to runflux-dev-de-distillend to end, so I can't post generated-image comparisons.Who can review?
@yiyixuxu @asomoza