Skip to content

Support true CFG in Flux ControlNet img2img and inpainting pipelines - #14670

Open
Devam0311 wants to merge 2 commits into
huggingface:mainfrom
Devam0311:flux-controlnet-true-cfg
Open

Support true CFG in Flux ControlNet img2img and inpainting pipelines#14670
Devam0311 wants to merge 2 commits into
huggingface:mainfrom
Devam0311:flux-controlnet-true-cfg

Conversation

@Devam0311

@Devam0311 Devam0311 commented Sep 1, 2026

Copy link
Copy Markdown

What does this PR do?

FluxControlNetPipeline supports true CFG via true_cfg_scale + negative_prompt, but FluxControlNetImg2ImgPipeline and FluxControlNetInpaintPipeline do not.

This blocks de-distilled Flux checkpoints (e.g. nyanko7/flux-dev-de-distill) on the img2img and inpainting ControlNet paths — de-distilled models need real conditional/unconditional guidance rather than the distilled guidance embedding. Inpainting is the case originally reported in #9635.

Solution

Ports the existing true-CFG implementation from FluxControlNetPipeline to both pipelines, following its conventions:

  • negative_prompt, negative_prompt_2, true_cfg_scale, negative_prompt_embeds and negative_pooled_prompt_embeds added to __call__
  • negative-prompt validation added to check_inputs, matching FluxImg2ImgPipeline
  • do_true_cfg = true_cfg_scale > 1 and has_neg_prompt, so behaviour is unchanged unless a negative prompt (or precomputed negative embeddings) is supplied
  • the unconditional pass in the denoising loop reuses the ControlNet residuals from the conditional pass, as FluxControlNetPipeline does

Note: negative_prompt / negative_prompt_2 / true_cfg_scale are placed after prompt_2 to match every other Flux pipeline, which shifts positional argument order in __call__. Happy to append them at the end instead if you'd prefer to preserve positional compatibility.

Testing

Three regression tests per pipeline in tests/pipelines/controlnet_flux/: test_*_true_cfg, test_*_true_cfg_requires_negative_prompt (asserts the no-negative-prompt path stays a no-op), and test_*_true_cfg_with_negative_embeds.

pytest tests/pipelines/controlnet_flux/ -q
# 68 passed, 42 skipped

ruff check <changed files>                       # All checks passed
ruff format --check <changed files>              # 6 files already formatted
python utils/check_forward_call_docstrings.py    # All forward/__call__ arguments are documented

utils/check_copies.py, check_dummies.py, check_repo.py and check_inits.py fail identically on a clean checkout of main in my environment (schedulers / bitsandbytes / missing modules), so they appear unrelated to this change.

Fixes #9635

Before submitting

  • Did you use an AI agent (Claude Code, Codex, Cursor, etc.) to help with this PR? If so:
    • Did you read the Coding with AI agents guide?
    • Did you run the self-review skill on the diff?
    • Did you share the final self-review notes in the PR description or a comment?
  • Did you read the contributor guideline?
  • Was this discussed/approved via a GitHub issue? — [Flux ControlNet] Add support for de-distilled models with CFG #9635
  • Did you make sure to update the documentation with your changes? — flux.md uses [[autodoc]], so the new arguments are picked up from the updated docstrings.
  • Did you write any new necessary tests?

Self-review notes

Reviewed against .ai/references/review-rules.md, pipelines.md and testing.md.

Blocking issues: none.

Non-blocking / for the reviewer:

  1. Positional-argument shift in __call__ (noted above). Consistency with the other Flux pipelines vs. positional back-compat — your call, easy to change.
  2. ControlNet residuals reused for the unconditional pass. Matches FluxControlNetPipeline and saves one ControlNet forward per step, but it does mean the control signal is not recomputed under the negative prompt.
  3. negative_prompt_embeds deliberately not added to _callback_tensor_inputs — no Flux pipeline does this, so leaving it unchanged keeps them consistent.

Intentionally not done: no slow/integration test against a real de-distilled checkpoint. Per testing.md, integration tests come later after discussion. Verification here was on CPU with tiny dummy components only — I don't have the hardware to run flux-dev-de-distill end to end, so I can't post generated-image comparisons.

Who can review?

@yiyixuxu @asomoza

FluxControlNetPipeline supports true classifier-free guidance via
`true_cfg_scale` and `negative_prompt`, but FluxControlNetImg2ImgPipeline
and FluxControlNetInpaintPipeline do not. This blocks de-distilled Flux
checkpoints on those paths, since they need real conditional/unconditional
guidance rather than the distilled `guidance` embedding.

Port the existing implementation from FluxControlNetPipeline, following its
conventions:

- add `negative_prompt`, `negative_prompt_2`, `true_cfg_scale`,
  `negative_prompt_embeds` and `negative_pooled_prompt_embeds` to `__call__`
- add negative-prompt validation to `check_inputs`
- gate on `do_true_cfg = true_cfg_scale > 1 and has_neg_prompt`, so behaviour
  is unchanged unless a negative prompt is supplied
- reuse the ControlNet residuals from the conditional pass for the
  unconditional pass, as FluxControlNetPipeline does

Add three regression tests per pipeline covering the enabled path, the
gating when no negative prompt is given, and the precomputed
negative-embeddings path.

Fixes huggingface#9635

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBYeq5vB4DNZDEaqUsroKR
@Devam0311

Copy link
Copy Markdown
Author

Hi! I implemented the functionality described in #9635 and added regression tests. When you have a chance, I'd appreciate any feedback on the approach.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Flux ControlNet] Add support for de-distilled models with CFG

1 participant