unfuse_lora(components=[...]) desyncs _merged_adapters/num_fused_loras from actual per-component PEFT merge state

[ErenAta16]

Describe the bug

LoraBaseMixin._merged_adapters (the set backing num_fused_loras/fused_loras) is a single set shared across the whole pipeline, but fuse_lora()/unfuse_lora() both accept a components argument that operates on a subset of _lora_loadable_modules (e.g. unet, text_encoder) independently. When the same adapter is fused into multiple components (the default, since fuse_lora() with no adapter_names fuses into every requested component) and later only some of those components are unfused via unfuse_lora(components=[...]), _merged_adapters drops the adapter name entirely, even though it's still physically merged into the base weights of the untouched component(s).

num_fused_loras/fused_loras then report the pipeline as having nothing (or less) fused, while the untouched component's weights still silently contain the baked-in LoRA delta. If I intend to submit a PR for this, and have one ready.

Reproduction

import torch.nn as nn
from peft import LoraConfig
from diffusers.loaders.lora_base import LoraBaseMixin
from diffusers.loaders.peft import PeftAdapterMixin
from diffusers.models.modeling_utils import ModelMixin
from diffusers.configuration_utils import ConfigMixin


class TinyModel(ModelMixin, ConfigMixin, PeftAdapterMixin):
    config_name = "config.json"

    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(8, 8)


class FakePipeline(LoraBaseMixin):
    _lora_loadable_modules = ["unet", "text_encoder"]

    def __init__(self, unet, text_encoder):
        self._merged_adapters = set()
        self.unet, self.text_encoder = unet, text_encoder


unet, text_encoder = TinyModel(), TinyModel()
lora_config = LoraConfig(r=4, lora_alpha=4, target_modules=["linear"], init_lora_weights=False)
unet.add_adapter(lora_config, adapter_name="my_adapter")
text_encoder.add_adapter(lora_config, adapter_name="my_adapter")
pipe = FakePipeline(unet, text_encoder)

pipe.fuse_lora(components=["unet", "text_encoder"], adapter_names=["my_adapter"])
print("after fuse (both):", pipe.num_fused_loras, pipe.fused_loras)

pipe.unfuse_lora(components=["text_encoder"])  # only unfuse text_encoder
print("after unfuse (text_encoder only):", pipe.num_fused_loras, pipe.fused_loras)

from peft.tuners.tuners_utils import BaseTunerLayer
unet_still_merged = any(isinstance(m, BaseTunerLayer) and len(m.merged_adapters) > 0 for m in unet.modules())
print("unet is ACTUALLY still merged at the PEFT level:", unet_still_merged)

Output:

after fuse (both): 1 {'my_adapter'}
after unfuse (text_encoder only): 0 set()
unet is ACTUALLY still merged at the PEFT level: True

num_fused_loras/fused_loras report 0/set() even though unet still has my_adapter merged into its base weights.

This is a real, documented, publicly-supported code path, every concrete pipeline mixin (e.g. StableDiffusionLoraLoaderMixin.unfuse_lora) exposes the components parameter, and the docstring for unfuse_lora documents it as "list of LoRA-injectable components to unfuse LoRA from," implying a subset is a normal, intended use.

I also checked the existing test suite (tests/lora/utils.py) and none of the fuse_lora/unfuse_lora tests exercise a partial-components combination, they always pass every loadable component at once (components=self.pipeline_class._lora_loadable_modules), which is why this gap wasn't caught.

Logs

No response, this is a silent correctness/bookkeeping bug, not a crash.

System Info

Reproduced against the current main branch source (src/diffusers/loaders/lora_base.py), peft 0.19.1, torch 2.x, Python 3.12.10, Windows 11. No GPU needed to reproduce, the bug is in pure Python set bookkeeping.

Who can help?

@sayakpaul (LoRA/PEFT integration maintainer, per the issue template's guidance for the "LoRA" area)