Skip to content

Fix two unreachable keys in VRAM_MANAGEMENT_MODULE_MAPS - #1666

Open
rakhimovv wants to merge 1 commit into
modelscope:mainfrom
rakhimovv:fix-vram-map-unreachable-keys
Open

Fix two unreachable keys in VRAM_MANAGEMENT_MODULE_MAPS#1666
rakhimovv wants to merge 1 commit into
modelscope:mainfrom
rakhimovv:fix-vram-map-unreachable-keys

Conversation

@rakhimovv

Copy link
Copy Markdown

Two entries in VRAM_MANAGEMENT_MODULE_MAPS are keyed on names that no model_configs.py entry registers, so they are never selected.

The lookup in ModelPool.fetch_module_map is by config["model_class"], and a key that doesn't match one falls through to wrapping the whole model as a single AutoWrappedModule.

1. joyai_image_dit.Transformer3DModelJoyAIImageDiT

vram_management_module_maps.py:342 names a class that doesn't exist in that module. The class is JoyAIImageDiT, which is what model_configs.py:1117 registers. The entry's contents already match that class (RMSNorm, ModulateWan, nn.Linear, nn.Conv3d, nn.LayerNorm), and the entry directly below it is keyed on the registered name joyai_image_text_encoder.JoyAIImageTextEncoder.

2. qwen_image_controlnet.BlockWiseControlBlockQwenImageBlockWiseControlNet

vram_management_module_maps.py:83 names the inner block rather than the registered model. model_configs.py lines 25 and 31 both register QwenImageBlockWiseControlNet. The entry's contents stay correct under the new key: BlockWiseControlBlock uses RMSNorm imported from general_modules, and qwen_image_dit re-exports that same class, so the existing qwen_image_dit.RMSNorm mapping still matches those submodules (checked by reading the imports, and confirmed with isinstance in a REPL).

Checking

I couldn't find a test hook for this, so I checked by enumerating the map at 4dbf980: of its 86 keys, exactly these 2 are absent from the set of registered model_class values, and 1 of the 2 additionally names a class that cannot be imported. With this patch both counts are 0, and the number of registered model classes with no per-layer entry goes from 20 to 18.

I ran the 8 Qwen-Image-Blockwise-ControlNet-* scripts (4 variants across model_inference and model_inference_low_vram) before and after; they produce output in both cases, which is what I'd expect given the fallback. I don't have a JoyAI-Image-Edit setup, so I haven't measured that half.

One thing I could not settle, and would value your read on: AutoWrappedModule.forward is what performs the onload, but the blockwise controlnet is invoked through process_controlnet_conditioning and blockwise_forward rather than forward, and __getattr__ delegates those to the wrapped module. I couldn't tell whether the whole-model fallback is fully equivalent for a model called that way, or whether it works only because load_models_to_device onloads in_iteration_models beforehand. If it's the latter, this patch matters more than the granularity difference. Happy to adjust if either key was deliberate.

The map is looked up by the model_class string registered in model_configs.py,
so an entry keyed on any other name is never selected and the model silently
falls back to whole-model wrapping.

- joyai_image_dit.Transformer3DModel: no such class exists; the registered
  class is JoyAIImageDiT.
- qwen_image_controlnet.BlockWiseControlBlock: that class is the inner block;
  the registered class is QwenImageBlockWiseControlNet.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant