Fix two unreachable keys in VRAM_MANAGEMENT_MODULE_MAPS - #1666
Open
rakhimovv wants to merge 1 commit into
Open
Conversation
The map is looked up by the model_class string registered in model_configs.py, so an entry keyed on any other name is never selected and the model silently falls back to whole-model wrapping. - joyai_image_dit.Transformer3DModel: no such class exists; the registered class is JoyAIImageDiT. - qwen_image_controlnet.BlockWiseControlBlock: that class is the inner block; the registered class is QwenImageBlockWiseControlNet.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two entries in
VRAM_MANAGEMENT_MODULE_MAPSare keyed on names that nomodel_configs.pyentry registers, so they are never selected.The lookup in
ModelPool.fetch_module_mapis byconfig["model_class"], and a key that doesn't match one falls through to wrapping the whole model as a singleAutoWrappedModule.1.
joyai_image_dit.Transformer3DModel→JoyAIImageDiTvram_management_module_maps.py:342names a class that doesn't exist in that module. The class isJoyAIImageDiT, which is whatmodel_configs.py:1117registers. The entry's contents already match that class (RMSNorm,ModulateWan,nn.Linear,nn.Conv3d,nn.LayerNorm), and the entry directly below it is keyed on the registered namejoyai_image_text_encoder.JoyAIImageTextEncoder.2.
qwen_image_controlnet.BlockWiseControlBlock→QwenImageBlockWiseControlNetvram_management_module_maps.py:83names the inner block rather than the registered model.model_configs.pylines 25 and 31 both registerQwenImageBlockWiseControlNet. The entry's contents stay correct under the new key:BlockWiseControlBlockusesRMSNormimported fromgeneral_modules, andqwen_image_ditre-exports that same class, so the existingqwen_image_dit.RMSNormmapping still matches those submodules (checked by reading the imports, and confirmed withisinstancein a REPL).Checking
I couldn't find a test hook for this, so I checked by enumerating the map at
4dbf980: of its 86 keys, exactly these 2 are absent from the set of registeredmodel_classvalues, and 1 of the 2 additionally names a class that cannot be imported. With this patch both counts are 0, and the number of registered model classes with no per-layer entry goes from 20 to 18.I ran the 8
Qwen-Image-Blockwise-ControlNet-*scripts (4 variants acrossmodel_inferenceandmodel_inference_low_vram) before and after; they produce output in both cases, which is what I'd expect given the fallback. I don't have a JoyAI-Image-Edit setup, so I haven't measured that half.One thing I could not settle, and would value your read on:
AutoWrappedModule.forwardis what performs the onload, but the blockwise controlnet is invoked throughprocess_controlnet_conditioningandblockwise_forwardrather thanforward, and__getattr__delegates those to the wrapped module. I couldn't tell whether the whole-model fallback is fully equivalent for a model called that way, or whether it works only becauseload_models_to_deviceonloadsin_iteration_modelsbeforehand. If it's the latter, this patch matters more than the granularity difference. Happy to adjust if either key was deliberate.