Add Ideogram support and improve BF16 dequantization handling - #459
Open
molbal wants to merge 16 commits into
Open
Add Ideogram support and improve BF16 dequantization handling#459molbal wants to merge 16 commits into
molbal wants to merge 16 commits into
Conversation
|
Great! I successfully ran it, but my device doesn't support bf16; it gets converted to fp32 computation, which makes it very slow. Can you make it run on my device in fp16? [INFO] got prompt |
Author
|
Hi @yu234567 - try now. It should work better now, can you verify please? |
Thank you so much, it worked! |
|
classifies Qwen3-VL-8B-Instruct q4_0 quant as _k quant and errors out ? is this expected or what quantization are you running for the te |
- Updated IMG_ARCH_LIST to include 'krea2'. - Introduced ModelKrea2 class with architecture details and tensor handling. - Enhanced convert_file function to support quantization types. - Added tools/convert_krea2_gguf.py for batch conversion of Krea-2 models to multiple GGUF quant levels.
…ation support limitations
…tensors and skip 0-dim scalars during GGUF conversion
Author
Fixed that was my bad |
Feature/molbal dynamic gguf
Add DynamicVRAM-aware GGUF loading and node variants. loader.py: attach mmap-backed file slices, preserve custom GGUF quant configs, and switch to dynamic handling (including BF16/quant paths). nodes.py: add DynamicVRAM loaders and GGUFModelPatcherDynamic, plus helpers to load Unet/CLIP via DynamicVRAM. quant_ops.py: implement GGML quantized tensor layout for Comfy's QuantizedTensor system. tools/convert.py: preserve safetensors metadata, introduce keys_noquant, and avoid quantizing specified small tensors. Add documentation pages for the new Dynamic VRAM loader nodes.
Add Dynamic VRAM support for GGUF loaders
## `lcpp.patch` changes (llama.cpp)
All additions follow the existing pattern used for the other image-model architectures (Flux, SD3, Aura, LTXV, HyVid, Wan, HiDream, Cosmos, Lumina2) --five/six standard insertion points plus two architecture-specific refinements (one a required correctness fix, one an optional quality tuning knob).
### Standard architecture registration (5 places in `src/llama.cpp`)
1. **`enum llm_arch`**: add `LLM_ARCH_KREA2,`
2. **`LLM_ARCH_NAMES` map**: add `{ LLM_ARCH_KREA2, "krea2" },`
3. **`LLM_TENSOR_NAMES` map**: add `{ LLM_ARCH_KREA2, {}},`
4. **`llm_load_hparams`**, "disable LLM metadata for image models" switch: add
`case LLM_ARCH_KREA2:` alongside the other image archs (skips LLM-style
hparam parsing, since this isn't a language model).
5. **`llama_model_quantize_internal`**, "rules for image models" section: add
a `LLM_ARCH_KREA2` block excluding the small/non-repeating tensors from
quantization (kept at their original F32/F16 precision):
```cpp
if (model.arch == LLM_ARCH_KREA2) {
image_model = true;
quantize &= name.find("first.") == std::string::npos;
quantize &= name.find("last.") == std::string::npos;
quantize &= name.find("tproj.") == std::string::npos;
quantize &= name.find("tmlp.") == std::string::npos;
quantize &= name.find("txtmlp.") == std::string::npos;
quantize &= name.find("txtfusion.projector.") == std::string::npos;
}
```
### The one architecture-specific fix: `txtfusion.projector.weight`
**Symptom:** ComfyUI failed to load the *quantized* GGUF (worked fine for the
BF16 intermediate file) with:
```
File ".../comfy/model_detection.py", line 917, in detect_unet_config
dit_config["txtlayers"] = state_dict['{}txtfusion.projector.weight'.format(key_prefix)].shape[1]
IndexError: tuple index out of range
```
**Root cause:** `txtfusion.projector.weight` has shape `(1, 12)` in the original checkpoint. When `llama-quantize` loads this into an actual `ggml_tensor` and writes it back out via `gguf_add_tensor()`, the tensor's dimensionality is derived from `ggml_n_dims()`, which scans the `ne[]` array from the highest index downward and drops trailing 1s. Since GGML's `ne` order is the reverse of the original torch shape, `(1, 12)` becomes `ne = [12, 1, 1, 1]` -- and `ne[1] == 1` gets trimmed away, collapsing the
tensor to 1D. The BF16 GGUF written directly by `convert.py`'s Python `gguf` library does *not* have this problem (it just serializes the given shape faithfully); the collapse is specific to the C++ `llama-quantize` step.
This is the same class of bug already handled for other architectures (SD3's `pos_embed`, AuraFlow's `positional_encoding`/`register_tokens`, Wan's `.modulation`/`img_emb.emb_pos`), all of which use a helper added to GGML for exactly this purpose: `gguf_set_tensor_ndim()`.
**Fix**, added to the same tensor-writing loop in `llama_model_quantize_internal` where the other archs' fixes live (right after the `LLM_ARCH_WAN` block):
```cpp
// Krea2's txtfusion.projector.weight has shape (1, 12) -- the leading
// dim of 1 becomes a trailing `ne` entry and gets truncated by
// ggml_n_dims() unless corrected explicitly.
if (model.arch == LLM_ARCH_KREA2) {
const std::string name = ggml_get_name(tensor);
if (name == "txtfusion.projector.weight" && tensor->ne[1] == 1) {
const int n_dim = 2;
gguf_set_tensor_ndim(ctx_outs[i_split], "txtfusion.projector.weight", n_dim);
LLAMA_LOG_INFO("\n%s: Correcting txtfusion.projector.weight shape for Krea2: [key:%s]\n", __func__, tensor->name);
}
}
```
Note the difference from the SD3/Aura/Wan precedents: those check `tensor->ne[2] == 1` and restore `n_dim = 3` because their tensors were originally 3D (e.g. `(1, H, W)`). Krea2's `txtfusion.projector.weight` is only 2D (`(1, 12)`), so the check is `tensor->ne[1] == 1` and the restored dimension is `n_dim = 2`.
**No other Krea2 tensors needed this fix.** All other tensors are either purely 1D, or 2D with no leading-1 dimension (verified against actual shapes pulled from the checkpoint).
### Optional refinement: `to_v` attention quant-type rules
llama.cpp's `img_tensor_get_type()` has a block of special-case rules that give the attention value projection ("to_v") preferential K-quant subtype treatment at certain `ftype`s, already applied to the other image archs (Flux, SD3, etc. -- matched via names like `.to_v.weight`, `.attn.w1v.weight`, `.attn.w2v.weight`). Krea2's equivalent tensor, `.attn.wv.weight`, wasn't originally in that match list. Added:
```cpp
if ( // Rules for to_v attention
(name.find("attn_v.weight") != std::string::npos) ||
(name.find(".to_v.weight") != std::string::npos) ||
(name.find(".v.weight") != std::string::npos) ||
(name.find(".attn.w1v.weight") != std::string::npos) ||
(name.find(".attn.w2v.weight") != std::string::npos) ||
+ (name.find(".attn.wv.weight") != std::string::npos) ||
(name.find("_attn.v_proj.weight") != std::string::npos)
){
```
Unlike the `txtfusion.projector.weight` fix above, this isn't a correctness issue -- without it, Krea2 quantizes and runs fine, `.attn.wv.weight` just gets the generic type-selection path instead of the to_v-specific one. It's a size/quality tuning knob, initially left out to keep the first working version minimal, then added and verified once the base support was confirmed stable.
Updated read_tensors.py to support tensor comparison and improved shape checking.
Enhance tensor reading and comparison functionality
Update tensor handling for Krea2 architecture
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This adds support for Ideogram GGUF models.
What Changed
ideogramto the supported image GGUF architectures.Notes
Tested on Windows 11, Python version: 3.12.11 (main, Jul 23 2025, 00:32:20) [MSC v.1944 64 bit (AMD64)] [INFO] Total VRAM 8192 MB, total RAM 48394 MB
[INFO] pytorch version: 2.12.0+cu130
[INFO] Set vram state to: LOW_VRAM
[INFO] Device: cuda:0 NVIDIA GeForce RTX 3080 Laptop GPU
Tested with Q4_0 gguf from https://huggingface.co/leejet/ideogram-4-GGUF
Other GGUF quant types still use the existing dequant paths.