feat:add Minimax h3 support ver1

docs/minimax-h3.md

@@ -0,0 +1,211 @@
+# MiniMax-H3
+
+edge-dit.cpp supports MiniMax-H3 video + audio generation through the standalone
+component-loading path. The implementation follows the MiniMax-H3 interface used
+by `stable-diffusion.cpp`: one diffusion checkpoint, Qwen3-VL text/vision encoder,
+video VAE, and optional audio VAE.
+
+## Model files
+
+MiniMax-H3 runs with four model components. The diffusion checkpoint changes by
+mode; the encoder and VAEs are shared.
+
+| Component | File name | Source |
+|---|---|---|
+| Qwen3-VL text/vision encoder | `qwen3vl_32b_minimax_h3-Q4_K_M.gguf` | [leejet/MiniMax-H3-GGUF](https://huggingface.co/leejet/MiniMax-H3-GGUF) |
+| Video VAE | `minimax_h3_video_vae_fp16.safetensors` | [Comfy-Org/MiniMax-H3](https://huggingface.co/Comfy-Org/MiniMax-H3) |
+| Audio VAE | `minimax_h3_audio_vae_fp32.safetensors` | [Comfy-Org/MiniMax-H3](https://huggingface.co/Comfy-Org/MiniMax-H3) |
+
+### FL2VA checkpoint
+
+| Component | File name | Source |
+|---|---|---|
+| Diffusion model | `minimax_h3_fl2va-Q4_K_M.gguf` | [leejet/MiniMax-H3-GGUF](https://huggingface.co/leejet/MiniMax-H3-GGUF) |
+
+Supported modes:
+
+| Mode | Inputs | Output |
+|---|---|---|
+| T2VA | Text prompt | Video + audio |
+| I2VA | Text prompt + first frame image | Video + audio |
+| FL2VA | Text prompt + first frame image + last frame image | Video + audio |
+
+### Ref2VA checkpoint
+
+| Component | File name | Source |
+|---|---|---|
+| Diffusion model | `minimax_h3_ref2va_pruned-Q4_K_M.gguf` | [leejet/MiniMax-H3-GGUF](https://huggingface.co/leejet/MiniMax-H3-GGUF) |
+
+Supported reference inputs:
+
+| Input | CLI flag | Notes |
+|---|---|---|
+| Reference image | `--ref-image <image>` | Repeatable. Presented as `<Picture N>` to Qwen3-VL. |
+| Reference video | `--ref-video <frame-dir>` | Repeatable. Directory of image frames sorted lexicographically; treated as 24 fps. |
+| Paired video audio | `--ref-video-audio <wav>` | Repeatable. The Nth WAV is paired with the Nth `--ref-video`. |
+| Standalone audio | `--ref-audio <wav>` | Repeatable. Independent audio reference, not attached to a video. |
+
+The reference inputs can be combined freely within Ref2VA. Ref2VA cannot be used
+with `--image` or `--end-img` in the same request.
+
+MiniMax-H3 control frames are not supported for either checkpoint.
+
+## Common parameters
+
+MiniMax-H3 is a 24 fps audio-video model. The usual full-resolution setting is:
+
+```bash
+-W 864 -H 480 --fps 24 --video-frames 56 --cfg-scale 1.0 --diffusion-fa --rng cpu
+```
+
+The frame count must satisfy `17k + 5` (for example `5`, `22`, `39`, `56`). The
+examples below use 56 frames and 20 steps.
+
+Use `--video-format mp4` for H.264/AAC MP4 output when `ffmpeg` is available.
+Without `--audio-vae`, the model can still generate video, but no generated audio
+is decoded or muxed.
+
+## FL2VA usage
+
+### Text to video + audio
+
+```bash
+ed-cli --video \
+  --diffusion-model minimax_h3_fl2va-Q4_K_M.gguf \
+  --vae minimax_h3_video_vae_fp16.safetensors \
+  --audio-vae minimax_h3_audio_vae_fp32.safetensors \
+  --llm qwen3vl_32b_minimax_h3-Q4_K_M.gguf \
+  -p "A cat surfs on an ocean wave with upbeat surf-rock music." \
+  --cfg-scale 1 -W 864 -H 480 --fps 24 --video-frames 56 --steps 20 \
+  --diffusion-fa --rng cpu --video-format mp4 \
+  -o minimax_h3_t2va.mp4
+```
+
+### First-frame image to video + audio
+
+```bash
+ed-cli --video \
+  --diffusion-model minimax_h3_fl2va-Q4_K_M.gguf \
+  --vae minimax_h3_video_vae_fp16.safetensors \
+  --audio-vae minimax_h3_audio_vae_fp32.safetensors \
+  --llm qwen3vl_32b_minimax_h3-Q4_K_M.gguf \
+  --image first_frame.png \
+  -p "Animate the provided first frame into a cinematic shot with matching sound." \
+  --cfg-scale 1 -W 864 -H 480 --fps 24 --video-frames 56 --steps 20 \
+  --diffusion-fa --rng cpu --video-format mp4 \
+  -o minimax_h3_i2va.mp4
+```
+
+### First + last frame to video + audio
+
+```bash
+ed-cli --video \
+  --diffusion-model minimax_h3_fl2va-Q4_K_M.gguf \
+  --vae minimax_h3_video_vae_fp16.safetensors \
+  --audio-vae minimax_h3_audio_vae_fp32.safetensors \
+  --llm qwen3vl_32b_minimax_h3-Q4_K_M.gguf \
+  --image first_frame.png --end-img last_frame.png \
+  -p "Create a smooth transition from the first frame to the last frame with natural audio." \
+  --cfg-scale 1 -W 864 -H 480 --fps 24 --video-frames 56 --steps 20 \
+  --diffusion-fa --rng cpu --video-format mp4 \
+  -o minimax_h3_fl2va.mp4
+```
+
+## Ref2VA usage
+
+When using references, write the prompt so it explicitly mentions the reference
+slots, such as `<Picture 1>`, `<Video 1>`, and `<Audio 1>`. This makes reference
+adherence easier to evaluate and avoids text-only prompts dominating the output.
+
+### Reference image
+
+```bash
+ed-cli --video \
+  --diffusion-model minimax_h3_ref2va_pruned-Q4_K_M.gguf \
+  --vae minimax_h3_video_vae_fp16.safetensors \
+  --audio-vae minimax_h3_audio_vae_fp32.safetensors \
+  --llm qwen3vl_32b_minimax_h3-Q4_K_M.gguf \
+  --ref-image reference.png \
+  -p "Use the landscape and lighting from <Picture 1> to create a cinematic video." \
+  --cfg-scale 1 -W 864 -H 480 --fps 24 --video-frames 56 --steps 20 \
+  --diffusion-fa --rng cpu --video-format mp4 \
+  -o minimax_h3_ref_image.mp4
+```
+
+### Reference video
+
+```bash
+ed-cli --video \
+  --diffusion-model minimax_h3_ref2va_pruned-Q4_K_M.gguf \
+  --vae minimax_h3_video_vae_fp16.safetensors \
+  --audio-vae minimax_h3_audio_vae_fp32.safetensors \
+  --llm qwen3vl_32b_minimax_h3-Q4_K_M.gguf \
+  --ref-video reference_frames \
+  -p "Follow the camera motion and scene layout from <Video 1>." \
+  --cfg-scale 1 -W 864 -H 480 --fps 24 --video-frames 56 --steps 20 \
+  --diffusion-fa --rng cpu --video-format mp4 \
+  -o minimax_h3_ref_video.mp4
+```
+
+### Reference video with paired audio
+
+```bash
+ed-cli --video \
+  --diffusion-model minimax_h3_ref2va_pruned-Q4_K_M.gguf \
+  --vae minimax_h3_video_vae_fp16.safetensors \
+  --audio-vae minimax_h3_audio_vae_fp32.safetensors \
+  --llm qwen3vl_32b_minimax_h3-Q4_K_M.gguf \
+  --ref-video reference_frames \
+  --ref-video-audio reference_soundtrack.wav \
+  -p "Follow the motion from <Video 1> and use its paired soundtrack as audio guidance." \
+  --cfg-scale 1 -W 864 -H 480 --fps 24 --video-frames 56 --steps 20 \
+  --diffusion-fa --rng cpu --video-format mp4 \
+  -o minimax_h3_ref_video_audio.mp4
+```
+
+### Standalone reference audio
+
+```bash
+ed-cli --video \
+  --diffusion-model minimax_h3_ref2va_pruned-Q4_K_M.gguf \
+  --vae minimax_h3_video_vae_fp16.safetensors \
+  --audio-vae minimax_h3_audio_vae_fp32.safetensors \
+  --llm qwen3vl_32b_minimax_h3-Q4_K_M.gguf \
+  --ref-audio reference_audio.wav \
+  -p "Generate a video with audio inspired by <Audio 1>." \
+  --cfg-scale 1 -W 864 -H 480 --fps 24 --video-frames 56 --steps 20 \
+  --diffusion-fa --rng cpu --video-format mp4 \
+  -o minimax_h3_ref_audio.mp4
+```
+
+### Mixed references
+
+```bash
+ed-cli --video \
+  --diffusion-model minimax_h3_ref2va_pruned-Q4_K_M.gguf \
+  --vae minimax_h3_video_vae_fp16.safetensors \
+  --audio-vae minimax_h3_audio_vae_fp32.safetensors \
+  --llm qwen3vl_32b_minimax_h3-Q4_K_M.gguf \
+  --ref-image reference.png \
+  --ref-video reference_frames \
+  --ref-video-audio reference_soundtrack.wav \
+  --ref-audio extra_audio.wav \
+  -p "Use the style from <Picture 1>, motion from <Video 1>, the paired video soundtrack, and additional ambience from <Audio 2>." \
+  --cfg-scale 1 -W 864 -H 480 --fps 24 --video-frames 56 --steps 20 \
+  --diffusion-fa --rng cpu --video-format mp4 \
+  -o minimax_h3_ref_mixed.mp4
+```
+
+In a mixed prompt, paired video audio is presented before the video as `<Audio 1>`.
+Standalone `--ref-audio` entries are numbered after video-paired audio, so the
+first standalone audio in the example is `<Audio 2>`.
+
+## Current limitations
+
+- Ref2VA behavior is functional but still under quality alignment with
+  `stable-diffusion.cpp`; matching seeds do not yet imply matching generated
+  videos.
+- Video reference conditioning can be strong. Use explicit prompts and visually
+  inspect whether the output is following the intended reference semantics.
+- Audio reference effects are easiest to judge with listening tests; contact
+  sheets only validate the video stream.

docs/models.md

@@ -135,6 +135,10 @@ Command example: [Qwen-Image-Edit CLI](cli.md#qwen-image-edit).
 
 Wan video generation uses `--video`, `--frames`, and `--fps`.
 
+MiniMax-H3 video + audio generation supports text, first-frame, first+last-frame,
+and Ref2VA conditioning through standalone component loading. See
+[MiniMax-H3 usage](minimax-h3.md).
+
 Supported output formats are `auto`, `avi`, `mp4`, `mov`, `mkv`, and `webm`.
 The CLI uses `ED_FFMPEG` when set and can also find imageio-ffmpeg binaries in
 an active Python environment.
@@ -204,4 +208,4 @@ profiling behavior.
 - [performance (RTX 4090)](performance-4090.md)
 - [performance (H200)](performance-H200.md)
 - [API and bindings](api.md)
-- [Development and contributing](development.md)
+- [Development and contributing](development.md)

examples/cli/cli_common.hpp

@@ -454,12 +454,20 @@ struct FluxCliArgs {
     const char* model_path = nullptr;
     const char* diffusion_model_path = nullptr;
     const char* vae_path = nullptr;
+    const char* audio_vae_path = nullptr;
     const char* clip_l_path = nullptr;
     const char* clip_g_path = nullptr;
     const char* t5xxl_path = nullptr;
+    const char* llm_path = nullptr;
+    const char* llm_vision_path = nullptr;
     const char* prompt = nullptr;
     const char* negative_prompt = nullptr;
     const char* image_path = nullptr;
+    const char* end_image_path = nullptr;
+    std::vector<std::string> ref_image_paths;
+    std::vector<std::string> ref_video_paths;
+    std::vector<std::string> ref_video_audio_paths;
+    std::vector<std::string> ref_audio_paths;
     const char* output_path = "output.png";
     const char* video_format = nullptr;
     const char* backend = nullptr;
@@ -550,6 +558,12 @@ inline bool parse_args(int argc, char** argv, FluxCliArgs* args) {
 
         if (std::strcmp(key, "--video") == 0) {
             args->video = true;
+        } else if (std::strcmp(key, "-M") == 0 || std::strcmp(key, "--mode") == 0) {
+            const char* v = require_value(key);
+            if (!v) return false;
+            if (std::strcmp(v, "vid_gen") == 0) {
+                args->video = true;
+            }
         } else if (std::strcmp(key, "--video-format") == 0) {
             args->video_format = require_value(key);
         } else if (std::strcmp(key, "--model") == 0 || std::strcmp(key, "--model_path") == 0) {
@@ -558,12 +572,18 @@ inline bool parse_args(int argc, char** argv, FluxCliArgs* args) {
             args->diffusion_model_path = require_value(key);
         } else if (std::strcmp(key, "--vae") == 0) {
             args->vae_path = require_value(key);
+        } else if (std::strcmp(key, "--audio-vae") == 0 || std::strcmp(key, "--audio_vae") == 0) {
+            args->audio_vae_path = require_value(key);
         } else if (std::strcmp(key, "--clip_l") == 0) {
             args->clip_l_path = require_value(key);
         } else if (std::strcmp(key, "--clip_g") == 0) {
             args->clip_g_path = require_value(key);
         } else if (std::strcmp(key, "--t5xxl") == 0) {
             args->t5xxl_path = require_value(key);
+        } else if (std::strcmp(key, "--llm") == 0) {
+            args->llm_path = require_value(key);
+        } else if (std::strcmp(key, "--llm_vision") == 0 || std::strcmp(key, "--llm-vision") == 0) {
+            args->llm_vision_path = require_value(key);
         } else if (std::strcmp(key, "--prompt") == 0 || std::strcmp(key, "-p") == 0) {
             args->prompt = require_value(key);
         } else if (std::strcmp(key, "--prompt_file") == 0 || std::strcmp(key, "--prompt-file") == 0) {
@@ -574,9 +594,30 @@ inline bool parse_args(int argc, char** argv, FluxCliArgs* args) {
                    std::strcmp(key, "--negative_prompt") == 0) {
             args->negative_prompt = require_value(key);
             if (!args->negative_prompt) return false;
-        } else if (std::strcmp(key, "--image") == 0 || std::strcmp(key, "-i") == 0) {
+        } else if (std::strcmp(key, "--image") == 0 || std::strcmp(key, "--init-img") == 0 ||
+                   std::strcmp(key, "--init_img") == 0 || std::strcmp(key, "-i") == 0) {
             args->image_path = require_value(key);
             if (!args->image_path) return false;
+        } else if (std::strcmp(key, "--end-img") == 0 || std::strcmp(key, "--end_img") == 0) {
+            args->end_image_path = require_value(key);
+            if (!args->end_image_path) return false;
+        } else if (std::strcmp(key, "--ref-image") == 0 || std::strcmp(key, "--ref_image") == 0 ||
+                   std::strcmp(key, "-r") == 0) {
+            const char* path = require_value(key);
+            if (!path) return false;
+            args->ref_image_paths.emplace_back(path);
+        } else if (std::strcmp(key, "--ref-video") == 0 || std::strcmp(key, "--ref_video") == 0) {
+            const char* path = require_value(key);
+            if (!path) return false;
+            args->ref_video_paths.emplace_back(path);
+        } else if (std::strcmp(key, "--ref-video-audio") == 0 || std::strcmp(key, "--ref_video_audio") == 0) {
+            const char* path = require_value(key);
+            if (!path) return false;
+            args->ref_video_audio_paths.emplace_back(path);
+        } else if (std::strcmp(key, "--ref-audio") == 0 || std::strcmp(key, "--ref_audio") == 0) {
+            const char* path = require_value(key);
+            if (!path) return false;
+            args->ref_audio_paths.emplace_back(path);
         } else if (std::strcmp(key, "--output") == 0 || std::strcmp(key, "-o") == 0) {
             args->output_path = require_value(key);
         } else if (std::strcmp(key, "--width") == 0 || std::strcmp(key, "-W") == 0) {
@@ -587,7 +628,8 @@ inline bool parse_args(int argc, char** argv, FluxCliArgs* args) {
             const char* v = require_value(key);
             if (!v) return false;
             args->height = parse_int_value(v, args->height);
-        } else if (std::strcmp(key, "--frames") == 0) {
+        } else if (std::strcmp(key, "--frames") == 0 || std::strcmp(key, "--video-frames") == 0 ||
+                   std::strcmp(key, "--video_frames") == 0) {
             const char* v = require_value(key);
             if (!v) return false;
             args->frames = parse_int_value(v, args->frames);
@@ -777,11 +819,18 @@ inline bool parse_args(int argc, char** argv, FluxCliArgs* args) {
             if (!v) return false;
             args->max_vram = parse_float_value(v, 0.0f);
         } else if (std::strcmp(key, "--flash-attention") == 0 ||
-                   std::strcmp(key, "--flash-attn") == 0) {
+                   std::strcmp(key, "--flash-attn") == 0 ||
+                   std::strcmp(key, "--diffusion-fa") == 0 ||
+                   std::strcmp(key, "--diffusion_fa") == 0) {
             args->flash_attention = true;
         } else if (std::strcmp(key, "--no-flash-attention") == 0 ||
                    std::strcmp(key, "--no-flash-attn") == 0) {
             args->flash_attention = false;
+        } else if (std::strcmp(key, "--rng") == 0) {
+            const char* ignored_rng = require_value(key);
+            if (!ignored_rng) return false;
+        } else if (std::strcmp(key, "-v") == 0 || std::strcmp(key, "--verbose") == 0) {
+            // Accepted for sd.cpp CLI compatibility; edge-dit logging is controlled externally.
         } else if (std::strcmp(key, "--cfg-parallel-size") == 0 ||
                    std::strcmp(key, "--cfg-size") == 0) {
             const char* v = require_value(key);
@@ -820,9 +869,13 @@ inline bool parse_args(int argc, char** argv, FluxCliArgs* args) {
         args->vae_path != nullptr && std::strlen(args->vae_path) > 0 &&
         args->clip_l_path != nullptr && std::strlen(args->clip_l_path) > 0 &&
         (args->no_t5 || (args->t5xxl_path != nullptr && std::strlen(args->t5xxl_path) > 0));
+    const bool has_minimax_h3_components =
+        args->diffusion_model_path != nullptr && std::strlen(args->diffusion_model_path) > 0 &&
+        args->vae_path != nullptr && std::strlen(args->vae_path) > 0 &&
+        args->llm_path != nullptr && std::strlen(args->llm_path) > 0;
 
-    if (!has_full_model && !has_components) {
-        std::fprintf(stderr, "--model or the full --diffusion-model/--vae/--clip_l/(--t5xxl or --no-t5) set is required\n");
+    if (!has_full_model && !has_components && !has_minimax_h3_components) {
+        std::fprintf(stderr, "--model, --diffusion-model/--vae/--clip_l/(--t5xxl or --no-t5), or --diffusion-model/--vae/--llm is required\n");
         return false;
     }
 
@@ -850,6 +903,11 @@ inline bool parse_args(int argc, char** argv, FluxCliArgs* args) {
         return false;
     }
 
+    if (args->ref_video_audio_paths.size() > args->ref_video_paths.size()) {
+        std::fprintf(stderr, "each --ref-video-audio needs a corresponding --ref-video\n");
+        return false;
+    }
+
     const std::string video_format = normalized_video_format(args->video_format);
     if (video_format != "auto" &&
         video_format != "avi" &&