diff --git a/docs.json b/docs.json index 3457c18be..2dad19e75 100644 --- a/docs.json +++ b/docs.json @@ -320,7 +320,8 @@ "tutorials/video/wan/fun-control", "tutorials/video/wan/fun-camera", "tutorials/video/wan/fun-inp", - "tutorials/video/wan/wan-flf" + "tutorials/video/wan/wan-flf", + "tutorials/video/wan/wan-infinitetalk" ] } ] diff --git a/tutorials/video/wan/wan-infinitetalk.mdx b/tutorials/video/wan/wan-infinitetalk.mdx new file mode 100644 index 000000000..6b88c5fb2 --- /dev/null +++ b/tutorials/video/wan/wan-infinitetalk.mdx @@ -0,0 +1,123 @@ +--- +title: "ComfyUI Wan2.1 InfiniteTalk Workflow Example" +description: "InfiniteTalk is an audio-driven full-body video dubbing model built on Wan2.1, enabling character lip-sync and body motion synchronization with any input audio." +sidebarTitle: "InfiniteTalk" +--- + +import UpdateReminder from '/snippets/tutorials/update-reminder.mdx' + +**Wan2.1 InfiniteTalk** is an open-source audio-driven video generation model built on Wan2.1, developed by [MeiGen-AI](https://github.com/MeiGen-AI/InfiniteTalk). It enables you to generate full-body talking videos from a single reference image and an audio input. The character's mouth movements and body motions are automatically synchronized to match the provided audio. + +**Key Features**: +- **Audio-Driven Lip Sync**: Generate natural mouth movements that match the input audio +- **Full-Body Motion**: Preserves identity, background, and camera movement while adding synchronized body motion +- **Dual Mode**: Supports both single-character and multi-character scenarios +- **ComfyUI Native**: Built-in `WanInfiniteTalkToVideo` node, no custom nodes required + +**Related Links**: +- [InfiniteTalk GitHub Repository](https://github.com/MeiGen-AI/InfiniteTalk) +- [Wan2.1 Code Repository](https://github.com/Wan-Video/Wan2.1) +- [Wan2.1 Model Repository](https://huggingface.co/Wan-AI) + + + This workflow uses Subgraph nodes for modular processing. Check out the Subgraph documentation to learn how to customize and extend the workflow. + + +## InfiniteTalk image-to-video workflow + + + + Open in Comfy Cloud + + + Download JSON or search "InfiniteTalk" in Template Library + + + +![Wan2.1 InfiniteTalk Workflow Preview](https://raw.githubusercontent.com/Comfy-Org/workflow_templates/main/templates/video_wan2_1_infinitetalk-1.webp) + + + +## Model Installation + +The following models need to be downloaded and placed in the correct directories: + +**diffusion_models** + +- [Wan2_1-I2V-14B-480p_fp8_e4m3fn_scaled_KJ.safetensors](https://huggingface.co/Kijai/WanVideo_comfy_fp8_scaled/resolve/main/I2V/Wan2_1-I2V-14B-480p_fp8_e4m3fn_scaled_KJ.safetensors) + +**text_encoders** + +- [umt5_xxl_fp8_e4m3fn_scaled.safetensors](https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/resolve/main/split_files/text_encoders/umt5_xxl_fp8_e4m3fn_scaled.safetensors) + +**model_patches** + +- [wan2.1_infiniteTalk_single_fp16.safetensors](https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/resolve/main/split_files/model_patches/wan2.1_infiniteTalk_single_fp16.safetensors): For single-character scenarios +- [wan2.1_infiniteTalk_multi_fp16.safetensors](https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/resolve/main/split_files/model_patches/wan2.1_infiniteTalk_multi_fp16.safetensors): For multi-character scenarios + +**audio_encoders** + +- [wav2vec2-chinese-base_fp16.safetensors](https://huggingface.co/Kijai/wav2vec2_safetensors/resolve/main/wav2vec2-chinese-base_fp16.safetensors) + +**vae** + +- [Wan2_1_VAE_bf16.safetensors](https://huggingface.co/Kijai/WanVideo_comfy/resolve/main/Wan2_1_VAE_bf16.safetensors) + +**loras** + +- [lightx2v_I2V_14B_480p_cfg_step_distill_rank64_bf16.safetensors](https://huggingface.co/Kijai/WanVideo_comfy/resolve/main/Lightx2v/lightx2v_I2V_14B_480p_cfg_step_distill_rank64_bf16.safetensors) + +### Model Storage Location + +``` +📂 ComfyUI/ +├── 📂 models/ +│ ├── 📂 diffusion_models/ +│ │ └── Wan2_1-I2V-14B-480p_fp8_e4m3fn_scaled_KJ.safetensors +│ ├── 📂 text_encoders/ +│ │ └── umt5_xxl_fp8_e4m3fn_scaled.safetensors +│ ├── 📂 model_patches/ +│ │ ├── wan2.1_infiniteTalk_single_fp16.safetensors +│ │ └── wan2.1_infiniteTalk_multi_fp16.safetensors +│ ├── 📂 audio_encoders/ +│ │ └── wav2vec2-chinese-base_fp16.safetensors +│ ├── 📂 vae/ +│ │ └── Wan2_1_VAE_bf16.safetensors +│ └── 📂 loras/ +│ └── lightx2v_I2V_14B_480p_cfg_step_distill_rank64_bf16.safetensors +``` + +### Sample Input Files + +Download these sample input files and drag them into the corresponding nodes: + + + + Character reference image + + + Audio track for character 1 + + + Audio track for character 2 + + + +## Workflow Steps + +1. **Load the input image**: Drag the character reference image to the `Load Image` node. For multi-character scenarios, use the Mask Editor to draw masks for each character. +2. **Load audio tracks**: Connect audio files to the `Load Audio` nodes (one per character). +3. **Load diffusion model**: Ensure the `Load Diffusion Model` node is using `Wan2_1-I2V-14B-480p_fp8_e4m3fn_scaled_KJ.safetensors`. +4. **Load model patches**: Load the appropriate InfiniteTalk patch (`single` or `multi` variant) via the `ModelPatchLoader` node. +5. **Configure InfiniteTalk**: Adjust the `WanInfiniteTalkToVideo` node parameters (video length, motion amount, etc.). +6. **Generate**: Run the workflow. The model will produce a full-body talking video synchronized with the input audio. + +### Extending Video Length + +Each **Video Extend** group extends the video by approximately 3.24 seconds (81 frames at 25 fps). If your audio is longer, you can: + +1. Box-select the "Video Extend" group +2. Press `Ctrl-C` (copy), then `Ctrl-Shift-V` (paste with connections) +3. Connect the `Batch Images` node's `IMAGE` output to the new `WanInfiniteTalkToVideo` node's `previous_frames` +4. Connect the `Batch Images` node's `IMAGE` output to the new `Batch Images` node's `images` +5. Adjust the connection between the previous group's `WanInfiniteTalkToVideo` output and the new group's `VAEDecode` input