[Bug] ROCm Q6_K pruned streaming degrades on OCuLink (missing prefetch)

[farawayso]

[Bug] ROCm Q6_K pruned streaming degrades on OCuLink (missing prefetch)

Environment

• OS: Windows 11
• GPU: AMD Radeon RX 7900 XTX 24GB (gfx1100 / RDNA3)
• iGPU: AMD Radeon 890M (integrated in 890M CPU)
• Connection: OCuLink external GPU dock (PCIe 4.0 x4,实测 bandwidth ~6.5 GB/s)
• sd-cli version: master-849 (d04e895), includes PR feat: --stream-layers for streaming weights from CPU during generation #1576 (--stream-layers)
• Missing: PR feat: prefetch streamed layers during compute #1905 (--layer-prefetch-depth, --resident-layers) not compiled
• Model: minimax_h3_fl2va_pruned-Q6_K.gguf (15.9 GB)
• Config: 480x864, 73 frames, 5 steps, Euler scheduler, CFG=1.0

Problem Description

When using --stream-layers + --params-backend diffusion=cpu streaming mode:

• Step 1: Normal (~40s)
• Steps 2-N: Extremely slow (~825s/it), outputs black screen (52KB video)

Expected: master-841 generated Q6_K normally at ~40s/step.

Log Evidence

[INFO ] model_manager.cpp:428  - model manager prepared params backend buffers (15902.72 MB, 532 tensors, 1 blocks, RAM) on ROCm_Host
[INFO ] model_manager.cpp:555  - model manager staged compute params (15902.75 MB, 532 tensors, 210 blocks) to ROCm0, taking 3.16s
[DEBUG] ggml_runner.cpp:919  - minimax_h3 executing segment 1/1: graph
# Subsequent steps hang or are extremely slow...

Key observations:

1. 15.9GB weights loaded as 1 blocks, RAM - bulk transfer
2. Graph cut produces 210 blocks, but no prefetch pipeline
3. Each step requires full staging, HIP driver degrades after multiple stagings

Root Cause Analysis

1. Missing Async Prefetch Pipeline

PR #1576 only provides basic residency framework:

• ✅ --stream-layers enabled
• ❌ No --layer-prefetch-depth (to hide PCIe/OCuLink transfer latency)
• ❌ No --resident-layers (minimal rolling window control)

PR #1905 was merged to upstream master (2026-09-06) but not compiled into local binary.

2. OCuLink Bandwidth Bottleneck

Metric	Value
Interface	OCuLink (PCIe 4.0 x4)
Measured bandwidth	~6.5 GB/s
Weight size	15.9 GB (Q6_K pruned)
Theoretical min transfer	15.9 / 6.5 ≈ 2.4s
Actual per-step time	825s (Steps 2-5)

Degradation cause:

• No prefetch → serial load-then-compute per step
• 210 blocks repeatedly transferred triggers HIP driver fallback
• OCuLink bandwidth is far below PCIe 5.0 x16 (~64 GB/s), latency-sensitive

3. Comparison: master-841 vs Current Binary

Feature	master-841	Current binary
--stream-layers	✅ Yes	✅ Yes
--layer-prefetch-depth	❌ No	❌ No
--resident-layers	❌ No	❌ No
Async prefetch pipeline	❌ No	❌ No
Performance (Q6_K 73f)	~40s/step	825s/step ✗

Conclusion: --stream-layers alone cannot solve performance issues in OCuLink bandwidth-limited environments. PR #1905's prefetch pipeline is needed to hide transfer latency.

Reproduction Steps

# Environment
export H3_QUANT=Q6_K
export H3_MAX_VRAM=23

# Run (will auto-enable streaming mode)
python scripts/sd_interactive_video.py t2va pruned short 480 864 3 1 5 auto

Or manual command:

./rocm/sd-cli.exe \
  --diffusion-model minimax_h3_fl2va_pruned-Q6_K.gguf \
  --llm qwen3vl_32b_minimax_h3-Q4_K_M.gguf \
  --vae minimax_h3_video_vae_fp16.safetensors \
  --lora-model-dir loras/ --lora-apply-mode at_runtime \
  -p "test prompt" \
  --steps 5 --cfg-scale 1.0 --sampling-method euler \
  -W 480 -H 864 --video-frames 73 --seed 42 \
  --backend llm=cpu,vae=ROCm0,diffusion=ROCm0 \
  --params-backend diffusion=cpu \
  --stream-layers \
  --max-vram ROCm0=23 \
  --temporal-tiling \
  -o out.mp4

Expected Behavior

• Steps 1-5 should each take ~40s consistently
• No black screen output
• --layer-prefetch-depth 1 should hide OCuLink transfer latency

Actual Behavior

• Step 1: ~40s ✅
• Steps 2-5: ~825s/it ❌ (20x slower)
• Output: 52KB black screen video ❌

Suggested Fixes

Short-term Workarounds (User-side)

# Option 1: Force eager mode (weights resident in GPU)
H3_FORCE_EAGER=1 H3_QUANT=Q6_K python scripts/sd_interactive_video.py ...

# Option 2: Downgrade to Q4_K_M
H3_QUANT=Q4_K_M python scripts/sd_interactive_video.py ...

# Option 3: Reduce frames to under 56
python scripts/sd_interactive_video.py ...  # Choose 56 frames

Long-term Fixes (Upstream)

1. Prioritize releasing PR feat: prefetch streamed layers during compute #1905 (--layer-prefetch-depth) and PR feat: pool VRAM for streamed layer #1906 (--stream-layer-pool)
2. Update documentation with OCuLink/low-bandwidth environment configuration tips
3. Consider adding auto-detection for low-bandwidth environments with eager fallback

Additional Notes

• This machine uses OCuLink external GPU dock, bandwidth is far below native PCIe
• On主板 directly (PCIe 5.0 x16), performance may be normal
• This is a combination issue: OCuLink bandwidth limitation + missing prefetch pipeline

Related Links

• PR feat: --stream-layers for streaming weights from CPU during generation #1576: feat: --stream-layers for streaming weights from CPU during generation
• PR feat: prefetch streamed layers during compute #1905: feat: prefetch streamed layers during compute (merged 2026-09-06)
• PR feat: pool VRAM for streamed layer #1906: feat: pool VRAM for streamed layer (open)