[metascroy]

Stack

Off-graph KV cache for the native ggml engine, through serving Muse Glimmer on it. Review bottom-up; each PR builds and passes its tests on its own.

• [Native] Launch the Muse Glimmer server on the native backend #24957 (this PR)
• [Native] Serve Muse Glimmer on the native ggml engine #24956
• [Native] Add a batching executor for native-backend packages #25055
• [Native] Select the logits rows an off-graph export returns #25009
• [Native] Gather the rows of x[:, rows] in the ggml engine #25008
• [Native] Serve many sequences from one ggml engine KV cache #25007
• [Native] Separate the ggml engine's K/V storage from its sequence cache #25006
• [Native] Publish the context length with off-graph packages #24955
• [Native] Share the off-graph cache geometry with other callers #24954
• [Native] Export Muse Glimmer through the native backend #24951
• [Native] Replay sample sets without decoding their tokens #24950
• [Native] Map Muse Glimmer GGUF tensors onto its HF model #24949
• [Native] Run GGUF Q5_K weights in the ggml engine #24948
• [Native] Load the first layers of a GGUF with --layers #24947
• [Native] Run the HF LLM export on newer transformers #24946
• [Native] Fuse broadcast RoPE frequencies and lower tanh in ggml #24945
• [Native] Support sliding-window attention with the off-graph KV cache #24944
• [Native] Carry named metadata in native packages #24943
• [Native] Add chunked prefill and interactive chat to the LLM runner #24639
• [Native] Run off-graph KV cache packages in the LLM runner #24638
• [Native] Serve kvcache::update_and_attend in the ggml engine #24637
• [Native] Add load-time options to engine contexts #24636
• [Native] Export HF LLMs with an off-graph KV cache #24635
• Move off-graph HF attention to extension/llm/cache #24634

Summary

serve_batching.py --backend native launches the Muse Glimmer server on the native engine.

• Passes the worker --native-device and --native-kv-dtype.
• Rejects what the native worker can't use, DFlash, the vision table and a --data-path, before starting the worker. Sessions and the prefix cache work as on the other backends.
• BATCHING_SERVING.md covers the native export, build and launch.

python -m executorch.examples.models.muse_glimmer.serving.serve_batching \
  --backend native --worker-bin .../muse_glimmer_batching_worker \
  --model-path mg.ptn --tokenizer-path <mg-hf>/tokenizer.json \
  --hf-tokenizer <mg-hf> --max-context 8192

Test plan

• test_serve_batching.py (33 pass): native options forwarded, omitted for other backends, DFlash, vision and a data path rejected.
• Served the full Muse-Glimmer-30B Q4_K_M export on an M4 Max (Metal) through /v1/chat/completions with --max-sessions 4 --prefix-cache-entries 4:
  • three concurrent requests ran at 31.2 tokens/s against 23.5 tokens/s one at a time, with answers identical to the sequential ones ("The capital of France is Paris.", "Red\nBlue\nYellow", "144");
  • a second request sharing an 848-token prompt reused 842 cached tokens and answered correctly.

---

Authored with Claude Code.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24957

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 63 New Failures, 4 Cancelled Jobs, 1 Pending, 4 Unrelated Failures, 23 Unclassified Failures

As of commit 25cd714 with merge base c7f7931 ():

NEW FAILURES - The following jobs have failed:

• Build documentation / build (buck2) / Build doc (gh)
• Build Presets / apple (ios-simulator) / build (gh)
• Build Presets / apple (ios) / build (gh)
• Build Presets / apple (llm) / build (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• Build Presets / apple (macos) / build (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• Build Presets / apple (profiling) / build (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• Build Presets / apple (pybind) / build (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• Build Presets / linux (linux, mt-l-arm64g4-16-62, executorch-ubuntu-22.04-gcc11-aarch64) / build (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Build Presets / linux (linux, mt-l-x86iavx512-8-64, executorch-ubuntu-22.04-clang12) / build (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Build Presets / linux (llm, mt-l-arm64g4-16-62, executorch-ubuntu-22.04-gcc11-aarch64) / build (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Build Presets / linux (llm, mt-l-x86iavx512-8-64, executorch-ubuntu-22.04-clang12) / build (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Build Presets / linux (pybind, mt-l-arm64g4-16-62, executorch-ubuntu-22.04-gcc11-aarch64) / build (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Build Presets / linux (pybind, mt-l-x86iavx512-8-64, executorch-ubuntu-22.04-clang12) / build (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Build Presets / windows (pybind) / build (gh)
• Build Presets / windows (windows) / build (gh)
• Build Presets / zephyr (zephyr) / build (gh)
• Cadence Build & Test / cpu-build / build (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• MLX / backend-tester (models) / test-mlx-backend-models (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / backend-tester (operators) / test-mlx-backend-operators (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx / test-mlx (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-dflash / test-mlx-dflash (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (google/gemma-4-E2B-it, gemma4-e2b, false, 4w) / test-mlx-llm-gemma4-e2b-4w (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (google/gemma-4-E2B-it, gemma4-e2b, true, 4w) / test-mlx-llm-gemma4-e2b-custom-4w (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/gemma-3-1b-it, gemma3-1b, false, 4w) / test-mlx-llm-gemma3-1b-4w (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/gemma-3-1b-it, gemma3-1b, false, nvfp4) / test-mlx-llm-gemma3-1b-nvfp4 (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/gemma-3-1b-it, gemma3-1b, true, 4w) / test-mlx-llm-gemma3-1b-custom-4w (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/gemma-3-1b-it, gemma3-1b, true, nvfp4) / test-mlx-llm-gemma3-1b-custom-nvfp4 (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/Llama-3.2-1B-Instruct, llama-1b, false, 4w) / test-mlx-llm-llama-1b-4w (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/Llama-3.2-1B-Instruct, llama-1b, false, nvfp4) / test-mlx-llm-llama-1b-nvfp4 (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/Llama-3.2-1B-Instruct, llama-1b, true, 4w) / test-mlx-llm-llama-1b-custom-4w (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/Llama-3.2-1B-Instruct, llama-1b, true, nvfp4) / test-mlx-llm-llama-1b-custom-nvfp4 (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/Qwen3-0.6B, qwen3-0.6b, false, 4w) / test-mlx-llm-qwen3-0.6b-4w (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/Qwen3-0.6B, qwen3-0.6b, false, nvfp4) / test-mlx-llm-qwen3-0.6b-nvfp4 (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/Qwen3-0.6B, qwen3-0.6b, true, 4w) / test-mlx-llm-qwen3-0.6b-custom-4w (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/Qwen3-0.6B, qwen3-0.6b, true, nvfp4) / test-mlx-llm-qwen3-0.6b-custom-nvfp4 (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm-offgraph (google/gemma-4-E2B-it, gemma4-e2b, gemma4) / test-mlx-llm-offgraph-gemma4-e2b (gh)
  ##[error]fatal: unable to access 'https://github.com/google/XNNPACK.git/': Failed to connect to github.com:443 after 8010 ms: Could not connect to server
• MLX / test-mlx-llm-offgraph (unsloth/gemma-3-1b-it, gemma3-1b, gemma) / test-mlx-llm-offgraph-gemma3-1b (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm-offgraph (unsloth/Llama-3.2-1B-Instruct, llama-1b, llama3) / test-mlx-llm-offgraph-llama-1b (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-muse-glimmer-build / test-mlx-muse-glimmer-build (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-nemotron3-diarization-build / test-mlx-nemotron3-diarization-build (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-parakeet / test-mlx-parakeet (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-qwen35-moe / test-mlx-qwen35-moe (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-stories110m / test-mlx-stories110m (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-supertonic / test-mlx-supertonic (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-voxtral / test-mlx-voxtral (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-voxtral-realtime / test-mlx-voxtral-realtime (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-whisper / test-mlx-whisper (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• Test ARM Backend / test-arm / test-backend-linux (arm_tosa_fp, models) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test ARM Backend / test-arm / test-backend-linux (arm_tosa_fp, operators) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test ARM Backend / test-arm-vgf / test-backend-linux (arm_vgf_fp, models) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test ARM Backend / test-arm-vgf / test-backend-linux (arm_vgf_fp, operators) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test CPU Backend / test-cpu / test-backend-linux (cpu_fp32, models) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test CPU Backend / test-cpu / test-backend-linux (cpu_fp32, operators) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test CUDA Builds / test-muse-glimmer-cuda-e2e-kquant-17gb-dflash-image / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test CUDA Builds / test-muse-glimmer-cuda-e2e-kquant-17gb-solo-text / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test CUDA Builds / test-muse-glimmer-cuda-e2e-kquant-17gb-solo-text-batching / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test CUDA Builds / test-muse-glimmer-cuda-e2e-kquant-dynamic-dflash-image / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test CUDA Builds / test-muse-glimmer-cuda-e2e-kquant-dynamic-solo-text / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test CUDA Builds / unittest-cuda-muse-glimmer-batching / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test QNN Backend / test-qnn / test-backend-linux (qnn, models) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test QNN Backend / test-qnn / test-backend-linux (qnn, operators) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test Vulkan Backend / test-vulkan / test-backend-linux (vulkan, models) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test Vulkan Backend / test-vulkan / test-backend-linux (vulkan, operators) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

• Build Aarch64 Linux CUDA Wheels / pytorch/executorch / build-wheel-py3_11-cuda-aarch6413_2-aarch64 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• Build Aarch64 Linux CUDA Wheels / pytorch/executorch / upload / upload-wheel-py3_11-cuda-aarch6413_2-aarch64 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build Aarch64 Linux Native Wheels / executorch-native / build-wheel-py3_11-cpu-aarch64 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  ModuleNotFoundError: No module named 'setuptools'
• Build Aarch64 Linux Native Wheels / executorch-native / upload / upload-wheel-py3_11-cpu-aarch64 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build Aarch64 Linux Wheels / pytorch/executorch / build-wheel-py3_11-cpu-aarch64 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• Build Aarch64 Linux Wheels / pytorch/executorch / upload / upload-wheel-py3_11-cpu-aarch64 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build Linux CUDA Wheels / pytorch/executorch / build-manywheel-py3_11-cuda13_2 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• Build Linux CUDA Wheels / pytorch/executorch / upload / upload-manywheel-py3_11-cuda13_2 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build Linux Native Wheels / executorch-native / build-manywheel-py3_11-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  ModuleNotFoundError: No module named 'setuptools'
• Build Linux Native Wheels / executorch-native / upload / upload-manywheel-py3_11-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build Linux Wheels / pytorch/executorch / build-manywheel-py3_11-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• Build Linux Wheels / pytorch/executorch / upload / upload-manywheel-py3_11-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build macOS Wheels / pytorch/executorch / build-wheel-py3_11-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• Build macOS Wheels / pytorch/executorch / upload / upload-wheel-py3_11-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build Windows Wheels / pytorch/executorch / build-wheel-py3_11-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• Build Windows Wheels / pytorch/executorch / upload / upload-wheel-py3_11-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• periodic / test-models-linux (buck2, mv3, portable, mt-l-x86iavx512-8-64, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• periodic / test-models-linux (buck2, mv3, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• periodic / test-models-linux (cmake, mv3, portable, mt-l-x86iavx512-8-64, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• periodic / test-models-linux (cmake, mv3, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• periodic / test-models-linux (cmake, vit, portable, mt-l-x86iavx512-8-64, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• periodic / test-models-linux (cmake, vit, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)

CANCELLED JOBS - The following jobs were cancelled. Please retry:

• Test Cortex-M Backend / Get changed files / get-changed-files (gh)
• Test WebGPU Backend / test-webgpu / package-golden-artifacts (gh)
• Test WebGPU Backend / test-webgpu / Resolve CI docker image / resolve (gh)
• Test XNNPACK Backend / test-xnnpack / package-golden-artifacts (gh)

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Cadence Build & Test / hifi-build / hifi4 (gh) (detected as infra flaky with no log or failing log classifier)
• Cadence Build & Test / hifi1s-build / hifi1s (gh) (detected as infra flaky with no log or failing log classifier)
• Cadence Build & Test / hifi5s-build / hifi5s (gh) (detected as infra flaky with no log or failing log classifier)
• Cadence Build & Test / vision-build / vision (gh) (detected as infra flaky with no log or failing log classifier)

This comment was automatically generated by Dr. CI and updates every 15 minutes.