[metascroy]

Stack

Off-graph KV cache for the native ggml engine, through serving Muse Glimmer on it. Review bottom-up; each PR builds and passes its tests on its own.

• [Native] Launch the Muse Glimmer server on the native backend #24957
• [Native] Serve Muse Glimmer on the native ggml engine #24956 (this PR)
• [Native] Add a batching executor for native-backend packages #25055
• [Native] Select the logits rows an off-graph export returns #25009
• [Native] Gather the rows of x[:, rows] in the ggml engine #25008
• [Native] Serve many sequences from one ggml engine KV cache #25007
• [Native] Separate the ggml engine's K/V storage from its sequence cache #25006
• [Native] Publish the context length with off-graph packages #24955
• [Native] Share the off-graph cache geometry with other callers #24954
• [Native] Export Muse Glimmer through the native backend #24951
• [Native] Replay sample sets without decoding their tokens #24950
• [Native] Map Muse Glimmer GGUF tensors onto its HF model #24949
• [Native] Run GGUF Q5_K weights in the ggml engine #24948
• [Native] Load the first layers of a GGUF with --layers #24947
• [Native] Run the HF LLM export on newer transformers #24946
• [Native] Fuse broadcast RoPE frequencies and lower tanh in ggml #24945
• [Native] Support sliding-window attention with the off-graph KV cache #24944
• [Native] Carry named metadata in native packages #24943
• [Native] Add chunked prefill and interactive chat to the LLM runner #24639
• [Native] Run off-graph KV cache packages in the LLM runner #24638
• [Native] Serve kvcache::update_and_attend in the ggml engine #24637
• [Native] Add load-time options to engine contexts #24636
• [Native] Export HF LLMs with an off-graph KV cache #24635
• Move off-graph HF attention to extension/llm/cache #24634

Summary

Muse Glimmer's backend factory takes --backend native. It creates the native backend's NativeExecutor (previous PR) over the ggml engine and its batched cache, then adds what's specific to Muse Glimmer, as it does around CUDA's CudaExecutor:

• a check that the vocabulary holds its special tokens;
• the preparation spec for text-only prompts.

The package comes from export_hf_llm.py --offgraph_cache --logits_to_keep selected. Sessions batch together and clone for prefix reuse; there's no DFlash or vision. The worker takes --native_device (cpu or gpu) and --native_kv_dtype (fp32 or fp16), rejects any other value, and rejects --data_path and --pos_embed_path, which the native backend doesn't use.

Build: the Muse Glimmer CMake project builds its batching targets from an installation with only EXECUTORCH_BUILD_NATIVE_GGML, skipping the CUDA/MLX-only runners; make muse-glimmer-batching-native builds it, with ggml's OpenMP off so the installed package doesn't depend on the build machine's OpenMP. Image hashing picks its SHA-256 by platform (CommonCrypto on Apple, portable elsewhere) rather than by CUDA or MLX.

Most of the CMakeLists.txt diff is re-indentation from wrapping the legacy runners in their backend condition.

Test plan

• Built from a fresh configure with the muse-glimmer-batching-native recipe's settings on macOS, linking the installed native_batching. The full Muse-Glimmer-30B-KQuant-17GB-Q4_K_M export, with selected logits, serves through the next PR's launcher with four sessions and a prefix cache (results there).
• Not built locally: MLX or CUDA.

---

Authored with Claude Code.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24956

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 172 New Failures, 1 Cancelled Job, 6 Unrelated Failures, 27 Unclassified Failures

As of commit b3f6fc5 with merge base c7f7931 ():

NEW FAILURES - The following jobs have failed:

• Build documentation / build (buck2) / Build doc (gh)
• Build Presets / apple (ios-simulator) / build (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• Build Presets / apple (ios) / build (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• Build Presets / apple (llm) / build (gh)
• Build Presets / apple (macos) / build (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• Build Presets / apple (profiling) / build (gh)
• Build Presets / apple (pybind) / build (gh)
• Build Presets / linux (linux, mt-l-arm64g4-16-62, executorch-ubuntu-22.04-gcc11-aarch64) / build (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Build Presets / linux (linux, mt-l-x86iavx512-8-64, executorch-ubuntu-22.04-clang12) / build (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Build Presets / linux (llm, mt-l-arm64g4-16-62, executorch-ubuntu-22.04-gcc11-aarch64) / build (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Build Presets / linux (llm, mt-l-x86iavx512-8-64, executorch-ubuntu-22.04-clang12) / build (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Build Presets / linux (pybind, mt-l-arm64g4-16-62, executorch-ubuntu-22.04-gcc11-aarch64) / build (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Build Presets / linux (pybind, mt-l-x86iavx512-8-64, executorch-ubuntu-22.04-clang12) / build (gh)
• Build Presets / windows (pybind) / build (gh)
• Build Presets / windows (windows) / build (gh)
• Build Presets / zephyr (zephyr) / build (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Cadence Build & Test / cpu-build / build (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• MLX / backend-tester (models) / test-mlx-backend-models (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / backend-tester (operators) / test-mlx-backend-operators (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx / test-mlx (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-dflash / test-mlx-dflash (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (google/gemma-4-E2B-it, gemma4-e2b, false, 4w) / test-mlx-llm-gemma4-e2b-4w (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (google/gemma-4-E2B-it, gemma4-e2b, true, 4w) / test-mlx-llm-gemma4-e2b-custom-4w (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/gemma-3-1b-it, gemma3-1b, false, 4w) / test-mlx-llm-gemma3-1b-4w (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/gemma-3-1b-it, gemma3-1b, false, nvfp4) / test-mlx-llm-gemma3-1b-nvfp4 (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/gemma-3-1b-it, gemma3-1b, true, 4w) / test-mlx-llm-gemma3-1b-custom-4w (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/gemma-3-1b-it, gemma3-1b, true, nvfp4) / test-mlx-llm-gemma3-1b-custom-nvfp4 (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/Llama-3.2-1B-Instruct, llama-1b, false, 4w) / test-mlx-llm-llama-1b-4w (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/Llama-3.2-1B-Instruct, llama-1b, false, nvfp4) / test-mlx-llm-llama-1b-nvfp4 (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/Llama-3.2-1B-Instruct, llama-1b, true, 4w) / test-mlx-llm-llama-1b-custom-4w (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/Llama-3.2-1B-Instruct, llama-1b, true, nvfp4) / test-mlx-llm-llama-1b-custom-nvfp4 (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/Qwen3-0.6B, qwen3-0.6b, false, 4w) / test-mlx-llm-qwen3-0.6b-4w (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/Qwen3-0.6B, qwen3-0.6b, false, nvfp4) / test-mlx-llm-qwen3-0.6b-nvfp4 (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/Qwen3-0.6B, qwen3-0.6b, true, 4w) / test-mlx-llm-qwen3-0.6b-custom-4w (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/Qwen3-0.6B, qwen3-0.6b, true, nvfp4) / test-mlx-llm-qwen3-0.6b-custom-nvfp4 (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm-offgraph (google/gemma-4-E2B-it, gemma4-e2b, gemma4) / test-mlx-llm-offgraph-gemma4-e2b (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm-offgraph (unsloth/gemma-3-1b-it, gemma3-1b, gemma) / test-mlx-llm-offgraph-gemma3-1b (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm-offgraph (unsloth/Llama-3.2-1B-Instruct, llama-1b, llama3) / test-mlx-llm-offgraph-llama-1b (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-muse-glimmer-build / test-mlx-muse-glimmer-build (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-nemotron3-diarization-build / test-mlx-nemotron3-diarization-build (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-parakeet / test-mlx-parakeet (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-qwen35-moe / test-mlx-qwen35-moe (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-stories110m / test-mlx-stories110m (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-supertonic / test-mlx-supertonic (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-voxtral / test-mlx-voxtral (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-voxtral-realtime / test-mlx-voxtral-realtime (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-whisper / test-mlx-whisper (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• pull / nxp-mcuxpresso-test / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-arduino-library / run / arduino-library (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-arm-backend-no-driver (test_pytest_models_tosa) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-arm-backend-no-driver (test_pytest_ops_no_target) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-arm-backend-no-driver (test_pytest_ops_tosa) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-arm-backend-no-driver (test_run_tosa) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-arm-backend-public-api-backward-compatibility / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-arm-backend-vkml (test_ootb_tests_vgf) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-arm-backend-vkml (test_pytest_ops_vkml) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-arm-backend-vkml (test_runtime_vgf) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-arm-cortex-m-size-test (bare_metal) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-arm-cortex-m-size-test (zephyr-preset) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-binary-size-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-binary-size-linux-gcc / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-build-wasm-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-custom-ops-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-eval_llama-wikitext-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-llama_runner_eager-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-llama-runner-linux (bf16, custom, mt-l-x86iavx512-8-64, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-llama-runner-linux (fp32, xnnpack+custom+qe, mt-l-arm64g4-16-62, executorch-ubuntu-22.04-gcc... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-llama-runner-linux (fp32, xnnpack+custom+qe, mt-l-x86iavx512-8-64, executorch-ubuntu-22.04-c... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, mt-l-arm64g4-16-62, executorch-ubuntu-... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, mt-l-x86iavx512-8-64, executorch-ubunt... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-llama-runner-linux (fp32, xnnpack+quantize_kv, mt-l-arm64g4-16-62, executorch-ubuntu-22.04-g... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-llama-runner-linux (fp32, xnnpack+quantize_kv, mt-l-x86iavx512-8-64, executorch-ubuntu-22.04... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-llama-runner-linux-android / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-lora-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-lora-multimethod-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-mcu-cortex-m-backend / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (add_mul, portable, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (add_mul, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (add, portable, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (add, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (emformer_join, portable, mt-l-x86iavx512-16-128) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (emformer_join, xnnpack-quantization-delegation, mt-l-x86iavx512-16-128) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (emformer_transcribe, portable, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (emformer_transcribe, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (ic3, portable, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (ic3, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (ic4, portable, mt-l-x86iavx512-16-128) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (ic4, xnnpack-quantization-delegation, mt-l-x86iavx512-16-128) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (linear, portable, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (linear, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (llama3_2_vision_encoder, portable, mt-l-x86iavx512-16-128) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (mobilebert, portable, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (mobilebert, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (mv2, portable, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (mv2, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (phi_4_mini, portable, mt-l-x86iavx512-16-128) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (resnet18, portable, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (resnet18, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (resnet50, portable, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (resnet50, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (w2l, portable, mt-l-x86iavx512-16-128) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux-basic (mv3, portable, buck2, mt-l-x86iavx512-8-64, executorch-ubuntu-22.04-clan... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux-basic (mv3, portable, cmake, mt-l-arm64g4-16-62, executorch-ubuntu-22.04-gcc11-... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux-basic (mv3, portable, cmake, mt-l-x86iavx512-8-64, executorch-ubuntu-22.04-clan... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, buck2, mt-l-x86iavx512-8-64, execu... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, cmake, mt-l-arm64g4-16-62, executo... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, cmake, mt-l-x86iavx512-8-64, execu... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux-basic (vit, portable, buck2, mt-l-x86iavx512-8-64, executorch-ubuntu-22.04-clan... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux-basic (vit, portable, cmake, mt-l-arm64g4-16-62, executorch-ubuntu-22.04-gcc11-... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux-basic (vit, portable, cmake, mt-l-x86iavx512-8-64, executorch-ubuntu-22.04-clan... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, buck2, mt-l-x86iavx512-8-64, execu... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, cmake, mt-l-arm64g4-16-62, executo... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, cmake, mt-l-x86iavx512-8-64, execu... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-moshi-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-multimodal-linux (gemma3-4b) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-openvino-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-parakeet-xnnpack-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-phi-3-mini-runner-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-pybindings-no-aten / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-qnn-buck-build-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-qnn-delegate-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-qnn-direct-build-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-qnn-models-linux (dl3) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-qnn-models-linux (mv2) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-qnn-models-linux (mv3) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-qnn-passes-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-qnn-python-imports-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-qnn-testsuite-linux / test-backend-linux (qnn, models) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-qnn-testsuite-linux / test-backend-linux (qnn, operators) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-qnn-wheel-packages-linux (3.11) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-qnn-wheel-packages-linux (3.12) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-qnn-wheel-packages-linux (3.13) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-quantized-aot-lib-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-selective-build-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-setup-linux-gcc / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-sqnr-static-llm-qnn-linux (smollm2_135m) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-static-llama-qnn-linux (stories_110m) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-static-llama-qnn-linux (stories_260k_bc) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-voxtral-realtime-xnnpack-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-vulkan-models-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-vulkan-operators-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / unittest / linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / unittest / macos / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• pull / unittest-buck / linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / unittest-buck / macos / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• pull / unittest-editable / linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / unittest-editable / macos / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• pull / unittest-nxp-neutron / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / unittest-wasm-bindings (--enable-etdump) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test ARM Backend / test-arm / test-backend-linux (arm_tosa_fp, models) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test ARM Backend / test-arm / test-backend-linux (arm_tosa_fp, operators) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test ARM Backend / test-arm-vgf / test-backend-linux (arm_vgf_fp, models) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test ARM Backend / test-arm-vgf / test-backend-linux (arm_vgf_fp, operators) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test CPU Backend / test-cpu / test-backend-linux (cpu_fp32, models) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test CPU Backend / test-cpu / test-backend-linux (cpu_fp32, operators) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test CUDA Builds / test-muse-glimmer-cuda-e2e-kquant-17gb-dflash-image / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test CUDA Builds / test-muse-glimmer-cuda-e2e-kquant-17gb-solo-text / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test CUDA Builds / test-muse-glimmer-cuda-e2e-kquant-17gb-solo-text-batching / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test CUDA Builds / test-muse-glimmer-cuda-e2e-kquant-dynamic-dflash-image / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test CUDA Builds / test-muse-glimmer-cuda-e2e-kquant-dynamic-solo-text / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test CUDA Builds / unittest-cuda-muse-glimmer-batching / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test QNN Backend / test-qnn / test-backend-linux (qnn, models) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test QNN Backend / test-qnn / test-backend-linux (qnn, operators) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test QNN Windows MSVC build / build-qnn-windows-arm64 (gh)
• Test Vulkan Backend / test-vulkan / test-backend-linux (vulkan, models) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test Vulkan Backend / test-vulkan / test-backend-linux (vulkan, operators) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test WebGPU Backend / test-webgpu / test-backend-linux (webgpu, models) / linux-job (gh)
  curl: (22) The requested URL returned error:
• Test WebGPU Backend / test-webgpu / test-backend-linux (webgpu, operators) / linux-job (gh)
  curl: (22) The requested URL returned error:
• Test XNNPACK Backend / test-xnnpack / test-backend-linux (xnnpack, models) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test XNNPACK Backend / test-xnnpack / test-backend-linux (xnnpack, operators) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

• Build Aarch64 Linux CUDA Wheels / pytorch/executorch / build-wheel-py3_11-cuda-aarch6413_2-aarch64 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• Build Aarch64 Linux CUDA Wheels / pytorch/executorch / upload / upload-wheel-py3_11-cuda-aarch6413_2-aarch64 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build Aarch64 Linux Native Wheels / executorch-native / build-wheel-py3_11-cpu-aarch64 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  ModuleNotFoundError: No module named 'setuptools'
• Build Aarch64 Linux Native Wheels / executorch-native / upload / upload-wheel-py3_11-cpu-aarch64 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build Aarch64 Linux Wheels / pytorch/executorch / build-wheel-py3_11-cpu-aarch64 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• Build Aarch64 Linux Wheels / pytorch/executorch / upload / upload-wheel-py3_11-cpu-aarch64 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build Linux CUDA Wheels / pytorch/executorch / build-manywheel-py3_11-cuda13_2 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• Build Linux CUDA Wheels / pytorch/executorch / upload / upload-manywheel-py3_11-cuda13_2 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build Linux Native Wheels / executorch-native / build-manywheel-py3_11-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  ModuleNotFoundError: No module named 'setuptools'
• Build Linux Native Wheels / executorch-native / upload / upload-manywheel-py3_11-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build Linux Wheels / pytorch/executorch / build-manywheel-py3_11-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• Build Linux Wheels / pytorch/executorch / upload / upload-manywheel-py3_11-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build macOS Wheels / pytorch/executorch / build-wheel-py3_11-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• Build macOS Wheels / pytorch/executorch / upload / upload-wheel-py3_11-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build Windows CUDA Wheels / bundle-cuda-windows-wheel-e2e-13.2 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build Windows CUDA Wheels / export-cuda-windows-wheel-program-13.2 / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Build Windows CUDA Wheels / pytorch/executorch / build-wheel-py3_11-cuda13_2 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• Build Windows CUDA Wheels / pytorch/executorch / upload / upload-wheel-py3_11-cuda13_2 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build Windows CUDA Wheels / test-cuda-windows-wheel-e2e-13.2 / windows-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build Windows Wheels / pytorch/executorch / build-wheel-py3_11-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• Build Windows Wheels / pytorch/executorch / upload / upload-wheel-py3_11-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• periodic / test-models-linux (buck2, mv3, portable, mt-l-x86iavx512-8-64, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• periodic / test-models-linux (buck2, mv3, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• periodic / test-models-linux (cmake, mv3, portable, mt-l-x86iavx512-8-64, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• periodic / test-models-linux (cmake, mv3, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• periodic / test-models-linux (cmake, vit, portable, mt-l-x86iavx512-8-64, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• periodic / test-models-linux (cmake, vit, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1

CANCELLED JOB - The following job was cancelled. Please retry:

• Test ROCm AOTI / test-voxtral-realtime-rocm-gfx950-rocm7.2 / linux-job (gh)
  ##[error]The operation was canceled.

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Cadence Build & Test / hifi-build / hifi4 (gh) (detected as infra flaky with no log or failing log classifier)
• Cadence Build & Test / hifi1s-build / hifi1s (gh) (detected as infra flaky with no log or failing log classifier)
• Cadence Build & Test / hifi5s-build / hifi5s (gh) (detected as infra flaky with no log or failing log classifier)
• Cadence Build & Test / vision-build / vision (gh) (detected as infra flaky with no log or failing log classifier)
• pull / unittest / windows / windows-job (gh) (detected as infra flaky with no log or failing log classifier)
• pull / unittest-editable / windows / windows-job (gh) (detected as infra flaky with no log or failing log classifier)

This comment was automatically generated by Dr. CI and updates every 15 minutes.