[metascroy]

Stack

Off-graph KV cache for the native ggml engine, through serving Muse Glimmer on it. Review bottom-up; each PR builds and passes its tests on its own.

• [Native] Launch the Muse Glimmer server on the native backend #24957
• [Native] Serve Muse Glimmer on the native ggml engine #24956
• [Native] Add a batching executor for native-backend packages #25055
• [Native] Select the logits rows an off-graph export returns #25009
• [Native] Gather the rows of x[:, rows] in the ggml engine #25008
• [Native] Serve many sequences from one ggml engine KV cache #25007 (this PR)
• [Native] Separate the ggml engine's K/V storage from its sequence cache #25006
• [Native] Publish the context length with off-graph packages #24955
• [Native] Share the off-graph cache geometry with other callers #24954
• [Native] Export Muse Glimmer through the native backend #24951
• [Native] Replay sample sets without decoding their tokens #24950
• [Native] Map Muse Glimmer GGUF tensors onto its HF model #24949
• [Native] Run GGUF Q5_K weights in the ggml engine #24948
• [Native] Load the first layers of a GGUF with --layers #24947
• [Native] Run the HF LLM export on newer transformers #24946
• [Native] Fuse broadcast RoPE frequencies and lower tanh in ggml #24945
• [Native] Support sliding-window attention with the off-graph KV cache #24944
• [Native] Carry named metadata in native packages #24943
• [Native] Add chunked prefill and interactive chat to the LLM runner #24639
• [Native] Run off-graph KV cache packages in the LLM runner #24638
• [Native] Serve kvcache::update_and_attend in the ggml engine #24637
• [Native] Add load-time options to engine contexts #24636
• [Native] Export HF LLMs with an off-graph KV cache #24635
• Move off-graph HF attention to extension/llm/cache #24634

Summary

The ggml engine's off-graph cache held one sequence, so a batching runtime couldn't put several sessions in one forward, clone a prefix, or share a pool between sessions.

• Cache: GgmlCellCache is the neutral CellCache over a GgmlKvPool whose rows are its cells, registered for kind::kBatched and kind::kBatchedCell. It offers BatchControl to the control plane.
• Engine: a context accepts either cache through llm_cache_registry_key. With a cell cache, the executor:
  • places the forward's tokens at their positions before building the graph (the control plane declared their sequences);
  • grows storage to the step's read extent;
  • fills each window's cache rows and mask straight from the cache's step: rows are the step's cells, and the mask keeps each query to its own sequence's earlier cells within its window.
• Windows: a windowed layer keeps every cell, so a rewind or clone never needs history the cache dropped.
• Step lifecycle: CellCache gains end_step() and abandon_step(), which the executor calls after each forward. A forward that succeeds ends its step, so running it again without a new declaration is refused rather than re-served. One that fails after placement gives its cells back: each of the step's sequences drops to its first position in the step, so they continue where they were. Other backends don't call these, so they're unaffected.

Test plan

New engine tests against a multi-sequence reference (naive attention per sequence):

• two sequences prefilling separately, then decoding in one forward, with and without graph caching, on fp32 and fp16 storage, starting from one row so storage regrows;
• a clone sharing its source's prefix as both continue, plus rewinds and closing the clone;
• sliding windows, including inside a prefill wider than the window;
• undeclared steps and positions that don't continue a sequence are refused;
• a sliding and a full layer served from the same cells in one two-layer forward;
• a full pool refusing a step until a rewind frees cells;
• a forward failing after placement (a host-side row check) gives its cells back; this fails without the undo;
• a forward that ran can't be replayed without a declaration, so a failing replay can't undo it; this fails without end_step().

cache_test.cpp adds CellEndedStepIsNotReServed and CellAbandonedStepGivesBackItsCells.

All pass on CPU, and on Metal through a temporary local change. Making the executor ignore the cache's mask fails three of the four.

---

Authored with Claude Code.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25007

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 169 New Failures, 7 Unrelated Failures, 27 Unclassified Failures

As of commit 0d8ca8f with merge base c7f7931 ():

NEW FAILURES - The following jobs have failed:

• Build documentation / build (buck2) / Build doc (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Build Presets / apple (ios-simulator) / build (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• Build Presets / apple (ios) / build (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• Build Presets / apple (llm) / build (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• Build Presets / apple (macos) / build (gh)
• Build Presets / apple (profiling) / build (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• Build Presets / apple (pybind) / build (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• Build Presets / linux (linux, mt-l-arm64g4-16-62, executorch-ubuntu-22.04-gcc11-aarch64) / build (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Build Presets / linux (linux, mt-l-x86iavx512-8-64, executorch-ubuntu-22.04-clang12) / build (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Build Presets / linux (llm, mt-l-arm64g4-16-62, executorch-ubuntu-22.04-gcc11-aarch64) / build (gh)
• Build Presets / linux (llm, mt-l-x86iavx512-8-64, executorch-ubuntu-22.04-clang12) / build (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Build Presets / linux (pybind, mt-l-arm64g4-16-62, executorch-ubuntu-22.04-gcc11-aarch64) / build (gh)
• Build Presets / linux (pybind, mt-l-x86iavx512-8-64, executorch-ubuntu-22.04-clang12) / build (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Build Presets / windows (pybind) / build (gh)
• Build Presets / windows (windows) / build (gh)
• Build Presets / zephyr (zephyr) / build (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Cadence Build & Test / cpu-build / build (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Lint / lintrunner (gh)
  >>> Lint for backends/native/test/runtime/ggml/test_ggml_engine.cpp:
• MLX / backend-tester (models) / test-mlx-backend-models (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / backend-tester (operators) / test-mlx-backend-operators (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx / test-mlx (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-dflash / test-mlx-dflash (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (google/gemma-4-E2B-it, gemma4-e2b, false, 4w) / test-mlx-llm-gemma4-e2b-4w (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (google/gemma-4-E2B-it, gemma4-e2b, true, 4w) / test-mlx-llm-gemma4-e2b-custom-4w (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/gemma-3-1b-it, gemma3-1b, false, 4w) / test-mlx-llm-gemma3-1b-4w (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/gemma-3-1b-it, gemma3-1b, false, nvfp4) / test-mlx-llm-gemma3-1b-nvfp4 (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/gemma-3-1b-it, gemma3-1b, true, 4w) / test-mlx-llm-gemma3-1b-custom-4w (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/gemma-3-1b-it, gemma3-1b, true, nvfp4) / test-mlx-llm-gemma3-1b-custom-nvfp4 (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/Llama-3.2-1B-Instruct, llama-1b, false, 4w) / test-mlx-llm-llama-1b-4w (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/Llama-3.2-1B-Instruct, llama-1b, false, nvfp4) / test-mlx-llm-llama-1b-nvfp4 (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/Llama-3.2-1B-Instruct, llama-1b, true, 4w) / test-mlx-llm-llama-1b-custom-4w (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/Llama-3.2-1B-Instruct, llama-1b, true, nvfp4) / test-mlx-llm-llama-1b-custom-nvfp4 (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/Qwen3-0.6B, qwen3-0.6b, false, 4w) / test-mlx-llm-qwen3-0.6b-4w (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/Qwen3-0.6B, qwen3-0.6b, false, nvfp4) / test-mlx-llm-qwen3-0.6b-nvfp4 (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/Qwen3-0.6B, qwen3-0.6b, true, 4w) / test-mlx-llm-qwen3-0.6b-custom-4w (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm (unsloth/Qwen3-0.6B, qwen3-0.6b, true, nvfp4) / test-mlx-llm-qwen3-0.6b-custom-nvfp4 (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm-offgraph (google/gemma-4-E2B-it, gemma4-e2b, gemma4) / test-mlx-llm-offgraph-gemma4-e2b (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm-offgraph (unsloth/gemma-3-1b-it, gemma3-1b, gemma) / test-mlx-llm-offgraph-gemma3-1b (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-llm-offgraph (unsloth/Llama-3.2-1B-Instruct, llama-1b, llama3) / test-mlx-llm-offgraph-llama-1b (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-muse-glimmer-build / test-mlx-muse-glimmer-build (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-nemotron3-diarization-build / test-mlx-nemotron3-diarization-build (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-parakeet / test-mlx-parakeet (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-qwen35-moe / test-mlx-qwen35-moe (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-stories110m / test-mlx-stories110m (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-supertonic / test-mlx-supertonic (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-voxtral / test-mlx-voxtral (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-voxtral-realtime / test-mlx-voxtral-realtime (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• MLX / test-mlx-whisper / test-mlx-whisper (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• pull / nxp-mcuxpresso-test / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-arduino-library / run / arduino-library (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-arm-backend-no-driver (test_pytest_models_tosa) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-arm-backend-no-driver (test_pytest_ops_no_target) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-arm-backend-no-driver (test_pytest_ops_tosa) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-arm-backend-no-driver (test_run_tosa) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-arm-backend-public-api-backward-compatibility / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-arm-backend-vkml (test_ootb_tests_vgf) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-arm-backend-vkml (test_pytest_ops_vkml) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-arm-backend-vkml (test_runtime_vgf) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-arm-cortex-m-size-test (bare_metal) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-arm-cortex-m-size-test (zephyr-preset) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-binary-size-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-binary-size-linux-gcc / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-build-wasm-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-custom-ops-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-eval_llama-wikitext-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-llama_runner_eager-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-llama-runner-linux (bf16, custom, mt-l-x86iavx512-8-64, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-llama-runner-linux (fp32, xnnpack+custom+qe, mt-l-arm64g4-16-62, executorch-ubuntu-22.04-gcc... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-llama-runner-linux (fp32, xnnpack+custom+qe, mt-l-x86iavx512-8-64, executorch-ubuntu-22.04-c... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, mt-l-arm64g4-16-62, executorch-ubuntu-... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, mt-l-x86iavx512-8-64, executorch-ubunt... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-llama-runner-linux (fp32, xnnpack+quantize_kv, mt-l-arm64g4-16-62, executorch-ubuntu-22.04-g... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-llama-runner-linux (fp32, xnnpack+quantize_kv, mt-l-x86iavx512-8-64, executorch-ubuntu-22.04... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-llama-runner-linux-android / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-lora-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-lora-multimethod-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-mcu-cortex-m-backend / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (add_mul, portable, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (add_mul, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (add, portable, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (add, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (emformer_join, portable, mt-l-x86iavx512-16-128) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (emformer_join, xnnpack-quantization-delegation, mt-l-x86iavx512-16-128) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (emformer_transcribe, portable, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (emformer_transcribe, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (ic3, portable, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (ic3, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (ic4, portable, mt-l-x86iavx512-16-128) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (ic4, xnnpack-quantization-delegation, mt-l-x86iavx512-16-128) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (linear, portable, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (linear, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (llama3_2_vision_encoder, portable, mt-l-x86iavx512-16-128) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (mobilebert, portable, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (mobilebert, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (mv2, portable, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (mv2, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (phi_4_mini, portable, mt-l-x86iavx512-16-128) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (resnet18, portable, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (resnet18, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (resnet50, portable, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (resnet50, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux (w2l, portable, mt-l-x86iavx512-16-128) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux-basic (mv3, portable, buck2, mt-l-x86iavx512-8-64, executorch-ubuntu-22.04-clan... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux-basic (mv3, portable, cmake, mt-l-arm64g4-16-62, executorch-ubuntu-22.04-gcc11-... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux-basic (mv3, portable, cmake, mt-l-x86iavx512-8-64, executorch-ubuntu-22.04-clan... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, buck2, mt-l-x86iavx512-8-64, execu... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, cmake, mt-l-arm64g4-16-62, executo... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, cmake, mt-l-x86iavx512-8-64, execu... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux-basic (vit, portable, buck2, mt-l-x86iavx512-8-64, executorch-ubuntu-22.04-clan... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux-basic (vit, portable, cmake, mt-l-arm64g4-16-62, executorch-ubuntu-22.04-gcc11-... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux-basic (vit, portable, cmake, mt-l-x86iavx512-8-64, executorch-ubuntu-22.04-clan... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, buck2, mt-l-x86iavx512-8-64, execu... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, cmake, mt-l-arm64g4-16-62, executo... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, cmake, mt-l-x86iavx512-8-64, execu... / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-moshi-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-multimodal-linux (gemma3-4b) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-openvino-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-parakeet-xnnpack-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-phi-3-mini-runner-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-pybindings-no-aten / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-qnn-buck-build-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-qnn-delegate-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-qnn-direct-build-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-qnn-models-linux (dl3) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-qnn-models-linux (mv2) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-qnn-models-linux (mv3) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-qnn-passes-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-qnn-python-imports-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-qnn-testsuite-linux / test-backend-linux (qnn, models) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-qnn-testsuite-linux / test-backend-linux (qnn, operators) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-qnn-wheel-packages-linux (3.11) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-qnn-wheel-packages-linux (3.12) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-qnn-wheel-packages-linux (3.13) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-quantized-aot-lib-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-selective-build-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-setup-linux-gcc / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-sqnr-static-llm-qnn-linux (smollm2_135m) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-static-llama-qnn-linux (stories_110m) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-static-llama-qnn-linux (stories_260k_bc) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-voxtral-realtime-xnnpack-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-vulkan-models-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / test-vulkan-operators-linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / unittest / linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / unittest / macos / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• pull / unittest-buck / linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / unittest-buck / macos / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• pull / unittest-editable / linux / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / unittest-editable / macos / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• pull / unittest-nxp-neutron / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / unittest-wasm-bindings (--enable-etdump) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / unittest-wasm-bindings / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test ARM Backend / test-arm / test-backend-linux (arm_tosa_fp, models) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test ARM Backend / test-arm / test-backend-linux (arm_tosa_fp, operators) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test ARM Backend / test-arm-vgf / test-backend-linux (arm_vgf_fp, models) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test ARM Backend / test-arm-vgf / test-backend-linux (arm_vgf_fp, operators) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test CPU Backend / test-cpu / test-backend-linux (cpu_fp32, models) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test CPU Backend / test-cpu / test-backend-linux (cpu_fp32, operators) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test QNN Backend / test-qnn / test-backend-linux (qnn, models) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test QNN Backend / test-qnn / test-backend-linux (qnn, operators) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test QNN Windows MSVC build / build-qnn-windows-arm64 (gh)
• Test QNN Windows MSVC build / build-qnn-windows-x64 / windows-job (gh)
• Test Vulkan Backend / test-vulkan / test-backend-linux (vulkan, models) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test Vulkan Backend / test-vulkan / test-backend-linux (vulkan, operators) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test WebGPU Backend / test-webgpu / test-backend-linux (webgpu, models) / linux-job (gh)
  curl: (22) The requested URL returned error:
• Test WebGPU Backend / test-webgpu / test-backend-linux (webgpu, operators) / linux-job (gh)
  curl: (22) The requested URL returned error:
• Test XNNPACK Backend / test-xnnpack / test-backend-linux (xnnpack, models) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test XNNPACK Backend / test-xnnpack / test-backend-linux (xnnpack, operators) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

• Build Aarch64 Linux CUDA Wheels / pytorch/executorch / build-wheel-py3_11-cuda-aarch6413_2-aarch64 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• Build Aarch64 Linux CUDA Wheels / pytorch/executorch / upload / upload-wheel-py3_11-cuda-aarch6413_2-aarch64 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build Aarch64 Linux Native Wheels / executorch-native / build-wheel-py3_11-cpu-aarch64 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  ModuleNotFoundError: No module named 'setuptools'
• Build Aarch64 Linux Native Wheels / executorch-native / upload / upload-wheel-py3_11-cpu-aarch64 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build Aarch64 Linux Wheels / pytorch/executorch / build-wheel-py3_11-cpu-aarch64 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• Build Aarch64 Linux Wheels / pytorch/executorch / upload / upload-wheel-py3_11-cpu-aarch64 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build Linux CUDA Wheels / pytorch/executorch / build-manywheel-py3_11-cuda13_2 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• Build Linux CUDA Wheels / pytorch/executorch / upload / upload-manywheel-py3_11-cuda13_2 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build Linux Native Wheels / executorch-native / build-manywheel-py3_11-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• Build Linux Native Wheels / executorch-native / upload / upload-manywheel-py3_11-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build Linux Wheels / pytorch/executorch / build-manywheel-py3_11-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• Build Linux Wheels / pytorch/executorch / upload / upload-manywheel-py3_11-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build macOS Wheels / pytorch/executorch / build-wheel-py3_11-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• Build macOS Wheels / pytorch/executorch / upload / upload-wheel-py3_11-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build Windows CUDA Wheels / bundle-cuda-windows-wheel-e2e-13.2 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build Windows CUDA Wheels / export-cuda-windows-wheel-program-13.2 / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Build Windows CUDA Wheels / pytorch/executorch / build-wheel-py3_11-cuda13_2 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• Build Windows CUDA Wheels / pytorch/executorch / upload / upload-wheel-py3_11-cuda13_2 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build Windows CUDA Wheels / test-cuda-windows-wheel-e2e-13.2 / windows-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build Windows Wheels / pytorch/executorch / build-wheel-py3_11-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• Build Windows Wheels / pytorch/executorch / upload / upload-wheel-py3_11-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• periodic / test-models-linux (buck2, mv3, portable, mt-l-x86iavx512-8-64, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• periodic / test-models-linux (buck2, mv3, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• periodic / test-models-linux (cmake, mv3, portable, mt-l-x86iavx512-8-64, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• periodic / test-models-linux (cmake, mv3, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• periodic / test-models-linux (cmake, vit, portable, mt-l-x86iavx512-8-64, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• periodic / test-models-linux (cmake, vit, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Cadence Build & Test / hifi-build / hifi4 (gh) (detected as infra flaky with no log or failing log classifier)
• Cadence Build & Test / hifi1s-build / hifi1s (gh) (detected as infra flaky with no log or failing log classifier)
• Cadence Build & Test / hifi5s-build / hifi5s (gh) (detected as infra flaky with no log or failing log classifier)
• Cadence Build & Test / vision-build / vision (gh) (detected as infra flaky with no log or failing log classifier)
• pull / unittest / windows / windows-job (gh) (detected as infra flaky with no log or failing log classifier)
• pull / unittest-editable / windows / windows-job (gh) (detected as infra flaky with no log or failing log classifier)
• Test Native Engines / native-ggml (gh) (detected as infra flaky with no log or failing log classifier)

This comment was automatically generated by Dr. CI and updates every 15 minutes.