Repository navigation
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24889
Note: Links to docs will display an error until the docs builds have been completed. ❌ 4 New Failures, 1 Pending, 2 Unrelated Failures, 5 Unclassified FailuresAs of commit b8f1ad5 with merge base 7b4cc86 ( NEW FAILURES - The following jobs have failed:
UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:
BROKEN TRUNK - The following jobs failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
This PR needs a
|
CudaWeightCollector skipped a later method's weight storage only when it compared equal, on the device, to the storage the first method wrote. Every method's preprocess ends with release_moved_tensors, which frees those GPU copies, so by the next method the remembered storage is empty, the size check fails, and the same weights are copied to the host, hashed and written again for every method (about 4 GB per method for a 5-layer Muse Glimmer). Record which CPU tensor each tensor was moved from when the program is moved to the GPU, and remember the source view each storage key was written from. A later method's copy of the same source view is reused without being read or written. The source storage is held weakly, so nothing matches once it is gone, and any other storage still goes through the existing on-device comparison and content-hash check.
…-only materialize, buck dep
After every CUDA-graph timing capture, _release_cached_memory_if_large called torch.cuda.memory_reserved() and torch.cuda.memory_allocated(). Each flattens the whole allocator stat tree into a Python dict, and that tree keeps one entry per CUDA-graph private pool ever captured, so the check gets slower with every capture of the export. Profiling a 5-layer Muse Glimmer batching export put about 80% of the candidate-benchmark time of forward_8..forward_64 (214 s of 267 s for forward_64) and nearly all of draft_forward's in this check. Read reserved_bytes.all.current and allocated_bytes.all.current from one nested stats call. The cached-bytes value and the empty_cache threshold are unchanged, so the timed candidates and the picks are the same; a probe with 2048 captured pools gave the same bytes about 20x faster.
Use the tightest visible cgroup CPU quota together with process affinity and host CPU count when sizing the ExecuTorch quantized GEMM compiler pool. Handle both cgroup versions and retain the existing 32-worker maximum on unrestricted hosts. CPU-only regression tests cover quota, affinity, namespace mount roots, parent limits and fallback behavior. Authored with OpenAI Codex.
Rename backends/cuda/_cpu_count.py to backends/cuda/utils/cpu_count.py and split cgroup parsing into small helpers so flake8 C901 passes. Behavior is unchanged.
SaturatedCpu started one spinner per host CPU. Under a cgroup quota that oversubscribes the quota now that the quantized GEMM compile pool is sized to the usable CPUs, which starved the pool and pushed the int4 under-load test past its job timeout. Spawn one spinner per usable CPU instead.
Every recent kquant-17gb dflash-image-batching run, on main and on pull requests, was cancelled at the 240 minute limit while export_dflash was still running (more than 3.5 hours into the export). Raise that matrix entry to 480 minutes and keep the two kquant-dynamic entries at 240.
test-muse-glimmer-cuda-e2e runs three A100 jobs, and the kquant-17gb dflash-image-batching export alone takes about five hours, so running it on every pull request push costs many A100 hours per iteration. Gate the job on pushes to main; pull requests, ciflow tags and manual dispatch no longer start it. Pull requests keep the unittest-cuda-* jobs, including unittest-cuda-muse-glimmer-batching. A CI script test pins the gate.
…hing e2e
Each forward_{N} is its own AOTI compile and autotune. On the 11-vCPU A100
runner the full 1..64 set does not fit 240 minutes (#25018 still timed out
with this alone, so the 480-minute timeout stays); 1,4,16,64 exercises the
same CUDA executor padding path with three fewer compiles.
Export forward_{1,4} and a single draft owner in CI, and drop the job's
480-minute timeout back to the shared 240. Even with the CPU-quota-aware
compile pool and forward_{1,4,16,64}, the export was still compiling the
draft methods at the 240-minute mark (#25018); forward_16/64 and
draft_forward_{2,3,4} account for about two of its hours. The run still
covers static and dynamic target steps, vision, and DFlash drafting on the
batching runtime.
…bserve the batching job
Gasoonjia
force-pushed
the
fix/muse-cgroup-compile-workers
branch
from
November 14, 2026 20:04
78b04f7 to
b8f1ad5
Compare
Contributor
Author
|
Rebased onto #25060 (stacked on #25059) to measure the kquant-17gb batching e2e with all three changes together. The top commit |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Quantized GEMM compilation sizes its worker pool using os.cpu_count(), which
continues to report host CPUs when a container limits CPU time with a cgroup
quota. Size the pool from the minimum of host CPU count, process affinity and
the tightest visible CPU quota, retaining the existing maximum of 32 workers.
The CPU-only helper handles cgroup v1 and v2, parent limits, mount roots and
cgroup namespace roots. Missing or unreadable quota information falls back to
the existing CPU count and affinity behavior. Fractional CPU quotas round up to
a positive worker count. The helper and its tests are included in Buck rules.
CI triggering and the four-hour Muse E2E timeout remain unchanged for the
initial CI verification.
Validation
14 CPU-only regression tests pass. Black and git diff --check pass.
Runtime probes on the same host using the patched _compile_workers():
Full Muse CI validation is pending PR creation. These probes validate worker
sizing; they do not establish end-to-end export performance with the fix.
Authored with OpenAI Codex.