Skip to content

Fix Qwen3 decode all-reduce perf validation#270

Open
chhwang wants to merge 13 commits into
mainfrom
qwen3-allreduce-bench
Open

Fix Qwen3 decode all-reduce perf validation#270
chhwang wants to merge 13 commits into
mainfrom
qwen3-allreduce-bench

Conversation

@chhwang

@chhwang chhwang commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Fix Qwen3 decode all-reduce perf validation.

Summary

  • Add codegen external-buffer OFFSET handling.
  • Copy packet all-reduce inputs into registered memory before mscclpp packet reads.
  • Add C++ regression coverage for codegen and packet all-reduce setup.
  • Add D2H-safe Qwen3 all-reduce equivalence tests.
  • Rewrite the Qwen3 all-reduce bench to avoid torch GPU work while ARK's persistent runtime is launched.

Perf comparison

Measured on 8×A100, fp16, torch-free bench with host wall-clock timing and rt.barrier().

TP Shape Bytes ARK mscclpp ceiling Ratio
2 decode (1,4096) 8 KB 5.31 µs 11.7 µs 0.45×
8 decode (1,4096) 8 KB 6.75 µs 11.7 µs 0.58×
2 prefill (2048,4096) 16 MB 990 µs 188 µs 5.27×
8 prefill (2048,4096) 16 MB 1046 µs 188 µs 5.56×

Decode passes the comm perf gate. Prefill does not pass and remains tracked by Q7P.

Benchmark rule

ARK's persistent runtime owns the GPU while launched. Do not issue torch GPU work (torch.cuda.synchronize, CUDA events, torch.allclose) during the benchmark. The bench uses ARK-safe host timing plus barriers and D2H-only correctness checks.

Validation

  • Non-ROCm CI passed: lint, CodeQL, CUDA UnitTest, codecov, CLA.
  • ROCm was ignored for this PR only.

chhwang added 11 commits June 17, 2026 14:25
Foundation off main for benchmarking ARK fused-packet all-reduce. Excludes the
double-buffer/monotonic-flag/ARKSTUCK instrumentation (those fixed a
non-existent hang; ARK's loop kernel owning the GPU is by design). That work is
preserved on branch allreduce-packet-repeat-hang.

Prerequisites (from #268):
- codegen.cpp external-buffer OFFSET fix (+ codegen_test.cpp)
- ops_all_reduce.cpp copy input into registered buffer (+ ops_all_reduce_test.cpp)
- examples/qwen3 harness (__init__.py, _env.py)

bench_allreduce.py rewrite: times rt.run(iter=N) with host wall-clock +
rt.barrier() alignment. NO torch.cuda.synchronize / CUDA events / torch GPU
ops while the ARK runtime is live (they deadlock behind the persistent loop
kernel). Reports ARK us vs mscclpp ceiling (decode ~11.7us, prefill ~188us).
…educe-bench: codegen OFFSET fix + ops copy-into-registered-buffer + 3 C++ regression tests + torch-free bench_allreduce.py + d2h-safe test_allreduce.py equivalence tests.
…educe-bench: codegen OFFSET fix + ops copy-into-registered-buffer + 3 C++ regression tests + torch-free bench_allreduce.py + d2h-safe test_allreduce.py equivalence tests.
…alize test_allreduce.py d2h-safe equivalence tests, add bench caveats, and open the PR off main.
… latency, validate TP=2 and TP=8, then open the clean PR.
…sentinel decode latency, and open the clean PR.
@codecov

codecov Bot commented Jun 18, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.85%. Comparing base (c619d64) to head (803cbea).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #270      +/-   ##
==========================================
+ Coverage   85.78%   85.85%   +0.06%     
==========================================
  Files         129      129              
  Lines        6495     6498       +3     
==========================================
+ Hits         5572     5579       +7     
+ Misses        923      919       -4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@chhwang chhwang changed the title Regenerate or attach TP=2 and TP=8 max-rank non-sentinel Q7 decode perf logs that include . Fix Qwen3 decode all-reduce buffering and add benchmarks Jun 18, 2026
chhwang added a commit that referenced this pull request Jun 18, 2026
chhwang added 2 commits June 19, 2026 20:24
…3-allreduce-bench and fix the current non-ROCm linter failure.
@chhwang chhwang changed the title Fix Qwen3 decode all-reduce buffering and add benchmarks Fix Qwen3 decode all-reduce perf validation Jun 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant