Skip to content

Use hosted NCU profiling by default without provider setup - #80

Merged
msaroufim merged 3 commits into
mainfrom
feat/modal-ncu-default
Sep 18, 2026
Merged

msaroufim merged 3 commits into
mainfrom
feat/modal-ncu-default

Conversation

@msaroufim

@msaroufim msaroufim commented Sep 17, 2026 •

Copy link
Copy Markdown
Member

popcorn submit submission.py --profile now submits through the authenticated GPU Mode API and downloads NCU reports. Users need only normal Popcorn registration: no Modal SDK, provider account, credentials, or profiling URL. B200 is the default when no GPU flag or directive is supplied.

--profile-brev explicitly selects the existing Brev service. Errors never switch providers, and --local evaluation remains separate. Both profiling paths accept benchmark selection and kernel-name/launch-count filters. Hosted results include .ncu-rep, text/CSV detail exports, and a manifest with selected benchmark specs, capture options, system information, and a configuration fingerprint. Failed or empty captures return errors.

Deployment dependency: merge and deploy gpu-mode/kernelbot#530 (API and GPU runners) before releasing this CLI. It adds the /profile/{leaderboard}/{gpu} endpoint and pins the working server-side NCU version. The new-problem guide is gpu-mode/reference-kernels#172. No production deployment is included here.

Validation:

  • 59 Rust tests passed; cargo fmt --check, cargo clippy --all-targets -- -D warnings, and release build passed.
  • Authenticated HTTP test verifies the Popcorn header and multipart capture options, with SSE errors propagated.
  • Isolated GPU integration exercised the real API/backend/runner with a fake database and ephemeral B200 launcher; the release CLI ran with an empty PATH and no MODAL_* values, then saved and extracted a 7,593,257-byte report and both exports.
  • Fixture: problems/linalg/qr_v2, benchmark 0 (batch=20, n=32, cond=1, seed=43214), reference-kernels 51e22db671d36c1c76091c43c36a44546ba324a1, regex:geqr2, demangled names, launch count 1. NCU 2025.2.1 collected 39 passes and valid occupancy, throughput, cache, and instruction counters. Six ctc__ metrics were unavailable. Other tasks require NCU-compatible evaluators and their own GPU validation.

docs/modal-ncu-validation.md records the integration boundary and the NaN-counter issue observed with NCU 2026.2, which motivated the backend's version pin.

@msaroufim msaroufim changed the title Add Modal NCU profiling with explicit --profile and --profile-brev Use hosted NCU profiling by default without provider setup Sep 18, 2026
@msaroufim
msaroufim merged commit 036b3ea into main Sep 18, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant