Use hosted NCU profiling by default without provider setup

.gitignore

@@ -6,3 +6,7 @@ scratch.md
 .codex/
 .popcorn/
 .DS_Store
+
+popcorn-profile-*/
+validation/
+__pycache__/

README.md

@@ -7,39 +7,38 @@ Tested on linux and mac but should just work on Windows as well.
 
 ## New: Nsight Compute Profiling
 
-Profile submissions on the hosted GPU Mode B200 Nsight Compute service.
+Profile submissions through GPU Mode (B200 by default).
+Use your normal Popcorn registration. `--profile` and `--mode profile` submit
+through GPU Mode and save NCU reports locally; no provider account or SDK is
+needed. Use `--profile-brev` to explicitly select the Brev service.
 See [docs/profiling.md](docs/profiling.md) for a complete copy-paste flow.
 
 Quick QR v2 example:
 
 ```bash
 curl -O https://raw.githubusercontent.com/gpu-mode/reference-kernels/main/problems/linalg/qr_v2/submission.py
-export POPCORN_BREV_PROFILER_URL=https://http--brev-profiler-proxy--dxfjds728w5v.code.run
-popcorn submit submission.py --leaderboard qr_v2 --profile-brev --benchmark-index 0 --no-tui
+popcorn submit submission.py --leaderboard qr_v2 --profile --benchmark-index 0 --no-tui
 ```
 
 Quick `eigh` dense row example:
 
 ```bash
 curl -O https://raw.githubusercontent.com/gpu-mode/reference-kernels/main/problems/linalg/eigh_py/submission.py
-export POPCORN_BREV_PROFILER_URL=https://http--brev-profiler-proxy--dxfjds728w5v.code.run
-popcorn submit submission.py --leaderboard eigh --profile-brev --benchmark-index 3 --no-tui
+popcorn submit submission.py --leaderboard eigh --profile --benchmark-index 3 --no-tui
 ```
 
 Quick `cholesky` example:
 
 ```bash
 curl -O https://raw.githubusercontent.com/gpu-mode/reference-kernels/main/problems/linalg/cholesky_py/submission.py
-export POPCORN_BREV_PROFILER_URL=https://http--brev-profiler-proxy--dxfjds728w5v.code.run
-popcorn submit submission.py --leaderboard cholesky --profile-brev --benchmark-index 0 --no-tui
+popcorn submit submission.py --leaderboard cholesky --profile --benchmark-index 0 --no-tui
 ```
 
 The CLI downloads and extracts `ncu-details.txt` and `ncu-details.csv` for
-agent-readable analysis. It also extracts the optional `.ncu-rep` GUI report and
-ends with a macOS command that opens it in Nsight Compute:
+agent-readable analysis. It also extracts the optional `.ncu-rep` GUI report which you can open in Nsight Compute:
 
 ```bash
-open -a "NVIDIA Nsight Compute" profile.0-.../profile.ncu-rep
+open -a "NVIDIA Nsight Compute" popcorn-profile-<run>/profile-0/profile.ncu-rep
 ```
 
 ## [NEW] Submit To The Linear Algebra Competition
@@ -139,14 +138,14 @@ popcorn submit solution.py
 # Direct submission with all options
 popcorn submit --leaderboard grayscale_v2 --gpu A100 --mode leaderboard solution.py
 
-# Nsight Compute profile on the hosted GPU Mode B200 profiler
-POPCORN_BREV_PROFILER_URL=https://http--brev-profiler-proxy--dxfjds728w5v.code.run popcorn submit --leaderboard qr_v2 --profile-brev solution.py
+# Nsight Compute profile through GPU Mode (B200 by default)
+popcorn submit --leaderboard qr_v2 --profile solution.py
 
 # Profile one QR v2 benchmark shape
-POPCORN_BREV_PROFILER_URL=https://http--brev-profiler-proxy--dxfjds728w5v.code.run popcorn submit --leaderboard qr_v2 --profile-brev --benchmark-index 0 solution.py
+popcorn submit --leaderboard qr_v2 --profile --benchmark-index 0 solution.py
 
 # Profile one eigh benchmark shape
-POPCORN_BREV_PROFILER_URL=https://http--brev-profiler-proxy--dxfjds728w5v.code.run popcorn submit --leaderboard eigh --profile-brev --benchmark-index 3 solution.py
+popcorn submit --leaderboard eigh --profile --benchmark-index 3 solution.py
 
 # Plain output mode (no TUI, good for CI/scripts)
 popcorn submit --no-tui --leaderboard grayscale_v2 --gpu A100 --mode test solution.py
@@ -200,7 +199,7 @@ if either lookup fails, it stops instead of risking a stale cached image.
 - `test` - Quick test run to check correctness
 - `benchmark` - Benchmark your solution (no leaderboard impact)
 - `leaderboard` - Official ranked submission
-- `profile` - Profile with Nsight Compute (limited availability)
+- `profile` - Profile with Nsight Compute through GPU Mode (default GPU: B200)
 
 ### Submissions
 

docs/linalg-qr-b200.md

@@ -22,8 +22,8 @@ popcorn submit --leaderboard qr_v2 --gpu B200 --mode test submission.py
 Profile the first benchmark shape with Nsight Compute:
 
 ```bash
-export POPCORN_BREV_PROFILER_URL=https://http--brev-profiler-proxy--dxfjds728w5v.code.run
-popcorn submit --leaderboard qr_v2 --profile-brev --benchmark-index 0 submission.py
+popcorn register discord
+popcorn submit --leaderboard qr_v2 --profile --benchmark-index 0 submission.py
 ```
 
 The CLI downloads a `.zip` and extracts `ncu-details.txt`, `ncu-details.csv`,

docs/modal-ncu-validation.md

@@ -0,0 +1,45 @@
+# Hosted NCU profiling validation
+
+`--profile` uses Popcorn authentication and the KernelBot API. KernelBot owns
+Modal credentials and executes NCU; the client only uploads code and saves
+returned artifacts. `--profile-brev` remains explicit. A capture error never
+switches providers.
+
+The CLI calls `POST /profile/{leaderboard}/{gpu}` with multipart capture options.
+This new endpoint requires the companion KernelBot update. Deploy the API and
+GPU runner before releasing the CLI; older services do not support this route.
+The existing `--local` evaluation workflow is independent of hosted profiling.
+
+Validation covers:
+
+- Rust tests for authenticated HTTP submission, multipart options, SSE errors,
+  artifact extraction, and malformed/missing captures.
+- KernelBot tests for authenticated API requests, option validation, benchmark
+  selection, child-process capture, and report exports.
+- An isolated integration run exercised the real FastAPI route, submission
+  preparation, KernelBackend, GPU `run_config`, and CLI artifact extraction,
+  with a fake database and an ephemeral B200 launcher. The CLI used normal
+  Popcorn header authentication with an empty `PATH` and no `MODAL_*` values.
+  It returned a 7,593,257-byte report plus text/CSV exports.
+- NCU 2025.2.1 captured 39 passes. The exported report contained 322.78 us
+  duration, 12.51% achieved occupancy, 0.42% SM throughput, 70.09% L2 hit rate,
+  and 1,565,428 executed instructions. Six `ctc__*` metrics were unavailable.
+- NCU 2026.2.0 from CUDA 13.3 produced many NaN counters on this Modal B200.
+  The server image therefore pins the tested 2025.2.1 version. The earlier
+  2026.2 report was evidence of artifact delivery, not healthy counter coverage.
+
+GPU fixture provenance: `problems/linalg/qr_v2`, benchmark 0
+(`batch=20, n=32, cond=1, seed=43214`), reference-kernels
+`51e22db671d36c1c76091c43c36a44546ba324a1` (the subsequent guide commit changes
+only documentation). The initial KernelBot baseline was
+`30ba5ce79107e5405b0cc1eda48ca551e7a51b16`.
+
+Only QR v2 has been tested end to end for this change. Other problems need an
+NCU-compatible evaluator; accepting `profile` with only a PyTorch profiler path
+is insufficient. Single-GPU NVIDIA capture is supported. Some requested metrics
+can be unavailable on a particular GPU.
+
+The tested image used CUDA 13.3 and PyTorch 2.12.0+cu130, with
+`regex:geqr2`, demangled kernel names, and launch count 1. The isolated test
+is not a production deployment. [Operator validation run](https://modal.com/apps/coreauto/main/ap-p8eodYfHkZAMp0ZT5btR1y)
+requires access to the operator's workspace.

docs/profiling.md

@@ -1,30 +1,51 @@
 # Nsight Compute Profiling
 
-This profiles GPU Mode submissions on the hosted B200 Nsight Compute service and
+The default profiler submits through GPU Mode on B200 and
 downloads agent-readable `ncu-details.txt` / `ncu-details.csv` artifacts. The
 full `.ncu-rep` GUI report is still included for local inspection.
 
-The profiler uses the `benchmarks:` list from the active `reference-kernels`
-checkout. `--benchmark-index N` profiles `benchmarks[N]`; omitting
+The profiler uses the `benchmarks:` list from the task synced into the hosted
+leaderboard configuration. `--benchmark-index N` profiles `benchmarks[N]`; omitting
 `--benchmark-index` profiles every benchmark entry for that leaderboard.
 
-## 1. Install and Register
+## Supported problems
+
+The task evaluator must implement `profile` mode and launch the submission in
+an NVTX push/pop range named `custom_kernel`. A PyTorch-profiler-only path is
+not sufficient. QR, QR v2, Eigh, Cholesky, and the shared NVIDIA evaluator have
+the required NCU path; only QR v2 has been tested end to end for this CLI change.
+Check the actual evaluator selected by the task, since task-specific copies may
+have different support. AMD and multi-GPU NCU profiling are unsupported.
+
+Problem authors can follow the reference-kernels
+[NCU integration guide](https://github.com/gpu-mode/reference-kernels/blob/a8044f1658acd4104558bedd3e78a8f096fd778a/docs/ncu-profiling.md).
+
+## 1. Install and register
 
 ```bash
 curl -fsSL https://raw.githubusercontent.com/gpu-mode/popcorn-cli/main/install.sh | bash
 popcorn register discord
 ```
 
-Restart your terminal if `popcorn` is not found after installation.
+`--profile` and `--mode profile` use the normal authenticated GPU Mode API and
+imply plain output. No Modal installation, provider account, provider token,
+or profiling URL is needed. The service owns the compute credentials. B200 is
+the default GPU; use `--gpu` or a submission GPU directive to select another
+supported GPU. The existing `--local` evaluation option is a separate workflow
+for users deliberately choosing their own compute account.
 
-## 2. Set the Hosted Profiler URL
+## 2. Benchmark selection and artifacts
 
-```bash
-export POPCORN_BREV_PROFILER_URL=https://http--brev-profiler-proxy--dxfjds728w5v.code.run
-```
+The hosted service uses the task/evaluator already synced to that leaderboard.
+Local reference-kernels edits and `POPCORN_REFERENCE_KERNELS_REF` do not change
+a hosted run. Problem authors should ask an operator to sync their published
+revision before validating it.
 
-`BREV_PROFILER_URL` is also accepted as a fallback, but
-`POPCORN_BREV_PROFILER_URL` is preferred.
+Each profile saves a `manifest.json` containing the leaderboard, GPU/system
+information, selected benchmark specs, capture options, and an evaluation-config
+SHA-256 digest. Reports go into a unique `popcorn-profile-*` directory.
+`--output` saves the text summary; artifacts remain in that directory. The
+config digest identifies the evaluated content; it is not a repository commit.
 
 ## 3. Profile QR v2
 
@@ -41,7 +62,7 @@ Profile one benchmark shape:
 ```bash
 popcorn submit submission.py \
   --leaderboard qr_v2 \
-  --profile-brev \
+  --profile \
   --benchmark-index 0 \
   --no-tui
 ```
@@ -67,14 +88,13 @@ Profile the dense `n=512` leverage row:
 ```bash
 popcorn submit submission.py \
   --leaderboard eigh \
-  --profile-brev \
+  --profile \
   --benchmark-index 3 \
   --no-tui
 ```
 
-The hosted profiler uses a deeper Nsight Compute launch window for `eigh` than
-for QR v2 so PyTorch/cuSOLVER submissions can reach solver-path kernels after
-clone/setup launches.
+The default capture window is 10 kernel launches per benchmark. For late solver
+kernels, select a kernel name or increase `--ncu-launch-count`.
 
 Current `eigh` benchmark index table from `reference-kernels` main
 `4a1153e`:
@@ -110,31 +130,41 @@ Profile the `batch=4096, n=32` benchmark:
 ```bash
 popcorn submit submission.py \
   --leaderboard cholesky \
-  --profile-brev \
+  --profile \
   --benchmark-index 0 \
   --no-tui
 ```
 
-The profiler will only profile the first 10 kernels, so if your solution uses more than that (the default Pytorch implementation does), then the profiler might be of limited use.
+Capture late kernels with the same filters on any leaderboard:
+
+```bash
+popcorn submit submission.py --leaderboard cholesky --profile --benchmark-index 0 \
+  --ncu-kernel-name 'regex:my_kernel' --ncu-kernel-name-base demangled \
+  --ncu-launch-count 2
+```
+
+NCU follows child processes, captures the evaluator's `custom_kernel` NVTX range,
+and leaves GPU clocks unchanged. Empty or failed captures return an error.
+Multi-GPU profiling is unsupported.
 
 ## 6. Read the Details
 
 After the run finishes, the CLI downloads and extracts files like:
 
 ```text
-profile.0-batch-20-n-32-cond-1-seed-43214.zip
-profile.0-batch-20-n-32-cond-1-seed-43214/ncu-details.txt
-profile.0-batch-20-n-32-cond-1-seed-43214/ncu-details.csv
-profile.0-batch-20-n-32-cond-1-seed-43214/profile.ncu-rep   # optional GUI report
+popcorn-profile-<run>/result-0/profile-0.zip
+popcorn-profile-<run>/result-0/profile-0/ncu-details.txt
+popcorn-profile-<run>/result-0/profile-0/ncu-details.csv
+popcorn-profile-<run>/result-0/profile-0/profile.ncu-rep   # optional GUI report
 ```
 
 Use `ncu-details.txt` or `ncu-details.csv` as the default artifact for AI
-analysis. The CLI prints clickable links for these detail files.
+analysis. The CLI prints local paths for the detail files and report.
 
-The last line printed by the CLI opens the optional GUI report on macOS:
+Open the GUI report on macOS:
 
 ```bash
-open -a "NVIDIA Nsight Compute" 'profile.0-batch-20-n-32-cond-1-seed-43214/profile.ncu-rep'
+open -a "NVIDIA Nsight Compute" 'popcorn-profile-<run>/result-0/profile-0/profile.ncu-rep'
 ```
 
 ## Profile All Benchmark Shapes
@@ -144,7 +174,7 @@ Omit `--benchmark-index`:
 ```bash
 popcorn submit submission.py \
   --leaderboard eigh \
-  --profile-brev \
+  --profile \
   --no-tui
 ```
 
@@ -165,3 +195,18 @@ For leaderboard submission:
 ```bash
 popcorn submit submission.py --leaderboard qr_v2 --gpu B200 --mode leaderboard --no-tui
 ```
+
+## Explicit Brev profiling
+
+Use `--profile-brev` to select the hosted Brev service explicitly. `--profile`
+uses the GPU Mode API; errors never trigger a switch to Brev. Brev requires Popcorn
+registration:
+
+```bash
+popcorn register discord
+export POPCORN_BREV_PROFILER_URL=https://http--brev-profiler-proxy--dxfjds728w5v.code.run
+popcorn submit submission.py --leaderboard qr_v2 --profile-brev --benchmark-index 0
+```
+
+`BREV_PROFILER_URL` is also accepted. Brev profiling uses that service's deployed
+reference-kernels checkout.