Skip to content

fix(pc-sampling): count _not_issued stall samples separately - #166

Merged
CodingInAVan merged 1 commit into
mainfrom
pc-sampling-not-issued-shares
Sep 30, 2026
Merged

CodingInAVan merged 1 commit into
mainfrom
pc-sampling-not-issued-shares

Conversation

@CodingInAVan

Copy link
Copy Markdown
Contributor

Follow-up to #164. This commit was reviewed in #165, but #165 targeted pc-sampling-sampled-launch-threshold after #164 had already merged it into main, so the change never reached main and CI never ran on it.

Problem

CUPTI's PC Sampling API reports every sample under its warp state (smsp__pcsamp_warps_issue_stalled_<state>) and, when the scheduler issued no instruction that cycle, again under <state>_not_issued. The _not_issued counts are a subset of the state's counts (Nsight Compute Profiling Guide, sections 2.4.7 and 2.4.8). The analyzer, both text reports and the collect-summary debug log summed the two, so each stall share came out at about half its value and per-kernel sample totals were about 1.9x CUPTI's count.

The analyzer also labelled PC Sampling API rows through the Activity API index table, so long_scoreboard (index 12 on an RTX 5060) appeared as "Barrier" and its _not_issued twin as "Sleeping".

Change

  • Stall shares use the warp states only (selected and not_selected included). These add up to totalSamples - nonUsrKernelsTotalSamples - droppedSamples.
  • _not_issued samples are a separate breakdown with their own total: extra columns in inspect_profile_samples and the Python report, a "Not issued" block in the C++ report, and inspect_stalls(not_issued=True).
  • Reasons are classified by the CUPTI name, not the index. smsp__pcsamp_sample_count and smsp__pcsamp_samples_data_dropped are not warp states and are left out.
  • The Activity API index table now matches CUpti_ActivityPCSamplingStallReason.
  • The collect-summary debug line prints N samples (+M not issued), so N lines up with CUPTI's counters.

Evidence

Four vector-add PcSampling captures on an RTX 5060 (CUDA 13.3), 200 to 6,000 launches:

  • The sum over warp states equals totalSamples - nonUsrKernelsTotalSamples - droppedSamples exactly in all four runs.
  • No (launch, PC, state) cell has more _not_issued samples than state samples.
  • Summing all reasons gives 1.90x to 1.93x that total.

On the 1,000-launch capture the analyzer's reason table showed "Barrier 49.2%" and "Sleeping 44.6%" before this change and shows long_scoreboard 93.4% after it.

Tests

  • New: tests/python/test_pc_stall_shares.py (5 tests) and TextReportTest.PcStallSharesExcludeNotIssuedSamples. All six fail on cf2e392 and pass with this change.
  • Python: 15 passed (new tests plus test_analyzer.py).
  • C++: TextReportTest 7/7 on a CUDA 13.3 build; full gpufl_tests 609/609 on a no-CUDA build.
  • GPU: EngineCoverageTest.EmitsExpectedEvents/PcSampling passes on an RTX 5060. Its collect summary reads 195316 samples (+132816 not issued) with totalSamples=195663 dropped=0 nonUsrKernels=347 (195,663 - 347 = 195,316).

🤖 Generated with Claude Code

CUPTI's PC Sampling API counts every sample under its warp state and,
when the scheduler issued nothing that cycle, again under the state's
_not_issued twin. The analyzer, both text reports and the collect log
summed the two, so each stall share came out at about half its value
(long_scoreboard at 93% showed as 49% plus 46%) and per-kernel sample
totals were about 1.9x CUPTI's count.

Shares now use the warp states only, which add up to totalSamples less
non-user-kernel and dropped samples. The _not_issued samples are shown
as a separate breakdown with their own total, and
inspect_stalls(not_issued=True) shows that view per kernel.

The analyzer also labelled PC Sampling API rows through the Activity API
index table, so long_scoreboard appeared as "Barrier". Rows now use the
CUPTI reason name, and the index table, still used for Activity API
rows, matches CUpti_ActivityPCSamplingStallReason.
@CodingInAVan
CodingInAVan merged commit d5429ca into main Sep 30, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant