[platelett]

Every MFC build with NVHPC 26.9, CPU or GPU, fails at its first link:

/usr/bin/ld: cannot find -lnvhpcwrapnvtx: No such file or directory
gmake[3]: *** [CMakeFiles/syscheck.dir/build.make:100: syscheck] Error 2

Once it links, 26.9 computes wrong hypoelastic results, because of a compiler bug. This PR fixes the link and works around the bug:

• Link. NVHPC 26.9 renamed its NVTX wrapper library to libnvhpcnvtx, and MFCTargets.cmake names the old one. This PR stops naming it: the -cudalib=nvtx3 flag already on that line links the wrapper under each release's own name. Checked on every NVHPC release in CI (25.5 to 26.3) plus 26.9.
• Hypoelasticity. At -O2 and above, 26.9 evaluates any(i == shear_indices) inside a loop as always true, so normal stresses get the shear (doubled) elastic energy. The five such comparisons are now written out as three == tests, which 26.9 compiles correctly.
• CI. Adds 26.9 CPU and GPU jobs, the same as for the other releases, and makes the CI rules name the cmake/ directory.

Change

cmake/MFCTargets.cmake. For CUDA ≥ 12.9 (no CUDA::nvToolsExt), the targets linked the wrapper twice:

target_link_libraries(${a_target} PRIVATE nvhpcwrapnvtx)
target_link_options(${a_target} PRIVATE "-cudalib=nvtx3")

• The first line is removed; the driver adds the wrapper itself.
• LINKER_LANGUAGE Fortran is set on the targets in that branch. Only nvfortran understands -cudalib, so this keeps it driving the link if C or C++ sources are ever added. MFC has none today, so nothing changes now.
• The CUDA ≤ 12.8 branch (CUDA::nvToolsExt) is unchanged. It has to stay: on the releases that take it, -cudalib=nvtx3 alone does not link (table below).

src/common/include/macros.fpp, m_variables_conversion.fpp, m_riemann_state.fpp. A new IS_SHEAR(idx) Fypp macro spells out the test as idx == shear_indices(1) .or. idx == shear_indices(2) .or. idx == shear_indices(3). It replaces any(... == shear_indices) at all five sites. The two forms mean the same thing; only the code 26.9 generates differs. The compiler bug and a reproducer are below.

.github/workflows/test.yml. Adds { nvhpc: '26.9', target: cpu } and { nvhpc: '26.9', target: gpu }. They run the same steps as the other releases: the CPU job runs the full suite, and the GPU job builds, including the case-optimized build.

.github/file-filter.yml. Adds 'cmake/**' to the cmakelist filter. The filter lists CMakeLists.txt and toolchain/cmake/**, but not the cmake/ directory added in #1552, so a PR that changes only cmake/*.cmake skips the build and test matrix. #2019 had the same line but was closed without merging.

toolchain/mfc/test/coverage.py. Adds cmake/ to ALWAYS_RUN_ALL_PREFIXES, next to toolchain/cmake/. This changes no test selection: a cmake/ change already ran the full suite through the "unattributable change" fallback. It makes the rule explicit, so cmake/ and toolchain/cmake/ are handled by the same named rule. A unit test covers it.

Side effects

• NVHPC ≤ 26.3 with CUDA ≥ 12.9: they link the same libraries as before; the explicit -lnvhpcwrapnvtx was a duplicate of what -cudalib=nvtx3 adds.
• CUDA ≤ 12.8: unchanged.
• Results: unchanged with GNU and NVHPC 26.3. Intel, Cray and AMD were not tested; the three comparisons mean exactly what any() means.
• CI: two more NVHPC jobs, and the build matrix now runs for PRs that change only cmake/.

Testing

On this PR's head, with NVHPC 26.9 and the CI jobs' flags (FFLAGS=-tp=px -Kieee -noswitcherror):

	Result
CPU job's test step, hypoelasticity only (--test-all --no-mpi --only Hypoelasticity)	61 passed, 0 failed
GPU job's case-optimized build (3D_sphbubcollapse, --gpu acc)	builds
Hypoelasticity tests on an H100 NVL (MPI on), --gpu acc	62 passed, 0 failed
Hypoelasticity tests on an H100 NVL (MPI on), --gpu mp	62 passed, 0 failed
The same, without the workaround commit (control)	13 of the first 23 fail in both modes, and the run aborts
./mfc.sh precheck	passes

CI builds 26.9 for GPUs but does not run it there; the H100 rows cover that.

Earlier, on the previous master (5791dd2f):

• NVHPC 26.9 full CPU suite, without MPI: 714 passed, 0 failed.
• NVHPC 26.9 full --gpu acc suite on an H100, without the workaround: 47 failed, all hypoelasticity.
• GNU + Open MPI full suite: 726 passed, 0 failed.
• NVHPC 26.3 hypoelasticity tests: 61 of 61 pass.

The NVHPC 26.9 any() miscompile

At -O2 and above, 26.9 evaluates any(i == shear_indices) inside a loop as always true. In 2D, shear_indices = [stress%beg + 1, 0, 0], so the normal stress τ_xx is counted as shear, and its elastic energy is added twice:

• in pre_process (s_convert_primitive_to_conservative_variables), so the initial energy is wrong;
• and in the same pattern in m_variables_conversion and m_riemann_state during the run.

For 34F3999B, the initial E is off by exactly τ²/(4G) in every cell (2.5e8 in patch 1, where G = 0.1).

• Not MFC's code.
  • The 20-line program below is standard Fortran.
  • It prints the right answer with gfortran (-O0/-O3, -fcheck=all), with every other NVHPC release tested (21.11 to 26.3) at -O2, and with 26.9 at -O1 or -O2 -Mnovect.
  • 26.9 at -O2 gets it wrong even with shear_indices = [0, 0, 0].
• The workaround holds up. The three comparisons compile correctly on 26.9, including after -Minline. A helper function that wraps any() still miscompiles once inlined, so the macro spells the comparisons out.

program r2
    implicit none
    integer :: sh(3)
    real(8) :: e, t(10)
    sh = [9, 0, 0]
    t = 0d0
    t(8) = 1d0
    e = 0d0
    call go(e, t, sh, 8, 10)
    print '(a, f4.1, a)', 'e = ', e, '   (expected 1.0)'
contains
    subroutine go(e, t, sh, lo, hi)
        real(8), intent(inout) :: e
        real(8), intent(in) :: t(:)
        integer, intent(in) :: sh(3), lo, hi
        integer :: i
        do i = lo, hi
            e = e + t(i)
            if (any(i == sh)) e = e + t(i)
        end do
    end subroutine go
end program r2

nvfortran -O2 r2.f90 && ./a.out prints e = 2.0 on 26.9, and 1.0 on 26.3 and 25.9.

Which releases take which NVTX branch, and what -cudalib=nvtx3 links

In the CI images' environment. "Link and run": MFC's src/common/m_nvtx.f90 linked with only -cudalib=nvtx3 and run, with -acc and with -mp=gpu.

Releases that take the -cudalib=nvtx3 branch (CUDA ≥ 12.9):

NVHPC	CUDA found	-cudalib=nvtx3 adds	Link and run	Without the flag	MFC --gpu acc / --gpu mp build
25.5	12.9	-lnvhpcwrapnvtx -lnvtx3interop	OK	undefined nvtxRangePushEx	syscheck and pre_process build with both
25.7	12.9	-lnvhpcwrapnvtx -lnvtx3interop	OK	undefined nvtxRangePushEx	syscheck and pre_process build with both
25.9	12.9	-lnvhpcwrapnvtx -lnvtx3interop	OK	—	--gpu acc builds (H100 node)
25.11	13.0	-lnvhpcwrapnvtx -lnvtx3interop	OK	undefined nvtxRangePushEx	syscheck and pre_process build with both
26.1	13.1	-lnvhpcwrapnvtx -lnvtx3interop	OK	undefined nvtxRangePushEx	syscheck and pre_process build with both
26.3	13.1	-lnvhpcwrapnvtx -lnvtx3interop	OK	undefined nvtxRangePushEx	all four targets build with both
26.9	13.3	-lnvhpcnvtx -lnvtx3interop	OK	—	all four targets build with both

Releases with CUDA ≤ 12.8, which take the CUDA::nvToolsExt branch (checked with CMake for 21.11–24.5):

NVHPC	CUDA	-cudalib=nvtx3 alone
21.7, 21.11, 22.2, 22.3	11.4–11.6	rejected: nvfortran-Error-Switch -cudalib with unknown keyword nvtx3
22.11, 23.11, 24.5	11.8–12.4	accepted, but the link fails: undefined nvtxRangePushA/nvtxRangePushEx (it adds only -lnvhpcwrapnvtx)

So the version check has to stay. Always passing -cudalib=nvtx3 would break the clusters in toolchain/modules that use NVHPC 21.11–22.11.

When CMake and nvfortran use different CUDA versions. A cuda_multi install bundles more than one CUDA:

• CMake's branch follows the CUDA that FindCUDAToolkit finds (PATH, CUDAToolkit_ROOT);
• nvfortran's CUDA follows -gpu=cudaX.Y or NVHPC_CUDA_HOME.

Every pairing was forced this way, and MFC's NVTX calls were linked and run:

NVHPC	nvfortran uses	CMake finds	Branch	master	this PR
25.5	12.9	12.9	-cudalib=nvtx3	links, runs	links, runs
25.5	12.9	11.8	CUDA::nvToolsExt (11.8's libnvToolsExt)	links, runs	same branch, unchanged
25.5	11.8	12.9	-cudalib=nvtx3	links, runs	links, runs (the wrapper alone resolves the calls)
25.5	11.8	11.8	CUDA::nvToolsExt	links, runs	same branch, unchanged
26.9	13.3	13.3	-cudalib=nvtx3	cannot find -lnvhpcwrapnvtx	links, runs
26.9	13.3	12.9	-cudalib=nvtx3	cannot find -lnvhpcwrapnvtx	links, runs
26.9	12.9	13.3	-cudalib=nvtx3	cannot find -lnvhpcwrapnvtx	links, runs
26.9	12.9	12.9	-cudalib=nvtx3	cannot find -lnvhpcwrapnvtx	links, runs

On 25.5, the reviewer's case (CMake finds 11.8, nvfortran uses 12.9) links 11.8's libnvToolsExt, which resolves the calls on its own. 26.9 bundles only CUDA 12.9 and 13.3, so every pairing takes this PR's branch, and master fails all four.

NVTX on GPUs

	master	this PR
NVHPC 26.9, --gpu acc, 2× H100 NVL	link fails (above)	all targets build
NVHPC 25.9, --gpu acc, 2× H100 NVL	builds	builds
NVTX ranges in nsys, H100 (./mfc.sh run ... --nsys, 2D_riemann_test, 50 steps)	—	present on 25.9 and 26.9: TIMESTEP ×50, COMPUTE-RHS ×150, RHS-RIEMANN-SOLVER ×300, RHS-RECONSTRUCTION ×300
NVHPC 26.9, CPU only	ld: cannot find -lnvhpcwrapnvtx	builds

Contribution Policy

All contributions are expected to demonstrate:

• A clear understanding of the codebase
• Alignment with product direction
• Thoughtful reasoning behind changes
• Evidence of real-world usage or hands-on experience with the problem

If these expectations are not met, we would prefer to implement the changes ourselves rather than spend time reviewing low-effort submissions.

Acknowledgement

• I confirm this PR meets the above expectations and reflects my own understanding and real-world context.

---

This PR is from Diablo (Tsinghua University) participating in the SC26 Student Cluster Competition.

[github-actions]

Claude Code Review

Head SHA: 9eb641e

Files changed:

• 3
• .github/file-filter.yml
• .github/workflows/test.yml
• cmake/MFCTargets.cmake

Findings:
No issues found. Checked in step 4: cmake/MFCTargets.cmake (the NVTX link branch, against its surrounding NVHPC/CUDAToolkit block).

[codecov]

Codecov Report

❌ Patch coverage is 40.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.95%. Comparing base (ce4b191) to head (5666e4f).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
src/simulation/m_riemann_state.fpp	0.00%	0 Missing and 2 partials ⚠️
src/common/m_variables_conversion.fpp	66.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2111   +/-   ##
=======================================
  Coverage   61.95%   61.95%           
=======================================
  Files          86       86           
  Lines       22909    22909           
  Branches     3379     3379           
=======================================
  Hits        14193    14193           
  Misses       6207     6207           
  Partials     2509     2509           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[sbryngelson]

Thanks for tracking this down. The main question is the inline comment on line 350: the fix relies on -cudalib=nvtx3 alone, and that has only been checked on three NVHPC releases. I would like that confirmed for the other CUDA >= 12.9 releases in the CI matrix before merging.

Separately, no CI job covers NVHPC 26.9, the release this PR fixes. Adding { nvhpc: '26.9', target: gpu } to the matrix in .github/workflows/test.yml would catch a regression here.

[sbryngelson]

This is my main concern. With the explicit nvhpcwrapnvtx removed, -cudalib=nvtx3 is now the only thing that puts the NVTX wrapper on the link line. The description says this was checked with -dryrun on 25.9, 26.3 and 26.9. Other releases also reach this branch (CUDA >= 12.9), for example 25.5, 25.7, 25.11 and 26.1, which CI builds. If any of them accepts -cudalib=nvtx3 but does not add the wrapper, nvtxRangePushA / nvtxRangePushEx (bound in src/common/m_nvtx.f90) end up unresolved and every GPU target fails to link. The old explicit library used to cover that case.

Could you run the same -dryrun check on the other releases in the CI matrix, or confirm that the GPU CI jobs for those versions actually link and not only configure?

Related: -cudalib only means something to nvfortran. If one of these targets is ever linked by a different compiler driver (for example after C/C++ sources are added), the flag is ignored and nothing else provides the NVTX symbols. Setting LINKER_LANGUAGE Fortran on these targets would guarantee the flag takes effect.

[platelett]

1. Compatibility with the other NVHPC releases

Checked on 25.5, 25.7, 25.9, 25.11, 26.1, 26.3 and 26.9. MFC's actual m_nvtx.f90, with calls to nvtxRangePushA, nvtxRangePushEx and nvtxRangePop, links and runs with only -cudalib=nvtx3 under both -acc and -mp=gpu.

Using the environment settings from the CI images, FindCUDAToolkit finds CUDA 12.9 or newer and reports no CUDA::nvToolsExt, so these configurations exercise the changed branch.

For MFC itself, syscheck and pre_process build with both --gpu acc and --gpu mp on 25.5, 25.7, 25.11 and 26.1. All four targets build with both on 26.3 and 26.9. The release-by-release results are in the description.

2. Fortran link driver

Added LINKER_LANGUAGE Fortran to the targets in the -cudalib=nvtx3 branch, so nvfortran continues to drive the link if C or C++ sources are added later.

[sbryngelson]

This branch is chosen by what FindCUDAToolkit finds (TARGET CUDA::nvToolsExt), not by the CUDA version nvfortran actually links against. On the NVHPC cuda_multi images (CI, docker.yml), CMake can find an older CUDA that still ships libnvToolsExt while nvfortran defaults to CUDA >= 12.9. Then the build links a mismatched NVTX library and skips -cudalib=nvtx3. Would it be simpler to always pass -cudalib=nvtx3 for NVHPC and let the driver choose?

[platelett]

I checked whether the branch could be removed. NVHPC 21.7, 21.11, 22.2 and 22.3 reject -cudalib=nvtx3. Versions 22.11, 23.11 and 24.5 accept it but leave the NVTX calls unresolved when it is used alone. Those releases still need the CUDA::nvToolsExt path.

I also forced all four pairings of CMake's CUDA and nvfortran's CUDA on each of two releases: 25.5 with CUDA 11.8/12.9, and 26.9 with CUDA 12.9/13.3. With this PR, the NVTX test calls link and run in all eight combinations.

In the case you described on 25.5—CMake finding 11.8 while nvfortran uses 12.9—the build links 11.8's libnvToolsExt, which resolves the calls. That branch is unchanged. The reverse pairing also works with -cudalib=nvtx3 alone. On 26.9, all four pairings use the changed branch and pass; master fails all four because it explicitly requests the old wrapper name.

These are checks of the NVTX calls, rather than full CFD regression runs for every pairing.

[platelett]

@claude full review

[platelett]

@claude full review

[platelett]

Thanks for tracking this down. The main question is the inline comment on line 350: the fix relies on -cudalib=nvtx3 alone, and that has only been checked on three NVHPC releases. I would like that confirmed for the other CUDA >= 12.9 releases in the CI matrix before merging.

Separately, no CI job covers NVHPC 26.9, the release this PR fixes. Adding { nvhpc: '26.9', target: gpu } to the matrix in .github/workflows/test.yml would catch a regression here.

Added { nvhpc: '26.9', target: gpu } to the matrix. The existing GPU job builds both OpenACC and OpenMP variants, so it covers compilation and linking on 26.9. It does not execute the numerical regression tests; the separate 26.9 hypoelasticity issue and its reproducer are documented in the PR description.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

[sbryngelson]

Thanks, the CMake change looks right to me: -cudalib=nvtx3 already pulls in the wrapper, so dropping the explicit library name is the clean fix. A few comments inline. The main one is that this turns a 26.9 link failure into a build that silently produces wrong hypoelastic results, so I'd like some guard for that before or alongside merging.

[sbryngelson]

Once this lands, NVHPC 26.9 builds succeed, but per the PR description 26.9 at -O2 miscompiles any(i == shear_indices) in m_variables_conversion.fpp and the hypoelasticity tests fail. Today users get a loud link error. After this, they get wrong physics with no diagnostic. Could we add a message(WARNING ...) (or FATAL_ERROR) when CMAKE_Fortran_COMPILER_VERSION is 26.9, or land the code workaround together with this?

[platelett]

I've included the workaround in this PR as a separate commit, before the link fix.

IS_SHEAR(idx) replaces all five any(... == shear_indices) checks with three explicit comparisons.

With the workaround, the hypoelasticity tests pass on the current code with NVHPC 26.9: 61 of 61 on CPU with the CI job's flags, and 62 of 62 on an H100 with both --gpu acc and --gpu mp, with MPI enabled.

Without the workaround, 13 of the first 23 tests fail on the H100 in each GPU mode before the run aborts. Failures were also reproduced on CPU.

The compiler bug and a reproducer are documented in the description.

[sbryngelson]

Two notes on this job:

1. It only builds (--dry-run), so nothing in CI runs tests on 26.9. A fix or regression of the hypoelasticity miscompile would go unnoticed. A follow-up 26.9 job that runs just the hypoelastic tests (allowed to fail) would cover that.
2. The matrix has continue-on-error: true, so this job cannot fail the workflow. It will only catch a link regression if someone reads its log. That policy predates this PR, so not a blocker, but worth knowing.

Also, the 26.9 testing in the description used the tarball, not nvcr.io/nvidia/nvhpc:26.9-devel-cuda_multi-ubuntu22.04. The job is still pending; let's confirm the image exists and fits on the runner before merging.

[platelett]

• A 26.9 test job: added as a regular { nvhpc: '26.9', target: cpu } entry. I read your suggestion as a job that runs the tests, which on the GitHub-hosted runners means the CPU target. With the workaround in, it is an ordinary matrix entry that runs the full suite like the other releases, not a separate hypoelasticity-only job. On 26.9 with that job's flags (--test-all --no-mpi, FFLAGS=-tp=px -Kieee -noswitcherror), the hypoelasticity tests give 61 passed, 0 failed. The full suite without MPI passed earlier on 26.9 (714 passed, 0 failed).

• The GPU job's steps on 26.9: the case-optimized 3D_sphbubcollapse build passes. If you meant running the tests on a GPU in CI: the self-hosted Phoenix NVHPC jobs use 25.5 (toolchain/modules), so that would need 26.9 there. Until then, the H100 runs above cover the workaround on a device.

• continue-on-error: agreed, left as is.

[sbryngelson]

This line is also in #2019; one of the two PRs should drop it. Related: ALWAYS_RUN_ALL_PREFIXES in toolchain/mfc/test/coverage.py lists toolchain/cmake/ but not cmake/. cmake/ changes still trigger the full run through the fallback rule, but adding the prefix there would keep the two lists in agreement.

[platelett]

Since #2019 was closed without merging, I've kept cmake/** here.

I've also added cmake/ to ALWAYS_RUN_ALL_PREFIXES, with a unit test for cmake/MFCTargets.cmake. As you noted, the fallback already selected the full suite, so this makes the rule explicit rather than changing test selection. The description now states that distinction.

[github-actions]

Lines of Code

File	Lines	Diff
src/common/include/macros.fpp	131	+3
src/common/m_variables_conversion.fpp	965	-1

Directory	Lines	Diff
common	10538	+2
total	47640	+2

[aliencaocao]

@platelett I noticed you have a workaround for the any() miscompile. My team has also discovered this and we have reported to NVIDIA earlier on, and received confirmation from them that this will be fixed in the next NVHPC SDK release. Just letting everyone here know.

[platelett]

@platelett I noticed you have a workaround for the any() miscompile. My team has also discovered this and we have reported to NVIDIA earlier on, and received confirmation from them that this will be fixed in the next NVHPC SDK release. Just letting everything here know.

Thanks for sharing and for reporting it to NVIDIA. Good to know a fix is on the way!