From ee95002f47b6bc9443ef4beb55f36086ccf6907c Mon Sep 17 00:00:00 2001 From: Andreas Adelmann Date: Tue, 8 Sep 2026 19:27:07 +0200 Subject: [PATCH 1/6] Enable Kokkos Kernels with host and GPU dependency configuration Enable Kokkos Kernels by default, supporting installed packages and FetchContent builds. Configure host LAPACKE/MKL independently of CUDA/HIP libraries and validate provider capabilities, linkage, and integer ABI. Add OPALX-inspired linear-map tests covering analytic eigenvalues and eigenvector residuals. Use Kernels GEMM and direct host LAPACKE because the upstream 5.2.0 eigenanalysis header fails with GCC 15 and Clang 21. Fix build-tree/install dependency propagation, add a standalone consumer test, document configuration, and forward dependency options through CI. Validated Serial and OpenMP builds, GCC and Clang, 1/2/4 MPI ranks, disabled configurations, and build-tree/install consumers. CUDA, HIP, MKL, and a working ILP64 provider remain unverified. Refs #600 --- CMakeLists.txt | 1 + README.md | 4 + ci/cscs/dashboard-configure-build.cmake | 9 + cmake/BuildSystem.md | 98 +++++++++ cmake/Dependencies.cmake | 12 ++ cmake/FindTPLLAPACKE.cmake | 19 ++ cmake/IPPLConfig.cmake.in | 93 +++----- cmake/KokkosKernelsDependency.cmake | 201 ++++++++++++++++++ .../KokkosKernelsConsumer/CMakeLists.txt | 5 + cmake/tests/KokkosKernelsConsumer/main.cpp | 24 +++ src/CMakeLists.txt | 5 + unit_tests/CMakeLists.txt | 3 + unit_tests/KokkosKernels/CMakeLists.txt | 7 + .../KokkosKernels/KokkosKernelsLinearMap.cpp | 196 +++++++++++++++++ 14 files changed, 613 insertions(+), 64 deletions(-) create mode 100644 cmake/FindTPLLAPACKE.cmake create mode 100644 cmake/KokkosKernelsDependency.cmake create mode 100644 cmake/tests/KokkosKernelsConsumer/CMakeLists.txt create mode 100644 cmake/tests/KokkosKernelsConsumer/main.cpp create mode 100644 unit_tests/KokkosKernels/CMakeLists.txt create mode 100644 unit_tests/KokkosKernels/KokkosKernelsLinearMap.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index 4990341c94..78460ccbe4 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -35,6 +35,7 @@ include(ProjectSetup) set(IPPL_PLATFORMS "SERIAL" CACHE STRING "Platforms to build IPPL for") option(BUILD_SHARED_LIBS "Build IPPL as a shared library" OFF) option(IPPL_ENABLE_UNIT_TESTS "Enable unit tests using GoogleTest" OFF) +option(IPPL_ENABLE_KOKKOS_KERNELS "Enable Kokkos Kernels linear algebra support" ON) option(IPPL_ENABLE_FFT "Enable FFT support" OFF) option(IPPL_ENABLE_CUFFTMP "Enable NVIDIA cuFFTMp FFT backend" OFF) option(IPPL_ENABLE_FINUFFT "Enable FINUFFT NUFFT backend" OFF) diff --git a/README.md b/README.md index 4b7e2ff7b2..faa8df8313 100644 --- a/README.md +++ b/README.md @@ -45,6 +45,10 @@ The [IPPL Manual](https://ippl-framework.github.io/Manual/) contains comprehensi For detailed API, class, and file documentation, please visit our [Doxygen site](https://ippl-framework.github.io/ippl/). +Kokkos Kernels is enabled by default and requires a host LAPACKE or MKL provider. +See [Kokkos Kernels configuration](cmake/BuildSystem.md#kokkos-kernels-and-host-eigenanalysis) +for dependency setup and options to disable host eigenanalysis or Kokkos Kernels. + **Feedback & Support** If you find any issues with the manual, please report them in the [Manual's GitHub repository](https://github.com/IPPL-framework/Manual). diff --git a/ci/cscs/dashboard-configure-build.cmake b/ci/cscs/dashboard-configure-build.cmake index 49ad872356..367ec84b5e 100644 --- a/ci/cscs/dashboard-configure-build.cmake +++ b/ci/cscs/dashboard-configure-build.cmake @@ -48,6 +48,15 @@ set(VARS_TO_FORWARD IPPL_ENABLE_SCRIPTS Heffte_VERSION Kokkos_VERSION + KokkosKernels_VERSION + IPPL_ENABLE_KOKKOS_KERNELS + IPPL_KOKKOS_KERNELS_HOST + IPPL_LAPACK_INTEGER_BYTES + LAPACKE_ROOT + LAPACKE_INCLUDE_DIRS + LAPACKE_LIBRARY_DIRS + LAPACKE_LIBRARIES + MKL_DIR MPIEXEC_EXECUTABLE MPIEXEC_PREFLAGS MPIEXEC_MAX_NUMPROCS diff --git a/cmake/BuildSystem.md b/cmake/BuildSystem.md index bb44185e75..6b0b10f526 100644 --- a/cmake/BuildSystem.md +++ b/cmake/BuildSystem.md @@ -172,3 +172,101 @@ target_link_libraries(app PRIVATE ippl::ippl) ``` + +## Kokkos Kernels and host eigenanalysis + +`IPPL_ENABLE_KOKKOS_KERNELS` defaults to `ON`. IPPL first finds an installed +Kokkos Kernels package (default minimum version 5.2.0), then falls back to +FetchContent. `KokkosKernels_VERSION=git.` requests a source build. +The package uses the Kokkos target already selected by IPPL. An external package +must have been built against a compatible Kokkos with the required backends. + +Host eigenanalysis is selected independently of `IPPL_PLATFORMS`: + +- `IPPL_KOKKOS_KERNELS_HOST=LAPACKE` (default): provide LAPACKE headers and a + complete LAPACKE/LAPACK/BLAS link line. Set `LAPACKE_ROOT`, or + `LAPACKE_INCLUDE_DIRS` and `LAPACKE_LIBRARIES`. Static link lines must include + transitive dependencies, such as the Fortran runtime. Library names can be + resolved through `LAPACKE_LIBRARY_DIRS`; absolute paths are supported. +- `IPPL_KOKKOS_KERNELS_HOST=MKL`: provide an MKL CMake package through + `CMAKE_PREFIX_PATH`/`MKL_DIR` (modern oneMKL). +- `IPPL_KOKKOS_KERNELS_HOST=NONE`: use portable/GPU kernels without requesting + host eigenanalysis; only the GEMM regression is registered in the test binary. +- `IPPL_ENABLE_KOKKOS_KERNELS=OFF`: omit the dependency and its tests entirely. + +Kokkos does not install these external libraries. On macOS, install a LAPACKE +provider, such as an OpenBLAS build with LAPACKE or reference LAPACK built with +LAPACKE. Apple's Accelerate alone does not supply this LAPACKE interface. +Installation paths belong in site presets/toolchain files, not repository CMake. +For example, source builds can use these preset cache entries: + +```json +{ + "IPPL_ENABLE_KOKKOS_KERNELS": "ON", + "IPPL_KOKKOS_KERNELS_HOST": "LAPACKE", + "LAPACKE_ROOT": "/site/path/to/lapacke", + "IPPL_LAPACK_INTEGER_BYTES": "4" +} +``` + +Use headers and libraries from one compatible provider. Integer size defaults to +4 bytes (LP64); 8 selects ILP64 and propagates `LAPACK_ILP64` or `MKL_ILP64` to IPPL +consumers. For MKL source builds it also selects `MKL_INTERFACE`. Changing this +option cannot convert an installed library to another ABI. Configure checks the +header integer size, links an actual `LAPACKE_dgeev` call, and, for native builds, +runs a small spectrum/status check. Cross builds defer runtime checks to tests. +These checks are smoke tests, not a substitute for a consistent vendor build. +Providers with renamed symbols (for example, `LAPACKE_dgeev64_`) need matching +headers exposing the standard LAPACKE call; integer-size selection alone does +not adapt symbol names. Incompatible installations are rejected by the link probe. + +CUDA source builds enable cuBLAS, cuSOLVER and cuSPARSE. HIP source builds enable +rocBLAS, rocSOLVER and rocSPARSE (the sparse libraries are required by the solver +TPL configuration). Supply `CUDAToolkit_ROOT` or ROCm's `CMAKE_PREFIX_PATH` in site +presets. Host LAPACKE/MKL remains necessary when host eigenanalysis is requested. +Installed Kernels configurations are inspected for the requested TPLs; changing +cache options cannot add a missing TPL to an installed package. All link +requirements propagate through `Kokkos::kokkoskernels` and `IPPL::ippl`. + +Source builds default to on-demand kernel instantiation rather than default ETI. +`KokkosKernels_ADD_DEFAULT_ETI=ON` can enable upstream's preinstantiations. +Supernodal SPTRSV defaults off because it requires LayoutLeft ETI; applications +needing it should configure the required ETI and enable it explicitly. + +The `KokkosKernelsLinearMap` unit test constructs coupled 4x4 maps using GEMM in +the default execution/memory space, explicitly copies results to host memory, +and verifies analytical eigenvalues and normalized right-eigenvector residuals. +It covers stable rotations, conjugate branches, coupled modes, real/complex +instabilities, and near-integer/neutral modes. Eigenvalue/residual tolerance is +1e-12 for small, well-conditioned double-precision maps; GEMM tolerance is 1e-14. +No production physics algorithm or reduction ordering is changed. + +The test calls the configured host `LAPACKE_dgeev` directly with column-major +Kokkos host views. Kokkos Kernels 5.2.0's experimental `SerialEigendecomposition` +header contains an invalid Householder template call rejected by GCC 15 and +Clang 21; its +device implementation is unfinished and its host wrapper discards LAPACKE's +status. IPPL does not patch upstream or suppress compiler diagnostics. This test +therefore validates portable map construction and an explicit host eigenanalysis +boundary, not a GPU eigensolver or the experimental wrapper. + +```sh +cmake --build build_openmp --target KokkosKernelsLinearMap +OMP_NUM_THREADS=2 ctest --test-dir build_openmp -R '^KokkosKernelsLinearMap$' --output-on-failure +``` + +Reference: [Kokkos Kernels 5.2.0 eigenanalysis implementation](https://github.com/kokkos/kokkos-kernels/blob/5.2.0/batched/dense/impl/KokkosBatched_Eigendecomposition_Serial_Internal.hpp). + +A standalone consumer test checks dependency propagation while linking only +`IPPL::ippl`. Configure it against the IPPL build directory, then repeat with +`IPPL_DIR=/lib/cmake/ippl` after installation: + +```sh +cmake -S cmake/tests/KokkosKernelsConsumer -B build/kernels-consumer -DIPPL_DIR="$PWD/build_openmp" +cmake --build build/kernels-consumer +OMP_NUM_THREADS=2 build/kernels-consumer/consumer +``` + +Use the same compiler/toolchain as the IPPL build. The CSCS dashboard script +forwards the host-provider and dependency-path options above; site images must +supply these dependencies before running default-enabled builds. diff --git a/cmake/Dependencies.cmake b/cmake/Dependencies.cmake index 001bcfe295..fa7c770a30 100644 --- a/cmake/Dependencies.cmake +++ b/cmake/Dependencies.cmake @@ -285,6 +285,18 @@ else() # git tag, please use one of these:\n" "${KOKKOS_GIT_TAGS}") endif() endif() +# ------------------------------------------------------------------------------ +# ~~~ +# Kokkos Kernels +# ------------------------------------------------------------------------------ +set(IPPL_KOKKOS_PACKAGE_HINT "${Kokkos_DIR}") +if(NOT Kokkos_FOUND) + set(IPPL_KOKKOS_PACKAGE_HINT "${kokkos_BINARY_DIR}") +endif() +if(IPPL_ENABLE_KOKKOS_KERNELS) + include(KokkosKernelsDependency) +endif() + # ------------------------------------------------------------------------------ # ~~~ # Heffte (only if FFT enabled) diff --git a/cmake/FindTPLLAPACKE.cmake b/cmake/FindTPLLAPACKE.cmake new file mode 100644 index 0000000000..2346f4ffcf --- /dev/null +++ b/cmake/FindTPLLAPACKE.cmake @@ -0,0 +1,19 @@ +# IPPL has already resolved the complete host link line. Preserve absolute paths +# and static transitive dependencies; upstream 5.2's finder treats them as names. +include(FindPackageHandleStandardArgs) +find_package_handle_standard_args(TPLLAPACKE REQUIRED_VARS LAPACKE_INCLUDE_DIRS LAPACKE_LIBRARIES) +set(_lapackeResolved) +foreach(_library IN LISTS LAPACKE_LIBRARIES) + if(IS_ABSOLUTE "${_library}") + if(NOT EXISTS "${_library}") + message(FATAL_ERROR "LAPACKE dependency does not exist: ${_library}") + endif() + list(APPEND _lapackeResolved "${_library}") + else() + unset(_lapackeLibrary CACHE) + find_library(_lapackeLibrary NAMES ${_library} HINTS ${LAPACKE_LIBRARY_DIRS} REQUIRED) + list(APPEND _lapackeResolved "${_lapackeLibrary}") + endif() +endforeach() +kokkoskernels_create_imported_tpl(LAPACKE INTERFACE + LINK_LIBRARIES "${_lapackeResolved}" INCLUDES "${LAPACKE_INCLUDE_DIRS}") diff --git a/cmake/IPPLConfig.cmake.in b/cmake/IPPLConfig.cmake.in index 2e262350fc..f50e6e3c32 100644 --- a/cmake/IPPLConfig.cmake.in +++ b/cmake/IPPLConfig.cmake.in @@ -1,76 +1,41 @@ @PACKAGE_INIT@ include(CMakeFindDependencyMacro) +find_dependency(MPI COMPONENTS CXX) -find_dependency(MPI REQUIRED) +# Source-built dependencies export packages in their build directories. Installed +# dependencies should be found relative to this package, including relocated installs. +set(_ipplPrefix "${PACKAGE_PREFIX_DIR}") +set(_ipplBuildTree FALSE) +if(CMAKE_CURRENT_LIST_DIR STREQUAL "@PROJECT_BINARY_DIR@" OR + CMAKE_CURRENT_LIST_DIR STREQUAL "@PROJECT_BINARY_DIR@/src") + set(_ipplBuildTree TRUE) +endif() -# ----------------------------------------------------------------------------- -# To support FetchContent downloaded kokkos/heffte: -# Detect if this file is being sourced from build tree or install tree -# ----------------------------------------------------------------------------- -get_filename_component(_ipplconfig_dir "${CMAKE_CURRENT_LIST_DIR}" REALPATH) -get_filename_component(_build_dir "@CMAKE_BINARY_DIR@" REALPATH) -get_filename_component(_install_dir "@CMAKE_INSTALL_PREFIX@" REALPATH) -if(_ipplconfig_dir MATCHES "^${_install_dir}") - set(_INSTALL_TREE 1) -elseif(_ipplconfig_dir MATCHES "^${_build_dir}") - set(_BUILD_TREE 1) +if(NOT TARGET Kokkos::kokkos) + if(_ipplBuildTree) + find_dependency(Kokkos CONFIG HINTS "@IPPL_KOKKOS_PACKAGE_HINT@") + else() + find_dependency(Kokkos CONFIG HINTS "${_ipplPrefix}" "@Kokkos_DIR@") + endif() endif() -# ----------------------------------------------------------------------------- -# Kokkos -# ----------------------------------------------------------------------------- -# "Kokkos_FOUND" is set when IPPL is configured if an external Kokkos was found -if (@Kokkos_FOUND@) - message(STATUS "Finding IPPL dependency : external Kokkos") - set(Kokkos_DIR @Kokkos_DIR@) - find_dependency(Kokkos REQUIRED) -else() - # Make sure Kokkos is corrrectly picked up, regardless of where we got it or built it - if(_INSTALL_TREE) - set(Kokkos_ROOT "${_install_dir}") - find_dependency(Kokkos REQUIRED) - message(STATUS "Using Kokkos_ROOT in the ippl install tree ${_install_dir}") - elseif(_BUILD_TREE) - set(Kokkos_ROOT "${Kokkos_BINARY_DIR}") - find_dependency(Kokkos REQUIRED) - message(STATUS - "You are using IPPL directly from the IPPL build tree with Kokkos compiled from source inside the IPPL tree ${Kokkos_ROOT}") - else() - message(FATAL_ERROR "IPPLConfig.cmake location does not match IPPL build or install tree") - endif() +if(@IPPL_ENABLE_KOKKOS_KERNELS@ AND NOT TARGET Kokkos::kokkoskernels) + if("@IPPL_KOKKOS_KERNELS_NEEDS_MKL@") + find_dependency(MKL CONFIG) + endif() + if(_ipplBuildTree) + find_dependency(KokkosKernels CONFIG HINTS "@IPPL_KOKKOS_KERNELS_PACKAGE_HINT@") + else() + find_dependency(KokkosKernels CONFIG HINTS "${_ipplPrefix}" "@KokkosKernels_DIR@") + endif() endif() -message(STATUS "Kokkos found version ${Kokkos_VERSION} ${Kokkos_DIR}") -# ----------------------------------------------------------------------------- -# Heffte -# ----------------------------------------------------------------------------- -if(@IPPL_ENABLE_FFT@) - if (@Heffte_FOUND@) - message(STATUS "Finding IPPL dependency : external Heffte") - set(Heffte_DIR @Heffte_DIR@) - find_dependency(Heffte REQUIRED) - else() - # Heffte was built "in-tree" via FetchContent - # we will instruct cmake to find Heffte either in build or install tree - if(_INSTALL_TREE) - set(Heffte_ROOT "${_install_dir}") - find_dependency(Heffte REQUIRED) - message(STATUS "Using Heffte_ROOT in the ippl install tree ${_install_dir}") - elseif(_BUILD_TREE) - set(Heffte_ROOT "${Heffte_BINARY_DIR}") - find_dependency(Heffte REQUIRED) - message(STATUS - "You are using IPPL directly from the IPPL build tree with Heffte compiled from source inside the IPPL tree ${Heffte_ROOT}") - else() - message(FATAL_ERROR "IPPLConfig.cmake location does not match IPPL build or install tree") - endif() - endif() - message(STATUS "Heffte found version ${Heffte_VERSION} ${Heffte_DIR}") +# Source-built Heffte is already part of IPPLTargets (see InstallIppl.cmake). +if(@IPPL_ENABLE_FFT@ AND "@Heffte_FOUND@" AND NOT TARGET Heffte::Heffte) + find_dependency(Heffte CONFIG HINTS "${_ipplPrefix}" "@Heffte_DIR@") endif() -# ----------------------------------------------------------------------------- -# MAke sure IPPL targets are setup -# ----------------------------------------------------------------------------- include("${CMAKE_CURRENT_LIST_DIR}/IPPLTargets.cmake") - +unset(_ipplPrefix) +unset(_ipplBuildTree) diff --git a/cmake/KokkosKernelsDependency.cmake b/cmake/KokkosKernelsDependency.cmake new file mode 100644 index 0000000000..aab845251c --- /dev/null +++ b/cmake/KokkosKernelsDependency.cmake @@ -0,0 +1,201 @@ +# Resolve host eigenanalysis independently of CUDA/HIP acceleration. +# This module is included only when IPPL_ENABLE_KOKKOS_KERNELS is enabled. +include(CheckCXXSourceCompiles) +include(CheckCXXSourceRuns) +include(CMakePushCheckState) + +set(KokkosKernels_VERSION "5.2.0" CACHE STRING "Kokkos Kernels release or git.tag/sha") +set(IPPL_KOKKOS_KERNELS_HOST "LAPACKE" CACHE STRING "Host eigenanalysis provider: LAPACKE, MKL, NONE") +set_property(CACHE IPPL_KOKKOS_KERNELS_HOST PROPERTY STRINGS LAPACKE MKL NONE) +set(IPPL_LAPACK_INTEGER_BYTES "4" CACHE STRING "Host LAPACK integer size: 4 (LP64) or 8 (ILP64)") +set_property(CACHE IPPL_LAPACK_INTEGER_BYTES PROPERTY STRINGS 4 8) +if(NOT IPPL_KOKKOS_KERNELS_HOST MATCHES "^(LAPACKE|MKL|NONE)$") + message(FATAL_ERROR "IPPL_KOKKOS_KERNELS_HOST must be LAPACKE, MKL, or NONE") +endif() +if(NOT IPPL_LAPACK_INTEGER_BYTES MATCHES "^(4|8)$") + message(FATAL_ERROR "IPPL_LAPACK_INTEGER_BYTES must be 4 (LP64) or 8 (ILP64)") +endif() + +extract_git_label(KokkosKernels_VERSION _ipplKernelsGit) +if(NOT TARGET Kokkos::kokkoskernels AND NOT _ipplKernelsGit) + # Its config checks Kokkos_FOUND, whereas IPPL may have built Kokkos in-tree. + # Keep the already selected target instead of searching for another Kokkos. + set(_ipplKokkosFound "${Kokkos_FOUND}") + set(Kokkos_FOUND TRUE) + find_package(KokkosKernels ${KokkosKernels_VERSION} CONFIG QUIET) + set(Kokkos_FOUND "${_ipplKokkosFound}") +endif() + +if(NOT TARGET Kokkos::kokkoskernels) + # These configure only source builds. Installed packages are checked below. + set(KokkosKernels_ENABLE_TPL_LAPACKE OFF CACHE BOOL "Host LAPACKE" FORCE) + set(KokkosKernels_ENABLE_TPL_MKL OFF CACHE BOOL "Host MKL" FORCE) + if(IPPL_KOKKOS_KERNELS_HOST STREQUAL "LAPACKE") + set(KokkosKernels_ENABLE_TPL_LAPACKE ON CACHE BOOL "Host LAPACKE" FORCE) + find_path(IPPL_LAPACKE_INCLUDE_DIR lapacke.h + HINTS ${LAPACKE_INCLUDE_DIRS} ${LAPACKE_ROOT}/include $ENV{LAPACKE_ROOT}/include) + if(NOT IPPL_LAPACKE_INCLUDE_DIR) + message(FATAL_ERROR "Host LAPACKE headers are required, including for CUDA/HIP. Set LAPACKE_ROOT or LAPACKE_INCLUDE_DIRS, select IPPL_KOKKOS_KERNELS_HOST=MKL, or use NONE to disable eigenanalysis.") + endif() + set(LAPACKE_INCLUDE_DIRS "${IPPL_LAPACKE_INCLUDE_DIR}") + if(NOT LAPACKE_LIBRARIES) + find_library(IPPL_LAPACKE_LIBRARY NAMES lapacke openblas + HINTS ${LAPACKE_LIBRARY_DIRS} ${LAPACKE_ROOT}/lib ${LAPACKE_ROOT}/lib64 + $ENV{LAPACKE_ROOT}/lib $ENV{LAPACKE_ROOT}/lib64) + if(NOT IPPL_LAPACKE_LIBRARY) + message(FATAL_ERROR "Host LAPACKE library missing. Set LAPACKE_ROOT or LAPACKE_LIBRARIES (including static transitive dependencies).") + endif() + set(BLA_SIZEOF_INTEGER "${IPPL_LAPACK_INTEGER_BYTES}") + find_package(LAPACK REQUIRED) + set(LAPACKE_LIBRARIES "${IPPL_LAPACKE_LIBRARY};${LAPACK_LIBRARIES}") + endif() + elseif(IPPL_KOKKOS_KERNELS_HOST STREQUAL "MKL") + set(KokkosKernels_ENABLE_TPL_MKL ON CACHE BOOL "Host MKL" FORCE) + if(IPPL_LAPACK_INTEGER_BYTES STREQUAL "8") + set(MKL_INTERFACE ilp64) + else() + set(MKL_INTERFACE lp64) + endif() + find_package(MKL CONFIG REQUIRED) + endif() + + foreach(_backend CUBLAS CUSOLVER CUSPARSE ROCBLAS ROCSOLVER ROCSPARSE) + set(_enabled OFF) + if(("CUDA" IN_LIST IPPL_PLATFORMS AND _backend MATCHES "^CU") OR + ("HIP" IN_LIST IPPL_PLATFORMS AND _backend MATCHES "^ROC")) + set(_enabled ON) + endif() + set(KokkosKernels_ENABLE_TPL_${_backend} ${_enabled} CACHE BOOL "IPPL GPU linear algebra backend" FORCE) + endforeach() + # Avoid the considerable ETI build cost by instantiating only kernels used by IPPL. + set(KokkosKernels_ADD_DEFAULT_ETI OFF CACHE BOOL "Preinstantiate default Kokkos Kernels types") + set(KokkosKernels_ENABLE_SUPERNODAL_SPTRSV OFF CACHE BOOL "Supernodal solver requires LayoutLeft ETI") + if(NOT _ipplKernelsGit) + set(_ipplKernelsGit "${KokkosKernels_VERSION}") + endif() + FetchContent_Declare(KokkosKernels + GIT_REPOSITORY https://github.com/kokkos/kokkos-kernels.git + GIT_TAG ${_ipplKernelsGit}) + FetchContent_MakeAvailable(KokkosKernels) + # Upstream installs an export but does not provide a build-tree export. + export(EXPORT KokkosKernelsTargets + FILE "${kokkoskernels_BINARY_DIR}/KokkosKernelsTargets.cmake" NAMESPACE Kokkos::) + set(IPPL_KOKKOS_KERNELS_PACKAGE_HINT "${kokkoskernels_BINARY_DIR}") + set(IPPL_KOKKOS_KERNELS_FETCHED ON) +else() + set(IPPL_KOKKOS_KERNELS_PACKAGE_HINT "${KokkosKernels_DIR}") + set(IPPL_KOKKOS_KERNELS_FETCHED OFF) +endif() + +# Find the actual generated configuration header: inspect the target, not the +# user's cache options, which cannot enable features in an installed package. +get_target_property(_kernelIncludes Kokkos::kokkoskernels INTERFACE_INCLUDE_DIRECTORIES) +set(_kernelConfig) +foreach(_include IN LISTS _kernelIncludes) + if(_include MATCHES "^\\$$") + set(_include "${CMAKE_MATCH_1}") + endif() + if(EXISTS "${_include}/KokkosKernels_config.h") + set(_kernelConfig "${_include}/KokkosKernels_config.h") + endif() +endforeach() +if(NOT _kernelConfig) + message(FATAL_ERROR "Cannot locate KokkosKernels_config.h on Kokkos::kokkoskernels") +endif() +file(READ "${_kernelConfig}" _kernelConfiguration) +foreach(_backend IN ITEMS LAPACKE MKL CUBLAS CUSOLVER CUSPARSE ROCBLAS ROCSOLVER ROCSPARSE) + set(_has${_backend} OFF) + if(_kernelConfiguration MATCHES "#define KOKKOSKERNELS_ENABLE_TPL_${_backend}([ \t\r\n]|$)") + set(_has${_backend} ON) + endif() +endforeach() +set(IPPL_KOKKOS_KERNELS_NEEDS_MKL ${_hasMKL}) +if(_hasMKL) + # Modern upstream MKL exports reference MKL::MKL, including when host + # eigenanalysis was disabled in IPPL but the external package still uses MKL. + find_package(MKL CONFIG REQUIRED) +endif() +foreach(_platform CUDA HIP) + if(_platform IN_LIST IPPL_PLATFORMS) + if((_platform STREQUAL "CUDA" AND (NOT _hasCUBLAS OR NOT _hasCUSOLVER OR NOT _hasCUSPARSE)) OR + (_platform STREQUAL "HIP" AND (NOT _hasROCBLAS OR NOT _hasROCSOLVER OR NOT _hasROCSPARSE))) + message(FATAL_ERROR "Selected Kokkos Kernels lacks the required ${_platform} BLAS/solver TPLs. Rebuild it with the matching backend.") + endif() + endif() +endforeach() + +set(IPPL_KOKKOS_KERNELS_EIGENANALYSIS OFF) +if(NOT IPPL_KOKKOS_KERNELS_HOST STREQUAL "NONE") + if(NOT _has${IPPL_KOKKOS_KERNELS_HOST}) + message(FATAL_ERROR "Selected Kokkos Kernels lacks ${IPPL_KOKKOS_KERNELS_HOST}. Rebuild it with this host TPL, or select IPPL_KOKKOS_KERNELS_HOST=NONE.") + endif() + set(_integerDefinition) + if(IPPL_LAPACK_INTEGER_BYTES STREQUAL "8") + if(IPPL_KOKKOS_KERNELS_HOST STREQUAL "MKL") + set(_integerDefinition MKL_ILP64) + else() + set(_integerDefinition LAPACK_ILP64) + endif() + endif() + # Export the integer interface with IPPL, also for installed-package consumers. + set(IPPL_HOST_LAPACK_DEFINITIONS "${_integerDefinition}") + if(IPPL_KOKKOS_KERNELS_FETCHED AND _integerDefinition) + target_compile_definitions(kokkoskernels PUBLIC ${_integerDefinition}) + endif() + + cmake_push_check_state(RESET) + if(IPPL_KOKKOS_KERNELS_HOST STREQUAL "MKL") + find_package(MKL CONFIG REQUIRED) + set(CMAKE_REQUIRED_LIBRARIES MKL::MKL) + elseif(IPPL_KOKKOS_KERNELS_FETCHED) + # try_compile accepts imported targets, not unbuilt local interface targets. + add_library(ippl_lapacke_probe INTERFACE IMPORTED) + get_target_property(_hostLinks LAPACKE INTERFACE_LINK_LIBRARIES) + get_target_property(_hostIncludes LAPACKE INTERFACE_INCLUDE_DIRECTORIES) + set_target_properties(ippl_lapacke_probe PROPERTIES + INTERFACE_LINK_LIBRARIES "${_hostLinks}" + INTERFACE_INCLUDE_DIRECTORIES "${_hostIncludes}") + set(CMAKE_REQUIRED_LIBRARIES ippl_lapacke_probe) + else() + set(CMAKE_REQUIRED_LIBRARIES Kokkos::kokkoskernels) + endif() + if(_integerDefinition) + set(CMAKE_REQUIRED_DEFINITIONS -D${_integerDefinition}) + endif() + set(_header lapacke.h) + if(IPPL_KOKKOS_KERNELS_HOST STREQUAL "MKL") + set(_header mkl.h) + endif() + # Recheck after provider/path changes rather than retaining a stale success. + unset(IPPL_HOST_EIGEN_LINKS CACHE) + set(_hostEigenProbe " + #include <${_header}> + #include + static_assert(sizeof(lapack_int) == ${IPPL_LAPACK_INTEGER_BYTES}, \"LAPACK integer ABI mismatch\"); + int main() { + double a[4] = {0, -1, 1, 0}, er[2], ei[2], left[4], right[4]; + const lapack_int info = LAPACKE_dgeev(LAPACK_COL_MAJOR, 'V', 'V', 2, a, 2, er, ei, left, 2, right, 2); + if (info || !std::isfinite(er[0]) || !std::isfinite(ei[0]) || + std::abs(er[0]) > 1e-12 || std::abs(er[1]) > 1e-12 || + std::abs(std::abs(ei[0]) - 1) > 1e-12 || std::abs(ei[0]+ei[1]) > 1e-12) return 1; + // Invalid layout returns -1 without touching the arrays. Check signed + // status handling as well as a valid solve across the integer interface. + return LAPACKE_dgeev(0, 'V', 'V', 2, a, 2, er, ei, left, 2, right, 2) != -1; + }") + check_cxx_source_compiles("${_hostEigenProbe}" IPPL_HOST_EIGEN_LINKS) + if(IPPL_HOST_EIGEN_LINKS AND NOT CMAKE_CROSSCOMPILING) + unset(IPPL_HOST_EIGEN_RUNS CACHE) + check_cxx_source_runs("${_hostEigenProbe}" IPPL_HOST_EIGEN_RUNS) + if(NOT IPPL_HOST_EIGEN_RUNS) + message(FATAL_ERROR "Host LAPACKE linked but failed the eigenvalue/ABI smoke test. Check the selected libraries and integer interface.") + endif() + elseif(CMAKE_CROSSCOMPILING) + message(STATUS "Cross-compiling: host LAPACKE compile/link checked; runtime ABI validation deferred to unit tests") + endif() + cmake_pop_check_state() + if(NOT IPPL_HOST_EIGEN_LINKS) + message(FATAL_ERROR "Host LAPACKE eigenanalysis failed to compile/link. Check matching headers, LP64/ILP64 libraries, and all static dependencies (including Fortran runtime). See CMake configure log.") + endif() + set(IPPL_KOKKOS_KERNELS_EIGENANALYSIS ON) +endif() +message(STATUS "IPPL Kokkos Kernels: host=${IPPL_KOKKOS_KERNELS_HOST}, eigenanalysis=${IPPL_KOKKOS_KERNELS_EIGENANALYSIS}") diff --git a/cmake/tests/KokkosKernelsConsumer/CMakeLists.txt b/cmake/tests/KokkosKernelsConsumer/CMakeLists.txt new file mode 100644 index 0000000000..90fc607cec --- /dev/null +++ b/cmake/tests/KokkosKernelsConsumer/CMakeLists.txt @@ -0,0 +1,5 @@ +cmake_minimum_required(VERSION 3.24) +project(IPPLConsumer LANGUAGES CXX) +find_package(IPPL CONFIG REQUIRED) +add_executable(consumer main.cpp) +target_link_libraries(consumer PRIVATE IPPL::ippl) diff --git a/cmake/tests/KokkosKernelsConsumer/main.cpp b/cmake/tests/KokkosKernelsConsumer/main.cpp new file mode 100644 index 0000000000..5c6ac5c160 --- /dev/null +++ b/cmake/tests/KokkosKernelsConsumer/main.cpp @@ -0,0 +1,24 @@ +#include + +#include +#include +#if defined(KOKKOSKERNELS_ENABLE_TPL_MKL) +#include +#elif defined(KOKKOSKERNELS_ENABLE_TPL_LAPACKE) +#include +#else +#error "This consumer test requires a host eigenanalysis provider" +#endif +#include +int main(int argc, char** argv) { + Kokkos::ScopeGuard guard(argc, argv); + Kokkos::View a("a", 2, 2), b("b", 2, 2), c("c", 2, 2); + Kokkos::deep_copy(a, 1.0); + Kokkos::deep_copy(b, 2.0); + KokkosBlas::gemm("N", "N", 1.0, a, b, 0.0, c); + auto h = Kokkos::create_mirror_view_and_copy(Kokkos::HostSpace(), c); + double m[4] = {0, -1, 1, 0}, er[2], ei[2], l[4], r[4]; + auto info = LAPACKE_dgeev(LAPACK_COL_MAJOR, 'V', 'V', 2, m, 2, er, ei, l, 2, r, 2); + return info || !std::isfinite(h(0, 0)) || !std::isfinite(ei[0]) || std::abs(h(0, 0) - 4) > 1e-14 + || std::abs(std::abs(ei[0]) - 1) > 1e-12; +} diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index ef43848271..8727680d19 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -105,6 +105,11 @@ include(${PROJECT_SOURCE_DIR}/cmake/PlatformOptions.cmake) target_link_libraries(ippl PUBLIC Kokkos::kokkos MPI::MPI_CXX) +if(IPPL_ENABLE_KOKKOS_KERNELS) + target_link_libraries(ippl PUBLIC Kokkos::kokkoskernels) + target_compile_definitions(ippl PUBLIC IPPL_ENABLE_KOKKOS_KERNELS ${IPPL_HOST_LAPACK_DEFINITIONS}) +endif() + if(IPPL_ENABLE_FFT) target_link_libraries(ippl PUBLIC Heffte::Heffte) endif() diff --git a/unit_tests/CMakeLists.txt b/unit_tests/CMakeLists.txt index e8e8ea29e1..c652c3cce7 100644 --- a/unit_tests/CMakeLists.txt +++ b/unit_tests/CMakeLists.txt @@ -35,3 +35,6 @@ add_subdirectory(Meshes) add_subdirectory(Particle) add_subdirectory(PIC) add_subdirectory(Utility) +if(IPPL_ENABLE_KOKKOS_KERNELS) + add_subdirectory(KokkosKernels) +endif() diff --git a/unit_tests/KokkosKernels/CMakeLists.txt b/unit_tests/KokkosKernels/CMakeLists.txt new file mode 100644 index 0000000000..5f90c11507 --- /dev/null +++ b/unit_tests/KokkosKernels/CMakeLists.txt @@ -0,0 +1,7 @@ +add_ippl_test(KokkosKernelsLinearMap LABELS kokkos-kernels) +if(IPPL_KOKKOS_KERNELS_EIGENANALYSIS) + target_compile_definitions(KokkosKernelsLinearMap PRIVATE IPPL_TEST_HOST_EIGENANALYSIS) + if(IPPL_KOKKOS_KERNELS_HOST STREQUAL "MKL") + target_compile_definitions(KokkosKernelsLinearMap PRIVATE IPPL_TEST_HOST_MKL) + endif() +endif() diff --git a/unit_tests/KokkosKernels/KokkosKernelsLinearMap.cpp b/unit_tests/KokkosKernels/KokkosKernelsLinearMap.cpp new file mode 100644 index 0000000000..9a6cec9c7d --- /dev/null +++ b/unit_tests/KokkosKernels/KokkosKernelsLinearMap.cpp @@ -0,0 +1,196 @@ +#include "Ippl.h" + +#include +#include +#include +#include +#include +#include + +#include "KokkosBlas3_gemm.hpp" +#include "gtest/gtest.h" + +#ifdef IPPL_TEST_HOST_EIGENANALYSIS +#include "KokkosKernels_config.h" +#ifdef IPPL_TEST_HOST_MKL +#include +#else +#include +#endif +#endif + +namespace { + constexpr int Dimension = 4; + using Matrix = std::array, Dimension>; + + // OPALX's transverse two-plane rotation convention; tunes are in turns. + Matrix rotations(double tune1, double tune2, double radius1 = 1, double radius2 = 1) { + Matrix matrix{}; + for (int plane = 0; plane < 2; ++plane) { + const double angle = 2 * Kokkos::numbers::pi * (plane ? tune2 : tune1); + const double radius = plane ? radius2 : radius1; + const int i = 2 * plane; + matrix[i][i] = matrix[i + 1][i + 1] = radius * std::cos(angle); + matrix[i][i + 1] = radius * std::sin(angle); + matrix[i + 1][i] = -radius * std::sin(angle); + } + return matrix; + } + + // Exercise GEMM in the configured execution/memory space, then explicitly + // transfer Q A Q^T to host memory for host-only eigenanalysis. + Matrix coupled(const Matrix& matrix) { + using View = Kokkos::View; + View a("map", Dimension, Dimension), q("coupling", Dimension, Dimension); + View work("work", Dimension, Dimension), result("result", Dimension, Dimension); + auto hostA = Kokkos::create_mirror_view(a); + auto hostQ = Kokkos::create_mirror_view(q); + const double c = std::cos(0.6), s = std::sin(0.6); + const Matrix rotation{{{c, 0, s, 0}, {0, c, 0, s}, {-s, 0, c, 0}, {0, -s, 0, c}}}; + for (int i = 0; i < Dimension; ++i) + for (int j = 0; j < Dimension; ++j) { + hostA(i, j) = matrix[i][j]; + hostQ(i, j) = rotation[i][j]; + } + Kokkos::deep_copy(a, hostA); + Kokkos::deep_copy(q, hostQ); + KokkosBlas::gemm("N", "N", 1.0, q, a, 0.0, work); + KokkosBlas::gemm("N", "T", 1.0, work, q, 0.0, result); + auto hostResult = Kokkos::create_mirror_view_and_copy(Kokkos::HostSpace(), result); + Matrix output{}; + for (int i = 0; i < Dimension; ++i) + for (int j = 0; j < Dimension; ++j) + output[i][j] = hostResult(i, j); + return output; + } + + TEST(KokkosKernels, CoupledMapGemm) { + const auto matrix = rotations(0.155, 0.31); + const auto result = coupled(matrix); + const double c = std::cos(0.6), s = std::sin(0.6); + // A few operations on unit-scale entries: 1e-14 accommodates FMA differences. + for (int i = 0; i < 2; ++i) + for (int j = 0; j < 2; ++j) { + EXPECT_NEAR(result[i][j], c * c * matrix[i][j] + s * s * matrix[i + 2][j + 2], + 1e-14); + EXPECT_NEAR(result[i + 2][j + 2], + s * s * matrix[i][j] + c * c * matrix[i + 2][j + 2], 1e-14); + EXPECT_NEAR(result[i][j + 2], c * s * (matrix[i + 2][j + 2] - matrix[i][j]), 1e-14); + EXPECT_NEAR(result[i + 2][j], result[i][j + 2], 1e-14); + } + } + +#ifdef IPPL_TEST_HOST_EIGENANALYSIS + void checkEigenpairs(const Matrix& input, std::vector> expected) { + using View = Kokkos::View; + using Vector = Kokkos::View; + View a("map", Dimension, Dimension), left("left", Dimension, Dimension); + View right("right", Dimension, Dimension); + Vector real("real", Dimension), imaginary("imaginary", Dimension); + for (int i = 0; i < Dimension; ++i) + for (int j = 0; j < Dimension; ++j) + a(i, j) = input[i][j]; + // Kokkos Kernels 5.2's experimental eigen header does not compile with GCC 15 + // (invalid Householder template call). Use its configured host TPL directly, + // retaining LAPACKE's INFO, with contiguous column-major host views. + Kokkos::deep_copy(real, std::numeric_limits::quiet_NaN()); + Kokkos::deep_copy(imaginary, std::numeric_limits::quiet_NaN()); + Kokkos::deep_copy(right, std::numeric_limits::quiet_NaN()); + ASSERT_EQ( + LAPACKE_dgeev(LAPACK_COL_MAJOR, 'V', 'V', Dimension, a.data(), Dimension, real.data(), + imaginary.data(), left.data(), Dimension, right.data(), Dimension), + 0); + // Small well-conditioned maps should have O(epsilon) errors; allow 1e-12 + // for variation among LAPACK providers without changing existing tolerances. + constexpr double Tolerance = 1e-12; + for (int k = 0; k < Dimension; ++k) { + ASSERT_TRUE(std::isfinite(real(k))); + ASSERT_TRUE(std::isfinite(imaginary(k))); + const std::complex lambda(real(k), imaginary(k)); + auto match = std::min_element(expected.begin(), expected.end(), [&](auto x, auto y) { + return std::abs(x - lambda) < std::abs(y - lambda); + }); + ASSERT_NE(match, expected.end()); + EXPECT_LT(std::abs(*match - lambda), Tolerance); + expected.erase(match); + // LAPACK packs each conjugate pair into adjacent real columns. + if (imaginary(k) > 0) { + ASSERT_LT(k + 1, Dimension); + EXPECT_NEAR(real(k + 1), real(k), Tolerance); + EXPECT_NEAR(imaginary(k + 1), -imaginary(k), Tolerance); + } + if (imaginary(k) < 0) { + ASSERT_GT(k, 0); + ASSERT_GT(imaginary(k - 1), 0); + } + std::array, Dimension> vector{}; + for (int i = 0; i < Dimension; ++i) { + vector[i] = imaginary(k) == 0 ? std::complex(right(i, k), 0) + : imaginary(k) > 0 + ? std::complex(right(i, k), right(i, k + 1)) + : std::complex(right(i, k - 1), -right(i, k)); + } + double norm = 0, residual = 0, matrixNorm = 0; + for (int i = 0; i < Dimension; ++i) { + std::complex av = 0; + for (int j = 0; j < Dimension; ++j) { + av += input[i][j] * vector[j]; + matrixNorm += input[i][j] * input[i][j]; + } + norm += std::norm(vector[i]); + residual += std::norm(av - lambda * vector[i]); + } + ASSERT_GT(norm, 0); + EXPECT_LT(std::sqrt(residual) + / ((std::sqrt(matrixNorm) + std::abs(lambda)) * std::sqrt(norm)), + Tolerance); + } + } + + std::vector> spectrum(double q1, double q2, double r1 = 1, double r2 = 1) { + const auto first = std::polar(r1, 2 * Kokkos::numbers::pi * q1); + const auto second = std::polar(r2, 2 * Kokkos::numbers::pi * q2); + return {first, std::conj(first), second, std::conj(second)}; + } + + TEST(KokkosKernels, StableRotationsAndConjugateBranches) { + checkEigenpairs(rotations(0.155, 0.89), spectrum(0.155, 0.89)); + } + + TEST(KokkosKernels, CoupledStableModes) { + checkEigenpairs(coupled(rotations(0.155, 0.31)), spectrum(0.155, 0.31)); + } + + TEST(KokkosKernels, RealAndComplexInstabilities) { + auto matrix = rotations(0.15, 0.3); + matrix[0][0] = 2; + matrix[1][1] = 0.5; + matrix[0][1] = matrix[1][0] = 0; + auto expected = spectrum(0.15, 0.3); + expected[0] = 2; + expected[1] = 0.5; + checkEigenpairs(matrix, expected); + checkEigenpairs(coupled(rotations(0.2, 0.2, 1.1, 1 / 1.1)), + spectrum(0.2, 0.2, 1.1, 1 / 1.1)); + checkEigenpairs(rotations(0.2, 0.3, 0.9, 0.8), spectrum(0.2, 0.3, 0.9, 0.8)); + } + + TEST(KokkosKernels, NearIntegerAndNeutralModes) { + checkEigenpairs(rotations(1e-8, 0.3), spectrum(1e-8, 0.3)); + Matrix identity{}; + for (int i = 0; i < Dimension; ++i) + identity[i][i] = 1; + checkEigenpairs(identity, {1, 1, 1, 1}); + identity[0][0] = identity[1][1] = -1; + checkEigenpairs(identity, {-1, -1, 1, 1}); + } +#endif +} // namespace + +int main(int argc, char* argv[]) { + ippl::initialize(argc, argv); + ::testing::InitGoogleTest(&argc, argv); + const int result = RUN_ALL_TESTS(); + ippl::finalize(); + return result; +} From 3a0eab590b6d1e43e74d7844f75a8e3832d197d1 Mon Sep 17 00:00:00 2001 From: Andreas Adelmann Date: Tue, 8 Sep 2026 22:04:11 +0200 Subject: [PATCH 2/6] Add source-build fallback for LAPACKE Fetch and build reference LAPACK when a compatible LAPACKE installation cannot be found. Build the host BLAS, LAPACK, and LAPACKE libraries in an isolated C/Fortran project so CUDA and HIP compiler settings do not leak into the host dependency. Export and install the fallback libraries through a relocatable CMake package, support LP64 and ILP64 interfaces, preserve explicitly supplied LAPACKE libraries, and allow automatic fetching to be disabled with IPPL_FETCH_LAPACKE. --- README.md | 3 +- ci/cscs/dashboard-configure-build.cmake | 4 ++ cmake/BuildSystem.md | 38 ++++++++-- cmake/FetchHostLapack.cmake | 72 +++++++++++++++++++ cmake/FindTPLLAPACKE.cmake | 19 ++++- cmake/IPPLConfig.cmake.in | 8 +++ cmake/KokkosKernelsDependency.cmake | 54 ++++++++++---- cmake/lapack-bootstrap/CMakeLists.txt | 50 +++++++++++++ .../IPPLHostLapackConfig.cmake.in | 11 +++ 9 files changed, 238 insertions(+), 21 deletions(-) create mode 100644 cmake/FetchHostLapack.cmake create mode 100644 cmake/lapack-bootstrap/CMakeLists.txt create mode 100644 cmake/lapack-bootstrap/IPPLHostLapackConfig.cmake.in diff --git a/README.md b/README.md index faa8df8313..e96cf16eb7 100644 --- a/README.md +++ b/README.md @@ -45,7 +45,8 @@ The [IPPL Manual](https://ippl-framework.github.io/Manual/) contains comprehensi For detailed API, class, and file documentation, please visit our [Doxygen site](https://ippl-framework.github.io/ippl/). -Kokkos Kernels is enabled by default and requires a host LAPACKE or MKL provider. +Kokkos Kernels is enabled by default. Missing host LAPACKE is built from source +using C and Fortran compilers; installed LAPACKE or MKL providers are also supported. See [Kokkos Kernels configuration](cmake/BuildSystem.md#kokkos-kernels-and-host-eigenanalysis) for dependency setup and options to disable host eigenanalysis or Kokkos Kernels. diff --git a/ci/cscs/dashboard-configure-build.cmake b/ci/cscs/dashboard-configure-build.cmake index 367ec84b5e..3cd8f1b951 100644 --- a/ci/cscs/dashboard-configure-build.cmake +++ b/ci/cscs/dashboard-configure-build.cmake @@ -52,6 +52,10 @@ set(VARS_TO_FORWARD IPPL_ENABLE_KOKKOS_KERNELS IPPL_KOKKOS_KERNELS_HOST IPPL_LAPACK_INTEGER_BYTES + IPPL_FETCH_LAPACKE + IPPL_LAPACKE_BUILD_JOBS + IPPL_LAPACKE_TOOLCHAIN_FILE + CMAKE_Fortran_COMPILER LAPACKE_ROOT LAPACKE_INCLUDE_DIRS LAPACKE_LIBRARY_DIRS diff --git a/cmake/BuildSystem.md b/cmake/BuildSystem.md index 6b0b10f526..337997ed74 100644 --- a/cmake/BuildSystem.md +++ b/cmake/BuildSystem.md @@ -194,9 +194,37 @@ Host eigenanalysis is selected independently of `IPPL_PLATFORMS`: host eigenanalysis; only the GEMM regression is registered in the test binary. - `IPPL_ENABLE_KOKKOS_KERNELS=OFF`: omit the dependency and its tests entirely. -Kokkos does not install these external libraries. On macOS, install a LAPACKE -provider, such as an OpenBLAS build with LAPACKE or reference LAPACK built with -LAPACKE. Apple's Accelerate alone does not supply this LAPACKE interface. +Kokkos does not install these external libraries. If host LAPACKE is not found, +IPPL now downloads reference LAPACK 3.12.1 (SHA256-verified) and builds its BLAS, +LAPACK and LAPACKE libraries. The fallback requires host C and Fortran compilers; +set `CMAKE_Fortran_COMPILER` or `FC` if automatic discovery cannot find gfortran. +The host C compiler is selected independently via `CC` or the fallback toolchain. +It supports LP64 and ILP64 through `IPPL_LAPACK_INTEGER_BYTES` and builds static +position-independent libraries for use by shared or static IPPL. + +The first configure builds this dependency in an isolated host project under +`_deps`, before the existing compile/link/runtime probes run. This can take a few +minutes; subsequent configures reuse the build. CUDA/HIP compiler launchers and +IPPL's directory flags are not applied to the host project. Its logs are in +`_deps/ippl-host-lapack--build/{configure,build,install}.log`. +`IPPL_LAPACKE_BUILD_JOBS` controls build parallelism (default 4). +For offline builds, point `FETCHCONTENT_SOURCE_DIR_IPPL_REFERENCE_LAPACK` at an +already unpacked reference LAPACK 3.12.1 source tree. + +Set `IPPL_FETCH_LAPACKE=OFF` to require an installed provider. An explicit +`LAPACKE_LIBRARIES` remains authoritative: missing headers or broken linkage +produce errors instead of silently substituting a different provider. A Kernels +package already built without its LAPACKE TPL still needs rebuilding; fetching +LAPACKE cannot enable features in an installed Kernels library. + +For a cross build, supply `IPPL_LAPACKE_TOOLCHAIN_FILE` with a C/Fortran toolchain +for the target CPU, or use an installed provider. Runtime checks remain deferred +in cross builds. Fetched libraries, headers, license and the relocatable +`IPPLHostLapack` CMake package are bundled with IPPL's installation; the compatible +Fortran compiler runtime remains a system dependency. + +On macOS, an installed OpenBLAS build with LAPACKE or reference LAPACK can also be +used. Apple's Accelerate alone does not supply this LAPACKE interface. Installation paths belong in site presets/toolchain files, not repository CMake. For example, source builds can use these preset cache entries: @@ -268,5 +296,5 @@ OMP_NUM_THREADS=2 build/kernels-consumer/consumer ``` Use the same compiler/toolchain as the IPPL build. The CSCS dashboard script -forwards the host-provider and dependency-path options above; site images must -supply these dependencies before running default-enabled builds. +forwards the host-provider, fallback, compiler and dependency-path options above. +Site images can supply LAPACKE or host C/Fortran compilers for the fallback. diff --git a/cmake/FetchHostLapack.cmake b/cmake/FetchHostLapack.cmake new file mode 100644 index 0000000000..d65025499a --- /dev/null +++ b/cmake/FetchHostLapack.cmake @@ -0,0 +1,72 @@ +# Bootstrap the host library at configure time so the normal link/runtime probes +# still validate a real library on the first configure. Subsequent builds reuse it. +function(ippl_fetch_host_lapack) + if(CMAKE_CROSSCOMPILING AND NOT IPPL_LAPACKE_TOOLCHAIN_FILE) + message(FATAL_ERROR "LAPACKE fallback in a cross build needs IPPL_LAPACKE_TOOLCHAIN_FILE for the target CPU, or an installed LAPACKE provider.") + endif() + set(IPPL_LAPACKE_BUILD_JOBS 4 CACHE STRING "Parallel jobs for reference LAPACK bootstrap") + set(IPPL_LAPACKE_TOOLCHAIN_FILE "" CACHE FILEPATH "Optional host LAPACK C/Fortran toolchain") + FetchContent_Declare(ippl_reference_lapack + URL https://codeload.github.com/Reference-LAPACK/lapack/tar.gz/refs/tags/v3.12.1 + URL_HASH SHA256=2ca6407a001a474d4d4d35f3a61550156050c48016d949f0da0529c0aa052422 + DOWNLOAD_EXTRACT_TIMESTAMP TRUE + SOURCE_SUBDIR ippl-bootstrap-only) + FetchContent_MakeAvailable(ippl_reference_lapack) + set(_build "${FETCHCONTENT_BASE_DIR}/ippl-host-lapack-${IPPL_LAPACK_INTEGER_BYTES}-build") + set(_prefix "${FETCHCONTENT_BASE_DIR}/ippl-host-lapack-${IPPL_LAPACK_INTEGER_BYTES}-install") + set(_options) + # Let the host project discover C independently (or use CC/the toolchain). + # IPPL may enable C only later through a dependency, so forwarding its C + # compiler would change the bootstrap compiler on the second configure. + foreach(_variable CMAKE_Fortran_COMPILER CMAKE_OSX_ARCHITECTURES CMAKE_OSX_SYSROOT CMAKE_OSX_DEPLOYMENT_TARGET) + if(DEFINED ${_variable} AND NOT "${${_variable}}" STREQUAL "") + if(_variable MATCHES "_COMPILER$") + list(APPEND _options "-D${_variable}:FILEPATH=${${_variable}}") + else() + list(APPEND _options "-D${_variable}:STRING=${${_variable}}") + endif() + endif() + endforeach() + if(IPPL_LAPACKE_TOOLCHAIN_FILE) + list(APPEND _options "-DCMAKE_TOOLCHAIN_FILE=${IPPL_LAPACKE_TOOLCHAIN_FILE}") + endif() + message(STATUS "Host LAPACKE not found: building reference LAPACK 3.12.1 (C/Fortran); logs in ${_build}") + file(MAKE_DIRECTORY "${_build}") + execute_process(COMMAND "${CMAKE_COMMAND}" + -S "${CMAKE_CURRENT_FUNCTION_LIST_DIR}/lapack-bootstrap" -B "${_build}" + "-DIPPL_REFERENCE_LAPACK_SOURCE_DIR=${ippl_reference_lapack_SOURCE_DIR}" + "-DIPPL_LAPACK_INTEGER_BYTES=${IPPL_LAPACK_INTEGER_BYTES}" + "-DCMAKE_INSTALL_PREFIX=${_prefix}" -DCMAKE_BUILD_TYPE=Release ${_options} + RESULT_VARIABLE _result OUTPUT_FILE "${_build}/configure.log" ERROR_FILE "${_build}/configure.log") + if(NOT _result EQUAL 0) + message(FATAL_ERROR "Host LAPACKE bootstrap configure failed. A host C and Fortran compiler (e.g. gfortran) is required. See ${_build}/configure.log") + endif() + execute_process(COMMAND "${CMAKE_COMMAND}" --build "${_build}" --config Release + --parallel "${IPPL_LAPACKE_BUILD_JOBS}" + RESULT_VARIABLE _result OUTPUT_FILE "${_build}/build.log" ERROR_FILE "${_build}/build.log") + if(NOT _result EQUAL 0) + message(FATAL_ERROR "Host LAPACKE bootstrap build failed; see ${_build}/build.log") + endif() + execute_process(COMMAND "${CMAKE_COMMAND}" --install "${_build}" --config Release + RESULT_VARIABLE _result OUTPUT_FILE "${_build}/install.log" ERROR_FILE "${_build}/install.log") + if(NOT _result EQUAL 0) + message(FATAL_ERROR "Host LAPACKE bootstrap install failed; see ${_build}/install.log") + endif() + # Use this exact bootstrap, including after a change of integer ABI. + set(IPPLHostLapack_DIR "${_prefix}/lib/cmake/IPPLHostLapack") + find_package(IPPLHostLapack CONFIG REQUIRED NO_DEFAULT_PATH PATHS "${IPPLHostLapack_DIR}") + # Imported targets created inside a function retain their directory visibility. + # Bundle only the private archives, public headers and our relocatable package; + # upstream pkg-config files contain the bootstrap prefix and are not exported. + install(DIRECTORY "${_prefix}/include/ippl-lapack" DESTINATION include) + install(DIRECTORY "${_prefix}/lib/ippl-lapack/" DESTINATION lib/ippl-lapack + FILES_MATCHING PATTERN "*.a" PATTERN "*.lib" + PATTERN "cmake" EXCLUDE PATTERN "pkgconfig" EXCLUDE) + install(FILES "${_prefix}/lib/cmake/IPPLHostLapack/IPPLHostLapackConfig.cmake" + DESTINATION lib/cmake/IPPLHostLapack) + install(FILES "${_prefix}/share/ippl/LAPACK-LICENSE" DESTINATION share/ippl) + set(LAPACKE_INCLUDE_DIRS "${_prefix}/include/ippl-lapack" PARENT_SCOPE) + set(LAPACKE_LIBRARIES IPPLHostLapack::lapacke PARENT_SCOPE) + set(IPPL_HOST_LAPACK_PREFIX "${_prefix}" PARENT_SCOPE) + set(IPPL_HOST_LAPACK_FETCHED ON PARENT_SCOPE) +endfunction() diff --git a/cmake/FindTPLLAPACKE.cmake b/cmake/FindTPLLAPACKE.cmake index 2346f4ffcf..3ce3c0973e 100644 --- a/cmake/FindTPLLAPACKE.cmake +++ b/cmake/FindTPLLAPACKE.cmake @@ -4,7 +4,9 @@ include(FindPackageHandleStandardArgs) find_package_handle_standard_args(TPLLAPACKE REQUIRED_VARS LAPACKE_INCLUDE_DIRS LAPACKE_LIBRARIES) set(_lapackeResolved) foreach(_library IN LISTS LAPACKE_LIBRARIES) - if(IS_ABSOLUTE "${_library}") + if(TARGET "${_library}") + list(APPEND _lapackeResolved "${_library}") + elseif(IS_ABSOLUTE "${_library}") if(NOT EXISTS "${_library}") message(FATAL_ERROR "LAPACKE dependency does not exist: ${_library}") endif() @@ -15,5 +17,18 @@ foreach(_library IN LISTS LAPACKE_LIBRARIES) list(APPEND _lapackeResolved "${_lapackeLibrary}") endif() endforeach() +set(_lapackeIncludes "${LAPACKE_INCLUDE_DIRS}") +if(IPPL_HOST_LAPACK_FETCHED) + # The imported host target carries relocatable include directories itself. + set(_lapackeIncludes) +endif() kokkoskernels_create_imported_tpl(LAPACKE INTERFACE - LINK_LIBRARIES "${_lapackeResolved}" INCLUDES "${LAPACKE_INCLUDE_DIRS}") + LINK_LIBRARIES "${_lapackeResolved}" INCLUDES "${_lapackeIncludes}") + +if(IPPL_HOST_LAPACK_FETCHED) + # Also make the Kernels package usable directly, including in a fresh IPPL + # build that finds this Kernels installation instead of fetching its sources. + kokkoskernels_append_config_line("include(CMakeFindDependencyMacro)") + kokkoskernels_append_config_line( + "find_dependency(IPPLHostLapack CONFIG HINTS \"\\\${PACKAGE_PREFIX_DIR}\" \"${IPPL_HOST_LAPACK_PREFIX}\")") +endif() diff --git a/cmake/IPPLConfig.cmake.in b/cmake/IPPLConfig.cmake.in index f50e6e3c32..ab10e60e1f 100644 --- a/cmake/IPPLConfig.cmake.in +++ b/cmake/IPPLConfig.cmake.in @@ -20,6 +20,14 @@ if(NOT TARGET Kokkos::kokkos) endif() endif() +if("@IPPL_HOST_LAPACK_FETCHED@" AND NOT TARGET IPPLHostLapack::lapacke) + if(_ipplBuildTree) + find_dependency(IPPLHostLapack CONFIG HINTS "@IPPL_HOST_LAPACK_PREFIX@") + else() + find_dependency(IPPLHostLapack CONFIG HINTS "${_ipplPrefix}") + endif() +endif() + if(@IPPL_ENABLE_KOKKOS_KERNELS@ AND NOT TARGET Kokkos::kokkoskernels) if("@IPPL_KOKKOS_KERNELS_NEEDS_MKL@") find_dependency(MKL CONFIG) diff --git a/cmake/KokkosKernelsDependency.cmake b/cmake/KokkosKernelsDependency.cmake index aab845251c..c9f0191b69 100644 --- a/cmake/KokkosKernelsDependency.cmake +++ b/cmake/KokkosKernelsDependency.cmake @@ -3,6 +3,8 @@ include(CheckCXXSourceCompiles) include(CheckCXXSourceRuns) include(CMakePushCheckState) +option(IPPL_FETCH_LAPACKE "Build reference LAPACKE when no installation is found" ON) +set(IPPL_HOST_LAPACK_FETCHED OFF) set(KokkosKernels_VERSION "5.2.0" CACHE STRING "Kokkos Kernels release or git.tag/sha") set(IPPL_KOKKOS_KERNELS_HOST "LAPACKE" CACHE STRING "Host eigenanalysis provider: LAPACKE, MKL, NONE") @@ -34,20 +36,29 @@ if(NOT TARGET Kokkos::kokkoskernels) set(KokkosKernels_ENABLE_TPL_LAPACKE ON CACHE BOOL "Host LAPACKE" FORCE) find_path(IPPL_LAPACKE_INCLUDE_DIR lapacke.h HINTS ${LAPACKE_INCLUDE_DIRS} ${LAPACKE_ROOT}/include $ENV{LAPACKE_ROOT}/include) - if(NOT IPPL_LAPACKE_INCLUDE_DIR) - message(FATAL_ERROR "Host LAPACKE headers are required, including for CUDA/HIP. Set LAPACKE_ROOT or LAPACKE_INCLUDE_DIRS, select IPPL_KOKKOS_KERNELS_HOST=MKL, or use NONE to disable eigenanalysis.") - endif() - set(LAPACKE_INCLUDE_DIRS "${IPPL_LAPACKE_INCLUDE_DIR}") - if(NOT LAPACKE_LIBRARIES) + if(LAPACKE_LIBRARIES) + # An explicit link line is authoritative. Never mix it with fetched headers. + if(NOT IPPL_LAPACKE_INCLUDE_DIR) + message(FATAL_ERROR "LAPACKE_LIBRARIES was supplied but lapacke.h was not found. Set LAPACKE_INCLUDE_DIRS or LAPACKE_ROOT, or clear LAPACKE_LIBRARIES to permit the fallback.") + endif() + set(LAPACKE_INCLUDE_DIRS "${IPPL_LAPACKE_INCLUDE_DIR}") + else() find_library(IPPL_LAPACKE_LIBRARY NAMES lapacke openblas HINTS ${LAPACKE_LIBRARY_DIRS} ${LAPACKE_ROOT}/lib ${LAPACKE_ROOT}/lib64 $ENV{LAPACKE_ROOT}/lib $ENV{LAPACKE_ROOT}/lib64) - if(NOT IPPL_LAPACKE_LIBRARY) - message(FATAL_ERROR "Host LAPACKE library missing. Set LAPACKE_ROOT or LAPACKE_LIBRARIES (including static transitive dependencies).") + if(IPPL_LAPACKE_INCLUDE_DIR AND IPPL_LAPACKE_LIBRARY) + set(BLA_SIZEOF_INTEGER "${IPPL_LAPACK_INTEGER_BYTES}") + find_package(LAPACK QUIET) + endif() + if(IPPL_LAPACKE_INCLUDE_DIR AND IPPL_LAPACKE_LIBRARY AND LAPACK_FOUND) + set(LAPACKE_INCLUDE_DIRS "${IPPL_LAPACKE_INCLUDE_DIR}") + set(LAPACKE_LIBRARIES "${IPPL_LAPACKE_LIBRARY};${LAPACK_LIBRARIES}") + elseif(IPPL_FETCH_LAPACKE) + include(FetchHostLapack) + ippl_fetch_host_lapack() + else() + message(FATAL_ERROR "Host LAPACKE was not found and IPPL_FETCH_LAPACKE=OFF. Set LAPACKE_ROOT or supply LAPACKE_INCLUDE_DIRS and LAPACKE_LIBRARIES.") endif() - set(BLA_SIZEOF_INTEGER "${IPPL_LAPACK_INTEGER_BYTES}") - find_package(LAPACK REQUIRED) - set(LAPACKE_LIBRARIES "${IPPL_LAPACKE_LIBRARY};${LAPACK_LIBRARIES}") endif() elseif(IPPL_KOKKOS_KERNELS_HOST STREQUAL "MKL") set(KokkosKernels_ENABLE_TPL_MKL ON CACHE BOOL "Host MKL" FORCE) @@ -76,7 +87,23 @@ if(NOT TARGET Kokkos::kokkoskernels) FetchContent_Declare(KokkosKernels GIT_REPOSITORY https://github.com/kokkos/kokkos-kernels.git GIT_TAG ${_ipplKernelsGit}) + # Upstream uses pre-CMP0126 cache semantics and otherwise replaces inherited + # discovery results with empty cache defaults. Seed them only for its configure, + # then restore the user's cache so automatic fallback values do not become + # authoritative user overrides on the next configure. + foreach(_variable LAPACKE_INCLUDE_DIRS LAPACKE_LIBRARIES) + get_property(_saved${_variable} CACHE ${_variable} PROPERTY VALUE) + get_property(_had${_variable} CACHE ${_variable} PROPERTY TYPE SET) + set(${_variable} "${${_variable}}" CACHE STRING "Host LAPACKE dependency" FORCE) + endforeach() FetchContent_MakeAvailable(KokkosKernels) + foreach(_variable LAPACKE_INCLUDE_DIRS LAPACKE_LIBRARIES) + if(_had${_variable}) + set(${_variable} "${_saved${_variable}}" CACHE STRING "Host LAPACKE dependency" FORCE) + else() + unset(${_variable} CACHE) + endif() + endforeach() # Upstream installs an export but does not provide a build-tree export. export(EXPORT KokkosKernelsTargets FILE "${kokkoskernels_BINARY_DIR}/KokkosKernelsTargets.cmake" NAMESPACE Kokkos::) @@ -152,9 +179,10 @@ if(NOT IPPL_KOKKOS_KERNELS_HOST STREQUAL "NONE") add_library(ippl_lapacke_probe INTERFACE IMPORTED) get_target_property(_hostLinks LAPACKE INTERFACE_LINK_LIBRARIES) get_target_property(_hostIncludes LAPACKE INTERFACE_INCLUDE_DIRECTORIES) - set_target_properties(ippl_lapacke_probe PROPERTIES - INTERFACE_LINK_LIBRARIES "${_hostLinks}" - INTERFACE_INCLUDE_DIRECTORIES "${_hostIncludes}") + set_target_properties(ippl_lapacke_probe PROPERTIES INTERFACE_LINK_LIBRARIES "${_hostLinks}") + if(_hostIncludes) + set_target_properties(ippl_lapacke_probe PROPERTIES INTERFACE_INCLUDE_DIRECTORIES "${_hostIncludes}") + endif() set(CMAKE_REQUIRED_LIBRARIES ippl_lapacke_probe) else() set(CMAKE_REQUIRED_LIBRARIES Kokkos::kokkoskernels) diff --git a/cmake/lapack-bootstrap/CMakeLists.txt b/cmake/lapack-bootstrap/CMakeLists.txt new file mode 100644 index 0000000000..f93fdeba44 --- /dev/null +++ b/cmake/lapack-bootstrap/CMakeLists.txt @@ -0,0 +1,50 @@ +cmake_minimum_required(VERSION 3.24) +# A separate host project prevents IPPL's CUDA/HIP launchers and flags from +# reaching reference LAPACK's C/Fortran compilation. +project(IPPLReferenceLapack LANGUAGES C Fortran) +set(BUILD_TESTING OFF CACHE BOOL "" FORCE) +set(BUILD_SHARED_LIBS OFF CACHE BOOL "" FORCE) +set(CMAKE_POSITION_INDEPENDENT_CODE ON) +set(LIBRARY_PREFIX ippl_) +set(CMAKE_INSTALL_LIBDIR lib/ippl-lapack CACHE STRING "" FORCE) +set(CMAKE_INSTALL_INCLUDEDIR include/ippl-lapack CACHE STRING "" FORCE) +set(LAPACKE ON CACHE BOOL "" FORCE) +set(CBLAS OFF CACHE BOOL "" FORCE) +set(LAPACKE_WITH_TMG OFF CACHE BOOL "" FORCE) +set(BUILD_INDEX64_EXT_API OFF CACHE BOOL "" FORCE) +set(USE_OPTIMIZED_BLAS OFF CACHE BOOL "" FORCE) +set(USE_OPTIMIZED_LAPACK OFF CACHE BOOL "" FORCE) +set(BUILD_DEPRECATED OFF CACHE BOOL "" FORCE) +if(IPPL_LAPACK_INTEGER_BYTES STREQUAL "8") + set(BUILD_INDEX64 ON CACHE BOOL "" FORCE) + set(_suffix 64) +else() + set(BUILD_INDEX64 OFF CACHE BOOL "" FORCE) + set(_suffix "") +endif() +add_subdirectory("${IPPL_REFERENCE_LAPACK_SOURCE_DIR}" lapack) + +# Consumers need not enable Fortran just to link the static host libraries. +set(IPPL_LAPACK_RUNTIME_LIBRARIES) +foreach(_library IN LISTS CMAKE_Fortran_IMPLICIT_LINK_LIBRARIES) + if(_library IN_LIST CMAKE_C_IMPLICIT_LINK_LIBRARIES) + continue() + endif() + unset(_runtimeLibrary) + find_library(_runtimeLibrary NAMES ${_library} + HINTS ${CMAKE_Fortran_IMPLICIT_LINK_DIRECTORIES} REQUIRED NO_CACHE) + list(APPEND IPPL_LAPACK_RUNTIME_LIBRARIES "${_runtimeLibrary}") +endforeach() +set(IPPL_LAPACKE_FILE "${CMAKE_STATIC_LIBRARY_PREFIX}lapacke${_suffix}${CMAKE_STATIC_LIBRARY_SUFFIX}") +set(IPPL_LAPACK_FILE "${CMAKE_STATIC_LIBRARY_PREFIX}lapack${_suffix}${CMAKE_STATIC_LIBRARY_SUFFIX}") +set(IPPL_BLAS_FILE "${CMAKE_STATIC_LIBRARY_PREFIX}blas${_suffix}${CMAKE_STATIC_LIBRARY_SUFFIX}") +# LIBRARY_PREFIX is applied inside the reference subdirectory, not this scope. +foreach(_kind LAPACKE LAPACK BLAS) + string(REPLACE "${CMAKE_STATIC_LIBRARY_PREFIX}" "${CMAKE_STATIC_LIBRARY_PREFIX}ippl_" + IPPL_${_kind}_FILE "${IPPL_${_kind}_FILE}") +endforeach() +configure_file(IPPLHostLapackConfig.cmake.in IPPLHostLapackConfig.cmake @ONLY) +install(FILES "${CMAKE_CURRENT_BINARY_DIR}/IPPLHostLapackConfig.cmake" + DESTINATION lib/cmake/IPPLHostLapack) +install(FILES "${IPPL_REFERENCE_LAPACK_SOURCE_DIR}/LICENSE" + DESTINATION share/ippl RENAME LAPACK-LICENSE) diff --git a/cmake/lapack-bootstrap/IPPLHostLapackConfig.cmake.in b/cmake/lapack-bootstrap/IPPLHostLapackConfig.cmake.in new file mode 100644 index 0000000000..cd35e6a64f --- /dev/null +++ b/cmake/lapack-bootstrap/IPPLHostLapackConfig.cmake.in @@ -0,0 +1,11 @@ +if(NOT TARGET IPPLHostLapack::lapacke) + get_filename_component(_ipplLapackPrefix "${CMAKE_CURRENT_LIST_DIR}/../../.." ABSOLUTE) + add_library(IPPLHostLapack::lapacke INTERFACE IMPORTED) + set_target_properties(IPPLHostLapack::lapacke PROPERTIES + INTERFACE_INCLUDE_DIRECTORIES "${_ipplLapackPrefix}/include/ippl-lapack" + INTERFACE_LINK_LIBRARIES "${_ipplLapackPrefix}/lib/ippl-lapack/@IPPL_LAPACKE_FILE@;${_ipplLapackPrefix}/lib/ippl-lapack/@IPPL_LAPACK_FILE@;${_ipplLapackPrefix}/lib/ippl-lapack/@IPPL_BLAS_FILE@;@IPPL_LAPACK_RUNTIME_LIBRARIES@") + if("@IPPL_LAPACK_INTEGER_BYTES@" STREQUAL "8") + set_property(TARGET IPPLHostLapack::lapacke PROPERTY INTERFACE_COMPILE_DEFINITIONS LAPACK_ILP64) + endif() + unset(_ipplLapackPrefix) +endif() From f694ba05c30e04b625232bbd72337902096228d5 Mon Sep 17 00:00:00 2001 From: Andreas Adelmann Date: Tue, 8 Sep 2026 22:10:03 +0200 Subject: [PATCH 3/6] I fixed FindTPLLAPACKE.cmake Preserve linker options beginning with -. A clean configure using an explicit LAPACKE link line containing -lm now passes, including the native eigenvalue probes. All five linear-map tests pass locally. --- cmake/FindTPLLAPACKE.cmake | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/cmake/FindTPLLAPACKE.cmake b/cmake/FindTPLLAPACKE.cmake index 3ce3c0973e..b22b66c1a9 100644 --- a/cmake/FindTPLLAPACKE.cmake +++ b/cmake/FindTPLLAPACKE.cmake @@ -6,6 +6,10 @@ set(_lapackeResolved) foreach(_library IN LISTS LAPACKE_LIBRARIES) if(TARGET "${_library}") list(APPEND _lapackeResolved "${_library}") + elseif(_library MATCHES "^-") + # CMake LAPACK providers may return linker items such as -lm or -pthread. + # Preserve them verbatim instead of treating them as find_library names. + list(APPEND _lapackeResolved "${_library}") elseif(IS_ABSOLUTE "${_library}") if(NOT EXISTS "${_library}") message(FATAL_ERROR "LAPACKE dependency does not exist: ${_library}") From e0412bf200688212624f3cef7047455d1a0545c5 Mon Sep 17 00:00:00 2001 From: Andreas Adelmann Date: Tue, 8 Sep 2026 22:29:38 +0200 Subject: [PATCH 4/6] LAPACKE headers are now system includes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Placing Kokkos’s bundled desul headers first. I reproduced the CSCS collision locally with an intentionally invalid desul header beside lapacke.h; Kokkos Kernels still compiled successfully. --- cmake/FindTPLLAPACKE.cmake | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/cmake/FindTPLLAPACKE.cmake b/cmake/FindTPLLAPACKE.cmake index b22b66c1a9..d78323a457 100644 --- a/cmake/FindTPLLAPACKE.cmake +++ b/cmake/FindTPLLAPACKE.cmake @@ -27,7 +27,12 @@ if(IPPL_HOST_LAPACK_FETCHED) set(_lapackeIncludes) endif() kokkoskernels_create_imported_tpl(LAPACKE INTERFACE - LINK_LIBRARIES "${_lapackeResolved}" INCLUDES "${_lapackeIncludes}") + LINK_LIBRARIES "${_lapackeResolved}") +if(_lapackeIncludes) + # Environment views may also contain an incompatible desul installation. + # Keep their broad include directory behind Kokkos's bundled TPL headers. + target_include_directories(LAPACKE SYSTEM INTERFACE ${_lapackeIncludes}) +endif() if(IPPL_HOST_LAPACK_FETCHED) # Also make the Kernels package usable directly, including in a fresh IPPL From 45083e80336a34e6fc904f5188b4f6bd70a1acdc Mon Sep 17 00:00:00 2001 From: Andreas Adelmann Date: Wed, 9 Sep 2026 20:23:33 +0200 Subject: [PATCH 5/6] default Kokkos Kernels are OFF --- CMakeLists.txt | 2 +- README.md | 5 +++-- cmake/BuildSystem.md | 13 +++++++------ 3 files changed, 11 insertions(+), 9 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 78460ccbe4..445955dcec 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -35,7 +35,7 @@ include(ProjectSetup) set(IPPL_PLATFORMS "SERIAL" CACHE STRING "Platforms to build IPPL for") option(BUILD_SHARED_LIBS "Build IPPL as a shared library" OFF) option(IPPL_ENABLE_UNIT_TESTS "Enable unit tests using GoogleTest" OFF) -option(IPPL_ENABLE_KOKKOS_KERNELS "Enable Kokkos Kernels linear algebra support" ON) +option(IPPL_ENABLE_KOKKOS_KERNELS "Enable Kokkos Kernels linear algebra support" OFF) option(IPPL_ENABLE_FFT "Enable FFT support" OFF) option(IPPL_ENABLE_CUFFTMP "Enable NVIDIA cuFFTMp FFT backend" OFF) option(IPPL_ENABLE_FINUFFT "Enable FINUFFT NUFFT backend" OFF) diff --git a/README.md b/README.md index e96cf16eb7..360b47aea2 100644 --- a/README.md +++ b/README.md @@ -45,10 +45,11 @@ The [IPPL Manual](https://ippl-framework.github.io/Manual/) contains comprehensi For detailed API, class, and file documentation, please visit our [Doxygen site](https://ippl-framework.github.io/ippl/). -Kokkos Kernels is enabled by default. Missing host LAPACKE is built from source +Kokkos Kernels support is optional and can be enabled with +`-DIPPL_ENABLE_KOKKOS_KERNELS=ON`. Missing host LAPACKE is then built from source using C and Fortran compilers; installed LAPACKE or MKL providers are also supported. See [Kokkos Kernels configuration](cmake/BuildSystem.md#kokkos-kernels-and-host-eigenanalysis) -for dependency setup and options to disable host eigenanalysis or Kokkos Kernels. +for dependency setup and options controlling host eigenanalysis. **Feedback & Support** If you find any issues with the manual, please report them in the [Manual's GitHub repository](https://github.com/IPPL-framework/Manual). diff --git a/cmake/BuildSystem.md b/cmake/BuildSystem.md index 337997ed74..3dbd1af1aa 100644 --- a/cmake/BuildSystem.md +++ b/cmake/BuildSystem.md @@ -175,11 +175,12 @@ target_link_libraries(app PRIVATE ippl::ippl) ## Kokkos Kernels and host eigenanalysis -`IPPL_ENABLE_KOKKOS_KERNELS` defaults to `ON`. IPPL first finds an installed -Kokkos Kernels package (default minimum version 5.2.0), then falls back to -FetchContent. `KokkosKernels_VERSION=git.` requests a source build. -The package uses the Kokkos target already selected by IPPL. An external package -must have been built against a compatible Kokkos with the required backends. +`IPPL_ENABLE_KOKKOS_KERNELS` defaults to `OFF`. Set it to `ON` to enable this +support. IPPL first finds an installed Kokkos Kernels package (default minimum +version 5.2.0), then falls back to FetchContent. +`KokkosKernels_VERSION=git.` requests a source build. The package uses +the Kokkos target already selected by IPPL. An external package must have been +built against a compatible Kokkos with the required backends. Host eigenanalysis is selected independently of `IPPL_PLATFORMS`: @@ -192,7 +193,7 @@ Host eigenanalysis is selected independently of `IPPL_PLATFORMS`: `CMAKE_PREFIX_PATH`/`MKL_DIR` (modern oneMKL). - `IPPL_KOKKOS_KERNELS_HOST=NONE`: use portable/GPU kernels without requesting host eigenanalysis; only the GEMM regression is registered in the test binary. -- `IPPL_ENABLE_KOKKOS_KERNELS=OFF`: omit the dependency and its tests entirely. +- `IPPL_ENABLE_KOKKOS_KERNELS=OFF` (default): omit the dependency and its tests entirely. Kokkos does not install these external libraries. If host LAPACKE is not found, IPPL now downloads reference LAPACK 3.12.1 (SHA256-verified) and builds its BLAS, From 274fa5ec5555ddd0d4e9a3b6be8935e1b5d1bfd3 Mon Sep 17 00:00:00 2001 From: Andreas Adelmann Date: Wed, 9 Sep 2026 20:28:26 +0200 Subject: [PATCH 6/6] Document Kokkos Kernels configuration options Describe the opt-in setting, host LAPACKE and MKL providers, LP64 and ILP64 selection, source-build fallback, offline FetchContent overrides, compiler selection, and automatic CUDA and HIP TPL configuration. --- cmake/BuildSystem.md | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/cmake/BuildSystem.md b/cmake/BuildSystem.md index 3dbd1af1aa..1c80f4e172 100644 --- a/cmake/BuildSystem.md +++ b/cmake/BuildSystem.md @@ -182,6 +182,41 @@ version 5.2.0), then falls back to FetchContent. the Kokkos target already selected by IPPL. An external package must have been built against a compatible Kokkos with the required backends. +The following CMake cache variables control this support: + +- `IPPL_ENABLE_KOKKOS_KERNELS` (`OFF`): enable Kokkos Kernels and its IPPL unit + test. The remaining variables in this list are used only when this is `ON`. +- `KokkosKernels_VERSION` (`5.2.0`): required package version. Prefix the value + with `git.` to request a specific source tag, branch, or commit. +- `IPPL_KOKKOS_KERNELS_HOST` (`LAPACKE`): select `LAPACKE`, `MKL`, or `NONE` for + host eigenanalysis. +- `IPPL_LAPACK_INTEGER_BYTES` (`4`): select the 4-byte LP64 or 8-byte ILP64 + integer interface. +- `IPPL_FETCH_LAPACKE` (`ON`): build reference LAPACK and LAPACKE when the + `LAPACKE` provider cannot be found. +- `IPPL_LAPACKE_BUILD_JOBS` (`4`): parallel jobs for the reference LAPACK build. + This cache entry is created only when the fallback is needed. +- `IPPL_LAPACKE_TOOLCHAIN_FILE` (empty): C/Fortran toolchain used by the fallback, + required when cross-compiling. This entry is also created only when needed. + +Installed dependencies can be selected with `KokkosKernels_DIR`, `LAPACKE_ROOT`, +`LAPACKE_INCLUDE_DIRS`, `LAPACKE_LIBRARY_DIRS`, `LAPACKE_LIBRARIES`, or `MKL_DIR`. +`LAPACKE_LIBRARIES` is a semicolon-separated complete link line and may contain +absolute paths, library names, imported targets, and linker items such as `-lm`. +The standard FetchContent overrides `FETCHCONTENT_SOURCE_DIR_KOKKOSKERNELS` and +`FETCHCONTENT_SOURCE_DIR_IPPL_REFERENCE_LAPACK` support offline source trees. +Set `CMAKE_Fortran_COMPILER` or the `FC` environment variable to choose the +fallback Fortran compiler; use `CC` to choose its C compiler. + +The minimum opt-in configuration is: + +```console +cmake -S . -B build -DIPPL_ENABLE_KOKKOS_KERNELS=ON +``` + +CUDA and HIP BLAS, sparse, and solver TPLs are selected automatically from +`IPPL_PLATFORMS`; they do not require additional IPPL cache variables. + Host eigenanalysis is selected independently of `IPPL_PLATFORMS`: - `IPPL_KOKKOS_KERNELS_HOST=LAPACKE` (default): provide LAPACKE headers and a