From 8ab07e6a9582ca819b94c3daabf11a8859f50848 Mon Sep 17 00:00:00 2001 From: Hugo Meiland <32266151+hmeiland@users.noreply.github.com> Date: Sat, 11 Jul 2026 09:38:44 +0200 Subject: [PATCH] Add blog post: a 1.7x faster HPL on a RISC-V SiFive U74 via EESSI --- docs/blog/.authors.yml | 5 + .../posts/2026/07/risc-v-u74-openblas-hpl.md | 285 ++++++++++++++++++ 2 files changed, 290 insertions(+) create mode 100644 docs/blog/posts/2026/07/risc-v-u74-openblas-hpl.md diff --git a/docs/blog/.authors.yml b/docs/blog/.authors.yml index 3d6212a572..1617a9eec2 100644 --- a/docs/blog/.authors.yml +++ b/docs/blog/.authors.yml @@ -129,3 +129,8 @@ authors: description: Jülich Supercomputing Centre avatar: https://avatars.githubusercontent.com/u/15690556?v=4 slug: https://github.com/SebastianAchilles + hmeiland: + name: Hugo Meiland + description: Microsoft + avatar: https://avatars.githubusercontent.com/u/32266151?v=4 + slug: https://github.com/hmeiland diff --git a/docs/blog/posts/2026/07/risc-v-u74-openblas-hpl.md b/docs/blog/posts/2026/07/risc-v-u74-openblas-hpl.md new file mode 100644 index 0000000000..5b6b59776d --- /dev/null +++ b/docs/blog/posts/2026/07/risc-v-u74-openblas-hpl.md @@ -0,0 +1,285 @@ +--- +authors: [hmeiland] +date: 2026-07-11 +slug: risc-v-u74-openblas-hpl +hide: + - toc +--- + +# A 1.7× faster HPL on a RISC-V SiFive U74, delivered through EESSI + +*A fresh-install, end-to-end walkthrough: from a bare VisionFive 2 to a ~1.7× faster +High-Performance Linpack run, using nothing but EESSI and a single `eb --from-pr`.* + + + +## TL;DR + +- The StarFive VisionFive 2 (SiFive **U74**, `rv64gc`) runs stock OpenBLAS at roughly + **3 GFLOP/s** on HPL — it falls back to a generic C GEMM kernel. +- A hand-tuned scalar **4×4 DGEMM micro-kernel** for the U74 lifts that to **~5.3 GFLOP/s + (~1.7× on HPL**; the raw DGEMM kernel itself is roughly **2×** faster). +- The kernel is upstreamed to OpenBLAS ([OpenMathLib/OpenBLAS#5903]) and packaged for + EESSI as an EasyBuild easyconfig ([easybuilders/easybuild-easyconfigs#26436]). +- This post reproduces the **whole thing from a clean machine**: set up EESSI, measure a + baseline HPL, build the optimized OpenBLAS straight from the PR with `eb --from-pr`, + load it as a module, and re-run HPL — **without recompiling HPL**. + +Everything below was run on a real VisionFive 2. Total wall-clock: a few minutes of +commands plus one longer `eb` build. + +## The board and the gap + +The [VisionFive 2](https://www.starfive.tech/en/site/boards) is a ~$70 single-board +computer built on the StarFive **JH7110**: four **SiFive U74** cores at 1.5 GHz, +`rv64gc` (RVA20-class, scalar — *no* vector extension), 8 GB RAM, running Ubuntu 24.04. + +Out of the box, OpenBLAS has no U74-specific GEMM kernel, so it selects the generic +`RISCV64_GENERIC` path — a portable C `2×2` kernel. It works, but it leaves a lot of the +(admittedly modest) FP throughput on the table. On HPL that lands around **3 GFLOP/s** +across the four cores. + +The optimization is a classic register-blocked GEMM micro-kernel, written in RV64 +assembly and tuned specifically for the U74's in-order, single-FP-pipe, one-load-port +pipeline: a software-pipelined **4×4** inner kernel with operand double-buffering and +load-before-FMA scheduling. On this core it lifts single-core DGEMM from ~1.4 to +**1.77 GFLOP/s** and, combined with a re-tuned block size, lifts HPL by **~1.7×** (the raw +4-core DGEMM rate is ~2× the generic kernel). (The full design write-up is in the +companion paper; this post is purely about *reproducing the result*.) + +The interesting part for practitioners isn't just the kernel — it's that you can get it +onto a machine through the **normal scientific-software supply chain**: EESSI provides the +stack, and EasyBuild pulls the not-yet-released kernel straight from a pull request. + +## What you need + +- A SiFive U74 / `rv64gc` RISC-V board (VisionFive 2 here), ~8 GB RAM, Ubuntu 24.04. +- [CVMFS](https://cvmfs.readthedocs.io/) + the [EESSI](https://www.eessi.io/) software + stack, specifically the RISC-V development repository + [`dev.eessi.io/riscv`][eessi-riscv]. + +That's it — no compilers, no MPI, no BLAS to install by hand. EESSI provides all of it. + +## Step 0 — Get EESSI + +If CVMFS + EESSI aren't already on the box, install them once. A CVMFS client for RISC-V is still +experimental — it's tracked in [cvmfs/cvmfs#3441][cvmfs-riscv-issue], with Ubuntu/riscv64 test +`.deb`s published at [`vavolkl/cvmfs-riscv`][cvmfs-riscv-rel] (release `cvmfs-2.14.0-riscv-test`). +Fetch the riscv64 client packages from there, add EESSI's architecture-independent config, and set up: + +```bash +# CVMFS RISC-V client — experimental test packages (see cvmfs/cvmfs#3441) +rel=https://github.com/vavolkl/cvmfs-riscv/releases/download/cvmfs-2.14.0-riscv-test +wget $rel/cvmfs_2.14.0+ubuntu24.04_riscv64.deb \ + $rel/cvmfs-libs_2.14.0+ubuntu24.04_riscv64.deb \ + $rel/cvmfs-fuse3_2.14.0+ubuntu24.04_riscv64.deb \ + $rel/cvmfs-config-default_2.2_all.deb +sudo apt install -y ./cvmfs*.deb +# EESSI CVMFS configuration (architecture-independent, from EESSI): +sudo dpkg -i cvmfs-config-eessi_*_all.deb +sudo cvmfs_config setup +``` + +Then initialize the EESSI environment. The RISC-V stack currently lives under a version +override: + +```bash +export EESSI_VERSION_OVERRIDE=2025.06-001 +source /cvmfs/software.eessi.io/versions/2025.06/init/lmod/bash +``` + +You should now be in the EESSI environment, targeting `riscv64/generic`: + +``` +$ echo $EESSI_SOFTWARE_SUBDIR +riscv64/generic +``` + +Everything from here on is `module load` + EasyBuild. + +## Step 1 — Baseline HPL with the stock stack + +EESSI ships a ready-to-run HPL. Load it — this pulls in HPL, OpenMPI, and FlexiBLAS +(whose default backend is the generic OpenBLAS 0.3.30): + +```bash +module load HPL/2.3-foss-2025b +``` + +Create a small problem (N=10000, NB=192, 2×2 process grid — one MPI rank per core): + +```bash +mkdir -p ~/hpl-run && cd ~/hpl-run +# HPL.dat: 1 problem size N=10000, NB=192, P=2, Q=2 (see the HPL docs for the full file) +``` + +Run it as pure MPI (one thread per rank): + +```bash +OMP_NUM_THREADS=1 mpirun -np 4 xhpl | grep -A1 '^WR' +``` + +On the VisionFive 2 this yields the baseline: + +``` +================================================================================ +T/V N NB P Q Time Gflops +-------------------------------------------------------------------------------- +WR11C2R4 10000 192 2 2 213.34 3.1257e+00 +================================================================================ +``` + +**~3.13 GFLOP/s** — the generic-kernel starting point. (`WR…` = the residual check +passed.) + +## Step 2 — Build the U74-optimized OpenBLAS *straight from the PR* + +Here's the payoff of doing this through EasyBuild: the optimized OpenBLAS isn't released +yet, but it lives in easyconfigs PR **#26436**, and `eb --from-pr` builds directly from it. + +Load EasyBuild, and set up an **EESSI-extend** user installation so the new module lands +in your home directory alongside (and reusing) the CVMFS stack: + +```bash +module load EasyBuild/5.3.0 + +mkdir -p $HOME/eessi-u74 # must exist BEFORE loading EESSI-extend +export EESSI_USER_INSTALL=$HOME/eessi-u74 +module load EESSI-extend/2025.06-easybuild +``` + +Now build the easyconfig from the PR: + +```bash +eb --from-pr 26436 --robot +``` + +`--from-pr` fetches `OpenBLAS-0.3.30-GCC-14.3.0-U74.eb` **and** its source patch from the +pull request, applies the patch (which adds the `U74` target and the 4×4 kernel to the +OpenBLAS source tree), and builds with `TARGET=U74 DYNAMIC_ARCH=0`. It reuses the +GCC/14.3.0 toolchain, Python, etc. already in CVMFS, and installs the result as the module +`OpenBLAS/0.3.30-GCC-14.3.0-U74` under `$EESSI_USER_INSTALL`. + +> **Note for the `dev.eessi.io/riscv` stack.** On that experimental stack, GCC 14.3.0's +> auto-optarch can emit a canonical-expanded `-march` (`…zaamo_zalrsc…`) that the very same +> compiler then rejects, breaking OpenBLAS's `getarch` check. It's a quirk of that +> experimental toolchain, *not* of the easyconfig or patch. Fix it by pinning a valid +> `-march` via `EASYBUILD_OPTARCH`: +> +> ```bash +> export EASYBUILD_OPTARCH='-march=rv64imafdc_zba_zbb' # valid scalar march for the U74 +> eb --from-pr 26436 --robot +> ``` + +## Step 3 — Load the module and re-run HPL (no HPL rebuild!) + +This is the neat trick. HPL was built against **FlexiBLAS**, which lets you switch the +BLAS backend at *runtime* — so we don't rebuild HPL at all. + +First, make the user-installed module visible and load it (module system, no hard-coded +paths): + +```bash +module use $EESSI_USER_INSTALL/versions/2025.06-001/software/linux/riscv64/generic/modules/all +module load OpenBLAS/0.3.30-GCC-14.3.0-U74 +``` + +Lmod recognizes the OpenBLAS family and swaps the stock module out of the way: + +``` +OpenBLAS/0.3.30-GCC-14.3.0 => OpenBLAS/0.3.30-GCC-14.3.0-U74 +``` + +**A caveat worth knowing** (and the answer to an obvious question): swapping the OpenBLAS +*module* is **not** enough for HPL to pick up the new library. EESSI binaries are linked +with **RPATH**, not `LD_LIBRARY_PATH`, and FlexiBLAS's `OPENBLAS` backend has an RPATH that +hard-pins the stock `libopenblas.so.0`. Loading a different OpenBLAS module only changes +`LIBRARY_PATH` (used at *compile* time) — it doesn't change what FlexiBLAS `dlopen`s at +runtime. So you have to point FlexiBLAS at the new backend explicitly. + +The clean way — and it means you never type a library path at run time — is to register +the module's library as a FlexiBLAS backend **once**, using the module's own +`$EBROOTOPENBLAS`, and make it your default: + +```bash +flexiblas add U74 $EBROOTOPENBLAS/lib/libopenblas.so +flexiblas default U74 # writes ~/.flexiblasrc +``` + +From here on, every FlexiBLAS-linked program — HPL, but also NumPy, SciPy, … — uses the +U74 library by default. Re-run the *exact same* `xhpl`, with **no environment variable and +no path**: + +```bash +cd ~/hpl-run +OMP_NUM_THREADS=1 mpirun -np 4 xhpl | grep -A1 '^WR' +``` + +``` +WR11C2R4 10000 192 2 2 126.19 5.2841e+00 +``` + +**5.28 GFLOP/s** — same binary, same input, same command; only the registered BLAS backend +changed. + +> Prefer not to touch `~/.flexiblasrc`? Select the backend per-invocation instead — still +> via the module, not a hard-coded path: +> ```bash +> FLEXIBLAS=$EBROOTOPENBLAS/lib/libopenblas.so OMP_NUM_THREADS=1 mpirun -np 4 xhpl +> ``` + +## Results + +| Configuration | HPL (N=10000, 4 cores) | Speedup | +|------------------------------------------------|------------------------|---------| +| Stock EESSI OpenBLAS 0.3.30 (generic `rv64gc`) | 3.13 GFLOP/s (213 s) | 1.00× | +| U74-optimized OpenBLAS (PR #26436) | 5.28 GFLOP/s (126 s) | 1.69× | + +For reference, raw DGEMM on the installed U74 module: **1.77 GFLOP/s** single-core, +**6.31 GFLOP/s** on 4 cores — versus ~1.4 single-core for the generic kernel. (HPL sits +below peak DGEMM because of its non-GEMM work — panel factorization, pivoting, +communication — so the ~1.7× end-to-end gain trails the ~2× raw-kernel gain.) + +The only thing that changed between the two HPL runs is which `libopenblas.so` FlexiBLAS +loaded. That's the whole point of shipping the kernel through EESSI/EasyBuild: it drops +into an existing, unmodified toolchain. + +## Why this matters + +- **Reproducible.** No hand-built BLAS, no `LD_PRELOAD` gymnastics, no patched HPL. The + optimized library is a first-class EasyBuild module built from a reviewable PR. +- **Runtime-swappable.** FlexiBLAS means one HPL (or NumPy, or any FlexiBLAS-linked app) + can switch between the stock and tuned backends — set once as your default with + `flexiblas default`, or per-run with a single environment variable. +- **On a path to "just works".** Once [OpenMathLib/OpenBLAS#5903] merges and ships in an + OpenBLAS release, the interim source patch disappears — a stock version-bump easyconfig + will build the U74 target with nothing but `buildopts`, and `eb --from-pr` won't be + needed at all. +- **Eventually, pre-optimized by default.** The bigger prize is for `U74` to become a + recognized **CPU target** in its own right: once the target is upstream in OpenBLAS and + the kernel matures, EESSI could detect a SiFive U74 (a `riscv64/sifive/u74`-style + software subdirectory, the way it already distinguishes x86-64 microarchitectures) and + **ship a pre-optimized OpenBLAS build automatically**. At that point a VisionFive 2 user + gets the tuned library from `module load` alone — no `TARGET`, no patch, no + `--from-pr`, no FlexiBLAS backend juggling. This walkthrough is the manual version of + what should eventually be transparent. + +## Links + +- OpenBLAS source PR (the U74 target + kernel): +- EasyBuild easyconfig PR (what this post builds): +- EESSI RISC-V dev repository: + +--- + +*Reproduced end-to-end on a StarFive VisionFive 2 (JH7110, 4× SiFive U74 @ 1.5 GHz, +Ubuntu 24.04) using EESSI `2025.06-001`. The kernel, patch, and easyconfig were produced +with AI assistance — an OpenCode agent driving Claude Opus 4.8, running on Microsoft +GitHub Copilot — with a human directing the work, operating the hardware, and validating +every result, per the [EasyBuild AI policy](https://docs.easybuild.io/policies/ai/).* + +[OpenMathLib/OpenBLAS#5903]: https://github.com/OpenMathLib/OpenBLAS/pull/5903 +[easybuilders/easybuild-easyconfigs#26436]: https://github.com/easybuilders/easybuild-easyconfigs/pull/26436 +[eessi-riscv]: https://www.eessi.io/docs/repositories/dev.eessi.io-riscv/ +[cvmfs-riscv-issue]: https://github.com/cvmfs/cvmfs/issues/3441 +[cvmfs-riscv-rel]: https://github.com/vavolkl/cvmfs-riscv/releases/tag/cvmfs-2.14.0-riscv-test