Testing

Testing

wolfTrust separates host behavior, architecture-accurate emulation, and physical-hardware evidence. A result from one environment must not be reported as a result from another.

Validation layers

Environment	What it validates	What it does not validate
Native host	State machines, manifests, IPC ownership, copied transfers, services, storage, crypto integration, recovery decisions, and negative inputs	Cortex-M exception return, CMSE, SAU, MPU, GTZC, or physical flash behavior
M33MU	Cortex-M33 instruction flow, TrustZone transitions, CMSE gateway calls, MPU faults, guest scheduling, authenticated boot, and target service interactions	STM32H563 peripherals, real option bytes, WRP, ST-Link, or silicon timing
STM32H563	The actual NUCLEO-H563ZI boot chain, Secure/Non-secure attribution, faults, flash, UART, and selected end-to-end behavior	Other devices, other provisioning states, peer-flash confidentiality, or adversarial peripheral and Non-secure NVIC ownership

Host tests

Run the complete native set:

make test

The suite list is generated by:

make -s -C tests/host print-suites

Current suites cover domain and manifest validation, lifecycle, guest verification, rollback decisions, IPC and FF-M behavior, SPM policy, gateway vectors, Secure Partition layout and recovery, crypto-engine relay and key isolation, vault and storage services, attestation and COSE integration, firmware update, runtime remeasurement, linked Secure layout, VNET, public PSA headers, boot-handoff record consumption, and negative paths. The attestation IAK suite runs wolfHSM NVM with both the default 8-byte and STM32H5 16-byte flash programming units.

Additional host checks:

make test-compilers
make test-sanitize
make test-valgrind

Valgrind requires the tool on the host. make test-compilers reruns all suites with the selected CC (default cc); it does not select multiple compilers. Run it once per compiler, for example make test-compilers CC=clang. Sanitizer support depends on the local toolchain.

LTO validation

Every Secure link generates wolftrust.map and runs the linked-image layout check. Compare optimized and diagnostic builds with separate output trees:

make BUILD_DIR=build-lto size-report
make BUILD_DIR=build-no-lto WT_LTO=0 size-report

The default M33MU and hardware commands exercise the LTO image. A release-size result is valid only when the applicable target scenarios pass without a fault marker. The cross-compilation workflow rebuilds one output tree from explicit WT_LTO=0 to the default WT_LTO=1, verifies the build stamp and object formats change, checks that isolation-critical objects remain non-LTO, and requires the optimized ELF to be smaller. It also injects a forbidden heap symbol twice to prove a rejected ELF, flat binary, map, and CMSE import library are deleted instead of being reused by the next Make invocation.

The per-PR core/port split workflow builds the CONFIG_VNET=y Secure image. The M33MU smoke tier builds and runs the WT_CONFORMANCE=1 confboot layout on every PR; the VNET layout runs with the full matrix (ci:h5 label, push to main, nightly), so both optional isolation-band configurations stay under the linked-image check in CI.

PSA FF conformance

make test-conformance

This target fetches the pinned Arm PSA architecture tests. When M33MU is available it runs the target FF-M IPC suite. Without M33MU it runs only the host-side client and policy subset and prints explicit warnings. Do not treat that fallback as target conformance.

The dedicated guest configurations also exercise the PSA Crypto, Storage, and Initial Attestation validation applications. Those are target scenarios, not part of a plain make test result.

M33MU scenarios

The baseline target command is:

make test-target

The engine defaults to native. Set WT_ENGINE to exercise the same target path with either backend:

WT_ENGINE=native make test-target
WT_ENGINE=hsm make test-target

It runs the port's smoke tier (STM32H563: positive, gtzcneg, crossdomain, bothpsa, confboot, devcrypto; WT_TIER=full runs every scenario). Detection accepts m33mu on PATH or a path in M33MU. If the emulator is unavailable, the target reports a skip rather than a pass.

Additional focused runs use:

WT_ENGINE=native tests/target/run_m33mu_scenario.sh positive
WT_ENGINE=hsm tests/target/run_m33mu_scenario.sh positive

The runner's usage output is the authoritative scenario list. It includes authenticated-boot failure, rollback, runtime remeasurement, Secure Partition recovery, key and vault isolation, storage recovery, attestation negatives, firmware update, manifest rejection, GTZC behavior, and VNET paths.

VNET has convenience targets:

make test-vnet
make test-vnet-target

test-vnet is host-only. test-vnet-target launches two authenticated wolfIP guests under M33MU.

Engine matrix

The full CI scenario list contains 28 scenarios and adds engine: [native, hsm] as a matrix dimension. There are 55 applicable engine/scenario cells: every scenario runs with both engines except hsmattackneg under native.

hsmattackneg drives the raw wolfHSM protocol from a compromised-guest probe. It checks that a forged wolfHSM client ID cannot select the attestation key and that a wolfHSM NVM-group packet cannot reach the rollback store. The native engine does not link the wolfHSM client wire, server, or message handlers, so that exact attack surface does not exist there. Native key and namespace behavior remains covered by the common positive, cross-domain, keystore, storage, attestation, and Crypto-validation rows.

Validation of the engine split completed under both engines with:

• the applicable M33MU scenario matrix;
• the Arm FF-M IPC suite at 85 passed, 4 heap-dependent tests skipped, and 0 failed;
• the current dev_apis Crypto schedule at 64 passed, 13 skipped, and 0 failed (77 scheduled tests; c047 is configuration-skipped in addition to the upstream schedule); and
• the STM32H563 positive, restart, cross-domain, and conformance hardware suite.

The engine dimension changes crypto dispatch, not what M33MU proves. Emulator results still do not establish STM32 attribution or physical flash behavior.

MIMXRT700 emulator scenarios

M33MU also models the i.MX RT700 (--cpu imxrt700), so the MIMXRT700 port has an emulator gate that mirrors its hardware runner:

make test-target TARGET=mimxrt700
tests/target/run_rt700_m33mu.sh positive
tests/target/run_rt700_m33mu.sh ahbscneg

positive boots the whole chain: wolfBoot verifies the wolfTrust image, and both bare-metal guests launch, reach the SPM through the SG veneers, and finish. ahbscneg adds the CPU isolation negative: guest 0 stores into guest 1's RAM window, which the per-dispatch SAU window keeps Secure, so the SAU refuses the store; the monitor contains the fault, relaunches guest 0 through its restart budget, quarantines it, and guest 1 keeps running. The runner asserts every step from the emulator log (the guest console lines and M33MU's protection-unit trace), never from a debugger.

The runner also carries the shared Secure-verdict negatives from tests/target/lib/scenario.sh, the per-scenario table both the STM32H563 and MIMXRT700 runners draw their Secure-image probe flags from. Each one ends on the verdict breakpoint its probe emits, asserted port-independently: rollbackneg (a downgraded boot is refused fail-closed and no guest enters a domain), remeasureneg (guest 0 measures clean at launch verification, its image is then tampered in flash, and the on-demand re-measure before dispatch must catch it and quarantine the domain), manifestneg (a corrupted manifest halts boot before anything is scheduled), and spbudgetneg (restart-budget exhaustion escalates to the fail-closed platform recovery). crossdomain and keystoreneg prove the Secure Partition MPU domains: an unprivileged storage-SP read of SPM-private RAM, or of the shared keystore band, MemManage-faults at the port's band address (shown by a second traced boot), the SP's wake never serves the guests' storage connect, and the guests' own lifecycle rides it out. spfaultneg and panicneg prove Secure Partition recovery: the crypto relay runs an undefined instruction on its first entry, or the storage SP closes an error-status handle, which the SPM must panic it for. Either way the SP UsageFaults exactly once, the SPM restarts it in place, the restarted SP serves the guests that follow, and both guests finish with no escalation. restart makes guest 0 read Secure RAM on every launch: the SAU refuses it, the monitor relaunches guest 0 through its restart budget and quarantines it, and guest 1 runs on. authneg flips one byte of guest 0's image after its digest was pinned, so launch verification refuses guest 0 while guest 1 boots and runs normally.

The remaining scenarios run the portable PSA test guest (tests/firmware/psa-guest/) in both Non-secure windows. It is the STM32H563 Zephyr guest's client lifecycle with no operating system underneath: wolfPSA over the SPM-mediated client for the PSA Crypto API, the OS-neutral FF-M, storage, and attestation clients, and the wolfCOSE verifier for the token. A port supplies a linker window and a console (boards/<target>/), nothing else. bothpsa runs the whole lifecycle from both guests in one boot: the mediated SHA-256 KAT, ITS and sealed PS set/get, volatile P-256 key-ops with a cross-key refusal, RNG, AES-CTR, and an attestation token verified against the IAK public key and wolfBoot's measurement of the signed Secure image. bothiso proves the SPM rejects a forged handle, an oversized vector, a vector inside the peer guest's window, and an unknown SID from both guests. attestneg adds the attestation negatives (invalid requests refused with the statuses Arm's tests expect; tampered and misattributed tokens fail the guest verify). fwustage stages a candidate into the wolfBoot update partition through SERVICE_FWU, arms it, and proves reject/clean restore READY. hsmattackneg (hsm engine only, it drives the raw wolfHSM client wire) proves a forged client id cannot reach the IAK and an NVM-group packet never reaches the server.

The conformance scenarios are the same drop-in proof the STM32H563 gives: confboot hosts Arm's unmodified psa-arch-tests FF-M IPC suite in the PSA guest against the conformance Secure image (WT_CONFORMANCE=1, the manifest that adds Arm's server, driver, and client partitions) and expects the same 85 passed, 4 heap tests skipped, 0 failed; devstorage, devcrypto, devattest, and devattestqcbor run the dev_apis storage, crypto, and initial-attestation suites; vaultrecover proves a foreign vault pool self-heals on a development device (the crypto suite passes after the reformat), and vaultrecoversec forces the SECURED lifecycle and proves the refusal is graceful on the emulator: no fault, and the guest still starts (the vault-not-wiped and attestation-degraded counters are hardware-runner evidence, read over the debug port as on the STM32H563). The port supplies port/mimxrt700/manifest-conformance.json and port/mimxrt700/conformance/ (the PAL bindings: console, watchdog, the flash-backed NVMEM boot flag, and the isolation MMIO windows); the val framework, its PAL, and the test lists are the upstream sources the Secure build fetches and generates.

The runner builds its own pinned emulator and wolfBoot first stage. The emulator is M33MU_REF plus tests/target/m33mu-imxrt700.patch, the model correction the chain needs until it lands upstream: a Secure AHBSC SRAM rule refuses Non-secure transactions but no longer overrides the SAU's NSC attribution, so the SG veneers in the Secure-ruled code-RAM band stay callable as the SRM's transaction check allows. The rules otherwise apply to CPU0 as documented, which is stricter than the EVK measured. wolfBoot is WOLFBOOT_REF built from config/examples/imx-rt700-tz.config plus tests/target/wolfboot-imxrt700-lifecycle.patch (the RT700 HAL's PSA lifecycle hook, which the attestation service needs; it goes once wolfBoot carries it), linked at the NOR base, the same offset the wolfBoot emulator tests use, so building it needs the MCUXpresso SDK or DFP like any RT700 wolfBoot build; set RT700_WOLFBOOT_DIR to reuse an existing one (the runner stops if that tree has no wolfboot.bin, rather than replace it). WT_ENGINE selects the crypto engine as for any build. A console boot ends on the emulator's wall-clock budget because the guests idle once done, which M33MU reports as exit status 127; a Secure-verdict boot stops on its breakpoint with status 0, as does a traced boot at its first delivered fault; any other status fails the run.

These are emulator results. They prove the SAU attribution and the monitor's containment on a faithful core model; the silicon ahbscneg run on the EVK is recorded separately.

STM32H563 hardware

The published hardware run requires:

• a Linux host with Bash and GNU userland;
• Docker for the documented container build;
• Python 3 on the host for the flash-phase measurement check;
• STM32_Programmer_CLI, found at its default install path or through STM32_CLI;
• pyOCD with STM32H563 support;
• a host-visible arm-none-eabi-nm, or an override in ARM_NM;
• a serial VCP, default /dev/ttyACM0 or H5_SERIAL; and
• a connected ST-Link.

The detector checks the board/programmer path, not every required host tool. It verifies the ST-Link USB ID only when lsusb is available. Without lsusb, a missing probe can be reported later as a flash failure rather than an initial skip.

Run the default hardware set with:

WT_ENGINE=native \
WT_H5_DOCKER_IMAGE=ghcr.io/wolfssl/wolfboot-ci-m33mu:v1.15 \
make test-hardware
WT_ENGINE=hsm \
WT_H5_DOCKER_IMAGE=ghcr.io/wolfssl/wolfboot-ci-m33mu:v1.15 \
make test-hardware

The target skips if hardware detection fails. It builds and flashes the positive, restart, cross-domain, and conformance scenarios by default. The suite wrapper forwards WT_ENGINE into its build container. Select a narrower set with WT_H5_SCENARIOS:

WT_H5_SCENARIOS="positive bootupdate" \
WT_H5_DOCKER_IMAGE=ghcr.io/wolfssl/wolfboot-ci-m33mu:v1.15 \
make test-hardware

The optional gtzcneg scenario must be selected explicitly. It verifies the GTZC peer-RAM curtain after a Non-secure MPU bypass; it does not establish peer flash confidentiality or adversarial peripheral and Non-secure NVIC ownership.

For the hardened guest-flash configuration, explicitly forward the build flag into the container and repeat it for the host flash run:

docker run --rm \
    -e WT_GUEST_FLASH_WRP=1 \
    -v "$PWD":/workspace \
    -w /workspace \
    ghcr.io/wolfssl/wolfboot-ci-m33mu:v1.15 \
    bash tests/target/run_h5_hardware.sh build positive
WT_GUEST_FLASH_WRP=1 \
    tests/target/run_h5_hardware.sh flash positive

With that variable set, the flash runner clears WRP so it can program the guests, then re-applies WRP before boot. The Secure image independently reads the live WRP register and refuses an incompletely protected guest. The supported provisioning_ctrl.sh set-wrp workflow deliberately applies WRP only while the board is Open; that is a conservative project workflow, not the complete silicon rule. The hardware runner does not check product state before clearing or reapplying WRP. Current ST guidance makes WRP nonmodifiable in TZ-Closed, Closed, and Locked, and RM0481 separately defines the FLASH_WRPSGNxR.UNLOCK condition. The current suite wrapper does not forward this variable into its Docker build, so the shorter make test-hardware form does not build a WRP-enforcing Secure image.

Use the disposable container for Secure and guest builds. The current direct-host build path adds safe.directory '*' to the user's global Git configuration and is not suitable as a published workflow.

Hardware test commands write flash and can reset the board. Review STM32H5 Guide before running them.

CI coverage

The workflows under .github/workflows/ separately run:

• host unit tests;
• compiler variants, sanitizers, and Valgrind;
• Cortex-M33 cross-compilation of both crypto engines;
• dependency integration;
• the core/port split guard and the docs guard (no internal-ledger or home-directory references in the published docs);
• fuzz targets;
• selected and nightly M33MU scenarios; and
• the MIMXRT700 chain under M33MU's RT700 model (the RT700 scenario matrix).

Trigger routing

The host, compiler, sanitizer, Valgrind, cross-compilation, integration, and core/port split checks run on every pull request, including drafts. The fuzz target also runs on pull requests as a 60-second libFuzzer smoke pass; the nightly schedule and manual dispatch run the 600-second soak instead.

The M33MU workflow (wolfBoot plus both guest lifecycles, then a select job that picks each port's tier and one matrix job that runs exactly those scenario groups) is tiered. Every pull request runs each port's smoke tier on both crypto engines (STM32H563: positive, gtzcneg, crossdomain, bothpsa, confboot, devcrypto; MIMXRT700: positive, ahbscneg, crossdomain, bothpsa, confboot, devcrypto). The full matrix runs on a push to main, on the nightly schedule, on manual dispatch (with a port input), and on a pull request that carries the ci:h5, ci:rt700, or ci:all label. The scenario groups per port and tier are in tests/target/lib/scenario_matrix.py; make test-target TARGET=<port> runs the same smoke tier locally and WT_TIER=full every scenario.

Running M33MU off a pull request

To run a port's full matrix against a branch without a PR, dispatch the workflow:

gh workflow run m33mu.yml --ref <branch> -f port=mimxrt700

To run a single scenario locally, use tests/target/run_m33mu_scenario.sh <key> (for example positive or crossdomain).

The GitHub-hosted workflows do not establish a physical-board result. Hardware output must come from the STM32H563 runner attached to a board.

Interpreting failures

• A host pass plus target failure usually points to architecture glue, image assembly, linker placement, or hardware policy rather than a neutral state machine.
• A missing target prints SKIP. It is not a successful run.
• Preserve the first failing marker and the generated log before rebuilding; target runners place detailed output under logs/ or the configured scenario log.
• Confirm that Secure, guest, manifest, flash, and emulator addresses were built from the same configuration.