FastFlow SDK

FastFlow SDK is a new software stack that brings together emerging capabilities, tools, and runtimes for developing custom GenAI applications for AMD XDNA Neural Processing Engines (NPUs).

Important

This is an early-access release that enables developers to experiment with the iron-tools compiler stack for custom model development and run out-of-the-box FastFlowLM models on AMD Ryzen AI XDNA2 NPUs using the new HRX runtime. Interfaces, packages, models, and supported hardware or driver combinations can change between snapshots. Use a GitHub Release for installable artifacts and for the driver versions validated with that build. These packages are not published to PyPI.

Tip

For a higher-level application-serving experience, use Lemonade Server, which provides a validated FastFlowLM integration.

It consists of:

• FastFlowLM (Fast Flow Language Models): NPU-optimized language-model inference
• HRX (HIP Runtime Extended): a runtime that executes NPU applications
• iron-tools: a programming framework and compiler for the AI Engine (AIE) array, formerly MLIR-AIE
• iron-ops: example machine-learning operators built with iron-tools

The native fastflow CLI ships with the runtime and reports device, driver, and runtime health. Building from this repository also pulls additional subproject requirements; each subproject has more details. See CONTRIBUTING.md if you want to change the stack.

Key terms

• XDNA2 / NPU2: the AMD Ryzen™ AI NPU generation targeted by this release.
• HRX: the runtime layer that loads and dispatches NPU applications.
• AIE: AMD's AI Engine array, the programmable compute fabric targeted by iron-tools.
• PDI: the programmable device image extracted from a compiled design and loaded by the NPU runtime.
• Peano: the LLVM-based AIE compiler toolchain used to generate device code.

Intended audience

This early-access release enables developers to use hands-on utilities for custom application development with the iron-tools compiler stack, run FastFlowLM models on AMD Ryzen AI XDNA2 NPUs using the HRX runtime, and validate the NPU runtime with fastflow.

Supported models

Each release includes a registry of FastFlowLM models validated for that build. Run flm list after installation to see the models available in your release.

The examples use qwen3:0.6b because it is the smallest recommended model for verifying NPU inference.

For the complete model catalog, see the FastFlowLM model documentation.

Supported platforms

This early-access release targets Linux and Windows on x86_64, with AMD Ryzen AI NPUs that implement XDNA2 (NPU2). Windows instructions are native PowerShell; WSL is not the supported Windows path.

Validated NPU driver versions are listed in each GitHub release. After installing or changing a driver, run fastflow self-test. If that fails, reboot and retry; a hot package switch can occasionally leave the device unhealthy even when the OS reports the new driver as installed. fastflow probe and fastflow doctor report the active driver, firmware, NPU power mode where supported, and runtime health. fastflow powermode reports the current mode. Use fastflow powermode <mode> to change it. Power-mode changes may require elevated privileges depending on the installed driver.

Runtime and FastFlowLM Python wheels require Python 3.11 or newer. Compiler wheels require Python 3.12, 3.13, or 3.14. The standalone FastFlowLM/HRX archives do not require Python.

Getting started

• I want to install a release and run models.
• I want to develop my own models and applications.
• I want to build FastFlow SDK from source.

Install a release and run models

You can either download and unpack binaries from a GitHub Release directly, or install everything into a Python virtual environment. The latter path is required if you plan to develop your own applications.

Option A: Standalone archive

Each release includes a relocatable FastFlowLM + HRX archive:

• Windows: fastflow-sdk-windows-x86_64.zip
• Linux: fastflow-sdk-linux-x86_64.tar.zst

Extract it and run the binaries from the archive root (bin/flm and bin/fastflow):

# Windows
Expand-Archive .\fastflow-sdk-windows-x86_64.zip -DestinationPath .
cd .\fastflow-sdk-windows-x86_64
.\bin\fastflow.exe version
.\bin\fastflow.exe probe --json
.\bin\fastflow.exe doctor
.\bin\fastflow.exe self-test
.\bin\flm.exe list
.\bin\flm.exe run qwen3:0.6b

# Linux
tar --zstd -xf fastflow-sdk-linux-x86_64.tar.zst
cd fastflow-sdk-linux-x86_64
./bin/fastflow version
./bin/fastflow probe --json
./bin/fastflow doctor
./bin/fastflow self-test
./bin/flm list
./bin/flm run qwen3:0.6b

fastflow self-test also dispatches a packaged NPU2 fixture. The first flm run downloads model assets and needs Hugging Face access. Inside the interactive flm run prompt, use /bye to exit. flm list shows the models included in that build.

The archive cannot compile NPU programs; use the wheels in Option B for that.

Option B: Python wheels

Wheels ship on the same GitHub release, not on PyPI. Installing fastflow-sdk-fastflowlm pulls the matching fastflow-sdk-runtime wheel.

Install Python and create a virtual environment. Activate the environment before every command below:

# Windows
winget install --exact --id Python.Python.3.12
py -3.12 -m venv $HOME\fastflow
& $HOME\fastflow\Scripts\Activate.ps1

# Linux
sudo apt-get install -y python3.12 python3.12-venv
python3.12 -m venv ~/fastflow
source ~/fastflow/bin/activate

Replace <tag> and <version> with the values from the release. For example, if the release tag is fastflow-sdk-v0.1.0.dev202610050001, then use <tag> = fastflow-sdk-v0.1.0.dev202610050001 and <version> = 0.1.0.dev202610050001.

python -m pip install --only-binary=:all: --find-links "https://github.com/ROCm/FastFlow-SDK/releases/expanded_assets/<tag>" "fastflow-sdk-fastflowlm==<version>"

To validate the runtime, use the following commands:

fastflow version
fastflow probe --json
fastflow doctor
fastflow self-test
flm list
flm run qwen3:0.6b

To also compile your own NPU programs, see Develop your own models and applications.

See native runtime CLI for fastflow commands.

Develop your own models and applications using iron-tools

iron-tools provides programming abstractions and a compiler toolchain that targets the AI Engine (AIE) array in the AMD NPU. FastFlowLM was built using iron-tools.

Note

iron-tools is currently undergoing renaming from its old name, MLIR-AIE, so you might see references to both of these terms. They mean the same thing.

iron-tools ships as a vendored part of FastFlow SDK in the fastflow-sdk-compiler package. It needs a Python 3.12, 3.13, or 3.14 virtual environment. Set up and activate the environment before running the commands below; instructions are the same as in the previous section.

Install fastflow-sdk-compiler and fastflow-sdk-runtime from a GitHub Release: the compiler builds the design, and the runtime dispatches it on the NPU. Replace <tag> and <version> with values from that release (tags look like fastflow-sdk-v<version>):

python -m pip install --only-binary=:all: --find-links "https://github.com/ROCm/FastFlow-SDK/releases/expanded_assets/<tag>" "fastflow-sdk-compiler==<version>" "fastflow-sdk-runtime==<version>"

For example:

python -m pip install --only-binary=:all: --find-links "https://github.com/ROCm/FastFlow-SDK/releases/expanded_assets/fastflow-sdk-v0.1.0.dev202610050001" "fastflow-sdk-compiler==0.1.0.dev202610050001" "fastflow-sdk-runtime==0.1.0.dev202610050001"

Install the IRON operator library

fastflow-sdk-ops is the IRON operator library (from iron.operators import ElementwiseAdd). It depends on fastflow-sdk-compiler at the same version, so a compiler install above is enough, or this command will pull the compiler if it is missing:

python -m pip install --only-binary=:all: --find-links "https://github.com/ROCm/FastFlow-SDK/releases/expanded_assets/<tag>" "fastflow-sdk-ops==<version>"

If you need bleeding-edge iron-tools features, you may also choose to install directly from the upstream repository instead of the vendored fastflow-sdk-compiler package; see instructions there.

Hello world example

tests/npu/jit_smoke.py contains a small IRON JIT example program that we can compile for the NPU using iron-tools. It copies a buffer DDR -> MemTile -> DDR: pure DMA, no compute kernel.

After installing fastflow-sdk-compiler and fastflow-sdk-runtime, clone or download this repository and run:

python tests/npu/jit_smoke.py
# PASS: 4096 int32 elements copied through the NPU

Learn the programming model

• MLIR-AIE programming guide -- the iron-tools tutorial series.
• MLIR-AIE programming examples -- designs from passthrough_kernel up to matrix multiplication.

Both assume an MLIR-AIE source checkout with Makefiles and XRT. The fastflow-sdk-compiler wheel already gives you the toolchain, and the runtime here is HRX, so read the IRON design code and skip their setup steps.

To build FastFlow SDK from this checkout instead of installing a release, follow the build from source guide. It covers the runtime, FastFlowLM, compiler stack, platform prerequisites, packaging, and tests.

Known limitations

• Preemption: FastFlowLM contains opt-in support for inserting NPU preemption points, but it is disabled by default. The native HRX/XADX flow does not yet support or validate the required preemption save/restore payloads end-to-end, so FastFlowLM preemption must remain disabled when using this flow.
• load_pdi and control scratchpad: FastFlow SDK packaging does not yet carry standalone PDIs, load_pdi relocations, or scratchpad parameter metadata. IRON sequences using load_pdi or ScratchpadParameter are therefore not currently supported.

Additional documentation

• Native runtime CLI: fastflow version, probe, doctor, self-test, and powermode
• tests/npu/jit_smoke.py: compiler/JIT smoke test and the smallest complete iron-tools design
• iron-tools programming: guide and examples
• Model qualification: optional fastflow qualify profiles and evidence bundles for validating releases
• Third-party notices
• Contributing

License

This repository is licensed under the MIT License.

Please file issues in this repository. See SECURITY.md to report a vulnerability. This is an early snapshot, so expect breaking changes between releases.