README.md

reComputer-RK-LLM

Docker-based deployment for pre-converted large language models (LLMs) and vision-language models (VLMs) on Seeed Studio reComputer boards with Rockchip RK3576 and RK3588-family processors. The bundled servers expose OpenAI-compatible APIs, with an Ollama-compatible chat endpoint for LLMs.

Hardware

This project targets reComputer RK3576 and reComputer RK3588 boards with a 64-bit Linux image, Docker, and Docker Buildx installed. The repository includes the ARM64 runtime libraries and Python wheel required by the image. The containers need access to the board's NPU devices.

Deployment guides

• LLM deployment
• VLM deployment

Architecture

The repository builds a reusable ARM64 environment image and separate model images:

runtime/ + app/ + scripts/  ->  environment image  ->  model image

The environment image contains the pinned RKLLM/RKNN runtimes and API servers. Model images add only the selected .rkllm and optional .rknn artifacts. Custom models can use the environment image directly by mounting their files.

Quick start

Run a published model image. This example starts the Qwen2.5 1.5B Instruct model for RK3576 with the w8a8 quantization:

sudo docker run --rm -it \
  --name recomputer-rk-llm \
  --privileged \
  -p 8001:8001 \
  -v /dev:/dev \
  -e INTERACTIVE_CHAT=true \
  -e LOG_LEVEL=warning \
  ghcr.io/seeed-projects/recomputer-rk-llm/llm/qwen2.5-1.5b-instruct:rk3576-w8a8

The command stays attached to the terminal and enables interactive chat for testing. If you want to run the same service in the background, use this command instead:

sudo docker run --rm -d \
  --name recomputer-rk-llm \
  --privileged \
  -p 8001:8001 \
  -v /dev:/dev \
  ghcr.io/seeed-projects/recomputer-rk-llm/llm/qwen2.5-1.5b-instruct:rk3576-w8a8

Choose another published image from Available model definitions. The image already contains the converted model and its matching runtime; use the LLM guide or VLM guide for custom model files and request examples.

Check readiness and call the OpenAI-compatible API:

curl http://localhost:8001/health
curl http://localhost:8001/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"rkllm-model","messages":[{"role":"user","content":"Hello"}],"stream":false}'

For a custom VLM, set MODEL_KIND=vlm and provide both MODEL_FILE and VISION_MODEL_FILE; see the VLM guide for the complete request format.

Build locally

To build the shared environment image on the target board or with an ARM64 Buildx builder:

docker buildx build --platform linux/arm64 \
  -f docker/Dockerfile \
  -t recomputer-rk-llm:env --load .

Available model definitions

The definitions under models/ are used by the Build model images GitHub Actions workflow. Each row below is a separate model image; the values show the available quantization tags for each board:

Type	Model	RK3576	RK3588
LLM	Qwen2.5 1.5B Instruct	w4a16, w8a8	w8a8
LLM	Qwen2.5 3B Instruct	w4a16, w8a8	w8a8
LLM	Qwen3 1.7B	w4a16, w8a8	w8a8
LLM	Qwen3 4B	w4a16, w8a8	w8a8
LLM	Gemma 3 4B IT	w4a16, w8a8	w8a8
VLM	Qwen3.5 2B	w4a16-g128, w8a8	w8a8
VLM	Qwen3.5 4B	w4a16-g128, w8a8	w8a8

Published image names follow this pattern:

ghcr.io/seeed-projects/recomputer-rk-llm/<type>/<model-id>:<platform>-<quantization>

To build a published model image, run Build RKLLM model images and select the desired scope, model, platform, and quantization.

Runtime

The bundled RKLLM runtime is v1.3.0. The ctypes definitions in app/fastapi_server_llm.py and app/fastapi_server_vlm.py match that ABI. Runtime artifacts are kept under runtime/.

Configuration

Variable	Default	Purpose
MODEL_PATH	/app/models/model.rkllm	Full model path
MODEL_FILE	empty	Model filename under /app/models
MODEL_KIND	llm	llm or vlm
VISION_MODEL_FILE	empty	VLM .rknn filename
TARGET_PLATFORM	auto	rk3576, rk3588, or rk3588s
RUN_FREQ_FIX	true	Apply board frequency setup
PORT	8001	HTTP port
API_MODEL_NAME	rkllm-model	Public API model name
INTERACTIVE_CHAT	false	Enable terminal chat for LLM

Repository layout

app/                         API servers
runtime/lib/                 ARM64 RKLLM/RKNN shared libraries
runtime/wheels/              ARM64 Python runtime wheels
models/<kind>/<id>/<board>/  Model metadata and download URLs
docker/Dockerfile            Reusable environment image
docker/Dockerfile.model      Thin model image
docker/entrypoint.sh         Runtime/model selection and validation
scripts/                     Board frequency setup
.github/workflows/           Environment and model image builds
docs/LLM.md                  LLM deployment guide
docs/VLM.md                  VLM deployment guide
tools/                       API performance test clients

Speed test

The performance clients measure time to first token (TTFT) and time per output token (TPOT):

python -m venv .env && source .env/bin/activate
pip install -r requirements.txt
python tools/llm_speed_test.py --help
python tools/vlm_speed_test.py --help

Safety

The examples use --privileged and -v /dev:/dev because Rockchip NPU access varies by board image. Keep the service on a trusted network; authentication and TLS are not included.

References

• airockchip/rknn-llm
• Seeed Studio reComputer

Community