diff --git a/.gitignore b/.gitignore index 7c3b9b1cc3..8244841d2d 100644 --- a/.gitignore +++ b/.gitignore @@ -1,7 +1,18 @@ +# OS files +.DS_Store + +# Dependencies +/node_modules/ + +# IDE / editor local state +/.vs/ + +# Local agent configuration +/.claude/ +/.opencode/ +/AGENTS.md +/opencode.json + # Auto-generated JSON files /static/json/baidu_tongji_data.json /static/json/commit_update_data.json -.DS_Store -/.vs -node_modules -.claude diff --git a/docs/aicore/dx-m1/windows/README.md b/docs/aicore/dx-m1/windows/README.md new file mode 100644 index 0000000000..45abab9693 --- /dev/null +++ b/docs/aicore/dx-m1/windows/README.md @@ -0,0 +1,126 @@ +--- +sidebar_position: 4 +doc_kind: page +locale: zh +board: dx-m1 +task_type: getting-started +last_verified: 2026-06-30 +--- + +# Windows 平台支持 + +智核 DX-M1 / DX-M1M 支持 Windows 10 / 11 x64 平台。本页介绍驱动安装、运行时配置及应用示例，帮助你快速在 Windows 上使用 NPU 推理。 + +## 使用前提 + +- **硬件**：智核 DX-M1 或 DX-M1M 模组，已通过 PCIe / M.2 接口接入主机 +- **操作系统**：Windows 10 或 Windows 11（x64） +- **运行库**：[Microsoft Visual C++ 2015-2022 Redistributable (x64)](https://aka.ms/vs/17/release/vc_redist.x64.exe) +- **权限**：管理员权限（驱动安装和服务注册需要） + +## 驱动安装 + +智核 DX-M1 / DX-M1M 在 Windows 上通过 PCIe 接口通信，需安装配套驱动。 + +1. 解压 `dx_rt_windows` 软件包，进入 `m1/v3.2.0/dxm1drv/` 目录 +2. 解压 `dxm1drv.zip`，得到 `dxm1drv.inf`、`dxm1drv.sys`、`dxm1drv.cat` +3. 右键点击 `dxm1drv.inf`，选择 **安装** +4. 安装完成后，打开设备管理器，在 **处理加速器 (Processing accelerators)** 下应能看到 **DEEPX DEVICE - M1 PCI CONTROLLER** + +## 运行时安装 + +驱动安装完成后，安装运行时守护进程。 + +1. 以管理员身份打开 **命令提示符 (cmd)** 或 **PowerShell** +2. 进入运行时目录（以 v3.2.0 为例）： + +```bash +cd m1\v3.2.0\dx_rt\bin +``` + +3. 注册并启动 DXRT 守护进程： + +```bash +dxrtd.exe --install +dxrtd.exe --start +``` + +`dxrtd.exe` 注册为 Windows 服务后，系统启动时会自动运行。你也可以使用 `services.msc` 管理该服务。 + +:::tip 版本选择 +`v3.2.0` 和 `v3.3.1` 使用相同的驱动，可按需选用。`v3.2.0` 提供完整的驱动 + 运行时 + 应用示例；`v3.3.1` 仅包含运行时。 +::: + +## 设备验证 + +使用 `dxrt-cli.exe` 验证 NPU 设备是否正常识别： + +```bash +dxrt-cli.exe -s +``` + +正常输出应显示设备信息，类似： + +```text +Device Information: + Device ID: 0 + Device Name: DX-M1 + Firmware Version: ... + Status: Ready +``` + +你也可以使用 `dxtop.exe` 实时监控 NPU 状态： + +```bash +dxtop.exe +``` + +## 应用示例 + +### 演示应用（v3.0.2，推荐） + +`m1/v3.2.0/dx_app/v3.0.2/` 提供预编译的演示程序，涵盖目标检测、人脸识别、姿态估计、语义分割等任务。 + +1. 首次使用需运行 `m1/v3.2.0/dx_app/setup.bat` 下载模型和示例视频 +2. 运行交互式菜单： + +```bash +cd m1\v3.2.0\dx_app\v3.0.2 +run_demo.bat +``` + +3. 按菜单提示选择对应模型即可运行推理演示 + +常用模型包括 `yolov5_sync`、`yolov8_sync`、`scrfd_sync`、`efficientnet_sync` 等，均提供同步 (`_sync`) 和异步 (`_async`) 两种模式。 + +### Python 接口 + +安装 Python wheel 包后可直接在 Python 中调用 NPU： + +```bash +# 根据 Python 版本选择对应的 wheel（支持 Python 3.10 ~ 3.14） +pip install m1\v3.2.0\dx_rt\python\dx_engine-1.1.4-cp312-cp312-win_amd64.whl +``` + +Python CLI 推理示例： + +```bash +python m1\v3.2.0\dx_rt\python\cli\run_model.py -m -l 100 +``` + +Python API 调用示例： + +```python +from dx_engine import InferenceEngine + +engine = InferenceEngine() +engine.load_model("model.dxnn") +engine.run() +``` + +## 故障排除 + +- **设备管理器中看不到 DEEPX 设备**：检查 PCIe / M.2 物理连接，确认模组已正确插入。尝试重新扫描硬件（`devmgmt.msc` → 操作 → 扫描检测硬件改动）。 +- **dxrt-cli.exe -s 报错**：确认 `dxrtd.exe` 已启动（`services.msc` 中查看 `DXRT Daemon` 服务状态）。 +- **运行 demo 无输出**：确认已执行 `setup.bat` 下载模型和视频文件，网络需能访问 `sdk.deepx.ai`。 +- **VC++ 报错**：确认已安装 Microsoft Visual C++ 2015-2022 Redistributable (x64)。 diff --git a/docs/fogwise/airbox-q900/ai-dev/qim-sdk/README.md b/docs/fogwise/airbox-q900/ai-dev/qim-sdk/README.md index 17c9c72d7e..25d630bcfd 100644 --- a/docs/fogwise/airbox-q900/ai-dev/qim-sdk/README.md +++ b/docs/fogwise/airbox-q900/ai-dev/qim-sdk/README.md @@ -126,14 +126,17 @@ gst-ai-classification --config-file=/etc/configs/config_classification.json | [事件编码器](./event-encoder.md) | 检测事件编码输出 | YOLOX | DSP | | [元数据解析](./metadata-parser.md) | 检测结果元数据导出 | YOLOX | DSP | | [多流推理](./multistream-inference.md) | 多路同步检测 | YOLOX | DSP | +| [多流批量推理](./multistream-batch-inference.md) | 批量多路检测 | YOLOv8 (batch=4) | DSP | +| [多输入多输出](./multi-input-output.md) | 多路输入输出检测 | YOLOv5 | DSP | | [人脸检测](./face-detection.md) | 人脸关键点检测 | Lightweight Face Detection | DSP | > 完整列表参见 [IM SDK 参考手册](https://docs.qualcomm.com/doc/80-70020-50SC/topic/download-model-and-label-files.html) 第 3 章。 ## 参考 -- [Python GStreamer 应用](./python-apps.md) — Python 绑定示例 -- [源码编译](./build-from-source.md) — 在设备上编译自定义 GStreamer 应用 +- [Python GStreamer 应用](./python-apps.md) — Python 方式，适合自定义前后处理 +- [源码编译](./build-from-source.md) — C/C++ 方式，适合深度定制插件 +- [模型导出](./model-export.md) — 自行导出 YOLOv5/YOLOv8 模型 ## 排障 @@ -144,7 +147,7 @@ gst-ai-classification --config-file=/etc/configs/config_classification.json ```bash -ls $XDG_RUNTIME_DIR/$WAYLAND_DISPLAY +ls $XDG_RUNTIME_DIR/wayland-0 ``` diff --git a/docs/fogwise/airbox-q900/ai-dev/qim-sdk/build-from-source.md b/docs/fogwise/airbox-q900/ai-dev/qim-sdk/build-from-source.md index ca68b0989b..6f31285cf3 100644 --- a/docs/fogwise/airbox-q900/ai-dev/qim-sdk/build-from-source.md +++ b/docs/fogwise/airbox-q900/ai-dev/qim-sdk/build-from-source.md @@ -1,61 +1,103 @@ --- -sidebar_position: 16 +sidebar_position: 18 doc_kind: page locale: zh board: airbox-q900 task_type: ai-dev -last_verified: 2026-06-24 +last_verified: 2026-06-30 --- # 源码编译 -QIM SDK 的所有 GStreamer 插件和示例应用已通过 [apt 安装](./README.md#安装) 提供预编译包，一般情况下无需自行编译。 +前面的 [Demo](./README.md#可用示例应用)（`gst-ai-*`）和 [Python 应用](./python-apps.md) 使用的是 apt 预编译的 GStreamer 插件。如果你需要**修改插件源码**（如定制后处理逻辑、添加自定义模型支持）或**写自己的 C/C++ 推理程序**，可以在设备上直接编译。 -你也可以在设备上直接编译源码，用于以下场景： +## Hello-QIM：最小 C/C++ 示例 -- 修改插件源码，定制后处理逻辑或添加自定义模型支持 -- 调试插件内部行为 -- 需要与预编译包不同的编译选项 +Hello-QIM 是一个最小可运行的 GStreamer 应用，展示如何用 C++ 创建 pipeline 并获取摄像头帧。适合作为自定义应用的起点。 -编译产物包括 34 个 GStreamer 插件动态库（`.so`），覆盖 AI 推理引擎（`mltflite`、`mlqnn`、`mlsnpe`）、预处理（`mlvconverter`）、后处理（`mlpostprocess` 等）、视频处理（`vcomposer` 等）。详见下方步骤。 +### 下载源码 -## 前提条件 + -- 已完成 [QIM SDK 安装](./README.md#安装) +```bash +git clone https://github.com/quic/sample-apps-for-qualcomm-linux +cd sample-apps-for-qualcomm-linux/Hello-QIM +``` -## 安装编译依赖 + + +### 准备源文件 ```bash -sudo apt-add-repository -s ppa:ubuntu-qcom-iot/qcom-ppa -sudo apt update -sudo apt install -y \ - qcom-adreno-dev \ - libgstreamer1.0-qcom-sample-apps-utils-dev +cp main.cc gst-appsink.cc ``` -## 编译 QIM 插件（从源码） +### 创建 Makefile + +```makefile +CC = g++ +CFLAGS = -Wall -g $(shell pkg-config --cflags gstreamer-1.0) +LDFLAGS = $(shell pkg-config --libs gstreamer-1.0) + +all: gst-appsink + +gst-appsink: gst-appsink.o + $(CC) -o $@ $^ $(LDFLAGS) + +gst-appsink.o: gst-appsink.cc + $(CC) $(CFLAGS) -c $< + +clean: + rm -f gst-appsink gst-appsink.o +``` -### 步骤 1：下载 QIM 源码 +### 编译并运行 ```bash -cd ~ -apt source gst-plugins-qti-oss -cd gst-plugins-qti-oss-* +make +./gst-appsink -w 1280 -h 720 ``` -### 步骤 2：CMake 编译 +预期输出： + +```text +Hello-QIM: Success creating pipeline and received camera frame. +``` + +> 如果设备没有连接摄像头，pipeline 创建仍然会成功，只是收不到帧。 + +## 编译 QIM 插件（进阶） + +如果需要修改 GStreamer 插件本身（而不仅仅是调用插件），可以从源码编译全部插件。 + +### 安装编译依赖 ```bash +sudo apt install -y \ + qcom-adreno-dev \ + libgstreamer1.0-qcom-sample-apps-utils-dev +``` + + + +### 下载并编译 + + + +```bash +cd ~ +apt source gst-plugins-qti-oss +cd gst-plugins-qti-oss-* mkdir build && cd build cmake \ @@ -71,27 +113,12 @@ cmake \ .. make -j$(nproc) -``` - - - -### 步骤 3：安装 - - - -```bash sudo make install ``` -### 步骤 4：验证 - -```bash -ls /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstqti*.so | wc -l -``` - -编译安装约 34 个插件动态库，输出如 `libgstqtimltflite.so`、`libgstqtimlvclassification.so` 等。 +编译安装约 34 个插件动态库（`libgstqtimltflite.so`、`libgstqtimlvclassification.so` 等）。 ## 排障 @@ -107,7 +134,7 @@ pkg-config --modversion gstreamer-1.0 -如果提示找不到，安装： +如果提示找不到： @@ -116,7 +143,3 @@ sudo apt install libgstreamer1.0-dev ``` - -## 参考 - -- [Python GStreamer 应用](./python-apps.md) — Python 开发方式 diff --git a/docs/fogwise/airbox-q900/ai-dev/qim-sdk/event-encoder.md b/docs/fogwise/airbox-q900/ai-dev/qim-sdk/event-encoder.md index f14193dad8..7e9792b0bb 100644 --- a/docs/fogwise/airbox-q900/ai-dev/qim-sdk/event-encoder.md +++ b/docs/fogwise/airbox-q900/ai-dev/qim-sdk/event-encoder.md @@ -1,5 +1,5 @@ --- -sidebar_position: 13 +sidebar_position: 15 doc_kind: page locale: zh board: airbox-q900 diff --git a/docs/fogwise/airbox-q900/ai-dev/qim-sdk/metadata-parser.md b/docs/fogwise/airbox-q900/ai-dev/qim-sdk/metadata-parser.md index 7e9a0d5ea8..a8ee0116af 100644 --- a/docs/fogwise/airbox-q900/ai-dev/qim-sdk/metadata-parser.md +++ b/docs/fogwise/airbox-q900/ai-dev/qim-sdk/metadata-parser.md @@ -1,5 +1,5 @@ --- -sidebar_position: 14 +sidebar_position: 16 doc_kind: page locale: zh board: airbox-q900 diff --git a/docs/fogwise/airbox-q900/ai-dev/qim-sdk/model-export.md b/docs/fogwise/airbox-q900/ai-dev/qim-sdk/model-export.md new file mode 100644 index 0000000000..7582822898 --- /dev/null +++ b/docs/fogwise/airbox-q900/ai-dev/qim-sdk/model-export.md @@ -0,0 +1,109 @@ +--- +sidebar_position: 19 +doc_kind: page +locale: zh +board: airbox-q900 +task_type: ai-dev +last_verified: 2026-06-29 +--- + +# 模型导出 + +IM SDK 的下载脚本 (`download_artifacts.sh`) 仅提供 10 个基础 TFLite 模型。以下 Demo 需要自行导出额外模型： + +| Demo | 所需模型 | 导出方式 | +| ---------------------------------------------------- | ------------------------------------ | ------------------ | +| [多输入多输出](./multi-input-output.md) | YOLOv5 (`yolov5.tflite`) | 从 YOLOv5 源码导出 | +| [多流批量推理](./multistream-batch-inference.md) | YOLOv8 batch=4 (`yolov8_det.tflite`) | Qualcomm AI Hub | +| [目标检测](./object-detection.md)（YOLOv8/YOLO-NAS） | YOLOv8 / YOLO-NAS | Qualcomm AI Hub | + +## 方式一：从 YOLOv5 源码导出（多输入多输出 Demo） + + + +```bash +git clone https://github.com/ultralytics/yolov5.git +cd yolov5 +python -m pip install -r requirements.txt tensorflow-cpu +python export.py --weights yolov5m.pt --img 320 --include tflite --int8 --data data/coco128.yaml +``` + + + +将导出的模型推送到设备： + + + +```bash +scp yolov5m-int8.tflite radxa@:/etc/models/yolov5.tflite +``` + + + + + +```bash +# 创建 yolov5 标签文件（复制 yolonas 标签） +sudo cp /etc/labels/yolonas.labels /etc/labels/yolov5.labels +``` + + + +> 此导出方式使用 `--img 320` 以匹配 Demo 输入分辨率。**不要**使用默认 640，否则会因输入尺寸不匹配导致推理失败。 + +## 方式二：Qualcomm AI Hub 导出 YOLOv8（目标检测、多流批量推理） + +### 1. 注册并获取 API Token + +[Qualcomm AI Hub](https://aihub.qualcomm.com/) → Settings → API 令牌。 + +### 2. 安装 qai-hub-models + + + +```bash +python -m venv qaihm +source qaihm/bin/activate +pip install qai-hub-models +``` + + + +### 3. 导出标准模型（batch=1） + + + +```bash +python -m qai_hub_models.models.yolov8_det.export \ + --quantize w8a8 \ + --target-runtime=tflite \ + --device "Dragonwing IQ-9075 EVK" +``` + + + +### 4. 导出批量模型（batch=4，用于多流批量推理） + + + +```bash +python -m qai_hub_models.models.yolov8_det.export \ + --quantize w8a8 \ + --target-runtime=tflite \ + --device "Dragonwing IQ-9075 EVK" \ + --batch-size 4 +``` + + + +> `--batch-size 4` 必须匹配 Demo 的流数量。Q900 (QCS9075) 最多支持 4 路。 + +### 5. 推送到设备 + + + +```bash +scp yolov8_det.tflite radxa@:/etc/models/ +``` + + diff --git a/docs/fogwise/airbox-q900/ai-dev/qim-sdk/multi-input-output.md b/docs/fogwise/airbox-q900/ai-dev/qim-sdk/multi-input-output.md new file mode 100644 index 0000000000..8efd1d8bb8 --- /dev/null +++ b/docs/fogwise/airbox-q900/ai-dev/qim-sdk/multi-input-output.md @@ -0,0 +1,157 @@ +--- +sidebar_position: 14 +doc_kind: page +locale: zh +board: airbox-q900 +task_type: ai-dev +last_verified: 2026-06-29 +--- + +# 多输入多输出推理 + +`gst-ai-multi-input-output-object-detection` 同时对多路输入源执行目标检测，每路独立推理，支持文件、RTSP 和摄像头输入组合，输出可到显示器、文件或 RTSP 流。 + +最大支持 6 路输入同时检测。 + +## 前提条件 + +- 已完成 [QIM SDK 安装](./README.md#安装) 和 [模型下载](./README.md#下载模型和测试资源) +- YOLOv5 TFLite 模型（默认下载不包含此模型，需通过 Qualcomm AI Hub 自行转换） + +## 步骤 + +### 1. 准备输入视频 + +配置默认使用 6 路文件输入。将测试视频复制为多份： + + + +```bash +for i in $(seq 1 6); do + sudo cp /etc/media/video.mp4 /etc/media/video${i}.mp4 +done +``` + + + +### 2. 准备 YOLOv5 模型 + +默认配置文件引用 `/etc/models/yolov5.tflite`，该模型未包含在下载脚本中，需从 YOLOv5 源码导出。 + +在主机上执行： + + + +```bash +git clone https://github.com/ultralytics/yolov5.git +cd yolov5 +python -m pip install -r requirements.txt tensorflow-cpu +python export.py --weights yolov5m.pt --img 320 --include tflite --int8 --data data/coco128.yaml +``` + + + +> **`--img 320` 是必须的**：此 Demo 的输入分辨率要求为 320×320。使用默认 640 会因尺寸不匹配导致推理失败。 + +将模型推送到设备并创建标签文件： + + + +```bash +scp yolov5m-int8.tflite radxa@:/etc/models/yolov5.tflite +``` + + + + + +```bash +sudo cp /etc/labels/yolonas.labels /etc/labels/yolov5.labels +``` + + + +> **注意**：此 Demo **不支持** YOLOX 模型（`yolox_quantized.tflite`）。下载脚本提供的 YOLOX 模型在其他 Demo（如 `gst-ai-object-detection`）中可正常使用，但本 Demo 必须使用 YOLOv5。 + +### 3. 修改配置文件 + + + +```bash +python3 -c " +import json +with open('/etc/configs/config-multi-input-output-object-detection.json') as f: + c = json.load(f) +c['model'] = '/etc/models/yolov5.tflite' +c['labels'] = '/etc/labels/yolov5.labels' +json.dump(c, open('/tmp/cfg_multi_in_out.json', 'w'), indent=2) +print('config written') +" +``` + + + +> 如果模型输出格式与 YOLOX 兼容，`labels` 可使用 `/etc/labels/yolox.json`。 + +### 4. 运行 + + + +```bash +gst-ai-multi-input-output-object-detection --config-file=/tmp/cfg_multi_in_out.json +``` + + + +按 `Ctrl + C` 停止。 + +## 预期输出 + +终端输出： + +```text +TARGET Can support file source, RTSP source and camera source +Run app with model: ... and labels: ... +IN Options: camera: 0 (id: 0), file: 6, rtsp: 0 +OUT Options: display: 1, file: (null), rtsp: 0 +VERBOSE: Replacing 334 out of 334 node(s) with delegate (TfLiteQnnDelegate) node +Pipeline state changed from NULL to READY: +Pipeline state changed from READY to PAUSED: +Pipeline state changed from PAUSED to PLAYING: +``` + +显示器上同时显示 6 路检测画面，每路独立绘制边界框。 + +## 验证 + +- 334 个算子全部委派到 DSP +- Pipeline 进入 `PLAYING` 状态 +- 显示器显示 6 路检测画面，每路独立绘制边界框 + +## 配置说明 + +| 字段 | 说明 | 默认值 | +| ------------------ | ------------------------------------- | -------------------------------- | +| `input-file-path` | 文件输入路径（JSON 数组，最多 6 路） | `["/etc/media/video1.mp4", ...]` | +| `input-rtsp-path` | RTSP 输入地址（JSON 数组，最多 6 路） | 空 | +| `num-camera` | 摄像头数量（0-2） | `0` | +| `camera-id` | 摄像头设备 ID | `0` | +| `model` | 目标检测模型路径 | `/etc/models/yolov5.tflite` | +| `labels` | 标签文件路径 | `/etc/labels/yolov5.json` | +| `output-display` | 是否输出到显示器 | `true` | +| `output-file-path` | 输出文件路径（可选） | 空 | +| `output-rtsp-path` | RTSP 输出地址（可选） | 空 | + +## Pipeline 流程 + +```text +filesrc × 6 → qtdemux → h264parse → v4l2h264dec → qtimlvconverter + ↓ + qtimltflite (DSP 推理) + ↓ + qtimlvdetection (YOLO 后处理) + ↓ + qtivcomposer + ↓ + waylandsink +``` diff --git a/docs/fogwise/airbox-q900/ai-dev/qim-sdk/multistream-batch-inference.md b/docs/fogwise/airbox-q900/ai-dev/qim-sdk/multistream-batch-inference.md new file mode 100644 index 0000000000..6ed7b0104d --- /dev/null +++ b/docs/fogwise/airbox-q900/ai-dev/qim-sdk/multistream-batch-inference.md @@ -0,0 +1,124 @@ +--- +sidebar_position: 13 +doc_kind: page +locale: zh +board: airbox-q900 +task_type: ai-dev +last_verified: 2026-06-29 +--- + +# 多流批量推理 + +`gst-ai-multistream-batch-inference` 同时对多路视频流执行批量 AI 推理（目标检测或图像分割），将流批量化后进行一次推理，提高吞吐量。 + +Q900 (QCS9075) 最多支持 24 路并发输入（6 批次 × batch-size 4）。 + +## 前提条件 + +- 已完成 [QIM SDK 安装](./README.md#安装) +- YOLOv8 TFLite 模型，batch-size=4（需通过 Qualcomm AI Hub 导出，详见 [模型导出](./model-export.md#方式二qualcomm-ai-hub-导出-yolov8目标检测多流批量推理)） + +## 步骤 + +### 1. 导出批量模型 + +使用 `--batch-size 4` 导出 YOLOv8 模型，命令详见 [模型导出](./model-export.md#4-导出批量模型batch4用于多流批量推理)。 + +将模型和标签推送到设备： + + + +```bash +scp yolov8_det.tflite radxa@:/etc/models/ +``` + + + +### 2. 创建配置文件 + + + +```bash +python3 -c " +import json +with open('/etc/configs/config-multistream-batch-inference.json') as f: + base = json.load(f) +entry = base['pipeline-info'][0] +entry['model-path'] = '/etc/models/yolov8_det.tflite' +entry['labels-path'] = '/etc/labels/yolov8.json' +# 生成 6 批次 × 4 流 = 24 路（id 最大 5） +base['pipeline-info'] = [] +for i in range(6): + e = json.loads(json.dumps(entry)) + e['id'] = i + base['pipeline-info'].append(e) +with open('/tmp/cfg_batch.json', 'w') as f: + json.dump(base, f, indent=2) +print('24-stream config written') +" +``` + + + +### 3. 运行 + + + +```bash +gst-ai-multistream-batch-inference --config-file=/tmp/cfg_batch.json +``` + + + +按 `Ctrl + C` 停止。 + +## 预期输出 + +终端输出： + +```text +VERBOSE: Replacing 282 out of 282 node(s) with delegate (TfLiteQnnDelegate) node +Pipeline state changed from NULL to READY: +Pipeline state changed from READY to PAUSED: +Pipeline state changed from PAUSED to PLAYING: +``` + +显示器上同时显示 24 路检测画面。 + +## 验证 + +- 全部算子委派到 DSP +- Pipeline 进入 `PLAYING` 状态 +- 24 路画面同时正确显示边界框 + +## 配置说明 + +| 字段 | 说明 | 默认值 | +| ------------------------------------- | ---------------------------------------------------- | ----------------- | +| `output-type` | 输出类型：`wayland` / `filesink` | `wayland` | +| `pipeline-info[].id` | 批次 ID（0-5） | `0` | +| `pipeline-info[].input-type` | 输入源类型 | `file` | +| `pipeline-info[].input-file-path` | 输入文件（每批次 4 路） | 见配置文件 | +| `pipeline-info[].mlframework` | 推理框架 | `tflite` | +| `pipeline-info[].model-path` | 模型路径 | 需替换为导出模型 | +| `pipeline-info[].labels-path` | 标签路径 | 需替换 | +| `pipeline-info[].constants` | LiteRT 量化常数 | 模型相关 | +| `pipeline-info[].post-process-plugin` | 后处理插件：`qtimlvdetection` / `qtimlvsegmentation` | `qtimlvdetection` | + +## Pipeline 流程 + +```text +filesrc × 4 → qtdemux → h264parse → v4l2h264dec → qtibatch (组批) + ↓ + qtimlvconverter + ↓ + qtimltflite (DSP 批量推理) + ↓ + qtimldemux (解复用) + ↓ + qtimlvdetection / qtimlvsegmentation + ↓ + qtivcomposer + ↓ + waylandsink +``` diff --git a/docs/fogwise/airbox-q900/ai-dev/qim-sdk/multistream-inference.md b/docs/fogwise/airbox-q900/ai-dev/qim-sdk/multistream-inference.md index d05729d470..a684247f74 100644 --- a/docs/fogwise/airbox-q900/ai-dev/qim-sdk/multistream-inference.md +++ b/docs/fogwise/airbox-q900/ai-dev/qim-sdk/multistream-inference.md @@ -63,7 +63,7 @@ python3 -c " import json with open('/etc/configs/config-multistream-inference.json') as f: c = json.load(f) -c['input-file-path'] = ['/etc/media/video_safe.mp4'] * 16 +c['input-file-path'] = ['/etc/media/video_safe.mp4'] * 32 json.dump(c, open('/tmp/cfg_multistream.json', 'w'), indent=2) print('config written') " diff --git a/docs/fogwise/airbox-q900/ai-dev/qim-sdk/python-apps.md b/docs/fogwise/airbox-q900/ai-dev/qim-sdk/python-apps.md index 78ba047eb8..c8b795fa85 100644 --- a/docs/fogwise/airbox-q900/ai-dev/qim-sdk/python-apps.md +++ b/docs/fogwise/airbox-q900/ai-dev/qim-sdk/python-apps.md @@ -1,44 +1,40 @@ --- -sidebar_position: 15 +sidebar_position: 17 doc_kind: page locale: zh board: airbox-q900 task_type: ai-dev -last_verified: 2026-06-24 +last_verified: 2026-06-30 --- # Python GStreamer 应用 -QIM SDK 提供 Python GStreamer 绑定示例，展示如何使用 Python 构建 AI/ML 和多媒体 pipeline。 +前面的 [Demo](./README.md#可用示例应用)（`gst-ai-*`）是预编译的命令行工具，通过 JSON 配置文件控制 pipeline。如果你需要在 pipeline 中插入自定义处理逻辑（如 OpenCV 预处理、自定义后处理），可以使用 Python GStreamer 绑定直接构建 pipeline。 + +QIM SDK 的 Python 示例通过 `gstreamer1.0-qcom-python-examples` 包安装，脚本位于 `/usr/bin/`，与 `gst-ai-*` Demo 使用**同一套底层 GStreamer 插件**（`qtimltflite`、`qtimlvdetection`、`qtivcomposer` 等），区别是用 argparse 命令行参数而非 config JSON。 ## 前提条件 - 已完成 [QIM SDK 安装](./README.md#安装) -Python 示例已通过 `gstreamer1.0-qcom-python-examples` 包安装。相关 Python 依赖（`python3-gst-1.0`、`python3-numpy`、`python3-opencv`）自动安装。 - -## 可用示例 - -| 应用 | 说明 | -| ------------------------ | ---------------------------- | -| 摄像头编码 | 从摄像头采集并编码为 H.264 | -| OpenCV 摄像头流 | 使用 OpenCV 读取摄像头流 | -| OpenCV 视频转换 | OpenCV 视频格式转换 | -| 并发视频播放（视频墙） | 多路视频同时播放 | -| 多摄像头流 | Python 控制多路摄像头 | -| 目标检测与显示 | 运行目标检测并在显示器上渲染 | -| RTSP 流解码和目标检测 | 从 RTSP 流解码并检测 | -| JPEG 图像解码 | 解码 JPEG 图像 | -| 目标检测和分类 | 级联检测+分类 | -| 转换和编码摄像头流 | 摄像头流转码 | -| 摄像头编码+目标检测+显示 | 端到端 pipeline | -| 目标检测+分类+分割 | 多任务 AI | -| 并行推理 | 多模型并行 | -| 菊花链检测和姿态检测 | 级联检测+姿态 | +Python 依赖（`python3-gst-1.0`、`python3-numpy`、`python3-opencv`）随包自动安装。 + +## 代表性示例 + +已安装 18 个可运行脚本，以下为几个代表性示例： + +| 脚本 | 说明 | 运行示例 | +| ----------------------------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `gst-ai-object-detection.py` | 目标检测 | `python3 /usr/bin/gst-ai-object-detection.py -s /etc/media/video.mp4 -f 2 -ml yolov5 -m /etc/models/yolov5m-int8.tflite -l /etc/labels/yolov5.json --use_dsp` | +| `gst-parallel-inference.py` | 多模型并行推理 | 同时运行检测+分类+分割 | +| `gst-daisychain-detection-pose.py` | 级联检测+姿态 | `--file /etc/media/video.mp4 --tflite-yolo-model ... --tflite-pose-model ...` | +| `gst-concurrent-videoplay-composition.py` | 视频墙 | `--infile /etc/media/video.mp4 -c 4`（4 路并发播放） | + +每个脚本使用 `--help` 查看完整参数。 ## 基本结构 -Python GStreamer 应用使用 PyGObject 绑定： +Python GStreamer 应用使用 PyGObject 绑定，最小示例： ```python import gi @@ -63,6 +59,8 @@ loop.run() ## 获取源代码 +所有 Python 示例的完整源码托管在 GitHub： + ```bash @@ -73,4 +71,5 @@ git clone https://github.com/quic/sample-apps-for-qualcomm-linux ## 参考 -- [源码编译](./build-from-source.md) — 在设备上编译自定义 C/C++ 应用 +- [Demo 应用](./README.md#可用示例应用) — 预编译 CLI 方式，通过 config JSON 控制 +- [源码编译](./build-from-source.md) — C/C++ 方式，深度定制插件 diff --git a/i18n/en/docusaurus-plugin-content-docs/current/aicore/dx-m1/windows/README.md b/i18n/en/docusaurus-plugin-content-docs/current/aicore/dx-m1/windows/README.md new file mode 100644 index 0000000000..efd6b943d0 --- /dev/null +++ b/i18n/en/docusaurus-plugin-content-docs/current/aicore/dx-m1/windows/README.md @@ -0,0 +1,126 @@ +--- +sidebar_position: 4 +doc_kind: page +locale: en +board: dx-m1 +task_type: getting-started +last_verified: 2026-06-30 +--- + +# Windows Platform Support + +The DX-M1 / DX-M1M AI accelerator supports Windows 10 / 11 x64. This guide covers driver installation, runtime setup, and application examples to help you get started with NPU inference on Windows. + +## Prerequisites + +- **Hardware**: DX-M1 or DX-M1M module installed via PCIe / M.2 interface +- **OS**: Windows 10 or Windows 11 (x64) +- **Runtime Library**: [Microsoft Visual C++ 2015-2022 Redistributable (x64)](https://aka.ms/vs/17/release/vc_redist.x64.exe) +- **Permissions**: Administrator privileges (required for driver installation and service registration) + +## Driver Installation + +The DX-M1 / DX-M1M communicates over PCIe on Windows and requires a companion driver. + +1. Extract the `dx_rt_windows` package and navigate to `m1/v3.2.0/dxm1drv/` +2. Extract `dxm1drv.zip` to obtain `dxm1drv.inf`, `dxm1drv.sys`, and `dxm1drv.cat` +3. Right-click `dxm1drv.inf` and select **Install** +4. After installation, open Device Manager. Under **Processing accelerators**, you should see **DEEPX DEVICE - M1 PCI CONTROLLER** + +## Runtime Installation + +Once the driver is installed, set up the runtime daemon. + +1. Open **Command Prompt (cmd)** or **PowerShell** as Administrator +2. Navigate to the runtime directory (using v3.2.0 as an example): + +```bash +cd m1\v3.2.0\dx_rt\bin +``` + +3. Register and start the DXRT daemon: + +```bash +dxrtd.exe --install +dxrtd.exe --start +``` + +`dxrtd.exe` is registered as a Windows service and starts automatically on system boot. You can also manage it via `services.msc`. + +:::tip Version Selection +Both `v3.2.0` and `v3.3.1` use the same driver. `v3.2.0` provides the full set: driver + runtime + application examples; `v3.3.1` includes only the runtime. +::: + +## Verification + +Use `dxrt-cli.exe` to verify that the NPU device is recognized: + +```bash +dxrt-cli.exe -s +``` + +Successful output should display device information similar to: + +```text +Device Information: + Device ID: 0 + Device Name: DX-M1 + Firmware Version: ... + Status: Ready +``` + +You can also use `dxtop.exe` for real-time NPU monitoring: + +```bash +dxtop.exe +``` + +## Application Examples + +### Demo Applications (v3.0.2, Recommended) + +`m1/v3.2.0/dx_app/v3.0.2/` provides prebuilt demo applications covering object detection, face recognition, pose estimation, semantic segmentation, and more. + +1. On first use, run `m1/v3.2.0/dx_app/setup.bat` to download models and sample videos +2. Launch the interactive menu: + +```bash +cd m1\v3.2.0\dx_app\v3.0.2 +run_demo.bat +``` + +3. Follow the menu prompts to select a model and run inference + +Common models include `yolov5_sync`, `yolov8_sync`, `scrfd_sync`, and `efficientnet_sync`, each available in synchronous (`_sync`) and asynchronous (`_async`) modes. + +### Python Interface + +Install the Python wheel to use the NPU directly from Python: + +```bash +# Choose the wheel matching your Python version (Python 3.10 ~ 3.14 supported) +pip install m1\v3.2.0\dx_rt\python\dx_engine-1.1.4-cp312-cp312-win_amd64.whl +``` + +Python CLI inference example: + +```bash +python m1\v3.2.0\dx_rt\python\cli\run_model.py -m -l 100 +``` + +Python API example: + +```python +from dx_engine import InferenceEngine + +engine = InferenceEngine() +engine.load_model("model.dxnn") +engine.run() +``` + +## Troubleshooting + +- **DEEPX device not visible in Device Manager**: Check the PCIe / M.2 physical connection and ensure the module is properly seated. Try rescanning for hardware changes (`devmgmt.msc` → Action → Scan for hardware changes). +- **`dxrt-cli.exe -s` fails**: Verify that `dxrtd.exe` is running (check `DXRT Daemon` service status in `services.msc`). +- **Demo produces no output**: Ensure `setup.bat` has been run to download models and video files. Network access to `sdk.deepx.ai` is required. +- **VC++ errors**: Confirm that Microsoft Visual C++ 2015-2022 Redistributable (x64) is installed. diff --git a/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/README.md b/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/README.md index 79cad9b8e1..157672a833 100644 --- a/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/README.md +++ b/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/README.md @@ -107,8 +107,6 @@ On success, the display shows the test video with classification labels overlaid Press `Ctrl + C` to stop. -> Normally you do **not** need to manually set `XDG_RUNTIME_DIR` or `WAYLAND_DISPLAY` — these are configured automatically on SSH login. - ## Available Sample Applications The following apps have been verified on Q900 (QCS9075): @@ -127,6 +125,8 @@ The following apps have been verified on Q900 (QCS9075): | [Daisy-Chain Detection & Pose](./daisychain-detection-pose.md) | Cascaded detection + pose | YOLOX + HRNet | DSP | | [Parallel AI Fusion](./parallel-inference.md) | 4-model simultaneous inference | YOLOX + InceptionV3 + HRNet + DeepLabV3+ | DSP | | [Multistream Inference](./multistream-inference.md) | Multi-stream concurrent detection | YOLOX | DSP | +| [Multistream Batch](./multistream-batch-inference.md) | Batch multi-stream detection | YOLOv8 (batch=4) | DSP | +| [Multi-Input Multi-Output](./multi-input-output.md) | Multi-input/output detection | YOLOv5 | DSP | | [Event Encoder](./event-encoder.md) | Detection event encoding | YOLOX | DSP | | [Metadata Parser](./metadata-parser.md) | Detection metadata export | YOLOX | DSP | @@ -134,8 +134,9 @@ The following apps have been verified on Q900 (QCS9075): ## Reference -- [Python GStreamer Apps](./python-apps.md) — Python binding examples -- [Build from Source](./build-from-source.md) — Compile custom GStreamer apps on-device +- [Python GStreamer Apps](./python-apps.md) — Python approach for custom pre/postprocessing +- [Build from Source](./build-from-source.md) — C/C++ approach for deep plugin customization +- [Model Export](./model-export.md) — Export custom YOLOv5/YOLOv8 models ## Troubleshooting @@ -146,7 +147,7 @@ Check that the Wayland socket exists: ```bash -ls $XDG_RUNTIME_DIR/$WAYLAND_DISPLAY +ls $XDG_RUNTIME_DIR/wayland-0 ``` diff --git a/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/build-from-source.md b/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/build-from-source.md index 48e05b5cb3..d9c147b51d 100644 --- a/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/build-from-source.md +++ b/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/build-from-source.md @@ -1,61 +1,103 @@ --- -sidebar_position: 16 +sidebar_position: 18 doc_kind: page locale: en board: airbox-q900 task_type: ai-dev -last_verified: 2026-06-24 +last_verified: 2026-06-30 --- # Build from Source -All QIM SDK GStreamer plugins and sample applications are already available as pre-built packages via [apt installation](./README.md#installation). In most cases, source compilation is unnecessary. +The [Demos](./README.md#available-sample-applications) (`gst-ai-*`) and [Python Apps](./python-apps.md) use precompiled GStreamer plugins from apt. If you need to **modify plugin source code** (e.g., customize postprocessing logic, add custom model support) or **write your own C/C++ inference programs**, you can compile directly on the device. -You can also compile from source directly on the device, which is useful for: +## Hello-QIM: Minimal C/C++ Example -- Modifying plugin source code to customize post-processing logic or add custom model support -- Debugging plugin internals -- Using build options that differ from the pre-built packages +Hello-QIM is a minimal runnable GStreamer application showing how to create a pipeline and capture camera frames in C++. Use it as a starting point for custom applications. -The build produces 34 GStreamer plugin shared libraries (`.so`), covering AI inference engines (`mltflite`, `mlqnn`, `mlsnpe`), preprocessing (`mlvconverter`), post-processing (`mlpostprocess` and others), and video processing (`vcomposer` and others). See the steps below. +### Download Source -## Prerequisites + -- Completed [QIM SDK Installation](./README.md#installation) +```bash +git clone https://github.com/quic/sample-apps-for-qualcomm-linux +cd sample-apps-for-qualcomm-linux/Hello-QIM +``` -## Install Build Dependencies + + +### Prepare Source File ```bash -sudo apt-add-repository -s ppa:ubuntu-qcom-iot/qcom-ppa -sudo apt update -sudo apt install -y \ - qcom-adreno-dev \ - libgstreamer1.0-qcom-sample-apps-utils-dev +cp main.cc gst-appsink.cc ``` -## Build QIM Plugins from Source +### Create Makefile + +```makefile +CC = g++ +CFLAGS = -Wall -g $(shell pkg-config --cflags gstreamer-1.0) +LDFLAGS = $(shell pkg-config --libs gstreamer-1.0) + +all: gst-appsink + +gst-appsink: gst-appsink.o + $(CC) -o $@ $^ $(LDFLAGS) + +gst-appsink.o: gst-appsink.cc + $(CC) $(CFLAGS) -c $< + +clean: + rm -f gst-appsink gst-appsink.o +``` -### Step 1: Download QIM Source +### Compile and Run ```bash -cd ~ -apt source gst-plugins-qti-oss -cd gst-plugins-qti-oss-* +make +./gst-appsink -w 1280 -h 720 ``` -### Step 2: CMake Build +Expected output: + +```text +Hello-QIM: Success creating pipeline and received camera frame. +``` + +> If the device has no camera connected, pipeline creation still succeeds — it just won't receive frames. + +## Compiling QIM Plugins (Advanced) + +If you need to modify the GStreamer plugins themselves (not just call them), compile all plugins from source. + +### Install Build Dependencies ```bash +sudo apt install -y \ + qcom-adreno-dev \ + libgstreamer1.0-qcom-sample-apps-utils-dev +``` + + + +### Download and Compile + + + +```bash +cd ~ +apt source gst-plugins-qti-oss +cd gst-plugins-qti-oss-* mkdir build && cd build cmake \ @@ -71,33 +113,18 @@ cmake \ .. make -j$(nproc) -``` - - - -### Step 3: Install - - - -```bash sudo make install ``` -### Step 4: Verify - -```bash -ls /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstqti*.so | wc -l -``` - -Approximately 34 plugin shared libraries are installed, such as `libgstqtimltflite.so`, `libgstqtimlvclassification.so`, etc. +This produces approximately 34 plugin shared libraries (`libgstqtimltflite.so`, `libgstqtimlvclassification.so`, etc.). ## Troubleshooting ### CMake cannot find GStreamer -Verify GStreamer development packages are installed: +Check that GStreamer development packages are installed: @@ -107,7 +134,7 @@ pkg-config --modversion gstreamer-1.0 -If not found, install: +If not found: @@ -116,7 +143,3 @@ sudo apt install libgstreamer1.0-dev ``` - -## Reference - -- [Python GStreamer Apps](./python-apps.md) — Python development approach diff --git a/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/event-encoder.md b/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/event-encoder.md index 9068c3b832..7d57246a7c 100644 --- a/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/event-encoder.md +++ b/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/event-encoder.md @@ -1,5 +1,5 @@ --- -sidebar_position: 13 +sidebar_position: 15 doc_kind: page locale: en board: airbox-q900 diff --git a/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/metadata-parser.md b/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/metadata-parser.md index 8984a9d305..2ec296f80a 100644 --- a/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/metadata-parser.md +++ b/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/metadata-parser.md @@ -1,5 +1,5 @@ --- -sidebar_position: 14 +sidebar_position: 16 doc_kind: page locale: en board: airbox-q900 diff --git a/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/model-export.md b/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/model-export.md new file mode 100644 index 0000000000..48275cb403 --- /dev/null +++ b/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/model-export.md @@ -0,0 +1,109 @@ +--- +sidebar_position: 19 +doc_kind: page +locale: en +board: airbox-q900 +task_type: ai-dev +last_verified: 2026-06-29 +--- + +# Model Export + +The IM SDK download script (`download_artifacts.sh`) provides only 10 base TFLite models. The following demos require custom-exported models: + +| Demo | Required Model | Export Method | +| --------------------------------------------------------------- | ------------------------------------ | ------------------------- | +| [Multi-Input Multi-Output](./multi-input-output.md) | YOLOv5 (`yolov5.tflite`) | Export from YOLOv5 source | +| [Multistream Batch Inference](./multistream-batch-inference.md) | YOLOv8 batch=4 (`yolov8_det.tflite`) | Qualcomm AI Hub | +| [Object Detection](./object-detection.md) (YOLOv8/YOLO-NAS) | YOLOv8 / YOLO-NAS | Qualcomm AI Hub | + +## Method 1: Export from YOLOv5 Source (for Multi-Input Multi-Output) + + + +```bash +git clone https://github.com/ultralytics/yolov5.git +cd yolov5 +python -m pip install -r requirements.txt tensorflow-cpu +python export.py --weights yolov5m.pt --img 320 --include tflite --int8 --data data/coco128.yaml +``` + + + +Push the exported model to the device: + + + +```bash +scp yolov5m-int8.tflite radxa@:/etc/models/yolov5.tflite +``` + + + + + +```bash +# Create yolov5 label file (copy yolonas labels) +sudo cp /etc/labels/yolonas.labels /etc/labels/yolov5.labels +``` + + + +> This method uses `--img 320` to match the demo's input resolution. **Do not** use the default 640, or inference will fail due to input size mismatch. + +## Method 2: Qualcomm AI Hub Export for YOLOv8 (Object Detection, Multistream Batch) + +### 1. Register and Get API Token + +[Qualcomm AI Hub](https://aihub.qualcomm.com/) → Settings → API Token. + +### 2. Install qai-hub-models + + + +```bash +python -m venv qaihm +source qaihm/bin/activate +pip install qai-hub-models +``` + + + +### 3. Export Standard Model (batch=1) + + + +```bash +python -m qai_hub_models.models.yolov8_det.export \ + --quantize w8a8 \ + --target-runtime=tflite \ + --device "Dragonwing IQ-9075 EVK" +``` + + + +### 4. Export Batch Model (batch=4, for Multistream Batch Inference) + + + +```bash +python -m qai_hub_models.models.yolov8_det.export \ + --quantize w8a8 \ + --target-runtime=tflite \ + --device "Dragonwing IQ-9075 EVK" \ + --batch-size 4 +``` + + + +> `--batch-size 4` must match the number of streams in the demo. Q900 (QCS9075) supports up to 4 streams. + +### 5. Push to Device + + + +```bash +scp yolov8_det.tflite radxa@:/etc/models/ +``` + + diff --git a/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/multi-input-output.md b/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/multi-input-output.md new file mode 100644 index 0000000000..12c9ba3296 --- /dev/null +++ b/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/multi-input-output.md @@ -0,0 +1,151 @@ +--- +sidebar_position: 14 +doc_kind: page +locale: en +board: airbox-q900 +task_type: ai-dev +last_verified: 2026-06-29 +--- + +# Multi-Input Multi-Output Inference + +`gst-ai-multi-input-output-object-detection` runs object detection on multiple input sources simultaneously, with independent inference per stream. It supports combinations of file, RTSP, and camera inputs, with output to display, file, or RTSP stream. + +Maximum 6 concurrent input streams. + +## Prerequisites + +- Completed [QIM SDK Installation](./README.md#installation) and [Model Download](./README.md#download-models-and-test-resources) +- YOLOv5 TFLite model (not included in the default download; convert via Qualcomm AI Hub) + +## Steps + +### 1. Prepare Input Videos + +The default configuration uses 6 file inputs. Copy the test video: + + + +```bash +for i in $(seq 1 6); do + sudo cp /etc/media/video.mp4 /etc/media/video${i}.mp4 +done +``` + + + +### 2. Prepare YOLOv5 Model + +The default configuration references `/etc/models/yolov5.tflite`, which is not included in the download script. Export from YOLOv5 source. + +On the host machine: + + + +```bash +git clone https://github.com/ultralytics/yolov5.git +cd yolov5 +python -m pip install -r requirements.txt tensorflow-cpu +python export.py --weights yolov5m.pt --img 320 --include tflite --int8 --data data/coco128.yaml +``` + + + +> **`--img 320` is required**: this demo's input resolution is 320×320. Using the default 640 will cause inference failure due to size mismatch. + +Push the model to the device and create the label file: + + + +```bash +scp yolov5m-int8.tflite radxa@:/etc/models/yolov5.tflite +``` + + + + + +```bash +sudo cp /etc/labels/yolonas.labels /etc/labels/yolov5.labels +``` + + + +> **Note**: This demo does NOT support the YOLOX model (`yolox_quantized.tflite`). The YOLOX model included in the download script works with other demos (e.g., `gst-ai-object-detection`), but this specific demo requires YOLOv5. + +### 3. Modify Configuration + + + +```bash +python3 -c " +import json +with open('/etc/configs/config-multi-input-output-object-detection.json') as f: + c = json.load(f) +c['model'] = '/etc/models/yolov5.tflite' +c['labels'] = '/etc/labels/yolov5.labels' +json.dump(c, open('/tmp/cfg_multi_in_out.json', 'w'), indent=2) +print('config written') +" +``` + +### 4. Run + + + +```bash +gst-ai-multi-input-output-object-detection --config-file=/tmp/cfg_multi_in_out.json +``` + + + +Press `Ctrl + C` to stop. + +## Expected Output + +```text +TARGET Can support file source, RTSP source and camera source +Run app with model: ... and labels: ... +IN Options: camera: 0 (id: 0), file: 6, rtsp: 0 +OUT Options: display: 1, file: (null), rtsp: 0 +VERBOSE: Replacing 334 out of 334 node(s) with delegate (TfLiteQnnDelegate) node +Pipeline state changed from NULL to READY: +Pipeline state changed from READY to PAUSED: +Pipeline state changed from PAUSED to PLAYING: +``` + +The display shows 6 concurrent detection streams, each with independent bounding boxes. + +## Validation + +- 334 ops all delegated to DSP +- Pipeline reaches `PLAYING` state +- Display shows 6 concurrent detection streams with independent bounding boxes + +## Configuration Reference + +| Field | Description | Default | +| ------------------ | ------------------------------------ | -------------------------------- | +| `input-file-path` | File input paths (JSON array, max 6) | `["/etc/media/video1.mp4", ...]` | +| `input-rtsp-path` | RTSP input URLs (JSON array, max 6) | empty | +| `num-camera` | Number of camera streams (0–2) | `0` | +| `camera-id` | Camera device ID | `0` | +| `model` | Object detection model path | `/etc/models/yolov5.tflite` | +| `labels` | Label file path | `/etc/labels/yolov5.json` | +| `output-display` | Enable display output | `true` | +| `output-file-path` | Output file path (optional) | empty | +| `output-rtsp-path` | RTSP output URL (optional) | empty | + +## Pipeline Flow + +```text +filesrc × 6 → qtdemux → h264parse → v4l2h264dec → qtimlvconverter + ↓ + qtimltflite (DSP inference) + ↓ + qtimlvdetection (YOLO postprocess) + ↓ + qtivcomposer + ↓ + waylandsink +``` diff --git a/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/multistream-batch-inference.md b/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/multistream-batch-inference.md new file mode 100644 index 0000000000..93123d42a8 --- /dev/null +++ b/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/multistream-batch-inference.md @@ -0,0 +1,122 @@ +--- +sidebar_position: 13 +doc_kind: page +locale: en +board: airbox-q900 +task_type: ai-dev +last_verified: 2026-06-29 +--- + +# Multistream Batch Inference + +`gst-ai-multistream-batch-inference` performs batch AI inference (object detection or image segmentation) on multiple video streams simultaneously, batching streams for a single inference pass to improve throughput. + +Q900 (QCS9075) supports up to 24 concurrent input streams (6 batches × batch-size 4). + +## Prerequisites + +- Completed [QIM SDK Installation](./README.md#installation) +- YOLOv8 TFLite model with batch-size=4 (export via Qualcomm AI Hub; see [Model Export](./model-export.md#method-2-qualcomm-ai-hub-export-for-yolov8-object-detection-multistream-batch)) + +## Steps + +### 1. Export Batch Model + +Export YOLOv8 with `--batch-size 4`. See [Model Export](./model-export.md#4-export-batch-model-batch4-for-multistream-batch-inference) for the full command. + +Push the model and labels to the device: + + + +```bash +scp yolov8_det.tflite radxa@:/etc/models/ +``` + + + +### 2. Create Configuration + + + +```bash +python3 -c " +import json +with open('/etc/configs/config-multistream-batch-inference.json') as f: + base = json.load(f) +entry = base['pipeline-info'][0] +entry['model-path'] = '/etc/models/yolov8_det.tflite' +entry['labels-path'] = '/etc/labels/yolov8.json' +# Generate 6 batches × 4 streams = 24 streams (id max=5) +base['pipeline-info'] = [] +for i in range(6): + e = json.loads(json.dumps(entry)) + e['id'] = i + base['pipeline-info'].append(e) +with open('/tmp/cfg_batch.json', 'w') as f: + json.dump(base, f, indent=2) +print('24-stream config written') +" +``` + + + +### 3. Run + + + +```bash +gst-ai-multistream-batch-inference --config-file=/tmp/cfg_batch.json +``` + + + +Press `Ctrl + C` to stop. + +## Expected Output + +```text +VERBOSE: Replacing 282 out of 282 node(s) with delegate (TfLiteQnnDelegate) node +Pipeline state changed from NULL to READY: +Pipeline state changed from READY to PAUSED: +Pipeline state changed from PAUSED to PLAYING: +``` + +The display shows 24 concurrent detection streams. + +## Validation + +- All ops delegated to DSP +- Pipeline reaches `PLAYING` state +- 24 streams display correct bounding boxes simultaneously + +## Configuration Reference + +| Field | Description | Default | +| ------------------------------------- | ------------------------------------------------------------- | ----------------- | +| `output-type` | Output type: `wayland` / `filesink` | `wayland` | +| `pipeline-info[].id` | Batch ID (0–5) | `0` | +| `pipeline-info[].input-type` | Input source type | `file` | +| `pipeline-info[].input-file-path` | Input files (4 per batch) | See config | +| `pipeline-info[].mlframework` | Inference framework | `tflite` | +| `pipeline-info[].model-path` | Model path | Must be replaced | +| `pipeline-info[].labels-path` | Label path | Must be replaced | +| `pipeline-info[].constants` | LiteRT quantization constants | Model-specific | +| `pipeline-info[].post-process-plugin` | Post-process plugin: `qtimlvdetection` / `qtimlvsegmentation` | `qtimlvdetection` | + +## Pipeline Flow + +```text +filesrc × 4 → qtdemux → h264parse → v4l2h264dec → qtibatch (batch) + ↓ + qtimlvconverter + ↓ + qtimltflite (DSP batch inference) + ↓ + qtimldemux (debatch) + ↓ + qtimlvdetection / qtimlvsegmentation + ↓ + qtivcomposer + ↓ + waylandsink +``` diff --git a/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/multistream-inference.md b/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/multistream-inference.md index 8ca4474d7c..abfb44b042 100644 --- a/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/multistream-inference.md +++ b/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/multistream-inference.md @@ -63,7 +63,7 @@ python3 -c " import json with open('/etc/configs/config-multistream-inference.json') as f: c = json.load(f) -c['input-file-path'] = ['/etc/media/video_safe.mp4'] * 16 +c['input-file-path'] = ['/etc/media/video_safe.mp4'] * 32 json.dump(c, open('/tmp/cfg_multistream.json', 'w'), indent=2) print('config written') " diff --git a/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/python-apps.md b/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/python-apps.md index e157a68a41..ff8c57687f 100644 --- a/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/python-apps.md +++ b/i18n/en/docusaurus-plugin-content-docs/current/fogwise/airbox-q900/ai-dev/qim-sdk/python-apps.md @@ -1,44 +1,40 @@ --- -sidebar_position: 15 +sidebar_position: 17 doc_kind: page locale: en board: airbox-q900 task_type: ai-dev -last_verified: 2026-06-24 +last_verified: 2026-06-30 --- # Python GStreamer Apps -QIM SDK provides Python GStreamer binding examples, demonstrating how to build AI/ML and multimedia pipelines using Python. +The [Demos](./README.md#available-sample-applications) (`gst-ai-*`) are precompiled CLI tools controlled via JSON config files. If you need to insert custom processing logic into the pipeline (e.g., OpenCV preprocessing, custom postprocessing), use the Python GStreamer bindings to build pipelines directly. + +QIM SDK Python examples are installed via the `gstreamer1.0-qcom-python-examples` package. Scripts are located in `/usr/bin/` and use the **same underlying GStreamer plugins** (`qtimltflite`, `qtimlvdetection`, `qtivcomposer`, etc.) as the `gst-ai-*` demos, but use argparse command-line arguments instead of config JSON. ## Prerequisites - Completed [QIM SDK Installation](./README.md#installation) -Python examples are installed via the `gstreamer1.0-qcom-python-examples` package. Related Python dependencies (`python3-gst-1.0`, `python3-numpy`, `python3-opencv`) are installed automatically. - -## Available Examples - -| Application | Description | -| ----------------------------------------- | ---------------------------------------- | -| Camera Encode | Capture from camera and encode to H.264 | -| OpenCV Camera Stream | Read camera stream via OpenCV | -| OpenCV Video Convert | OpenCV video format conversion | -| Concurrent Video Playback (Video Wall) | Multi-stream simultaneous playback | -| Multi-Camera Stream | Python-controlled multi-camera | -| Object Detection & Display | Run object detection and display results | -| RTSP Stream Decode & Detection | Decode from RTSP and detect | -| JPEG Image Decode | Decode JPEG images | -| Object Detection & Classification | Cascaded detection + classification | -| Convert & Encode Camera Stream | Camera stream transcoding | -| Camera Encode + Detection + Display | End-to-end pipeline | -| Detection + Classification + Segmentation | Multi-task AI | -| Parallel Inference | Multi-model parallel | -| Daisy-Chain Detection & Pose | Cascaded detection + pose | +Python dependencies (`python3-gst-1.0`, `python3-numpy`, `python3-opencv`) are installed automatically. + +## Representative Examples + +18 runnable scripts are installed. Here are a few representative ones: + +| Script | Description | Example | +| ----------------------------------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `gst-ai-object-detection.py` | Object detection | `python3 /usr/bin/gst-ai-object-detection.py -s /etc/media/video.mp4 -f 2 -ml yolov5 -m /etc/models/yolov5m-int8.tflite -l /etc/labels/yolov5.json --use_dsp` | +| `gst-parallel-inference.py` | Multi-model parallel inference | Run detection + classification + segmentation simultaneously | +| `gst-daisychain-detection-pose.py` | Cascaded detection + pose | `--file /etc/media/video.mp4 --tflite-yolo-model ... --tflite-pose-model ...` | +| `gst-concurrent-videoplay-composition.py` | Video wall | `--infile /etc/media/video.mp4 -c 4` (4 concurrent streams) | + +Use `--help` on any script to see its full parameter list. ## Basic Structure -Python GStreamer apps use PyGObject bindings: +Python GStreamer apps use PyGObject bindings. Minimal example: ```python import gi @@ -63,6 +59,8 @@ loop.run() ## Get Source Code +Full source code for all Python examples is on GitHub: + ```bash @@ -73,4 +71,5 @@ git clone https://github.com/quic/sample-apps-for-qualcomm-linux ## Reference -- [Build from Source](./build-from-source.md) — Compile custom C/C++ apps on-device +- [Demos](./README.md#available-sample-applications) — Precompiled CLI, controlled via config JSON +- [Build from Source](./build-from-source.md) — C/C++ approach for deep plugin customization