Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -15,5 +15,9 @@ venv
.coverage
dist
build
/*.pt
/yolo11n.onnx
/yolo26n.onnx
/mobileclip*.ts
*.log
terminals
7 changes: 4 additions & 3 deletions .github/workflows/backend-metadata.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,11 @@ jobs:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.11'
- run: pip install 'numpy<2' tflite-runtime==2.14.0 fastapi pillow
python-version: '3.14'
- run: pip install numpy==2.5.4 ai-edge-litert==2.3.0 fastapi==0.162.5 pillow==13.0.0
- run: python -m unittest discover -s tests -p test_backend_model_info.py -v
- run: python -m unittest discover -s tests -p test_runtime_requirements.py -v
- name: Install API startup test dependencies
run: pip install pydantic-settings uvicorn httpx python-multipart shapely sentry-sdk
run: pip install pydantic-settings uvicorn httpx2 python-multipart shapely sentry-sdk
- name: Verify default readiness and lazy optional engines
run: python -m unittest discover -s tests -p test_startup.py -v
1 change: 1 addition & 0 deletions .github/workflows/docker-publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ jobs:
uses: docker/build-push-action@v6
with:
context: .
target: runtime
push: true
platforms: linux/amd64,linux/arm64
tags: ${{ steps.meta.outputs.tags }}
Expand Down
8 changes: 7 additions & 1 deletion .github/workflows/hosting-runtime.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@ on:
branches: [main]
pull_request:
paths:
- Dockerfile
- Pipfile
- Pipfile.lock
- requirements-runtime.txt
- backends/onnx/**
- scripts/hosting_*.py
- scripts/test_hosting_release.py
- scripts/wait_hosting_release.py
Expand Down Expand Up @@ -34,7 +39,8 @@ jobs:
id: build
with:
context: .
file: k8s/local-hosting/Dockerfile
file: Dockerfile
target: hosting
build-args: |
SENTRY_RELEASE=light-object-detect@${{ github.sha }}
platforms: linux/amd64
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/model-install-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,5 @@ jobs:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- run: python -m unittest discover -s tests -p test_model_download.py -v
python-version: '3.14'
- run: python -m unittest discover -s tests -p 'test_model_*.py' -v
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ backends/tflite/models/

# PyTorch weights (used only for export, not deployed)
*.pt
/mobileclip*.ts

# Logs
*.log
Expand Down
45 changes: 23 additions & 22 deletions BACKEND_CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ This guide explains how to configure different detection backends for both light
## Overview

The light-object-detect API supports three detection backends:
- **ONNX Runtime**: Best performance with CUDA, excellent accuracy with YOLOv8
- **ONNX Runtime**: Default YOLO26 backend, with CPU, CUDA, and OpenVINO provider options
- **TFLite**: Lightweight, good for embedded systems, moderate accuracy
- **OpenCV DNN**: Good balance, supports many formats, decent performance

Expand All @@ -32,23 +32,20 @@ This will download the model to `backends/tflite/models/` and create the labels

### ONNX Models

Download YOLOv8 ONNX models:
Download YOLO26 ONNX models:

```bash
# Using the download script (requires wget)
bash scripts/download_models.sh --model-size n --output-dir models

# Or using Python (requires ultralytics package)
pip install ultralytics
python3 scripts/download_models.py --model-size n --output-dir models
# Using the model authoring environment
pipenv sync --dev
pipenv run python scripts/download_models.py --family yolo26 --model-size n
```

Available model sizes:
- `n` (nano): ~6MB, fastest
- `s` (small): ~22MB, balanced
- `m` (medium): ~52MB, better accuracy
- `l` (large): ~87MB, high accuracy
- `x` (xlarge): ~136MB, best accuracy
- `n` (nano): fastest
- `s` (small): balanced
- `m` (medium): better accuracy
- `l` (large): high accuracy
- `x` (xlarge): highest accuracy and cost

### OpenCV DNN Models

Expand Down Expand Up @@ -83,11 +80,11 @@ TFLITE_CONFIDENCE_THRESHOLD=0.5
**For ONNX:**
```bash
DEFAULT_BACKEND=onnx
ONNX_MODEL_PATH=models/yolov8n.onnx
ONNX_LABELS_PATH=models/coco_labels.txt
ONNX_MODEL_PATH=backends/onnx/models/yolo26n-distill.onnx
ONNX_LABELS_PATH=backends/onnx/models/coco.txt
ONNX_CONFIDENCE_THRESHOLD=0.5
ONNX_IOU_THRESHOLD=0.45
ONNX_MODEL_TYPE=yolov8
ONNX_MODEL_TYPE=yolo26
```

**For OpenCV:**
Expand Down Expand Up @@ -220,7 +217,7 @@ cd light-object-detect
bash scripts/download_models.sh --model-size n --output-dir models

# Verify the file exists
ls -lh models/yolov8n.onnx
ls -lh backends/onnx/models/yolo26n-distill.onnx
```

### lightNVR Not Calling API
Expand Down Expand Up @@ -324,18 +321,22 @@ needs the `intel-opencl-icd` OpenCL driver *inside the image* and `--device
`intel-driver-compiler-npu` / `intel-level-zero-npu` packages and
`--device /dev/accel`.

In Docker, build with both args:
`onnxruntime-openvino` 1.24.1 currently has no Python 3.14 wheel. Use the
Python 3.13 Bookworm variant for Intel GPU builds until a compatible wheel is
published. The normal ONNX CPU/CUDA image uses Python 3.14.

In Docker, build with these args:

```bash
docker build \
--build-arg ONNXRUNTIME_PACKAGE=onnxruntime-openvino \
--build-arg BASE_IMAGE=python:3.13.18-slim-bookworm \
--build-arg ONNXRUNTIME_PACKAGE=onnxruntime-openvino==1.24.1 \
--build-arg INSTALL_INTEL_GPU=1 .
```

`INSTALL_INTEL_GPU=1` requires a Debian bookworm base — trixie dropped
`intel-opencl-icd` from its repositories. The Dockerfile pins
`BASE_IMAGE=python:3.11-slim-bookworm` for exactly this reason; if you override
`BASE_IMAGE` with a trixie variant, the GPU driver install will fail with a
`intel-opencl-icd` from its repositories. If you use a trixie variant, the
GPU driver install will fail with a
message telling you so. Without the driver the container still runs, but
OpenVINO sees no GPU and silently uses the CPU device.

Expand Down
59 changes: 21 additions & 38 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,11 +1,7 @@
# Overridable so the INSTALL_INTEL_GPU build can pin an older Debian. The
# default stays on the floating tag (currently trixie) because the arm64 Coral
# runtime installed below is a trixie .deb; INSTALL_INTEL_GPU needs
# intel-opencl-icd, which trixie dropped, so that build must pass
# --build-arg BASE_IMAGE=python:3.11-slim-bookworm. The two are mutually
# exclusive, which is harmless in practice: Intel iGPU builds are never arm64.
ARG BASE_IMAGE=python:3.11-slim
FROM ${BASE_IMAGE}
# Python 3.14 is the newest release supported by ONNX Runtime wheels.
# The arm64 Coral runtime below targets Debian trixie.
ARG BASE_IMAGE=python:3.14.10-slim-trixie
FROM ${BASE_IMAGE} AS base

ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
Expand Down Expand Up @@ -38,29 +34,12 @@ RUN apt-get update \
# onnxruntime-openvino Intel CPU / iGPU / NPU via the OpenVINO provider
# onnxruntime-gpu NVIDIA CUDA
# Usage: docker build --build-arg ONNXRUNTIME_PACKAGE=onnxruntime-openvino ...
ARG ONNXRUNTIME_PACKAGE=onnxruntime
ARG ONNXRUNTIME_PACKAGE=onnxruntime==1.31.1

# Core Python deps (ONNX is now the default backend)
COPY Pipfile Pipfile.lock ./
COPY requirements-runtime.txt ./
RUN python -m pip install --upgrade pip \
&& python -m pip install \
sentry-sdk==2.73.0 \
fastapi \
uvicorn \
python-multipart \
pydantic \
pydantic-settings \
pillow \
exceptiongroup \
"numpy<2.0" \
"$ONNXRUNTIME_PACKAGE" \
opencv-python \
scipy \
shapely

# Install tflite-runtime for Coral Edge TPU support
# Compatible with feranick's libedgetpu 16.0TF2.19.1-1
RUN pip install tflite-runtime==2.14.0
&& python -m pip install -r requirements-runtime.txt "$ONNXRUNTIME_PACKAGE"

# Optional: Intel GPU userspace driver, required for OpenVINO's GPU device.
# The OpenVINO wheel ships the GPU plugin but still needs the host OpenCL
Expand All @@ -72,29 +51,24 @@ RUN if [ "$INSTALL_INTEL_GPU" = "1" ]; then \
&& { apt-get install -y --no-install-recommends intel-opencl-icd clinfo \
|| { echo "ERROR: intel-opencl-icd is not in this base image's repos."; \
echo "Debian trixie dropped it — build with"; \
echo " --build-arg BASE_IMAGE=python:3.11-slim-bookworm"; \
echo " --build-arg BASE_IMAGE=python:3.13.18-slim-bookworm"; \
exit 1; }; } \
&& rm -rf /var/lib/apt/lists/*; \
fi

# Optional: install tensorflow for tflite backend support
# Usage: docker build --build-arg INSTALL_TENSORFLOW=1 ...
ARG INSTALL_TENSORFLOW=0
RUN if [ "$INSTALL_TENSORFLOW" = "1" ]; then python -m pip install tensorflow; fi

# Optional: install moondream (torch + transformers) for VLM backend
# Usage: docker build --build-arg INSTALL_MOONDREAM=1 ...
ARG INSTALL_MOONDREAM=0
RUN if [ "$INSTALL_MOONDREAM" = "1" ]; then \
python -m pip install \
"transformers>=4.51.1,<5.0" \
"torch>=2.7.0" \
"accelerate>=1.10.0"; \
"transformers==4.57.6" \
"torch==2.15.0" \
"accelerate==1.15.0"; \
fi

COPY . .

# Optionally bake the TFLite model for the default tflite-runtime interpreter
# Optionally bake the TFLite model for LiteRT
# Usage: docker build --build-arg DOWNLOAD_DEFAULT_MODEL=1 ...
ARG DOWNLOAD_DEFAULT_MODEL=0
RUN if [ "$DOWNLOAD_DEFAULT_MODEL" = "1" ]; then python scripts/download_model.py; fi
Expand All @@ -109,3 +83,12 @@ ENV SENTRY_DSN=https://8bd44d815901211012fcae52d51371f9@o4512337837686784.ingest
SENTRY_RELEASE=${SENTRY_RELEASE}

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

# The hosting image uses the same Python and dependencies as the published
# runtime, with its CPU quota settings and the verified distilled model.
FROM base AS hosting
ENV ONNX_INTRA_OP_THREADS=2
RUN python -c 'from backends.onnx.backend import ONNXBackend; from PIL import Image; b = ONNXBackend("backends/onnx/models/yolo26n-distill.onnx", "backends/onnx/models/coco.txt", model_type="yolo26"); assert b.session.get_session_options().intra_op_num_threads == 2; assert [o.shape for o in b.session.get_outputs()] == [[1, 300, 6]]; b.detect(Image.open("test_image.jpg"))'

# Keep the ordinary runtime as the default docker build target.
FROM base AS runtime
44 changes: 22 additions & 22 deletions Pipfile
Original file line number Diff line number Diff line change
Expand Up @@ -4,32 +4,32 @@ verify_ssl = true
name = "pypi"

[packages]
sentry-sdk = "==2.73.0"
fastapi = "*"
uvicorn = "*"
python-multipart = "*"
pydantic = "*"
pydantic-settings = "*"
pillow = "*"
exceptiongroup = "*"
numpy = "*"
tensorflow = "*"
sentry-sdk = "==2.75.0"
fastapi = "==0.162.5"
uvicorn = "==0.54.0"
python-multipart = "==0.0.33"
pydantic = "==2.14.3"
pydantic-settings = "==2.16.2"
pillow = "==13.0.0"
numpy = "==2.5.4"
# Swap for onnxruntime-openvino (Intel CPU/iGPU/NPU) or onnxruntime-gpu (CUDA)
# to enable those execution providers. They replace this wheel; do not install
# more than one. See BACKEND_CONFIGURATION.md.
onnxruntime = "==1.23.2"
opencv-python = "*"
scipy = "*"
shapely = "*"
transformers = ">=4.51.1,<5.0"
torch = ">=2.7.0"
accelerate = ">=1.10.0"
# more than one. The OpenVINO wheel currently requires Python 3.13 or older.
# See BACKEND_CONFIGURATION.md.
onnxruntime = "==1.31.1"
opencv-python = "==5.0.0.93"
shapely = "==2.2.0"
ai-edge-litert = "==2.3.0"

[dev-packages]
ultralytics = "*"
# Optional: the suites in tests/ are stdlib-only and self-run under plain
# `python3 tests/test_*.py`, so pytest is a convenience, not a requirement.
ultralytics = "==8.4.211"
transformers = ">=4.51.1,<5.0"
accelerate = ">=1.10.0,<2.0"
onnx = "*"
onnxslim = "*"
# The unit suites use unittest; pytest is a convenience for local runs.
pytest = "*"
httpx2 = "*"

[requires]
python_version = "3.13"
python_version = "3.14"
Loading
Loading