A backend for computer-vision dataset curation and training, with a fast inference API underneath. Ingest images, detect and crop regions, label them with people and a vision-language model, import and combine datasets, export, train and promote models. It works for any image domain, not one built-in use case.
Underneath is a unified REST API on NVIDIA Triton Inference Server with TensorRT engines: object detection, face recognition, visual search, OCR and embeddings.
The optional web UI, Cropwright, is a frontend for the curation API. Every
route also works from curl, httpx or the generated OpenAPI client. See
docs/VISION_AND_GOALS.md for the project's scope
and standards.
Where to start. Not sure whether you need the inference API or the curation
tools? Read Choose your path.
The user documentation lives in docs-site/docs/; this
README is the overview, the install and a first result.
Inference API (port 4603, /v1 twin for the core routes)
- Object detection (YOLO11 and YOLO26 side by side), batch up to 64 images.
- Face detection, ArcFace embeddings, 1:1 verify, search and 1:N identify.
- MobileCLIP image and text embeddings, visual search over OpenSearch k-NN.
- PP-OCRv5 text detection and recognition.
- Combined analysis, ingest into the visual-search indexes, FAISS clustering and albums.
Curation and training (/curation, all project scoped)
- Isolated projects. Each project has its own indexes, directories, class registry, settings and jobs. Isolation is enforced at the OpenSearch transport layer.
- Config you can version. Prompt packs, region profiles, keymaps and the VLM choice are revisioned documents with activate and roll back.
- One or many regions per item.
region_boxesis always a list; each box has its own state, text, cluster and embedding. - Ingest policy. The stock detector stores every detection of its label space as an unlabeled proposal. A per-project policy decides what is kept and what gets a vector.
- VLM labeling. A deployment-wide registry of OpenAI-compatible endpoints with a local model catalog; each project activates one. Remote endpoints need an explicit acknowledgement because crops leave the host.
- Dataset import, combine, export, train, promote. YOLO, COCO and OpenProcessor-export layouts; merge up to eight projects; bake-off model comparison; promotion back into Triton.
- Class identity by name. Import, combine, export, train, promote and model sharing map classes by name, never by index.
- The lock rule. A human-set or import-validated label or box, and anything in the frozen test holdout, is never overwritten by an automated writer.
The whole curation surface is in
contracts/openapi/curation.json; the
guides are under docs-site/docs/guides/.
curl -fsSL https://raw.githubusercontent.com/davidamacey/OpenProcessor/main/setup-openprocessor.sh | bashThis installs the latest published release into ./openprocessor/: no git
clone, no local image build, no host Python. The script you pipe in only
resolves the release, downloads that release's own setup-openprocessor.sh
and SHA256SUMS, checks the checksum, and runs the verified copy. Images are
pinned by digest (images.lock) and checked after the pull. It asks which
tiers you want, picks GPUs, exports the TensorRT engines inside the
containers, starts everything and runs a health check.
Needs: Linux, Docker with Compose v2, an NVIDIA GPU with the NVIDIA
Container Toolkit, and disk for the tiers you pick (about 60 GB for core,
about 135 GB for everything). First install takes 30-60 minutes, mostly image
pulls and TensorRT export.
| Tier | What you get | Extra images | Needs |
|---|---|---|---|
core |
Triton, the API (port 4603), OpenSearch | ~53 GB | ~16 GB VRAM (--profile minimal for 6-8 GB cards) |
curation |
curation workers, PE-Core embeddings, evaluator | + ~15 GB | + ~2 GB VRAM; implies core |
segmenter |
SAM 3 segmenter | + ~7 GB | 2-8 GB VRAM; a HuggingFace token with SAM 3 access (gated); implies curation |
vlm |
local vLLM serving a model from the VLM catalog | + ~19 GB | ~23 GB VRAM for the default; implies curation |
trainer |
training service + MLflow | + ~9 GB | >= 16 GB free VRAM while training; implies curation |
cropwright |
the Cropwright web UI (a profile of the same compose project) | + ~65 MB | implies curation |
Monitoring (Prometheus, Grafana, Loki) is not a tier: add --with-monitoring.
Its dashboards are default-open, so it is off unless you ask.
curl -fsSL https://raw.githubusercontent.com/davidamacey/OpenProcessor/main/setup-openprocessor.sh \
| bash -s -- --tiers core,curation,cropwright --unattended--unattended never prompts: every choice comes from flags or their defaults,
and a step that needs consent (a non-loopback --bind, an external VLM, a
purge) fails unless its consent variable is set. Use --all for every tier,
--version vX.Y.Z to pin a release, --dry-run to see every command without
running any. All flags: INSTALLATION.md.
V=vX.Y.Z # the release you want
curl -fsSLO https://github.com/davidamacey/OpenProcessor/releases/download/$V/setup-openprocessor.sh
curl -fsSLO https://github.com/davidamacey/OpenProcessor/releases/download/$V/SHA256SUMS
grep ' setup-openprocessor.sh$' SHA256SUMS | sha256sum -c -
bash setup-openprocessor.sh --version "$V"SHA256SUMS comes from the same place as the files it covers. It proves
integrity (the download is complete and uncorrupted), not authenticity:
anyone who could replace the release files could replace SHA256SUMS too. The
same holds for images.lock (it pins exact digests
and is covered by SHA256SUMS). Signed releases are follow-up work.
Cropwright is reachable on your LAN by default, for homelab or
small-business use. The API itself stays bound to 127.0.0.1. There is no login
on Cropwright — a warning is shown. Pass --local-only to opt out and keep
everything on 127.0.0.1.
Do not port-forward Cropwright (or any OpenProcessor port) to the public
internet. If you need access beyond a trusted network, put a reverse proxy
with authentication in front. --bind <ip> publishes the API ports on that
address instead (with a warning and a typed confirmation), and a specific
address also narrows Cropwright to that interface. See
SECURITY.md.
cd openprocessor
./openprocessor status # services and health
./openprocessor logs api -f # live logs
./openprocessor sample coco # fetch a public COCO sample (200 images)
./openprocessor vlm list # the local VLM catalog
./openprocessor upgrade # to the latest release (backs up first)
./setup-openprocessor.sh --repair | --rollback | --uninstallcurl http://127.0.0.1:4603/health
curl -X POST http://127.0.0.1:4603/detect -F "image=@your-image.jpg"Every subcommand of the openprocessor CLI is listed in
INSTALLATION.md. The installer sizes
the OpenSearch heap from your RAM (half of it, 2 to 30 GB) and prints it in the
summary; see INSTALLATION.md.
For development, or to build the images yourself:
git clone https://github.com/davidamacey/OpenProcessor.git && cd OpenProcessor && ./scripts/setup.shscripts/setup.sh detects your GPU, picks a profile, downloads the models,
exports them to TensorRT and starts the services. Add --yes for no prompts,
or --profile=standard --gpu=0 --yes to choose explicitly. In a checkout,
manage the stack with make up, make curation-up, make status and
make down. The ./openprocessor CLI also works there once .env has a
COMPOSE_PROJECT_NAME line (the .env that setup.sh writes has none). See
INSTALLATION.md for manual steps.
Images are published on Docker Hub under versioned tags. The installer never
uses :latest; it runs the digests in the release's images.lock:
docker pull davidamacey/openprocessor:<version> # FastAPI service
docker pull davidamacey/openprocessor-triton:<version> # Triton server<version> is the release number without the leading v (the VERSION file).
curl http://127.0.0.1:4603/health
curl -X POST http://127.0.0.1:4603/detect -F "image=@your-image.jpg"Interactive API docs are at http://127.0.0.1:4603/docs. The core routes are
listed under API Endpoints.
Curation routes live under /curation/projects/{project}/.... The default
project exists after the first start; create more with
POST /curation/projects. The slug is 2-32 characters: lowercase letters,
digits and single hyphens, starting with a letter.
API=http://127.0.0.1:4603/curation
# 1. create a project (slug is permanent; display_name is editable)
curl -X POST $API/projects -H 'Content-Type: application/json' \
-d '{"slug": "cars", "display_name": "Cars"}'
# 2. add classes (class identity is the name)
curl -X POST $API/projects/cars/classes -H 'Content-Type: application/json' \
-d '{"name": "car"}'Try it with a public sample. No dataset ships in this repo (nothing proprietary is bundled anywhere). Fetch a small, license-filtered COCO 2017 subset instead:
make sample-coco-readme # 200 images, 20 per class, ~1-2 min on a fast link
# or, from an installed deployment: ./openprocessor sample cocoThis writes data/samples/coco_va_readme/ (images + ATTRIBUTION.csv +
coco_gt.json), gitignored, from a pinned, deterministic selection filtered to
Flickr licenses safe to redistribute crops of (Attribution,
Attribution-ShareAlike, "No known copyright restrictions", "United States
Government Work"; never NonCommercial or NoDerivs). Then, in a source checkout:
# 1. The stock detector stores every detection of its whole label space (all
# 80 COCO classes) as an unlabeled proposal, and you choose what to keep
# later. To store or embed less, set the per-project ingest policy
# (PUT /curation/projects/{project}/ingest/policy); the old
# OP_INGEST_PRIMARY_CLASS_IDS env var is retired and ignored.
# 2. Point OP_SOURCE_ROOT_HOST at data/samples in .env (the compose mount
# target is fixed at /data/source), then recreate the services that mount
# it: api, curation-detection-worker, curation-auto-label-worker.
echo 'OP_SOURCE_ROOT_HOST=./data/samples' >> .env
make up && make curation-up
# 3. --folder is a container path under the /data/source mount, not a
# host-relative one:
docker compose -p openprocessor exec api python scripts/curation/bulk_ingest.py \
--folder /data/source/coco_va_readme/images --project carsAn installed directory already has OP_SOURCE_ROOT_HOST=./data, so skip step 2
and use --folder /data/source/samples/coco_va_readme/images. The full
walkthrough, with the expected result after each step, is the
Quick start.
make sample-coco (800 images plus side sets), make sample-plates (300 Open
Images V7 images) and make sample-clean use the same tool. For a dev
deployment with every profile, make dev-up builds and starts it, make dev-ps
inspects it and make dev-down stops it (volumes are kept). Set
COMPOSE_PROJECT_NAME, OP_IMAGE_REPO and OP_IMAGE_TAG in .env to keep
locally built images separate from published ones.
examples/ holds a complete, public, multi-region example: the primary
detector finds cars, then the region stage finds the wheels on each car.
make sample-coco-cars # 60 CC BY car images into data/samples/coco_car
.venv/bin/python scripts/examples/wheel_example_live.py \
--api http://localhost:4603 --project wheels \
--container-dir /data/source/coco_car/imagesThe script creates the project, creates and activates
examples/region_profiles/vehicle_wheel.json (max_regions_per_item: 4,
text-free) and examples/prompt_packs/vehicle_wheel.json, ingests the images,
waits for the detection worker and exports the wheel boxes. It needs the live
stack with the segmenter and an activated VLM. The same walk runs offline in
tests/integration/test_wheel_example_e2e.py.
| Profile | VRAM | GPUs |
|---|---|---|
| minimal | 6-8GB | RTX 3060, RTX 4060 |
| standard | 12-24GB | RTX 3080, RTX 4090 |
| full | 48GB+ | A6000, A100 |
Choose the profile at install time (--profile on the installer, or
scripts/setup.sh --profile=NAME). Measured throughput is in
docs/PERFORMANCE.md; size your own card with
GPU sizing. Engines are compiled
once and cached, so only the first start is slow.
Dual YOLO families. One Triton instance and one API serve YOLO11 and YOLO26.
-
YOLO11 exports through the EfficientNMS end2end toolchain (
export/export_models.py, GPU-NMS engines, the default path). -
YOLO26 exports through the stock Ultralytics toolchain (
export/export_yolo26.py, natively NMS-free engines):docker compose -p openprocessor exec api python /app/export/export_yolo26.py --models small curl -X POST http://localhost:4603/models/yolo26_small_trt/load -
The API reads each model's output format from Triton metadata, so
/detect?model_name=yolo26_small_trtand the YOLO11 default work interchangeably. SetYOLO_MODEL=yolo26_small_trtto change the default detector. Models load and unload at runtime withPOST /models/{name}/loadandPOST /models/{name}/unload.
All endpoints are on port 4603. The routes below are also under /v1.
| Endpoint | Method | Description |
|---|---|---|
/detect |
POST | YOLO object detection (single image) |
/detect/batch |
POST | Batch detection (up to 64 images) |
| Endpoint | Method | Description |
|---|---|---|
/faces/detect |
POST | Face detection with landmarks (SCRFD) |
/faces/recognize |
POST | Detection + ArcFace 512-dim embeddings |
/faces/verify |
POST | 1:1 face comparison (two images) |
/faces/search |
POST | Find similar faces in index |
/faces/identify |
POST | 1:N face identification |
| Endpoint | Method | Description |
|---|---|---|
/embed/image |
POST | MobileCLIP image embedding (512-dim) |
/embed/text |
POST | MobileCLIP text embedding (512-dim) |
/embed/batch |
POST | Batch image embeddings |
/embed/boxes |
POST | Per-box crop embeddings |
| Endpoint | Method | Description |
|---|---|---|
/search/image |
POST | Image-to-image similarity search |
/search/text |
POST | Text-to-image search |
/search/face |
POST | Face similarity search |
/search/ocr |
POST | Search images by text content |
/search/object |
POST | Object-level search (vehicles, people) |
| Endpoint | Method | Description |
|---|---|---|
/ingest |
POST | Ingest image (auto-indexes faces, OCR, objects) |
/ingest/batch |
POST | Batch ingest (up to 64 images) |
/ingest/directory |
POST | Bulk ingest from server directory |
| Endpoint | Method | Description |
|---|---|---|
/ocr/predict |
POST | Extract text from image (PP-OCRv5) |
/ocr/batch |
POST | Batch OCR processing |
| Endpoint | Method | Description |
|---|---|---|
/analyze |
POST | All models on single image (YOLO + faces + CLIP + OCR) |
/analyze/batch |
POST | Batch combined analysis |
| Endpoint | Method | Description |
|---|---|---|
/clusters/train/{index} |
POST | Train FAISS clustering for an index |
/clusters/stats/{index} |
GET | Get cluster statistics |
/clusters/{index}/{id} |
GET | Get cluster members |
/clusters/albums |
GET | List auto-generated albums |
| Endpoint | Method | Description |
|---|---|---|
/query/image/{id} |
GET | Get stored image data/metadata |
/query/stats |
GET | Index statistics for all indexes |
/query/duplicates |
GET | List duplicate groups |
| Endpoint | Method | Description |
|---|---|---|
/health |
GET | Service health check |
/ready |
GET | Readiness probe: Triton + OpenSearch reachability |
The /curation routes are not mounted under /v1; they exist only at
/curation (the prefix is OP_API_PREFIX). Every curation route is listed in
contracts/openapi/curation.json, grouped
and explained in
docs/design/curation_api_contract.md,
and the TypeScript item types are in contracts/.
This service, the core API and the curation subsystem alike, has no authentication, authorization, or rate limiting. It is built to run on a trusted machine or private network behind your own reverse proxy, never exposed directly to a network you don't trust (and never to the public internet). See SECURITY.md for the routes that are dangerous without access control and the other default-open components (Grafana, OpenSearch) you should lock down before wider deployment.
import requests
# Object Detection
with open('image.jpg', 'rb') as f:
resp = requests.post('http://localhost:4603/detect', files={'image': f})
result = resp.json()
# {"detections": [{"x1": 0.1, "y1": 0.2, "x2": 0.3, "y2": 0.4, "confidence": 0.95, "class_id": 0, "class_name": "person"}], ...}
# Face Recognition
with open('photo.jpg', 'rb') as f:
resp = requests.post('http://localhost:4603/faces/recognize', files={'image': f})
print(resp.json())
# {"num_faces": 2, "faces": [...], "embeddings": [[...512 floats...], ...]}
# Image Embedding
with open('image.jpg', 'rb') as f:
resp = requests.post('http://localhost:4603/embed/image', files={'image': f})
embedding = resp.json()['embedding'] # 512-dim vector
# Text-to-Image Search (query params, not a JSON body -- `text`, not `query`)
resp = requests.post('http://localhost:4603/search/text',
params={'text': 'a red sports car', 'top_k': 10})
results = resp.json()['results']
# Image Ingestion (auto-indexes everything)
with open('photo.jpg', 'rb') as f:
resp = requests.post('http://localhost:4603/ingest',
files={'image': f},
data={'image_id': 'photo_001'})
print(resp.json())
# {"status": "indexed", "image_id": "photo_001", "indexed": {"global": true, "faces": 2, "vehicles": 1}}
# OCR
with open('document.jpg', 'rb') as f:
resp = requests.post('http://localhost:4603/ocr/predict', files={'image': f})
print(resp.json())
# {"num_texts": 5, "texts": ["Invoice", "Total: $100"], ...}
# Combined Analysis (everything in one call)
with open('scene.jpg', 'rb') as f:
resp = requests.post('http://localhost:4603/analyze', files={'image': f})
result = resp.json()
# {"detections": [...], "faces": [...], "global_embedding": [...], "ocr": {...}}# Detection
curl -X POST http://localhost:4603/detect -F "image=@photo.jpg"
# Face Recognition
curl -X POST http://localhost:4603/faces/recognize -F "image=@face.jpg"
# Text Search (query params, not a JSON body -- `text`, not `query`)
curl -X POST "http://localhost:4603/search/text?text=sunset+beach&top_k=10"
# Ingestion
curl -X POST http://localhost:4603/ingest \
-F "image=@photo.jpg" \
-F "image_id=my_photo_001"{
"detections": [
{
"x1": 0.094, "y1": 0.278, "x2": 0.870, "y2": 0.989,
"confidence": 0.918,
"class_id": 0,
"class_name": "person"
}
],
"image": {"width": 1920, "height": 1080},
"inference_time_ms": 12.5
}Note: Coordinates are normalized (0.0-1.0). Multiply by image width/height for pixels.
{
"num_faces": 2,
"faces": [
{
"box": {"x1": 0.30, "y1": 0.10, "x2": 0.50, "y2": 0.40},
"confidence": 0.98,
"landmarks": [0.35, 0.20, 0.45, 0.20, 0.40, 0.28, 0.36, 0.35, 0.44, 0.35]
}
],
"embeddings": [[...512 floats...]],
"inference_time_ms": 25.3
}The search, ingest and curation response shapes are in the API schema
(http://localhost:4603/docs) and in
docs/design/curation_api_contract.md.
Client (Port 4603)
|
v
+----------+ +-----------------------------+
| api |---->| curation workers (profile) |
+----------+ | detection, VLM, auto-label, |
| | cluster refresh, evaluator |
v +-----------------------------+
+--------------+ +------------+
| triton-server| | opensearch |
| (GPU) | | (k-NN) |
+--------------+ +------------+
Services (compose profile in brackets):
api(4603): FastAPI service handling all requests.triton-server(4600-4602): Triton with TensorRT models.opensearch(4607): vector database and the curation datastore.curation-detection-worker,curation-vlm-worker,curation-auto-label-worker,curation-cluster-refresh[curation];curation-evaluator[evaluator, on demand].segmenter(4611) [segmenter],vlm(4612) [vlm].curation-trainer,curation-mlflow(4609) [training].- Prometheus (4604), Grafana (4605), Loki (4606), OpenSearch Dashboards (4608),
DCGM exporter (4610), Alloy [
monitoring]. Opt in withmake up-monitoringor--with-monitoring;make updoes not start them.make dev-monitoring-upadds them to the dev stack.
More: docs/ARCHITECTURE.md.
| Model | Purpose | Backend |
|---|---|---|
| YOLO11 / YOLO26 | Object detection | TensorRT |
| SCRFD-10G | Face detection + landmarks | TensorRT |
| ArcFace | Face embeddings (512-dim) | TensorRT |
| MobileCLIP | Image/text embeddings (512-dim) | TensorRT |
| PP-OCRv5 | Text detection + recognition | TensorRT |
| PE-Core-L14-336 | Curation embeddings (1024-dim, image and text) | TensorRT image, Triton CPU text |
| Segmenter | Region proposals from a text prompt (optional) | HTTP service |
| VLM | Class labels and region verdicts (optional) | OpenAI-compatible endpoint |
Core models use FP16 with dynamic batching. Export steps: export/README.md.
Minimum:
- NVIDIA GPU with 8GB+ VRAM (Ampere or newer)
- 16GB RAM, 16 CPU cores
- Docker with NVIDIA Container Toolkit
Recommended:
- NVIDIA A100/A6000/RTX 4090 (16GB+)
- 64GB RAM, 48+ CPU cores
- NVMe SSD for image storage
Every setting is in .env (template: env.template). The
"Curation quick-config" block lists the keys the curation tiers need. The
documented reference is
Configuration (basic) and
Configuration (advanced).
- GPU placement.
TRITON_GPU_ID,API_GPU_ID,SEGMENTER_GPU_ID,EVALUATOR_GPU_IDandVLM_GPU_IDpick a GPU per service;./openprocessor gpu planprints them.TRITON_GPU_ID=0,2also runs the PE image encoder on GPU 2; see INSTALLATION.md. - Image source.
OP_SOURCE_ROOT_HOSTis the host folder the API and the workers read images from, mounted read-only at/data/source. Recreateapi,curation-detection-workerandcuration-auto-label-workerafter changing it (./openprocessor start); a restart does not re-read.env. - Workers. The API runs
uvicorn --workers=32(seedocker-compose.yml). - Triton instance counts live in each model's
config.pbtxt. Raise a count when a hot path needs it and the card has room; checknvidia-smiafter loading. - Segmenter VRAM.
SEGMENTER_INSTANCES(default 2) andSEGMENTER_SHARED_WEIGHTS(default 1) trade VRAM for throughput.
Run the offline suite (no Docker, no GPU needed):
.venv/bin/python -m pytest tests/ -qtests/test_full_system.py and tests/validate_visual_results.py hit the
running stack over HTTP, so they read their target ports from the environment
(API_PORT, TRITON_HTTP_PORT, OPENSEARCH_PORT; defaults 4603/4600/4607):
# Full system test (all endpoints) — host venv path
.venv/bin/python tests/test_full_system.py 2>&1 | tee test_results/test_results.txt
# Visual validation (draws bounding boxes on test images)
.venv/bin/python tests/validate_visual_results.py 2>&1 | tee test_results/visual_validation.txtDocker-only path. The production api image installs only
requirements.txt (no pytest, no requirements-test.txt), so a bare
docker compose exec api pytest ... fails with executable file not found. Install the test deps into the running container first (not persisted
across a recreate):
docker compose -p openprocessor exec api pip install -r requirements-test.txt
docker compose -p openprocessor exec api python -m pytest tests/ -q --ignore=tests/liveSee CONTRIBUTING.md for the live write-path harness and the pre-commit hooks.
cd benchmarks
./build.sh
./triton_bench --mode quick # 30-second test
./triton_bench --mode full # Full benchmarkSee benchmarks/README.md and docs/PERFORMANCE.md.
- User and operator docs:
docs-site/docs/, a Docusaurus site (make docs-buildbuilds it). - Maintainer docs: docs/
- docs/ARCHITECTURE.md: components, data model, runtime topology
- docs/design/curation_api_contract.md: route and wire-model reference
- docs/VISION_AND_GOALS.md: scope and standards
- docs/PERFORMANCE.md: measurement log
- docs/opensearch_schema_design.md: index schemas
- docs/OCR.md: OCR model setup
- docs/FACE_RECOGNITION_IMPLEMENTATION.md: face recognition details
- contracts/: generated API schema (OpenAPI and TypeScript types), regenerated by
scripts/codegen/generate_contracts.py - INSTALLATION.md: installer, CLI and source install
- export/README.md: model export
- scripts/README.md: scripts and curation tooling
- benchmarks/README.md: benchmark tool
- SECURITY.md: read before exposing the API beyond a trusted network
- CONTRIBUTING.md: dev setup, tests, commit conventions
- CLAUDE.md: orientation for AI coding agents
- CHANGELOG.md: release history
This project uses:
- NVIDIA Triton Inference Server
- Ultralytics YOLO
- levipereira/ultralytics fork for End2End TensorRT export
- Apple MobileCLIP
- InsightFace ArcFace
- InsightFace SCRFD
- PaddleOCR
- Meta Perception Encoder (PE-Core)
- Meta Segment Anything 3 (optional segmenter)
See ATTRIBUTION.md for complete licensing information.
This project is licensed under the GNU Affero General Public License
v3.0 or later (AGPL-3.0-or-later) — see LICENSE. It vendors an
AGPL-3.0 Ultralytics fork (src/ultralytics_patches/) whose copyleft terms
propagate to the combined work. Third-party components retain their own
licenses (BSD, Apache-2.0, MIT, and others) — see
ATTRIBUTION.md for the full per-component table.
