Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,15 @@ Dockerfile*
docker-compose*.yml
.dockerignore

# Local and sibling trees no root-context image reads (the frontend and the docs site build
# from their own contexts)
artifacts_local/
docs-site/
frontend/
.claude/
.install/
export_*/

# Documentation (not needed in containers)
docs/
*.md
Expand Down
19 changes: 19 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,25 @@ history of this codebase and was never published. This release is `[0.5.0]`.

### Changed

- **BREAKING**: the Cropwright variables are now `OP_CROPWRIGHT_*`: `CROPWRIGHT_IMAGE`,
`CROPWRIGHT_BIND_ADDRESS`, `CROPWRIGHT_PORT`, `CROPWRIGHT_INGEST_MAX_REQUEST_MB` and
`CROPWRIGHT_DATASET_UPLOAD_MAX_MB` became `OP_CROPWRIGHT_IMAGE`, `OP_CROPWRIGHT_BIND_ADDRESS`,
`OP_CROPWRIGHT_PORT`, `OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB` and
`OP_CROPWRIGHT_DATASET_UPLOAD_MAX_MB` (the live-test and docs-screenshot helpers moved the same
way). A `.env` that still sets an old name is refused: `./openprocessor` and
`setup-openprocessor.sh` exit 2 and print the new name, and the cropwright container exits
at start. Rename the keys in `.env` and run `./openprocessor restart`.
- Every Dockerfile base image is pinned by digest (`image:tag@sha256:...`); the bump procedure is
in `docs/releases/RELEASE_CHECKLIST.md`. The evaluator and segmenter images build their Python
environment in a separate stage (no compilers or pip cache in the evaluator runtime; the
segmenter keeps gcc for `torch.compile`), the fake VLM test image runs as `nobody`, and the
api image copies only the `scripts/` it runs (`scripts/curation`, `scripts/datasets`, six shell
libraries) instead of the whole tree. The root `.dockerignore` also excludes `artifacts_local/`,
`docs-site/`, `frontend/`, `.claude/`, `.install/` and `export_*/`.
- `check_docs_vs_code.py` requires an assignment line in `env.template` for every compose
`${VAR}` (a prose mention no longer counts) and gains a `template` check: a non-`OP_` setting
that no compose file, script or code reads is rejected. The unread `LOG_LEVEL` and the unread
`OP_CROPWRIGHT_TAG` documentation are removed.
- Repo cleanup, PR 5b (duplicates): one `now_iso()` (`src/utils/timeutil.py`, UTC, microsecond
precision) replaces 21 private timestamp helpers with three different bodies; the class-history
and edit-history entries used `timespec='seconds'` and now carry microseconds like every other
Expand Down
13 changes: 10 additions & 3 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
# -----------------------------------------------------------------------------
# Stage 1: Build - Install Python dependencies with compilation tools
# -----------------------------------------------------------------------------
FROM python:3.13-slim-trixie AS builder
FROM python:3.13-slim-trixie@sha256:70729b46c69b4f1e97c4822c1af3df53a1476cf5ddc6c087c0c10bc3a5678c2f AS builder

WORKDIR /build

Expand Down Expand Up @@ -62,7 +62,7 @@ RUN rm -rf /root/.local/lib/python3.13/site-packages/pip \
# -----------------------------------------------------------------------------
# Stage 2: Runtime - Minimal image with only runtime dependencies
# -----------------------------------------------------------------------------
FROM python:3.13-slim-trixie
FROM python:3.13-slim-trixie@sha256:70729b46c69b4f1e97c4822c1af3df53a1476cf5ddc6c087c0c10bc3a5678c2f

LABEL org.opencontainers.image.title="OpenProcessor FastAPI Service" \
org.opencontainers.image.description="Visual AI API with object detection, face recognition, embeddings, and OCR" \
Expand Down Expand Up @@ -137,7 +137,14 @@ ENV PATH=/home/appuser/.local/bin:$PATH \

# Copy application code
COPY --chown=appuser:appuser src/ ./src/
COPY --chown=appuser:appuser scripts/ ./scripts/
# Only what runs inside the container: the curation workers and the CLIs operators run with
# `docker compose exec api python -m scripts.curation...`, the public COCO sample fetcher, and the
# shell libraries `openprocessor models install` sources in the container (config.sh and what it
# sources, download.sh). tests/test_dockerfile_hygiene.py checks this set against the imports.
COPY --chown=appuser:appuser scripts/curation/ ./scripts/curation/
COPY --chown=appuser:appuser scripts/datasets/ ./scripts/datasets/
COPY --chown=appuser:appuser scripts/lib/colors.sh scripts/lib/config.sh scripts/lib/config_models.sh \
scripts/lib/download.sh scripts/lib/gpu.sh scripts/lib/opensearch_heap.sh ./scripts/lib/
COPY --chown=appuser:appuser VERSION ./VERSION
COPY --chown=appuser:appuser config_templates/ ./config_templates/

Expand Down
2 changes: 1 addition & 1 deletion Dockerfile.triton
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
# re-exporting every engine — see docs/MIGRATION_TRITON_26.md.
# =============================================================================

FROM nvcr.io/nvidia/tritonserver:26.06-py3
FROM nvcr.io/nvidia/tritonserver:26.06-py3@sha256:58df7489c3f2276f9591d500a012dee03e23d35543ce3c390b4c001e6bf90794

LABEL org.opencontainers.image.title="OpenProcessor Inference Server" \
org.opencontainers.image.description="NVIDIA Triton Inference Server with TensorRT models for visual AI" \
Expand Down
6 changes: 3 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -1127,9 +1127,9 @@ docs-build: ## Build docs-site (npm ci + docusaurus build; broken links and anch
DOCS_PROJECT ?= coco-bench-2k

.PHONY: docs-screenshots
docs-screenshots: ## Capture the Cropwright docs screenshots (CROPWRIGHT_URL=... required; public sample project only)
@test -n "$(CROPWRIGHT_URL)" || { echo "Set CROPWRIGHT_URL to a Cropwright instance that holds only public sample data."; exit 2; }
cd frontend && e2e/.venv/bin/python scripts/capture_docs_screenshots.py --base-url $(CROPWRIGHT_URL) --project $(DOCS_PROJECT)
docs-screenshots: ## Capture the Cropwright docs screenshots (OP_CROPWRIGHT_URL=... required; public sample project only)
@test -n "$(OP_CROPWRIGHT_URL)" || { echo "Set OP_CROPWRIGHT_URL to a Cropwright instance that holds only public sample data."; exit 2; }
cd frontend && e2e/.venv/bin/python scripts/capture_docs_screenshots.py --base-url $(OP_CROPWRIGHT_URL) --project $(DOCS_PROJECT)

.PHONY: docs
docs: info ## Alias for info
Expand Down
2 changes: 1 addition & 1 deletion SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ everything on 127.0.0.1.**

`setup-openprocessor.sh` targets one computer **and** its local network
(homelab, small office). With the `cropwright` tier, the Cropwright web UI
is published on all interfaces (`CROPWRIGHT_BIND_ADDRESS=0.0.0.0`) so other
is published on all interfaces (`OP_CROPWRIGHT_BIND_ADDRESS=0.0.0.0`) so other
computers on the LAN can open it; its nginx proxies to the API over the
Docker network. Every backend port (API, Triton, OpenSearch, ...) stays
bound to `127.0.0.1` unless you pass `--bind` and confirm it. A specific
Expand Down
29 changes: 7 additions & 22 deletions docker-compose.dev.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,8 @@ services:
dockerfile: Dockerfile.triton

api:
build:
# The root Dockerfile build, shared by the curation workers below.
build: &root-build
context: .
dockerfile: Dockerfile
args:
Expand Down Expand Up @@ -62,46 +63,30 @@ services:
build:
context: ./frontend
dockerfile: Dockerfile
image: ${CROPWRIGHT_IMAGE:-openprocessor-cropwright:dev}
image: ${OP_CROPWRIGHT_IMAGE:-openprocessor-cropwright:dev}

curation-detection-worker:
build:
context: .
dockerfile: Dockerfile
args:
OP_BUILD_SHA: "${OP_BUILD_SHA:-}"
build: *root-build
volumes:
- ./src:/app/src:ro
- ./scripts:/app/scripts:ro
- ./examples:/app/examples:ro

curation-vlm-worker:
build:
context: .
dockerfile: Dockerfile
args:
OP_BUILD_SHA: "${OP_BUILD_SHA:-}"
build: *root-build
volumes:
- ./src:/app/src:ro
- ./scripts:/app/scripts:ro

curation-auto-label-worker:
build:
context: .
dockerfile: Dockerfile
args:
OP_BUILD_SHA: "${OP_BUILD_SHA:-}"
build: *root-build
volumes:
- ./src:/app/src:ro
- ./scripts:/app/scripts:ro
- ./examples:/app/examples:ro

curation-cluster-refresh:
build:
context: .
dockerfile: Dockerfile
args:
OP_BUILD_SHA: "${OP_BUILD_SHA:-}"
build: *root-build
volumes:
- ./src:/app/src:ro
- ./scripts:/app/scripts:ro
Expand Down
21 changes: 14 additions & 7 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ x-gateway-wrap: &gateway_wrap |
esac
case "$${OP_UI_PUBLISH_ADDRESS:-}" in
127.*|localhost|::1|'[::1]') ;;
*) echo "gateway-wrap: refusing to start $$tag: OP_GATEWAY_SUBPATHS=true but the UI ports are published on '$$OP_UI_PUBLISH_ADDRESS'. Unset OP_UI_BIND_ADDRESS (or set it to 127.0.0.1) and publish only the gateway (CROPWRIGHT_BIND_ADDRESS)." >&2; exit 1 ;;
*) echo "gateway-wrap: refusing to start $$tag: OP_GATEWAY_SUBPATHS=true but the UI ports are published on '$$OP_UI_PUBLISH_ADDRESS'. Unset OP_UI_BIND_ADDRESS (or set it to 127.0.0.1) and publish only the gateway (OP_CROPWRIGHT_BIND_ADDRESS)." >&2; exit 1 ;;
esac
case "$$tag" in
grafana)
Expand Down Expand Up @@ -330,8 +330,8 @@ services:
# ===================================================================
cropwright:
profiles: ['cropwright']
# CROPWRIGHT_IMAGE (a full reference, e.g. a digest pin) wins over CROPWRIGHT_TAG.
image: ${CROPWRIGHT_IMAGE:-${OP_IMAGE_REPO:-davidamacey}/cropwright:${OP_IMAGE_TAG:-0.5.0}}
# OP_CROPWRIGHT_IMAGE (a full reference, e.g. a digest pin) wins over the OP_IMAGE_TAG default.
image: ${OP_CROPWRIGHT_IMAGE:-${OP_IMAGE_REPO:-davidamacey}/cropwright:${OP_IMAGE_TAG:-0.5.0}}
container_name: ${COMPOSE_PROJECT_NAME:-openprocessor}-cropwright
# Soft scheduling floor for the static-file nginx, not a cap.
mem_reservation: 64m
Expand All @@ -354,17 +354,24 @@ services:
# different origin directly from the browser.
- PUBLIC_TRITON_API_URL=${PUBLIC_TRITON_API_URL:-}
# Deployment-owned nginx upload caps: /ingest (MB) and dataset archives (MB).
- CROPWRIGHT_INGEST_MAX_REQUEST_MB=${CROPWRIGHT_INGEST_MAX_REQUEST_MB:-256}
- CROPWRIGHT_DATASET_UPLOAD_MAX_MB=${CROPWRIGHT_DATASET_UPLOAD_MAX_MB:-2048}
- OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB=${OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB:-256}
- OP_CROPWRIGHT_DATASET_UPLOAD_MAX_MB=${OP_CROPWRIGHT_DATASET_UPLOAD_MAX_MB:-2048}
# Where nginx proxies /OpenProcessor/ (the docs site): the docs service of docker-compose.docs.yml, when run.
- DOCS_UPSTREAM=${DOCS_UPSTREAM:-http://docs:8080}
# Gateway mode: also serve the monitoring UIs under /grafana/, /prometheus/,
# /dashboards/ and /mlflow/ of this one origin (see x-gateway-wrap below).
- OP_GATEWAY_SUBPATHS=${OP_GATEWAY_SUBPATHS:-false}
# Retired names (renamed to OP_CROPWRIGHT_*): passed through only so the entrypoint
# can refuse to start while a .env still sets one.
- CROPWRIGHT_IMAGE=${CROPWRIGHT_IMAGE:-}
- CROPWRIGHT_BIND_ADDRESS=${CROPWRIGHT_BIND_ADDRESS:-}
- CROPWRIGHT_PORT=${CROPWRIGHT_PORT:-}
- CROPWRIGHT_INGEST_MAX_REQUEST_MB=${CROPWRIGHT_INGEST_MAX_REQUEST_MB:-}
- CROPWRIGHT_DATASET_UPLOAD_MAX_MB=${CROPWRIGHT_DATASET_UPLOAD_MAX_MB:-}
ports:
# A UI: CROPWRIGHT_BIND_ADDRESS opens just this one to a LAN (the installer's default for
# A UI: OP_CROPWRIGHT_BIND_ADDRESS opens just this one to a LAN (the installer's default for
# homelabs), OP_UI_BIND_ADDRESS opens it with the other UIs, and the API stays on loopback.
- '${CROPWRIGHT_BIND_ADDRESS:-${OP_UI_BIND_ADDRESS:-${OP_BIND_ADDRESS:-127.0.0.1}}}:${CROPWRIGHT_PORT:-5184}:8080'
- '${OP_CROPWRIGHT_BIND_ADDRESS:-${OP_UI_BIND_ADDRESS:-${OP_BIND_ADDRESS:-127.0.0.1}}}:${OP_CROPWRIGHT_PORT:-5184}:8080'
healthcheck:
# 127.0.0.1, not localhost: localhost resolves to ::1 and nginx listens on IPv4 only.
test: ['CMD', 'wget', '-qO-', 'http://127.0.0.1:8080/']
Expand Down
64 changes: 37 additions & 27 deletions docker/evaluator/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@
# Standalone image (no shared trainer base image exists in this repo):
# installs the same detection-stack pins as the core API image
# (requirements.txt) plus the harness-only extras
# (scripts/curation/bakeoff/requirements.txt).
# (scripts/curation/bakeoff/requirements.txt). Two stages: the compilers that
# build wheel dependencies stay in the builder and never reach the runtime image.
#
# Build:
# docker compose --profile curation build curation-evaluator
Expand All @@ -19,18 +20,39 @@
# One-shot job:
# docker compose --profile curation run --rm curation-evaluator \
# python -m scripts.curation.bakeoff.bakeoff_runner --job /eval_jobs/x.job.json
#
# WAIVER (release checklist): still a single
# `FROM`, not multi-stage. build-essential/cmake/pkg-config stay in the
# final image to compile wheel deps at `pip install` time, which the
# other images' builder-stage split avoids. Deferred: the current scope is the
# compose/mount split, not image-size hardening, and this image is
# on-demand only (`docker compose ... run --rm`), never a long-running
# service. Track the multi-stage conversion as release-checklist debt
# (§3.4) rather than doing it silently here.
# =============================================================================

FROM python:3.13-slim-trixie
# -----------------------------------------------------------------------------
# Stage 1: build -- resolve every wheel into a relocatable --user tree
# -----------------------------------------------------------------------------
FROM python:3.13-slim-trixie@sha256:70729b46c69b4f1e97c4822c1af3df53a1476cf5ddc6c087c0c10bc3a5678c2f AS builder

WORKDIR /build

# hadolint ignore=DL3008
RUN apt-get update && apt-get upgrade -y \
&& apt-get install -y --no-install-recommends \
build-essential \
cmake \
git \
pkg-config \
&& rm -rf /var/lib/apt/lists/*

COPY requirements.txt requirements.txt
COPY scripts/curation/bakeoff/requirements.txt bakeoff-requirements.txt

# hadolint ignore=DL3013
RUN pip install --user --no-cache-dir --upgrade pip \
&& pip install --user --no-cache-dir --no-warn-script-location \
--extra-index-url https://pypi.nvidia.com \
-r requirements.txt \
&& pip install --user --no-cache-dir --no-warn-script-location -r bakeoff-requirements.txt \
&& pip install --user --no-cache-dir --no-warn-script-location mlflow

# -----------------------------------------------------------------------------
# Stage 2: runtime
# -----------------------------------------------------------------------------
FROM python:3.13-slim-trixie@sha256:70729b46c69b4f1e97c4822c1af3df53a1476cf5ddc6c087c0c10bc3a5678c2f

LABEL org.opencontainers.image.title="OpenProcessor Curation Evaluator" \
org.opencontainers.image.description="On-demand detector bake-off harness runner" \
Expand All @@ -47,11 +69,6 @@ LABEL org.opencontainers.image.revision=${OP_BUILD_SHA}
# hadolint ignore=DL3008
RUN apt-get update && apt-get upgrade -y \
&& apt-get install -y --no-install-recommends \
build-essential \
cmake \
curl \
git \
pkg-config \
libgl1 \
libglib2.0-0t64 \
libgomp1 \
Expand All @@ -65,24 +82,17 @@ RUN groupadd -r appuser && \

WORKDIR /app

COPY requirements.txt requirements.txt
COPY scripts/curation/bakeoff/requirements.txt bakeoff-requirements.txt

# hadolint ignore=DL3013
RUN pip install --no-cache-dir --upgrade pip \
&& pip install --no-cache-dir --extra-index-url https://pypi.nvidia.com \
-r requirements.txt \
&& pip install --no-cache-dir -r bakeoff-requirements.txt \
&& pip install --no-cache-dir mlflow
COPY --from=builder --chown=appuser:appuser /root/.local /home/appuser/.local

ENV PYTHONUNBUFFERED=1 \
ENV PATH=/home/appuser/.local/bin:$PATH \
PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
PYTHONPATH=/app \
HF_HOME=/home/appuser/.cache/huggingface \
TORCH_HOME=/home/appuser/.cache/torch

COPY --chown=appuser:appuser src/ ./src/
COPY --chown=appuser:appuser scripts/ ./scripts/
COPY --chown=appuser:appuser scripts/curation/ ./scripts/curation/
# scripts/curation/bakeoff/profile.py reads opt-in baseline profiles under
# the repo's examples/ tree (path given explicitly via a job's
# baselines_path) -- bake it in so a non-checkout deploy doesn't need a
Expand Down
4 changes: 3 additions & 1 deletion docker/hardened/triton/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,10 @@
# Build: docker build -f docker/hardened/triton/Dockerfile -t triton-hardened:26.06 .
# Scan: trivy image --scanners vuln --severity CRITICAL,HIGH triton-hardened:26.06

# Bump TRITON_TAG and TRITON_DIGEST together (docs/releases/RELEASE_CHECKLIST.md, base image digests).
ARG TRITON_TAG=26.06-py3
FROM nvcr.io/nvidia/tritonserver:${TRITON_TAG}
ARG TRITON_DIGEST=sha256:58df7489c3f2276f9591d500a012dee03e23d35543ce3c390b4c001e6bf90794
FROM nvcr.io/nvidia/tritonserver:${TRITON_TAG}@${TRITON_DIGEST}

# hadolint ignore=DL3008,DL3009,DL4006
RUN set -eux; \
Expand Down
Loading
Loading