From fdf6f8dced5490ee4842514abba668b1c83f847b Mon Sep 17 00:00:00 2001 From: davidamacey Date: Wed, 14 Oct 2026 18:09:34 -0400 Subject: [PATCH 1/8] feat(env)!: rename CROPWRIGHT_* to OP_CROPWRIGHT_* and refuse the old names BREAKING: CROPWRIGHT_IMAGE, _BIND_ADDRESS, _PORT, _INGEST_MAX_REQUEST_MB and _DATASET_UPLOAD_MAX_MB are now OP_CROPWRIGHT_*. The deploy CLI and the installer exit 2 naming the new variable when a .env still sets an old name; the cropwright container entrypoint refuses one passed through by compose. The remaining names (live-test URL/project, docs screenshot URL) are renamed the same way. The unread OP_CROPWRIGHT_TAG and the container-name variables only the deleted standalone compose used are dropped from the docs. The docs check now requires an assignment line in env.template for every compose ${VAR} (a prose mention no longer counts), and a new template check rejects a non-OP_ setting nothing reads; the unread LOG_LEVEL is removed. --- Makefile | 6 +- SECURITY.md | 2 +- docker-compose.dev.yml | 2 +- docker-compose.yml | 21 ++-- docs-site/docs/configuration/basic.mdx | 1 - .../configuration/environment-variables.md | 7 +- .../configuration/second-instance.md | 2 +- .../cropwright/developer-guide/testing.md | 2 +- .../cropwright/getting-started/quick-start.md | 6 +- .../docs/cropwright/operations/deployment.md | 10 +- .../cropwright/operations/troubleshooting.md | 6 +- .../cropwright/user-guide/dataset-import.md | 2 +- .../docs/developer-guide/screenshots.mdx | 4 +- docs-site/docs/getting-started/installer.mdx | 2 +- .../docs/getting-started/second-stack.mdx | 2 +- docs-site/docs/operations/monitoring.mdx | 2 +- env.template | 20 ++-- frontend/.env.example | 29 +---- frontend/CLAUDE.md | 10 +- frontend/README.md | 28 +++-- frontend/docker-entrypoint.sh | 23 +++- frontend/e2e/live/conftest.py | 12 +- frontend/e2e/live/test_data_agreement.py | 2 +- frontend/e2e/live/test_route_sweep.py | 2 +- frontend/e2e/stubbed/test_ingest.py | 2 +- frontend/nginx.conf | 4 +- frontend/scripts/capture_docs_screenshots.py | 10 +- frontend/scripts/run-e2e.mjs | 2 +- openprocessor | 7 ++ release-manifest.txt | 1 + scripts/docs/check_docs_vs_code.py | 60 ++++++++-- scripts/lib/image_keys.sh | 2 +- scripts/lib/retired_env.sh | 34 ++++++ setup-openprocessor.sh | 62 +++++------ .../test_ports_names_and_project_name.py | 2 +- tests/installer/shims/docker | 6 +- tests/installer/test_lifecycle.py | 2 +- .../installer/test_retired_cropwright_env.py | 104 ++++++++++++++++++ .../test_setup_bind_and_cropwright.py | 20 ++-- tests/installer/test_static.py | 1 + tests/installer/test_tty_prompts.py | 2 +- tests/test_compose_gateway.py | 2 +- tests/test_compose_ui_bind.py | 4 +- tests/test_docs_vs_code.py | 50 ++++++++- tests/test_env_surface.py | 7 ++ 45 files changed, 414 insertions(+), 173 deletions(-) create mode 100755 scripts/lib/retired_env.sh create mode 100644 tests/installer/test_retired_cropwright_env.py diff --git a/Makefile b/Makefile index d265657c6..769de140c 100644 --- a/Makefile +++ b/Makefile @@ -1127,9 +1127,9 @@ docs-build: ## Build docs-site (npm ci + docusaurus build; broken links and anch DOCS_PROJECT ?= coco-bench-2k .PHONY: docs-screenshots -docs-screenshots: ## Capture the Cropwright docs screenshots (CROPWRIGHT_URL=... required; public sample project only) - @test -n "$(CROPWRIGHT_URL)" || { echo "Set CROPWRIGHT_URL to a Cropwright instance that holds only public sample data."; exit 2; } - cd frontend && e2e/.venv/bin/python scripts/capture_docs_screenshots.py --base-url $(CROPWRIGHT_URL) --project $(DOCS_PROJECT) +docs-screenshots: ## Capture the Cropwright docs screenshots (OP_CROPWRIGHT_URL=... required; public sample project only) + @test -n "$(OP_CROPWRIGHT_URL)" || { echo "Set OP_CROPWRIGHT_URL to a Cropwright instance that holds only public sample data."; exit 2; } + cd frontend && e2e/.venv/bin/python scripts/capture_docs_screenshots.py --base-url $(OP_CROPWRIGHT_URL) --project $(DOCS_PROJECT) .PHONY: docs docs: info ## Alias for info diff --git a/SECURITY.md b/SECURITY.md index e27aa4de8..84556bdeb 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -43,7 +43,7 @@ everything on 127.0.0.1.** `setup-openprocessor.sh` targets one computer **and** its local network (homelab, small office). With the `cropwright` tier, the Cropwright web UI -is published on all interfaces (`CROPWRIGHT_BIND_ADDRESS=0.0.0.0`) so other +is published on all interfaces (`OP_CROPWRIGHT_BIND_ADDRESS=0.0.0.0`) so other computers on the LAN can open it; its nginx proxies to the API over the Docker network. Every backend port (API, Triton, OpenSearch, ...) stays bound to `127.0.0.1` unless you pass `--bind` and confirm it. A specific diff --git a/docker-compose.dev.yml b/docker-compose.dev.yml index 317d94988..cc46c40be 100644 --- a/docker-compose.dev.yml +++ b/docker-compose.dev.yml @@ -62,7 +62,7 @@ services: build: context: ./frontend dockerfile: Dockerfile - image: ${CROPWRIGHT_IMAGE:-openprocessor-cropwright:dev} + image: ${OP_CROPWRIGHT_IMAGE:-openprocessor-cropwright:dev} curation-detection-worker: build: diff --git a/docker-compose.yml b/docker-compose.yml index bcc5f3d3c..5bd542bc0 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -41,7 +41,7 @@ x-gateway-wrap: &gateway_wrap | esac case "$${OP_UI_PUBLISH_ADDRESS:-}" in 127.*|localhost|::1|'[::1]') ;; - *) echo "gateway-wrap: refusing to start $$tag: OP_GATEWAY_SUBPATHS=true but the UI ports are published on '$$OP_UI_PUBLISH_ADDRESS'. Unset OP_UI_BIND_ADDRESS (or set it to 127.0.0.1) and publish only the gateway (CROPWRIGHT_BIND_ADDRESS)." >&2; exit 1 ;; + *) echo "gateway-wrap: refusing to start $$tag: OP_GATEWAY_SUBPATHS=true but the UI ports are published on '$$OP_UI_PUBLISH_ADDRESS'. Unset OP_UI_BIND_ADDRESS (or set it to 127.0.0.1) and publish only the gateway (OP_CROPWRIGHT_BIND_ADDRESS)." >&2; exit 1 ;; esac case "$$tag" in grafana) @@ -330,8 +330,8 @@ services: # =================================================================== cropwright: profiles: ['cropwright'] - # CROPWRIGHT_IMAGE (a full reference, e.g. a digest pin) wins over CROPWRIGHT_TAG. - image: ${CROPWRIGHT_IMAGE:-${OP_IMAGE_REPO:-davidamacey}/cropwright:${OP_IMAGE_TAG:-0.5.0}} + # OP_CROPWRIGHT_IMAGE (a full reference, e.g. a digest pin) wins over the OP_IMAGE_TAG default. + image: ${OP_CROPWRIGHT_IMAGE:-${OP_IMAGE_REPO:-davidamacey}/cropwright:${OP_IMAGE_TAG:-0.5.0}} container_name: ${COMPOSE_PROJECT_NAME:-openprocessor}-cropwright # Soft scheduling floor for the static-file nginx, not a cap. mem_reservation: 64m @@ -354,17 +354,24 @@ services: # different origin directly from the browser. - PUBLIC_TRITON_API_URL=${PUBLIC_TRITON_API_URL:-} # Deployment-owned nginx upload caps: /ingest (MB) and dataset archives (MB). - - CROPWRIGHT_INGEST_MAX_REQUEST_MB=${CROPWRIGHT_INGEST_MAX_REQUEST_MB:-256} - - CROPWRIGHT_DATASET_UPLOAD_MAX_MB=${CROPWRIGHT_DATASET_UPLOAD_MAX_MB:-2048} + - OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB=${OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB:-256} + - OP_CROPWRIGHT_DATASET_UPLOAD_MAX_MB=${OP_CROPWRIGHT_DATASET_UPLOAD_MAX_MB:-2048} # Where nginx proxies /OpenProcessor/ (the docs site): the docs service of docker-compose.docs.yml, when run. - DOCS_UPSTREAM=${DOCS_UPSTREAM:-http://docs:8080} # Gateway mode: also serve the monitoring UIs under /grafana/, /prometheus/, # /dashboards/ and /mlflow/ of this one origin (see x-gateway-wrap below). - OP_GATEWAY_SUBPATHS=${OP_GATEWAY_SUBPATHS:-false} + # Retired names (renamed to OP_CROPWRIGHT_*): passed through only so the entrypoint + # can refuse to start while a .env still sets one. + - CROPWRIGHT_IMAGE=${CROPWRIGHT_IMAGE:-} + - CROPWRIGHT_BIND_ADDRESS=${CROPWRIGHT_BIND_ADDRESS:-} + - CROPWRIGHT_PORT=${CROPWRIGHT_PORT:-} + - CROPWRIGHT_INGEST_MAX_REQUEST_MB=${CROPWRIGHT_INGEST_MAX_REQUEST_MB:-} + - CROPWRIGHT_DATASET_UPLOAD_MAX_MB=${CROPWRIGHT_DATASET_UPLOAD_MAX_MB:-} ports: - # A UI: CROPWRIGHT_BIND_ADDRESS opens just this one to a LAN (the installer's default for + # A UI: OP_CROPWRIGHT_BIND_ADDRESS opens just this one to a LAN (the installer's default for # homelabs), OP_UI_BIND_ADDRESS opens it with the other UIs, and the API stays on loopback. - - '${CROPWRIGHT_BIND_ADDRESS:-${OP_UI_BIND_ADDRESS:-${OP_BIND_ADDRESS:-127.0.0.1}}}:${CROPWRIGHT_PORT:-5184}:8080' + - '${OP_CROPWRIGHT_BIND_ADDRESS:-${OP_UI_BIND_ADDRESS:-${OP_BIND_ADDRESS:-127.0.0.1}}}:${OP_CROPWRIGHT_PORT:-5184}:8080' healthcheck: # 127.0.0.1, not localhost: localhost resolves to ::1 and nginx listens on IPv4 only. test: ['CMD', 'wget', '-qO-', 'http://127.0.0.1:8080/'] diff --git a/docs-site/docs/configuration/basic.mdx b/docs-site/docs/configuration/basic.mdx index 32a5a0e4f..2589420b1 100644 --- a/docs-site/docs/configuration/basic.mdx +++ b/docs-site/docs/configuration/basic.mdx @@ -122,7 +122,6 @@ See [Running a second, isolated stack](../getting-started/second-stack.mdx). | Variable | Meaning | |---|---| | `DEBUG` | `1` for verbose logging | -| `LOG_LEVEL` | Listed in `env.template` for uvicorn; the shipped compose command does not read it | ## Core model selection diff --git a/docs-site/docs/cropwright/configuration/environment-variables.md b/docs-site/docs/cropwright/configuration/environment-variables.md index 444bae394..105b2d98e 100644 --- a/docs-site/docs/cropwright/configuration/environment-variables.md +++ b/docs-site/docs/cropwright/configuration/environment-variables.md @@ -36,11 +36,10 @@ Compose setting: the published image and the Compose stack never read them. | Variable | Purpose | Default | | --- | --- | --- | | `API_UPSTREAM` | Where nginx proxies `PUBLIC_API_PREFIX/*` — the OpenProcessor API container, by name over the shared docker network | `http://op-api:8000` | -| `CROPWRIGHT_PORT` | Host port this instance publishes nginx on | `5184` | +| `OP_CROPWRIGHT_PORT` | Host port this instance publishes nginx on | `5184` | | `OP_UI_BIND_ADDRESS` | Host address the Cropwright port is published on (the optional monitoring UIs follow it). Defaults to `OP_BIND_ADDRESS`, then loopback. Set `0.0.0.0` to allow your local network; the API has no authentication, so keep it on a trusted network. | `127.0.0.1` | -| `CROPWRIGHT_TAG` | Which published image tag to pull; pin a version for a reproducible deploy | `latest` | -| `CROPWRIGHT_DATASET_UPLOAD_MAX_MB` | Upload cap for a dataset archive on [dataset import](../user-guide/dataset-import.md) (nginx `client_max_body_size` for the dataset-upload route, in MB), with a one-hour read timeout. Substituted into both nginx.conf and the served client config; the client refuses a larger file before sending it. | `2048` | -| `CROPWRIGHT_INGEST_MAX_REQUEST_MB` | Deployment-owned upload cap for `/ingest` (nginx `client_max_body_size`, in MB); substituted into both nginx.conf and the served client config so the two can't drift | `256` | +| `OP_CROPWRIGHT_DATASET_UPLOAD_MAX_MB` | Upload cap for a dataset archive on [dataset import](../user-guide/dataset-import.md) (nginx `client_max_body_size` for the dataset-upload route, in MB), with a one-hour read timeout. Substituted into both nginx.conf and the served client config; the client refuses a larger file before sending it. | `2048` | +| `OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB` | Deployment-owned upload cap for `/ingest` (nginx `client_max_body_size`, in MB); substituted into both nginx.conf and the served client config so the two can't drift | `256` | Both upload caps must be positive integers; the container refuses to start otherwise. Values are substituted when the container starts, so one image diff --git a/docs-site/docs/cropwright/configuration/second-instance.md b/docs-site/docs/cropwright/configuration/second-instance.md index 292231937..6fa0ef69c 100644 --- a/docs-site/docs/cropwright/configuration/second-instance.md +++ b/docs-site/docs/cropwright/configuration/second-instance.md @@ -9,7 +9,7 @@ Give it its own compose project name, container name and host port so it never collides with a running instance: ```bash -CROPWRIGHT_PORT=5190 docker compose -p second --profile cropwright up -d +OP_CROPWRIGHT_PORT=5190 docker compose -p second --profile cropwright up -d ``` This is the same image and compose file, pointed at a different backend diff --git a/docs-site/docs/cropwright/developer-guide/testing.md b/docs-site/docs/cropwright/developer-guide/testing.md index bc9834f4f..648b4cde7 100644 --- a/docs-site/docs/cropwright/developer-guide/testing.md +++ b/docs-site/docs/cropwright/developer-guide/testing.md @@ -27,7 +27,7 @@ silent `200 {}`). Never touches a live backend. `e2e/live/` — drives the real deployed build against a **real** OpenProcessor backend, structurally read-only (every non-GET/HEAD request is aborted and any attempt fails the test). Skipped entirely unless -`CROPWRIGHT_LIVE_URL` is set — never runs in CI. Also saves full-page +`OP_CROPWRIGHT_LIVE_URL` is set — never runs in CI. Also saves full-page screenshots per route at two viewports; **a human must actually look at them** after a run — the assertions catch what's mechanically checkable, not a layout regression. diff --git a/docs-site/docs/cropwright/getting-started/quick-start.md b/docs-site/docs/cropwright/getting-started/quick-start.md index d130ff4c3..6f75d0ade 100644 --- a/docs-site/docs/cropwright/getting-started/quick-start.md +++ b/docs-site/docs/cropwright/getting-started/quick-start.md @@ -23,7 +23,7 @@ the stack: | -------------------- | --------------------------------------------------------------------------------- | -------------------------------- | | `API_UPSTREAM` | Where nginx proxies API calls, by service name over the compose network | `http://op-api:8000` | | `PUBLIC_API_PREFIX` | Must equal the backend's own `OP_API_PREFIX` | `/curation` | -| `CROPWRIGHT_PORT` | Host port to publish | `5184` | +| `OP_CROPWRIGHT_PORT` | Host port to publish | `5184` | | `OP_UI_BIND_ADDRESS` | Address the Cropwright port (and the monitoring UIs) is published on | `OP_BIND_ADDRESS`, then loopback | See [Environment variables](../configuration/environment-variables.md) for @@ -41,9 +41,9 @@ From a source checkout, `docker-compose.dev.yml` builds the image from `./fronte docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile cropwright up -d --build cropwright ``` -Open `http://localhost:5184` (or whatever `CROPWRIGHT_PORT` you set). The +Open `http://localhost:5184` (or whatever `OP_CROPWRIGHT_PORT` you set). The image runs nginx as a non-root user (uid 101) listening on container port -8080; Compose maps `CROPWRIGHT_PORT` to it. The same origin also serves the backend's API +8080; Compose maps `OP_CROPWRIGHT_PORT` to it. The same origin also serves the backend's API reference (`/docs`, `/redoc`, `/openapi.json`) and, when the docs service is running (`docker-compose.docs.yml`), this documentation; the top-bar **Resources** menu links to them. diff --git a/docs-site/docs/cropwright/operations/deployment.md b/docs-site/docs/cropwright/operations/deployment.md index cd27df3be..3e3903b37 100644 --- a/docs-site/docs/cropwright/operations/deployment.md +++ b/docs-site/docs/cropwright/operations/deployment.md @@ -9,7 +9,7 @@ Cropwright ships as a single Docker image: SvelteKit's static adapter build served by nginx. - **Non-root**: the image runs nginx as uid 101, listening on container - port `8080`. Compose maps `CROPWRIGHT_PORT` (default `5184`) to it. + port `8080`. Compose maps `OP_CROPWRIGHT_PORT` (default `5184`) to it. - **Docs on the same origin**: nginx proxies the API's Swagger UI (`/docs`), ReDoc (`/redoc`) and `/openapi.json`, and serves the one documentation site at `/OpenProcessor/` when the `docs` service of `docker-compose.docs.yml` is running. Point `DOCS_UPSTREAM` @@ -38,10 +38,10 @@ differences worth knowing when you put another proxy in front: - The general API location has a 120 s read timeout. - Ingest routes (`.../projects//ingest/...`) get a 600 s read timeout - and the `CROPWRIGHT_INGEST_MAX_REQUEST_MB` body cap, because a batch with + and the `OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB` body cap, because a batch with detection and embedding can run long. - The dataset-archive upload route (`.../projects//datasets/uploads`) - gets a one-hour read timeout and the `CROPWRIGHT_DATASET_UPLOAD_MAX_MB` + gets a one-hour read timeout and the `OP_CROPWRIGHT_DATASET_UPLOAD_MAX_MB` body cap. An authenticating proxy in front of Cropwright must pass these through @@ -49,7 +49,7 @@ unchanged, or large uploads will fail with a 413 or a gateway timeout. ## Network exposure -The compose file publishes on `${OP_UI_BIND_ADDRESS:-${OP_BIND_ADDRESS:-127.0.0.1}}:${CROPWRIGHT_PORT:-5184}`, +The compose file publishes on `${OP_UI_BIND_ADDRESS:-${OP_BIND_ADDRESS:-127.0.0.1}}:${OP_CROPWRIGHT_PORT:-5184}`, so by default Cropwright answers on this machine only. Set `OP_UI_BIND_ADDRESS=0.0.0.0` (or a LAN address) to let your local network open it; that also opens the optional monitoring UIs, and the API port stays on `OP_BIND_ADDRESS`. @@ -66,6 +66,6 @@ the container). Released images are published to Docker Hub as `davidamacey/cropwright` (`X.Y.Z`, `X.Y` and `latest`), multi-arch for `linux/amd64` and -`linux/arm64`. `docker-compose.yml` only pulls; set `CROPWRIGHT_TAG` (or a full `CROPWRIGHT_IMAGE` +`linux/arm64`. `docker-compose.yml` only pulls; set `OP_IMAGE_TAG` (or a full `OP_CROPWRIGHT_IMAGE` reference) to pin a version. To build from the checkout instead, add the dev overlay: `docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile cropwright up -d --build cropwright`. diff --git a/docs-site/docs/cropwright/operations/troubleshooting.md b/docs-site/docs/cropwright/operations/troubleshooting.md index 5a81e7fb2..2785dd30f 100644 --- a/docs-site/docs/cropwright/operations/troubleshooting.md +++ b/docs-site/docs/cropwright/operations/troubleshooting.md @@ -11,7 +11,7 @@ title: Troubleshooting exactly. - Confirm `API_UPSTREAM` resolves — the `cropwright` service must be in the same compose project as the API (`docker compose -p ps`). -- `curl http://localhost:/curation/health` (swap in your +- `curl http://localhost:/curation/health` (swap in your prefix) from the host to bypass the browser entirely. ## A route/section I expect isn't showing up @@ -30,7 +30,7 @@ and on a current OpenProcessor release. ## Upload fails with a 413 Either the server-path batch's own item cap, or nginx's -`client_max_body_size` (`CROPWRIGHT_INGEST_MAX_REQUEST_MB`). The upload +`client_max_body_size` (`OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB`). The upload run controller retries once at half the chunk size on the backend's own JSON 413; an nginx HTML 413 stops the run outright rather than retrying — raise the cap and re-run. @@ -78,7 +78,7 @@ recorded first. Use **Settings → Models** to acknowledge. See ## A dataset archive upload fails A file larger than the smaller of the backend's limit and -`CROPWRIGHT_DATASET_UPLOAD_MAX_MB` is refused before it is sent. Raise the +`OP_CROPWRIGHT_DATASET_UPLOAD_MAX_MB` is refused before it is sent. Raise the env var and restart the container, or import from a folder path the backend can read. diff --git a/docs-site/docs/cropwright/user-guide/dataset-import.md b/docs-site/docs/cropwright/user-guide/dataset-import.md index c60e98809..86471594e 100644 --- a/docs-site/docs/cropwright/user-guide/dataset-import.md +++ b/docs-site/docs/cropwright/user-guide/dataset-import.md @@ -26,7 +26,7 @@ serves. 1. **Source.** Type a folder path the backend can read (the allowed roots are listed under the field, and the preview tells you if the path isn't usable), or upload an archive. An archive larger than the smaller of the - backend's own limit and the deployment's `CROPWRIGHT_DATASET_UPLOAD_MAX_MB` + backend's own limit and the deployment's `OP_CROPWRIGHT_DATASET_UPLOAD_MAX_MB` is refused before it is sent. Pick a format, or leave **Detect automatically**. 2. **Preview.** A moment after any change, the backend previews exactly what diff --git a/docs-site/docs/developer-guide/screenshots.mdx b/docs-site/docs/developer-guide/screenshots.mdx index 715afb538..7e0a0d6e0 100644 --- a/docs-site/docs/developer-guide/screenshots.mdx +++ b/docs-site/docs/developer-guide/screenshots.mdx @@ -58,8 +58,8 @@ names, file paths and class names that are not sample data. # Once: creates frontend/e2e/.venv with Playwright. cd frontend && npm run test:e2e -# From the repository root. CROPWRIGHT_URL has no default. -make docs-screenshots CROPWRIGHT_URL=http://localhost: +# From the repository root. OP_CROPWRIGHT_URL has no default. +make docs-screenshots OP_CROPWRIGHT_URL=http://localhost: ``` `make docs-screenshots` runs the script with `--project $(DOCS_PROJECT)`, diff --git a/docs-site/docs/getting-started/installer.mdx b/docs-site/docs/getting-started/installer.mdx index c30355d97..d042e6257 100644 --- a/docs-site/docs/getting-started/installer.mdx +++ b/docs-site/docs/getting-started/installer.mdx @@ -108,7 +108,7 @@ Cropwright is built from `frontend/` in this repository and released with the ba Its image (`davidamacey/cropwright`) is pinned in `images.lock` like every other image, and the `cropwright` tier is a profile of the main compose project, so the installer needs nothing from a second repository. Its port and bind address go in the install -`.env` as `CROPWRIGHT_PORT` and `CROPWRIGHT_BIND_ADDRESS`. +`.env` as `OP_CROPWRIGHT_PORT` and `OP_CROPWRIGHT_BIND_ADDRESS`. The Cropwright documentation is a section of this docs site, and its UI links into it. The `/OpenProcessor/` route of Cropwright's nginx proxies to the docs container diff --git a/docs-site/docs/getting-started/second-stack.mdx b/docs-site/docs/getting-started/second-stack.mdx index 7f8043859..bc666089e 100644 --- a/docs-site/docs/getting-started/second-stack.mdx +++ b/docs-site/docs/getting-started/second-stack.mdx @@ -52,7 +52,7 @@ MLFLOW_PORT=4709 DCGM_PORT=4710 SEGMENTER_PORT=4711 VLM_PORT=4712 -CROPWRIGHT_PORT=5284 +OP_CROPWRIGHT_PORT=5284 ``` `make` reads `.env` too (`-include .env`), so `make up`, `make test-detect` diff --git a/docs-site/docs/operations/monitoring.mdx b/docs-site/docs/operations/monitoring.mdx index 406877e66..5ba9eb249 100644 --- a/docs-site/docs/operations/monitoring.mdx +++ b/docs-site/docs/operations/monitoring.mdx @@ -265,7 +265,7 @@ nginx proxy the UIs under sub-paths. The UIs stay on `127.0.0.1`. COMPOSE_PROFILES=cropwright,monitoring,training OP_GATEWAY_SUBPATHS=true GF_SECURITY_ADMIN_PASSWORD= -CROPWRIGHT_BIND_ADDRESS=0.0.0.0 # the gateway; leave OP_UI_BIND_ADDRESS unset +OP_CROPWRIGHT_BIND_ADDRESS=0.0.0.0 # the gateway; leave OP_UI_BIND_ADDRESS unset ``` One switch does everything: compose starts Grafana, Prometheus and OpenSearch diff --git a/env.template b/env.template index 9064b4ccb..f990c82ba 100644 --- a/env.template +++ b/env.template @@ -134,15 +134,15 @@ OPENSEARCH_HEAP=2g # GF_SECURITY_ADMIN_PASSWORD=admin # # Cropwright (the labeling web UI, compose profile `cropwright`) has its own -# publish address and host port. CROPWRIGHT_BIND_ADDRESS falls back to +# publish address and host port. OP_CROPWRIGHT_BIND_ADDRESS falls back to # OP_UI_BIND_ADDRESS, then OP_BIND_ADDRESS. Cropwright has no login: bind it to # a LAN address only on a network you trust. -# CROPWRIGHT_BIND_ADDRESS= -# CROPWRIGHT_PORT=5184 +# OP_CROPWRIGHT_BIND_ADDRESS= +# OP_CROPWRIGHT_PORT=5184 # # Gateway mode (OP_GATEWAY_SUBPATHS): Cropwright's nginx also serves the four # monitoring UIs under /grafana/, /prometheus/, /dashboards/ and /mlflow/ of its -# one published port (CROPWRIGHT_PORT), so only that port needs to be reachable +# one published port (OP_CROPWRIGHT_PORT), so only that port needs to be reachable # and the UIs stay published on 127.0.0.1 only. Requires the `cropwright` # profile. One switch for the whole stack: compose reads it for Cropwright # (which adds the routes), Grafana, Prometheus, OpenSearch Dashboards and MLflow @@ -234,10 +234,6 @@ VLM_PORT=4612 # Enable debug mode (verbose logging) # DEBUG=1 -# Log level for uvicorn -# Options: critical, error, warning, info, debug, trace -# LOG_LEVEL=info - # ============================================================================= # Curation quick-config (installer plan §6.3/§8) -- START HERE # ============================================================================= @@ -1068,9 +1064,9 @@ VLM_PORT=4612 # OPENSEARCH_IMAGE=opensearchproject/opensearch:3.6.0 # OPENSEARCH_DASHBOARDS_IMAGE=opensearchproject/opensearch-dashboards:3.6.0 -# Cropwright (profile: cropwright). CROPWRIGHT_IMAGE is a full image reference +# Cropwright (profile: cropwright). OP_CROPWRIGHT_IMAGE is a full image reference # (for example a digest pin) and wins over OP_IMAGE_REPO/OP_IMAGE_TAG. -# CROPWRIGHT_IMAGE= +# OP_CROPWRIGHT_IMAGE= # Must equal the API's OP_API_PREFIX: the API builds some URLs (region thumbnails) itself. # PUBLIC_API_PREFIX=/curation # Where Cropwright's nginx proxies the API and the docs site. @@ -1080,8 +1076,8 @@ VLM_PORT=4612 # directly from the browser. # PUBLIC_TRITON_API_URL= # Upload caps in MB: one /ingest request, and a dataset archive. -# CROPWRIGHT_INGEST_MAX_REQUEST_MB=256 -# CROPWRIGHT_DATASET_UPLOAD_MAX_MB=2048 +# OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB=256 +# OP_CROPWRIGHT_DATASET_UPLOAD_MAX_MB=2048 # The optional docs-site overlay (docker-compose.docs.yml). # OP_DOCS_IMAGE=openprocessor-docs:local diff --git a/frontend/.env.example b/frontend/.env.example index adc2d81fc..04125175f 100644 --- a/frontend/.env.example +++ b/frontend/.env.example @@ -37,35 +37,18 @@ # OP_DOCKER_NETWORK=openprocessor_triton_net # Host port this instance publishes nginx on. Defaults to 5184. -# CROPWRIGHT_PORT=5184 +# OP_CROPWRIGHT_PORT=5184 # Host address the port is published on. Defaults to 0.0.0.0, so this # machine and other machines on the local network can both reach it. Set # 127.0.0.1 to allow this machine only. The curation API has no # authentication, so keep it on a trusted network and never expose it to # the internet. -# CROPWRIGHT_BIND_ADDRESS=0.0.0.0 - -# Which published davidamacey/cropwright tag to pull (multi-arch: -# linux/amd64 + linux/arm64). Defaults to "latest". Pin a version for a -# reproducible deploy, e.g. CROPWRIGHT_TAG=0.2.0. -# CROPWRIGHT_TAG=latest +# OP_CROPWRIGHT_BIND_ADDRESS=0.0.0.0 # A full image reference, e.g. davidamacey/cropwright@sha256:. Wins -# over CROPWRIGHT_TAG when set (the OpenProcessor installer pins it). -# CROPWRIGHT_IMAGE= - -# Container name for `docker compose up`. Defaults to "cropwright". -# Give a second, independent instance its own value (and its own -# `docker compose -p `) so it doesn't collide with a running -# instance — see docker-compose.yml's header comment for a worked -# example. -# CROPWRIGHT_CONTAINER_NAME=cropwright - -# Container name of the bundled documentation service (no published port; -# it is reached at /cropwright/ on the app origin). Give a second instance -# its own value too. -# CROPWRIGHT_DOCS_CONTAINER_NAME=cropwright-docs +# over the default image tag when set (the OpenProcessor installer pins it). +# OP_CROPWRIGHT_IMAGE= # Where nginx proxies /cropwright/ (the docs service) to. Same shape as # API_UPSTREAM. Default matches docker-compose.yml's `docs` service. @@ -75,10 +58,10 @@ # in MB). Defaults to 256. Raise it for a deployment expecting very # large batch uploads; docker-entrypoint.sh substitutes this into both # nginx.conf and the served client config so the two never drift apart. -# CROPWRIGHT_INGEST_MAX_REQUEST_MB=256 +# OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB=256 # Upload cap for a dataset archive on /datasets/import (nginx # client_max_body_size for the dataset-upload route, in MB). Defaults to # 2048, matching the backend's own default. The client refuses a larger # file before sending it. -# CROPWRIGHT_DATASET_UPLOAD_MAX_MB=2048 +# OP_CROPWRIGHT_DATASET_UPLOAD_MAX_MB=2048 diff --git a/frontend/CLAUDE.md b/frontend/CLAUDE.md index 3a0f2b0ad..fa6619558 100644 --- a/frontend/CLAUDE.md +++ b/frontend/CLAUDE.md @@ -182,7 +182,7 @@ record. to invent independently, and the operator reads the served "Worklog drained as of HH:MM:SS" note. `RegionDrainPanel.svelte` shows the verdict and how long it's held (`stable for Ns`). -- **Deployment-owned upload cap.** `CROPWRIGHT_INGEST_MAX_REQUEST_MB` +- **Deployment-owned upload cap.** `OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB` (default 256, `docker-entrypoint.sh`) is substituted into both `nginx.conf`'s `client_max_body_size` and `window.__CROPWRIGHT_INGEST_MAX_REQUEST_MB__` (`src/app.html`), so @@ -405,7 +405,7 @@ Built against the frozen W10 spec before the backend ships it. - **Wizard** (`ImportWizard`, `importWizardController.svelte.ts`): a server path (the served `ingest/config` `batch.source_roots` listed) or an archive upload (`POST /datasets/uploads`, refused client-side above - `min(served upload.max_bytes, CROPWRIGHT_DATASET_UPLOAD_MAX_MB)`), a + `min(served upload.max_bytes, OP_CROPWRIGHT_DATASET_UPLOAD_MAX_MB)`), a 400 ms-debounced `POST /datasets/preview` of exactly what the operator chose (no `project` in any body), the served splits/totals/estimate/ OP-export facts/issues (`DatasetIssueList`, grouped by served severity @@ -459,7 +459,7 @@ false`; the served post-write crop is adopted by `CropDetailModal` and embed options (sent only once touched), and `ReprocessCounts` shows each scope's served `detail`; see "Detector, ingest policy and embedding state". - **Proxy.** `nginx.conf` has a `…/projects/*/datasets/uploads` location - with `client_max_body_size` from `CROPWRIGHT_DATASET_UPLOAD_MAX_MB` + with `client_max_body_size` from `OP_CROPWRIGHT_DATASET_UPLOAD_MAX_MB` (default 2048) and a one-hour timeout. - **Contract.** Types in `src/lib/types_import.ts`, pinned key-for-key to the vendored OpenAPI (OpenProcessor f582aa05) by @@ -2566,7 +2566,7 @@ deployed** build against a **real** OpenProcessor backend instead of the stubbed in-browser fixtures — the one class of bug `e2e/stubbed/` structurally cannot see: the frontend and the live backend's own wire shape/data actually disagreeing. Point it at the deployment with -`CROPWRIGHT_LIVE_URL=http://localhost:5184 npm run test:live` (or any +`OP_CROPWRIGHT_LIVE_URL=http://localhost:5184 npm run test:live` (or any other reachable Cropwright nginx origin) — unset, `_require_live_url` (`e2e/live/conftest.py`, session-scoped + autouse) skips every test in the tier before a browser is ever launched, so this never runs in CI or @@ -2736,7 +2736,7 @@ Measured 2026-10-03: sourceBadge 72.7%, uploadPlanner 87.5%, classPicker The production build runs in a non-root `nginxinc/nginx-unprivileged:1.31.2-alpine3.23` container (uid 101, nginx on -container port 8080), host port 5184 (`CROPWRIGHT_PORT`, mapped to +container port 8080), host port 5184 (`OP_CROPWRIGHT_PORT`, mapped to 8080). `docker-compose.yml` is pull-only (`docker compose pull && docker compose up -d`, no repo checkout needed — see README's "Quick start"); building from source uses the `docker-compose.build.yml` overlay diff --git a/frontend/README.md b/frontend/README.md index f62d76d64..b6797afe1 100644 --- a/frontend/README.md +++ b/frontend/README.md @@ -77,8 +77,8 @@ Edit `.env` and set, at minimum: - `OP_DOCKER_NETWORK` — the OpenProcessor backend's docker network name (`docker network ls`; defaults to OpenProcessor's own default network name). -- `CROPWRIGHT_PORT` — host port to publish (default `5184`). -- `CROPWRIGHT_BIND_ADDRESS` — host address the port is published on +- `OP_CROPWRIGHT_PORT` — host port to publish (default `5184`). +- `OP_CROPWRIGHT_BIND_ADDRESS` — host address the port is published on (default `0.0.0.0`: this machine and the local network). Set `127.0.0.1` for this machine only. The API has no authentication, so keep it on a trusted network. @@ -89,12 +89,12 @@ Then: docker compose -f docker-compose.yml -f docker-compose.build.yml up -d --build ``` -Open `http://localhost:5184` (or whatever `CROPWRIGHT_PORT` you set). +Open `http://localhost:5184` (or whatever `OP_CROPWRIGHT_PORT` you set). The app origin also serves the API's own Swagger UI at `/docs` (ReDoc at `/redoc`); the top bar's **Resources** menu links to them, to the documentation when it is running, and to any dashboards the backend serves. The image runs nginx as a non-root user (uid 101) listening on port 8080 -inside the container; compose maps `CROPWRIGHT_PORT` to it. +inside the container; compose maps `OP_CROPWRIGHT_PORT` to it. **Verify it's connected:** @@ -105,12 +105,11 @@ inside the container; compose maps `CROPWRIGHT_PORT` to it. ### Running a second, independent instance -Give it its own compose project name, container name and host port so +Give it its own compose project name and host port so it never collides with a running instance: ```bash -CROPWRIGHT_PORT=5190 CROPWRIGHT_CONTAINER_NAME=cw-second \ - CROPWRIGHT_DOCS_CONTAINER_NAME=cw-second-docs \ +OP_CROPWRIGHT_PORT=5190 \ OP_DOCKER_NETWORK=some_other_openprocessor_net \ docker compose -p cw-second up -d ``` @@ -204,11 +203,10 @@ key a registered region slot's own keymap declares. | `PUBLIC_TRITON_API_URL` | _(empty)_ | `npm run dev` | Legacy name (predates the OpenProcessor rename). Points the dev server straight at a backend origin, bypassing the nginx proxy Docker gets for free. Leave empty in Docker. | | `API_UPSTREAM` | `http://op-api:8000` | Docker | Where nginx proxies `PUBLIC_API_PREFIX/*` to, by container name over the shared docker network. | | `OP_DOCKER_NETWORK` | `openprocessor_triton_net` | Docker | The OpenProcessor backend's docker network name (must already exist). | -| `CROPWRIGHT_PORT` | `5184` | Docker | Host port nginx is published on. | -| `CROPWRIGHT_BIND_ADDRESS` | `0.0.0.0` | Docker | Host address the port is published on: this machine and the local network. `127.0.0.1` limits it to this machine. Keep it on a trusted network. | -| `CROPWRIGHT_CONTAINER_NAME` | `cropwright` | Docker | Container name — set uniquely for a second instance (see above). | -| `CROPWRIGHT_INGEST_MAX_REQUEST_MB` | `256` | Docker | Upload cap for `/ingest` (nginx `client_max_body_size`); kept in lockstep with the client-side chunk planner. | -| `CROPWRIGHT_DATASET_UPLOAD_MAX_MB` | `2048` | Docker | Upload cap for a dataset archive on `/datasets/import` (nginx `client_max_body_size`); the client refuses a larger file before sending. | +| `OP_CROPWRIGHT_PORT` | `5184` | Docker | Host port nginx is published on. | +| `OP_CROPWRIGHT_BIND_ADDRESS` | `0.0.0.0` | Docker | Host address the port is published on: this machine and the local network. `127.0.0.1` limits it to this machine. Keep it on a trusted network. | +| `OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB` | `256` | Docker | Upload cap for `/ingest` (nginx `client_max_body_size`); kept in lockstep with the client-side chunk planner. | +| `OP_CROPWRIGHT_DATASET_UPLOAD_MAX_MB` | `2048` | Docker | Upload cap for a dataset archive on `/datasets/import` (nginx `client_max_body_size`); the client refuses a larger file before sending. | | `PUBLIC_APP_NAME` / `PUBLIC_APP_BADGE` | `Cropwright` / `CW` | build time | Top-bar wordmark/badge, for a white-labeled deployment. | The app is a pure SPA consumer of the OpenProcessor API — there is @@ -246,7 +244,7 @@ npm run test:e2e # Playwright against an in-browser stub backend (own venv, auto ``` **Live, read-only tests** against a real deployed backend: -`CROPWRIGHT_LIVE_URL=http://localhost:5184 npm run test:live` — skipped +`OP_CROPWRIGHT_LIVE_URL=http://localhost:5184 npm run test:live` — skipped entirely unless that env var is set; structurally read-only (aborts any non-GET/HEAD request to the API and fails the test if one was even attempted). @@ -303,8 +301,8 @@ them. | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | Top-bar status chip red / "API unavailable" | Backend not reachable at `API_UPSTREAM` (Docker) or `PUBLIC_TRITON_API_URL` (dev); or `{API_PREFIX}/health` returns 5xx | | A write returns 409 "no region profile is configured" | The backend has no region profile set — region-specific actions (confirm/reject a sub-box, etc.) are unavailable until it does | -| Upload to `/ingest` fails with 413 | File(s) exceed `CROPWRIGHT_INGEST_MAX_REQUEST_MB` (nginx) or the backend's own configured upload cap — raise the env var, rebuild | -| Port already in use | Another service already publishes that host port — set `CROPWRIGHT_PORT` (and give the instance its own `CROPWRIGHT_CONTAINER_NAME`) | +| Upload to `/ingest` fails with 413 | File(s) exceed `OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB` (nginx) or the backend's own configured upload cap — raise the env var, rebuild | +| Port already in use | Another service already publishes that host port — set `OP_CROPWRIGHT_PORT` (and give the instance its own `CROPWRIGHT_CONTAINER_NAME`) | | Empty cluster grid / review queues | The backend's index isn't populated yet — ingest some images first | | Build fails: `Cannot find module 'svelte-dnd-action'` | `npm install` wasn't run, or `node_modules` is stale — `rm -rf node_modules && npm install` | diff --git a/frontend/docker-entrypoint.sh b/frontend/docker-entrypoint.sh index 21c21668f..75d93ff42 100755 --- a/frontend/docker-entrypoint.sh +++ b/frontend/docker-entrypoint.sh @@ -15,6 +15,17 @@ # path prefix, in both the bundled JS/HTML and nginx.conf's proxy # `location`, so the two stay in lockstep. See API_PREFIX below. set -eu + +# These names were renamed to OP_CROPWRIGHT_*. A leftover one would be ignored and the +# service would quietly run on defaults, so refuse to start instead. +for retired in IMAGE BIND_ADDRESS PORT INGEST_MAX_REQUEST_MB DATASET_UPLOAD_MAX_MB; do + eval "value=\${CROPWRIGHT_${retired}-}" + if [ -n "$value" ]; then + echo "[entrypoint] CROPWRIGHT_${retired} was renamed to OP_CROPWRIGHT_${retired}: update .env" >&2 + exit 2 + fi +done + TARGET_URL="${PUBLIC_TRITON_API_URL-}" # Empty is the default (relative URLs). Anything else is substituted into a # JS string literal through sed, so it is restricted to characters that are @@ -69,10 +80,10 @@ fi # (src/app.html) so the client-side chunk planner and the proxy's actual # limit can never drift apart. Must be a bare positive integer (MB) — # it's interpolated directly into nginx config. -INGEST_MAX_REQUEST_MB="${CROPWRIGHT_INGEST_MAX_REQUEST_MB:-256}" +INGEST_MAX_REQUEST_MB="${OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB:-256}" case "$INGEST_MAX_REQUEST_MB" in ''|*[!0-9]*) - echo "[entrypoint] CROPWRIGHT_INGEST_MAX_REQUEST_MB must be a positive integer, got: $INGEST_MAX_REQUEST_MB" >&2 + echo "[entrypoint] OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB must be a positive integer, got: $INGEST_MAX_REQUEST_MB" >&2 exit 1 ;; esac @@ -80,10 +91,10 @@ esac # The dataset-archive upload cap (W10 `POST .../datasets/uploads`): # nginx's client_max_body_size for that one location, and # window.__CROPWRIGHT_DATASET_UPLOAD_MAX_MB__ for the client's pre-check. -DATASET_UPLOAD_MAX_MB="${CROPWRIGHT_DATASET_UPLOAD_MAX_MB:-2048}" +DATASET_UPLOAD_MAX_MB="${OP_CROPWRIGHT_DATASET_UPLOAD_MAX_MB:-2048}" case "$DATASET_UPLOAD_MAX_MB" in ''|*[!0-9]*) - echo "[entrypoint] CROPWRIGHT_DATASET_UPLOAD_MAX_MB must be a positive integer, got: $DATASET_UPLOAD_MAX_MB" >&2 + echo "[entrypoint] OP_CROPWRIGHT_DATASET_UPLOAD_MAX_MB must be a positive integer, got: $DATASET_UPLOAD_MAX_MB" >&2 exit 1 ;; esac @@ -144,6 +155,6 @@ echo "[entrypoint] PUBLIC_TRITON_API_URL=${TARGET_URL:- str | None: - url = os.environ.get("CROPWRIGHT_LIVE_URL") + url = os.environ.get("OP_CROPWRIGHT_LIVE_URL") return url.rstrip("/") if url else None @pytest.fixture(scope="session", autouse=True) def _require_live_url() -> None: """Skip the entire tier, before any fixture that could touch a - browser or network runs, unless CROPWRIGHT_LIVE_URL is set. + browser or network runs, unless OP_CROPWRIGHT_LIVE_URL is set. Session-scoped + autouse: pytest instantiates session-scoped fixtures before function-scoped ones (`page`, `browser`, …) @@ -91,7 +91,7 @@ def _require_live_url() -> None: """ if _env_live_url() is None: pytest.skip( - "live tier skipped: set CROPWRIGHT_LIVE_URL (e.g. " + "live tier skipped: set OP_CROPWRIGHT_LIVE_URL (e.g. " "http://localhost:5184) to run it against a real deployment — " "see CLAUDE.md's 'Live read-only tier' section. Never set in " "CI or by `npm run test:e2e`." @@ -133,9 +133,9 @@ def live_project(live_url: str) -> dict[str, Any]: backend, but this tier fails closed rather than guessing a prefix). """ payload = global_api_get(live_url, "/projects") - # CROPWRIGHT_LIVE_PROJECT points the (read-only) tier at a project that + # OP_CROPWRIGHT_LIVE_PROJECT points the (read-only) tier at a project that # holds data when the served default is empty; it must still be a served slug. - default_slug = os.environ.get("CROPWRIGHT_LIVE_PROJECT") or payload["default_slug"] + default_slug = os.environ.get("OP_CROPWRIGHT_LIVE_PROJECT") or payload["default_slug"] match = next((p for p in payload["projects"] if p["slug"] == default_slug), None) if match is None: pytest.skip( diff --git a/frontend/e2e/live/test_data_agreement.py b/frontend/e2e/live/test_data_agreement.py index ef9826d8b..a8368d3e8 100644 --- a/frontend/e2e/live/test_data_agreement.py +++ b/frontend/e2e/live/test_data_agreement.py @@ -41,7 +41,7 @@ def test_dashboard_cluster_count_agrees_with_stats_dataset( page = guarded_page.page page.goto(f"{live_url}{page_path(live_project, '/dashboard')}", wait_until="domcontentloaded") # `data-testid="dataset-cluster-count"` (DatasetStats.svelte) is only - # in source as of this change — the deployment at CROPWRIGHT_LIVE_URL + # in source as of this change — the deployment at OP_CROPWRIGHT_LIVE_URL # is a prior build without it (this tier is instructed not to rebuild # or restart the container), so select by the stable dt/dd label # pairing instead, which works against both the old and new build. diff --git a/frontend/e2e/live/test_route_sweep.py b/frontend/e2e/live/test_route_sweep.py index d9b64d22a..c5b1be330 100644 --- a/frontend/e2e/live/test_route_sweep.py +++ b/frontend/e2e/live/test_route_sweep.py @@ -2,7 +2,7 @@ deployment without erroring. Each route in ROUTES is opened in a real browser pointed at -`CROPWRIGHT_LIVE_URL` (nginx on :5184 by default, proxying `/curation` to +`OP_CROPWRIGHT_LIVE_URL` (nginx on :5184 by default, proxying `/curation` to a live OpenProcessor backend). For each we assert: * no `pageerror` (guaranteed by `guarded_page`'s teardown — see diff --git a/frontend/e2e/stubbed/test_ingest.py b/frontend/e2e/stubbed/test_ingest.py index 82c5647bd..e0574f328 100644 --- a/frontend/e2e/stubbed/test_ingest.py +++ b/frontend/e2e/stubbed/test_ingest.py @@ -237,7 +237,7 @@ def test_nginx_style_413_html_stops_run(stub, page, app_url): page.wait_for_selector("text=3 files selected") page.get_by_role("button", name="Start", exact=True).click() - page.wait_for_selector("text=CROPWRIGHT_INGEST_MAX_REQUEST_MB") + page.wait_for_selector("text=OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB") def test_drain_gate(stub, page, app_url): diff --git a/frontend/nginx.conf b/frontend/nginx.conf index 35c986ad9..db2799762 100644 --- a/frontend/nginx.conf +++ b/frontend/nginx.conf @@ -48,7 +48,7 @@ server { # a regex location inside another regex location, so this is a # sibling block rather than the nested one an earlier draft of this # plan sketched). __INGEST_MAX_REQUEST_MB__ is substituted at - # container start from CROPWRIGHT_INGEST_MAX_REQUEST_MB (default + # container start from OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB (default # 256) — the same value the bundled JS reads via # window.__CROPWRIGHT_INGEST_MAX_REQUEST_MB__ so the client-side # planner and this limit can never drift apart. @@ -75,7 +75,7 @@ server { # Dataset archive uploads (OpenProcessor W10, docs/design/ # w10-import-reprocess-ui-plan-2026-09-27.md §6): one multipart # .zip/.tar/.tar.gz, streamed to the backend, up to - # __DATASET_UPLOAD_MAX_MB__ (CROPWRIGHT_DATASET_UPLOAD_MAX_MB, default + # __DATASET_UPLOAD_MAX_MB__ (OP_CROPWRIGHT_DATASET_UPLOAD_MAX_MB, default # 2048 — the backend's own default cap is 2 GiB). The client reads the # same value via window.__CROPWRIGHT_DATASET_UPLOAD_MAX_MB__ and # refuses a larger file before sending. Declared before the general diff --git a/frontend/scripts/capture_docs_screenshots.py b/frontend/scripts/capture_docs_screenshots.py index aa7a00342..9d05cf01c 100755 --- a/frontend/scripts/capture_docs_screenshots.py +++ b/frontend/scripts/capture_docs_screenshots.py @@ -23,8 +23,8 @@ npm run test:e2e # once, to provision e2e/.venv with Playwright e2e/.venv/bin/python scripts/capture_docs_screenshots.py \\ --base-url http://localhost: - # or: CROPWRIGHT_URL=... e2e/.venv/bin/python scripts/capture_docs_screenshots.py - # or, from the repo root: make docs-screenshots CROPWRIGHT_URL=... + # or: OP_CROPWRIGHT_URL=... e2e/.venv/bin/python scripts/capture_docs_screenshots.py + # or, from the repo root: make docs-screenshots OP_CROPWRIGHT_URL=... The capture is read-only: every non-GET/HEAD request is aborted, so a screenshot pass can never write to the backend it points at. @@ -912,14 +912,14 @@ def main() -> int: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument( "--base-url", - default=os.environ.get("CROPWRIGHT_URL"), - required="CROPWRIGHT_URL" not in os.environ, + default=os.environ.get("OP_CROPWRIGHT_URL"), + required="OP_CROPWRIGHT_URL" not in os.environ, help="Base URL of a Cropwright instance pointed at a PUBLIC-sample-data " "OpenProcessor backend only. Never a real deployment.", ) parser.add_argument( "--project", - default=os.environ.get("CROPWRIGHT_PROJECT"), + default=os.environ.get("OP_CROPWRIGHT_PROJECT"), help="Project slug holding the public sample data. Without it bare routes " "redirect to the default project.", ) diff --git a/frontend/scripts/run-e2e.mjs b/frontend/scripts/run-e2e.mjs index 4b85d9401..dc3647ffb 100755 --- a/frontend/scripts/run-e2e.mjs +++ b/frontend/scripts/run-e2e.mjs @@ -9,7 +9,7 @@ * `npm run test:live` reuses this exact venv/browser setup, just pointed * at `e2e/live` instead — pass the target directory as argv[2] (see * package.json). The live tier never builds/serves anything itself; it - * talks to whatever `CROPWRIGHT_LIVE_URL` points at (its own conftest.py + * talks to whatever `OP_CROPWRIGHT_LIVE_URL` points at (its own conftest.py * skips the whole tier when that's unset), and gets `--screenshot= * only-on-failure` so a failure leaves a screenshot in * artifacts_local/cw-live/live-tier/ instead of nothing. diff --git a/openprocessor b/openprocessor index 23f31e441..dcca52205 100755 --- a/openprocessor +++ b/openprocessor @@ -29,6 +29,7 @@ source "${SCRIPT_DIR}/lib/ports.sh" source "${SCRIPT_DIR}/lib/bind_dirs.sh" source "${SCRIPT_DIR}/lib/vlm_catalog.sh" source "${SCRIPT_DIR}/lib/vlm_switch.sh" +source "${SCRIPT_DIR}/lib/retired_env.sh" # Installer plan section 5.1: every compose call goes through dc(), which # always passes -p, --env-file and --project-directory. Section 5.5: @@ -344,6 +345,12 @@ main() { local command="${1:-help}" shift 2>/dev/null || true + # Only here: dc() often runs inside a command substitution, where an exit is swallowed. + case "$command" in + help|--help|-h|version) ;; + *) refuse_retired_env "$PROJECT_DIR/.env" ;; + esac + case "$command" in start) cmd_start "$@" diff --git a/release-manifest.txt b/release-manifest.txt index afb7caa30..8e8750b16 100644 --- a/release-manifest.txt +++ b/release-manifest.txt @@ -22,6 +22,7 @@ scripts/lib/vlm_catalog.sh scripts/lib/vlm_switch.sh scripts/lib/image_keys.sh scripts/lib/opensearch_heap.sh +scripts/lib/retired_env.sh examples/vlm/catalog.tsv export/build_pe_trt.sh exec monitoring/** optional diff --git a/scripts/docs/check_docs_vs_code.py b/scripts/docs/check_docs_vs_code.py index 5f96ffca7..487b0df4c 100755 --- a/scripts/docs/check_docs_vs_code.py +++ b/scripts/docs/check_docs_vs_code.py @@ -1,15 +1,18 @@ #!/usr/bin/env python3 """Check the documentation against the code. -Four checks, each reporting every miss (not just the first): +Five checks, each reporting every miss (not just the first): * routes - every ``METHOD /path`` written in a doc exists in the generated OpenAPI (``contracts/openapi/*.json``) or, for non-curation routes, in the assembled FastAPI app. * env - every ``OP_*`` token written in a doc exists in code, ``env.template``, compose files or the installer scripts. -* compose - every ``${VAR}`` a root compose file reads appears in ``env.template`` - (commented counts), so an operator can find every knob. +* compose - every ``${VAR}`` a root compose file reads is assigned in ``env.template`` + (commented counts, a prose mention does not), so an operator can find every knob. + Names the installer refuses (``RETIRED_ENV_NAMES``) are exempt. +* template - every non-``OP_`` setting ``env.template`` assigns is read by a compose file, + script or code, so nothing advertised does nothing. * links - every relative link resolves to a file, and every ``#anchor`` that points into a markdown file matches a heading in that file. @@ -64,7 +67,15 @@ # Resolution prefixes for abbreviated paths, tried in order. _ROUTE_PREFIXES = ('', '/curation', '/curation/projects/{project}') -_CODE_ENV_ROOTS = ('src', 'scripts', 'docker', 'openprocessor', 'config_templates') +_CODE_ENV_ROOTS = ( + 'src', + 'scripts', + 'docker', + 'openprocessor', + 'config_templates', + 'frontend/e2e', + 'frontend/scripts', +) _CODE_ENV_FILES = ( 'env.template', 'setup-openprocessor.sh', @@ -195,7 +206,7 @@ def _env_corpus() -> str: for p in base.rglob('*'): if not p.is_file() or set(p.relative_to(REPO_ROOT).parts) & _SKIP_DIR_NAMES: continue - if p.suffix in {'.py', '.sh', '.yml', '.yaml', '.json', '.toml', ''}: + if p.suffix in {'.py', '.sh', '.mjs', '.yml', '.yaml', '.json', '.toml', ''}: parts.append(p.read_text(encoding='utf-8', errors='replace')) parts.extend( p.read_text(encoding='utf-8', errors='replace') @@ -219,21 +230,55 @@ def check_env() -> list[str]: _COMPOSE_VAR_RE = re.compile(r'(? set[str]: + """Names env.template assigns, commented out or not; prose mentions do not count.""" + text = (REPO_ROOT / 'env.template').read_text(encoding='utf-8', errors='replace') + return set(_TEMPLATE_ASSIGN_RE.findall(text)) + + +def _retired_env_names() -> set[str]: + """Names the installer refuses in a .env (renamed); compose may still pass them through.""" + script = REPO_ROOT / 'setup-openprocessor.sh' + if not script.is_file(): + return set() + block = _RETIRED_BLOCK_RE.search(script.read_text(encoding='utf-8', errors='replace')) + return set(block.group(1).split()) if block else set() def check_compose_vars() -> list[str]: - template = (REPO_ROOT / 'env.template').read_text(encoding='utf-8', errors='replace') + documented = _template_names() | _retired_env_names() errors = [] for compose in sorted(REPO_ROOT.glob('docker-compose*.yml')): text = compose.read_text(encoding='utf-8', errors='replace') for var in sorted(set(_COMPOSE_VAR_RE.findall(text))): - if not re.search(rf'\b{re.escape(var)}\b', template): + if var not in documented: errors.append( # noqa: PERF401 # the append carries a formatted message built next to its check f'{_rel(compose)}: ${{{var}}} is not in env.template (commented is fine)' ) return errors +def check_template_names() -> list[str]: + """A non-OP_ setting in env.template that no compose file, script or code reads is + advertised but does nothing (OP_ names are covered by tests/test_env_surface.py).""" + template = (REPO_ROOT / 'env.template').read_text(encoding='utf-8', errors='replace') + corpus = _env_corpus() + errors = [] + for name in sorted(_template_names()): + if name.startswith('OP_'): + continue + word = re.compile(rf'\b{re.escape(name)}\b') + if len(word.findall(corpus)) <= len(word.findall(template)): + errors.append( + f'env.template: {name} is set by no compose file, script or code' + ) # the append carries a formatted message built next to its check + return errors + + _LINK_RE = re.compile(r'(? list[str]: 'routes': check_routes, 'env': check_env, 'compose': check_compose_vars, + 'template': check_template_names, 'links': check_links, } diff --git a/scripts/lib/image_keys.sh b/scripts/lib/image_keys.sh index c60e9ac64..62c5ab926 100755 --- a/scripts/lib/image_keys.sh +++ b/scripts/lib/image_keys.sh @@ -23,7 +23,7 @@ IMAGE_KEYS_TABLE=( "evaluator|OP_EVALUATOR_IMAGE|build|docker/evaluator/Dockerfile|openprocessor-evaluator" "segmenter|OP_SEGMENTER_IMAGE|build|docker/segmenter/Dockerfile|openprocessor-segmenter" "trainer|OP_TRAINER_IMAGE|build|docker/trainer/Dockerfile|openprocessor-trainer" - "cropwright|CROPWRIGHT_IMAGE|build|frontend/Dockerfile|cropwright" + "cropwright|OP_CROPWRIGHT_IMAGE|build|frontend/Dockerfile|cropwright" "vlm_gemma4|VLM_IMAGE|third|vllm/vllm-openai:gemma4-cu130" "vlm_generic|VLM_IMAGE|third|vllm/vllm-openai:gemma4-cu130" "opensearch|OPENSEARCH_IMAGE|third|opensearchproject/opensearch:3.6.0" diff --git a/scripts/lib/retired_env.sh b/scripts/lib/retired_env.sh new file mode 100755 index 000000000..e97af71dc --- /dev/null +++ b/scripts/lib/retired_env.sh @@ -0,0 +1,34 @@ +#!/bin/bash +# ============================================================================= +# retired_env.sh - refuse a .env that still sets a renamed variable +# ============================================================================= +# Sourced by the deploy CLI (openprocessor) and by setup-openprocessor.sh from the verified +# release. A retired name is not read any more, so leaving it in .env would look like a working +# config that silently does nothing; the compose stack's cropwright entrypoint refuses it too +# (frontend/docker-entrypoint.sh). tests/installer/test_retired_cropwright_env.py keeps the +# three lists in step. + +[[ -n "${_RETIRED_ENV_SH_LOADED:-}" ]] && return 0 +_RETIRED_ENV_SH_LOADED=1 + +# Renamed to OP_ (the Cropwright variables joined the OP_ prefix). +RETIRED_ENV_NAMES=( + CROPWRIGHT_IMAGE + CROPWRIGHT_BIND_ADDRESS + CROPWRIGHT_PORT + CROPWRIGHT_INGEST_MAX_REQUEST_MB + CROPWRIGHT_DATASET_UPLOAD_MAX_MB +) + +# refuse_retired_env FILE -- exit 2 naming the new variable for every retired name set in FILE +refuse_retired_env() { + local file="$1" name found=0 + [[ -f "$file" ]] || return 0 + for name in "${RETIRED_ENV_NAMES[@]}"; do + if awk -v k="$name" '{ i = index($0, "=") } i > 1 && substr($0, 1, i - 1) == k { f = 1 } END { exit !f }' "$file"; then + echo "[ERROR] ${name} was renamed to OP_${name}: update ${file}" >&2 + found=1 + fi + done + (( found == 0 )) || exit 2 +} diff --git a/setup-openprocessor.sh b/setup-openprocessor.sh index 1da9f8f41..35089e9d6 100755 --- a/setup-openprocessor.sh +++ b/setup-openprocessor.sh @@ -148,8 +148,8 @@ dc() { echo "DRY: ${cmd[*]//"$dir"/"${OP_REAL_DIR:-$dir}"}" return 0 fi - env -u COMPOSE_PROFILES -u COMPOSE_FILE -u COMPOSE_ENV_FILES -u CROPWRIGHT_BIND_ADDRESS \ - -u CROPWRIGHT_IMAGE -u CROPWRIGHT_PORT COMPOSE_PROJECT_NAME="$project" "${cmd[@]}" + env -u COMPOSE_PROFILES -u COMPOSE_FILE -u COMPOSE_ENV_FILES -u OP_CROPWRIGHT_BIND_ADDRESS \ + -u OP_CROPWRIGHT_IMAGE -u OP_CROPWRIGHT_PORT COMPOSE_PROJECT_NAME="$project" "${cmd[@]}" } # docker_mut ARGS... -- a state-changing plain docker call (pull/run/rmi) @@ -1845,38 +1845,35 @@ _version_ge() { [[ "$(printf '%s\n%s\n' "$2" "$1" | sort -V | head -n1)" == "$2" ]] } -# choose_cropwright_bind -> CROPWRIGHT_BIND (owner decision, plan 7 addendum): -# the web UI is for this computer AND the local LAN, so it binds 0.0.0.0 by -# default; --local-only, or "no" at the prompt, keeps it on 127.0.0.1. The -# backend ports stay on OP_BIND_ADDRESS (127.0.0.1); LAN browsers reach the -# API through Cropwright's nginx on the docker network. A specific -# non-loopback --bind (say 10.0.0.5) narrows Cropwright to that interface -# too: never wider than the one the user chose. +# choose_cropwright_bind -> OP_CROPWRIGHT_BIND: the web UI serves this computer AND the local LAN +# (0.0.0.0) unless --local-only, or "no" at the prompt, keeps it on 127.0.0.1. Backend ports stay on +# OP_BIND_ADDRESS; LAN browsers reach the API through Cropwright's nginx on the docker network. A +# specific non-loopback --bind (say 10.0.0.5) narrows Cropwright to that interface too. choose_cropwright_bind() { local reply prev - prev="$(read_env_var "${OP_DIR}/.env" CROPWRIGHT_BIND_ADDRESS 2>/dev/null || true)" + prev="$(read_env_var "${OP_DIR}/.env" OP_CROPWRIGHT_BIND_ADDRESS 2>/dev/null || true)" if [[ "$OP_LOCAL_ONLY" == 1 ]]; then - CROPWRIGHT_BIND=127.0.0.1 + OP_CROPWRIGHT_BIND=127.0.0.1 elif [[ "$OP_BIND_ADDRESS" != 0.0.0.0 ]] && ! is_loopback_ipv4 "$OP_BIND_ADDRESS"; then - CROPWRIGHT_BIND="$OP_BIND_ADDRESS" + OP_CROPWRIGHT_BIND="$OP_BIND_ADDRESS" elif [[ -n "$prev" ]] && is_ipv4 "$prev"; then - CROPWRIGHT_BIND="$prev" + OP_CROPWRIGHT_BIND="$prev" elif [[ "$OP_UNATTENDED" != 1 ]] && tty_usable; then prompt_line reply "Let other computers on your LAN open the Cropwright web UI? [Y/n]: " "--local-only" - if [[ -z "$reply" || "${reply,,}" == y* ]]; then CROPWRIGHT_BIND=0.0.0.0; else CROPWRIGHT_BIND=127.0.0.1; fi + if [[ -z "$reply" || "${reply,,}" == y* ]]; then OP_CROPWRIGHT_BIND=0.0.0.0; else OP_CROPWRIGHT_BIND=127.0.0.1; fi else - CROPWRIGHT_BIND=0.0.0.0 + OP_CROPWRIGHT_BIND=0.0.0.0 fi } setup_cropwright() { local port - port="$(read_env_var "${OP_DIR}/.env" CROPWRIGHT_PORT || true)" + port="$(read_env_var "${OP_DIR}/.env" OP_CROPWRIGHT_PORT || true)" [[ "$port" =~ ^[0-9]+$ ]] || port=5184 port="$(next_free_port "$port" "${TAKEN_PORTS}")" || die "no free port for Cropwright" - CROPWRIGHT_PORT="$port" - env_set CROPWRIGHT_PORT "$port" - env_set CROPWRIGHT_BIND_ADDRESS "$CROPWRIGHT_BIND" + OP_CROPWRIGHT_PORT="$port" + env_set OP_CROPWRIGHT_PORT "$port" + env_set OP_CROPWRIGHT_BIND_ADDRESS "$OP_CROPWRIGHT_BIND" # Check what Compose will really do with the cropwright service: the # published host IP must be the chosen one, and the image the pinned one. @@ -1885,11 +1882,11 @@ setup_cropwright() { hosts="$(printf '%s\n' "$json" | awk '/"cropwright": \{/ {on=1} on && /"host_ip":/ {gsub(/.*"host_ip": *"|".*/, ""); print; exit}')" # A port without a host IP is published on every interface. [[ -z "$hosts" ]] && hosts="0.0.0.0" - if [[ "$hosts" != "$CROPWRIGHT_BIND" ]]; then - die "the cropwright service would publish on '${hosts}', not ${CROPWRIGHT_BIND} (its compose entry must honour CROPWRIGHT_BIND_ADDRESS)" "$EXIT_VERIFY" + if [[ "$hosts" != "$OP_CROPWRIGHT_BIND" ]]; then + die "the cropwright service would publish on '${hosts}', not ${OP_CROPWRIGHT_BIND} (its compose entry must honour OP_CROPWRIGHT_BIND_ADDRESS)" "$EXIT_VERIFY" fi assert_container_names_free "${OP_PROJECT}-cropwright" - log_success "Cropwright configured on ${CROPWRIGHT_BIND}:${port}" + log_success "Cropwright configured on ${OP_CROPWRIGHT_BIND}:${port}" } # ----------------------------------------------------------------------------- @@ -2022,8 +2019,8 @@ run_health() { wait_http "http://${h}:${p}/health" 180 || fails+=("mlflow not healthy: ./openprocessor logs curation-mlflow") fi if _has_tier cropwright; then - wait_http "http://${h}:${CROPWRIGHT_PORT}/" 120 || fails+=("cropwright / not 200") - wait_http "http://${h}:${CROPWRIGHT_PORT}/curation/health" 60 || fails+=("cropwright cannot reach the API through its proxy (network ${OP_PROJECT}_triton_net)") + wait_http "http://${h}:${OP_CROPWRIGHT_PORT}/" 120 || fails+=("cropwright / not 200") + wait_http "http://${h}:${OP_CROPWRIGHT_PORT}/curation/health" 60 || fails+=("cropwright cannot reach the API through its proxy (network ${OP_PROJECT}_triton_net)") fi if [[ "$OP_WITH_MONITORING" == 1 ]]; then local unreadable @@ -2354,11 +2351,11 @@ print_summary() { if _has_tier vlm; then vlmp="$(read_env_var "$ENV_FILE" VLM_PORT)"; echo " VLM http://${h}:${vlmp}/v1/models (${VLM_PICK})"; fi if _has_tier trainer; then ml="$(read_env_var "$ENV_FILE" MLFLOW_PORT)"; echo " MLflow http://${h}:${ml}"; fi if _has_tier cropwright; then - echo " Cropwright http://${h}:${CROPWRIGHT_PORT}" - if ! is_loopback_ipv4 "${CROPWRIGHT_BIND:-127.0.0.1}"; then - local lan="$CROPWRIGHT_BIND" + echo " Cropwright http://${h}:${OP_CROPWRIGHT_PORT}" + if ! is_loopback_ipv4 "${OP_CROPWRIGHT_BIND:-127.0.0.1}"; then + local lan="$OP_CROPWRIGHT_BIND" [[ "$lan" == 0.0.0.0 ]] && lan="$(hostname -I 2>/dev/null | awk '{ print $1 }' || true)" - echo " Cropwright on your LAN: http://${lan:-}:${CROPWRIGHT_PORT}" + echo " Cropwright on your LAN: http://${lan:-}:${OP_CROPWRIGHT_PORT}" log_warn "Cropwright is reachable from your LAN and has NO login: use it only on a trusted network," log_warn "never port-forward it to the internet, and put a reverse proxy with authentication in front" log_warn "for anything wider (SECURITY.md). Re-run with --local-only to keep it on this computer." @@ -2374,7 +2371,7 @@ print_summary() { fi if ! is_loopback_ipv4 "$OP_BIND_ADDRESS"; then log_warn "SECURITY: services are published on ${OP_BIND_ADDRESS} with NO authentication. See SECURITY.md." - elif _has_tier cropwright && ! is_loopback_ipv4 "${CROPWRIGHT_BIND:-127.0.0.1}"; then + elif _has_tier cropwright && ! is_loopback_ipv4 "${OP_CROPWRIGHT_BIND:-127.0.0.1}"; then echo " Security: every OpenProcessor API port is bound to ${OP_BIND_ADDRESS}; Cropwright is the exception (your LAN, above)." echo " The API has no auth; use a reverse proxy before exposing it." else @@ -2789,6 +2786,9 @@ do_install() { source "${OP_DIR}/scripts/lib/image_keys.sh" # shellcheck source=scripts/lib/opensearch_heap.sh source "${OP_DIR}/scripts/lib/opensearch_heap.sh" + # shellcheck source=scripts/lib/retired_env.sh + source "${OP_DIR}/scripts/lib/retired_env.sh" + refuse_retired_env "$ENV_FILE" env_create_or_merge env_set COMPOSE_PROJECT_NAME "$OP_PROJECT" @@ -3152,8 +3152,8 @@ main() { TRITON_DIGEST="" IMAGE_DIGESTS=() PINNED_IMAGES=() - CROPWRIGHT_PORT="" - CROPWRIGHT_BIND="" + OP_CROPWRIGHT_PORT="" + OP_CROPWRIGHT_BIND="" ACTIVE_IMAGES=() VLM_PICK="" RESOLVED_MODE="" diff --git a/tests/compose/test_ports_names_and_project_name.py b/tests/compose/test_ports_names_and_project_name.py index de8bf0b79..652bb1de5 100644 --- a/tests/compose/test_ports_names_and_project_name.py +++ b/tests/compose/test_ports_names_and_project_name.py @@ -129,7 +129,7 @@ def test_every_published_port_uses_op_bind_address() -> None: text = str(entry) # Cropwright has its own knob so the installer can open only the web UI to a LAN. if not text.startswith( - ('${OP_BIND_ADDRESS', '${OP_UI_BIND_ADDRESS', '${CROPWRIGHT_BIND_ADDRESS') + ('${OP_BIND_ADDRESS', '${OP_UI_BIND_ADDRESS', '${OP_CROPWRIGHT_BIND_ADDRESS') ): bad.append(f'{name}: {text!r}') assert not bad, 'published ports missing ${OP_BIND_ADDRESS prefix:\n' + '\n'.join(bad) diff --git a/tests/installer/shims/docker b/tests/installer/shims/docker index e7a2525a9..d12739f3c 100755 --- a/tests/installer/shims/docker +++ b/tests/installer/shims/docker @@ -132,7 +132,7 @@ cmd_compose() { [[ "$profiles" == *,curation,* ]] && wanted+=" OP_EVALUATOR_IMAGE:openprocessor-evaluator" [[ "$profiles" == *,segmenter,* ]] && wanted+=" OP_SEGMENTER_IMAGE:openprocessor-segmenter" [[ "$profiles" == *,training,* ]] && wanted+=" OP_TRAINER_IMAGE:openprocessor-trainer" - [[ "$profiles" == *,cropwright,* ]] && wanted+=" CROPWRIGHT_IMAGE:cropwright" + [[ "$profiles" == *,cropwright,* ]] && wanted+=" OP_CROPWRIGHT_IMAGE:cropwright" for k in $wanted; do v="$(env_val "$envfile" "${k%%:*}")" [[ -n "$v" ]] || v="${repo}/${k#*:}:${tag}" @@ -157,8 +157,8 @@ cmd_compose() { fi local cbind cport cservice="" if [[ ",$(env_val "$envfile" COMPOSE_PROFILES)," == *,cropwright,* ]]; then - cbind="${CROPWRIGHT_BIND_ADDRESS:-$(env_val "$envfile" CROPWRIGHT_BIND_ADDRESS)}" - cport="$(env_val "$envfile" CROPWRIGHT_PORT)" + cbind="${OP_CROPWRIGHT_BIND_ADDRESS:-$(env_val "$envfile" OP_CROPWRIGHT_BIND_ADDRESS)}" + cport="$(env_val "$envfile" OP_CROPWRIGHT_PORT)" [[ -e "${state}/cw_ignores_bind" ]] && cbind=0.0.0.0 cservice=" \"cropwright\": { \"container_name\": \"${cpn}-cropwright\", diff --git a/tests/installer/test_lifecycle.py b/tests/installer/test_lifecycle.py index 79680d2c0..a392d363e 100644 --- a/tests/installer/test_lifecycle.py +++ b/tests/installer/test_lifecycle.py @@ -465,7 +465,7 @@ def test_release_dir_install_pins_and_enables_cropwright(shimmed: Shimmed, tmp_p assert result.returncode == 0, result.stderr[-2000:] env_text = (shimmed.root / 'inst' / '.env').read_text() assert 'cropwright' in env_text.split('COMPOSE_PROFILES=')[1].splitlines()[0].split(',') - assert f'CROPWRIGHT_IMAGE=davidamacey/cropwright@{fake_digest("cropwright")}' in env_text + assert f'OP_CROPWRIGHT_IMAGE=davidamacey/cropwright@{fake_digest("cropwright")}' in env_text assert not (shimmed.root / 'inst' / 'cropwright').exists() assert [ln for ln in shimmed.log_lines('curl') if 'cw.test' in ln] == [] diff --git a/tests/installer/test_retired_cropwright_env.py b/tests/installer/test_retired_cropwright_env.py new file mode 100644 index 000000000..a9e100346 --- /dev/null +++ b/tests/installer/test_retired_cropwright_env.py @@ -0,0 +1,104 @@ +"""The Cropwright variables moved under OP_CROPWRIGHT_*; a .env that still sets an old +name is refused by the installer and the deploy CLI, naming the new variable.""" + +from __future__ import annotations + +import re +import subprocess +from typing import TYPE_CHECKING + +import pytest +from _setup_dry_run_helpers import configure, dry +from installer_harness import REPO_ROOT + + +if TYPE_CHECKING: + from pathlib import Path + + from installer_harness import Shimmed + +RETIRED = { + 'CROPWRIGHT_IMAGE': 'OP_CROPWRIGHT_IMAGE', + 'CROPWRIGHT_BIND_ADDRESS': 'OP_CROPWRIGHT_BIND_ADDRESS', + 'CROPWRIGHT_PORT': 'OP_CROPWRIGHT_PORT', + 'CROPWRIGHT_INGEST_MAX_REQUEST_MB': 'OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB', + 'CROPWRIGHT_DATASET_UPLOAD_MAX_MB': 'OP_CROPWRIGHT_DATASET_UPLOAD_MAX_MB', +} + + +def _install(shimmed: Shimmed) -> Path: + assert configure(shimmed).returncode == 0 + return (shimmed.root / 'inst').resolve() + + +def _cli(shimmed: Shimmed, inst: Path, *args: str) -> subprocess.CompletedProcess: + return subprocess.run( + [str(inst / 'openprocessor'), *args], + check=False, + cwd=str(shimmed.root), + env=shimmed.env(), + capture_output=True, + text=True, + timeout=60, + start_new_session=True, + ) + + +@pytest.mark.parametrize(('old', 'new'), sorted(RETIRED.items())) +def test_installer_refuses_a_retired_name_in_env(shimmed: Shimmed, old: str, new: str) -> None: + inst = _install(shimmed) + with (inst / '.env').open('a') as fh: + fh.write(f'{old}=1\n') + result = dry(shimmed) + assert result.returncode == 2, result.stderr + assert f'{old} was renamed to {new}' in result.stderr + + +@pytest.mark.parametrize(('old', 'new'), sorted(RETIRED.items())) +def test_cli_refuses_a_retired_name_in_env(shimmed: Shimmed, old: str, new: str) -> None: + inst = _install(shimmed) + with (inst / '.env').open('a') as fh: + fh.write(f'{old}=1\n') + result = _cli(shimmed, inst, 'status') + assert result.returncode == 2, result.stderr + assert f'{old} was renamed to {new}' in result.stdout + result.stderr + + +def test_cli_accepts_a_current_env(shimmed: Shimmed) -> None: + inst = _install(shimmed) + with (inst / '.env').open('a') as fh: + fh.write('OP_CROPWRIGHT_PORT=5190\n') + result = _cli(shimmed, inst, 'status') + assert 'renamed' not in result.stdout + result.stderr + + +def _names(text: str) -> set[str]: + return set(re.findall(r'\bCROPWRIGHT_[A-Z_]+', text)) + + +def test_retired_lists_agree_everywhere() -> None: + lib = (REPO_ROOT / 'scripts' / 'lib' / 'retired_env.sh').read_text() + compose = (REPO_ROOT / 'docker-compose.yml').read_text() + entrypoint = (REPO_ROOT / 'frontend' / 'docker-entrypoint.sh').read_text() + block = re.search(r'RETIRED_ENV_NAMES=\((.*?)\)', lib, re.S) + assert block, 'scripts/lib/retired_env.sh has no RETIRED_ENV_NAMES' + assert _names(block.group(1)) == set(RETIRED) + passthrough = set(re.findall(r'^\s+- (CROPWRIGHT_[A-Z_]+)=\$\{\1:-\}$', compose, re.M)) + assert passthrough == set(RETIRED) + loop = re.search(r'for retired in ([A-Z_ ]+); do', entrypoint) + assert loop, 'the cropwright entrypoint has no retired-name loop' + assert {f'CROPWRIGHT_{n}' for n in loop.group(1).split()} == set(RETIRED) + + +@pytest.mark.parametrize('old', sorted(RETIRED)) +def test_cropwright_entrypoint_refuses_a_retired_name(old: str) -> None: + result = subprocess.run( + ['sh', str(REPO_ROOT / 'frontend' / 'docker-entrypoint.sh')], + check=False, + env={'PATH': '/usr/bin:/bin', old: '1'}, + capture_output=True, + text=True, + timeout=30, + ) + assert result.returncode == 2 + assert f'{old} was renamed to OP_{old}' in result.stderr diff --git a/tests/installer/test_setup_bind_and_cropwright.py b/tests/installer/test_setup_bind_and_cropwright.py index c303b6486..c4ee02359 100644 --- a/tests/installer/test_setup_bind_and_cropwright.py +++ b/tests/installer/test_setup_bind_and_cropwright.py @@ -92,8 +92,8 @@ def test_cropwright_is_reachable_from_the_lan_by_default_with_a_warning(shimmed: result = configure(shimmed, tiers='cropwright') assert result.returncode == 0, result.stderr cw_env = _cw_env(shimmed) - assert 'CROPWRIGHT_BIND_ADDRESS=0.0.0.0' in cw_env - assert f'CROPWRIGHT_IMAGE=davidamacey/cropwright@{fake_digest("cropwright")}' in cw_env + assert 'OP_CROPWRIGHT_BIND_ADDRESS=0.0.0.0' in cw_env + assert f'OP_CROPWRIGHT_IMAGE=davidamacey/cropwright@{fake_digest("cropwright")}' in cw_env assert 'cropwright' in env_file(shimmed)['COMPOSE_PROFILES'].split(',') assert stat.S_IMODE((shimmed.root / 'inst' / '.env').stat().st_mode) == 0o600 assert not (shimmed.root / 'inst' / 'cropwright').exists() @@ -109,23 +109,23 @@ def test_cropwright_is_reachable_from_the_lan_by_default_with_a_warning(shimmed: def test_local_only_keeps_cropwright_on_this_computer(shimmed: Shimmed) -> None: result = configure(shimmed, '--local-only', tiers='cropwright') assert result.returncode == 0, result.stderr - assert 'CROPWRIGHT_BIND_ADDRESS=127.0.0.1' in _cw_env(shimmed) + assert 'OP_CROPWRIGHT_BIND_ADDRESS=127.0.0.1' in _cw_env(shimmed) assert 'reachable from your LAN' not in result.stderr def test_a_shell_bind_cannot_override_the_chosen_cropwright_bind(shimmed: Shimmed) -> None: result = configure( - shimmed, '--local-only', tiers='cropwright', CROPWRIGHT_BIND_ADDRESS='0.0.0.0' + shimmed, '--local-only', tiers='cropwright', OP_CROPWRIGHT_BIND_ADDRESS='0.0.0.0' ) assert result.returncode == 0, result.stderr - assert 'CROPWRIGHT_BIND_ADDRESS=127.0.0.1' in _cw_env(shimmed) + assert 'OP_CROPWRIGHT_BIND_ADDRESS=127.0.0.1' in _cw_env(shimmed) def test_cropwright_compose_that_ignores_the_bind_is_refused(shimmed: Shimmed) -> None: shimmed.flag('cw_ignores_bind') result = configure(shimmed, '--local-only', tiers='cropwright') assert result.returncode == 7 - assert 'must honour CROPWRIGHT_BIND_ADDRESS' in result.stderr + assert 'must honour OP_CROPWRIGHT_BIND_ADDRESS' in result.stderr def test_summary_security_line_excepts_a_lan_cropwright(shimmed: Shimmed) -> None: @@ -152,7 +152,7 @@ def test_specific_bind_address_also_applies_to_cropwright(shimmed: Shimmed) -> N shimmed, '--bind', '10.10.10.20', tiers='cropwright', OP_ALLOW_PUBLIC_BIND='1' ) assert result.returncode == 0, result.stderr - assert 'CROPWRIGHT_BIND_ADDRESS=10.10.10.20' in _cw_env(shimmed) + assert 'OP_CROPWRIGHT_BIND_ADDRESS=10.10.10.20' in _cw_env(shimmed) assert 'Cropwright on your LAN: http://10.10.10.20:' in result.stdout assert 'reachable from your LAN and has NO login' in result.stderr @@ -164,7 +164,7 @@ def test_rerun_without_bind_keeps_a_specific_cropwright_bind(shimmed: Shimmed) - assert first.returncode == 0, first.stderr again = configure(shimmed, tiers='cropwright') assert again.returncode == 0, again.stderr - assert 'CROPWRIGHT_BIND_ADDRESS=10.10.10.20' in _cw_env(shimmed) + assert 'OP_CROPWRIGHT_BIND_ADDRESS=10.10.10.20' in _cw_env(shimmed) def test_local_only_wins_over_a_specific_bind_for_cropwright(shimmed: Shimmed) -> None: @@ -177,10 +177,10 @@ def test_local_only_wins_over_a_specific_bind_for_cropwright(shimmed: Shimmed) - OP_ALLOW_PUBLIC_BIND='1', ) assert result.returncode == 0, result.stderr - assert 'CROPWRIGHT_BIND_ADDRESS=127.0.0.1' in _cw_env(shimmed) + assert 'OP_CROPWRIGHT_BIND_ADDRESS=127.0.0.1' in _cw_env(shimmed) def test_wildcard_bind_keeps_cropwright_on_every_interface(shimmed: Shimmed) -> None: result = configure(shimmed, '--bind', '0.0.0.0', tiers='cropwright', OP_ALLOW_PUBLIC_BIND='1') assert result.returncode == 0, result.stderr - assert 'CROPWRIGHT_BIND_ADDRESS=0.0.0.0' in _cw_env(shimmed) + assert 'OP_CROPWRIGHT_BIND_ADDRESS=0.0.0.0' in _cw_env(shimmed) diff --git a/tests/installer/test_static.py b/tests/installer/test_static.py index 37dfbf7bf..9153050b9 100644 --- a/tests/installer/test_static.py +++ b/tests/installer/test_static.py @@ -177,6 +177,7 @@ def test_installer_sources_only_verified_install_files() -> None: 'source "${OP_DIR}/scripts/lib/model_setup.sh"', 'source "${OP_DIR}/scripts/lib/image_keys.sh"', 'source "${OP_DIR}/scripts/lib/opensearch_heap.sh"', + 'source "${OP_DIR}/scripts/lib/retired_env.sh"', } body = text[text.index('\ndo_install() {') :] assert body.index('install_staged "${OP_DIR}/.install/staging"') < body.index( diff --git a/tests/installer/test_tty_prompts.py b/tests/installer/test_tty_prompts.py index 98dd945fd..cd350bb5d 100644 --- a/tests/installer/test_tty_prompts.py +++ b/tests/installer/test_tty_prompts.py @@ -167,4 +167,4 @@ def test_answering_no_to_lan_access_keeps_cropwright_local(shimmed: Shimmed) -> assert b'Let other computers on your LAN' in tty_out assert rc == 0, err[-2000:] cw_env = (shimmed.root / 'inst' / '.env').read_text() - assert 'CROPWRIGHT_BIND_ADDRESS=127.0.0.1' in cw_env + assert 'OP_CROPWRIGHT_BIND_ADDRESS=127.0.0.1' in cw_env diff --git a/tests/test_compose_gateway.py b/tests/test_compose_gateway.py index 0dbe70eee..bcbbc59f9 100644 --- a/tests/test_compose_gateway.py +++ b/tests/test_compose_gateway.py @@ -92,7 +92,7 @@ def test_gateway_switch_reaches_the_api_the_gateway_and_each_ui(tmp_path: Path) def test_ui_ports_stay_on_loopback_in_gateway_mode(tmp_path: Path) -> None: services = _services( - tmp_path, {'OP_GATEWAY_SUBPATHS': 'true', 'CROPWRIGHT_BIND_ADDRESS': '0.0.0.0'} + tmp_path, {'OP_GATEWAY_SUBPATHS': 'true', 'OP_CROPWRIGHT_BIND_ADDRESS': '0.0.0.0'} ) for name in UI_SERVICES: assert {p['host_ip'] for p in services[name]['ports']} == {'127.0.0.1'}, name diff --git a/tests/test_compose_ui_bind.py b/tests/test_compose_ui_bind.py index 476ad7603..19314cdbf 100644 --- a/tests/test_compose_ui_bind.py +++ b/tests/test_compose_ui_bind.py @@ -72,7 +72,7 @@ def test_ui_variable_defaults_to_bind_address(tmp_path: Path) -> None: def test_cropwright_variable_moves_only_cropwright(tmp_path: Path) -> None: """The installer opens just the web UI to a LAN (homelab default); the API, OpenSearch and the monitoring UIs stay on loopback.""" - hosts = _hosts(tmp_path, {'CROPWRIGHT_BIND_ADDRESS': '0.0.0.0'}) + hosts = _hosts(tmp_path, {'OP_CROPWRIGHT_BIND_ADDRESS': '0.0.0.0'}) for name, h in hosts.items(): expected = {'0.0.0.0'} if name == 'cropwright' else {'127.0.0.1'} assert h == expected, (name, h) @@ -80,7 +80,7 @@ def test_cropwright_variable_moves_only_cropwright(tmp_path: Path) -> None: def test_cropwright_variable_wins_over_the_ui_variable(tmp_path: Path) -> None: hosts = _hosts( - tmp_path, {'OP_UI_BIND_ADDRESS': '0.0.0.0', 'CROPWRIGHT_BIND_ADDRESS': '127.0.0.1'} + tmp_path, {'OP_UI_BIND_ADDRESS': '0.0.0.0', 'OP_CROPWRIGHT_BIND_ADDRESS': '127.0.0.1'} ) assert hosts['cropwright'] == {'127.0.0.1'} assert hosts['grafana'] == {'0.0.0.0'} diff --git a/tests/test_docs_vs_code.py b/tests/test_docs_vs_code.py index 708f37915..7ddc30fd5 100644 --- a/tests/test_docs_vs_code.py +++ b/tests/test_docs_vs_code.py @@ -31,7 +31,7 @@ def _reset_only(): checker.set_only(None) -@pytest.mark.parametrize('name', ['routes', 'env', 'compose']) +@pytest.mark.parametrize('name', ['routes', 'env', 'compose', 'template']) def test_real_docs_match_the_code(name: str) -> None: errors = checker.CHECKS[name]() assert errors == [], f'{name} check found {len(errors)} problem(s):\n' + '\n'.join(errors) @@ -85,6 +85,54 @@ def test_compose_check_rejects_a_variable_missing_from_env_template( ] +def test_compose_check_wants_an_assignment_not_a_prose_mention( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + (tmp_path / 'docker-compose.yml').write_text( + 'services:\n a:\n image: ${ONLY_IN_PROSE:-x}\n', encoding='utf-8' + ) + (tmp_path / 'env.template').write_text( + '# Set ONLY_IN_PROSE to pin the image, see the docs.\n', encoding='utf-8' + ) + monkeypatch.setattr(checker, 'REPO_ROOT', tmp_path) + + assert checker.check_compose_vars() == [ + 'docker-compose.yml: ${ONLY_IN_PROSE} is not in env.template (commented is fine)' + ] + + +def test_compose_check_skips_retired_names_the_installer_refuses( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + (tmp_path / 'docker-compose.yml').write_text( + 'services:\n a:\n environment:\n - OLD_NAME=${OLD_NAME:-}\n', encoding='utf-8' + ) + (tmp_path / 'env.template').write_text('# NEW_NAME=1\n', encoding='utf-8') + (tmp_path / 'setup-openprocessor.sh').write_text( + 'RETIRED_ENV_NAMES=(\n OLD_NAME\n)\n', encoding='utf-8' + ) + monkeypatch.setattr(checker, 'REPO_ROOT', tmp_path) + + assert checker.check_compose_vars() == [] + + +def test_template_check_rejects_a_setting_nothing_reads( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + (tmp_path / 'docker-compose.yml').write_text( + 'services:\n a:\n image: ${READ_BY_COMPOSE:-x}\n', encoding='utf-8' + ) + (tmp_path / 'env.template').write_text( + 'READ_BY_COMPOSE=x\n# READ_NOWHERE=info\nOP_SKIPPED=1\n', encoding='utf-8' + ) + monkeypatch.setattr(checker, 'REPO_ROOT', tmp_path) + monkeypatch.setattr(checker, '_CODE_ENV_ROOTS', ()) + + assert checker.check_template_names() == [ + 'env.template: READ_NOWHERE is set by no compose file, script or code' + ] + + def test_link_check_rejects_missing_file_and_anchor(tmp_path: Path) -> None: (tmp_path / 'other.md').write_text('# Real Heading\n', encoding='utf-8') assert _check_one(tmp_path, '[ok](other.md#real-heading)\n', 'links') == [] diff --git a/tests/test_env_surface.py b/tests/test_env_surface.py index 37d32dbd1..e0d715234 100644 --- a/tests/test_env_surface.py +++ b/tests/test_env_surface.py @@ -99,6 +99,13 @@ 'OP_SHARDS_PER_HEAP_GB', # docker-compose.docs.yml's own ${OP_DOCS_IMAGE:-...} interpolation is the reader. 'OP_DOCS_IMAGE', + # The Cropwright service's own ${VAR:-default} interpolation in docker-compose.yml (and + # frontend/docker-entrypoint.sh in the container) reads these, not Python code. + 'OP_CROPWRIGHT_IMAGE', + 'OP_CROPWRIGHT_BIND_ADDRESS', + 'OP_CROPWRIGHT_PORT', + 'OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB', + 'OP_CROPWRIGHT_DATASET_UPLOAD_MAX_MB', # `openprocessor vlm use` waits for the VLM; read by scripts/lib/vlm_switch.sh (bash). 'OP_VLM_WAIT_S', 'OP_VLM_POLL_S', From 75c293792533014273ded8a4dc9efb05ad8e9631 Mon Sep 17 00:00:00 2001 From: davidamacey Date: Wed, 14 Oct 2026 18:42:46 -0400 Subject: [PATCH 2/8] build(docker): pin bases by digest, multi-stage side-cars and a runtime scripts set Every FROM is image:tag@sha256 (bump procedure in the release checklist). The evaluator and segmenter build their Python environment in a separate stage, the fake VLM test image runs as nobody, and the api image copies scripts/curation, scripts/datasets and the six shell libraries it runs in the container instead of all of scripts/ (the evaluator copies only scripts/curation). The root .dockerignore excludes artifacts_local, docs-site, frontend, .claude, .install and export_*. tests/test_dockerfile_hygiene.py checks the pins, the non-root user, the stages and that the copied scripts cover every import. --- .dockerignore | 9 ++ Dockerfile | 13 +- Dockerfile.triton | 2 +- docker/evaluator/Dockerfile | 64 +++++----- docker/hardened/triton/Dockerfile | 4 +- docker/segmenter/Dockerfile | 65 +++++----- docker/test/Dockerfile.fake-vlm | 4 +- docker/trainer/Dockerfile | 4 +- docs-site/Dockerfile | 4 +- docs/releases/RELEASE_CHECKLIST.md | 19 +++ frontend/Dockerfile | 4 +- tests/test_dockerfile_hygiene.py | 189 +++++++++++++++++++++++++++++ 12 files changed, 315 insertions(+), 66 deletions(-) create mode 100644 tests/test_dockerfile_hygiene.py diff --git a/.dockerignore b/.dockerignore index d4b97fd8e..14c0322af 100644 --- a/.dockerignore +++ b/.dockerignore @@ -45,6 +45,15 @@ Dockerfile* docker-compose*.yml .dockerignore +# Local and sibling trees no root-context image reads (the frontend and the docs site build +# from their own contexts) +artifacts_local/ +docs-site/ +frontend/ +.claude/ +.install/ +export_*/ + # Documentation (not needed in containers) docs/ *.md diff --git a/Dockerfile b/Dockerfile index 1390ba890..e79a6683c 100644 --- a/Dockerfile +++ b/Dockerfile @@ -14,7 +14,7 @@ # ----------------------------------------------------------------------------- # Stage 1: Build - Install Python dependencies with compilation tools # ----------------------------------------------------------------------------- -FROM python:3.13-slim-trixie AS builder +FROM python:3.13-slim-trixie@sha256:70729b46c69b4f1e97c4822c1af3df53a1476cf5ddc6c087c0c10bc3a5678c2f AS builder WORKDIR /build @@ -62,7 +62,7 @@ RUN rm -rf /root/.local/lib/python3.13/site-packages/pip \ # ----------------------------------------------------------------------------- # Stage 2: Runtime - Minimal image with only runtime dependencies # ----------------------------------------------------------------------------- -FROM python:3.13-slim-trixie +FROM python:3.13-slim-trixie@sha256:70729b46c69b4f1e97c4822c1af3df53a1476cf5ddc6c087c0c10bc3a5678c2f LABEL org.opencontainers.image.title="OpenProcessor FastAPI Service" \ org.opencontainers.image.description="Visual AI API with object detection, face recognition, embeddings, and OCR" \ @@ -137,7 +137,14 @@ ENV PATH=/home/appuser/.local/bin:$PATH \ # Copy application code COPY --chown=appuser:appuser src/ ./src/ -COPY --chown=appuser:appuser scripts/ ./scripts/ +# Only what runs inside the container: the curation workers and the CLIs operators run with +# `docker compose exec api python -m scripts.curation...`, the public COCO sample fetcher, and the +# shell libraries `openprocessor models install` sources in the container (config.sh and what it +# sources, download.sh). tests/test_dockerfile_hygiene.py checks this set against the imports. +COPY --chown=appuser:appuser scripts/curation/ ./scripts/curation/ +COPY --chown=appuser:appuser scripts/datasets/ ./scripts/datasets/ +COPY --chown=appuser:appuser scripts/lib/colors.sh scripts/lib/config.sh scripts/lib/config_models.sh \ + scripts/lib/download.sh scripts/lib/gpu.sh scripts/lib/opensearch_heap.sh ./scripts/lib/ COPY --chown=appuser:appuser VERSION ./VERSION COPY --chown=appuser:appuser config_templates/ ./config_templates/ diff --git a/Dockerfile.triton b/Dockerfile.triton index 535fcd6c3..97a3ee54e 100644 --- a/Dockerfile.triton +++ b/Dockerfile.triton @@ -9,7 +9,7 @@ # re-exporting every engine — see docs/MIGRATION_TRITON_26.md. # ============================================================================= -FROM nvcr.io/nvidia/tritonserver:26.06-py3 +FROM nvcr.io/nvidia/tritonserver:26.06-py3@sha256:58df7489c3f2276f9591d500a012dee03e23d35543ce3c390b4c001e6bf90794 LABEL org.opencontainers.image.title="OpenProcessor Inference Server" \ org.opencontainers.image.description="NVIDIA Triton Inference Server with TensorRT models for visual AI" \ diff --git a/docker/evaluator/Dockerfile b/docker/evaluator/Dockerfile index e4d75aa71..da0e65f88 100644 --- a/docker/evaluator/Dockerfile +++ b/docker/evaluator/Dockerfile @@ -9,7 +9,8 @@ # Standalone image (no shared trainer base image exists in this repo): # installs the same detection-stack pins as the core API image # (requirements.txt) plus the harness-only extras -# (scripts/curation/bakeoff/requirements.txt). +# (scripts/curation/bakeoff/requirements.txt). Two stages: the compilers that +# build wheel dependencies stay in the builder and never reach the runtime image. # # Build: # docker compose --profile curation build curation-evaluator @@ -19,18 +20,39 @@ # One-shot job: # docker compose --profile curation run --rm curation-evaluator \ # python -m scripts.curation.bakeoff.bakeoff_runner --job /eval_jobs/x.job.json -# -# WAIVER (release checklist): still a single -# `FROM`, not multi-stage. build-essential/cmake/pkg-config stay in the -# final image to compile wheel deps at `pip install` time, which the -# other images' builder-stage split avoids. Deferred: the current scope is the -# compose/mount split, not image-size hardening, and this image is -# on-demand only (`docker compose ... run --rm`), never a long-running -# service. Track the multi-stage conversion as release-checklist debt -# (§3.4) rather than doing it silently here. # ============================================================================= -FROM python:3.13-slim-trixie +# ----------------------------------------------------------------------------- +# Stage 1: build -- resolve every wheel into a relocatable --user tree +# ----------------------------------------------------------------------------- +FROM python:3.13-slim-trixie@sha256:70729b46c69b4f1e97c4822c1af3df53a1476cf5ddc6c087c0c10bc3a5678c2f AS builder + +WORKDIR /build + +# hadolint ignore=DL3008 +RUN apt-get update && apt-get upgrade -y \ + && apt-get install -y --no-install-recommends \ + build-essential \ + cmake \ + git \ + pkg-config \ + && rm -rf /var/lib/apt/lists/* + +COPY requirements.txt requirements.txt +COPY scripts/curation/bakeoff/requirements.txt bakeoff-requirements.txt + +# hadolint ignore=DL3013 +RUN pip install --user --no-cache-dir --upgrade pip \ + && pip install --user --no-cache-dir --no-warn-script-location \ + --extra-index-url https://pypi.nvidia.com \ + -r requirements.txt \ + && pip install --user --no-cache-dir --no-warn-script-location -r bakeoff-requirements.txt \ + && pip install --user --no-cache-dir --no-warn-script-location mlflow + +# ----------------------------------------------------------------------------- +# Stage 2: runtime +# ----------------------------------------------------------------------------- +FROM python:3.13-slim-trixie@sha256:70729b46c69b4f1e97c4822c1af3df53a1476cf5ddc6c087c0c10bc3a5678c2f LABEL org.opencontainers.image.title="OpenProcessor Curation Evaluator" \ org.opencontainers.image.description="On-demand detector bake-off harness runner" \ @@ -47,11 +69,6 @@ LABEL org.opencontainers.image.revision=${OP_BUILD_SHA} # hadolint ignore=DL3008 RUN apt-get update && apt-get upgrade -y \ && apt-get install -y --no-install-recommends \ - build-essential \ - cmake \ - curl \ - git \ - pkg-config \ libgl1 \ libglib2.0-0t64 \ libgomp1 \ @@ -65,24 +82,17 @@ RUN groupadd -r appuser && \ WORKDIR /app -COPY requirements.txt requirements.txt -COPY scripts/curation/bakeoff/requirements.txt bakeoff-requirements.txt - -# hadolint ignore=DL3013 -RUN pip install --no-cache-dir --upgrade pip \ - && pip install --no-cache-dir --extra-index-url https://pypi.nvidia.com \ - -r requirements.txt \ - && pip install --no-cache-dir -r bakeoff-requirements.txt \ - && pip install --no-cache-dir mlflow +COPY --from=builder --chown=appuser:appuser /root/.local /home/appuser/.local -ENV PYTHONUNBUFFERED=1 \ +ENV PATH=/home/appuser/.local/bin:$PATH \ + PYTHONUNBUFFERED=1 \ PYTHONDONTWRITEBYTECODE=1 \ PYTHONPATH=/app \ HF_HOME=/home/appuser/.cache/huggingface \ TORCH_HOME=/home/appuser/.cache/torch COPY --chown=appuser:appuser src/ ./src/ -COPY --chown=appuser:appuser scripts/ ./scripts/ +COPY --chown=appuser:appuser scripts/curation/ ./scripts/curation/ # scripts/curation/bakeoff/profile.py reads opt-in baseline profiles under # the repo's examples/ tree (path given explicitly via a job's # baselines_path) -- bake it in so a non-checkout deploy doesn't need a diff --git a/docker/hardened/triton/Dockerfile b/docker/hardened/triton/Dockerfile index 003c8cbcf..481e7abd5 100644 --- a/docker/hardened/triton/Dockerfile +++ b/docker/hardened/triton/Dockerfile @@ -18,8 +18,10 @@ # Build: docker build -f docker/hardened/triton/Dockerfile -t triton-hardened:26.06 . # Scan: trivy image --scanners vuln --severity CRITICAL,HIGH triton-hardened:26.06 +# Bump TRITON_TAG and TRITON_DIGEST together (docs/releases/RELEASE_CHECKLIST.md, base image digests). ARG TRITON_TAG=26.06-py3 -FROM nvcr.io/nvidia/tritonserver:${TRITON_TAG} +ARG TRITON_DIGEST=sha256:58df7489c3f2276f9591d500a012dee03e23d35543ce3c390b4c001e6bf90794 +FROM nvcr.io/nvidia/tritonserver:${TRITON_TAG}@${TRITON_DIGEST} # hadolint ignore=DL3008,DL3009,DL4006 RUN set -eux; \ diff --git a/docker/segmenter/Dockerfile b/docker/segmenter/Dockerfile index 3688d02e2..4d1b7f3e2 100644 --- a/docker/segmenter/Dockerfile +++ b/docker/segmenter/Dockerfile @@ -20,7 +20,7 @@ # Isolated so neither git nor the clone's history reaches the runtime # image. Pin to a known-good upstream SHA for reproducible rebuilds; # `main` is the default only because upstream publishes no releases. -FROM alpine/git:v2.54.0 AS sam3-src +FROM alpine/git:v2.54.0@sha256:ce84fd6a057b4dc5cf5009da2dd832f1c6d096c1984bf50496cf3c256402264d AS sam3-src ARG SAM3_SHA=main WORKDIR /src @@ -28,10 +28,42 @@ RUN git clone https://github.com/facebookresearch/sam3.git sam3 WORKDIR /src/sam3 RUN git checkout "$SAM3_SHA" && rm -rf .git -# -- Stage 2: runtime -------------------------------------------------------- +# -- Stage 2: build the Python environment ------------------------------------ +# Wheels resolve into a relocatable --user tree here; the runtime stage copies only that tree, so +# neither pip's cache nor the SAM 3 source checkout reaches the final image. gcc/g++ are present +# so a dependency without a wheel can still compile. +# This base is Ubuntu 24.04 with a PEP 668 system Python (older bases used conda), so pip +# refuses to install without PIP_BREAK_SYSTEM_PACKAGES. Its torch lives in the system +# site-packages, which a --user install resolves against. +FROM pytorch/pytorch:2.14.1-cuda12.6-cudnn9-runtime@sha256:1262abc856a428933c0477b4be87079173a0b087b7691c47fcb396b7b5775c56 AS builder + +ENV PIP_BREAK_SYSTEM_PACKAGES=1 + +# hadolint ignore=DL3008 +RUN apt-get update && apt-get install -y --no-install-recommends gcc g++ \ + && rm -rf /var/lib/apt/lists/* + +COPY --from=sam3-src /src/sam3 /opt/sam3 + +# hadolint ignore=DL3013 +RUN pip install --user --no-cache-dir --upgrade pip \ + && pip install --user --no-cache-dir --no-warn-script-location /opt/sam3 \ + && pip install --user --no-cache-dir --no-warn-script-location \ + 'fastapi==0.115.4' \ + 'uvicorn[standard]==0.32.0' \ + 'pydantic>=2.7,<3' \ + 'numpy>=1.26,<2.0' \ + 'pillow>=10.0' \ + 'opencv-python-headless>=4.10' \ + 'scikit-image>=0.24' \ + 'huggingface-hub>=0.24' \ + 'einops>=0.8' \ + 'pycocotools>=2.0.7' + +# -- Stage 3: runtime -------------------------------------------------------- # No -slim base is available for a CUDA + cuDNN + PyTorch runtime; this # is the smallest official image that carries the stack the model needs. -FROM pytorch/pytorch:2.14.1-cuda12.6-cudnn9-runtime +FROM pytorch/pytorch:2.14.1-cuda12.6-cudnn9-runtime@sha256:1262abc856a428933c0477b4be87079173a0b087b7691c47fcb396b7b5775c56 LABEL org.opencontainers.image.title="OpenProcessor Segmenter" \ org.opencontainers.image.description="Promptable segmentation (SAM 3) for the detection cascade" \ @@ -53,29 +85,6 @@ RUN apt-get update && apt-get upgrade -y \ && rm -rf /var/lib/apt/lists/* \ && apt-get clean -COPY --from=sam3-src /src/sam3 /opt/sam3 - -# This base is Ubuntu 24.04 with a PEP 668 system Python (older bases used -# conda), so pip refuses to install without this. The image is single-purpose -# and its torch lives in the system site-packages, so a venv would only add -# python3-venv (not in the base) and a PATH indirection. -ENV PIP_BREAK_SYSTEM_PACKAGES=1 - -# hadolint ignore=DL3013 -RUN pip install --no-cache-dir --upgrade pip \ - && pip install --no-cache-dir /opt/sam3 \ - && pip install --no-cache-dir \ - 'fastapi==0.115.4' \ - 'uvicorn[standard]==0.32.0' \ - 'pydantic>=2.7,<3' \ - 'numpy>=1.26,<2.0' \ - 'pillow>=10.0' \ - 'opencv-python-headless>=4.10' \ - 'scikit-image>=0.24' \ - 'huggingface-hub>=0.24' \ - 'einops>=0.8' \ - 'pycocotools>=2.0.7' - # Ubuntu 24.04 bases ship an `ubuntu` user at uid 1000; appuser must own that # uid to read the host-owned HF cache bind mount. No-op on bases without it. RUN if getent passwd ubuntu >/dev/null; then userdel -r ubuntu; fi && \ @@ -85,9 +94,11 @@ RUN if getent passwd ubuntu >/dev/null; then userdel -r ubuntu; fi && \ chown -R appuser:appuser /app /home/appuser/.cache WORKDIR /app +COPY --from=builder --chown=appuser:appuser /root/.local /home/appuser/.local COPY --chown=appuser:appuser main.py sam3_backend.py /app/ -ENV PYTHONUNBUFFERED=1 \ +ENV PATH=/home/appuser/.local/bin:$PATH \ + PYTHONUNBUFFERED=1 \ PYTHONDONTWRITEBYTECODE=1 \ SEGMENTER_DEVICE=cuda:0 \ SEGMENTER_LISTEN_PORT=8000 \ diff --git a/docker/test/Dockerfile.fake-vlm b/docker/test/Dockerfile.fake-vlm index 74d18860b..be215960d 100644 --- a/docker/test/Dockerfile.fake-vlm +++ b/docker/test/Dockerfile.fake-vlm @@ -3,12 +3,14 @@ # harness. Standard library only — no pip install, no network at build # time beyond the base image. # ============================================================================= -FROM python:3.12-slim +FROM python:3.12-slim@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1 WORKDIR /app COPY fake_vlm.py /app/fake_vlm.py +USER nobody + EXPOSE 8000 HEALTHCHECK --interval=5s --timeout=5s --start-period=5s --retries=10 \ diff --git a/docker/trainer/Dockerfile b/docker/trainer/Dockerfile index 1750c7612..729e8a82e 100644 --- a/docker/trainer/Dockerfile +++ b/docker/trainer/Dockerfile @@ -31,7 +31,7 @@ # ----------------------------------------------------------------------------- # Stage 1: build -- resolve every wheel into a relocatable --user tree # ----------------------------------------------------------------------------- -FROM python:3.12-slim-trixie AS builder +FROM python:3.12-slim-trixie@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1 AS builder WORKDIR /build @@ -67,7 +67,7 @@ RUN pip install --user --no-cache-dir --upgrade pip \ # ----------------------------------------------------------------------------- # Stage 2: runtime # ----------------------------------------------------------------------------- -FROM python:3.12-slim-trixie +FROM python:3.12-slim-trixie@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1 LABEL org.opencontainers.image.title="OpenProcessor Curation Trainer" \ org.opencontainers.image.description="YOLO training service driven by the curation job-file protocol" \ diff --git a/docs-site/Dockerfile b/docs-site/Dockerfile index 6fe011f15..7afa86ee7 100644 --- a/docs-site/Dockerfile +++ b/docs-site/Dockerfile @@ -1,5 +1,5 @@ # Multi-stage build: Node.js build -> nginx:alpine serve, non-root. -FROM node:26-alpine AS build +FROM node:26-alpine@sha256:6f08a523443ddedf52941874c203475b0ad503e859ad70cb26ab8be9a517dbd5 AS build WORKDIR /app COPY package*.json ./ @@ -10,7 +10,7 @@ RUN npm run build # nginx-unprivileged: listens on 8080 as a non-root user out of the box, # same base family as the other project images. -FROM nginxinc/nginx-unprivileged:1.31-alpine +FROM nginxinc/nginx-unprivileged:1.31-alpine@sha256:378d0c32f6562f7fd9629170797f57d2b953aa09be983b3daaf4809bb039ca00 LABEL org.opencontainers.image.title="OpenProcessor Documentation" \ org.opencontainers.image.description="Docusaurus documentation site for OpenProcessor" \ diff --git a/docs/releases/RELEASE_CHECKLIST.md b/docs/releases/RELEASE_CHECKLIST.md index 2daabcd86..a5ccfcf17 100644 --- a/docs/releases/RELEASE_CHECKLIST.md +++ b/docs/releases/RELEASE_CHECKLIST.md @@ -33,6 +33,25 @@ Cropwright is no longer released from its own repository or its own lock. There tarball; do not run `frontend/scripts/release/*` for 0.5.0. The `cropwright` tier is a compose profile of the main project and takes its image from `images.lock`. +## Base image digests: the bump procedure + +Every `FROM` in a tracked Dockerfile is `image:tag@sha256:`: the tag says what +was meant, the digest is what Docker pulls, so a rebuild never silently changes the base. +`tests/test_dockerfile_hygiene.py` fails on a `FROM` without a digest. `docker/hardened/triton` +pins through the `TRITON_TAG` and `TRITON_DIGEST` build args, which are bumped together. + +To bump a base (a Dependabot docker PR does the same edit): + +1. Look up the multi-arch index digest of the new tag: + `docker buildx imagetools inspect --format '{{json .Manifest.Digest}}'` + (or `docker manifest inspect ` and hash the index). Use the index digest, not a + per-platform one, so amd64 and arm64 builds both resolve. +2. Replace the tag and the digest together in every Dockerfile that names the image + (`rg 'python:3.13-slim-trixie@' --glob '*Dockerfile*'`), keeping the `AS ` suffix. +3. Rebuild the touched images (`docker compose -p build api`, and the side-cars), + run `hadolint` (2.15.1) and `tests/test_dockerfile_hygiene.py`, then scan the new images + with Trivy (`make release-dry-run`, step 3 of Steps) before they are published. + ## Release decisions (#44, #63): all DECIDED Carried forward from 0.4.1; nothing here changes for 0.5.0 unless noted. diff --git a/frontend/Dockerfile b/frontend/Dockerfile index f7a7e07af..e10199f76 100644 --- a/frontend/Dockerfile +++ b/frontend/Dockerfile @@ -1,7 +1,7 @@ # The bundle is architecture-independent static files, so build it once on # the native build platform; only the nginx stage below is per-arch. This # keeps a multi-arch (amd64 + arm64) build from running npm under emulation. -FROM --platform=$BUILDPLATFORM node:26-alpine AS builder +FROM --platform=$BUILDPLATFORM node:26-alpine@sha256:6f08a523443ddedf52941874c203475b0ad503e859ad70cb26ab8be9a517dbd5 AS builder WORKDIR /app COPY package*.json ./ RUN npm ci @@ -13,7 +13,7 @@ RUN npm run build # Runtime: nginx as a non-root user (uid 101) listening on 8080. # Pinned to an exact patch + Alpine tag (not a floating minor) so a base # rebuild never silently changes what's shipped; Dependabot bumps this. -FROM nginxinc/nginx-unprivileged:1.31.2-alpine3.23 +FROM nginxinc/nginx-unprivileged:1.31.2-alpine3.23@sha256:6320020c7da8714feab524e02c08c5a1958675c4e68700e93a2fd8970b065786 ARG VERSION=0.5.0 ARG REVISION=unknown diff --git a/tests/test_dockerfile_hygiene.py b/tests/test_dockerfile_hygiene.py new file mode 100644 index 000000000..506ac622d --- /dev/null +++ b/tests/test_dockerfile_hygiene.py @@ -0,0 +1,189 @@ +"""Dockerfile and build-context rules: pinned bases, non-root runtime, slim stages, and a +`COPY scripts/...` set that covers what the container actually imports.""" + +from __future__ import annotations + +import ast +import re +import subprocess +from pathlib import Path + +import pytest + + +REPO_ROOT = Path(__file__).resolve().parents[1] + +# `COPY` instructions with a trailing backslash continuation are joined before parsing. +_INSTRUCTION_RE = re.compile(r'^(FROM|COPY|USER|RUN)\s+(.*)$', re.M) +_PINNED_RE = re.compile(r'@sha256:[0-9a-f]{64}\b') + +# A final stage that sets no USER inherits one from its base: nginx-unprivileged runs as 101. +_USER_FROM_BASE = {'docs-site/Dockerfile': 'nginx-unprivileged sets USER 101'} + +# Images built from the repository root; a path under an ignored tree cannot be COPY-ed there. +_ROOT_CONTEXT = ( + 'Dockerfile', + 'Dockerfile.triton', + 'docker/evaluator/Dockerfile', + 'docker/trainer/Dockerfile', +) + + +def _dockerfiles() -> list[str]: + out = subprocess.run( # nosec B603 B607 - fixed git invocation + ['git', 'ls-files', '*Dockerfile*'], + cwd=REPO_ROOT, + check=True, + capture_output=True, + text=True, + ).stdout.split() + return sorted(out) + + +def _instructions(rel: str) -> list[tuple[str, str]]: + text = (REPO_ROOT / rel).read_text(encoding='utf-8').replace('\\\n', ' ') + return [(m.group(1), m.group(2)) for m in _INSTRUCTION_RE.finditer(text)] + + +def _stages(rel: str) -> list[list[tuple[str, str]]]: + stages: list[list[tuple[str, str]]] = [] + for kind, rest in _instructions(rel): + if kind == 'FROM': + stages.append([]) + stages[-1].append((kind, rest)) + return stages + + +@pytest.mark.parametrize('rel', _dockerfiles()) +def test_every_base_image_is_pinned_by_digest(rel: str) -> None: + stage_names = set() + for kind, rest in _instructions(rel): + if kind != 'FROM': + continue + words = [w for w in rest.split() if not w.startswith('--')] + ref = words[0] + if ref in stage_names or ref == 'scratch': + pass + else: + # An ARG-built reference is pinned by a sha256 ARG default (hardened triton). + expanded = ref + for name, default in re.findall( + r'^ARG (\w+)=(\S+)$', (REPO_ROOT / rel).read_text(), re.M + ): + expanded = expanded.replace('${' + name + '}', default) + assert _PINNED_RE.search(expanded), f'{rel}: FROM {ref} is not pinned by digest' + if len(words) == 3 and words[1].upper() == 'AS': + stage_names.add(words[2]) + + +@pytest.mark.parametrize('rel', [r for r in _dockerfiles() if r not in _USER_FROM_BASE]) +def test_final_stage_runs_as_a_non_root_user(rel: str) -> None: + users = [rest.split()[0] for kind, rest in _stages(rel)[-1] if kind == 'USER'] + assert users, f'{rel}: the final stage never sets USER' + assert users[-1] not in {'root', '0'}, f'{rel}: the final stage ends as root' + + +@pytest.mark.parametrize('rel', ['docker/evaluator/Dockerfile', 'docker/segmenter/Dockerfile']) +def test_image_builds_dependencies_in_a_separate_stage(rel: str) -> None: + stages = _stages(rel) + assert len(stages) >= 3 if 'segmenter' in rel else len(stages) >= 2 + installs = [rest for kind, rest in stages[-1] if kind == 'RUN' and 'pip install' in rest] + assert installs == [], f'{rel}: the runtime stage runs pip install' + + +def test_root_dockerignore_excludes_trees_no_build_needs() -> None: + ignored = { + ln.strip().rstrip('/') + for ln in (REPO_ROOT / '.dockerignore').read_text().splitlines() + if ln.strip() and not ln.startswith(('#', '!')) + } + for entry in ('artifacts_local', 'docs-site', 'frontend', '.claude', '.install', 'export_*'): + assert entry in ignored, f'.dockerignore does not exclude {entry}/' + for rel in _ROOT_CONTEXT: + for kind, rest in _instructions(rel): + if kind != 'COPY' or '--from=' in rest: + continue + for src in [w for w in rest.split() if not w.startswith('--')][:-1]: + assert src.split('/')[0] not in ignored, f'{rel}: COPY {src} is dockerignored' + + +def _copied_scripts(rel: str) -> tuple[set[str], set[str]]: + """(directories, files) under scripts/ that the Dockerfile copies into the image.""" + dirs: set[str] = set() + files: set[str] = set() + for kind, rest in _instructions(rel): + if kind != 'COPY' or '--from=' in rest: + continue + words = [w for w in rest.split() if not w.startswith('--')] + for src in words[:-1]: + if src == 'scripts/': + dirs.add('scripts') + elif src.startswith('scripts/') and src.endswith('/'): + dirs.add(src.rstrip('/')) + elif src.startswith('scripts/'): + files.add(src) + return dirs, files + + +def _covered(path: str, dirs: set[str], files: set[str]) -> bool: + return path in files or any(path == d or path.startswith(d + '/') for d in dirs) + + +def _scripts_modules_imported(roots: list[str]) -> set[str]: + """Repo-relative files under scripts/ reachable by import from the given trees.""" + seen: set[Path] = set() + todo = [p for r in roots for p in (REPO_ROOT / r).rglob('*.py')] if roots else [] + needed: set[str] = set() + while todo: + path = todo.pop() + if path in seen: + continue + seen.add(path) + for node in ast.walk(ast.parse(path.read_text(encoding='utf-8'))): + names: list[str] = [] + if isinstance(node, ast.ImportFrom) and node.module and node.level == 0: + names = [node.module, *[f'{node.module}.{a.name}' for a in node.names]] + elif isinstance(node, ast.Import): + names = [a.name for a in node.names] + for name in names: + if name.split('.')[0] != 'scripts': + continue + base = REPO_ROOT / name.replace('.', '/') + for cand in (base.with_suffix('.py'), base / '__init__.py'): + if cand.is_file(): + needed.add(str(cand.relative_to(REPO_ROOT))) + todo.append(cand) + return needed + + +def test_api_image_copies_every_scripts_file_it_imports_or_sources() -> None: + dirs, files = _copied_scripts('Dockerfile') + assert 'scripts' not in dirs, 'the api image must copy a runtime subset of scripts/' + roots = ['src', 'export', 'models', 'config_templates', 'scripts/curation'] + needed = _scripts_modules_imported(roots) + # `openprocessor models install` and `sample coco` run these inside the container + # (scripts/lib/model_setup.sh). + needed |= _scripts_modules_imported(['scripts/datasets']) + needed |= { + 'scripts/datasets/fetch_coco_subset.py', + 'scripts/datasets/manifests/coco_va_200.json', + 'scripts/datasets/manifests/coco_va_800.json', + 'scripts/lib/config.sh', + 'scripts/lib/download.sh', + } + for lib in sorted(needed): + if lib.endswith('.sh'): + text = (REPO_ROOT / lib).read_text() + needed |= { + f'scripts/lib/{m}' for m in re.findall(r'source "\$\{\w+\}/(\w+\.sh)"', text) + } + missing = sorted(n for n in needed if not _covered(n, dirs, files)) + assert missing == [], f'Dockerfile does not COPY: {missing}' + + +def test_evaluator_image_copies_every_scripts_file_it_imports() -> None: + dirs, files = _copied_scripts('docker/evaluator/Dockerfile') + assert 'scripts' not in dirs + needed = _scripts_modules_imported(['src', 'scripts/curation/bakeoff']) + missing = sorted(n for n in needed if not _covered(n, dirs, files)) + assert missing == [], f'evaluator Dockerfile does not COPY: {missing}' From c9bffcfed6d0494764538b6e421605cb3f5b0a2d Mon Sep 17 00:00:00 2001 From: davidamacey Date: Wed, 14 Oct 2026 18:43:07 -0400 Subject: [PATCH 3/8] refactor(compose): share the root Dockerfile build across the api and workers One YAML anchor replaces five identical build blocks; docker compose config renders byte-identically for every file and profile combination the Makefile and CLI use. --- docker-compose.dev.yml | 27 ++++++--------------------- 1 file changed, 6 insertions(+), 21 deletions(-) diff --git a/docker-compose.dev.yml b/docker-compose.dev.yml index cc46c40be..0b3a14681 100644 --- a/docker-compose.dev.yml +++ b/docker-compose.dev.yml @@ -34,7 +34,8 @@ services: dockerfile: Dockerfile.triton api: - build: + # The root Dockerfile build, shared by the curation workers below. + build: &root-build context: . dockerfile: Dockerfile args: @@ -65,43 +66,27 @@ services: image: ${OP_CROPWRIGHT_IMAGE:-openprocessor-cropwright:dev} curation-detection-worker: - build: - context: . - dockerfile: Dockerfile - args: - OP_BUILD_SHA: "${OP_BUILD_SHA:-}" + build: *root-build volumes: - ./src:/app/src:ro - ./scripts:/app/scripts:ro - ./examples:/app/examples:ro curation-vlm-worker: - build: - context: . - dockerfile: Dockerfile - args: - OP_BUILD_SHA: "${OP_BUILD_SHA:-}" + build: *root-build volumes: - ./src:/app/src:ro - ./scripts:/app/scripts:ro curation-auto-label-worker: - build: - context: . - dockerfile: Dockerfile - args: - OP_BUILD_SHA: "${OP_BUILD_SHA:-}" + build: *root-build volumes: - ./src:/app/src:ro - ./scripts:/app/scripts:ro - ./examples:/app/examples:ro curation-cluster-refresh: - build: - context: . - dockerfile: Dockerfile - args: - OP_BUILD_SHA: "${OP_BUILD_SHA:-}" + build: *root-build volumes: - ./src:/app/src:ro - ./scripts:/app/scripts:ro From 2623fab23eb6028426efb959af056711e71287e1 Mon Sep 17 00:00:00 2001 From: davidamacey Date: Wed, 14 Oct 2026 18:43:23 -0400 Subject: [PATCH 4/8] docs(changelog): note the env rename, Dockerfile pins and the docs check --- CHANGELOG.md | 19 +++++++++++++++++++ frontend/CHANGELOG.md | 6 ++++++ 2 files changed, 25 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index a194697e6..000504eff 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -139,6 +139,25 @@ history of this codebase and was never published. This release is `[0.5.0]`. ### Changed +- **BREAKING**: the Cropwright variables are now `OP_CROPWRIGHT_*`: `CROPWRIGHT_IMAGE`, + `CROPWRIGHT_BIND_ADDRESS`, `CROPWRIGHT_PORT`, `CROPWRIGHT_INGEST_MAX_REQUEST_MB` and + `CROPWRIGHT_DATASET_UPLOAD_MAX_MB` became `OP_CROPWRIGHT_IMAGE`, `OP_CROPWRIGHT_BIND_ADDRESS`, + `OP_CROPWRIGHT_PORT`, `OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB` and + `OP_CROPWRIGHT_DATASET_UPLOAD_MAX_MB` (the live-test and docs-screenshot helpers moved the same + way). A `.env` that still sets an old name is refused: `./openprocessor` and + `setup-openprocessor.sh` exit 2 and print the new name, and the cropwright container exits + at start. Rename the keys in `.env` and run `./openprocessor restart`. +- Every Dockerfile base image is pinned by digest (`image:tag@sha256:...`); the bump procedure is + in `docs/releases/RELEASE_CHECKLIST.md`. The evaluator and segmenter images build their Python + environment in a separate stage (no compilers or pip cache in the evaluator runtime; the + segmenter keeps gcc for `torch.compile`), the fake VLM test image runs as `nobody`, and the + api image copies only the `scripts/` it runs (`scripts/curation`, `scripts/datasets`, six shell + libraries) instead of the whole tree. The root `.dockerignore` also excludes `artifacts_local/`, + `docs-site/`, `frontend/`, `.claude/`, `.install/` and `export_*/`. +- `check_docs_vs_code.py` requires an assignment line in `env.template` for every compose + `${VAR}` (a prose mention no longer counts) and gains a `template` check: a non-`OP_` setting + that no compose file, script or code reads is rejected. The unread `LOG_LEVEL` and the unread + `OP_CROPWRIGHT_TAG` documentation are removed. - Repo cleanup, PR 7: the oversize export scripts, the trainer and OCR model modules, the shell programs (`openprocessor`, `scripts/setup.sh`, `scripts/lib/config.sh`, `export.sh`, `export_paddleocr.sh`, `acceptance_run.sh`) and 19 oversize test files are split by concern into diff --git a/frontend/CHANGELOG.md b/frontend/CHANGELOG.md index 5e373514a..2fe49dbdf 100644 --- a/frontend/CHANGELOG.md +++ b/frontend/CHANGELOG.md @@ -22,6 +22,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- **BREAKING**: the container reads `OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB` and + `OP_CROPWRIGHT_DATASET_UPLOAD_MAX_MB` (were `CROPWRIGHT_*`), and `docker-entrypoint.sh` exits 2 at + start while a retired `CROPWRIGHT_*` name is still set. The browser-side `window.__CROPWRIGHT_*__` + globals are unchanged. `.env.example` and the README drop the container-name variables only the + removed standalone compose used. + - `ProjectCounts.images` and `.items` are `number | null` (the API serves `null` when OpenSearch could not be counted); the project list already renders `null` as an em dash. - Work-package ids and history names are gone from comments, JSDoc and test titles in `src/` and `e2e/`, From 83d9198a5fa56715525ae1831665016ab1fa9ccd Mon Sep 17 00:00:00 2001 From: davidamacey Date: Wed, 14 Oct 2026 18:59:52 -0400 Subject: [PATCH 5/8] fix(docs): read the retired env names from the shell library --- scripts/docs/check_docs_vs_code.py | 6 +++--- tests/test_docs_vs_code.py | 3 ++- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/scripts/docs/check_docs_vs_code.py b/scripts/docs/check_docs_vs_code.py index 487b0df4c..7c6aab4ca 100755 --- a/scripts/docs/check_docs_vs_code.py +++ b/scripts/docs/check_docs_vs_code.py @@ -10,7 +10,7 @@ compose files or the installer scripts. * compose - every ``${VAR}`` a root compose file reads is assigned in ``env.template`` (commented counts, a prose mention does not), so an operator can find every knob. - Names the installer refuses (``RETIRED_ENV_NAMES``) are exempt. + Names the installer and CLI refuse (``RETIRED_ENV_NAMES``) are exempt. * template - every non-``OP_`` setting ``env.template`` assigns is read by a compose file, script or code, so nothing advertised does nothing. * links - every relative link resolves to a file, and every ``#anchor`` that @@ -241,8 +241,8 @@ def _template_names() -> set[str]: def _retired_env_names() -> set[str]: - """Names the installer refuses in a .env (renamed); compose may still pass them through.""" - script = REPO_ROOT / 'setup-openprocessor.sh' + """Names the installer and CLI refuse in a .env (renamed); compose may still pass them through.""" + script = REPO_ROOT / 'scripts' / 'lib' / 'retired_env.sh' if not script.is_file(): return set() block = _RETIRED_BLOCK_RE.search(script.read_text(encoding='utf-8', errors='replace')) diff --git a/tests/test_docs_vs_code.py b/tests/test_docs_vs_code.py index 7ddc30fd5..37581111e 100644 --- a/tests/test_docs_vs_code.py +++ b/tests/test_docs_vs_code.py @@ -108,7 +108,8 @@ def test_compose_check_skips_retired_names_the_installer_refuses( 'services:\n a:\n environment:\n - OLD_NAME=${OLD_NAME:-}\n', encoding='utf-8' ) (tmp_path / 'env.template').write_text('# NEW_NAME=1\n', encoding='utf-8') - (tmp_path / 'setup-openprocessor.sh').write_text( + (tmp_path / 'scripts' / 'lib').mkdir(parents=True) + (tmp_path / 'scripts' / 'lib' / 'retired_env.sh').write_text( 'RETIRED_ENV_NAMES=(\n OLD_NAME\n)\n', encoding='utf-8' ) monkeypatch.setattr(checker, 'REPO_ROOT', tmp_path) From 4fc676b1c3748340e38cdf083a69921ae4e5331e Mon Sep 17 00:00:00 2001 From: davidamacey Date: Wed, 14 Oct 2026 19:05:19 -0400 Subject: [PATCH 6/8] docs(frontend): use the renamed live-test URL variable --- frontend/CLAUDE.md | 4 ++-- frontend/README.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/frontend/CLAUDE.md b/frontend/CLAUDE.md index f3c04fa64..ea3f6949d 100644 --- a/frontend/CLAUDE.md +++ b/frontend/CLAUDE.md @@ -62,7 +62,7 @@ npm run lint # prettier --check + eslint npx vitest run # unit and component tests (jsdom) npm run build # static build to build/ npm run test:e2e # stubbed Playwright suite (own venv at e2e/.venv, auto-provisioned) -npm run test:live # live read-only tier; needs CROPWRIGHT_LIVE_URL, skipped otherwise +npm run test:live # live read-only tier; needs OP_CROPWRIGHT_LIVE_URL, skipped otherwise npm run test:mutation # Stryker, 15-20 minutes ``` @@ -254,7 +254,7 @@ Under `src/lib/contract/`: `artifacts_local/e2e-failures//`; read those first when a test flakes. - A `py_compile` hook and CI step gate `scripts/*.py` and `e2e/**/*.py` syntax. - **Live read-only tier (`e2e/live/`):** drives the real deployed build against a real - backend (`CROPWRIGHT_LIVE_URL=... npm run test:live`; skipped when unset, never in CI). + backend (`OP_CROPWRIGHT_LIVE_URL=... npm run test:live`; skipped when unset, never in CI). Structurally read-only: the `guarded_page` fixture aborts any non-GET/HEAD request under the API prefix and its teardown fails if one was attempted. It is project-aware (`live_project` resolves the served default slug; reads go through diff --git a/frontend/README.md b/frontend/README.md index 12cdc7be0..ba832a43a 100644 --- a/frontend/README.md +++ b/frontend/README.md @@ -188,7 +188,7 @@ npm run test:e2e # Playwright against an in-browser stub backend (own venv, auto ``` **Live, read-only tests** against a real deployed backend: -`CROPWRIGHT_LIVE_URL=http://localhost:5184 npm run test:live` — skipped +`OP_CROPWRIGHT_LIVE_URL=http://localhost:5184 npm run test:live` — skipped entirely unless that env var is set; structurally read-only (aborts any non-GET/HEAD request to the API and fails the test if one was even attempted). From b9f6fc504c2a6cd268218ecab7f00b8b39254a09 Mon Sep 17 00:00:00 2001 From: davidamacey Date: Wed, 14 Oct 2026 19:11:10 -0400 Subject: [PATCH 7/8] fix(frontend): print the renamed OP_CROPWRIGHT_ upload-cap variable in the 413 message --- frontend/src/app.html | 4 ++-- frontend/src/lib/datasets/uploadCap.ts | 2 +- frontend/src/lib/ingest/ingestConfig.ts | 2 +- frontend/src/lib/ingest/ingestRunController.svelte.ts | 2 +- frontend/src/lib/ingest/ingestRunController.test.ts | 2 +- 5 files changed, 6 insertions(+), 6 deletions(-) diff --git a/frontend/src/app.html b/frontend/src/app.html index 45dc60293..d56a537cc 100644 --- a/frontend/src/app.html +++ b/frontend/src/app.html @@ -8,7 +8,7 @@ Cropwright diff --git a/frontend/src/lib/datasets/uploadCap.ts b/frontend/src/lib/datasets/uploadCap.ts index c9050d737..54fd3fc6a 100644 --- a/frontend/src/lib/datasets/uploadCap.ts +++ b/frontend/src/lib/datasets/uploadCap.ts @@ -2,7 +2,7 @@ * The archive-upload size the client allows: the lower of the served * `GET /datasets/formats` `upload.max_bytes` and this deployment's own * nginx `client_max_body_size` for `/datasets/uploads` - * (`CROPWRIGHT_DATASET_UPLOAD_MAX_MB`, substituted into + * (`OP_CROPWRIGHT_DATASET_UPLOAD_MAX_MB`, substituted into * `window.__CROPWRIGHT_DATASET_UPLOAD_MAX_MB__` by docker-entrypoint.sh). * A file above either one would be refused anyway, after a long upload. */ diff --git a/frontend/src/lib/ingest/ingestConfig.ts b/frontend/src/lib/ingest/ingestConfig.ts index c0ea9a58e..45df186b5 100644 --- a/frontend/src/lib/ingest/ingestConfig.ts +++ b/frontend/src/lib/ingest/ingestConfig.ts @@ -21,7 +21,7 @@ export const PATH_LOOKUP_MAX = 10_000; /** * Deployment-owned (not a backend value at all — see A.4): the nginx * body-size cap this Cropwright deployment was built/started with. - * `CROPWRIGHT_INGEST_MAX_REQUEST_MB` (docker-entrypoint.sh) substitutes + * `OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB` (docker-entrypoint.sh) substitutes * the runtime value into `window.__CROPWRIGHT_INGEST_MAX_REQUEST_MB__`; * unset (local dev, or the literal unresolved placeholder) falls back * to nginx.conf's default. diff --git a/frontend/src/lib/ingest/ingestRunController.svelte.ts b/frontend/src/lib/ingest/ingestRunController.svelte.ts index 612a383a1..8af9b2cf8 100644 --- a/frontend/src/lib/ingest/ingestRunController.svelte.ts +++ b/frontend/src/lib/ingest/ingestRunController.svelte.ts @@ -328,7 +328,7 @@ export function createIngestRun(deps: IngestRunDeps): IngestRun { state = 'error'; errorReason = `Upload rejected by the Cropwright proxy — request larger than ` + - `CROPWRIGHT_INGEST_MAX_REQUEST_MB`; + `OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB`; cancelRequested = true; toastStore.error(errorReason); markChunk(chunk, 'not_sent'); diff --git a/frontend/src/lib/ingest/ingestRunController.test.ts b/frontend/src/lib/ingest/ingestRunController.test.ts index db044c825..fecb3d58a 100644 --- a/frontend/src/lib/ingest/ingestRunController.test.ts +++ b/frontend/src/lib/ingest/ingestRunController.test.ts @@ -547,7 +547,7 @@ describe('createIngestRun — response handling', () => { skipLookup: true, }); expect(run.state).toBe('error'); - expect(run.errorReason).toMatch(/CROPWRIGHT_INGEST_MAX_REQUEST_MB/); + expect(run.errorReason).toMatch(/OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB/); expect(upload).toHaveBeenCalledTimes(1); }); From 2ed277b4a64663a54ca86e35b5bf56a8c0141a38 Mon Sep 17 00:00:00 2001 From: davidamacey Date: Wed, 14 Oct 2026 19:13:06 -0400 Subject: [PATCH 8/8] style(frontend): format the README env table --- frontend/README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/frontend/README.md b/frontend/README.md index ba832a43a..1cef445dd 100644 --- a/frontend/README.md +++ b/frontend/README.md @@ -158,10 +158,10 @@ key a registered region slot's own keymap declares. | `PUBLIC_API_PREFIX` | `/curation` | both | Path prefix the backend serves its curation endpoints under. Must equal the backend's own `OP_API_PREFIX`. | | `PUBLIC_TRITON_API_URL` | _(empty)_ | `npm run dev` | Legacy name (predates the OpenProcessor rename). Points the dev server straight at a backend origin, bypassing the nginx proxy Docker gets for free. Leave empty in Docker. | | `API_UPSTREAM` | `http://op-api:8000` | Docker | Where nginx proxies `PUBLIC_API_PREFIX/*` to, by container name over the shared docker network. | -| `OP_CROPWRIGHT_PORT` | `5184` | Docker | Host port nginx is published on. | -| `OP_CROPWRIGHT_BIND_ADDRESS` | `0.0.0.0` | Docker | Host address the port is published on: this machine and the local network. `127.0.0.1` limits it to this machine. Keep it on a trusted network. | -| `OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB` | `256` | Docker | Upload cap for `/ingest` (nginx `client_max_body_size`); kept in lockstep with the client-side chunk planner. | -| `OP_CROPWRIGHT_DATASET_UPLOAD_MAX_MB` | `2048` | Docker | Upload cap for a dataset archive on `/datasets/import` (nginx `client_max_body_size`); the client refuses a larger file before sending. | +| `OP_CROPWRIGHT_PORT` | `5184` | Docker | Host port nginx is published on. | +| `OP_CROPWRIGHT_BIND_ADDRESS` | `0.0.0.0` | Docker | Host address the port is published on: this machine and the local network. `127.0.0.1` limits it to this machine. Keep it on a trusted network. | +| `OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB` | `256` | Docker | Upload cap for `/ingest` (nginx `client_max_body_size`); kept in lockstep with the client-side chunk planner. | +| `OP_CROPWRIGHT_DATASET_UPLOAD_MAX_MB` | `2048` | Docker | Upload cap for a dataset archive on `/datasets/import` (nginx `client_max_body_size`); the client refuses a larger file before sending. | | `PUBLIC_APP_NAME` / `PUBLIC_APP_BADGE` | `Cropwright` / `CW` | build time | Top-bar wordmark/badge, for a white-labeled deployment. | The app is a pure SPA consumer of the OpenProcessor API — there is @@ -213,7 +213,7 @@ repository root builds and publishes it (see `docs/releases/RELEASE_CHECKLIST.md | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | Top-bar status chip red / "API unavailable" | Backend not reachable at `API_UPSTREAM` (Docker) or `PUBLIC_TRITON_API_URL` (dev); or `{API_PREFIX}/health` returns 5xx | | A write returns 409 "no region profile is configured" | The backend has no region profile set — region-specific actions (confirm/reject a sub-box, etc.) are unavailable until it does | -| Upload to `/ingest` fails with 413 | File(s) exceed `OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB` (nginx) or the backend's own configured upload cap — raise the env var, rebuild | +| Upload to `/ingest` fails with 413 | File(s) exceed `OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB` (nginx) or the backend's own configured upload cap — raise the env var, rebuild | | Empty cluster grid / review queues | The backend's index isn't populated yet — ingest some images first | | Build fails: `Cannot find module 'svelte-dnd-action'` | `npm install` wasn't run, or `node_modules` is stale — `rm -rf node_modules && npm install` |