Skip to content

chore(infra)!: rename CROPWRIGHT_* env to OP_CROPWRIGHT_*, pin Docker bases by digest, trim build contexts, env.template checks - #314

Merged
davidamacey merged 11 commits into
mainfrom
chore/cleanup-infra
Oct 15, 2026
Merged

davidamacey merged 11 commits into
mainfrom
chore/cleanup-infra

Conversation

@davidamacey

Copy link
Copy Markdown
Owner

chore/cleanup-infra: env rename, env.template checks, Dockerfile pins and runtime image content (PR 5c of 9 of the repo cleanup)

Plan: docs/design/repo_cleanup_plan.md section 7 (F3 env part, F5), decisions D25, D31, D32.
No src/ logic, no frontend/src, no file splits. BREAKING: the Cropwright env names changed.

Commits

Commit Content
feat(env)! D25: CROPWRIGHT_{IMAGE,BIND_ADDRESS,PORT,INGEST_MAX_REQUEST_MB,DATASET_UPLOAD_MAX_MB} are OP_CROPWRIGHT_* in compose, env.template, the installer, frontend/docker-entrypoint.sh, tests and docs (live-test and docs-screenshot helper vars too). scripts/lib/retired_env.sh holds the retired list: ./openprocessor (every command except help/version) and setup-openprocessor.sh (after the release is installed, before .env is written) exit 2 with CROPWRIGHT_PORT was renamed to OP_CROPWRIGHT_PORT: update <file>; compose passes the retired names through to the cropwright container whose entrypoint exits 2. Docs check: assignment line required in env.template, new template check, LOG_LEVEL (read by nothing) removed, docs-only OP_CROPWRIGHT_TAG removed, container-name variables dropped from frontend/.env.example and README
build(docker) D31 + D32 (below)
refactor(compose) one YAML anchor for the five identical root build: blocks in docker-compose.dev.yml
docs(changelog), fix(docs) CHANGELOG (BREAKING note), frontend/CHANGELOG.md; the checker reads the retired list from the lib

Failing first

tests/installer/test_retired_cropwright_env.py   assert 0 == 2   (installer exited 0 on a .env setting CROPWRIGHT_BIND_ADDRESS)
                                                 CLI case: 'CROPWRIGHT_PORT was renamed to ...' in ''  (status ran, no refusal)
tests/test_docs_vs_code.py                        6 failed: compose prose-mention accepted, retired names not exempt,
                                                 check_template_names missing, real-repo compose/template checks
tests/test_dockerfile_hygiene.py (vs the old Dockerfiles)  15 failed, 7 passed:
    9x FROM not pinned by digest; fake-vlm final stage never sets USER; evaluator and segmenter run pip install in the
    runtime stage; .dockerignore lacks artifacts_local/docs-site/frontend/...; Dockerfile and evaluator COPY all of scripts/

The entrypoint refusal test was written together with the entrypoint change (not seen failing alone).
After: tests/installer, tests/compose, docs checks and the hygiene tests pass; full offline suite
8250 passed, 19 skipped in 617 s (-n 6) with one failure that I fixed afterwards in fix(docs)
(test_real_docs_match_the_code[compose]: the checker still read the retired list from the installer after I moved it to the lib;
15 passed after). pre-commit run --all-files green (34 hooks).

Plant-a-violation proofs

Guard Plant Result
compose check ${PLANTED_VAR} in docker-compose.yml, and # set PLANTED_VAR somewhere (prose) in env.template docker-compose.yml: ${PLANTED_VAR} is not in env.template (prose does not satisfy it); removed, 0 problems
template check # PLANTED_SETTING=1 in env.template env.template: PLANTED_SETTING is set by no compose file, script or code; removed, 0 problems
digest pin FROM python:3.13-slim-trixie AS builder in Dockerfile Dockerfile: FROM python:3.13-slim-trixie is not pinned by digest; restored
scripts COPY deleted COPY scripts/datasets/ Dockerfile does not COPY: ['scripts/datasets/_common.py', 'scripts/datasets/bench_set.py']; restored

D31 / D32 details

  • Digests are multi-arch index digests (docker buildx imagetools inspect), written image:tag@sha256:... (the tag stays in
    the reference, which Dependabot also updates). docker/hardened/triton pins through TRITON_TAG + new TRITON_DIGEST.
    Bump procedure: new section in docs/releases/RELEASE_CHECKLIST.md. hadolint 2.15.1: no new findings (info DL3066 as before).
  • docker/evaluator: two stages (builder with build-essential/cmake/git; runtime with only the three shared libs, no git/curl;
    _code_sha() already returns unknown without git). docker/segmenter: three stages (git clone, pip builder, runtime);
    runtime keeps gcc/g++ for torch.compile, drops the /opt/sam3 source copy and the pip cache. Dockerfile.fake-vlm: USER nobody.
  • Root .dockerignore: artifacts_local/ docs-site/ frontend/ .claude/ .install/ export_*/. Checked first: no root-context
    Dockerfile or compose build COPYs from them (frontend and docs-site build from their own contexts; export/ is untouched).
  • Runtime scripts/ set, derived by AST import closure over src export models config_templates scripts/curation plus the
    commands scripts/lib/model_setup.sh runs in the container: scripts/curation/, scripts/datasets/ (fetch_coco_subset, manifests),
    and six shell libs (config.sh and what it sources, download.sh). Evaluator: scripts/curation/ only.
    tests/test_dockerfile_hygiene.py recomputes the closure, so a new import outside the set fails CI.

Measured (built with -p cleanup-infra, tags cleanup-infra-local/*:{before,after}, never the live project):

Image before after
api (build api) 21,214,992,927 B 21,171,892,377 B (-43 MB; /app/scripts 4 MB to 3 MB, rest is the newer pinned base)
evaluator 14,822,214,566 B 14,352,074,113 B (-470 MB)
segmenter 8,240,867,291 B 8,162,332,302 B (-78 MB)

Smoke in the built images: import of scripts.curation.worker, .worker.__main__, .vlm_worker, .auto_label_worker,
.bakeoff.bakeoff_runner, scripts.datasets.fetch_coco_subset, src.main: all ok; python -m scripts.curation.worker --help and
scripts/curation/cluster_refresh_daemon.py --help print usage; source /app/scripts/lib/config.sh and download.sh work as appuser;
/app/scripts is appuser-owned and writable. Evaluator: imports mlflow, onnxruntime, ultralytics, tritonclient, pycocotools and the
runner, --help prints usage. Segmenter: sam3 (with bpe_simple_vocab_16e6.txt.gz in site-packages), sam3_backend, runs as uid 1000,
gcc present. The live write-path harness was not run (needs GPUs 0/2 and the test stack).

Compose rendering

docker compose -p cleanup-infra config before/after the anchor commit is byte-identical for: base, base + all profiles, dev,
dev + all profiles, the Makefile DEV_COMPOSE (dev + gpu-arbiter + all profiles) and base + docs. No dead services, profiles, volumes or
build args found (every named volume and network is used; OP_BUILD_SHA is consumed by all seven builds).

Findings, not fixed here

  • docker-compose.yml + docker-compose.gpu-clustering.yml (make cluster-gpu) does not render on Compose 2.29.7 on main either:
    services.curation-auto-label-worker: can't set distinct values on 'mem_reservation' and 'deploy.resources.reservations.memory'.
    Needs its own issue (drop mem_reservation for deploy.resources.reservations.memory or the reverse).
  • Compose cannot refuse a set variable natively (${A:+${B:?}} evaluates the inner expression eagerly), so refusal at compose level is the
    cropwright entrypoint; a raw docker compose up with an old name starts the container which then exits 2 and restarts. The CLI and installer refuse before compose.
  • frontend/src/lib/ingest/ingestRunController.svelte.ts:331 and its test still print CROPWRIGHT_INGEST_MAX_REQUEST_MB in a user-facing message
    (outside this PR's scope); it should say OP_CROPWRIGHT_INGEST_MAX_REQUEST_MB. The window.__CROPWRIGHT_*__ globals are unchanged.
  • setup-openprocessor.sh stays at its 3214-line baseline: the refusal logic is in the new lib and only the source plus call are added; the
    7-line comment on choose_cropwright_bind was condensed to 4 to make room.

Conflicts to expect

PR 5a (#308) edits frontend/.env.example, frontend/README.md, frontend/CLAUDE.md and scripts/docs/check_docs_vs_code.py (drops the
frontend/docker-compose*.yml corpus entry; I added frontend/e2e and frontend/scripts roots next to it). I did not touch files 5a deletes
(frontend/docker-compose.yml, frontend/scripts/release/*). Resolve by keeping both sides. Not rebased onto 5a/PR 4.

… names

BREAKING: CROPWRIGHT_IMAGE, _BIND_ADDRESS, _PORT, _INGEST_MAX_REQUEST_MB and
_DATASET_UPLOAD_MAX_MB are now OP_CROPWRIGHT_*. The deploy CLI and the
installer exit 2 naming the new variable when a .env still sets an old name;
the cropwright container entrypoint refuses one passed through by compose.
The remaining names (live-test URL/project, docs screenshot URL) are renamed
the same way. The unread OP_CROPWRIGHT_TAG and the container-name variables
only the deleted standalone compose used are dropped from the docs.

The docs check now requires an assignment line in env.template for every
compose ${VAR} (a prose mention no longer counts), and a new template check
rejects a non-OP_ setting nothing reads; the unread LOG_LEVEL is removed.
…me scripts set

Every FROM is image:tag@sha256 (bump procedure in the release checklist).
The evaluator and segmenter build their Python environment in a separate
stage, the fake VLM test image runs as nobody, and the api image copies
scripts/curation, scripts/datasets and the six shell libraries it runs in
the container instead of all of scripts/ (the evaluator copies only
scripts/curation). The root .dockerignore excludes artifacts_local, docs-site,
frontend, .claude, .install and export_*. tests/test_dockerfile_hygiene.py
checks the pins, the non-root user, the stages and that the copied scripts
cover every import.
… workers

One YAML anchor replaces five identical build blocks; docker compose config
renders byte-identically for every file and profile combination the Makefile
and CLI use.
# Conflicts:
#	frontend/CHANGELOG.md
#	frontend/CLAUDE.md
#	frontend/README.md
# Conflicts:
#	frontend/CHANGELOG.md
# Conflicts:
#	CHANGELOG.md
@davidamacey
davidamacey merged commit 47bee9d into main Oct 15, 2026
11 checks passed
@davidamacey
davidamacey deleted the chore/cleanup-infra branch October 15, 2026 00:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant