diff --git a/CHANGELOG.md b/CHANGELOG.md index fd003ca00..91768d298 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,56 +7,98 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 Releases 0.2.0, 0.2.1 and 0.3.0 are published on GitHub; their entries below are the content of those tags. The entry headed `Pre-release 0.1.0` is the earliest private -history of this codebase and was never published. This release is `[0.5.0]`. +history of this codebase and was never published. This release is `[0.6.0]`. ## [Unreleased] -### Fixed -- Ingest no longer fails a JPEG that Pillow reads but libjpeg-turbo's header read refuses (`tjDecompressHeader3(): Could not determine subsampling level`, 2 of 100,000 Open Images files, `decode_failed`): that error alone sends the image to the Pillow decode; a damaged file still fails. -- Dispatcher-started bulk-ingest runs (poll, cron, webhook) never started: the tick ran in a worker thread and both the run and the poll called `asyncio.create_task` there (`ingest_dispatch_failed`, `no running event loop`). The tick now runs on the loop. A run whose spawn fails is marked `failed` instead of staying `queued` (which kept every later start, and the project's delete, answering 409), and `POST /ingest/bulk/{run_id}/cancel` finalizes an orphan (a run `queued` past 10 s with no task, or `interrupted` after a restart) as `cancelled`. -- `DELETE /projects/{p}` (and archive) answering `409 project_busy` right after an ingest finished (#249): the detection worker refreshed its in-flight count only every 15 s, so the last batch read as busy for up to that long after it had landed. The worker now rewrites a project's liveness within a second of its count changing, and the `project_busy` message names each running job and says when a retry is the answer. -- `make pe-build-trt` (`export/build_pe_trt.sh`, #277) no longer ships an FP32 engine silently: a failed or FP32-falling-back FP16 bake, a failed FP16 build or a failed uint8 input step now exits 4 with the cause. `ALLOW_FP32=1` opts in to an FP32 engine knowingly. +## [0.6.0] - 2026-10-16 + +### Highlights + +0.6.0 makes ingest much faster and adds a way to point the VLM at exactly the crops you choose. Images are decoded once and (for large photos) on the GPU, detector and embedding inputs travel to Triton as uint8, the embedding engine is FP16, the PE image encoder can spread across several GPUs, and ingest writes to OpenSearch are batched; on the pinned public COCO set one RTX A6000 went from 8.9 to 17.4 images/s through `/ingest/upload` on the FP16 engine alone, with the rest of the gains measured in `docs/PERFORMANCE.md`. The new **assist runs** (`/assist/runs`) send a crop, a selection, a cluster, a sub-cluster or a filter to the VLM, with cluster label proposals you apply explicitly. Bulk ingest runs in the background from a folder, a manifest or S3, with a preview first, and ingest can be detection-only. A project calls no VLM until you activate one. The repository itself was cleaned up (guards, dead code removed, large files split, one typed env module, one docs source of truth). + +### Upgrade notes / behaviour changes + +Read these before upgrading from 0.5.0. + +- **New projects start with the VLM off** (#282). A project calls no VLM until it activates one (`GET .../vlm/endpoints/active` answers `source: "off"`). Projects that exist already are pinned once, at the first start of this version, to what they had (`env` when `OP_VLM_URL` was set, else `off`), so their behaviour does not change. +- **`make` and `model` are no longer hard-coded VLM outputs.** The reply keys, `vlm_item_make` and `vlm_item_model` are gone (old values stay in old documents, unread). Declare such facts as prompt-pack `attributes`; assist runs store them in the item's `assist` object. +- **`POST /vlm/label_cluster/{cluster_id}` and `POST /vlm/verify_regions` are removed.** Use `POST /assist/runs` (a `cluster` target, or `verify` on a `crops` or `boxes` target), or `?cluster_id=` on `auto_label/start`. +- **`curation-evaluator` moved from the `curation` profile to its own `evaluator` profile**, so `--profile curation up -d` no longer reserves a GPU for it. Start it with `./openprocessor curation up curation-evaluator`. +- **`CROPWRIGHT_*` env names are now `OP_CROPWRIGHT_*`** (`_IMAGE`, `_BIND_ADDRESS`, `_PORT`, `_INGEST_MAX_REQUEST_MB`, `_DATASET_UPLOAD_MAX_MB`). A `.env` that still sets an old name is refused at startup (`./openprocessor` and the installer exit 2, the container exits). Rename the keys and run `./openprocessor restart`. +- **Region-profile fields `aspect_min` and `aspect_max` are retired** (and `OP_REGION_DETECTION_ASPECT_MIN` / `_MAX`). A stored profile that carries either is rejected by name, and skipped with a warning when the registry loads; re-save it without them. +- **Rebuild the detector and PE image encoder engines** (`./openprocessor models repair`): both now take uint8 input, and a model whose plan and `config.pbtxt` disagree does not reach READY. +- **GPU decode is on by default** for the `full` profile (`OP_INGEST_GPU_DECODE`; `minimal` and `standard` default off). It adds about 8.5 GB to the Triton card. +- **New projects use the Lucene vector engine** (`ef_construction` 128); existing projects keep their faiss fields (no migration; re-ingest to move one). +- **Tests default `OPENSEARCH_URL` to a refused local address**, so an offline test run never reaches a live OpenSearch; set `OPENSEARCH_URL` for the live suite. +- **Retired region statuses `pending` and `pending_verify`, `regions.by_human` (use `regions.by_human_drew`) and `scripts/curation/region_worker_main.py` are removed.** `scripts/curation/ingest_walker.py` is replaced by `scripts/curation/bulk_ingest.py` (`--folder` instead of the old flags). +- **`TRITON_WORKERS`, `MAX_BATCH_SIZE` and `SHM_SIZE` are gone** from `.env`, `env.template` and the profile JSON (nothing read them); `setup.sh` no longer writes `docker-compose.override.yml`. +- **New environment variables** (all optional, documented in `env.template`): `OP_ASSIST_JOBS_DIR`, `OP_BULK_INGEST_DIR`, `OP_API_LOOPBACK_URL`, `OP_CPU_POOL_THREADS`, `OP_INGEST_INFLIGHT_MB`, `OP_INGEST_GPU_DECODE`, `OP_INGEST_GPU_MIN_MEGAPIXELS`, `OP_INGEST_DISPATCH_S`, `OP_CLUSTER_REFRESH_INTERVAL_S`, `_GROWTH_THRESHOLD`, `_MAX_DEFERRAL_S`, `_RETRY_BACKOFF_S` and `_AUTO_LABEL`, `OP_S3_ALLOWED`, `OP_S3_ENDPOINT_URL`, `OP_S3_REGION`, `OP_S3_PATH_STYLE`, `OP_S3_FETCH_WORKERS`, `OP_S3_LIST_WORKERS`, `OP_S3_INFLIGHT_MB`, `OP_S3_MAX_OBJECT_MB`, `OP_SOURCE_ROOT_HOST`, `OP_VLM_POLL_S`, `OP_VLM_WAIT_S`, plus the renamed `OP_CROPWRIGHT_*` set. `OP_HNSW_EF_CONSTRUCTION` now documents the new default of 128. +- **Compose changes**: the `evaluator` profile above; Triton gets `CUDA_DEVICE_ORDER=PCI_BUS_ID` (so `TRITON_GPU_ID=0,2` means the lowest listed id first), `CUPY_CACHE_DIR` and glibc malloc tuning for the API; Triton runs with `--log-verbose=0`; the API's memory reservation uses the `deploy.resources.reservations` form; an optional git-ignored `docker-compose.local.yml` is loaded after the stock files; the `gateway-wrap` message names `OP_CROPWRIGHT_BIND_ADDRESS`. Image defaults now point at `0.6.0`. ### Added +- Stage timers `file_read` and `gpu_decode` in `op_pipeline_stage_seconds` and the bulk run's `stage_s`, so time spent reading source files and in the GPU decode call no longer lands in `opensearch_write`. +- `OP_OPENSEARCH_DATA` (compose): the OpenSearch data directory as a Docker volume name (default) or a host path; docs-site Operations > Storage and sizing recommends local NVMe with measured numbers and the steps to move. + +- `scripts/bench/scale_run.py`, a driver for 100k-image ingest benchmarks (forced stop, resume, 15 s progress polls and 30 s resource samples, `--folder` or `--manifest` source), and `scripts/datasets/fetch_openimages_bench.py`, a seeded, checksum-pinned fetcher for a CC BY 2.0 Open Images v7 train set (pin in `scripts/datasets/manifests/openimages_bench_100000.sha256`). Results are in `docs/PERFORMANCE.md`, "Scale ladder: 10k and 100k". + - `scripts/docs/check_docs_vs_code.py` gains an `envnames` check: a non-`OP_` variable a doc names in a code span, on a line about env or variables, must be read by a compose file, script or code (#287). + - Example prompt pack `examples/prompt_packs/general_objects.json`: the generic item prompts plus three domain-neutral item attributes (`brand`, `colour`, `material`) for COCO-style objects; validated by the pack validation in a test. + - `GET /clusters` cards serve `subcluster_ids` (the sorted sub-cluster ids, at most 200; `n_subclusters` stays the exact count). + - The item wire serves the accuracy audit state (#166): `audit_sample`, `audit_batch_id`, `audit_sampled_at`, `audit_label_name`, `audit_label_source`, `audit_outcome` and `audit_human_name` (null for an item never drawn), so `GET /audit/queue` items can show the machine label they were drawn with. + - Cropwright **Assist…** (cluster page, one crop, one box, Matching items): a dialog to pick what the VLM sees (selected, all, a representative sample), the actions (classify, attributes, verify, segment) and the VLM and prompt pack, with the server's dry run; the cluster and sub-cluster label proposal card with the agreement, an explicit dry-run-first **Apply** and the `scope_changed` refusal; dashed "VLM suggests" attribute chips; and a **Region verification** switch (`region_verify`) on the VLM scope panel. See the guide [Selective VLM and segmenter assist](docs-site/docs/guides/selective-assist.mdx). - Pack attributes in whole-project VLM runs: `vlm_attributes=true` on `POST /pipeline/auto_label` and `/start` makes the VLM stage use the pack's combined `assist` call and store the item attributes in each item's `assist` slot. The region worker's combined call asks for the pack's item attributes on the items it classifies and writes them to `assist.attributes` too. + - Selectable segmenter verification: the VLM policy takes `region_verify` (`vlm`, the default, or `none`) for the region worker, and an assist run with the `segment` action re-segments chosen crops, a cluster or a filter, with `segment.verify` (`vlm` or `none`) for those items only. The run waits in the new status `waiting_region` and reports `result.segment`. Refusals: `409 region_profile_required`, `no_segmenter_leg`, `region_stage_paused`, `verify_endpoint_mismatch`. With the worker's VLM requested but unreachable, an item is written as `detection_failed` (`vlm_unavailable`). + - Assist runs on boxes: a `boxes` target (chosen `{crop_id, box_id}` pairs) takes `verify` and `attributes`. The `verify` action replaces `POST /vlm/verify_regions` (one verdict per box, only the named boxes, never a human- or import-owned one); box attributes the pack declares with `applies_to: box` are stored under the item's `assist.boxes[]`, apart from the box. + - Cluster and sub-cluster label proposals from an assist run: `result.proposal` (top class, agreement, vote distribution, new-class proposals by name, attribute summary) with an agreement gate (at least 3 answers, `min_agreement`), `scope_now` on every read (membership overlap against the members the run resolved, so a re-refine shows as a changed scope), `GET /assist/runs` for a cluster or sub-cluster, and `POST /assist/runs/{run_id}/apply`: the explicit, dry-run-first action that labels the scope's current members as a human would, skips locked and validated ones and refuses a changed scope (`409 scope_changed`). Assist class suggestions leave the item in its cluster. + - Assist runs, `POST /assist/runs` (dry run by default), `GET /assist/runs/{run_id}` and `POST /assist/runs/{run_id}/cancel`: one request shape sends a crop, a selection, a cluster, a sub-cluster or an item filter to the VLM, whole or as a seeded, size-scaled representative sample stratified by sub-cluster, with `classify` and `attributes` actions. Class answers go through the guarded VLM class writer (a locked, validated or holdout item keeps its class); every answer, a locked item's included, is stored in the item's new `assist` object (shown on the item wire), apart from the labels. One run per project (`409 assist_busy`); `OP_ASSIST_JOBS_DIR`. + - Prompt packs declare `attributes` (up to 16 free-form facts about an item or a box, with an optional closed vocabulary) and carry the `assist` (classify plus item attributes) and `attributes` (attributes alone) prompts. The labeler gains `assist_batch` and `attributes_batch`, and `POST /prompt_packs/test` accepts `call: assist` and `attributes`. Pack validation checks the attribute list and the prompts it needs (`pack_attribute_invalid`, `pack_attribute_duplicate`). The `vehicle_wheel` example pack declares `make`, `model`, `body_style` and a wheel `rim_material`. + - `GET /crops?cluster_subid=` and the item filter's `cluster_subid` select one AHC sub-cluster on the server, so a sub-cluster larger than a page can be listed whole. + - Detection-only ingest (#279): ingest policy `embedding.mode: "none"` stores boxes with no vector of any kind (the whole-frame vector included) and `clustering.automatic: false` keeps ingest and the embed scope from placing items in clusters, the `curation-cluster-refresh` daemon from refreshing the project and the auto-label worker's idle retrain from running. Embed and cluster later on demand. Cropwright's ingest-policy page offers both. + - The cluster-refresh daemon reads `OP_CLUSTER_REFRESH_INTERVAL_S`, `_GROWTH_THRESHOLD`, `_MAX_DEFERRAL_S`, `_RETRY_BACKOFF_S` and `_AUTO_LABEL` (#292); a command-line flag overrides each. No compose override file is needed. + - Pipeline modes guide: a recipe, run on public data, for "ingest without the VLM, cluster, name clusters by hand, then run the VLM for one cluster, filter or batch" (`vlm=` per run). - `./openprocessor` and `make` load an optional, git-ignored `docker-compose.local.yml` after the stock compose files: a second read-only source root, a changed `shm_size` or a mounted credentials file no longer need hand-built `docker compose` commands (#283). + - `./openprocessor curation up [service...]` and `make curation-up SERVICES="..."` start a subset of the curation services; a name that is not a curation service is refused (#285). + - `capture_docs_screenshots.py` requires `--public-projects` (`OP_CROPWRIGHT_PUBLIC_PROJECTS`), refuses a `--project` outside it, and refuses the pages that list every project (project list, combine, copy-settings, clone-from, model sharing) while the backend holds any project outside it (#284). + - `scripts/docs/check_docs_vs_code.py` checks `./openprocessor` commands and flags, `make` targets, `` file names and `NAME=` lines in fenced blocks against the code; the version test scans the frontend docs; `docs.yml` also runs on `docs/**` and type-checks the site; a weekly `docs-links.yml` checks external links (#287, #293). + - `scripts/codegen/check_cited_paths.py` (pre-commit): a comment or docstring that cites a `docs/`, `scripts/` or similar file cites one that exists. Citations of design notes that left the tree in the frontend sit in a shrink-only baseline (#289). + - Bulk-ingest preflight (#278): `POST /curation/projects/{project}/ingest/bulk/preview`, `scripts/curation/bulk_ingest.py --preview` and a Preview button on the Cropwright Ingest page report, without ingesting anything, a folder's, manifest's, S3 prefix's or registered source's file count, total @@ -65,16 +107,19 @@ history of this codebase and was never published. This release is `[0.5.0]`. uses only figures measured in `docs/PERFORMANCE.md`, served (with the section each came from) in `GET .../ingest/bulk/config` under `preview`. The walk is capped (200,000 files, 60 seconds) and answers `partial: true` with the reason, stops when the client disconnects, and refuses an unreadable root. + - `docs/design/engineering_standards.md`: the portable standards and lessons from the repo cleanup (principles, the lint, type and test config that worked, every guard with its baseline format, `--tighten` workflow and plant-a-violation proof, an ordered port checklist for a sister repo, pitfalls, before/after numbers). `CONTRIBUTING.md` points to it. + - `scripts/bench/cluster_bench.py` (#218): times the production cluster pass at a chosen pool size (`--sizes`, `--methods`, `--reps`) on real public COCO embeddings in a throw-away project and records method, wall time and counts; a pass that did not cluster the whole pool is refused. `docs/PERFORMANCE.md` gains a summary table of the optimization milestone's results, the k-NN native-memory formula checked against the stored 100k data (#56) and the go / no-go of the write ceiling against 100 images/s (#230). + - Ingest from S3 and S3-compatible stores (#264): `--s3` and `--source-id` on `scripts/curation/bulk_ingest.py`, an `s3` or `source_id` start on `POST /curation/projects/{project}/ingest/bulk`, and registered ingest sources under @@ -85,6 +130,7 @@ history of this codebase and was never published. This release is `[0.5.0]`. already stored. S3 is off until `OP_S3_ALLOWED` is set; credentials come only from the `api` container's environment, a mounted file or a role. A dispatcher in the API acts on due triggers (`OP_INGEST_DISPATCH_S`). Guide: Ingest from S3 and watched sources. + - `scripts/bench/ingest_transport_bench.py` and the "Ingest transports (#263)" measurement in `docs/PERFORMANCE.md`: HTTP/1.1 multipart against `grpc.aio` streaming and unary, transport only, on public images. A gRPC ingest channel saves under 1.2 percent of the pipeline's API CPU per image and the @@ -110,18 +156,22 @@ history of this codebase and was never published. This release is `[0.5.0]`. - A checksum-pinned public benchmark set: `fetch_coco_subset.py --bench-set N` draws a seeded, license-filtered, all-class COCO 2017 set and pins per-image SHA-256 and bytes (`scripts/datasets/manifests/coco_bench_2000.json`); `--verify-only` re-hashes a download offline (#45). + - Ingest stage timers for the work that ran untimed: `hash`, `persist`, `letterbox`, `triton_request`, `whole_frame_preprocess`, `blur`, `doc_build`, `bulk_serialize` and `admission_wait` join the `op_pipeline_stage_seconds` stages, and `resize` is renamed `pe_preprocess` (it is the PE crop preprocessing). On 12-24 MP photos the timers name 93 percent of the API CPU, on 640 x 480 frames 54 percent; the table and the native profile of the rest are in `docs/PERFORMANCE.md` (#210). + - `OP_CPU_POOL_THREADS` (default 8) and `OP_INGEST_INFLIGHT_MB` (default 1024): the threads per API worker for pixel work on large images, and the megabytes of decoded pixels a worker admits into ingest at once (#241). + - `scripts/bench/baseline_suite.py` and `suite_report.py`: one entry point that records the environment, ingest through `/ingest/upload` and `/ingest/batch`, Triton statistics, perf_analyzer points, endpoints and VLM throughput as one JSON; the v0.5.0 numbers are in `docs/PERFORMANCE.md` and `docs/benchmarks/` (#45, #40). + - Wave 0 full-pipeline throughput probes: `scripts/bench/ingest_sweep_suite.py` (client concurrency and batch sweep, Triton instance-count, queue-delay and preferred-batch arms applied and restored through the model store, per-endpoint stage cost for faces, OCR and the combined analyze route, @@ -130,6 +180,7 @@ history of this codebase and was never published. This release is `[0.5.0]`. `ingest_sweep_report.py` (the tables). `scripts/datasets/fetch_hires_set.py` pins a 500-image set of 12 to 24 MP public photos (`manifests/commons_hires_500.json`). Results and the binding resource per stage are in `docs/PERFORMANCE.md` ("Wave 0: full-pipeline throughput") (#45, #210, #230). + - `scripts/bench/parity.py` with `capture` and `compare` subcommands: the accuracy gate every v0.6.0 change passes. `capture` ingests the pinned COCO set into a fresh project and stores the per-stage outputs (detector boxes, crop rectangles, PE crop and whole-frame vectors, cluster @@ -139,11 +190,13 @@ history of this codebase and was never published. This release is `[0.5.0]`. neighbour overlap, cluster agreement and purity against the COCO ground truth, label agreement), applies the thresholds of `scripts/bench/parity_gates.json` and exits non-zero on failure. The FP32 against FP16 PE engine results are in `docs/PERFORMANCE.md` (#208). + - Measured INT8 evaluation of the PE image encoder on Ampere (RTX A6000 and RTX 3080 Ti), recorded in `docs/PERFORMANCE.md` and `docs/benchmarks/int8_pe_eval.json`. Result: explicit Q/DQ INT8 is 1.2-1.4x faster than FP16 but fails the parity gates (crop cosine median 0.94-0.99 against 0.999, neighbour overlap 0.52-0.82 against 0.95), and weight-only INT8 passes the gates with no speed or size gain. INT8 is not offered; FP16 stays the only PE precision and no code changed (#213). + - Measured system shared memory between the API and Triton for the detector and PE image batches, recorded in `docs/PERFORMANCE.md` and `docs/benchmarks/shared_memory_transport.json`. Result: it removes about 6 ms of client CPU per frame when the transport is measured alone, but ingest API CPU @@ -163,6 +216,7 @@ history of this codebase and was never published. This release is `[0.5.0]`. repeated card, mixed compute capabilities, or a card without room; the installer takes `--gpu-plan triton=0+2` and refuses the same cases. `scripts/bench/pe_gpu_parity.py` compares embeddings of the same crops per card (#228). + - GPU decode measured and planned (`docs/design/gpu_decode_plan.md`, numbers in `docs/PERFORMANCE.md`): on 12-24 MP photos the CPU JPEG decode caps ingest at about 26 photos/s on this host; GPU decode cuts the decode's host cost 3.7x and a GPU decode cut API CPU per photo by @@ -171,6 +225,7 @@ history of this codebase and was never published. This release is `[0.5.0]`. references while AP and kNN precision are unchanged, so it waits for an owner decision. New probe tools `scripts/bench/gpu_decode_bench.py` and `scripts/bench/gpu_decode_parity.py` with tested kernels and helpers (#217). No product code changed. + - GPU decode for photos, opt-in with `OP_INGEST_GPU_DECODE` and gated by `OP_INGEST_GPU_MIN_MEGAPIXELS` (3.0): the `gpu_image_prep` Triton model decodes each JPEG once on the GPU (nvJPEG with GPU Huffman decoding), builds the detector letterbox and the PE squares with fused @@ -210,19 +265,25 @@ history of this codebase and was never published. This release is `[0.5.0]`. `.env` and names the real restart command. `setup.sh` no longer writes `docker-compose.override.yml`, and `TRITON_WORKERS`, `MAX_BATCH_SIZE` and `SHM_SIZE` (read by nothing) are gone from `.env`, `env.template`, the profile JSON and the docs (#281). + - `curation-evaluator` moved from the `curation` profile to its own `evaluator` profile, so `--profile curation up -d` no longer reserves a GPU for it. Start it with `./openprocessor curation up curation-evaluator` or `docker compose run --rm curation-evaluator` (#285). + - In a checkout, `./openprocessor` with no `COMPOSE_PROJECT_NAME` in `.env` uses the documented default project `openprocessor` and says so, instead of exiting 3. An install directory still refuses (#291). + - `bulk_ingest.py` defaults `--api-base` to `localhost:8000` inside the container and `localhost:4603` on the host (#288). + - `docs-site/.nvmrc` follows the Node version of `docs-site/Dockerfile` (#289). + - Repo cleanup, PR 9a (typing): mypy now runs with `disallow_untyped_defs`, `disallow_incomplete_defs` and `warn_return_any` over `src scripts docker export` (`tests/` is outside its scope), and the nine-error-code relax override for the ML modules is gone. 328 errors were fixed with real types (route return annotations that match each `response_model`, `Protocol`s for the FAISS index and the ingest outcome, typed mixin attributes); the OpenAPI contracts are unchanged. + - Repo cleanup, PR 6 (backend split by concern): every `src/` and `scripts/` file of 600 lines or more is split into cohesive modules (routers, `src/main.py` into `src/app/`, the Triton and PE clients, curation, projects, training and config-store services, `src/config/curation.py`, three scripts), @@ -230,6 +291,7 @@ history of this codebase and was never published. This release is `[0.5.0]`. contracts and behavior are unchanged. Twenty-three functions over complexity 15 are decomposed behind characterization tests and leave the complexity baseline. `duplicate_detection/__init__.py` no longer holds logic, and the two exporters share `export_models` and `export_writers`. + - **BREAKING**: the Cropwright variables are now `OP_CROPWRIGHT_*`: `CROPWRIGHT_IMAGE`, `CROPWRIGHT_BIND_ADDRESS`, `CROPWRIGHT_PORT`, `CROPWRIGHT_INGEST_MAX_REQUEST_MB` and `CROPWRIGHT_DATASET_UPLOAD_MAX_MB` became `OP_CROPWRIGHT_IMAGE`, `OP_CROPWRIGHT_BIND_ADDRESS`, @@ -238,6 +300,7 @@ history of this codebase and was never published. This release is `[0.5.0]`. way). A `.env` that still sets an old name is refused: `./openprocessor` and `setup-openprocessor.sh` exit 2 and print the new name, and the cropwright container exits at start. Rename the keys in `.env` and run `./openprocessor restart`. + - Every Dockerfile base image is pinned by digest (`image:tag@sha256:...`); the bump procedure is in `docs/releases/RELEASE_CHECKLIST.md`. The evaluator and segmenter images build their Python environment in a separate stage (no compilers or pip cache in the evaluator runtime; the @@ -245,10 +308,12 @@ history of this codebase and was never published. This release is `[0.5.0]`. api image copies only the `scripts/` it runs (`scripts/curation`, `scripts/datasets`, six shell libraries) instead of the whole tree. The root `.dockerignore` also excludes `artifacts_local/`, `docs-site/`, `frontend/`, `.claude/`, `.install/` and `export_*/`. + - `check_docs_vs_code.py` requires an assignment line in `env.template` for every compose `${VAR}` (a prose mention no longer counts) and gains a `template` check: a non-`OP_` setting that no compose file, script or code reads is rejected. The unread `LOG_LEVEL` and the unread `OP_CROPWRIGHT_TAG` documentation are removed. + - Repo cleanup, PR 5b (duplicates): one `now_iso()` (`src/utils/timeutil.py`, UTC, microsecond precision) replaces 21 private timestamp helpers with three different bodies; the class-history and edit-history entries used `timespec='seconds'` and now carry microseconds like every other @@ -261,6 +326,7 @@ history of this codebase and was never published. This release is `[0.5.0]`. `curation_umap.py` moved under `src/routers/curation/` (`train/`, `images.py`, `umap.py`), `src/services/clustering.py` to `src/services/visual_search/clustering.py`; routes and the OpenAPI contract are unchanged. `requirements-dev.txt` is gone (no consumer). + - Environment variables are read through one typed module, `src/config/env.py` (`env_str`/`env_int`/`env_float`/`env_bool`/`env_path`), and ruff `TID251` bans `os.environ`/`os.getenv` in `src/`. Behavior change: a variable that is set but malformed or @@ -287,9 +353,11 @@ history of this codebase and was never published. This release is `[0.5.0]`. `docs/benchmarks/v060_wave0_*.json` keep their names), and the review-id file and test names under `tests/curation/` lost their suffixes. Comments and docstrings no longer carry wave, round or work-package ids (`scripts/codegen/check_process_ids.py` now enforces that with an empty baseline). + - The region field that keeps the status an item had before a region requeue is now `region_status_before_requeue` (was `region_status_legacy`). Values already stored under the old name are not read; they only recorded a past requeue. + - Repo guards, the first step of the repo cleanup (`docs/design/repo_cleanup_plan.md`). Hooks: pre-commit-hooks 6, gitleaks 8.30, bandit 1.9, shellcheck 0.11, hadolint 2.15 and ruff 0.17 (the hook id is now `ruff-check`; Dockerfile health checks use the JSON form hadolint wants). @@ -310,7 +378,6 @@ history of this codebase and was never published. This release is `[0.5.0]`. `--root`, no progress file: the run resumes from the server). The parallel directory walk moved to `src/services/curation/bulk_ingest/walk.py`. - - The detector engines (`trt`, `trt_end2end`, YOLO26 and dual-head exports) take uint8 pixels and do the `/255` in the TensorRT graph, so a detector request carries a quarter of the bytes of the FP32 tensor it replaces (4.9 MB to 1.2 MB per image). `export/uint8_input.py` puts the cast and divide in @@ -319,6 +386,7 @@ history of this codebase and was never published. This release is `[0.5.0]`. detector engine (`./openprocessor models repair`); a model whose plan and config disagree on the input type does not reach READY, so the installer rebuilds it. Promoted trained models (ORT, `output0`) and the bake-off engines keep float32 input (#214). + - The PE image encoder takes uint8 pixels and applies `(x / 255 - 0.5) / 0.5` in the TensorRT graph (`build_pe_trt.sh` and `build_pe_ort_fallback.sh` run `export/uint8_input.py` after the FP16 bake), so a PE request carries a quarter of the bytes of the FP32 tensor it replaces (10.0 MB to 2.5 MB @@ -329,12 +397,14 @@ history of this codebase and was never published. This release is `[0.5.0]`. 49 percent (0.118 to 0.062 CPU-s), and images/s rise 10 to 12 percent (19.1 to 21.0 upload); the gates pass against a rebuilt FP32 control, and rebuilding the detector engine alone already moves detections past the box gate of the v0.5.0 reference (`docs/PERFORMANCE.md`) (#214). + - `ocr_pipeline` ships two Python instances instead of one, and the installer profile now sets its count (`ocr_pipeline_instances`: minimal 1, standard 2, full 4). The Python BLS model runs one request per instance, so OCR was capped at about 21 images/s; Wave 0 measured 42 with two instances and 80 with four. A warm instance costs about 256 MiB of GPU memory (measured on a 12 GB card); `GPU_MIB_PER_INSTANCE` in the model config and `python_backend_budget_mib` per profile bound the count, and `tests/test_triton_instance_budget.py` enforces it (#240). + - Response serialization no longer uses FastAPI's deprecated `ORJSONResponse` (about 15 routers, the error handlers and four routes that built it directly). Routes with a response model serialize through Pydantic straight to bytes; the curation routes that return a @@ -343,6 +413,7 @@ history of this codebase and was never published. This release is `[0.5.0]`. about 4 ms instead of about 69 ms and the acceptance log no longer repeats the deprecation warning. Response bytes are unchanged; the one contract change is that `POST .../datasets/imports/{import_id}/undo` now documents its response schema (#153). + - Ingest on 12-24 MP photos is no longer bound by one event-loop thread per worker. Decode, the detector letterbox, PE and whole-frame preprocessing, blur scoring and crop encoding of a large image run on a bounded per-worker thread pool (work under 4 MiB of pixels stays in place: a pool hop cost @@ -350,6 +421,7 @@ history of this codebase and was never published. This release is `[0.5.0]`. decoded megabytes fit under `OP_INGEST_INFLIGHT_MB`. On the 500-photo set at 8 images per request throughput is 21.5, 27.3, 23.6 and 24.1 images/s at 8, 16, 32 and 64 clients against 5.2, 8.7, 10.8 and 4.9, and no longer falls past 32 clients (#241). + - API CPU per 640 x 480 frame is down 29 to 33 percent (0.170-0.192 to 0.113-0.136 CPU-s; the 40 percent target was not met) with no change to any stored value: vectors reach OpenSearch through `ndarray.tolist()` and an orjson serializer (34 to 1 ms per image), PE normalization is a lookup table @@ -359,8 +431,10 @@ history of this codebase and was never published. This release is `[0.5.0]`. checks each distinct index once, and the API container keeps freed buffers instead of trimming them (`MALLOC_TRIM_THRESHOLD_`, `MALLOC_TOP_PAD_`, `MALLOC_MMAP_THRESHOLD_` in `docker-compose.yml`, up to about 128 MB resident per worker) (#211). + - Triton runs with `--log-verbose=0` instead of logging an allocation and a free for every tensor of every request (#211). + - The PE image encoder engine is 17 to 25 percent faster (181 to 225 images/s through Triton at batch 16, same output: crop cosine median 0.99996, every `precision` parity gate passes). `export/pe_rope_layout.py`, the last step of `export/export_pe_image_encoder.py`, permutes the query and key channels of every @@ -410,6 +484,7 @@ history of this codebase and was never published. This release is `[0.5.0]`. that used to stop at about 26 images/s now sustains 100 (and 175 to 187 at 200), with 62 percent less API CPU per image at 100 and k-NN p95 under the same writes down from 449 to 28 ms at 25. The upsert moved to `src/clients/occ_upsert.py` and returns an outcome per document (#248). + - New projects build their vector fields on the Lucene engine with `ef_construction` 128 (was faiss 512): the items index is 48 percent smaller on the product stack, filtered recall stays near 1.0, the cosine score is unchanged, and the review-default overlap@10 against faiss is 0.992 (gate 0.95) with cluster @@ -450,6 +525,7 @@ history of this codebase and was never published. This release is `[0.5.0]`. the status and body the old `HTTPException` carried. No route, status code or response body changed. An auto-label job queued before the upgrade names the old function path and fails to resolve it: requeue it. + - The frontend's oversize files are split by concern: `api.ts` into per-domain `api_.ts` modules, the review, train, export, clusters, cluster-detail and classes pages into a controller plus components, `types.ts`, the slot-config parser, the large vitest files, `e2e/conftest.py` @@ -459,13 +535,17 @@ history of this codebase and was never published. This release is `[0.5.0]`. ### Removed - `POST /vlm/verify_regions`: use `POST /assist/runs` with action `verify` on a `crops` or `boxes` target. + - The hard-coded `make` and `model` VLM outputs: the reply keys, `VlmClassPrediction.make/model`, the combined reply's `make/model`, and the `vlm_item_make` / `vlm_item_model` item fields (never on the wire). A prompt pack declares such facts as `attributes` instead; they are stored in the item's `assist` object by assist runs. Existing values in `vlm_item_make` / `vlm_item_model` stay in old documents and are not read. + - `POST /vlm/label_cluster/{cluster_id}`: use `POST /assist/runs` with a `cluster` target (and `sample.mode: "all"`), or `?cluster_id=` on `auto_label/start` for the whole-cluster job. Cropwright's cluster-page "Run VLM" and the dashboard's "Run VLM on cluster" dialog are replaced by **Assist…** on the cluster page, which uses the assist run. + - The region-profile fields `aspect_min` and `aspect_max` (and `OcrRegion.is_region_shaped`, which was their only reader and had no caller): from the profile dataclass, the profile editor schema, the license plate example, the docs and the OpenAPI contract. A stored profile that still carries either key is rejected by name and skipped with a logged warning when the registry loads, so re-save it without them. `OP_REGION_DETECTION_ASPECT_MIN` and `_MAX` no longer exist. + - Dead code, found by the new vulture guard (`scripts/codegen/check_vulture.py`) and the plan's list: `src/services/cluster_maintenance.py`, the unused `ImageService`, the unused `region_lean` module (and the `paddleocr` requirement only it imported), the unused @@ -474,30 +554,47 @@ history of this codebase and was never published. This release is `[0.5.0]`. and other helpers only a test called, and the `src.utils` package re-exports (import from the submodule). Reviewed dynamic-dispatch cases are in `scripts/codegen/vulture_whitelist.py` with a reason each. + - The `scripts/curation/region_worker_main.py` compatibility entry point: start the detection worker with `python -m scripts.curation.worker` (the compose file already does). + - `regions.by_human` in `GET /curation/projects/{project}/stats/dataset`: it was the old name of `regions.by_human_drew`. + - The retired region statuses `pending` and `pending_verify`: queries, the worker and the queue gauges now match only the `RegionStatus` values, which is all any writer has emitted since the rename. + - The `check-no-literal-region-fields` hook and `scripts/codegen/check_no_literal_region_fields.py`: the naming-leak scan and `tests/test_no_legacy_region_scalars.py` cover the same ground. ### Fixed +- Bulk ingest of large photos was held back by its own admission gate and by file reads on the event loop (see `docs/design/ingest_write_path_investigation.md`). The per-process gate now charges a photo the GPU decode takes its encoded size, not the decoded pixels it never holds in the API (a 16-photo request of 20 MP counted 915 of 1,024 MiB, so a second request on the same worker waited, 41 percent of request time in the 100k 20 MP run). `/ingest/batch` and `/ingest/image` read their files in parallel on a per-process pool (`OP_INGEST_READ_THREADS`, default 8) instead of one after another on the loop thread. + +- Ingest no longer fails a JPEG that Pillow reads but libjpeg-turbo's header read refuses (`tjDecompressHeader3(): Could not determine subsampling level`, 2 of 100,000 Open Images files, `decode_failed`): that error alone sends the image to the Pillow decode; a damaged file still fails. + +- Dispatcher-started bulk-ingest runs (poll, cron, webhook) never started: the tick ran in a worker thread and both the run and the poll called `asyncio.create_task` there (`ingest_dispatch_failed`, `no running event loop`). The tick now runs on the loop. A run whose spawn fails is marked `failed` instead of staying `queued` (which kept every later start, and the project's delete, answering 409), and `POST /ingest/bulk/{run_id}/cancel` finalizes an orphan (a run `queued` past 10 s with no task, or `interrupted` after a restart) as `cancelled`. + +- `DELETE /projects/{p}` (and archive) answering `409 project_busy` right after an ingest finished (#249): the detection worker refreshed its in-flight count only every 15 s, so the last batch read as busy for up to that long after it had landed. The worker now rewrites a project's liveness within a second of its count changing, and the `project_busy` message names each running job and says when a retry is the answer. + +- `make pe-build-trt` (`export/build_pe_trt.sh`, #277) no longer ships an FP32 engine silently: a failed or FP32-falling-back FP16 bake, a failed FP16 build or a failed uint8 input step now exits 4 with the cause. `ALLOW_FP32=1` opts in to an FP32 engine knowingly. - `POST /reprocess` with a filter target and `limit` / `sample` capped only the image scopes; the `region` and `vlm` scopes ran on every match. They now act on the capped, seeded selection, and the dry run counts it. + - CI stability: the offline suite no longer builds its app-lifespan OpenSearch client against the in-cluster hostname, which resolved slowly on the runner (10 to 15 s per lookup, several per startup) and made integration test setup hit the 400 s timeout, doubling the suite's run time. Tests now default `OPENSEARCH_URL` to a refused loopback port. The stubbed e2e harness gains a default 404 for `GET /vlm/endpoints/active`. - `PUT /ingest/policy` dropped the `clustering` block: saving `clustering.automatic: false` answered 200 with `true` and stored `true`, so detection-only clustering could never be switched off. + - The auto-label VLM stage no longer reports `completed` with `predicted: 0` when it could label nothing: it adds `failed` and `unreadable` counts, and fails the job when every selected crop failed (an unreachable VLM, or source images the worker cannot read). + - `docs-site` `npm run typecheck` passes: `baseUrl` (deprecated, TS5101) is dropped for `paths` alone, and the missing `@types/react-dom` is added (#293). + - Monitoring alerts (#290): the Grafana rules **Triton Model Not Ready** (`nv_model_ready_state`), **API High Error Rate** (`http_requests_total`) and **OpenSearch Cluster Unhealthy** (`opensearch_cluster_status`) queried series nothing exports and could never fire; the first and last are @@ -509,10 +606,13 @@ history of this codebase and was never published. This release is `[0.5.0]`. it), and the two contact points are renamed `unconfigured-default` and `unconfigured-critical` because they post to a localhost address nothing listens on: nobody is notified until you configure a receiver (`monitoring/README.md`, `docs-site/docs/operations/monitoring.mdx`). + - A bulk-ingest folder walk that was stopped (a cancelled run, a closed walk) kept listing the rest of the tree and waited for a stuck directory listing before returning; it now stops at once. + - Starting a bulk ingest on a folder the API cannot read is refused (`bulk_ingest_invalid_source`, "not readable") instead of completing with zero images discovered. + - The auto-label, cluster-refresh and VLM workers no longer log `Unclosed client session` on shutdown: each opened a second OpenSearch client for its project registry that nothing closed. `script_project_registry` now takes the worker's own client, which the worker already closes. @@ -521,13 +621,16 @@ history of this codebase and was never published. This release is `[0.5.0]`. raises `IndexError` and fails the whole batch. Only the affected image is refused, with the reason `face_landmarks_length_mismatch`; `detect_faces`, `recognize_faces` and the single-image face paths report the same reason. + - App shutdown (#319): each lifespan shutdown step (background tasks, pools, executors, Triton and OpenSearch clients) now runs independently; a failing step is logged as `shutdown_step_failed` with its step name and no longer skips the steps after it. + - `make cluster-gpu` renders again on Compose 2.29.7 (#315): the auto-label worker states its 2 GB memory floor under `deploy.resources.reservations`, which the GPU overlay extends, instead of `mem_reservation`, which Compose rejects next to it. A new test renders every compose file combination the Makefile and CLI use. + - The `vlm use` interrupt test passes when pytest is started in the background (#304): the child inherited an ignored SIGINT, which bash cannot trap; the test now restores the default. @@ -545,25 +648,31 @@ history of this codebase and was never published. This release is `[0.5.0]`. a cluster run raise instead of returning `0`. A crashed background job is logged at shutdown and the ingest dispatch tick runs off the event loop. About thirty best-effort OpenSearch cleanup handlers now catch only OpenSearch errors. + - A job state file that exists but cannot be read counts as a running job in every start guard (`FileJob.is_live`, and the auto-label guard), so a second job is not started beside one it cannot see (#161). + - The cluster-refresh daemon keeps its per-project growth state (items at the last retrain, the job in flight, the previous poll's count) in the project's settings document, so a restart neither retrains a project that was just retrained nor starts a retrain in the middle of an ingest (#235). An audit-gate `409` from `auto_promote` no longer stops the retrain from being queued (#280). + - An auto-label job still queued after `queue_max_age_seconds` (one hour) fails with "auto_label worker not running" and its trigger is withdrawn; the status serves `queue_age_seconds` and `queue_max_age_seconds` (#236). + - An unreachable VLM endpoint no longer leaves the VLM worker spinning with a full queue and nothing reported. `POST /curation/projects/{project}/vlm/label_batch` answers `503 vlm_unreachable` when no crop got an answer (nothing is written, no crop is marked failed); the worker releases its in-flight crops, backs off (poll interval, doubling to 60 s; the same for an unreachable API) and resumes by itself when the endpoint answers. The stall reason is logged and served in the worker's heartbeat, so the compose healthcheck output reads `ok: ... stalled: VLM endpoint unreachable: ...` while the worker stays healthy. + - The VLM worker skips a project that has no class to label as, so its crops are never claimed in flight and no longer fill the queue and starve the other projects; it logs the change once per project and picks the project up on the next poll after its first class. + - An image that does not decode reports `not a decodable image` (or Pillow's own error for a damaged file) instead of `No module named 'pi_heif'` from the Ultralytics `Image.open` patch (#297). @@ -583,6 +692,7 @@ history of this codebase and was never published. This release is `[0.5.0]`. channel multiplier, so `bake_fp16_onnx` keeps those four convs in FP32. The engine is 75 MB against 147 MB, 1.5x (batch 8) to 2.1x (batch 32) faster per image, cosine similarity to the FP32 engine at least 0.9998 on 200 COCO images (#224). + - The `curation-vlm-worker` no longer polls every project each cycle and takes a 409 `vlm_not_configured` from the ones with no VLM (#207, about 2.7 idle cores at 18 projects). Each cycle it asks the registry and each project's activation which projects have an active VLM, @@ -594,10 +704,12 @@ history of this codebase and was never published. This release is `[0.5.0]`. per embedding, ingest 17.4 against 8.9 images/s on the 2,000-image COCO set, cosine similarity to FP32 0.9997 mean). Before, the FP16 build failed at ONNX parse and the installer silently shipped the FP32 engine. + - The installer no longer reports a precision fallback as a clean success: the FP32 retry warns with the model, the FP16 failure reason and the consequence, is recorded in `.install/precision.tsv`, marks the model group `degraded` (a re-run retries FP16) and shows under "Degraded precision" in `./openprocessor models status` and in the install summary. + - `frontend/scripts/capture_docs_screenshots.py` no longer aborts after the first image when `--out` is outside `frontend/` (it printed the path with `relative_to`, which raised `ValueError`). @@ -607,8 +719,10 @@ history of this codebase and was never published. This release is `[0.5.0]`. expecting TYPE_UINT8". `export/check_input_dtype.py` compares the built input datatype to `config.pbtxt` and fails the install on a mismatch; the build stamp records `input=uint8`, so an engine built before this fix is rebuilt on the next `models install`. + - `./openprocessor models install|repair` in a checkout without `.install/state.json` no longer exits silently: it logs that every model group will be built and goes on (#270). + - The committed `yolov11_small_trt_end2end` config declares the FP16 build's `det_scores` datatype, so a default export leaves the tree clean; a test compares each committed exporter-written config with its renderer. The installer no longer builds the unused standard `yolov11_small_trt` engine, and that @@ -1022,7 +1136,6 @@ optional Cropwright 0.1.0 UI that ships staged in the deploy bundle. - Full gate on the release commit: 6,943 tests passed (12 skipped), `pre-commit run --all-files` and `generate_contracts.py --check` clean. - Measured on the public COCO set (2,000-image subset, one run): 13.39 images/s, 0 failed, 1.37 crops per image (`docs/PERFORMANCE.md`). - ### Added - Every compose service rotates its logs (`json-file`, 10 MB x 5) and has a `mem_reservation` (OpenSearch reserves its heap size). - Installer warns when `vm.max_map_count` is below 262144, with the fix. @@ -1251,7 +1364,6 @@ optional Cropwright 0.1.0 UI that ships staged in the deploy bundle. - Auto-label's `vlm` stage reports `skipped` with an explicit `reason` when it selects nothing or `run_vlm` is off (#116). - ReDoc no longer loads its footer logo from `cdn.redoc.ly`; a test scans the vendored bundles for external fetches (#117). - - The config-store poll no longer logs a 404 WARNING per request for projects that never wrote `meta:config_revision`: optional documents (revision counter, activations, curation settings) are read through one shared reader that treats a miss as absent without a transport error, and an absent revision is remembered for one poll interval (`OP_CONFIG_POLL_S`). - `resource_links` no longer lists `triton_metrics` and `dcgm_metrics` (and `OP_TRITON_METRICS_*` / `OP_DCGM_*` are gone): they are Prometheus scrape endpoints bound to localhost, not browser UIs; Grafana and Prometheus show their data. - API `/metrics` now aggregates every uvicorn worker: compose sets `PROMETHEUS_MULTIPROC_DIR` to a tmpfs on `yolo-api`, so the `op_*` and request metrics behind the ingest and API dashboards reflect the whole API instead of one worker. diff --git a/CLAUDE.md b/CLAUDE.md index 67ca09ae0..324146a26 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -7,7 +7,7 @@ where things live, and the commands and rules that matter. Before making design decisions or judgment calls, read [`docs/VISION_AND_GOALS.md`](docs/VISION_AND_GOALS.md): what OpenProcessor is -for, the v0.5.0 feature scope, and the standards the code is held to (no dead +for, the v0.6.0 feature scope, and the standards the code is held to (no dead code, no compatibility shims, fail-closed isolation, class identity by name never index, review depth matched to real risk). diff --git a/README.md b/README.md index 86c0d759d..7257207bf 100644 --- a/README.md +++ b/README.md @@ -53,12 +53,27 @@ README is the overview, the install and a first result. - **Dataset import, combine, export, train, promote.** YOLO, COCO and OpenProcessor-export layouts; merge up to eight projects; bake-off model comparison; promotion back into Triton. +- **Selective VLM assist.** `POST /assist/runs` sends a crop, a box, a + selection, a cluster, a sub-cluster or a filter to the VLM, whole or as a + representative sample, and proposes a cluster label you apply explicitly + (see [Selective VLM and segmenter assist](docs-site/docs/guides/selective-assist.mdx)). + A project calls no VLM until it activates one. +- **Bulk ingest.** A folder, a manifest or an S3 bucket runs as a background + job with a preview first; ingest can also be detection-only. - **Class identity by name.** Import, combine, export, train, promote and model sharing map classes by name, never by index. - **The lock rule.** A human-set or import-validated label or box, and anything in the frozen test holdout, is never overwritten by an automated writer. +**What's new in 0.6.0.** Faster ingest (images decoded once, uint8 inputs to +Triton, an FP16 embedding engine, GPU decode for large photos, multi-GPU +embedding, batched writes; numbers in [`docs/PERFORMANCE.md`](docs/PERFORMANCE.md)), +assist runs, bulk and S3 ingest, and a cleaned-up codebase. Upgrading from 0.5.0: +read the upgrade notes in [`CHANGELOG.md`](CHANGELOG.md) (new projects start +with the VLM off, `CROPWRIGHT_*` env names are now `OP_CROPWRIGHT_*`, the +evaluator has its own `evaluator` profile). + The whole curation surface is in [`contracts/openapi/curation.json`](contracts/openapi/curation.json); the guides are under [`docs-site/docs/guides/`](docs-site/docs/guides/). diff --git a/VERSION b/VERSION index 8f0916f76..a918a2aa1 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.5.0 +0.6.0 diff --git a/docker-compose.yml b/docker-compose.yml index 394cf9de2..a3d2a780f 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -64,7 +64,7 @@ x-gateway-wrap: &gateway_wrap | services: triton-server: - image: ${OP_TRITON_IMAGE:-${OP_IMAGE_REPO:-davidamacey}/openprocessor-triton:${OP_IMAGE_TAG:-0.5.0}} + image: ${OP_TRITON_IMAGE:-${OP_IMAGE_REPO:-davidamacey}/openprocessor-triton:${OP_IMAGE_TAG:-0.6.0}} container_name: ${COMPOSE_PROJECT_NAME:-openprocessor}-triton logging: *default-logging restart: always @@ -139,7 +139,7 @@ services: - triton_net api: - image: ${OP_API_IMAGE:-${OP_IMAGE_REPO:-davidamacey}/openprocessor:${OP_IMAGE_TAG:-0.5.0}} + image: ${OP_API_IMAGE:-${OP_IMAGE_REPO:-davidamacey}/openprocessor:${OP_IMAGE_TAG:-0.6.0}} container_name: ${COMPOSE_PROJECT_NAME:-openprocessor}-api logging: *default-logging restart: always @@ -331,7 +331,7 @@ services: cropwright: profiles: ['cropwright'] # OP_CROPWRIGHT_IMAGE (a full reference, e.g. a digest pin) wins over the OP_IMAGE_TAG default. - image: ${OP_CROPWRIGHT_IMAGE:-${OP_IMAGE_REPO:-davidamacey}/cropwright:${OP_IMAGE_TAG:-0.5.0}} + image: ${OP_CROPWRIGHT_IMAGE:-${OP_IMAGE_REPO:-davidamacey}/cropwright:${OP_IMAGE_TAG:-0.6.0}} container_name: ${COMPOSE_PROJECT_NAME:-openprocessor}-cropwright # Soft scheduling floor for the static-file nginx, not a cap. mem_reservation: 64m @@ -399,7 +399,7 @@ services: # =================================================================== curation-detection-worker: - image: ${OP_API_IMAGE:-${OP_IMAGE_REPO:-davidamacey}/openprocessor:${OP_IMAGE_TAG:-0.5.0}} + image: ${OP_API_IMAGE:-${OP_IMAGE_REPO:-davidamacey}/openprocessor:${OP_IMAGE_TAG:-0.6.0}} container_name: ${COMPOSE_PROJECT_NAME:-openprocessor}-detection-worker logging: *default-logging mem_reservation: 1g @@ -465,7 +465,7 @@ services: - triton_net curation-vlm-worker: - image: ${OP_API_IMAGE:-${OP_IMAGE_REPO:-davidamacey}/openprocessor:${OP_IMAGE_TAG:-0.5.0}} + image: ${OP_API_IMAGE:-${OP_IMAGE_REPO:-davidamacey}/openprocessor:${OP_IMAGE_TAG:-0.6.0}} container_name: ${COMPOSE_PROJECT_NAME:-openprocessor}-vlm-worker logging: *default-logging mem_reservation: 512m @@ -504,7 +504,7 @@ services: - triton_net curation-auto-label-worker: - image: ${OP_API_IMAGE:-${OP_IMAGE_REPO:-davidamacey}/openprocessor:${OP_IMAGE_TAG:-0.5.0}} + image: ${OP_API_IMAGE:-${OP_IMAGE_REPO:-davidamacey}/openprocessor:${OP_IMAGE_TAG:-0.6.0}} container_name: ${COMPOSE_PROJECT_NAME:-openprocessor}-auto-label-worker logging: *default-logging # deploy form, not mem_reservation: Compose rejects the two together, and @@ -573,7 +573,7 @@ services: - triton_net curation-cluster-refresh: - image: ${OP_API_IMAGE:-${OP_IMAGE_REPO:-davidamacey}/openprocessor:${OP_IMAGE_TAG:-0.5.0}} + image: ${OP_API_IMAGE:-${OP_IMAGE_REPO:-davidamacey}/openprocessor:${OP_IMAGE_TAG:-0.6.0}} container_name: ${COMPOSE_PROJECT_NAME:-openprocessor}-cluster-refresh logging: *default-logging mem_reservation: 1g @@ -610,7 +610,7 @@ services: # profile, so `--profile curation up -d` does not reserve a GPU for it. # `docker compose run --rm curation-evaluator`. curation-evaluator: - image: ${OP_EVALUATOR_IMAGE:-${OP_IMAGE_REPO:-davidamacey}/openprocessor-evaluator:${OP_IMAGE_TAG:-0.5.0}} + image: ${OP_EVALUATOR_IMAGE:-${OP_IMAGE_REPO:-davidamacey}/openprocessor-evaluator:${OP_IMAGE_TAG:-0.6.0}} container_name: ${COMPOSE_PROJECT_NAME:-openprocessor}-evaluator logging: *default-logging profiles: @@ -652,7 +652,7 @@ services: # OP_SEGMENTER_URL=http://segmenter:8000 \ # docker compose --profile curation --profile segmenter up -d segmenter: - image: ${OP_SEGMENTER_IMAGE:-${OP_IMAGE_REPO:-davidamacey}/openprocessor-segmenter:${OP_IMAGE_TAG:-0.5.0}} + image: ${OP_SEGMENTER_IMAGE:-${OP_IMAGE_REPO:-davidamacey}/openprocessor-segmenter:${OP_IMAGE_TAG:-0.6.0}} container_name: ${COMPOSE_PROJECT_NAME:-openprocessor}-segmenter logging: *default-logging profiles: @@ -721,7 +721,7 @@ services: # =================================================================== curation-trainer: - image: ${OP_TRAINER_IMAGE:-${OP_IMAGE_REPO:-davidamacey}/openprocessor-trainer:${OP_IMAGE_TAG:-0.5.0}} + image: ${OP_TRAINER_IMAGE:-${OP_IMAGE_REPO:-davidamacey}/openprocessor-trainer:${OP_IMAGE_TAG:-0.6.0}} container_name: ${COMPOSE_PROJECT_NAME:-openprocessor}-trainer logging: *default-logging profiles: @@ -1102,7 +1102,7 @@ services: soft: 65536 hard: 65536 volumes: - - opensearch_data:/usr/share/opensearch/data + - ${OP_OPENSEARCH_DATA:-opensearch_data}:/usr/share/opensearch/data ports: - "${OP_BIND_ADDRESS:-127.0.0.1}:${OPENSEARCH_PORT:-4607}:9200" # REST API healthcheck: diff --git a/docs-site/docs/about/vision-and-goals.mdx b/docs-site/docs/about/vision-and-goals.mdx index 0b544f57d..9d4537c81 100644 --- a/docs-site/docs/about/vision-and-goals.mdx +++ b/docs-site/docs/about/vision-and-goals.mdx @@ -21,7 +21,7 @@ The pipeline, end to end: ingest, detection, region stage (one or many boxes per embedding and clustering, human and AI review, import of labeled datasets, export, training, bake-off, and promotion back into serving. -## What v0.5.0 delivers +## What v0.6.0 delivers - Isolated projects, each with its own indexes, directories and class registry. - Editable prompt packs and region profiles, versioned and revertible. @@ -31,6 +31,8 @@ training, bake-off, and promotion back into serving. - Dataset import (YOLO, COCO, OpenProcessor exports) and project combine, with classes mapped by name. - A one-line installer. +- Assist runs: send a crop, a selection, a cluster or a filter to the VLM and apply the proposed label explicitly. +- Bulk ingest from a folder, a manifest or S3, with a preview, and measured ingest speed-ups (decode once, uint8 inputs, FP16 embeddings, GPU decode, multi-GPU embedding). - Documentation for every feature, and a public, reproducible cars-and-wheels example. ## Standards the code is held to @@ -44,10 +46,11 @@ training, bake-off, and promotion back into serving. operation. - Work is merged in waves, each with its own gate. -## Not in v0.5.0 +## Not in v0.6.0 -Deep GPU and Triton pipeline optimization (issue #40), benchmarks, a technical white -paper and more domain showcases are planned after the release. +The full zero-copy GPU pipeline (issue #40), the 1M-image scale step, request-driven +processing plans, faces and OCR on resident crops, a technical white paper and more +domain showcases are planned for the next release and after. ## More detail diff --git a/docs-site/docs/api-reference/curation.mdx b/docs-site/docs/api-reference/curation.mdx index 01128997a..252fbfba4 100644 --- a/docs-site/docs/api-reference/curation.mdx +++ b/docs-site/docs/api-reference/curation.mdx @@ -6,7 +6,7 @@ title: Curation routes # Curation routes :::info[Experimental] -The curation subsystem is experimental in v0.5.0. It is real and tested, +The curation subsystem is experimental in v0.6.0. It is real and tested, but backward compatibility across releases is not guaranteed until it graduates. It ships opt-in behind the `curation` compose profile. ::: diff --git a/docs-site/docs/getting-started/introduction.mdx b/docs-site/docs/getting-started/introduction.mdx index eda744680..00bf76783 100644 --- a/docs-site/docs/getting-started/introduction.mdx +++ b/docs-site/docs/getting-started/introduction.mdx @@ -54,7 +54,7 @@ The curation subsystem is **domain-agnostic**: nothing about "what an item or a region of interest is" is hardcoded. It is **bring-your-own-model**: you supply the item detector, and optionally a VLM, a region detector and training data. It ships **opt-in** behind the `curation` compose profile and, -like the rest of the API, has **no authentication**. v0.5.0 keeps no +like the rest of the API, has **no authentication**. v0.6.0 keeps no compatibility shims: a renamed environment variable is rejected at startup, not translated. diff --git a/docs-site/docs/getting-started/quick-start.mdx b/docs-site/docs/getting-started/quick-start.mdx index fdd80c66f..00e994241 100644 --- a/docs-site/docs/getting-started/quick-start.mdx +++ b/docs-site/docs/getting-started/quick-start.mdx @@ -62,7 +62,7 @@ both `triton` and `opensearch` report `"ok": true`: ```json { "status": "ready", - "version": "0.5.0", + "version": "0.6.0", "api_version": "v1", "services": { "triton": {"ok": true, "detail": "is_server_live OK", ...}, diff --git a/docs-site/docs/guides/ingest-large-folder.mdx b/docs-site/docs/guides/ingest-large-folder.mdx index a80eaedd3..9e73b2213 100644 --- a/docs-site/docs/guides/ingest-large-folder.mdx +++ b/docs-site/docs/guides/ingest-large-folder.mdx @@ -443,6 +443,8 @@ Measured rates, all on an RTX A6000, from | 12 to 24 MP photos, `/ingest/batch`, CPU decode, detect and embed | 27.0 images/s | Decode once and the GPU decode | | The same photos with GPU decode | 35.0 images/s | Decode once and the GPU decode | | The same photos, GPU decode, detect only | 44.0 images/s | Decode once and the GPU decode | +| 100,000 public Open Images (up to 0.7 MP), one A6000, stopped and resumed once | 35.96 images/s | Scale ladder: 10k and 100k | +| 100,000 camera-original 20 MP JPEGs on a NAS, one A6000, stopped and resumed once | 21.00 images/s | Scale ladder: 10k and 100k | Host CPU per photo at bulk was 1.12 s on the CPU path and 0.34 s with GPU decode. diff --git a/docs-site/docs/operations/storage.mdx b/docs-site/docs/operations/storage.mdx new file mode 100644 index 000000000..3a8714ad9 --- /dev/null +++ b/docs-site/docs/operations/storage.mdx @@ -0,0 +1,77 @@ +--- +sidebar_position: 10 +title: Storage and sizing +--- + +# Storage and sizing + +**Put the OpenSearch data directory on local NVMe.** Do not put it on a network share (NFS, SMB, a NAS +volume), and keep it off the disk that holds your source images when that disk is shared. OpenSearch +syncs its transaction log on every bulk request, flushes segments and merges them in the background; +those are many small synced writes, which a local SSD absorbs and a network share does not under load. + +## What was measured + +One OpenSearch 3.6.0 node (2 GB heap, the project's index bodies and Lucene HNSW fields, random +1024-dimension vectors in documents shaped like real crop and image documents, 60,000 crop documents +plus 9,375 image documents per run, 4 bulk writers of 1,000 documents), data on local NVMe against +data on the NAS array (md RAID6, ext4), same host and same moment. Raw files: +`docs/benchmarks/write_path_*.json`; method and every number: `docs/design/ingest_write_path_investigation.md`. + +| Case | NVMe | NAS | +|---|---:|---:| +| Documents per second, nothing else running (4 writers) | 3,199 | 2,631 | +| Documents per second, 8 writers | 5,334 | 3,542 | +| Flush time per run (node counter) | 3.4 s | 14.8 s | +| Same run while two readers stream source photos from the same NAS (167 MB/s) | 2,511 docs/s, bulk p95 2.6 s | 1,106 docs/s, bulk p95 14.7 s, flush 26 s | + +Alone, the NAS costs about 18 percent of write speed. Sharing the array with the image reads +costs 2.3 times, and the slowest bulk request grows from 2.6 to 14.7 seconds. A 100,000-image +run at 36 images per second needs about 200 documents per second, so OpenSearch on a NAS keeps up +with ingest; the damage is the tail (stalls during merges and while other jobs use the array), and +every other consumer of that array pays for the merge traffic. + +## Sizing + +| Quantity | Measured | +|---|---:| +| Index size per 100,000 small images (about 0.7 MP, 4.9 crops each) | 8.9 GiB (about 91 KB per image) | +| Index size per 100,000 camera photos (20 MP, 2.4 crops each) | 4.4 GiB (about 47 KB per image) | +| Index bytes per crop | about 19.5 KB | +| Segments after a 100,000-image run (before any merge) | 128 to 159 | + +Plan for about 20 KB per crop plus the same again for merge headroom: a million small images is +about 90 GiB of index and needs about 180 GiB free. Merges rewrite data many times over (one +real index merged 68 GB in total to hold a few GB), so free space is a hard requirement, not a +comfort. + +## Moving the data directory to NVMe + +`OP_OPENSEARCH_DATA` in `.env` is either a Docker volume name (the default, `opensearch_data`) or +an absolute host path. To move an existing deployment: + +```bash +./openprocessor stop # or: make down +sudo mkdir -p /srv/opensearch-data +sudo docker run --rm -v _opensearch_data:/from -v /srv/opensearch-data:/to \ + alpine sh -c 'cp -a /from/. /to/' # is COMPOSE_PROJECT_NAME +sudo chown -R 1000:1000 /srv/opensearch-data # OpenSearch runs as uid 1000 +echo 'OP_OPENSEARCH_DATA=/srv/opensearch-data' >> .env +./openprocessor start +curl -s localhost:4607/_cluster/health # green, then check the document counts +``` + +Keep the old volume until the counts match, then remove it. + +## Ingest settings that depend on where the files are + +- `OP_INGEST_READ_THREADS` (default 8): threads per API worker that read a batch's source files in + parallel. One read after another from a share ran at 155 MB/s; 16 in parallel at 700 MB/s. +- `OP_INGEST_INFLIGHT_MB` (default 1024): decoded megabytes one worker holds. Photos the GPU + decode takes are counted by their encoded size, so a 16-photo request of 20 MP no longer fills + the whole budget. + +Index-level levers that were tried and are not recommended for bulk runs on this index type +(`refresh_interval` -1 or 30 s were 28 percent slower, because the HNSW graph is then built in +one large flush; `translog.durability: async` gained 3 to 12 percent and gives up the crash +guarantee) are in the investigation file. diff --git a/docs-site/package-lock.json b/docs-site/package-lock.json index 12d24b070..b8e0e9813 100644 --- a/docs-site/package-lock.json +++ b/docs-site/package-lock.json @@ -1,13 +1,13 @@ { "name": "openprocessor-docs", - "version": "0.5.0", + "version": "0.6.0", "lockfileVersion": 3, "requires": true, "license": "AGPL-3.0-or-later", "packages": { "": { "name": "openprocessor-docs", - "version": "0.5.0", + "version": "0.6.0", "license": "AGPL-3.0-or-later", "dependencies": { "@docusaurus/core": "3.10.2", diff --git a/docs-site/package.json b/docs-site/package.json index 71c71931c..1caf53c97 100644 --- a/docs-site/package.json +++ b/docs-site/package.json @@ -1,6 +1,6 @@ { "name": "openprocessor-docs", - "version": "0.5.0", + "version": "0.6.0", "private": true, "license": "AGPL-3.0-or-later", "scripts": { diff --git a/docs-site/sidebars.ts b/docs-site/sidebars.ts index 2c8ead9c4..e945d82b8 100644 --- a/docs-site/sidebars.ts +++ b/docs-site/sidebars.ts @@ -88,6 +88,7 @@ const sidebars: SidebarsConfig = { 'operations/training-and-promote', 'operations/monitoring', 'operations/release-acceptance', + 'operations/storage', 'operations/upgrade-and-backup', ], }, diff --git a/docs-site/src/data/roadmap.json b/docs-site/src/data/roadmap.json index 69de2e012..b57756fd9 100644 --- a/docs-site/src/data/roadmap.json +++ b/docs-site/src/data/roadmap.json @@ -3,28 +3,25 @@ "note": "Hand-maintained, not derived from an issue tracker. 'Shipped' means it is on main today; 'Planned' items are design work that has not landed and may change after measurement. Update this file when scope actually changes; never describe work that isn't planned.", "releases": [ { - "version": "v0.5.0", - "headline": "The monorepo release: backend and Cropwright in one repository, label confirmation, one-port gateway", + "version": "v0.6.0", + "headline": "Faster ingest and assist runs: selective VLM labeling, bulk ingest, measured GPU pipeline steps", "stage": "shipped", - "summary": "The Triton-backed core API and the opt-in /curation active-learning subsystem, now released together with the Cropwright frontend (frontend/) from one repository and one docs site. Adds label confirmation (VLM scope policy, detector class kept beside VLM labels, accuracy audit, audit-gated auto-promote), the gateway, and a refreshed dependency set.", + "summary": "The measured first half of the Triton throughput plan (decode once, uint8 inputs, FP16 PE image encoder, GPU decode, multi-GPU embedding, coalesced writes) plus assist runs, bulk and S3 ingest, detection-only ingest and a repository cleanup. Numbers are in docs/PERFORMANCE.md.", "items": [ - "Backend and the Cropwright web UI in one repository, built and pinned together; one docs site", - "Label confirmation: per-project VLM scope policy, the detector's own class kept beside VLM labels, a detector_disagreements review tab, an accuracy audit with Wilson 95% intervals, and an audit gate on auto-promote", - "Optional detector hint for VLM prompt packs (detector_hint_min_confidence_pct, off by default; +7.8 points overall accuracy on a COCO oracle)", - "Gateway: Grafana, Prometheus, OpenSearch Dashboards and MLflow on sub-paths of the single Cropwright port (OP_GATEWAY_SUBPATHS)", - "Breaking: the compose service yolo-api is now api; the unread ocr_det_* region profile fields are removed", - "/curation: 266 method routes in the generated OpenAPI contract", - "Refreshed dependencies: FastAPI 0.143 and Pydantic 2.14, SvelteKit 3 and TypeScript 6, MLflow 3.17, PyTorch 2.14.1 segmenter base, Playwright 1.63", - "Opt-in compose profiles: curation, segmenter, vlm, training, monitoring, benchmark, cropwright" + "Assist runs (/assist/runs): send a crop, box, selection, cluster, sub-cluster or filter to the VLM; cluster label proposals applied explicitly", + "Prompt-pack attributes replace the hard-coded make and model; a project calls no VLM until one is activated", + "Bulk ingest of a folder, manifest or S3 with a preview; detection-only ingest", + "Ingest speed-ups: decode once, uint8 inputs, FP16 PE image encoder, GPU decode, data-parallel PE encoding, coalesced writes, Lucene vector engine for new projects", + "Breaking: CROPWRIGHT_* env names are OP_CROPWRIGHT_*; /vlm/label_cluster and /vlm/verify_regions are replaced by /assist/runs; the evaluator has its own compose profile", + "Repository cleanup: guards, dead code removed, large files split, one typed env module, one docs source of truth" ] }, { - "version": "Next", + "version": "v0.7.0", "headline": "Triton throughput: keep images on the GPU", "stage": "next", - "summary": "A measured, wave-by-wave optimization plan for the inference and ingest pipeline. The goal is to decode each image once on the GPU and keep it resident across the models that need it, with every change behind a flag, a parity gate against today's outputs, and a p50/p99 benchmark gate. None of this has shipped.", + "summary": "The rest of the measured optimization plan: keep each decoded image resident on the GPU across the models that need it, with every change behind a flag, a parity gate against today's outputs and a p50/p99 benchmark gate. The 1M-image scale step is a goal here, not a claim. None of this has shipped.", "items": [ - "CPU-side fixes first: decode once, stop blocking the event loop, shorter Triton queue delays", "Smaller model outputs (end2end secondary detector, in-graph OCR argmax) and a GPU-resident OCR BLS", "GPU JPEG decode (DALI or nvJPEG, chosen by bake-off) and static Triton ensembles for /detect and /embed", "A GPU-resident curation ingest pipeline (BLS) that cuts crops and embeddings from the full-resolution frame", diff --git a/docs-site/static/img/screenshots/cropwright/assist-dialog-1600.png b/docs-site/static/img/screenshots/cropwright/assist-dialog-1600.png index 1598bea16..0b2d67e8d 100644 Binary files a/docs-site/static/img/screenshots/cropwright/assist-dialog-1600.png and b/docs-site/static/img/screenshots/cropwright/assist-dialog-1600.png differ diff --git a/docs-site/static/img/screenshots/cropwright/classes-1600.png b/docs-site/static/img/screenshots/cropwright/classes-1600.png index d70c6c917..8dea1ae76 100644 Binary files a/docs-site/static/img/screenshots/cropwright/classes-1600.png and b/docs-site/static/img/screenshots/cropwright/classes-1600.png differ diff --git a/docs-site/static/img/screenshots/cropwright/cluster-detail-1600.png b/docs-site/static/img/screenshots/cropwright/cluster-detail-1600.png index 35e1018ad..20903f715 100644 Binary files a/docs-site/static/img/screenshots/cropwright/cluster-detail-1600.png and b/docs-site/static/img/screenshots/cropwright/cluster-detail-1600.png differ diff --git a/docs-site/static/img/screenshots/cropwright/clusters-1600.png b/docs-site/static/img/screenshots/cropwright/clusters-1600.png index 2e6c2bfb5..cc44cf84c 100644 Binary files a/docs-site/static/img/screenshots/cropwright/clusters-1600.png and b/docs-site/static/img/screenshots/cropwright/clusters-1600.png differ diff --git a/docs-site/static/img/screenshots/cropwright/dashboard-1600.png b/docs-site/static/img/screenshots/cropwright/dashboard-1600.png index 3e5f749e0..9f3525fe6 100644 Binary files a/docs-site/static/img/screenshots/cropwright/dashboard-1600.png and b/docs-site/static/img/screenshots/cropwright/dashboard-1600.png differ diff --git a/docs-site/static/img/screenshots/cropwright/dataset-imports-1600.png b/docs-site/static/img/screenshots/cropwright/dataset-imports-1600.png index 4cf189774..e265cf7cd 100644 Binary files a/docs-site/static/img/screenshots/cropwright/dataset-imports-1600.png and b/docs-site/static/img/screenshots/cropwright/dataset-imports-1600.png differ diff --git a/docs-site/static/img/screenshots/cropwright/export-1600.png b/docs-site/static/img/screenshots/cropwright/export-1600.png index f3dc5a778..2c72e0738 100644 Binary files a/docs-site/static/img/screenshots/cropwright/export-1600.png and b/docs-site/static/img/screenshots/cropwright/export-1600.png differ diff --git a/docs-site/static/img/screenshots/cropwright/ingest-policy-1600.png b/docs-site/static/img/screenshots/cropwright/ingest-policy-1600.png index 387974676..be716a22b 100644 Binary files a/docs-site/static/img/screenshots/cropwright/ingest-policy-1600.png and b/docs-site/static/img/screenshots/cropwright/ingest-policy-1600.png differ diff --git a/docs-site/static/img/screenshots/cropwright/prompt-packs-list-1600.png b/docs-site/static/img/screenshots/cropwright/prompt-packs-list-1600.png index 5ca4b20b9..e8d6c8636 100644 Binary files a/docs-site/static/img/screenshots/cropwright/prompt-packs-list-1600.png and b/docs-site/static/img/screenshots/cropwright/prompt-packs-list-1600.png differ diff --git a/docs-site/static/img/screenshots/cropwright/region-profiles-list-1600.png b/docs-site/static/img/screenshots/cropwright/region-profiles-list-1600.png index ff8b6fef6..2e4fc1314 100644 Binary files a/docs-site/static/img/screenshots/cropwright/region-profiles-list-1600.png and b/docs-site/static/img/screenshots/cropwright/region-profiles-list-1600.png differ diff --git a/docs-site/static/img/screenshots/cropwright/reprocess-dialog-1600.png b/docs-site/static/img/screenshots/cropwright/reprocess-dialog-1600.png index 26010f237..6ea8f2493 100644 Binary files a/docs-site/static/img/screenshots/cropwright/reprocess-dialog-1600.png and b/docs-site/static/img/screenshots/cropwright/reprocess-dialog-1600.png differ diff --git a/docs-site/static/img/screenshots/cropwright/review-1600.png b/docs-site/static/img/screenshots/cropwright/review-1600.png index a8e66b8a7..eb7e1280c 100644 Binary files a/docs-site/static/img/screenshots/cropwright/review-1600.png and b/docs-site/static/img/screenshots/cropwright/review-1600.png differ diff --git a/docs-site/static/img/screenshots/cropwright/settings-keymap-1600.png b/docs-site/static/img/screenshots/cropwright/settings-keymap-1600.png index d4a30d5b2..69c82e379 100644 Binary files a/docs-site/static/img/screenshots/cropwright/settings-keymap-1600.png and b/docs-site/static/img/screenshots/cropwright/settings-keymap-1600.png differ diff --git a/docs/CURATION.md b/docs/CURATION.md index c4babf457..04d19f036 100644 --- a/docs/CURATION.md +++ b/docs/CURATION.md @@ -5,7 +5,7 @@ This page is a map. The user guides live in the documentation site page that explains it. Headings that code, compose files and scripts cite are kept, so those references still resolve. -> **Status: experimental for v0.5.0.** The subsystem is working and tested, but its +> **Status: experimental for v0.6.0.** The subsystem is working and tested, but its > API can still change between releases. It ships opt-in, behind Docker Compose > profiles. diff --git a/docs/PERFORMANCE.md b/docs/PERFORMANCE.md index a30016ba3..05a9dc4c0 100644 --- a/docs/PERFORMANCE.md +++ b/docs/PERFORMANCE.md @@ -2,10 +2,11 @@ Complete guide for optimizing FastAPI and Triton Inference Server performance. -> **Status (v0.5.0):** this guide covers the API-layer tuning, the measured results of the GPU and +> **Status (v0.6.0):** this guide covers the API-layer tuning, the measured results of the GPU and > Triton pipeline optimization milestone (summary table below; plan in > [`design/triton_pipeline_optimization_plan.md`](design/triton_pipeline_optimization_plan.md)), -> and the benchmark protocol. The scale ladder (10k, 100k, 1M images) is not in this guide yet. +> and the benchmark protocol. The scale ladder at 10k and 100k images is in +> [Scale ladder: 10k and 100k](#scale-ladder-10k-and-100k); the 1M step is planned for the next release. --- @@ -403,8 +404,8 @@ ingested at **13.39 images/s** (0 failed, 1.37 crops per image, batch p50 10.0 s 4,000-image mixed set that is half 12 to 20 MP JPEGs ingested at 4.90 images/s (about 15 for the small photos, about 2.5 for the large ones; decode and resize dominate). Those runs used the narrow vehicle detector; the full-vocabulary -policy rows above are computed, not measured. The v0.5.0 re-measurement on the pinned -2,000-image set is in "v0.5.0 baseline" below. +policy rows above are computed, not measured. The 0.5.0 re-measurement on the pinned +2,000-image set is in "0.5.0 baseline" below. ### Measured VLM labeling speed @@ -510,10 +511,10 @@ Triton compute_infer, A all: pe_image_encoder 1,590 s total for 33,388 inference A second baseline on a set of real high-resolution (about 20 MP) photos was measured privately; its numbers are not published. It showed the same pipeline is decode and I/O bound at that image size (the GPU was mostly idle), which is what the GPU decode and crop-at-model-size waves in the optimization plan target. Single run per row; a 3-round median is planned before the optimization work starts. -### v0.5.0 baseline (set A, public COCO, 2,000 images) +### 0.5.0 baseline (set A, public COCO, 2,000 images) The Wave 0 baseline of `docs/design/triton_pipeline_optimization_plan.md`, measured on the -published v0.5.0 images before any optimization (issues #45 and #40). Raw JSON: +published 0.5.0 images before any optimization (issues #45 and #40). Raw JSON: `docs/benchmarks/v050_baseline.json` (environment, ingest, Triton statistics, perf_analyzer points, endpoints, VLM) and `docs/benchmarks/v050_storage.json` (the storage run); idle-CPU readings in `docs/benchmarks/v050_idle_background.json`. (The VLM model label in the raw JSON metrics is replaced by `catalog-default`.) Every @@ -524,7 +525,7 @@ unless stated. **8.9 images/s** through `POST /curation/projects/{project}/ingest/upload` and **9.2 images/s** through `.../ingest/batch`, with the GPU 91 percent busy. The run-to-run spread is under 6 percent. 90 percent of the GPU time is -the PE image encoder (7.4 embeddings per image). That engine is FP32, and a fresh v0.5.0 +the PE image encoder (7.4 embeddings per image). That engine is FP32, and a fresh 0.5.0 install builds the same FP32 engine (see Caveats), so this is the true "before" for the default install. @@ -543,11 +544,11 @@ default install. Software under test: the images pinned by `images.lock` (API `openprocessor:0.5.0` `sha256:b0af838e...`, Triton `openprocessor-triton:0.5.0` `sha256:1ac56ac9...`, -OpenSearch `3.6.0`), installed with `setup-openprocessor.sh --version v0.5.0 --tiers +OpenSearch `3.6.0`), installed with `setup-openprocessor.sh --version 0.5.0 --tiers core,curation --local-only --bind 127.0.0.1 --skip-models` into an isolated compose project on its own ports, GPU 0 only. The VLM worker was stopped for the measured ingest runs. OpenSearch heap 8 GB, no replicas, one node. Models: TensorRT engines built on 2026-09-26 and copied -from an earlier v0.5.0 install on the same host. The PE image encoder engine is FP32 +from an earlier 0.5.0 install on the same host. The PE image encoder engine is FP32 (1.27 GB); every other engine is consistent with FP16 by file size. #### Method @@ -723,7 +724,7 @@ already seen is much faster (27 to 31 crops/s) because of vLLM's multimodal cach image) that build fails at parse time on both `Einsum` nodes: `IEinsumLayer must have all inputs of same type. Input 1 has type Half and input 0 has type Float` (`/visual/Einsum`: the FP16 graph rewrite converted its float `Constant` to half but left the explicit float32 `Cast` that - feeds the other input). The installer then retries from the FP32 ONNX, which builds. A fresh v0.5.0 + feeds the other input). The installer then retries from the FP32 ONNX, which builds. A fresh 0.5.0 install therefore ends up with the same FP32 engine as this baseline, so the baseline is the true "before" for the default install. The fix and its measured effect are in the next section. - OpenSearch data and the upload store live on a RAID array (Docker data root), not on NVMe. @@ -746,7 +747,7 @@ already seen is much faster (27 to 31 crops/s) because of vLLM's multimodal cach The first change of `docs/design/triton_pipeline_optimization_plan.md` after the baseline above: the PE image encoder is built as a real FP16 engine. Same pinned set, same harness -(`scripts/bench/baseline_suite.py --phases env,ingest`), same host and GPU 0, same published v0.5.0 +(`scripts/bench/baseline_suite.py --phases env,ingest`), same host and GPU 0, same published 0.5.0 images; the only difference is the engine behind `pe_image_encoder`. Raw JSON: `docs/benchmarks/v060_fp16_pe.json` (ingest, Triton statistics, GPU) and `docs/benchmarks/v060_fp16_pe_engine.json` (engine size, build time, parity). Medians with the @@ -759,7 +760,7 @@ stay equivalent to the FP32 engine (cosine similarity 0.9996 mean, 0.9976 minimu #### Root cause and fix -The v0.5.0 FP16 bake (`bake_fp16_onnx` in `export/trt_utils.py`) never produced a buildable graph on +The 0.5.0 FP16 bake (`bake_fp16_onnx` in `export/trt_utils.py`) never produced a buildable graph on TensorRT 11.1, and the installer hid it by retrying from FP32. Two defects, both found by building the baked ONNX with `trtexec`: @@ -783,7 +784,7 @@ Caveats). #### Engine and parity -| Item | FP32 (v0.5.0) | FP16 (this change) | +| Item | FP32 (0.5.0) | FP16 (this change) | |---|---:|---:| | Plan size | 1,272,018,292 B | 644,778,380 B (0.51x) | | `trtexec` engine build | 38 s | 81 s | @@ -847,7 +848,7 @@ The earlier private FP16 measurement of about 170 embeddings/s (5.9 ms) matches - Parity covers 200 public images on whole-frame preprocessing. Retrieval and clustering gates on labelled data (plan section 7) are open. - The fix bakes FP16 for every model that calls `bake_fp16_onnx`. On TensorRT 11.1 the MobileCLIP - image encoder's v0.5.0 bake also failed at parse (`ElementWiseOperation DIV must have same input + image encoder's 0.5.0 bake also failed at parse (`ElementWiseOperation DIV must have same input types`); with the reconcile pass it parses, but the build stopped at `Could not find any implementation for node .../reparam_conv/Conv + PWN(...)` (#224). The failing layers are the depthwise 7x7 stride-2 convs with a channel multiplier of 2 (the three @@ -1064,8 +1065,8 @@ reference: [parity_references.md](benchmarks/parity_references.md). #### FP16 against FP32 PE engine -Same isolated stack (the published v0.5.0 images, GPU 0, own ports, no VLM), only the file behind -`pe_image_encoder` changes. The FP32 plan is the one of the v0.5.0 baseline (1,272,018,292 B). The FP16 plan +Same isolated stack (the published 0.5.0 images, GPU 0, own ports, no VLM), only the file behind +`pe_image_encoder` changes. The FP32 plan is the one of the 0.5.0 baseline (1,272,018,292 B). The FP16 plan is built through the fixed installer path (`export/trt_utils.py` bake, then `trtexec` with the shapes of `scripts/lib/model_setup.sh`): 641,185,140 B, built in 95 s. Each configuration was captured twice, once with 4 and once with 1 upload request in flight (run 1 and run 2), and the FP32 plan was captured @@ -1132,7 +1133,7 @@ Reading the numbers: #### Caveats -- One host, one GPU shared with other projects, the published v0.5.0 images; the container image ids are +- One host, one GPU shared with other projects, the published 0.5.0 images; the container image ids are in each `capture.json`. The summaries of the four comparisons are in `docs/benchmarks/parity/`. - The VLM gate and the label agreement of a registry-labelled project are not exercised: no VLM is configured and the project has no class registry, so stored class names are empty. The label gates here @@ -1186,8 +1187,8 @@ parentheses. | GPU | one RTX A6000 (GPU 2, 300 W cap, 48 GB), driver 615.71.09, CUDA 13.4; foreign SM load on it 0.0 percent in every sample of all 198 measured runs | | Other GPUs | GPU 0 holds another project (16 GB resident, up to 95 percent utilization during one of my windows), GPU 1 another (7 GB): neither was used | | Host | 2 x Xeon E5-2680 v3, 48 logical cores, 504 GB RAM, load average 5 to 59 over the runs (other projects), whole-host CPU busy 10 to 77 percent (columns below) | -| Stack | published v0.5.0 images (`images.lock` digests), isolated compose project, own ports, loopback only, OpenSearch heap 8 GB, no replicas, 32 API workers, curation workers and VLM not running | -| PE engine | FP16, baked with the fixed `export/trt_utils.py` and built with `trtexec` as in Wave 1: plan 644,771,508 bytes; all other engines as installed by v0.5.0 | +| Stack | published 0.5.0 images (`images.lock` digests), isolated compose project, own ports, loopback only, OpenSearch heap 8 GB, no replicas, 32 API workers, curation workers and VLM not running | +| PE engine | FP16, baked with the fixed `export/trt_utils.py` and built with `trtexec` as in Wave 1: plan 644,771,508 bytes; all other engines as installed by 0.5.0 | | Set A | the pinned COCO set (2,000 images, 640 x 480 median 166 KB); sweeps use its first 1,000 images, the headline rows all 2,000 | | Set B | 500 public photos from the Wikimedia Commons featured-picture category, CC0, public domain, CC BY and CC BY-SA, 12.0 to 24.0 MP (median 17.0, mean 17.4), 5.2 GB (mean 10.4 MB), pinned in `scripts/datasets/manifests/commons_hires_500.json` (manifest sha256 `c013c1f9...`, seed 20261010, fetched with `scripts/datasets/fetch_hires_set.py`) | | Protocol | the Wave 1 protocol: fresh project per repetition, warm-up project first, 3 repetitions, policy `all`; clients are threads of one Python process; cluster training and the storage merge are skipped in these runs (`--skip-cluster --skip-settle`) so that the window is the ingest alone | @@ -1572,7 +1573,7 @@ those costs, fixes the ones that measure as worth fixing, and repeats the Wave 0 Medians with the minimum and maximum of 3 repetitions in parentheses. **Method.** The Wave 0 stack and protocol on GPU 2 of the same host (an isolated compose project, -loopback ports, the published v0.5.0 images with the branch's `src/` mounted over `/app/src`, +loopback ports, the published 0.5.0 images with the branch's `src/` mounted over `/app/src`, the FP16 PE plan of the Wave 0 reference, plan sha256 `b9a6f397...`, 32 uvicorn workers, a fresh OpenSearch for each series). Set A is the pinned COCO set (first 1,000 images, `/ingest/upload`, 32 images per request), set B the 500 photos of `commons_hires_500.json` (`/ingest/batch`, 8 images per @@ -1801,13 +1802,13 @@ Runtime backend (`output0`, used by the region detector) and the bake-off engine #### Method -One RTX A6000 (GPU 2), the published v0.5.0 images with the checkout's `src/` mounted over `/app/src`, +One RTX A6000 (GPU 2), the published 0.5.0 images with the checkout's `src/` mounted over `/app/src`, an isolated compose project on loopback, OpenSearch 8 GB heap, 32 API workers, a fresh stack per configuration, policy `all`. Set A: the pinned COCO set (2,000 images, `coco_bench_2000.json`), `scripts/bench/baseline_suite.py --phases env,ingest --reps 3 --skip-cluster --skip-settle` (32 images per request, 4 clients; upload and batch). Set B: the 500 pinned photos, `scripts/bench/ingest_sweep_suite.py --phases sweep --sweep-points 8x16 --modes batch`. Configurations: **base** is `origin/main` with the -detector plan of the v0.5.0 install (Sep 26 build) and an FP16 PE plan from `build_pe_trt.sh`; +detector plan of the 0.5.0 install (Sep 26 build) and an FP16 PE plan from `build_pe_trt.sh`; **detector uint8** is the first commit of this change (PE still FP32); **final** is both. The uint8 detector plan was built with the real exporter path (`export_models.export_trt_end2end`), the PE plan with `build_pe_trt.sh`, both on the TensorRT 11.1 of the Triton image. @@ -1911,7 +1912,7 @@ Three findings, in order of importance. crop rectangles are identical on 98.7 percent of items against the control and 99.1 to 99.2 percent between two runs of the same engine. 2. **Rebuilding the detector engine, with no input change, fails the gates against `ref-fp16`.** - The v0.5.0 detector plan (Sep 26) was built from an ONNX baked before the FP16 cast fix of the FP16 + The 0.5.0 detector plan (Sep 26) was built from an ONNX baked before the FP16 cast fix of the FP16 PE section; the plan rebuilt today from the same weights moves 6.5 percent of images beyond the box gate, 2 confident boxes go unmatched, and crop-vector agreement drops to 0.978 (1st percentile). This is independent of the input type (the "rebuilt FP32" row has no uint8 in it) and applies to every @@ -1957,7 +1958,7 @@ shipped**: no setting, no compose change, no client code. The implementation and commits on the local branch `perf/shared-memory-impl` (`4de40001`, `8e20afe5`), for reference only. **Method.** The isolated stack of the uint8 section (GPU 2, foreign load 0.0 percent in every sampled -run, published v0.5.0 images with the branch `src/` mounted, uint8 detector and PE plans, PE plan sha256 +run, published 0.5.0 images with the branch `src/` mounted, uint8 detector and PE plans, PE plan sha256 `a4feb7df...`, a fresh project per repetition, policy `all`), one extra tmpfs volume mounted as `/dev/shm` in both containers and `--allow-client-shm=true` on Triton. Wire and shared memory were the same code with the transport switch off or on, the API container recreated between launches, arms interleaved (wire, shm, shm, @@ -2731,7 +2732,7 @@ does the packing across concurrent requests; the fix is to have concurrent reque |---|---| | GPU | RTX A6000, GPU 2, 0.0 percent foreign SM share in every counted repetition; another project held GPU 0 at 100 percent and was not touched. SM clock 1,416-1,444 MHz (c16), 1,449-1,482 MHz (c4) | | Host | 48 logical cores shared with other projects; 1-minute load 5-9 on the c16 and c4 repetitions, 12-20 at c32, 14-25 on set B; host 12-15 percent busy (set A, c4 and c16) | -| Stack | project `xbatch`: Triton, API (the published v0.5.0 image with this checkout's `src/` mounted), OpenSearch 8 GB heap, 32 API workers, loopback ports; the uint8 FP16 PE plan of `ref-u8fp16` (sha256 `a4feb7df...`), `pe_image_encoder` dynamic batcher `preferred_batch_size: [4, 8, 16, 32]`, `max_queue_delay_microseconds: 15000` (unchanged) | +| Stack | project `xbatch`: Triton, API (the published 0.5.0 image with this checkout's `src/` mounted), OpenSearch 8 GB heap, 32 API workers, loopback ports; the uint8 FP16 PE plan of `ref-u8fp16` (sha256 `a4feb7df...`), `pe_image_encoder` dynamic batcher `preferred_batch_size: [4, 8, 16, 32]`, `max_queue_delay_microseconds: 15000` (unchanged) | | Load | set A: 2,000 COCO images, `POST /ingest/batch`, 32 images per request, policy `all`, 1 warm-up and 3 counted repetitions (`ingest_sweep_suite.py --phases env,sweep --sweep-points 32x16,32x4 --modes batch --skip-cluster --skip-settle`); set B: 500 photos, 8 per request, 16 clients; the embed bench is `scripts/bench/embed_missing_bench.py` (500 images ingested, then 3 rewrites of every crop and frame vector) | #### Measurement first: PE batch sizes Triton sees today (set A, unchanged ingest code) @@ -2973,7 +2974,7 @@ finish at 100,000 and HDBSCAN was not run there (see above), so only `ivf` belon Question: does decoding JPEGs on the GPU, and building each model's input there, speed up ingest? The full design, the dataflow map and the implementation plan are in [design/gpu_decode_plan.md](design/gpu_decode_plan.md); this section keeps the numbers. One RTX -A6000 (GPU 2; SM86 has no hardware JPEG decoder), the published v0.5.0 images, the `ref-u8fp16` +A6000 (GPU 2; SM86 has no hardware JPEG decoder), the published 0.5.0 images, the `ref-u8fp16` engines, an isolated compose project, only this work's processes on GPU 2 during every timed run. Tools: `scripts/bench/gpu_decode_bench.py`, `scripts/bench/gpu_decode_parity.py`. @@ -3461,6 +3462,197 @@ Reproduce: --pipeline-cpu --concurrency 4,16,32 --reps 3 --out transport.json ``` +### Scale ladder: 10k and 100k + +Three runs of the bulk ingest runner ([Bulk ingest runner](#bulk-ingest-runner)) at larger sizes than the 2,000-image +baseline: 10,000 public COCO images, 100,000 public Open Images, and 100,000 camera-original photos. All three +use the 0.6.0 defaults (policy `all`, the 3-stage index policy defaults, 16 images per request, 16 requests in +flight, FP16 PE engine, GPU decode on) and a fresh project. The 1M-image step is not run (planned for the next release). + +#### Sets and environment + +| Run | Images | Source | +|---|---|---| +| coco-10k | 10,000, all under 1 MP, 1.65 GB | public COCO train and val 2017, seed 20261009 (`--bench-set 10000`, same command as the 2k pin) | +| oi-100k | 100,000, each at most 1,024 px (about 0.7 MP) | public Open Images v7 train, every image CC BY 2.0, seed 20261009; manifest sha256 `b9deca207878b569...` | +| hires-100k | 100,000, median about 20 MP, about 6.8 MB each | a private collection of 20 MP camera-original JPEGs, not redistributable; aggregates only | + +One RTX A6000 (GPU 2) for Triton, VLM stopped. OpenSearch data and the source images are on a NAS (RAID6 over a +network share), Docker data is on the NAS too, and the host is shared with other workloads. One earlier attempt +window was invalid because the NAS saturated on IO from another job; the runs below are the repeat. Each run +was stopped once with a forced cancel part-way and resumed with the same source (see Stop and resume). + +#### Results + +| Measure | coco-10k | oi-100k | hires-100k | +|---|---:|---:|---:| +| Throughput, images/s | 29.04 to 29.08 | 35.96 | 21.00 | +| Share of the 2k baseline (29.0 images/s) | 100 % | 124 % | 72 % | +| Wall time, ingest | 311 s (run A, one leg); 155 s second leg (run B) | 2,781 s | 4,761 s | +| Legs (forced stop, then resume) | 2 (run B) | 1,384 s + 1,397 s | 2,582 s + 2,179 s | +| Failures | 0 | 2 (decode, see below) | 0 | +| Duplicate images | 0 | 0 | 2,120 | +| Crops indexed and embedded | 64,415 | 490,804 | 236,689 | +| Crops per stored image | 6.44 | 4.91 | 2.42 | +| Crops embedded per second | about 187 | about 177 | about 50 | +| Cluster pass (CPU, after the last ingest) | 269 s at 64,415 crops | 2,042 s at 490,804 | 1,802 s at 236,689 | +| Index size on disk | not measured (preview estimate 653 MB) | 8.86 GiB | 4.35 GiB | +| Index bytes per input image | not measured | 95.2 KB | 46.7 KB | +| Index bytes per crop | not measured | 19.4 KB | 19.7 KB | +| Peak GPU utilization and memory (card 2) | not recorded | 100 % and 21.1 GiB | 100 % and 25.8 GiB | +| Mean GPU utilization over the samples | not recorded | 94 % | 51 % | +| OpenSearch heap peak, segments peak | not recorded | 79 %, 159 | 79 %, 128 | +| Peak host CPU and memory | not recorded | 90 %, 152 GB | 79 %, 159 GB | + +Images per second is images processed in the whole run divided by the ingest wall time including both legs +(the legs include the cost of the forced stop and the restart). The 29.0 baseline is the 2k pin on the same +build. Index bytes are the OpenSearch store size the driver sampled at the end of the run; the 2,120 duplicates in hires-100k are not stored, so its per-unique-image figure is 47.7 KB. +The two decode failures in oi-100k were fixed after this run by the Pillow header fallback (#333); the run was +not repeated. +The 10k cluster pass is the `cluster_residuals` stage (4 m 29 s); the other stages in it took 2 s or less. +The bytes per crop are the same in both 100k runs (about 19.5 KB), so the index grows with the crop count, +not the pixel count: a 20 MP photo costs the index less than a 0.7 MP one because it holds fewer objects. + +#### Throughput over time + +From the 15 s progress polls in `scale/oi-100k.jsonl` and `scale/hires-100k.jsonl` (images stored per second +over a rolling 5-minute window, per leg; a leg's first minutes include model and cache warm-up). + +| Run and leg | Quarter 1 | Quarter 2 | Quarter 3 | Quarter 4 | 5-min window min, median, max | +|---|---:|---:|---:|---:|---| +| oi-100k leg 1 | 37.0 | 36.8 | 36.9 | 36.2 | 35.9, 36.9, 37.6 | +| oi-100k leg 2 | 36.6 | 37.3 | 36.3 | 31.9 | 32.1, 36.6, 37.7 | +| hires-100k leg 1 | 22.7 | 23.5 | 19.1 | 12.8 | 4.6, 19.5, 30.4 | +| hires-100k leg 2 | 23.5 | 22.4 | 21.5 | 23.7 | 15.1, 22.4, 28.6 | + +- **oi-100k does not decay.** Leg 1 is flat to within 2 percent over 1,368 s (37.0 to 36.2 images/s). Leg 2 is flat for + three quarters and drops to 31.9 in the last quarter, as the index passed 159 segments with merges running + (the sampler saw three merges in flight in the second half); the whole-run rate of 35.96 includes it. +- **hires-100k is noisier and leg 1 decayed.** Leg 1 held 23 images/s for 20 minutes, then fell to 19, 16 and + 9 images/s over the last ten minutes before the forced stop (the minimum 5-minute window was 4.6). Leg 2, after the + restart and on the same index, held 21.5 to 23.7 for its whole length. A restart that cures the slowdown points + at something that builds up in a long-running process or on the host (the leg 1 GPU utilization also fell to 11 to 51 percent + in that stretch) rather than at index size, because leg 2 started with the bigger index and did not slow down. This run + cannot say which. The shared host and the NAS are both candidates (see Caveats). + +#### Stop and resume + +| Run | Stop | Resume | Result | +|---|---|---|---| +| coco-10k run A | cancel sent at about 7 % (the threshold used the growing discovery count; fixed in the driver) | restart returned the same run id, 944 images skipped | 10,000 stored, 0 failed, 29.08 images/s | +| coco-10k run B | forced cancel at 5,232 stored | the same run resumed (attempts 2), 5,488 skipped from the ledger | 10,000 stored, 0 failed, 29.04 images/s, 155 s second leg | +| oi-100k | forced cancel at 50,000 (50,654 stored by the time it took effect) | skipped 50,654, no re-embedding | 99,998 stored plus 2 failed | +| hires-100k | forced cancel at 50,000 (50,272 stored) | skipped 50,272 | 97,880 stored plus 2,120 duplicates, 0 failed | + +In every case the stop and the restart changed no count: images stored plus skipped plus duplicates plus failed +equals the input exactly, and the project holds 99,998 images and 490,804 crops (oi-100k), 97,880 images and +236,689 crops (hires-100k), all embedded (`n_not_embedded` 0, `n_embed_failed` 0). `resumed` is only in the +response of the start call; `attempts` 2 on the stored run is the record of a resume. + +#### Where the time goes + +`stage_s` in the run record is cumulative worker-seconds summed over every concurrent worker, not wall seconds. +Dividing a stage by the run's `elapsed_s` gives its average concurrency; dividing by the 16 requests in flight gives +a loose share. Stages overlap (a request's time includes the stages inside it), so the columns do not add to 100 %. +For the resumed runs the record's `elapsed_s` (1,394.7 s, 2,176.7 s) matches the last leg, so these are the last +leg's figures. + +| Stage (worker-s) | coco-10k | oi-100k | hires-100k | +|---|---:|---:|---:| +| `batch_requests` (whole request) | 4,899 | 21,813 | 33,999 | +| as average concurrency (over `elapsed_s`) | 15.7 | 15.6 | 15.6 | +| `embed` | 51,662 | 171,182 | 35 | +| `opensearch_write` | 5,042 | 28,971 | 49,432 | +| `admission_wait` | 0.04 | 0.2 | 13,919 | +| `decode` | 47 | 500 | 15 | +| `letterbox` | 26 | 590 | 7 | +| `whole_frame_preprocess` | 59 | 508 | 4 | +| `jpeg_encode` | 30 | 220 | 1 | +| `blur` | 28 | 309 | 3 | +| `triton_request`, `hash`, `crop`, `doc_build`, `bulk_serialize`, `pe_preprocess` (each) | under 45 | under 220 | under 18 | + +- **All three runs keep 16 requests in flight all the time** (15.6 to 15.7 average), so the client side never + starves; what differs is what the requests wait on. +- **The small-image runs wait on the GPU.** `embed` is 3.5 to 5.7 worker-seconds per image, summed over the + crops of a request waiting in the Triton queue (about 123 waiters at once in oi-100k), while the CPU stages are + each under 1 percent of it. Mean GPU utilization is 94 % and 100 % is the peak. The crops-per-second figures + fit the same picture: 187 (coco-10k) and 177 (oi-100k) crops embedded per second on different image + sets. That is why oi-100k beats the 29.0 baseline (124 %): its images carry 4.91 crops against 6.44, so + at the same crop rate it stores 22 percent more images per second. The images-per-second figure is a property of the + crops per image as much as of the pipeline. +- **The 20 MP run does not wait on the GPU.** `embed` is 35 worker-seconds, mean GPU utilization is 51 % (median 53), + and the crop rate is 50 per second, a quarter of the small-image runs. The time sits in `opensearch_write` + (49,432, 22.7 workers) and `admission_wait` (13,919, 6.4 workers). `admission_wait` is time spent waiting for + the in-flight bytes budget (`OP_INGEST_INFLIGHT_MB`): at 6.8 MB per photo and 16 images per request, a share of requests waited for + memory budget, about 0.28 s per image. At 21 images/s the runner moves about 143 MB/s of source files off the NAS. +- **What the stage timers cannot say.** In the 20 MP run `decode` is 15 s and `embed` 35 s, which are not credible + as the cost of the work, so on the GPU decode path these timers do not cover the work and its time shows up + in the stage that awaits it (`opensearch_write` here). Reading `opensearch_write` literally as the OpenSearch + commit time would contradict probe 5 (one writer sustains the equivalent of 137 images/s). The defensible + statements are: the GPU is idle half the time, so the limit is upstream of it (NAS reads of 6.8 MB files, + the GPU decode and the in-flight bytes cap, or a combination) and the OpenSearch write stage is not shown to be + the limit. Storage is on the same NAS as the sources, so a NAS-bound reading fits but is not proven; a run with + the sources on local NVMe, or the same photos downsized, would separate them. +- **The small-image runs are not storage-write bound.** `opensearch_write` is 0.5 to 0.6 worker-seconds per image + in coco-10k and oi-100k (21 workers in oi-100k) with the GPU saturated, and the throughput matches the crop rate + of the GPU-bound baseline, so the NAS-hosted OpenSearch kept up with 177 documents per second of crops at + 100,000 images (heap peak 79 %, 159 segments). + +#### Caveats + +- One card, one run per configuration; the 100k runs are single observations, not medians. The coco-10k pair + (29.08 and 29.04) is the only repeat, and it agrees within 0.2 percent. +- The host and the NAS are shared with other workloads. An earlier first attempt was lost to a NAS IO saturation + episode. The hires-100k leg 1 slowdown may be the same kind of interference; the data cannot exclude it. +- The hires-100k set is private and cannot be reproduced; only its aggregates are published. Its crops + per photo (2.42) depend on its content and do not transfer to other photo sets. +- `stage_s` for a resumed run covers only the final leg (see above), and the stage timers cover different work + on the CPU and GPU decode paths. +- The cluster pass runs on the CPU and shares the host. Its time per crop was 4.2 ms in oi-100k and 7.6 ms in + hires-100k, and the cause of the difference was not found. +- The coco-10k run recorded no GPU, heap or index-size samples; the preview route estimated 653 MB for it. +- Both 100k indexes were left unmerged (128 and 159 segments); size on disk after a force merge was not measured. + +#### Bottlenecks and next steps + +1. **Decode once on the GPU, stay on the GPU (#40).** The 20 MP run leaves the GPU half idle while the host + moves 6.8 MB files and decodes; the small-image runs are GPU bound on embedding, so keeping the decoded frame + resident removes the repeated host work. +2. **Multi-size views of one decode (#219).** Work crops at model size for the VLM and region consumers so the + embed and region stages share one decode. +3. **Faces and OCR on resident crops (#220).** Ingest does not call them today (probe 3); on resident crops they cost + a model pass, not a decode. +4. **ortloom head-to-head against Triton on the ingest-path models (#231)**, since the small-image runs are bound by + the embedding model's throughput. +5. **The embed reprocess pass stops at about 58 percent GPU (#259)**, the same underuse the 20 MP ingest shows. +6. **Raise or tune the in-flight bytes gate (`OP_INGEST_INFLIGHT_MB`)** for photo-sized sets and re-run hires-100k + with the sources on local storage, to split the NAS from the pipeline. Not run in 0.6.0. +7. The 1M-image step follows once 1 to 5 land; at 36 images/s it would be about 7.7 hours of ingest and a + cluster pass that grows faster than linearly (269 s at 64k crops, 2,042 s at 491k). + +#### Reproduce + +```bash +# Public Open Images v7 train, seed 20261009, 100,000 images, CC BY 2.0 (checksum-pinned; the 39 MB pin is not in git) +python scripts/datasets/fetch_openimages_bench.py --out data/bench100k --n 100000 --seed 20261009 \ + --cache-dir data/oi-cache --manifest data/oi_bench_100000.json +python scripts/datasets/fetch_openimages_bench.py --verify-only --out data/bench100k --manifest data/oi_bench_100000.json +# pin: manifest_sha256 b9deca207878b569b7cd6e649dbd68d1b34b70db0a0fb0f5d43cc4d554da9f50 +# (scripts/datasets/manifests/openimages_bench_100000.sha256) + +# 10k COCO set (same command as the 2k pin, larger --bench-set) +# the driver: ingest, one forced cancel at 50 %, resume, 15 s polls, 30 s resource samples, cluster pass timing +.venv/bin/python scripts/bench/scale_run.py --api http://127.0.0.1:4603 \ + --project oi-100k --folder /data/bench100k/images --label oi-100k \ + --stop-at 0.5 --baseline-img-s 29.0 --out-dir artifacts_local/scale +# --manifest replaces --folder for a manifest source +``` + +The driver writes `