[davidamacey]

Closes #282
Closes #279
Closes #292

The question this answers

"Is the VLM running on every ingest? If I just want to ingest images, detect cars and
people, cluster, and label clusters with custom names, do I have to wait for the VLM, or can
I use it selectively?"

Before: ingest never waited for the VLM, but with OP_VLM_URL set every project that never
activated an endpoint inherited the env VLM and curation-vlm-worker labeled every embedded
item in the background (scope all). A pure detection-only project was not selectable
(lazy mode still stored the whole-frame vector; clustering had two triggers and no off
switch). After: a project calls no VLM until it activates one, detection-only is one policy,
clustering has a per-project switch, and the selective VLM run is a documented, checked
recipe.

Behavior change (CHANGELOG, Changed): a project calls no VLM until it activates one (#282)

Decision (the safest of the two options in the brief): explicit state in the activation
doc, with a one-time pin for old projects.

• active_vlm_endpoint() returns None for a project with no activation (it used to return
  the env endpoint). GET .../vlm/endpoints/active answers source: "off"; Cropwright
  already renders what the backend serves ("off: no VLM runs for this project"), so no
  client re-derivation was added. PUT /settings defaults.vlm: null now means off.
• New projects: lifecycle.create writes an explicit off activation while the project is
  still building (src/services/projects/vlm_default.py).
• Projects created by an earlier release: legacy_repair (runs at API start, under the
  per-project flock, active projects only) pins a project with no activation doc to what it
  inherited: env when OP_VLM_URL is set, else off. Their effective behavior does not
  change, and a later OP_VLM_URL grants nothing. Idempotent: a project with any
  activation doc is never touched. The pin is a quiet write (no config.changed event).
• Caveat, stated in the docs: the default project of a fresh install is pinned by the same
  rule on first start (env if the installer set OP_VLM_URL); every project created after
  that starts off.
• The continuous VLM worker (projects_with_active_vlm, current_want) and the auto-label
  VLM stage (run_labeler / resolve_run_vlm) go through active_vlm_endpoint(), so they
  respect it. A per-run vlm=<name> still works and does not change the project's setting.
• scripts/release/acceptance_run.sh vlm_label phase now activates the env VLM first.

Detection-only mode and the clustering switch (#279)

Minimal shape, consistent with the existing embedding.mode values:

• embedding.mode: "none": store boxes with no vector of any kind (detections deferred,
  no whole-frame vector, the GPU-decode path drops the whole-frame vector it computes in its
  one pass). Labeled items from a dataset import still embed (the existing rule). lazy is
  unchanged (the whole-frame vector is still stored).
• clustering.automatic: false (new ClusteringPolicy, default true): ingest and the
  embed reprocess scope place no item into a cluster, curation-cluster-refresh skips
  the project, and the auto-label worker's idle retrain skips it. Explicit clustering
  (/clusters/*, auto-label jobs) always runs. Later embed/cluster on demand:
  auto_label/start?embed_missing=true&train_clusters=true.
• Cropwright's ingest-policy page offers the none mode and a "Cluster automatically"
  checkbox.

Cluster-refresh flags from the environment (#292)

OP_CLUSTER_REFRESH_INTERVAL_S, _GROWTH_THRESHOLD, _MAX_DEFERRAL_S, _RETRY_BACKOFF_S,
_AUTO_LABEL are the flag defaults; a flag still overrides. The service has
env_file: .env, so no compose change was needed (cli-and-tooling's compose and CLI
files are untouched). Documented in env.template, configuration/advanced.mdx and
pipeline-modes. No env var was renamed or removed, so retired_env.py is unchanged.

Gap found by running the selective recipe, fixed

Running the recipe live, a VLM run against an unreachable endpoint finished completed with
predicted: 0: indistinguishable from "nothing to do". The auto-label VLM stage now reports
failed (VLM request failures) and unreadable (crops whose image could not be read) and
fails the job with those counts when every selected crop failed.

Live recipe (throwaway public project, deleted afterwards)

Run 2026-10-15 on the opmono API with 40 COCO images copied from coco-bench-2k into a
throwaway cwlife-recipe project (deleted; the project list is back to the five originals).
That API predates this change, so the first step deactivated the project's VLM to stand in
for the new default (it confirmed the old behavior first: source: "env" for a brand-new
project). Outputs are in docs-site/docs/guides/pipeline-modes.mdx, "Recipe". Ran: ingest
policy (car, person), upload (40 images, 162 boxes kept, 155 dropped), cluster job (VLM stage
skipped), clusters/refine (10 sub-clusters), create class spoiler, crops/batch_label
(28 human-validated), vlm/label_cluster/10000 without an endpoint (409
vlm_not_configured, fail closed), with ?vlm=env (job selected the 134 unvalidated members,
none of the 28). Not run live: anything that needs the new code (new-project default off,
embedding.mode none, the new failure report), because the opmono image does not contain this
branch and I was told not to rebuild it; those are covered by the offline tests below. That
stack's VLM endpoint was unreachable, so no VLM labels were produced.

Tests written first and seen failing

(before the fix, git stash of src and scripts)

tests/curation/test_vlm_explicit_per_project.py::test_a_project_that_never_activated_one_calls_no_vlm  FAILED
tests/curation/test_vlm_explicit_per_project.py::test_settings_null_means_off_not_inherit  FAILED
tests/curation/test_worker_vlm_hot_reload.py::test_a_project_that_never_activated_one_calls_no_vlm_even_with_an_env_builtin  FAILED
tests/curation/test_vlm_profile_configured.py::test_the_env_url_does_not_count_once_the_project_switched_the_vlm_off  FAILED
tests/curation/test_vlm_review_contract.py::test_the_active_response_is_the_shared_activation_shape  FAILED
5 failed, 32 passed

tests/curation/test_detection_only_mode.py  (5 FAILED)  tests/curation/test_cluster_refresh_controls.py  (4 FAILED)
9 failed, 3 passed     e.g. test_defaults_come_from_the_environment: the flags ignored the environment

tests/release/test_acceptance_run.py::test_confirm_one_class_then_train_single_class_subset  FAILED
  (no activation call before vlm/label_batch)

tests/curation/test_vlm_stage_nothing_labeled.py  3 FAILED
  KeyError: 'failed'   /   DID NOT RAISE RuntimeError (all requests failed, run "completed")

Frontend: ingestPolicyPage.test.ts "saves detection-only mode with automatic clustering off"
failed before the form change (Cannot read properties of null (reading 'checked')).

Test-suite note: many existing tests relied on "never activated means env". A vlm_env
fixture now only sets the env endpoint; vlm_env_active (and curation/vlm_opt_in.py)
double a project that opted in, for tests whose fake storage cannot hold the activation doc.
test_wheel_example_e2e activates the env VLM through the real route instead.

Checks

• Full offline suite once (-n 6), make contracts regenerated (contracts/openapi/curation.json,
  the clustering policy field and none mode) and contracts-check, mypy, pre-commit
  --all-files: results at the end of the final reply.
• Ratchets: legacy_repair.py broad-except baseline tightened 2 to 1 (the repair steps are now
  one loop) and the index_items complexity entry dropped (the whole-frame step is a helper).
  No new guard, so no plant-a-violation proof.
• Frontend: npm run check, lint, knip, vitest (451 files, 4759 tests), build, stubbed
  e2e for the ingest-policy page. Page opened in Chromium on DISPLAY=:11 at 1600 and 430 px
  and the screenshots viewed (stub-driven, not committed). Docs screenshots were not
  refreshed: that needs the capture script against a live stack running this branch.

Decisions and things to know

• defaults.vlm: null on PUT /settings used to mean "clear my choice, inherit env"; it now
  means off (there is nothing to inherit).
• GET /settings defaults.vlm is unchanged for a project with no activation (absent).
• The legacy pin only runs where the activation doc is absent. A project whose activation
  read fails at start is retried at the next start (logged as legacy_project_repair
  with vlm_pin_error), never guessed.
• Not done: sub-cluster scope for label_cluster (use label_batch with chosen crop ids);
  a "VLM: inherited from env" label was not needed because inheritance no longer exists.

Also in this PR (folded in to avoid repeated CI runs)

• fix/worker-client-close (was fix(workers): close the registry's OpenSearch client at shutdown #326): workers share one OpenSearch client with their registry; no more Unclosed client session on shutdown.
• fix/cli-and-tooling (was fix(cli): CLI and tooling fixes (profile, local overrides, curation subset, project default, docs guards) #327): CLI profile, local compose override, curation subset, project default, docs guards.

Closes #281
Closes #283
Closes #284
Closes #285
Closes #288
Closes #291
Closes #293