Skip to content

fix(cli): CLI and tooling fixes (profile, local overrides, curation subset, project default, docs guards) - #327

Closed
davidamacey wants to merge 6 commits into
mainfrom
fix/cli-and-tooling
Closed

davidamacey wants to merge 6 commits into
mainfrom
fix/cli-and-tooling

Conversation

@davidamacey

Copy link
Copy Markdown
Owner

Branch fix/cli-and-tooling, 4 commits on origin/main (bbe929a). No src/ behavior change (only comments).

Closes #281
Closes #283
Closes #284
Closes #285
Closes #288
Closes #291
Closes #293

Already fixed, close (parent closes)

  • test(docs): close the docs-guard gaps #287: closed by this PR, all five items (see below), not by earlier work.
  • chore: stale code comments and citations #289: closed by this PR plus earlier cleanup. Evidence of what was already gone on main: the
    openprocessor "start never touches curation workers" comment no longer exists;
    .pre-commit-config.yaml line 1 no longer says "Triton API"; frontend/docker-compose.yml no
    longer exists; slow_request_threshold_ms is used (src/app/factory.py:164), so it stays. Done
    here: docs-site/.nvmrc 20 -> 26 (matches docs-site/Dockerfile), TS5101 (chore(docs-site): npm run typecheck fails with TS5101 on a clean origin/main #293), the
    embedding_state.py comment (said not_selected was written by no path and deferred only for a
    project that could not use the vector; ingest policy writes both), and the stale citations
    (check_cited_paths.py, below).

So: close #287 and #289 with this PR too (add Closes #287 / Closes #289 when opening if preferred).

What changed

#281 openprocessor profile

Removed the dead parts rather than wiring them: TRITON_WORKERS, MAX_BATCH_SIZE, SHM_SIZE (read by
nothing) are gone from the generated .env, env.template and the profile JSON (workers,
shm_size); generate_compose_override and the docker-compose.override.yml it wrote (never loaded)
are gone from setup.sh and the CLI. profile <name> now rewrites the Triton configs (as before) and
sets GPU_PROFILE and OP_INGEST_GPU_DECODE in the existing .env (new apply_profile_to_env,
other lines untouched), and its hint is ./openprocessor restart triton-server. Docs (INSTALLATION,
gpu-sizing, basic) updated. The override-file heap test was removed with the function.

#283 your own compose overrides

./openprocessor and make load docker-compose.local.yml (git-ignored) after the stock files when it
exists (install dirs and checkouts). Documented in INSTALLATION (new section "More than one source
root and your own compose overrides") and docs-site ingest-large-folder section 3. Render-only
check: docker compose -p cli-and-tooling -f docker-compose.yml -f <local> config shows the extra
bind mount. Not done: the issue's "startup check that every alias path exists in each container"
(a src/ change; workflow-controls owns src/ behavior). It is not hidden: the doc tells the user to
check the merged config.

#284 screenshot public-data guard

frontend/scripts/docs_screenshots/public_data.py (stdlib, unit-tested). capture_docs_screenshots.py
now requires --public-projects / OP_CROPWRIGHT_PUBLIC_PROJECTS, --project must be in it, and
GET /curation/projects?include_archived=true is read: if any project is outside the list, the pages
that list every project (routes under /projects, states projects-delete-dry-run,
projects-copy-settings, combine-wizard, combine-job, models-sharing, models-unshare-force,
clone-from-project) are refused. make docs-screenshots passes DOCS_PUBLIC_PROJECTS (defaults to
DOCS_PROJECT). The browser flow itself was not run (needs a Cropwright instance); the decision logic
is covered by tests/test_docs_screenshot_public_data_guard.py.

#285 curation subset

./openprocessor curation up [service...] and make curation-up SERVICES="..."; unknown names are
refused before any docker call. curation-evaluator moved to its own evaluator profile
(profiles: [evaluator]), so --profile curation up -d no longer starts the GPU-reserving evaluator;
naming it starts it. Docs/CLAUDE.md/README/ARCHITECTURE/Dockerfile comments updated.

#287 docs guard

check_docs_vs_code.py gains cli (commands and flags of ./openprocessor), make (targets in code
spans/fences), screenshots (file exists; the <file> placeholder is skipped) and envfiles
(NAME= lines in fenced blocks). tests/test_version_consistency.py scans frontend/*.md and
frontend/docs/** excluding CHANGELOG.md and docs/releases/. docs.yml also triggers on
docs/** and runs npm run typecheck. New weekly docs-links.yml (lychee, external links, also
workflow_dispatch); it has not run (needs GitHub).

#288 bulk_ingest

--api-base defaults to localhost:8000 inside a container (/.dockerenv), localhost:4603 on the
host. Docs no longer pass --api-base.

#289 citations

scripts/codegen/check_cited_paths.py (pre-commit; baseline baselines/cited_paths.txt, 193 frontend
citations of design notes that left the tree, shrink-only). Fixed all 30 citations outside the frontend
(docs/design/openprocessor_internal/* -> the design by name).

#291 fresh checkout

A checkout with no COMPOSE_PROJECT_NAME and no OP_PROJECT uses openprocessor and warns. The
ownership check (assert_project_owned) still refuses a project whose containers belong to another
directory, so a live stack with that name is not touched. An install dir still exits 3.

#293 docs-site typecheck

baseUrl is set by @docusaurus/tsconfig itself, so the fix is "baseUrl": null plus our own paths
(@site/* relative to the tsconfig). Removing the override exposed a second error (missing
@types/react-dom), added as a devDependency. npm run typecheck && npm run build pass; typecheck is a CI step.

Tests written first, seen failing

  • tests/installer/test_cli_profile_command.py: 5 failed (env keys not updated, override file written,
    hint ./scripts/openprocessor.sh, dead keys still generated) before the fix.
  • tests/installer/test_cli_compose_files.py: 7 failed before the fix (no local file in the -f list,
    exit 3 on a fresh checkout, no up -d <names>, unknown name accepted: assert 0 != 0).
  • tests/test_makefile_curation_targets.py + evaluator profile test: 4 failed before the fix.
  • tests/test_docs_screenshot_public_data_guard.py: ImportError: cannot import name 'public_data' before the module.
  • tests/curation/test_script_api_prefix.py (bulk_ingest default): written with the fix; fails on main by
    construction (no _in_container).
  • tests/test_docs_vs_code.py: KeyError: 'cli' before the checks existed.
  • tests/codegen/test_check_cited_paths.py: one test initially failed on a wrong test case (a path with no
    extension), fixed in the test.

Plant-a-violation proofs (planted, caught, removed): README with ./openprocessor frobnicate --nope,
make nonesuch, a missing <Screenshot> and NOT_A_SETTING=1 -> cli: 2, make: 1, screenshots: 1,
envfiles: 1 problems; Shipped in v0.1.0. appended to frontend/README.md -> frontend/README.md: v0.1.0
fails test_docs_name_only_the_current_release; a docs/design/ghost_plan.md comment in
src/config/projects.py -> new violation, not in the baseline.

Verification

  • Full offline suite once: 8765 passed, 20 skipped (-n 6, 11m56s). Installer tests ran in it without a hang.
  • PATH=/tmp/hl:$PATH .venv/bin/pre-commit run --all-files: green (each commit passed its hooks).
  • .venv/bin/python -m mypy: Success: no issues found in 939 source files.
  • scripts/docs/check_docs_vs_code.py: all nine checks 0 problems.
  • shellcheck clean for openprocessor, scripts/lib/*.sh, scripts/setup.sh.
  • cd docs-site && npm run typecheck && npm run build: pass.
  • Docker: render-only (docker compose -p cli-and-tooling ... config --services): evaluator absent from the
    curation profile; the local override's mount renders. No container started.

Notes for the next PR / reviewers

  • Makefile gained CURATION_SERVICES and the docker-compose.local.yml append; COMPOSE shows it.
  • setup.sh no longer creates docker-compose.override.yml; a leftover one in an old checkout is simply ignored (it never was loaded).
  • The evaluator profile changes what COMPOSE_PROFILES=curation starts in existing installs (the evaluator stops
    being started); that is the point of feat(cli): start a subset of curation workers #285.
  • frontend/ citations (193 in the baseline) are left for whoever next edits those files (the workflow-controls PR touches frontend).

Adds check_cited_paths.py (pre-commit, shrink-only baseline for the frontend citations of
design notes that left the tree) and rewords the 30 citations outside the frontend.
…bulk_ingest default, docs-site typecheck

- capture_docs_screenshots.py: required public-project allowlist; listing pages refused
  while the backend holds a project outside it (#284)
- check_docs_vs_code.py: CLI commands and flags, make targets, screenshot files, NAME= lines;
  version test scans frontend docs; docs.yml runs on docs/** and type-checks; weekly external
  link check (#287)
- bulk_ingest.py: --api-base defaults to localhost:8000 inside the container (#288)
- docs-site: drop deprecated baseUrl, add @types/react-dom, .nvmrc follows the Dockerfile (#293, #289)
… a subset of curation services, default the project name

- profile: sets GPU_PROFILE and OP_INGEST_GPU_DECODE in an existing .env; the unread
  TRITON_WORKERS / MAX_BATCH_SIZE / SHM_SIZE keys and the never-loaded generated override
  are removed (#281)
- ./openprocessor and make load an optional docker-compose.local.yml (#283)
- curation up [service...] and make curation-up SERVICES=...; curation-evaluator gets its
  own evaluator profile (#285)
- a checkout without COMPOSE_PROJECT_NAME uses the project openprocessor with a warning (#291)
@davidamacey

Copy link
Copy Markdown
Owner Author

Folded into #329 (merge commits preserved) to avoid repeated full CI runs and changelog conflicts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment