Repository navigation
fix(cli): CLI and tooling fixes (profile, local overrides, curation subset, project default, docs guards) - #327
Closed
davidamacey wants to merge 6 commits into
Closed
davidamacey wants to merge 6 commits into
davidamacey wants to merge 6 commits into
Conversation
Adds check_cited_paths.py (pre-commit, shrink-only baseline for the frontend citations of design notes that left the tree) and rewords the 30 citations outside the frontend.
…bulk_ingest default, docs-site typecheck - capture_docs_screenshots.py: required public-project allowlist; listing pages refused while the backend holds a project outside it (#284) - check_docs_vs_code.py: CLI commands and flags, make targets, screenshot files, NAME= lines; version test scans frontend docs; docs.yml runs on docs/** and type-checks; weekly external link check (#287) - bulk_ingest.py: --api-base defaults to localhost:8000 inside the container (#288) - docs-site: drop deprecated baseUrl, add @types/react-dom, .nvmrc follows the Dockerfile (#293, #289)
… a subset of curation services, default the project name - profile: sets GPU_PROFILE and OP_INGEST_GPU_DECODE in an existing .env; the unread TRITON_WORKERS / MAX_BATCH_SIZE / SHM_SIZE keys and the never-loaded generated override are removed (#281) - ./openprocessor and make load an optional docker-compose.local.yml (#283) - curation up [service...] and make curation-up SERVICES=...; curation-evaluator gets its own evaluator profile (#285) - a checkout without COMPOSE_PROJECT_NAME uses the project openprocessor with a warning (#291)
# Conflicts: # CHANGELOG.md
Owner
Author
|
Folded into #329 (merge commits preserved) to avoid repeated full CI runs and changelog conflicts. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Branch
fix/cli-and-tooling, 4 commits onorigin/main(bbe929a). Nosrc/behavior change (only comments).Closes #281
Closes #283
Closes #284
Closes #285
Closes #288
Closes #291
Closes #293
Already fixed, close (parent closes)
openprocessor"start never touches curation workers" comment no longer exists;.pre-commit-config.yamlline 1 no longer says "Triton API";frontend/docker-compose.ymlnolonger exists;
slow_request_threshold_msis used (src/app/factory.py:164), so it stays. Donehere:
docs-site/.nvmrc20 -> 26 (matchesdocs-site/Dockerfile), TS5101 (chore(docs-site):npm run typecheckfails with TS5101 on a clean origin/main #293), theembedding_state.pycomment (saidnot_selectedwas written by no path anddeferredonly for aproject that could not use the vector; ingest policy writes both), and the stale citations
(
check_cited_paths.py, below).So: close #287 and #289 with this PR too (add
Closes #287/Closes #289when opening if preferred).What changed
#281
openprocessor profileRemoved the dead parts rather than wiring them:
TRITON_WORKERS,MAX_BATCH_SIZE,SHM_SIZE(read bynothing) are gone from the generated
.env,env.templateand the profile JSON (workers,shm_size);generate_compose_overrideand thedocker-compose.override.ymlit wrote (never loaded)are gone from
setup.shand the CLI.profile <name>now rewrites the Triton configs (as before) andsets
GPU_PROFILEandOP_INGEST_GPU_DECODEin the existing.env(newapply_profile_to_env,other lines untouched), and its hint is
./openprocessor restart triton-server. Docs (INSTALLATION,gpu-sizing, basic) updated. The override-file heap test was removed with the function.
#283 your own compose overrides
./openprocessorandmakeloaddocker-compose.local.yml(git-ignored) after the stock files when itexists (install dirs and checkouts). Documented in INSTALLATION (new section "More than one source
root and your own compose overrides") and docs-site
ingest-large-foldersection 3. Render-onlycheck:
docker compose -p cli-and-tooling -f docker-compose.yml -f <local> configshows the extrabind mount. Not done: the issue's "startup check that every alias path exists in each container"
(a src/ change; workflow-controls owns src/ behavior). It is not hidden: the doc tells the user to
check the merged config.
#284 screenshot public-data guard
frontend/scripts/docs_screenshots/public_data.py(stdlib, unit-tested).capture_docs_screenshots.pynow requires
--public-projects/OP_CROPWRIGHT_PUBLIC_PROJECTS,--projectmust be in it, andGET /curation/projects?include_archived=trueis read: if any project is outside the list, the pagesthat list every project (routes under
/projects, statesprojects-delete-dry-run,projects-copy-settings,combine-wizard,combine-job,models-sharing,models-unshare-force,clone-from-project) are refused.make docs-screenshotspassesDOCS_PUBLIC_PROJECTS(defaults toDOCS_PROJECT). The browser flow itself was not run (needs a Cropwright instance); the decision logicis covered by
tests/test_docs_screenshot_public_data_guard.py.#285 curation subset
./openprocessor curation up [service...]andmake curation-up SERVICES="..."; unknown names arerefused before any docker call.
curation-evaluatormoved to its ownevaluatorprofile(
profiles: [evaluator]), so--profile curation up -dno longer starts the GPU-reserving evaluator;naming it starts it. Docs/CLAUDE.md/README/ARCHITECTURE/Dockerfile comments updated.
#287 docs guard
check_docs_vs_code.pygainscli(commands and flags of./openprocessor),make(targets in codespans/fences),
screenshots(file exists; the<file>placeholder is skipped) andenvfiles(
NAME=lines in fenced blocks).tests/test_version_consistency.pyscansfrontend/*.mdandfrontend/docs/**excludingCHANGELOG.mdanddocs/releases/.docs.ymlalso triggers ondocs/**and runsnpm run typecheck. New weeklydocs-links.yml(lychee, external links, alsoworkflow_dispatch); it has not run (needs GitHub).#288 bulk_ingest
--api-basedefaults tolocalhost:8000inside a container (/.dockerenv),localhost:4603on thehost. Docs no longer pass
--api-base.#289 citations
scripts/codegen/check_cited_paths.py(pre-commit; baselinebaselines/cited_paths.txt, 193 frontendcitations of design notes that left the tree, shrink-only). Fixed all 30 citations outside the frontend
(
docs/design/openprocessor_internal/*-> the design by name).#291 fresh checkout
A checkout with no
COMPOSE_PROJECT_NAMEand noOP_PROJECTusesopenprocessorand warns. Theownership check (
assert_project_owned) still refuses a project whose containers belong to anotherdirectory, so a live stack with that name is not touched. An install dir still exits 3.
#293 docs-site typecheck
baseUrlis set by@docusaurus/tsconfigitself, so the fix is"baseUrl": nullplus our ownpaths(
@site/*relative to the tsconfig). Removing the override exposed a second error (missing@types/react-dom), added as a devDependency.npm run typecheck && npm run buildpass; typecheck is a CI step.Tests written first, seen failing
tests/installer/test_cli_profile_command.py:5 failed(env keys not updated, override file written,hint
./scripts/openprocessor.sh, dead keys still generated) before the fix.tests/installer/test_cli_compose_files.py:7 failedbefore the fix (no local file in the-flist,exit 3 on a fresh checkout, no
up -d <names>, unknown name accepted:assert 0 != 0).tests/test_makefile_curation_targets.py+ evaluator profile test:4 failedbefore the fix.tests/test_docs_screenshot_public_data_guard.py:ImportError: cannot import name 'public_data'before the module.tests/curation/test_script_api_prefix.py(bulk_ingest default): written with the fix; fails on main byconstruction (no
_in_container).tests/test_docs_vs_code.py:KeyError: 'cli'before the checks existed.tests/codegen/test_check_cited_paths.py: one test initially failed on a wrong test case (a path with noextension), fixed in the test.
Plant-a-violation proofs (planted, caught, removed): README with
./openprocessor frobnicate --nope,make nonesuch, a missing<Screenshot>andNOT_A_SETTING=1->cli: 2,make: 1,screenshots: 1,envfiles: 1problems;Shipped in v0.1.0.appended tofrontend/README.md->frontend/README.md: v0.1.0fails
test_docs_name_only_the_current_release; adocs/design/ghost_plan.mdcomment insrc/config/projects.py->new violation, not in the baseline.Verification
8765 passed, 20 skipped(-n 6, 11m56s). Installer tests ran in it without a hang.PATH=/tmp/hl:$PATH .venv/bin/pre-commit run --all-files: green (each commit passed its hooks)..venv/bin/python -m mypy:Success: no issues found in 939 source files.scripts/docs/check_docs_vs_code.py: all nine checks 0 problems.openprocessor,scripts/lib/*.sh,scripts/setup.sh.cd docs-site && npm run typecheck && npm run build: pass.docker compose -p cli-and-tooling ... config --services): evaluator absent from thecurationprofile; the local override's mount renders. No container started.Notes for the next PR / reviewers
CURATION_SERVICESand thedocker-compose.local.ymlappend;COMPOSEshows it.setup.shno longer createsdocker-compose.override.yml; a leftover one in an old checkout is simply ignored (it never was loaded).evaluatorprofile changes whatCOMPOSE_PROFILES=curationstarts in existing installs (the evaluator stopsbeing started); that is the point of feat(cli): start a subset of curation workers #285.
frontend/citations (193 in the baseline) are left for whoever next edits those files (the workflow-controls PR touches frontend).