Repository navigation
feat(ingest): bulk-ingest preflight, and honest monitoring alerts - #328
Merged
Merged
Conversation
…otifications are unconfigured Closes #290 Remove Triton Model Not Ready and OpenSearch Cluster Unhealthy, repoint the error-rate rule to http_request_duration_seconds_count and the temperature rule to DCGM, extend the recorded-metrics test to the Grafana rules, rename the alert folder and the unconfigured contact points.
Closes #278 Read-only preflight of a folder, manifest, S3 prefix or registered source: counts, bytes, extensions, header-sampled megapixel histogram, skipped entries and an estimate from measured PERFORMANCE.md figures. Capped, cancellable walk; a stopped walk no longer keeps listing, and an unreadable folder is refused. Changelogs cover both PR fixes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #278
Closes #290
Branch
feat/ingest-preflight-alerts. Two independent fixes, committed separately.#278 Preflight for bulk ingest
POST /curation/projects/{project}/ingest/bulk/preview,scripts/curation/bulk_ingest.py --previewand a Preview button on the Cropwright Ingest page. Read-only: nothing is ingested or written.
folder,manifest,s3(listing only, insideOP_S3_ALLOWED) orsource_id, samesingle-field rule as a start. Out-of-root path, unreadable root, S3 off or not allowed are refused
(422
bulk_ingest_invalid_source/s3_not_allowed, 503s3_not_configured, 404 for an unknown source).seeded 300-file sample (and the share at or above the GPU-decode gate), unreadable directories (first
paths), symlinks not followed, missing / out-of-root manifest lines, and an estimate (time, vectors,
OpenSearch bytes).
GET .../ingest/bulk/configunderpreview) end the walk withpartial: trueandstop_reason; a client disconnect stops it.docs/PERFORMANCE.md, served with the section each came from(
src/services/curation/bulk_ingest/preflight_rates.py): 29.0 / 27.0 / 35.0 images/s (CPU small, CPUlarge, GPU-decode large; "Decode once and the GPU decode (perf(triton): GPU decode once and a GPU-resident ingest pipeline (WP-2.4) #217)"), 7.4 embeddings per image
("v0.5.0 baseline"), 65,298 bytes per image ("Storage per image and per vector"). A test checks every
figure and section name is still in PERFORMANCE.md. Unknown image sizes (S3) give a min-max range
across the measured routes.
walk.py,source.py): a stopped or closed walk kept listing therest of the tree and waited on a stuck directory listing (the pool is now shut down without waiting and
workers check the stop flag); the walk could not notice a stop while nothing matched (
should_stoppoll); a run on an unreadable folder "completed" with 0 discovered (now refused).
iter_image_pathswas split into a
_Walkclass (complexity ratchet entry for it deleted: baseline only shrinks).bulkIngestContract.test.ts;BulkPreviewResult.svelterenders only served values; second press cancels the scan.
ingest-large-folder.mdx(the "no preflight exists" text replaced), API contract doc.Not refreshed: docs screenshots (the opmono API image has no new route, so no live capture was possible).
Verified by offline tests only (the opmono API image lacks the route). The sample output in the guide is
the service plus the CLI renderer run in a script on the public 10-image COCO sample, marked as such.
#290 Monitoring alerts
nv_model_ready_state) andOpenSearch Cluster Unhealthy (
opensearch_cluster_status, no exporter ships). Repointed API HighError Rate (
http_requests_total->http_request_duration_seconds_count) and High GPUTemperature (
nv_gpu_temperature-> DCGMDCGM_FI_DEV_GPU_TEMP). The first and the last two werenot in the issue; found by the new check.
tests/test_monitoring_metrics.pyalready pinned dashboards and Prometheus rules to arecorded list of exported names but never read
grafana-alerting.yml(queries sit underdata[].model.expr). It now does (no second list was added). Triton names were re-read live (read-only)and match the recorded list.
Visual AI Alerts->OpenProcessor Alerts. Contact points renamedunconfigured-default/unconfigured-criticaland the file header,monitoring/README.mdanddocs-site/docs/operations/monitoring.mdxsay plainly that nobody is notified until a receiver isconfigured and what the default does. A test requires any localhost receiver to carry the
unconfigured-name. The compose file does not passALERT_WEBHOOK_URL/SLACK_WEBHOOK_URLintoGrafana (compose is out of scope here), so the docs show the override file.
Tests written first, seen failing
Collection (service tests, before
preflight.pyexisted):Routes, before the route existed:
6 failed, 14 deselected(config preview block, folder preview,outside-root 422, exactly-one-source, S3-off 503, unknown source 404). CLI
--preview:1 failed, 3 passedbefore the flag (the readable-units test). Frontend panel, before the component:
5 failed | 7 passed(BulkIngestPanel preview tests). Monitoring, against the original
grafana-alerting.yml:Not strictly test-first: the controller and contract tests of the frontend were written after their
code (they passed first time); the panel tests were written first.
Plant-a-violation: the new metric check is the existing guard extended; the planted violation is the
original file above (4 dead rules caught).
Checks
See the final report of the run: full offline suite,
make contracts-check, pre-commit--all-files,mypy, frontend check / lint / knip / vitest / build, stubbed e2e (
test_bulk_ingest.py,test_ingest.py,test_ingest_sources.py), docs checker.Viewed screenshots of the Ingest page with a preview at 1600 px and 430 px (no horizontal overflow).
Not verified: loading the new
grafana-alerting.ymlinto a real Grafana (a throwaway containerdid not finish its database migrations inside the time budget; the change is deletions, renames and two
expression edits).