Skip to content

feat(ingest): bulk-ingest preflight, and honest monitoring alerts - #328

Merged
davidamacey merged 2 commits into
mainfrom
feat/ingest-preflight-alerts
Oct 15, 2026
Merged

davidamacey merged 2 commits into
mainfrom
feat/ingest-preflight-alerts

Conversation

@davidamacey

Copy link
Copy Markdown
Owner

Closes #278
Closes #290

Branch feat/ingest-preflight-alerts. Two independent fixes, committed separately.

#278 Preflight for bulk ingest

POST /curation/projects/{project}/ingest/bulk/preview, scripts/curation/bulk_ingest.py --preview
and a Preview button on the Cropwright Ingest page. Read-only: nothing is ingested or written.

  • Sources: folder, manifest, s3 (listing only, inside OP_S3_ALLOWED) or source_id, same
    single-field rule as a start. Out-of-root path, unreadable root, S3 off or not allowed are refused
    (422 bulk_ingest_invalid_source / s3_not_allowed, 503 s3_not_configured, 404 for an unknown source).
  • Returns file count, total bytes, per-extension counts and bytes, a header-only megapixel histogram of a
    seeded 300-file sample (and the share at or above the GPU-decode gate), unreadable directories (first
    paths), symlinks not followed, missing / out-of-root manifest lines, and an estimate (time, vectors,
    OpenSearch bytes).
  • Bounded and cancellable: caps of 200,000 files / 60 s (served in GET .../ingest/bulk/config under
    preview) end the walk with partial: true and stop_reason; a client disconnect stops it.
  • Estimates use only measured figures in docs/PERFORMANCE.md, served with the section each came from
    (src/services/curation/bulk_ingest/preflight_rates.py): 29.0 / 27.0 / 35.0 images/s (CPU small, CPU
    large, GPU-decode large; "Decode once and the GPU decode (perf(triton): GPU decode once and a GPU-resident ingest pipeline (WP-2.4) #217)"), 7.4 embeddings per image
    ("v0.5.0 baseline"), 65,298 bytes per image ("Storage per image and per vector"). A test checks every
    figure and section name is still in PERFORMANCE.md. Unknown image sizes (S3) give a min-max range
    across the measured routes.
  • Gaps found and fixed on the way (walk.py, source.py): a stopped or closed walk kept listing the
    rest of the tree and waited on a stuck directory listing (the pool is now shut down without waiting and
    workers check the stop flag); the walk could not notice a stop while nothing matched (should_stop
    poll); a run on an unreadable folder "completed" with 0 discovered (now refused). iter_image_paths
    was split into a _Walk class (complexity ratchet entry for it deleted: baseline only shrinks).
  • Frontend: types pinned to the OpenAPI by bulkIngestContract.test.ts; BulkPreviewResult.svelte
    renders only served values; second press cancels the scan.
  • Docs: ingest-large-folder.mdx (the "no preflight exists" text replaced), API contract doc.
    Not refreshed: docs screenshots (the opmono API image has no new route, so no live capture was possible).

Verified by offline tests only (the opmono API image lacks the route). The sample output in the guide is
the service plus the CLI renderer run in a script on the public 10-image COCO sample, marked as such.

#290 Monitoring alerts

  • Removed Grafana rules on series nothing exports: Triton Model Not Ready (nv_model_ready_state) and
    OpenSearch Cluster Unhealthy (opensearch_cluster_status, no exporter ships). Repointed API High
    Error Rate
    (http_requests_total -> http_request_duration_seconds_count) and High GPU
    Temperature
    (nv_gpu_temperature -> DCGM DCGM_FI_DEV_GPU_TEMP). The first and the last two were
    not in the issue; found by the new check.
  • Root cause: tests/test_monitoring_metrics.py already pinned dashboards and Prometheus rules to a
    recorded list of exported names but never read grafana-alerting.yml (queries sit under
    data[].model.expr). It now does (no second list was added). Triton names were re-read live (read-only)
    and match the recorded list.
  • Folder Visual AI Alerts -> OpenProcessor Alerts. Contact points renamed unconfigured-default /
    unconfigured-critical and the file header, monitoring/README.md and
    docs-site/docs/operations/monitoring.mdx say plainly that nobody is notified until a receiver is
    configured and what the default does. A test requires any localhost receiver to carry the
    unconfigured- name. The compose file does not pass ALERT_WEBHOOK_URL/SLACK_WEBHOOK_URL into
    Grafana (compose is out of scope here), so the docs show the override file.

Tests written first, seen failing

Collection (service tests, before preflight.py existed):

ImportError: cannot import name 'preflight' from 'src.services.curation.bulk_ingest'

Routes, before the route existed: 6 failed, 14 deselected (config preview block, folder preview,
outside-root 422, exactly-one-source, S3-off 503, unknown source 404). CLI --preview: 1 failed, 3 passed
before the flag (the readable-units test). Frontend panel, before the component: 5 failed | 7 passed
(BulkIngestPanel preview tests). Monitoring, against the original grafana-alerting.yml:

AssertionError: Grafana rule(s) query metric(s) nothing exports: {'API High Error Rate: ... http_requests_total',
  'Triton Model Not Ready: nv_model_ready_state', 'High GPU Temperature: nv_gpu_temperature',
  'OpenSearch Cluster Unhealthy: opensearch_cluster_status'}
AssertionError: assert {'Visual AI Alerts'} == {'OpenProcessor Alerts'}
AssertionError: default-notifications  (contact point with a localhost receiver not named unconfigured-*)

Not strictly test-first: the controller and contract tests of the frontend were written after their
code (they passed first time); the panel tests were written first.

Plant-a-violation: the new metric check is the existing guard extended; the planted violation is the
original file above (4 dead rules caught).

Checks

See the final report of the run: full offline suite, make contracts-check, pre-commit --all-files,
mypy, frontend check / lint / knip / vitest / build, stubbed e2e (test_bulk_ingest.py, test_ingest.py,
test_ingest_sources.py), docs checker.
Viewed screenshots of the Ingest page with a preview at 1600 px and 430 px (no horizontal overflow).

Not verified: loading the new grafana-alerting.yml into a real Grafana (a throwaway container
did not finish its database migrations inside the time budget; the change is deletions, renames and two
expression edits).

…otifications are unconfigured

Closes #290

Remove Triton Model Not Ready and OpenSearch Cluster Unhealthy, repoint the error-rate rule to http_request_duration_seconds_count and the temperature rule to DCGM, extend the recorded-metrics test to the Grafana rules, rename the alert folder and the unconfigured contact points.
Closes #278

Read-only preflight of a folder, manifest, S3 prefix or registered source: counts, bytes, extensions, header-sampled megapixel histogram, skipped entries and an estimate from measured PERFORMANCE.md figures. Capped, cancellable walk; a stopped walk no longer keeps listing, and an unreadable folder is refused. Changelogs cover both PR fixes.
@davidamacey
davidamacey merged commit 7d03797 into main Oct 15, 2026
11 checks passed
@davidamacey
davidamacey deleted the feat/ingest-preflight-alerts branch October 15, 2026 09:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(monitoring): dead Grafana alert rules feat(ingest): preflight for bulk ingest (file count, bytes, estimated time and disk)

1 participant