You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(ingest): S3 / S3-compatible object store source for bulk ingest, with watch mode #264
Owner goal: first-class ingestion of images from API endpoints and bulk from the backend, and the same from an S3 bucket in the cloud, including watching a bucket for new files. Today the bulk runner (#229) supports only a folder or a manifest of local paths under OP_SOURCE_ROOT; there is no S3 code. Design questions to settle first (plan file, then build): (1) Source kind 's3' (bucket + prefix, region/endpoint, S3-compatible such as MinIO for local tests) in src/services/curation/bulk_ingest/source.py; the pipeline reads local paths today, so decide between streaming object bytes straight into the ingest pipeline (no temp files; best for throughput and for the GPU/CPU decode routing from #217) and staging to a local cache; either way originals must stay servable for review/crops (presigned URL or proxy) and the stored image reference must identify bucket/key/version-or-etag. (2) Listing at scale: ListObjectsV2 paging with a checkpoint (StartAfter / last-modified), parallel prefix listing, and ranged/concurrent GETs with bounded in-flight bytes (reuse the decoded-megabyte admission gate). (3) Watch mode: periodic poll with a durable checkpoint per project/source first (simple, resumable, works with any S3-compatible store); S3 Event Notifications to SQS as a later option; schedule via the existing job mechanism so a daily/hourly batch is just a recurring run. (4) Idempotency: content hash ledger already skips stored images; use ETag/version to avoid re-downloading. (5) Security: the API has no authentication and is loopback-only by default (SECURITY.md): cloud credentials widen the blast radius, so read them from the environment / instance role / a mounted secret file, never from the request or the tree, scope them to read-only on the configured bucket/prefix allow-list, fail closed on an unconfigured bucket. (6) Surfaces: same runner behind the CLI (scripts/curation/bulk_ingest.py), the /ingest/bulk route and the Cropwright panel; progress and failures as today; contracts regenerated. (7) Tests: MinIO in an isolated compose project for the live tests, offline unit tests with a fake client; public data only. Sequence: after the decode/ingest speed work (#217, #226, #248) so it is built on the final pipeline.
Owner decisions (2026-10-12): (1) no API auth or extra security hardening for now: the service runs locally and processing work comes first (the credential handling in point 5 is reduced to: credentials from the environment or instance role, never from a request or the tree; auth comes later). (2) Triggers: S3 event webhooks when files arrive (S3 Event Notifications to a webhook endpoint, directly or via SNS/SQS; also S3-compatible stores such as MinIO bucket notifications) AND cron/scheduled runs; both in addition to the poll-with-checkpoint watch. Webhook receiver must be idempotent (dedupe by bucket/key/etag), verify the notification shape, enqueue into the bulk runner and return fast. (3) Build it rolled together with the ingest pipeline (decode routing, OpenSearch write path) so it inherits the speed work. Sequence unchanged: plan file first, built after #217/#226/#248.
Tracked with the triggers and the gRPC decision in #271 (plan: docs/design/ingest_surfaces_plan.md, PR #269): staged bytes with the object reference in source_identifier, one durable per-source queue behind watcher, webhook, cron and manual runs, one dispatcher, Cropwright sources panel.
Verified live against a real MinIO (300 public COCO images): register source, preview, bulk ingest (300 ingested, 0 failed, image served), fail-closed allow-list (422 s3_not_allowed), Cropwright sources panel at 1600/430 px, and after the dispatcher fix (PR #333) the webhook, poll and cron paths ingest new objects with no dispatch errors. Report: artifacts_local/review/s3-live-check.md (local).
Activity
davidamacey commentedon Oct 12, 2026
Owner decisions (2026-10-12): (1) no API auth or extra security hardening for now: the service runs locally and processing work comes first (the credential handling in point 5 is reduced to: credentials from the environment or instance role, never from a request or the tree; auth comes later). (2) Triggers: S3 event webhooks when files arrive (S3 Event Notifications to a webhook endpoint, directly or via SNS/SQS; also S3-compatible stores such as MinIO bucket notifications) AND cron/scheduled runs; both in addition to the poll-with-checkpoint watch. Webhook receiver must be idempotent (dedupe by bucket/key/etag), verify the notification shape, enqueue into the bulk runner and return fast. (3) Build it rolled together with the ingest pipeline (decode routing, OpenSearch write path) so it inherits the speed work. Sequence unchanged: plan file first, built after #217/#226/#248.
docs(plan): ingest surfaces plan carries the owner decisions, as-desi…
davidamacey commentedon Oct 13, 2026
Tracked with the triggers and the gRPC decision in #271 (plan: docs/design/ingest_surfaces_plan.md, PR #269): staged bytes with the object reference in source_identifier, one durable per-source queue behind watcher, webhook, cron and manual runs, one dispatcher, Cropwright sources panel.
Merge pull request #269 from davidamacey/docs/ingest-surfaces-plan
test(ingest): list the source events route in the cross-project sweep (…
refactor(ingest): start the dispatcher with the sweeper through backg…
test(ingest): the fetch gate bounds encoded bytes across concurrent b…
docs(ingest): S3 and watched sources guide, configuration table and c…
fix(ingest): keep the bound project across object fetch threads and f…
feat(frontend): finish the ingest sources panel, confirm removal, fix…
docs(frontend): sources panel steps, changelog and a screenshot state (…
fix(projects): delete removes bulk-ingest, ingest-source and reproces…
test(ingest): MinIO overlay, live S3 check results, source screenshot…
fix(ingest): settle every queue row of one object when a retried stal…
Merge pull request #296 from davidamacey/feat/ingest-surfaces
davidamacey commentedon Oct 16, 2026
Verified live against a real MinIO (300 public COCO images): register source, preview, bulk ingest (300 ingested, 0 failed, image served), fail-closed allow-list (422 s3_not_allowed), Cropwright sources panel at 1600/430 px, and after the dispatcher fix (PR #333) the webhook, poll and cron paths ingest new objects with no dispatch errors. Report: artifacts_local/review/s3-live-check.md (local).