Skip to content

feat(ingest): S3 / S3-compatible object store source for bulk ingest, with watch mode #264

Closed

Description

@davidamacey

Owner goal: first-class ingestion of images from API endpoints and bulk from the backend, and the same from an S3 bucket in the cloud, including watching a bucket for new files. Today the bulk runner (#229) supports only a folder or a manifest of local paths under OP_SOURCE_ROOT; there is no S3 code. Design questions to settle first (plan file, then build): (1) Source kind 's3' (bucket + prefix, region/endpoint, S3-compatible such as MinIO for local tests) in src/services/curation/bulk_ingest/source.py; the pipeline reads local paths today, so decide between streaming object bytes straight into the ingest pipeline (no temp files; best for throughput and for the GPU/CPU decode routing from #217) and staging to a local cache; either way originals must stay servable for review/crops (presigned URL or proxy) and the stored image reference must identify bucket/key/version-or-etag. (2) Listing at scale: ListObjectsV2 paging with a checkpoint (StartAfter / last-modified), parallel prefix listing, and ranged/concurrent GETs with bounded in-flight bytes (reuse the decoded-megabyte admission gate). (3) Watch mode: periodic poll with a durable checkpoint per project/source first (simple, resumable, works with any S3-compatible store); S3 Event Notifications to SQS as a later option; schedule via the existing job mechanism so a daily/hourly batch is just a recurring run. (4) Idempotency: content hash ledger already skips stored images; use ETag/version to avoid re-downloading. (5) Security: the API has no authentication and is loopback-only by default (SECURITY.md): cloud credentials widen the blast radius, so read them from the environment / instance role / a mounted secret file, never from the request or the tree, scope them to read-only on the configured bucket/prefix allow-list, fail closed on an unconfigured bucket. (6) Surfaces: same runner behind the CLI (scripts/curation/bulk_ingest.py), the /ingest/bulk route and the Cropwright panel; progress and failures as today; contracts regenerated. (7) Tests: MinIO in an isolated compose project for the live tests, offline unit tests with a fake client; public data only. Sequence: after the decode/ingest speed work (#217, #226, #248) so it is built on the final pipeline.

Activity

  1. davidamacey commented on Oct 12, 2026

    @davidamacey
    OwnerAuthor

    Owner decisions (2026-10-12): (1) no API auth or extra security hardening for now: the service runs locally and processing work comes first (the credential handling in point 5 is reduced to: credentials from the environment or instance role, never from a request or the tree; auth comes later). (2) Triggers: S3 event webhooks when files arrive (S3 Event Notifications to a webhook endpoint, directly or via SNS/SQS; also S3-compatible stores such as MinIO bucket notifications) AND cron/scheduled runs; both in addition to the poll-with-checkpoint watch. Webhook receiver must be idempotent (dedupe by bucket/key/etag), verify the notification shape, enqueue into the bulk runner and return fast. (3) Build it rolled together with the ingest pipeline (decode routing, OpenSearch write path) so it inherits the speed work. Sequence unchanged: plan file first, built after #217/#226/#248.

  2. davidamacey commented on Oct 13, 2026

    @davidamacey
    OwnerAuthor

    Tracked with the triggers and the gRPC decision in #271 (plan: docs/design/ingest_surfaces_plan.md, PR #269): staged bytes with the object reference in source_identifier, one durable per-source queue behind watcher, webhook, cron and manual runs, one dispatcher, Cropwright sources panel.

  3. davidamacey commented on Oct 16, 2026

    @davidamacey
    OwnerAuthor

    Verified live against a real MinIO (300 public COCO images): register source, preview, bulk ingest (300 ingested, 0 failed, image served), fail-closed allow-list (422 s3_not_allowed), Cropwright sources panel at 1600/430 px, and after the dispatcher fix (PR #333) the webhook, poll and cron paths ingest new objects with no dispatch errors. Report: artifacts_local/review/s3-live-check.md (local).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions