You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Plan: docs/design/ingest_surfaces_plan.md (PR #269, docs only; the file lives on main once that PR merges). Covers #264 (S3 source with watch mode) and #263 (gRPC ingest channel, only if it wins by measurement), built on the bulk runner of #229.
Goal: every way of getting images in ends in one pipeline (ingest_batch): HTTP API requests (exists), bulk from a folder or manifest (exists), S3 and S3-compatible stores (new), triggers (new), gRPC (decided by measurement).
Owner decisions already made: no API authentication for now (local use), credentials only from the environment, an instance role or a mounted secret file, fail closed on an unconfigured bucket or prefix; webhook and cron and poll-watch are all in scope; built on the final pipeline (decode routing, coalesced writer, bulk concurrency 16, project isolation, lock rule and OCC, content-hash ledger); live tests on MinIO in an isolated compose project, offline tests on a fake client, public data only.
Work slices, in order (each ends with passing tests; contracts regenerated in the same commit):
S3 source and checkpointed watcher: object store seam, fetch with bounded in-flight bytes, objects route, per-source durable queue keyed by (key, etag), paged and prefix-parallel resumable scan, folder watcher, source registry and run-now, one-shot bulk start for a bucket, CLI.
Webhook receiver: S3/MinIO event shape validation, idempotent by bucket/key/etag, fast 202.
Schedules and dispatcher: cron, poll interval, webhook drains, backoff, one acting worker per project.
Cropwright: sources panel (status, triggers, last run, failures, run now, retry), served blocked_message and poll_after_s.
gRPC decision: transport-only measurement with an Amdahl bound first, written decision rule, prototype only if it passes (see plan section 8).
One live end-to-end check (MinIO -> webhook -> bulk run -> OpenSearch -> Cropwright, full-page screenshots at desktop and 430 px), persist stage share recorded, teardown proven.
The plan also lists the file map, interfaces, route shapes, edge cases, the test plan and the traps found while sketching an implementation (section 12).
Plan gist (secret, unlisted): https://gist.github.com/davidamacey/63b4234e9c893855f1f15fc252b07953 . Source of truth is docs/design/ingest_surfaces_plan.md (PR #269). Work in progress is kept on branch feat/ingest-surfaces (S3 source + checkpointed queue, webhook receiver, dispatcher, WIP Cropwright sources panel); two frontend tests still fail in the WIP commit (contract pin needs make contracts; retry button case).
Verified live against a real MinIO (300 public COCO images): register source, preview, bulk ingest (300 ingested, 0 failed, image served), fail-closed allow-list (422 s3_not_allowed), Cropwright sources panel at 1600/430 px, and after the dispatcher fix (PR #333) the webhook, poll and cron paths ingest new objects with no dispatch errors. Report: artifacts_local/review/s3-live-check.md (local).
Activity
davidamacey commentedon Oct 13, 2026
Plan gist (secret, unlisted): https://gist.github.com/davidamacey/63b4234e9c893855f1f15fc252b07953 . Source of truth is docs/design/ingest_surfaces_plan.md (PR #269). Work in progress is kept on branch feat/ingest-surfaces (S3 source + checkpointed queue, webhook receiver, dispatcher, WIP Cropwright sources panel); two frontend tests still fail in the WIP commit (contract pin needs make contracts; retry button case).
davidamacey commentedon Oct 13, 2026
Plan merged to main in #269 (docs/design/ingest_surfaces_plan.md). Build from branch feat/ingest-surfaces when scheduled.
davidamacey commentedon Oct 16, 2026
Verified live against a real MinIO (300 public COCO images): register source, preview, bulk ingest (300 ingested, 0 failed, image served), fail-closed allow-list (422 s3_not_allowed), Cropwright sources panel at 1600/430 px, and after the dispatcher fix (PR #333) the webhook, poll and cron paths ingest new objects with no dispatch errors. Report: artifacts_local/review/s3-live-check.md (local).