Skip to content

feat(capture): captures.store — mirror captures from an S3/GCS/Azure bucket - #40

Merged
babaliauskas merged 17 commits into
mainfrom
feat/remote-capture-store-cli
Oct 9, 2026
Merged

feat(capture): captures.store — mirror captures from an S3/GCS/Azure bucket#40
babaliauskas merged 17 commits into
mainfrom
feat/remote-capture-store-cli

Conversation

@babaliauskas

@babaliauskas babaliauskas commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator

What this changes

  • New captures.store in evalshift.yaml (s3://<bucket>/<prefix>, gs://<bucket>/<prefix>, az://<account>/<container>/<prefix>). With it set, capture sync and capture list mirror new captures and toolset sidecars from the bucket into .evalshift/ before they run; everything else reads the local mirror unchanged. Without it, nothing changes.
  • New capture fetch [--suite] [--since 30m|24h|7d|ISO] [-c] does only the mirror step; sync gains --since / --offline, list gains -c/--config / --offline.
  • Incremental and safe: only objects missing locally are downloaded (captures immutable, toolsets content-addressed), promoted ids are skipped so capture clean never causes a re-download, writes are atomic, and bucket keys are untrusted — anything not shaped like the SDK's layout is counted malformed and never written, and every destination is checked to stay inside .evalshift/. A failed fetch exits 1 before anything is promoted; a re-run resumes.
  • Read adapters for S3 / GCS / Azure Blob behind optional extras evalshift[s3], [gcs], [azure] (lazy imports; missing extra → exit 1 naming the pip extra). doctor reports whether the store is installed (fail) and listable (warn).
  • capture list --json keeps stdout pure JSON (fetch line and warnings go to stderr). Store URIs are never echoed when rejected; error hints are per cloud.
  • Docs: DOCS.md, llms-full.txt, docs/configuration.md (## captures), docs/sdk.md, docs/hosted.md, docs/getting-started.md, docs/index.md, AGENTS.md, README.md, CHANGELOG.

Why

Production agents usually run on hosts whose disk does not outlive them (Fargate, Lambda, Kubernetes pods). The companion SDK PR (evalshift/evalshift-sdk#12) lets the capture SDK ship captures to a bucket the user owns with EVALSHIFT_SINK; this PR lets the CLI pull them back with no flags — "the layout is the interface, on disk or in a bucket". Design: spec §4, §6, §7, §9 (remote capture store design, 2026-10-06).

Test plan

  • pytest passes — 2742 passed, 1 skipped (no server checkout); coverage 94.8% (floor 90)
  • ruff check, ruff format --check clean; pre-commit run --all-files clean (pre-push CI mirror passed)
  • mypy --strict src/evalshift_cli clean
  • CHANGELOG.md updated under ## [Unreleased]
  • docs updated
  • Optional: manual smoke against a real bucket before release (captures: {store: s3://…} + evalshift capture fetch)

Release notes: ship with SDK 0.5.0. Older CLIs reject the captures: key (config is strict), so bump any pinned GitHub Action before adding it.

🤖 Generated with Claude Code

babaliauskas and others added 17 commits October 9, 2026 19:08
@babaliauskas @claude
feat(captures): store URI grammar shared with the SDK
2ff25b6 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@babaliauskas @claude
feat(config): captures.store names the object store captures are mirr…
569a5eb 
…ored from

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@babaliauskas @claude
feat(captures): incremental fetch from a remote object store into the…
9028285 
… local mirror

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@babaliauskas @claude
fix(captures): refuse colon suite segments and contain fetch destinat…
58725c2 
…ions

On Windows a suite segment like `D:` or `a:b` joins as a drive-relative path or
an alternate data stream and escapes <base>. The suite predicate now refuses
`:`, and every destination must also normalize inside its root before a
download is queued.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@babaliauskas @claude
feat(captures): S3, GCS and Azure Blob read adapters behind optional …
c57ce21 
…extras; open_store

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@babaliauskas @claude
feat(capture): fetch from captures.store; sync and list mirror the bu…
015c9a7 
…cket first

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@babaliauskas @claude
fix(capture): escape the store URI, keep list --json stdout pure, sta…
9751e18 
…y silent on configs without captures

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@babaliauskas @claude
feat(doctor): report whether captures.store is installed and reachable
52a6d00 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@babaliauskas @claude
docs: captures.store, capture fetch, bucket-mirroring sync and list
a350467 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@babaliauskas @claude
fix(capture): warn about a skipped store only for a top-level capture…
d93d44e 
…s key

evalshift init writes "captures" into its comments, so the substring test
warned on every invalid init-generated config. Match a column-0 captures:
key instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@babaliauskas @claude
fix(config): never echo a rejected captures.store value
51cc97d 
The unsupported-scheme, no-bucket and Azure no-container messages quoted the
whole value, so a pasted Azure connection string landed in the config error
and CI logs. Name only the parsed scheme. Message strings match the SDK.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@babaliauskas @claude
fix(capture): per-cloud fetch hints, local write errors, strict bucke…
06d3376 
…t keys

- A list or download failure's hint names the scheme's credential chain and
  points --offline at capture sync / capture list (capture fetch has none).
- A local write failure (mkdir, temp file, write, rename) raises its own
  error naming the file, with a disk-space-and-permissions hint.
- Capture and toolset basenames use fullmatch, so a key with a trailing
  newline is malformed instead of written.
- Folder-marker keys ending in / are skipped without counting as malformed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@babaliauskas @claude
fix(capture): reject an out-of-range --since duration instead of a tr…
bd064f2 
…aceback

parse_since("99999999d") raised OverflowError; it now raises the usual
--since ValueError, so the CLI exits 2 with its message.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@babaliauskas @claude
fix(capture): exit 1 with one line when capture fetch cannot read the…
a5c53ca 
… config

An unreadable or non-UTF-8 evalshift.yaml re-raised OSError or
UnicodeDecodeError from capture fetch as a traceback. Print the escaped path
and the error type only, and exit 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@babaliauskas @claude
chore(mypy): drop the unused boto3.* override
8544f53 
Only the top-level boto3 module is imported; mypy --warn-unused-configs
flagged [mypy-boto3.*] as unused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@babaliauskas @claude
docs: count the capture bucket in the privacy and SDK-contract statem…
a1cc62a 
…ents

index.md said providers were the only outbound calls; hosted.md and
llms-full.txt said "exactly two kinds" and then a third; AGENTS.md said disk
is the only SDK/CLI interface. Each now accounts for captures.store.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@babaliauskas @claude
docs(configuration): --since wording and the older-CLI note for captures
c42d32b 
--since limits a fetch to recently written captures, not "recent objects".
Older CLIs reject the captures: key (strict config), so bump the CLI and a
pinned action first; say so in configuration.md and the changelog.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@babaliauskas babaliauskas mentioned this pull request Oct 9, 2026
Merged
2 of 3 tasks
@babaliauskas
babaliauskas merged commit c166702 into main Oct 9, 2026
4 checks passed
@babaliauskas
babaliauskas deleted the feat/remote-capture-store-cli branch October 9, 2026 19:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@babaliauskas