babaliauskas · Oct 9, 2026 · Oct 9, 2026 · Oct 9, 2026 · Oct 9, 2026 · Oct 9, 2026
diff --git a/AGENTS.md b/AGENTS.md
@@ -34,10 +34,10 @@ repos (`evalshift-sdk`, `evalshift-action`); never edit them from here.
 - **CLI** — this repo. PyPI `evalshift`. Full reference: [DOCS.md](DOCS.md).
 - **SDK** — PyPI `evalshift-sdk`, import name `evalshift`, stdlib-only, Python
   3.10+. Records production agent runs to `.evalshift/captures/`; the CLI
-  promotes them with `evalshift capture sync`. Disk is the only interface — the
-  two packages never call each other. The CLI imports as `evalshift_cli` and
-  depends on the SDK, so both install into one environment. See
-  [docs/sdk.md](docs/sdk.md).
+  promotes them with `evalshift capture sync`. The layout is the interface (on
+  disk or mirrored from a bucket) — the two packages never call each other. The
+  CLI imports as `evalshift_cli` and depends on the SDK, so both install into
+  one environment. See [docs/sdk.md](docs/sdk.md).
 - **GitHub Action** — `evalshift/evalshift-action@v0`. Runs the pipeline on
   pull requests, pushes the run, keeps one PR comment updated, sets the
   `evalshift/regression` commit status. See [docs/github-action.md](docs/github-action.md).

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -9,6 +9,40 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- `captures.store` in `evalshift.yaml`: name the object store the capture SDK
+  ships captures to — `s3://<bucket>/<prefix>`, `gs://<bucket>/<prefix>` or
+  `az://<account>/<container>/<prefix>` — and `capture sync` and `capture list`
+  mirror new captures and toolset sidecars from it into `.evalshift/` before
+  they run, with no flags. Only objects missing locally are downloaded
+  (captures are immutable, toolsets content-addressed), captures already
+  promoted are skipped so `capture clean` never causes a re-download, and
+  every file is written atomically. New `capture fetch` does only that step;
+  `--since 30m|24h|7d|<ISO>` limits a fetch to recently written captures and
+  `--offline` skips it. Keys from a bucket are validated against the SDK's
+  layout and anything else is ignored as malformed — including a suite
+  directory containing `:`, `/`, `\`, NUL or `..`, or starting or ending with
+  `.` or a space, and a key with a trailing newline — and no destination can
+  land outside `.evalshift/`; folder-marker keys ending in `/` (the S3
+  console's "Create folder", ADLS directories) are skipped without being
+  counted. A list or download failure exits 1 with a hint naming the
+  provider's credential chain (`aws sso login` / `AWS_PROFILE` / the CI OIDC
+  role, `gcloud auth application-default login`, `az login`); a local write
+  failure names the file and asks you to check disk space and permissions on
+  its directory instead. Extras:
+  `evalshift[s3]`, `[gcs]`, `[azure]` (azure-storage-blob and azure-identity);
+  a missing one exits 1 naming the `pip install` to run. Credentials come from
+  each provider's default chain and are rejected in the URI: a value with `@`
+  or `?` fails to load, naming the accepted forms. No rejected value is echoed
+  back (it may be a pasted key or connection string); at most its scheme is
+  named.
+  `capture list --json` keeps stdout pure JSON (the fetch line and warnings
+  go to stderr). `doctor` reports whether the store is installed (`fail`, exit 1,
+  when the extra is missing) and reachable (`warn` when it cannot be listed).
+  Without `captures.store` nothing changes; a `sync` or `list` whose
+  `evalshift.yaml` cannot be read or does not validate skips the fetch,
+  warning only when the file has a top-level `captures:` key. Older CLIs
+  reject the `captures:` key, so bump the CLI (and a pinned CI action's
+  `evalshift-version`) before adding it.
 - `evaluators.trace_invariants`: hand-written rules over tool-call traces —
   `forbidden`, `required`, `order`, `call_count` (`min_calls` and/or
   `max_calls` per tool; equal bounds = an exact count) and

diff --git a/DOCS.md b/DOCS.md
diff --git a/README.md b/README.md
@@ -310,7 +310,8 @@ assets, works offline) has:
 
 Your prompts and suite stay local for `doctor`, `run`, `evaluate`, `analyze`,
 and `report`. The only outbound calls in local mode are to the LLM providers you
-configure — any provider LiteLLM supports, called with your own API keys.
+configure — any provider LiteLLM supports, called with your own API keys — and,
+if you set `captures.store`, reads from your own capture bucket.
 Anthropic, DeepSeek, Google and OpenAI ids additionally get a curated pricing and
 capability entry; everything else is passed through with the provider inferred
 from the id.

diff --git a/docs/configuration.md b/docs/configuration.md
@@ -21,6 +21,7 @@ evaluators: {...}         # optional (but at least one is needed for `evaluate`)
 suites: {...}             # optional named suites for `run --suite-name`, each with
                           # its own optional `evaluators:` block
 retention: {...}          # optional run-history pruning policy
+captures: {...}           # optional, see below
 ```
 
 Unknown keys are rejected (`extra: forbid` everywhere) so typos fail
@@ -1013,6 +1014,50 @@ evalshift runs clean --suite main_chat    # restrict to one suite
 `--keep` beats `EVALSHIFT_MAX_RUNS`, which beats the config value. `runs clean` works even without a
 valid `evalshift.yaml` (it falls back to the defaults), so it's always available for disk cleanup.
 
+## `captures`
+
+Where the capture SDK's recordings live when they are not on this machine. Omit the block
+entirely for the default: captures are read from `.evalshift/captures/` (or `EVALSHIFT_DIR`)
+on local disk and nothing here is needed.
+
+Production agents on Fargate, Lambda or Kubernetes lose their disk when they stop, so the
+SDK can ship captures to an object store you own instead (`EVALSHIFT_SINK=<uri>` in the
+agent's environment — see [Capture SDK](sdk.md)). Name the same store here and the CLI
+pulls new captures down before it promotes them, with no flags to remember:
+
+```yaml
+captures:
+  store: s3://acme-evals/support-agent
+```
+
+| Field   | Type          | Default | Description |
+| ------- | ------------- | ------- | ----------- |
+| `store` | string (URI)  | (none)  | Object store the SDK writes to. `capture sync` and `capture list` fetch new captures and toolset sidecars from it into the local directory first; `capture fetch` does only that step. |
+
+CLI versions before this release reject the `captures:` key (the config is strict), so bump
+the CLI — and a pinned GitHub Action's `evalshift-version` — before adding it.
+
+Accepted URI forms — the same grammar the SDK uses:
+
+| Form | Store | Install | Credentials |
+| --- | --- | --- | --- |
+| `s3://<bucket>/<prefix>` | Amazon S3; MinIO, Cloudflare R2, Backblaze B2 via boto3's `AWS_ENDPOINT_URL` | `pip install "evalshift[s3]"` | IAM role in CI (OIDC), `aws sso login` locally |
+| `gs://<bucket>/<prefix>` | Google Cloud Storage | `pip install "evalshift[gcs]"` | Workload Identity in CI, `gcloud auth application-default login` locally |
+| `az://<account>/<container>/<prefix>` | Azure Blob Storage | `pip install "evalshift[azure]"` | Managed Identity / federated credential in CI, `az login` locally |
+
+`<prefix>` is optional. Credentials never go in the URI: a value containing `@` or `?` fails
+to load, naming the accepted forms. A rejected value is never echoed back (it may be a pasted
+key or connection string); the error names at most its scheme. Under the prefix the layout is exactly the local one —
+`captures/<suite>/cap_<hex>.json` and `toolsets/<hex>.json` — so the bucket is a mirror of
+`.evalshift/`, not a different format.
+
+What the fetch does: it lists the bucket, downloads only objects missing locally (captures
+are immutable and toolsets content-addressed, so "exists" means "current"), skips captures
+already promoted into a suite (so `capture clean` never causes a re-download), and writes
+each file atomically. `--since 24h` (or `7d`, `30m`, an ISO date) limits a fetch to recently
+written captures; `--offline` skips it and works with what is already local. Retention in the bucket
+is yours to set with a lifecycle rule; the SDK's `max_captures` does not apply there.
+
 ## Capture lifecycle
 
 The companion `evalshift-sdk` package records real agent runs to
@@ -1021,11 +1066,16 @@ the base). The CLI consumes them via the `capture` commands:
 
 ```shell
 evalshift capture list                          # see what the SDK recorded
+evalshift capture fetch                         # (captures.store only) mirror new captures locally
 evalshift capture sync --input-var query        # promote every capture + wire suites:
 evalshift run --suite-name support_agent --yes  # score a candidate model against it
 evalshift capture clean                         # prune already-promoted captures
 ```
 
+With [`captures.store`](#captures) configured, `list` and `sync` run the same fetch first
+and print one summary line (`fetched 12 capture(s) and 1 toolset(s) from s3://…`); pass
+`--offline` to skip it. `promote`, `diff` and `clean` operate on the local mirror only.
+
 `capture clean [<suite>]` deletes promoted captures by default (`--promoted`);
 `--all` deletes every capture, promoted or not. It never touches promoted
 suites, then offers to sweep toolset sidecars no surviving capture or suite

diff --git a/docs/getting-started.md b/docs/getting-started.md
@@ -123,6 +123,11 @@ Then exercise the agent with capture turned on:
 EVALSHIFT_CAPTURE=1 python your_agent.py   # writes .evalshift/captures/
 ```
 
+Running on Fargate, Lambda or Kubernetes, where the disk does not outlive the task? Add
+`EVALSHIFT_SINK=s3://<bucket>/<prefix>` (or `gs://…` / `az://…`) next to it and put the same
+URI under `captures: {store: …}` in `evalshift.yaml`; step 6 then fetches from the bucket
+first. See [`captures`](configuration.md#captures).
+
 Captures are off unless `EVALSHIFT_CAPTURE=1` is set, so the decorator can
 stay in production code. If the agent calls OpenAI, Anthropic or Google GenAI
 directly, wrap the client once — `wrap_openai(OpenAI())`, `wrap_anthropic`,

diff --git a/docs/hosted.md b/docs/hosted.md
@@ -271,8 +271,8 @@ to clear a push with a security or compliance team, this is the section to
 hand them.
 
 The CLI contains **no telemetry**: no analytics, no crash reporting, no
-phone-home of any kind. It opens exactly two kinds of network connections,
-both initiated by you:
+phone-home of any kind. It opens two kinds of network connections (three with
+`captures.store`), all initiated by you:
 
 1. **Your model providers** (whichever you configure — any provider LiteLLM
    supports), using your own API keys: `run` sends the rendered prompts and
@@ -285,6 +285,11 @@ both initiated by you:
    commands — `doctor`, `run`, `evaluate`, `analyze`, `report`, `bundle` —
    send nothing to EvalShift-operated services.
 
+With [`captures.store`](configuration.md#captures) configured, `capture sync`,
+`capture list`, `capture fetch` and `doctor` also read from that object store —
+a bucket you own, with your own cloud credentials. Nothing from it goes to
+EvalShift.
+
 ### What `push` sends, block by block
 
 `push` uploads `run_bundle.json.gz` plus two pieces of request metadata: the
@@ -322,7 +327,9 @@ The bundle itself contains:
   imported agent traces (`traces.jsonl`; only the verdicts computed on them
   upload — `agent_trace` scores and `trace_invariants` rule results — never
   the timelines; see the `examples[]` row above), the SQLite response cache,
-  `.evalshift/captures/`, `state.json`, `report.json`, and `report.html`.
+  `.evalshift/captures/` (and, with `captures.store` configured, the bucket
+  *you* named that they are mirrored from — read with your credentials, never
+  uploaded to EvalShift), `state.json`, `report.json`, and `report.html`.
 
 The content hashes that replace this data (`dataset_hash`, `examples_hash`,
 `prompts[].content_hash`) are SHA-256 digests, so hosted diffs and baselines

diff --git a/docs/index.md b/docs/index.md
@@ -98,7 +98,8 @@ evalshift runs clean                    # prune old runs (--keep, --older-than,
 
 For local commands, your prompts and your suite never leave your machine.
 The only outbound calls are to the LLM providers you configure — any provider
-LiteLLM supports — using your own API keys. `bundle` packages artifacts locally;
+LiteLLM supports — using your own API keys — and, if you set `captures.store`,
+reads from your own capture bucket with your own cloud credentials. `bundle` packages artifacts locally;
 hosted upload happens only when you run `push` or `compare --push`.
 
 ## For AI coding agents

diff --git a/docs/sdk.md b/docs/sdk.md
@@ -14,20 +14,25 @@ dense LLM reference at <https://www.evalshift.dev/sdk-llms-full.txt>.
 
 ```
 your agent (evalshift-sdk)  →  .evalshift/captures/<suite>/cap_<hex>.json
-                            →  evalshift capture sync
+                            →  or, EVALSHIFT_SINK=s3://… : <prefix>/captures/<suite>/cap_<hex>.json
+                            →  evalshift capture sync       (fetches from captures.store first)
                             →  .evalshift/suites/<suite>/golden.jsonl + evalshift.yaml
 ```
 
-- **Disk is the only interface.** The SDK never imports or calls the CLI, and
-  the CLI reads captures from disk without calling SDK code (`doctor` only
-  checks which package the `evalshift` import name resolves to).
+- **The layout is the interface.** The SDK never imports or calls the CLI, and the CLI reads
+  captures from disk without calling SDK code (`doctor` only checks which package the
+  `evalshift` import name resolves to). The same layout — `captures/<suite>/cap_<hex>.json`
+  plus `toolsets/<hex>.json` — is what the SDK writes to local disk *or* under a prefix in an
+  object store you own; the CLI mirrors the latter into `.evalshift/` and reads it unchanged.
 - **One environment or two.** The CLI imports as `evalshift_cli` and depends on
   the SDK, so `pip install evalshift` gives you both and `import evalshift` is
   always the SDK. A production agent that only records captures installs
   `evalshift-sdk` alone.
 - **Off by default.** Nothing is recorded unless `EVALSHIFT_CAPTURE=1` is set,
   so the instrumentation is safe to leave in production code permanently.
-- **No network.** The SDK writes local files only. Python 3.10+, stdlib-only.
+- **No network by default.** The SDK writes local files unless `EVALSHIFT_SINK` names a bucket
+  you own (`s3://`, `gs://`, `az://`, SDK 0.5.0+), and then it writes only there — never to
+  EvalShift. Python 3.10+, stdlib-only; the cloud clients are optional extras.
 
 ## 1. Instrument the agent
 
@@ -103,12 +108,26 @@ newest captures. `EVALSHIFT_MAX_CAPTURES`, `EVALSHIFT_DEDUP`,
 `EVALSHIFT_CAPTURE_TTL` and `EVALSHIFT_SAMPLE_RATE` tune that; they are read by
 the SDK, in your agent's process.
 
+```bash
+# on a host whose disk does not outlive it (Fargate, Lambda, pods):
+EVALSHIFT_CAPTURE=1 EVALSHIFT_SINK=s3://acme-evals/support-agent python your_agent.py
+```
+
+The SDK ships each capture (and its toolset sidecar) to the bucket on a background thread,
+fail-open; see the SDK docs for the `SIGTERM` / Lambda flush note. Bucket retention is a
+lifecycle rule; `EVALSHIFT_MAX_CAPTURES` applies to local disk only.
+
 ## 3. Promote captures into suites
 
-Run the CLI from the directory holding `.evalshift/` (or set `EVALSHIFT_DIR`):
+Run the CLI from the directory holding `.evalshift/` (or set `EVALSHIFT_DIR`).
+If the captures live in a bucket, name it once in `evalshift.yaml`
+(`captures: {store: s3://acme-evals/support-agent}`) and install `evalshift[s3]` (or `[gcs]` /
+`[azure]`); `list` and `sync` then fetch new captures first. `evalshift capture fetch` does
+only that step.
 
 ```bash
 evalshift capture list                  # what was recorded (--json for machine output)
+evalshift capture fetch                 # mirror new captures from captures.store
 evalshift capture sync                  # promote ALL captures → suites + wire evalshift.yaml
 evalshift capture promote cap_ab12 --as refund_case_1
 evalshift capture diff cap_ab12 cap_cd34