From 2c7abf1758def2de97f936a1715aa0249799dda9 Mon Sep 17 00:00:00 2001 From: Lukas Babaliauskas Date: Fri, 9 Oct 2026 21:53:57 +0200 Subject: [PATCH 1/8] docs(spec): object-store sink fails loudly on a missing client library Co-Authored-By: Claude Fable 5.1 --- ...-09-sink-dependency-loud-failure-design.md | 173 ++++++++++++++++++ 1 file changed, 173 insertions(+) create mode 100644 docs/superpowers/specs/2026-10-09-sink-dependency-loud-failure-design.md diff --git a/docs/superpowers/specs/2026-10-09-sink-dependency-loud-failure-design.md b/docs/superpowers/specs/2026-10-09-sink-dependency-loud-failure-design.md new file mode 100644 index 00000000..17131b7a --- /dev/null +++ b/docs/superpowers/specs/2026-10-09-sink-dependency-loud-failure-design.md @@ -0,0 +1,173 @@ +# Object-store sink: fail loudly on a missing client library, name the package + +**Date:** 2026-10-09 · **Status:** draft for maintainer review · **Repos:** evalshift-sdk, evalshift-cli (behaviour); evalshift-client (docs only); evalshift-action (nothing) + +## Problem + +SDK 0.5.0 / CLI 1.3.0 shipped the object-store sink behind pip extras (`evalshift-sdk[s3]`, +`[gcs]`, `[azure]`; `evalshift[s3]` …). Two things are wrong with how that fails: + +1. **The SDK fails quietly.** `EVALSHIFT_SINK=s3://…` with no boto3 installed logs one + `WARNING` at import and writes captures to local disk. On the ephemeral hosts the feature + exists for, that is the exact loss it was meant to prevent, and a green deploy hides it. +2. **The fix is phrased in packaging jargon.** The error and every doc say "install the `[s3]` + extra". A user who installed `evalshift-sdk` the normal way does not know what an extra is, + and `pip install "evalshift-sdk[s3]"` needs quotes in zsh. + +Side effect on the CLI: `doctor` returns `fail` for `captures.store` when the library is +missing, `compare` exits 1 on any `fail`, and the GitHub Action installs the bare package, so +every repo that sets `captures.store` has a red CI (reproduced 2026-10-09). + +## Decision + +- **Misconfiguration raises; runtime faults stay fail-open.** A sink the SDK cannot build from + `EVALSHIFT_SINK` — client library missing, URI malformed, store failing to open — is refused + at the agent's startup with one exception naming the fix. Network errors, failed puts, full + queues and exit-flush timeouts keep today's fail-open behaviour. This is a documented carve-out + from "fail-open is the core promise", in the same table as redaction and the read side. +- **Raise at the first explicit touch, never on bare import.** `import evalshift` stays safe + (the CLI imports it; a developer may have the env vars exported in a shell). The raise happens + where the user's own code first asks the SDK to capture. +- **Name the library, not the extra.** Every message and every doc says `pip install boto3` + (`google-cloud-storage`, `azure-storage-blob azure-identity`). The extras stay in + `pyproject.toml` as tested version floors; nothing user-facing mentions them. +- **The CLI keeps exit 1 on the capture commands, but `doctor` only warns.** A CI run on a + committed suite never touches the bucket. + +Not doing: bundling the cloud SDKs into the base package (three heavy dependency trees into +every agent; "stdlib-only" is a documented promise), an extras input on the Action (unneeded +once `doctor` warns), vendoring a client. + +## SDK (evalshift-sdk 0.6.0) + +### Startup validation + +`_Config` is built at import (`config.py:181`) and `_env_sink()` runs then. Change `_env_sink()` +so a misconfigured `EVALSHIFT_SINK` is **recorded, not swallowed**: `_Config` gains +`sink_error: SinkConfigurationError | None`; `sink` stays `None` (the `FileSink` default is not +used while an error is recorded — see "Hot path"). Nothing is raised at import. The current +`WARNING` lines go away; the error carries the same text. + +A blank or unset `EVALSHIFT_SINK` records nothing (unchanged). + +### Touch points that raise + +When `EVALSHIFT_CAPTURE` is truthy **and** `sink_error` is recorded, these raise it: + +| Touch point | When it runs in a real agent | +|---|---| +| `capture.agent(...)`, `capture.agent_session(...)`, `capture.tool(...)` applied as decorators | the user's module import, i.e. process startup | +| `wrap_openai(...)`, `wrap_anthropic(...)`, `wrap_genai(...)` | client construction, startup | +| `EvalShiftCallbackHandler(...)` | handler construction, startup | +| `configure(...)` called **without** an explicit `sink` | startup | + +`configure(sink=)` clears `sink_error`: the user replaced the env sink with +something that exists. The gate is read live, so a process started with capture off never raises. + +The check is one function in `config.py`, `require_sink_ready()`, that every touch point calls; +it is a no-op when the gate is off or nothing is recorded. + +### Hot path + +If a capture is attempted while `sink_error` is recorded (possible only when the gate turned on +after every touch point ran), the SDK **behaves as if the gate were off** for that invocation — +no span, no sidecar, no disk write — and logs the error text at `WARNING` once per process. +Rule: a sink the user asked for is never silently replaced by local disk. + +### The explicit-sink path + +`S3Store(...)`, `GCSStore(...)`, `AzureBlobStore(...)` import their client lazily. On +`ImportError` they raise `MissingStoreDependencyError` (renamed from `MissingExtraError`, same +base `ImportError`, same module `evalshift.stores.uri`) whose text names the pip package(s), not +the extra. `open_store()` raises the same. No alias for the old name: 0.5.0 shipped today and the +rename is listed under *Changed* in the changelog. + +### Messages + +All raised and logged text follows the existing rule: never the raw env value. Scheme, variable +name, package names and accepted forms only. A grammar error is raised `from None` so the +parser's quoted-URI message cannot appear as a chained cause. + +| Case | `SinkConfigurationError` text | +|---|---| +| library missing | `EVALSHIFT_SINK names an s3:// store, but boto3 is not installed. Run: pip install boto3 — or unset EVALSHIFT_SINK to capture to local disk.` | +| malformed URI | `EVALSHIFT_SINK is not a valid store URI. Accepted forms: s3:///, gs:///, az:////. Credentials never go in the URI.` | +| store failed to open | `EVALSHIFT_SINK could not be opened (). Unset it to capture to local disk.` | + +Scheme → pip packages, kept **verbatim in both repos'** `uri.py` next to `_REQUIRED_MODULES`: + +| scheme | required modules | `pip install` | +|---|---|---| +| `s3` | `boto3` | `boto3` | +| `gs` | `google.cloud.storage` | `google-cloud-storage` | +| `az` | `azure.storage.blob`, `azure.identity` | `azure-storage-blob azure-identity` | + +The Azure message names both packages in one command even if only one is missing. + +### Public API changes + +- New `evalshift.SinkConfigurationError` (subclass of `RuntimeError`), exported at top level. +- `evalshift.stores.uri.MissingExtraError` → `MissingStoreDependencyError`. +- `StoreURI.extra` stays (the extras still exist) but gains `StoreURI.packages` → the + `pip install` argument string above; messages use `packages`. +- Fail-open table in DOCS.md gains a row: *Sink configuration — raises `SinkConfigurationError` + at startup — a bucket you asked for and did not get is the loss this feature exists to + prevent.* DECISIONS.md D-stores: the "Warning-level logging" and "Adapters … Extras" bullets + are rewritten accordingly (new sub-decision D-stores-b). + +## CLI (evalshift 1.3.1) + +- `RemoteStoreUnavailable` text: summary `s3:// captures store needs boto3, which is not + installed`, hint `install it: pip install boto3` (packages from the shared table). +- `doctor`'s `captures.store` row: library missing → `warn` with that text; bucket not + listable → `warn` (unchanged); listable → `ok`. `doctor` exits 1 only on an invalid existing + config again. Docstrings and DOCS.md:872 updated. +- `capture fetch` / `list` / `sync`: unchanged — exit 1 with the new text when the library is + missing (the user asked for a store the command cannot read). +- Consequence: `compare` and the Action no longer fail on a config that names a store. + +## Docs + +Every surface that says "extra", "`[s3]`" or "falls back to local disk" changes. Enumerate with +`git grep` at implementation time; known sites today: + +- **sdk:** README.md, DOCS.md (install, env table, `ObjectStoreSink` section, fail-open table, + troubleshooting), llms.txt, llms-full.txt, docs/DECISIONS.md, CHANGELOG.md. +- **cli:** docs/configuration.md (`captures` URI table "Install" column), docs/sdk.md, DOCS.md + (`doctor` paragraph, capture section, exit-code sentence), llms-full.txt, CHANGELOG.md, + `doctor.py` / `remote.py` / `uri.py` docstrings. +- **client (after both releases):** SdkConfig.tsx, Captures.tsx, Configuration.tsx, Changelog.tsx, + version.ts, then `npm run sync:llms`. +- **action:** nothing (its docs never mention the extras; the troubleshooting entry in PR #27 is + about pin skew and stays true). + +Open PRs: **sdk #14** (README extras) is superseded — close it and fold the README rewrite into +the 0.6.0 PR. **client #55** documents what is live today (0.5.0 / 1.3.0) and merges as is; the +site follows the releases with the follow-up above. + +## Release order + +SDK 0.6.0 and CLI 1.3.1 are independent; no server or action change. Ship both, then the client +follow-up, then `npm run sync:llms`. + +## Testing + +- **SDK** `tests/test_config_sink_env.py` is rewritten: bare import with a broken sink does not + raise; each touch point raises `SinkConfigurationError` with the gate on and not with it off; + `configure(sink=MemorySink())` clears the error; the secret-bearing-value test keeps asserting + the value never appears in the exception text or the chained cause; the hot-path rule (gate + turned on late → no file written, one `WARNING`); the Azure message names both packages; + `MissingStoreDependencyError` from `S3Store()` names `pip install boto3`. The subprocess test + gains a case proving `import evalshift` exits 0 with a broken `EVALSHIFT_SINK`. +- **CLI** `doctor` test: missing library → `warn`, exit 0; capture-command tests assert the new + hint text; the docs-currency test gains `"[s3]"`-style extras and `"optional dependency"` as + retired terms across the prose files. + +## Judgement calls to confirm + +1. Hot-path rule drops the capture rather than writing to disk when the gate turns on late. + Alternative: keep today's disk fallback with the warning for that one case. +2. `MissingExtraError` is renamed without an alias. +3. `SinkConfigurationError` subclasses `RuntimeError`, not `ImportError`, because the malformed + URI case is not an import problem; the library-missing case keeps `ImportError` semantics + through `MissingStoreDependencyError` on the explicit path only. From 23928b5254c5805db94050935da74418fe66ebd4 Mon Sep 17 00:00:00 2001 From: Lukas Babaliauskas Date: Fri, 9 Oct 2026 21:59:13 +0200 Subject: [PATCH 2/8] docs(spec): gate-off never raises; the docs PRs are merged Co-Authored-By: Claude Fable 5.1 --- .../2026-10-09-sink-dependency-loud-failure-design.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/superpowers/specs/2026-10-09-sink-dependency-loud-failure-design.md b/docs/superpowers/specs/2026-10-09-sink-dependency-loud-failure-design.md index 17131b7a..75fae4da 100644 --- a/docs/superpowers/specs/2026-10-09-sink-dependency-loud-failure-design.md +++ b/docs/superpowers/specs/2026-10-09-sink-dependency-loud-failure-design.md @@ -48,7 +48,10 @@ so a misconfigured `EVALSHIFT_SINK` is **recorded, not swallowed**: `_Config` ga used while an error is recorded — see "Hot path"). Nothing is raised at import. The current `WARNING` lines go away; the error carries the same text. -A blank or unset `EVALSHIFT_SINK` records nothing (unchanged). +A blank or unset `EVALSHIFT_SINK` records nothing (unchanged). Recording is independent of the +gate (as building the env sink is today), but **nothing below happens unless `EVALSHIFT_CAPTURE` +is truthy**: with capture off, a broken `EVALSHIFT_SINK` never raises, never logs, never changes +behaviour. ### Touch points that raise @@ -141,9 +144,9 @@ Every surface that says "extra", "`[s3]`" or "falls back to local disk" changes. - **action:** nothing (its docs never mention the extras; the troubleshooting entry in PR #27 is about pin skew and stays true). -Open PRs: **sdk #14** (README extras) is superseded — close it and fold the README rewrite into -the 0.6.0 PR. **client #55** documents what is live today (0.5.0 / 1.3.0) and merges as is; the -site follows the releases with the follow-up above. +The 2026-10-09 docs PRs (sdk #14, cli #42, action #27, server #36, client #55) are merged and +describe the extras-based install story of 0.5.0 / 1.3.0 accurately; the 0.6.0 and 1.3.1 PRs +rewrite those passages on top, and the site follows the releases with the follow-up above. ## Release order From 2f5f2c4f1992b2de89bd765c5985ff92379ff581 Mon Sep 17 00:00:00 2001 From: Lukas Babaliauskas Date: Fri, 9 Oct 2026 22:11:59 +0200 Subject: [PATCH 3/8] =?UTF-8?q?docs(plan):=20sink=20dependency=20loud=20fa?= =?UTF-8?q?ilure=20=E2=80=94=20implementation=20plan?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- ...2026-10-09-sink-dependency-loud-failure.md | 1262 +++++++++++++++++ ...-09-sink-dependency-loud-failure-design.md | 15 +- 2 files changed, 1271 insertions(+), 6 deletions(-) create mode 100644 docs/superpowers/plans/2026-10-09-sink-dependency-loud-failure.md diff --git a/docs/superpowers/plans/2026-10-09-sink-dependency-loud-failure.md b/docs/superpowers/plans/2026-10-09-sink-dependency-loud-failure.md new file mode 100644 index 00000000..2b979676 --- /dev/null +++ b/docs/superpowers/plans/2026-10-09-sink-dependency-loud-failure.md @@ -0,0 +1,1262 @@ +# Sink Dependency Loud Failure Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** A misconfigured `EVALSHIFT_SINK` (client library missing, URI malformed) stops the agent at startup with one exception that says `pip install boto3`, instead of logging a warning and writing captures to local disk; the CLI names packages the same way and `doctor` stops failing CI over it. + +**Architecture:** The SDK records the problem when its process-wide config is built at import, raises it from `config.require_sink_ready()` at every explicit touch point (the `capture.*` decorators, the client wrappers, the LangChain handler, `configure()` without a sink) while `EVALSHIFT_CAPTURE` is truthy, and makes `is_capture_enabled()` return `False` (one warning) as a backstop if the gate turns on later. The CLI changes only messages and the `doctor` row status. Both repos keep the store-URI grammar module as the shared, verbatim home of the scheme → package tables. + +**Tech Stack:** Python 3.10+ (SDK, stdlib-only, `uv run pytest` / `ruff` / `mypy`), Python 3.11+ (CLI, `pytest` / `ruff` / `mypy --strict` via `.venv`), Typer, Rich, pytest `monkeypatch` + `caplog`. + +**Spec:** `docs/superpowers/plans/../specs/2026-10-09-sink-dependency-loud-failure-design.md` (this repo: `evalshift-cli/docs/superpowers/specs/2026-10-09-sink-dependency-loud-failure-design.md`) + +## Global Constraints + +- SDK stays stdlib-only: `dependencies = []` in `evalshift-sdk/pyproject.toml`; cloud clients are imported lazily and only checked with `importlib.util.find_spec`. +- The extras `[s3]`, `[gcs]`, `[azure]` stay in both `pyproject.toml` files; nothing user-facing (messages, docs) mentions them again. +- No raised or logged text ever contains the raw `EVALSHIFT_SINK` / `captures.store` value; grammar errors are raised `from None`. +- Nothing raises, logs or changes behaviour unless `EVALSHIFT_CAPTURE` is truthy (`1`/`true`/`yes`/`on`, case-insensitive). Bare `import evalshift` never raises. +- Scheme → module → package tables are identical in `evalshift-sdk/src/evalshift/stores/uri.py` and `evalshift-cli/src/evalshift_cli/captures/store_uri.py`. +- Messages: SDK missing library → `EVALSHIFT_SINK points at an object store (s3://), but boto3 is not installed. Run: pip install boto3, or unset EVALSHIFT_SINK to capture to local disk.`; CLI → summary `s3:// captures store needs boto3, which is not installed`, hint `install it: pip install boto3`. Azure names both packages: `pip install azure-storage-blob azure-identity`. +- Gates: SDK `uv run ruff check && uv run ruff format --check && uv run mypy && uv run pytest`; CLI `ruff check . && ruff format --check . && mypy --strict src/evalshift_cli && pytest` with `PATH=$PWD/.venv/bin:$PATH` (the commit and push hooks need it). +- Conventional Commits; every commit message ends with `Co-Authored-By: Claude Fable 5.1 `. Run `git` with `-C ` or from inside the repo; never run two repos' git commands in parallel shells. +- Versions: SDK 0.6.0, CLI 1.3.1, bumped in the release task only (not in feature commits). + +## Review Focus + +1. `EVALSHIFT_SINK=ftp://x` with `EVALSHIFT_CAPTURE` unset: decorating, wrapping, `configure()` and `is_capture_enabled()` must neither raise nor log — Task 4 `test_gate_off_never_raises_or_warns`. +2. A malformed value carrying a secret (`az://acct?sig=SECRET`, an Azure connection string): the raised exception's full formatted traceback must not contain it — Task 3 `test_recorded_error_never_leaks_a_secret_bearing_value`. +3. The CLI process (`evalshift doctor` imports `evalshift`) with both env vars exported in the shell and no boto3: must still exit normally — Task 5 `test_import_with_a_broken_sink_exits_zero_and_logs_nothing`. +4. Azure with `azure-storage-blob` installed but not `azure-identity`: the message names both packages in one `pip install` — Task 1 `test_open_store_azure_names_both_packages_when_identity_is_missing`. +5. Gate turned on after every touch point ran (env set mid-process): no capture file on disk, one warning, agent output unchanged — Task 3 `test_gate_turned_on_late_drops_captures_instead_of_writing_to_disk`. + +--- + +# Part A — evalshift-sdk (0.6.0) + +Work in `/home/lukas/repos/evalshift/evalshift-sdk` on branch `feat/sink-loud-failure` from `main` (`git -C /home/lukas/repos/evalshift/evalshift-sdk checkout -b feat/sink-loud-failure main`). All paths below are relative to that repo. + +### Task 1: Package tables and `MissingStoreDependencyError` in the URI grammar + +**Files:** +- Modify: `src/evalshift/stores/uri.py` (lines 29, 50-53, 96-115, 118-140, 143-150) +- Modify: `tests/test_stores_adapters.py:18,141-156` +- Grep-and-fix: every `MissingExtraError` in `src/`, `tests/`, `DOCS.md`, `llms.txt`, `llms-full.txt` (`git grep -n MissingExtraError`) + +**Interfaces:** +- Produces: `_PACKAGE_FOR_MODULE: dict[str, str]`, `_PACKAGES_FOR_SCHEME: dict[str, str]`, `StoreURI.packages -> str`, `class MissingStoreDependencyError(ImportError)` with attributes `scheme`, `module`, `package`, `packages`, `extra`, and `require_store_modules(scheme: Scheme) -> None`. `open_store(uri)` raises `MissingStoreDependencyError` (was `MissingExtraError`). + +- [ ] **Step 1: Write the failing tests** + +Replace `test_open_store_missing_extra_names_the_pip_extra` and `test_open_store_azure_needs_azure_identity_too` in `tests/test_stores_adapters.py` (the file already imports `from evalshift.stores import uri as uri_module` and patches `uri_module._installed`; keep that) with: + +```python +def test_open_store_missing_library_names_the_package(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(uri_module, "_installed", lambda module: False) + with pytest.raises(MissingStoreDependencyError) as info: + open_store("gs://b/p") + err = info.value + assert str(err) == ( + "gs:// store needs google-cloud-storage, which is not installed; " + "run: pip install google-cloud-storage" + ) + assert (err.scheme, err.module, err.package, err.packages, err.extra) == ( + "gs", "google.cloud.storage", "google-cloud-storage", "google-cloud-storage", "gcs" + ) + assert "[gcs]" not in str(err) + + +def test_open_store_azure_names_both_packages_when_identity_is_missing( + monkeypatch: pytest.MonkeyPatch, +) -> None: + # azure-storage-blob alone is not enough: DefaultAzureCredential lives in azure-identity, + # and the first put would fail in the background. One pip command installs both. + monkeypatch.setattr(uri_module, "_installed", lambda module: module != "azure.identity") + with pytest.raises(MissingStoreDependencyError) as info: + open_store("az://a/c/p") + assert info.value.package == "azure-identity" + assert str(info.value).endswith("run: pip install azure-storage-blob azure-identity") + + +def test_require_store_modules_is_a_no_op_when_everything_is_installed( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(uri_module, "_installed", lambda module: True) + require_store_modules("az") + + +def test_store_uri_packages_property() -> None: + assert parse_store_uri("s3://b/p").packages == "boto3" + assert parse_store_uri("gs://b/p").packages == "google-cloud-storage" + assert parse_store_uri("az://a/c/p").packages == "azure-storage-blob azure-identity" +``` + +Update the import at line 18 to `from evalshift.stores.uri import MissingStoreDependencyError, open_store, parse_store_uri, require_store_modules`. + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `cd /home/lukas/repos/evalshift/evalshift-sdk && uv run pytest tests/test_stores_adapters.py -q` +Expected: ImportError on `MissingStoreDependencyError`. + +- [ ] **Step 3: Implement** + +In `src/evalshift/stores/uri.py`, after `_EXTRA_FOR_SCHEME` (line 29) add: + +```python +#: The pip distribution that provides each client module. Shared verbatim with the CLI. +_PACKAGE_FOR_MODULE: dict[str, str] = { + "boto3": "boto3", + "google.cloud.storage": "google-cloud-storage", + "azure.storage.blob": "azure-storage-blob", + "azure.identity": "azure-identity", +} + +#: The ``pip install`` argument that installs everything a scheme needs, in one command. +#: Named in every message instead of the extras: a user who installed the SDK the normal way +#: has no reason to know what an extra is. Shared verbatim with the CLI. +_PACKAGES_FOR_SCHEME: dict[str, str] = { + "s3": "boto3", + "gs": "google-cloud-storage", + "az": "azure-storage-blob azure-identity", +} +``` + +Add to `StoreURI` after the `extra` property: + +```python + @property + def packages(self) -> str: + """The ``pip install`` argument that installs this scheme's client library (or libraries).""" + return _PACKAGES_FOR_SCHEME[self.scheme] +``` + +Replace `class MissingExtraError` (lines 107-115) with: + +```python +class MissingStoreDependencyError(ImportError): + """The client library for a store scheme is not installed. + + Attributes: + scheme: ``"s3"``, ``"gs"`` or ``"az"``. + module: The first module that could not be found. + package: The pip distribution that provides ``module``. + packages: The ``pip install`` argument that installs everything the scheme needs. + extra: The pip extra that pins the same packages, for callers that prefer it. + + The message names ``package`` and ``packages`` only -- never an extra, never a URI. + """ + + def __init__(self, scheme: Scheme, module: str) -> None: + self.scheme = scheme + self.module = module + self.package = _PACKAGE_FOR_MODULE[module] + self.packages = _PACKAGES_FOR_SCHEME[scheme] + self.extra = _EXTRA_FOR_SCHEME[scheme] + super().__init__( + f"{scheme}:// store needs {self.package}, which is not installed; " + f"run: pip install {self.packages}" + ) +``` + +Replace the module-check loop at the top of `open_store` with a call to a new helper placed right after `_installed`: + +```python +def require_store_modules(scheme: Scheme) -> None: + """Raise :class:`MissingStoreDependencyError` unless every module ``scheme`` needs is importable. + + Uses ``importlib.util.find_spec`` only, so nothing is imported and nothing touches the + network: safe at process start and at store construction. + """ + for module in _REQUIRED_MODULES[scheme]: + if not _installed(module): + raise MissingStoreDependencyError(scheme, module) +``` + +and in `open_store`: + +```python + parsed = parse_store_uri(uri) + require_store_modules(parsed.scheme) +``` + +Update `open_store`'s docstring (`MissingStoreDependencyError: when a module the scheme's adapter needs is not installed; it names the first missing one and the package to install.`), the `_REQUIRED_MODULES` comment (`-- both ship in the ``[azure]`` extra` → `-- installed together by ``pip install azure-storage-blob azure-identity``), and `__all__`: remove `"MissingExtraError"`, add `"MissingStoreDependencyError"` and `"require_store_modules"`. Run `git grep -n MissingExtraError` and rename every remaining reference in `src/` and `tests/` (docs are rewritten in Task 6; leave them for now). + +- [ ] **Step 4: Run the tests** + +Run: `uv run pytest tests/test_stores_adapters.py tests/test_stores_uri.py -q` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/evalshift/stores/uri.py tests/test_stores_adapters.py +git commit -m "feat(stores): name the package to install, not the extra + +MissingExtraError -> MissingStoreDependencyError; its message and new +.package/.packages attributes say 'pip install boto3' (both Azure packages +in one command). The scheme tables are shared verbatim with the CLI. + +Co-Authored-By: Claude Fable 5.1 " +``` + +### Task 2: Store constructors check their library at construction + +**Files:** +- Modify: `src/evalshift/stores/s3.py:1-9,18-31`, `src/evalshift/stores/gcs.py` (docstring, `__init__`, line 29), `src/evalshift/stores/azure.py` (docstring, `__init__`, lines 31-32) +- Test: `tests/test_stores_adapters.py` + +**Interfaces:** +- Consumes: `require_store_modules` from Task 1. +- Produces: `S3Store(bucket, prefix="", *, client=None)`, `GCSStore(...)`, `AzureBlobStore(account, container, prefix="", *, client=None)` raise `MissingStoreDependencyError` at construction when `client is None` and the library is absent. Behaviour with an injected `client` is unchanged. + +- [ ] **Step 1: Write the failing tests** + +Append to `tests/test_stores_adapters.py`: + +```python +@pytest.mark.parametrize( + ("build", "package"), + [ + (lambda: S3Store("b", "p"), "boto3"), + (lambda: GCSStore("b", "p"), "google-cloud-storage"), + (lambda: AzureBlobStore("a", "c", "p"), "azure-storage-blob azure-identity"), + ], +) +def test_store_without_a_client_requires_its_library_at_construction( + monkeypatch: pytest.MonkeyPatch, build: Callable[[], object], package: str +) -> None: + # The client is still built lazily on the first put (a background thread, where a raise + # would only be a logged failure); the *presence* of the library is checked here, where + # user code constructs the store and a raise is loud. + monkeypatch.setattr(uri_module, "_installed", lambda module: False) + with pytest.raises(MissingStoreDependencyError, match=rf"pip install {package}$"): + build() + + +def test_store_with_an_injected_client_needs_no_library(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(uri_module, "_installed", lambda module: False) + S3Store("b", "p", client=object()) + GCSStore("b", "p", client=object()) + AzureBlobStore("a", "c", "p", client=object()) +``` + +Add `from collections.abc import Callable` to the test imports if missing. + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `uv run pytest tests/test_stores_adapters.py -q -k "at_construction or injected_client"` +Expected: the first test FAILS (no exception raised). + +- [ ] **Step 3: Implement** + +`src/evalshift/stores/s3.py`: change the module docstring's first line to `"""Amazon S3 (and S3-compatible) ``ObjectStore``. Needs ``boto3`` (``pip install boto3``).` and add the import and check: + +```python +from typing import Any + +from evalshift.stores.uri import require_store_modules + + +class S3Store: + """Put objects under ``s3:///``.""" + + def __init__(self, bucket: str, prefix: str = "", *, client: Any | None = None) -> None: + if client is None: + require_store_modules("s3") # loud here, in user code, not on the first background put + self.bucket = bucket +``` + +Change the lazy-import comment to `import boto3 # lazy: optional, presence checked in __init__`. + +`src/evalshift/stores/gcs.py`: same shape with `require_store_modules("gs")`, docstring `Needs ``google-cloud-storage`` (``pip install google-cloud-storage``)`, comment on line 29 `# lazy: optional, presence checked in __init__`. + +`src/evalshift/stores/azure.py`: same shape with `require_store_modules("az")`, docstring `Needs ``azure-storage-blob`` and ``azure-identity`` (``pip install azure-storage-blob azure-identity``)`, comment on line 31 `# lazy: optional, presence checked in __init__`. + +The import `evalshift.stores.uri` from the three store modules creates no cycle: `uri.py` imports the store modules only inside `open_store`. + +- [ ] **Step 4: Run the tests** + +Run: `uv run pytest tests/test_stores_adapters.py tests/test_object_store_sink.py -q` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/evalshift/stores/s3.py src/evalshift/stores/gcs.py src/evalshift/stores/azure.py tests/test_stores_adapters.py +git commit -m "feat(stores): a store built without a client checks its library at construction + +Co-Authored-By: Claude Fable 5.1 " +``` + +### Task 3: `SinkConfigurationError`, recorded at import, raised by `require_sink_ready()` + +**Files:** +- Modify: `src/evalshift/config.py:36,108-179,181-221,328-339` +- Modify: `src/evalshift/__init__.py:19,34-50` +- Modify: `tests/test_config_sink_env.py` (replace lines 1-2, 18, 51-133) + +**Interfaces:** +- Consumes: `MissingStoreDependencyError` (Task 1). +- Produces: `class SinkConfigurationError(RuntimeError)` exported from `evalshift` and `evalshift.config`; `_Config.sink_error: SinkConfigurationError | None`; `require_sink_ready() -> None` (raises when `_gate_on()` and an error is recorded); `is_capture_enabled()` now `_gate_on() and not _sink_blocked()`; `configure(sink=...)` clears `sink_error`, `configure()` without `sink` calls `require_sink_ready()`; `reset_config()` resets the once-per-process warning flag. + +- [ ] **Step 1: Write the failing tests** + +In `tests/test_config_sink_env.py`: change the module docstring to `"""``EVALSHIFT_SINK`` selects an ObjectStoreSink at config construction; an unusable value is recorded there and raised by ``require_sink_ready()`` once capture is on. ``configure(sink=...)`` still beats the env."""`, change line 18 to `from evalshift.stores.uri import MissingStoreDependencyError`, add `import traceback` and `from evalshift import SinkConfigurationError, capture` and `from evalshift.config import require_sink_ready, is_capture_enabled` to the imports, delete `test_env_sink_bad_grammar_warns_once_and_falls_back`, `test_env_sink_missing_extra_warns_and_falls_back`, `test_env_sink_unexpected_error_warns_once_and_falls_back` and `test_env_sink_warning_does_not_leak_secret_bearing_value`, and add: + +```python +def _recorded() -> SinkConfigurationError: + err = cfg._CONFIG.sink_error + assert isinstance(err, SinkConfigurationError) + return err + + +def test_bad_grammar_is_recorded_at_reset_not_raised( + monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture +) -> None: + monkeypatch.setenv("EVALSHIFT_SINK", "ftp://bucket/prefix") + with caplog.at_level(logging.DEBUG, logger="evalshift"): + reset_config() # must not raise: this runs at `import evalshift` + err = _recorded() + assert str(err) == ( + "EVALSHIFT_SINK is not a valid store URI. Accepted forms: s3:///, " + "gs:///, az:////. " + "Credentials never go in the URI." + ) + assert not caplog.records + assert cfg._CONFIG.sink is None + + +def test_missing_library_is_recorded_naming_the_package(monkeypatch: pytest.MonkeyPatch) -> None: + def _raise(uri: str) -> object: + raise MissingStoreDependencyError("s3", "boto3") + + monkeypatch.setattr(cfg, "open_store", _raise) + monkeypatch.setenv("EVALSHIFT_SINK", "s3://bucket/prefix") + reset_config() + assert str(_recorded()) == ( + "EVALSHIFT_SINK points at an object store (s3://), but boto3 is not installed. " + "Run: pip install boto3, or unset EVALSHIFT_SINK to capture to local disk." + ) + + +def test_missing_azure_library_names_both_packages(monkeypatch: pytest.MonkeyPatch) -> None: + def _raise(uri: str) -> object: + raise MissingStoreDependencyError("az", "azure.identity") + + monkeypatch.setattr(cfg, "open_store", _raise) + monkeypatch.setenv("EVALSHIFT_SINK", "az://acct/c/p") + reset_config() + assert "Run: pip install azure-storage-blob azure-identity," in str(_recorded()) + + +def test_unexpected_open_error_is_recorded_without_its_text(monkeypatch: pytest.MonkeyPatch) -> None: + def _raise(uri: str) -> object: + raise RuntimeError(f"boom {uri}") + + monkeypatch.setattr(cfg, "open_store", _raise) + monkeypatch.setenv("EVALSHIFT_SINK", "s3://bucket/SECRETPREFIX") + reset_config() + err = _recorded() + assert str(err) == ( + "EVALSHIFT_SINK could not be opened (RuntimeError). Unset it to capture to local disk." + ) + assert "SECRETPREFIX" not in "".join(traceback.format_exception(err)) + + +@pytest.mark.parametrize( + ("value", "secret"), + [ + ("az://acct/c?sv=1&sig=SECRETSIG", "SECRETSIG"), + ( + "DefaultEndpointsProtocol=https;AccountName=acct;AccountKey=SECRETKEY==;" + "EndpointSuffix=core.windows.net", + "SECRETKEY", + ), + ("az://acct#sig=SECRETSIG", "SECRETSIG"), + ], +) +def test_recorded_error_never_leaks_a_secret_bearing_value( + monkeypatch: pytest.MonkeyPatch, value: str, secret: str +) -> None: + # The parser's grammar errors may quote the value they reject; the recorded error must + # carry a fixed message and no chained cause or context that reaches a traceback. + monkeypatch.setenv("EVALSHIFT_SINK", value) + reset_config() + err = _recorded() + assert secret not in "".join(traceback.format_exception(err)) + assert err.__cause__ is None and err.__suppress_context__ + + +def test_require_sink_ready_raises_only_with_the_gate_on(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("EVALSHIFT_SINK", "ftp://bucket/prefix") + monkeypatch.delenv("EVALSHIFT_CAPTURE", raising=False) + reset_config() + require_sink_ready() # gate off: nothing + monkeypatch.setenv("EVALSHIFT_CAPTURE", "1") + with pytest.raises(SinkConfigurationError, match="EVALSHIFT_SINK is not a valid store URI"): + require_sink_ready() + with pytest.raises(SinkConfigurationError): + require_sink_ready() # raising twice must work (the stored instance is reused) + + +def test_configure_with_a_sink_clears_the_error_and_without_one_raises( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setenv("EVALSHIFT_SINK", "ftp://bucket/prefix") + monkeypatch.setenv("EVALSHIFT_CAPTURE", "1") + reset_config() + with pytest.raises(SinkConfigurationError): + configure(dedup=False) + assert cfg._CONFIG.dedup is True # the raise comes first; nothing was merged + memory = MemorySink() + configure(sink=memory) + assert cfg._CONFIG.sink_error is None + require_sink_ready() + assert is_capture_enabled() is True + assert _unwrap(active_sink()) is memory + + +def test_is_capture_enabled_is_false_while_the_sink_is_blocked( + monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture +) -> None: + monkeypatch.setenv("EVALSHIFT_SINK", "ftp://bucket/prefix") + monkeypatch.setenv("EVALSHIFT_CAPTURE", "1") + reset_config() + with caplog.at_level(logging.WARNING, logger="evalshift"): + assert is_capture_enabled() is False + assert is_capture_enabled() is False + warnings = [r for r in caplog.records if r.levelno == logging.WARNING] + assert len(warnings) == 1 + assert warnings[0].getMessage().startswith("evalshift: capture disabled: EVALSHIFT_SINK") + + +def test_gate_turned_on_late_drops_captures_instead_of_writing_to_disk( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture +) -> None: + monkeypatch.setenv("EVALSHIFT_DIR", str(tmp_path)) + monkeypatch.setenv("EVALSHIFT_SINK", "ftp://bucket/prefix") + monkeypatch.delenv("EVALSHIFT_CAPTURE", raising=False) + reset_config() + + @capture.agent(suite="s", redact=False, tools=[]) # gate off: decorating must not raise + def run(q: str) -> str: + return q.upper() + + monkeypatch.setenv("EVALSHIFT_CAPTURE", "1") + with caplog.at_level(logging.WARNING, logger="evalshift"): + assert run("hi") == "HI" + assert run("again") == "AGAIN" + assert not list(tmp_path.rglob("*.json")) + assert len([r for r in caplog.records if r.levelno == logging.WARNING]) == 1 +``` + +Add `from pathlib import Path` to the imports. + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `uv run pytest tests/test_config_sink_env.py -q` +Expected: ImportError on `SinkConfigurationError`. + +- [ ] **Step 3: Implement `config.py`** + +Line 36: `from evalshift.stores.uri import STORE_URI_FORMS, MissingStoreDependencyError, open_store`. + +Before `_env_sink` (line 108) add: + +```python +class SinkConfigurationError(RuntimeError): + """``EVALSHIFT_SINK`` names a store the SDK cannot build. + + Recorded when the process-wide config is built (at ``import evalshift``) and raised by + :func:`require_sink_ready` at the first explicit touch -- a ``capture.*`` decorator, a client + wrapper, the LangChain handler, or :func:`configure` without an explicit ``sink`` -- while + ``EVALSHIFT_CAPTURE`` is on. Never raised on bare import, never when capture is off. The + message names the variable, the scheme and the fix; never the value, which may carry a + pasted credential. + """ +``` + +Replace `_env_sink` (lines 108-151) with: + +```python +def _env_sink() -> Sink | None: + """Build the sink ``EVALSHIFT_SINK`` names, or ``None`` for the default ``FileSink``. + + Blank counts as unset. The store's client is built lazily, so this never touches the network + or the credential chain. + + Raises: + SinkConfigurationError: when the value is set but unusable. The text never includes the + raw value, directly or through a chained cause: several of the parser's grammar + errors quote the URI they reject, so every branch raises ``from None`` with a fixed + message naming only the variable, the scheme, the accepted forms or the package to + install. + """ + raw = os.environ.get(SINK_ENV, "").strip() + if not raw: + return None + try: + return ObjectStoreSink(open_store(raw)) + except MissingStoreDependencyError as exc: + raise SinkConfigurationError( + f"{SINK_ENV} points at an object store ({exc.scheme}://), but {exc.package} is not " + f"installed. Run: pip install {exc.packages}, or unset {SINK_ENV} to capture to " + "local disk." + ) from None + except ValueError: + raise SinkConfigurationError( + f"{SINK_ENV} is not a valid store URI. Accepted forms: {STORE_URI_FORMS}. " + "Credentials never go in the URI." + ) from None + except Exception as exc: + # Not a grammar or missing-library error, so its text is unknown and may quote the value. + raise SinkConfigurationError( + f"{SINK_ENV} could not be opened ({type(exc).__name__}). " + "Unset it to capture to local disk." + ) from None +``` + +In `_Config` replace the `sink` field with two plain fields and add `__post_init__`; update the class docstring's first sentence about `sink` to: *``sink`` defaults from ``EVALSHIFT_SINK`` (an object-store URI selects an :class:`~evalshift.sinks.object_store.ObjectStoreSink`; unset or blank leaves it ``None``, i.e. the default ``FileSink``; an unusable value leaves it ``None`` and records ``sink_error`` instead); an explicit ``configure(sink=...)`` still wins and clears ``sink_error``.* + +```python + sink: Sink | None = None + sink_error: SinkConfigurationError | None = None + ... # the hygiene fields stay exactly as they are + + def __post_init__(self) -> None: + if self.sink is not None or self.sink_error is not None: + return + try: + self.sink = _env_sink() + except SinkConfigurationError as exc: + self.sink_error = exc # raised later by require_sink_ready(); import stays safe + except Exception: # pragma: no cover - _env_sink converts everything; belt and braces + safety.logger.debug("evalshift: sink env failed (swallowed)", exc_info=True) +``` + +Replace `is_capture_enabled` (lines 184-186) with: + +```python +#: Whether the once-per-process "capture disabled" warning has been logged. Reset by reset_config(). +_SINK_BLOCK_WARNED = False + + +def _gate_on() -> bool: + """The raw gate: ``EVALSHIFT_CAPTURE`` is set to a truthy value.""" + return os.environ.get(CAPTURE_ENV, "").strip().lower() in _TRUTHY + + +def _sink_blocked() -> bool: + """True while a recorded ``EVALSHIFT_SINK`` error must stop captures; warns once per process.""" + global _SINK_BLOCK_WARNED + if _CONFIG.sink_error is None: + return False + if not _SINK_BLOCK_WARNED: + _SINK_BLOCK_WARNED = True + safety.logger.warning("evalshift: capture disabled: %s", _CONFIG.sink_error) + return True + + +def is_capture_enabled() -> bool: + """True iff ``EVALSHIFT_CAPTURE`` is truthy and no unusable ``EVALSHIFT_SINK`` blocks capture. + + A sink the user asked for is never silently replaced by local disk: with the gate on and a + recorded sink error this is ``False`` (one ``WARNING`` per process), so every entry point + behaves as if capture were off. Normally the error is raised earlier, at startup, by + :func:`require_sink_ready`; this is the backstop for a gate that turned on after that. + """ + return _gate_on() and not _sink_blocked() + + +def require_sink_ready() -> None: + """Raise the recorded ``EVALSHIFT_SINK`` error if capture is on; a no-op otherwise. + + Every explicit touch point calls this -- the ``capture.*`` decorators, the client wrappers, + the LangChain handler and :func:`configure` without a ``sink`` -- so in a real agent the + raise lands at process start, in the deploy logs, before any traffic. + """ + if _CONFIG.sink_error is not None and _gate_on(): + raise _CONFIG.sink_error.with_traceback(None) +``` + +In `configure`, replace the `sink` branch with: + +```python + if sink is not _UNSET: + _CONFIG.sink = sink + _CONFIG.sink_error = None # an explicit sink replaces whatever EVALSHIFT_SINK named + else: + require_sink_ready() +``` + +(keep it as the first statement of the function so a raise merges nothing). In `reset_config`, add `global _CONFIG, _SINK_BLOCK_WARNED` and `_SINK_BLOCK_WARNED = False` before `dedup.reset_registry()`. Add `"SinkConfigurationError"` and `"require_sink_ready"` to `__all__`. In `src/evalshift/__init__.py` change line 19 to `from evalshift.config import SinkConfigurationError, configure, flush_captures` and add `"SinkConfigurationError"` to `__all__` (alphabetical, after `"SCHEMA_VERSION"`). + +- [ ] **Step 4: Run the tests** + +Run: `uv run pytest tests/test_config_sink_env.py tests/test_config.py -q && uv run mypy` +Expected: PASS; mypy clean. + +- [ ] **Step 5: Commit** + +```bash +git add src/evalshift/config.py src/evalshift/__init__.py tests/test_config_sink_env.py +git commit -m "feat(config): an unusable EVALSHIFT_SINK is raised, not quietly replaced by disk + +The error is recorded when the config is built at import and raised by +require_sink_ready() once EVALSHIFT_CAPTURE is on; configure(sink=...) +clears it. If the gate turns on after that, is_capture_enabled() is False +with one warning, so nothing is ever written to disk in its place. + +Co-Authored-By: Claude Fable 5.1 " +``` + +### Task 4: The touch points call `require_sink_ready()` + +**Files:** +- Modify: `src/evalshift/capture/api.py:771` (`agent`, right after `redactor = resolve_redactor(redact)`), `:842` (`agent_session`, same spot), `:908` (`agent_session_async`, same spot), `:989-995` (`tool`, first statement of the method body) +- Modify: `src/evalshift/adapters/openai.py:316-334`, `src/evalshift/adapters/anthropic.py:311-…`, `src/evalshift/adapters/genai.py:410-…` (each `wrap_*` body, before `return cast(C, ClientProxy(...))`) +- Modify: `src/evalshift/adapters/langchain.py:281` (after `super().__init__()`) +- Create: `tests/test_sink_touch_points.py` + +**Interfaces:** +- Consumes: `config.require_sink_ready()` (Task 3). + +- [ ] **Step 1: Write the failing tests** + +Create `tests/test_sink_touch_points.py`: + +```python +"""Every explicit touch point raises the recorded ``EVALSHIFT_SINK`` error while capture is on.""" + +from __future__ import annotations + +import asyncio +import importlib +import logging + +import pytest + +from evalshift import SinkConfigurationError, capture, configure +from evalshift.config import is_capture_enabled, require_sink_ready, reset_config + + +@pytest.fixture +def broken_sink(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("EVALSHIFT_SINK", "ftp://bucket/prefix") + monkeypatch.setenv("EVALSHIFT_CAPTURE", "1") + reset_config() # records, never raises + + +def test_agent_decorator_raises_at_decoration(broken_sink: None) -> None: + with pytest.raises(SinkConfigurationError, match="EVALSHIFT_SINK is not a valid store URI"): + capture.agent(suite="s", redact=False, tools=[]) + + +def test_tool_decorator_raises_at_decoration(broken_sink: None) -> None: + with pytest.raises(SinkConfigurationError): + + @capture.tool + def lookup() -> None: ... + + with pytest.raises(SinkConfigurationError): + capture.tool(name="lookup") + + +def test_agent_session_raises_on_entry(broken_sink: None) -> None: + with pytest.raises(SinkConfigurationError): + with capture.agent_session(suite="s", redact=False, tools=[]): + pass + + +def test_agent_session_async_raises_on_entry(broken_sink: None) -> None: + async def run() -> None: + async with capture.agent_session_async(suite="s", redact=False, tools=[]): + pass + + with pytest.raises(SinkConfigurationError): + asyncio.run(run()) + + +@pytest.mark.parametrize( + ("module", "name"), + [ + ("evalshift.adapters.openai", "wrap_openai"), + ("evalshift.adapters.anthropic", "wrap_anthropic"), + ("evalshift.adapters.genai", "wrap_genai"), + ], +) +def test_client_wrappers_raise_at_wrap_time(broken_sink: None, module: str, name: str) -> None: + wrap = getattr(importlib.import_module(module), name) + with pytest.raises(SinkConfigurationError): + wrap(object()) + + +def test_langchain_handler_raises_at_construction(broken_sink: None) -> None: + pytest.importorskip("langchain_core") + from evalshift.adapters.langchain import EvalShiftCallbackHandler + + with pytest.raises(SinkConfigurationError): + EvalShiftCallbackHandler(suite="s", redact=False, tools=[]) + + +def test_gate_off_never_raises_or_warns( + monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture +) -> None: + monkeypatch.setenv("EVALSHIFT_SINK", "ftp://bucket/prefix") + monkeypatch.delenv("EVALSHIFT_CAPTURE", raising=False) + reset_config() + with caplog.at_level(logging.DEBUG, logger="evalshift"): + require_sink_ready() + capture.agent(suite="s", redact=False, tools=[]) + capture.tool(name="x") + with capture.agent_session(suite="s", redact=False, tools=[]) as tree: + assert tree is None + from evalshift.adapters.openai import wrap_openai + + wrap_openai(object()) + configure(dedup=False) + assert is_capture_enabled() is False + assert not caplog.records +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `uv run pytest tests/test_sink_touch_points.py -q` +Expected: every `raises` test FAILS with "DID NOT RAISE"; `test_gate_off_never_raises_or_warns` passes already. + +- [ ] **Step 3: Implement** + +`src/evalshift/capture/api.py` — in `agent`, after `redactor = resolve_redactor(redact) # eager: ...` add: + +```python + config.require_sink_ready() # eager too: a sink the SDK cannot build fails at startup +``` + +In `agent_session` and `agent_session_async`, after their `redactor = resolve_redactor(redact)` line add the same call (it runs on `with` entry, before the gate check). In `tool`, make it the first statement of the method body: + +```python + def tool(self, fn: Callable[..., Any] | None = None, *, name: str | None = None) -> Any: + """Decorator that records a tool span (no-op when no agent session is active). + + Supports both ``@capture.tool`` and ``@capture.tool(name="...")``. Like :meth:`agent`, an + unusable ``EVALSHIFT_SINK`` raises here, at decoration time, when capture is on. + """ + config.require_sink_ready() +``` + +`src/evalshift/adapters/openai.py`: add `from evalshift import config` to the imports and, in `wrap_openai`, before the `return`: + +```python + config.require_sink_ready() # a sink the SDK cannot build fails here, at startup + return cast(C, ClientProxy(client, _OVERRIDES)) +``` + +Same two-line change in `wrap_anthropic` (`anthropic.py`, which already imports `safety` from `evalshift`: extend to `from evalshift import config, safety`) and `wrap_genai` (`genai.py`). In `langchain.py`'s `EvalShiftCallbackHandler.__init__`, after `super().__init__()` add `config.require_sink_ready()` (the module already imports `config`). Add one sentence to each wrapper's docstring: *Raises :class:`~evalshift.SinkConfigurationError` when capture is on and ``EVALSHIFT_SINK`` cannot be built.* + +- [ ] **Step 4: Run the tests** + +Run: `uv run pytest tests/test_sink_touch_points.py tests/test_capture_agent.py tests/test_capture_tool.py tests/test_capture_gate.py tests/adapters -q && uv run mypy && uv run ruff check` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/evalshift/capture/api.py src/evalshift/adapters/openai.py src/evalshift/adapters/anthropic.py src/evalshift/adapters/genai.py src/evalshift/adapters/langchain.py tests/test_sink_touch_points.py +git commit -m "feat(capture): decorators, wrappers and the LangChain handler raise on an unusable sink + +Co-Authored-By: Claude Fable 5.1 " +``` + +### Task 5: Process-level proof: bare import is safe, decoration is loud + +**Files:** +- Modify: `tests/test_object_store_sink_subprocess.py` (append; reuse `_run_child` and its `child_env` construction) + +- [ ] **Step 1: Write the tests** + +```python +def test_import_with_a_broken_sink_exits_zero_and_logs_nothing() -> None: + # The CLI imports the SDK, and a developer may have both env vars exported in a shell: + # `import evalshift` must stay silent and successful whatever EVALSHIFT_SINK holds. + result = _run_child( + "import evalshift", + env={"EVALSHIFT_SINK": "ftp://bucket/prefix", "EVALSHIFT_CAPTURE": "1"}, + ) + assert "EVALSHIFT_SINK" not in result.stderr + + +def test_decorating_with_a_broken_sink_fails_the_process_naming_the_fix() -> None: + child_env = {k: v for k, v in os.environ.items() if k != "EVALSHIFT_SINK"} + child_env.update({"EVALSHIFT_SINK": "ftp://bucket/prefix", "EVALSHIFT_CAPTURE": "1"}) + result = subprocess.run( + [ + sys.executable, + "-c", + "from evalshift import capture\ncapture.agent(suite='s', redact=False, tools=[])", + ], + env=child_env, + capture_output=True, + text=True, + timeout=60, + check=False, + ) + assert result.returncode != 0 + assert "SinkConfigurationError" in result.stderr + assert "Accepted forms: s3:///" in result.stderr +``` + +- [ ] **Step 2: Run them** + +Run: `uv run pytest tests/test_object_store_sink_subprocess.py -q` +Expected: PASS (Tasks 3–4 are in place). If the first test fails because `_run_child` prepends a `pythonpath`, pass `pythonpath=Path("src")` the way the file's existing tests do. + +- [ ] **Step 3: Commit** + +```bash +git add tests/test_object_store_sink_subprocess.py +git commit -m "test(config): bare import survives a broken EVALSHIFT_SINK; decoration does not + +Co-Authored-By: Claude Fable 5.1 " +``` + +### Task 6: SDK docs, decision record and changelog + +**Files:** +- Modify: `README.md:50-56,112-121`, `DOCS.md:52-60,210-216,566,649-663,1086-1100` plus every `MissingExtraError` hit, `llms.txt:8`, `llms-full.txt:7-10,55,259-263`, `docs/DECISIONS.md:526-533,545-549` (+ new sub-decision), `CHANGELOG.md:8` + +Enumerate first: `git grep -n -e '\[s3\]' -e '\[gcs\]' -e '\[azure\]' -e 'falls back to local disk' -e 'local disk instead' -e 'keeps writing to local disk' -e MissingExtraError -e 'pip extra' -- README.md DOCS.md llms.txt llms-full.txt docs/` and fix every hit; the list below is what that grep returns today. + +- [ ] **Step 1: README.md** + +Replace lines 50-56 with: + +```markdown +Optional object-store sinks (`EVALSHIFT_SINK=s3://…` / `gs://…` / `az://…`, see below) use the +provider's own client library — install it like any other package: + +```bash +pip install boto3 # s3:// — Amazon S3 and S3-compatible stores +pip install google-cloud-storage # gs:// +pip install azure-storage-blob azure-identity # az:// +``` +``` + +In the "Hosts whose disk does not outlive them" section replace `are accepted; install the matching extra above.` with `are accepted; install the provider's client library (above).` and replace the sentence `Uploads run on a bounded background thread and never raise into the agent; an invalid `EVALSHIFT_SINK` or a missing extra logs one warning and falls back to local disk.` with: + +```markdown +Uploads run on a bounded background thread and never raise into the agent. Configuration is the +one thing that does raise: if `EVALSHIFT_SINK` is set but the library is missing or the value is +malformed, the first `capture.*` decorator, client wrapper or `configure()` call raises +`SinkConfigurationError` — at startup, naming the `pip install` to run — instead of quietly +writing to local disk. With `EVALSHIFT_CAPTURE` off nothing happens at all. +``` + +- [ ] **Step 2: DOCS.md** + +Lines 55-60 (install block): same replacement as the README's. Line 52 keeps "Every adapter module is import-guarded". + +Lines 210-216: change the lead-in to `Three deliberate exceptions to fail-open:` and add a row: + +```markdown +| Sink configuration (`EVALSHIFT_SINK`) | **Raises** `SinkConfigurationError` at the first explicit touch while capture is on — never on bare import, never with capture off | A bucket you asked for and did not get is the loss this feature exists to prevent; a crash in the deploy log beats captures quietly written to a disk that dies with the task | +``` + +Line 566 (env table row): replace the description cell with: + +```markdown +Ship captures to an object store you own: `s3:///`, `gs:///` or `az:////`. Needs the provider's client library (`pip install boto3` / `google-cloud-storage` / `azure-storage-blob azure-identity`). An unusable value — library missing, malformed URI — is recorded here and raised as `SinkConfigurationError` by the first `capture.*` decorator, client wrapper, LangChain handler or `configure()` call while `EVALSHIFT_CAPTURE` is on; the message names the variable, the scheme and the fix, never the value. With capture off nothing happens. +``` + +Lines 659-661 (URI table): rename the `Extra` column to `Install` and put `pip install boto3`, `pip install google-cloud-storage`, `pip install azure-storage-blob azure-identity` in it. Line 663: replace from `If `EVALSHIFT_SINK` is invalid` to the end of the paragraph with: + +```markdown +If `EVALSHIFT_SINK` is invalid, its library is missing, or the store cannot be opened for any other reason, the SDK records the problem at import and raises it as `SinkConfigurationError` the first time your code asks it to capture — applying a `capture.*` decorator, wrapping a client, building the LangChain handler or calling `configure()` without a `sink` — while `EVALSHIFT_CAPTURE` is on. In a real agent that is process start. Bare `import evalshift` never raises, and with capture off nothing happens. Should the gate turn on only after those ran, captures are dropped with one `WARNING` rather than written to local disk: a store you asked for is never silently replaced. The message names the variable, the scheme and the package to install (`Run: pip install boto3`), never the value. +``` + +Lines 1086-1100 (API reference): after the `ObjectStoreSink` block add: + +```markdown +`evalshift.SinkConfigurationError` (`RuntimeError`) — raised by `evalshift.config.require_sink_ready()`, which every touch point calls; `evalshift.stores.uri.MissingStoreDependencyError` (`ImportError`; `.scheme`, `.module`, `.package`, `.packages`, `.extra`) — raised by `open_store()`, `require_store_modules(scheme)` and a store constructed without a `client` when its library is absent. +``` + +Replace every remaining `MissingExtraError` with `MissingStoreDependencyError` and every `pip install "evalshift-sdk[]"`-style phrase with the package form. + +- [ ] **Step 3: llms.txt and llms-full.txt** + +`llms.txt:8`: add `SinkConfigurationError` to the top-level public API list. `llms-full.txt:7-10`: replace the three store extras lines with `Object stores: pip install boto3 (s3://) | google-cloud-storage (gs://) | azure-storage-blob azure-identity (az://) -- plain packages, no extra needed`. Line 55: rewrite the `EVALSHIFT_SINK` row to `| EVALSHIFT_SINK | unset (disk) | Ship captures to s3:///, gs:/// or az:////; needs the provider's client library (pip install boto3 / google-cloud-storage / azure-storage-blob azure-identity). Unusable value (library missing, bad URI) -> recorded at import, raised as SinkConfigurationError by the first capture.* decorator / client wrapper / LangChain handler / configure() while EVALSHIFT_CAPTURE is on; never on bare import, never with capture off; if the gate turns on later, captures are dropped with ONE warning (never written to disk instead). Message names the var, scheme and package, never the value. | Config construction |`. Lines 259-263: `MissingExtraError (ImportError, .extra) naming pip install "evalshift-sdk[]"` → `MissingStoreDependencyError (ImportError; .scheme/.module/.package/.packages/.extra) naming "pip install "`. Add after it: `evalshift.config.require_sink_ready() -> None: raises the recorded SinkConfigurationError iff EVALSHIFT_CAPTURE is on. evalshift.SinkConfigurationError(RuntimeError).` + +- [ ] **Step 4: docs/DECISIONS.md** + +In D-stores, replace the sentence ending `(or, for a missing client library, the pip extra).` with `accepted forms and the package to install -- never the value.`, replace `Extras `[s3]`, `[gcs]`, `[azure]`;` with `Client libraries are plain packages (`pip install boto3` / `google-cloud-storage` / `azure-storage-blob azure-identity`); the extras of the same names stay in pyproject as tested floors but are never advertised;`, and replace the bullet `**Warning-level logging, deliberately**: ... and an invalid or unusable `EVALSHIFT_SINK`, log at `WARNING`` so it no longer lists the invalid sink. Then append a sub-decision: + +```markdown +### D-stores-b — an unusable `EVALSHIFT_SINK` raises at startup; it is never replaced by disk +A warning on a green deploy is read by nobody, and a sink the user asked for and did not get is +the loss D-stores exists to prevent. So configuration is a third carve-out from fail-open (with +redaction and the read side): `_Config` records a `SinkConfigurationError` at import and +`config.require_sink_ready()` raises it at the first explicit touch -- `capture.agent` / +`agent_session` / `tool`, `wrap_openai` / `wrap_anthropic` / `wrap_genai`, +`EvalShiftCallbackHandler()`, `configure()` without a `sink` -- iff `EVALSHIFT_CAPTURE` is on. +Bare `import evalshift` never raises (the CLI imports the SDK; a shell may export the vars), and +with capture off nothing happens. Backstop: `is_capture_enabled()` is `False` (one `WARNING`) +while an error is recorded, so a gate that turns on late drops captures rather than writing them +to local disk. Messages name packages, not extras (`pip install boto3`), because a user who +installed the SDK normally does not know what an extra is, and `"evalshift-sdk[s3]"` needs quotes +in zsh. Runtime faults (puts, queue, exit flush) stay fail-open. `MissingExtraError` became +`MissingStoreDependencyError`, no alias (0.5.0 was one day old). +``` + +- [ ] **Step 5: CHANGELOG.md** + +Under `## [Unreleased]`: + +```markdown +### Changed + +- **An unusable `EVALSHIFT_SINK` now raises instead of falling back to local disk.** When + `EVALSHIFT_CAPTURE` is on and the value names a store the SDK cannot build — its client + library is not installed, or the URI is malformed — the first `capture.*` decorator, client + wrapper (`wrap_openai` / `wrap_anthropic` / `wrap_genai`), `EvalShiftCallbackHandler()` or + `configure()` without a `sink` raises `SinkConfigurationError` (new, exported from + `evalshift`), naming the variable, the scheme and the fix. In a real agent that is process + start. Bare `import evalshift` never raises, and with capture off nothing happens. If the gate + turns on only after that, captures are dropped with one `WARNING` rather than written to disk. + Previously the SDK logged one warning and wrote to local disk, which on the ephemeral hosts + this feature exists for lost every capture behind a green deploy. +- Every message names the package to install, not a pip extra: `pip install boto3`, + `google-cloud-storage`, `azure-storage-blob azure-identity` (both Azure packages in one + command). The `[s3]` / `[gcs]` / `[azure]` extras still exist as tested version floors but + are no longer documented. `evalshift.stores.uri.MissingExtraError` is renamed + `MissingStoreDependencyError` (still an `ImportError`; new `.package` / `.packages`), and a + store constructed without a `client` (`S3Store(...)` etc.) raises it at construction instead + of failing its first background upload. +``` + +- [ ] **Step 6: Verify and commit** + +Run: `git grep -n -e '\[s3\]' -e '\[gcs\]' -e '\[azure\]' -e MissingExtraError -e 'falls back to local disk' -e 'local disk instead' -- README.md DOCS.md llms.txt llms-full.txt docs/` — expected: only `pyproject.toml`-free, historical `CHANGELOG.md` 0.5.0 lines remain (none of these files should match). Then `uv run ruff check && uv run ruff format --check && uv run mypy && uv run pytest -q` — all green. + +```bash +git add README.md DOCS.md llms.txt llms-full.txt docs/DECISIONS.md CHANGELOG.md +git commit -m "docs(sdk): the sink raises on misconfiguration; install the package, not the extra + +Co-Authored-By: Claude Fable 5.1 " +``` + +Push and open the PR against `main` (`gh pr create -R evalshift/evalshift-sdk`), body ending with `🤖 Generated with [Claude Code](https://claude.com/claude-code)`. + +--- + +# Part B — evalshift-cli (1.3.1) + +Work in `/home/lukas/repos/evalshift/evalshift-cli` on branch `fix/store-dependency-messages` from `main`. Export `PATH=$PWD/.venv/bin:$PATH` first (the hooks need it). Paths relative to that repo. + +### Task 7: Package tables in `store_uri.py`; `open_store` names the package + +**Files:** +- Modify: `src/evalshift_cli/captures/store_uri.py:30,50-53` +- Modify: `src/evalshift_cli/captures/remote.py:155-173` +- Modify: `tests/unit/test_captures_stores.py:207-220`, `tests/unit/test_cli_capture_remote.py:322-346`, `tests/unit/test_captures_store_uri.py` (append) + +**Interfaces:** +- Produces: `_PACKAGE_FOR_MODULE`, `_PACKAGES_FOR_SCHEME` (verbatim copies of the SDK's), `StoreURI.packages -> str`, `package_for_module(module: str) -> str`. `RemoteStoreUnavailable.summary == ":// captures store needs , which is not installed"`, `.hint == "install it: pip install "`. + +- [ ] **Step 1: Write the failing tests** + +Replace `test_open_store_missing_extra_names_pip_extra` and `test_open_store_azure_needs_identity_too` in `tests/unit/test_captures_stores.py` with: + +```python +def test_open_store_missing_library_names_the_package(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(remote_module, "_installed", lambda module: False) + with pytest.raises(RemoteStoreUnavailable) as info: + open_store(parse_store_uri("gs://b/p")) + assert info.value.summary == ( + "gs:// captures store needs google-cloud-storage, which is not installed" + ) + assert info.value.hint == "install it: pip install google-cloud-storage" + + +def test_open_store_azure_names_both_packages_when_identity_is_missing( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(remote_module, "_installed", lambda module: module != "azure.identity") + with pytest.raises(RemoteStoreUnavailable) as info: + open_store(parse_store_uri("az://a/c/p")) + assert info.value.summary == "az:// captures store needs azure-identity, which is not installed" + assert info.value.hint == "install it: pip install azure-storage-blob azure-identity" +``` + +Append to `tests/unit/test_captures_store_uri.py`: + +```python +def test_packages_names_every_library_the_scheme_needs() -> None: + assert parse_store_uri("s3://b/p").packages == "boto3" + assert parse_store_uri("gs://b/p").packages == "google-cloud-storage" + assert parse_store_uri("az://a/c/p").packages == "azure-storage-blob azure-identity" +``` + +In `tests/unit/test_cli_capture_remote.py` change the `missing_extra` fixture's raise to + +```python + raise RemoteStoreUnavailable( + "s3:// captures store needs boto3, which is not installed", + hint="install it: pip install boto3", + ) +``` + +rename it `missing_library` (and the fixture parameter in the test below it), rename the test to `test_missing_library_exits_1_naming_the_package`, and change its assertion to `assert "pip install boto3" in result.stdout`. + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `pytest tests/unit/test_captures_stores.py tests/unit/test_captures_store_uri.py tests/unit/test_cli_capture_remote.py -q --no-cov` +Expected: FAIL on the new strings and on `packages`. + +- [ ] **Step 3: Implement** + +`src/evalshift_cli/captures/store_uri.py` after line 30: + +```python +#: The pip distribution that provides each client module. Shared verbatim with the SDK. +_PACKAGE_FOR_MODULE: dict[str, str] = { + "boto3": "boto3", + "google.cloud.storage": "google-cloud-storage", + "azure.storage.blob": "azure-storage-blob", + "azure.identity": "azure-identity", +} + +#: The ``pip install`` argument that installs everything a scheme needs, in one command. +#: Named in every message instead of the extras: a user who installed the CLI the normal way +#: has no reason to know what an extra is. Shared verbatim with the SDK. +_PACKAGES_FOR_SCHEME: dict[str, str] = { + "s3": "boto3", + "gs": "google-cloud-storage", + "az": "azure-storage-blob azure-identity", +} + + +def package_for_module(module: str) -> str: + """The pip distribution that provides ``module`` (one of the client modules above).""" + return _PACKAGE_FOR_MODULE[module] +``` + +and in `StoreURI`: + +```python + @property + def packages(self) -> str: + """The ``pip install`` argument that installs this scheme's client library (or libraries).""" + return _PACKAGES_FOR_SCHEME[self.scheme] +``` + +`src/evalshift_cli/captures/remote.py`: import `package_for_module` from `evalshift_cli.captures.store_uri` (next to the existing `StoreURI` import) and replace the raise in `open_store` with: + +```python + for module in _REQUIRED_MODULES[parsed.scheme]: + if not _installed(module): + raise RemoteStoreUnavailable( + f"{parsed.scheme}:// captures store needs {package_for_module(module)}, " + "which is not installed", + hint=f"install it: pip install {parsed.packages}", + ) +``` + +Update the docstring's `Raises:` line to `RemoteStoreUnavailable: when a client module the scheme needs is not installed. The summary names the package that provides the first missing module; the hint names the pip command that installs everything the scheme needs.` + +- [ ] **Step 4: Run the tests** + +Run: `pytest tests/unit/test_captures_stores.py tests/unit/test_captures_store_uri.py tests/unit/test_cli_capture_remote.py -q --no-cov && mypy --strict src/evalshift_cli && ruff check .` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/evalshift_cli/captures/store_uri.py src/evalshift_cli/captures/remote.py tests/unit/test_captures_stores.py tests/unit/test_captures_store_uri.py tests/unit/test_cli_capture_remote.py +git commit -m "fix(captures): a missing store library is reported as the package to install + +Co-Authored-By: Claude Fable 5.1 " +``` + +### Task 8: `doctor`'s `captures.store` row warns instead of failing + +**Files:** +- Modify: `src/evalshift_cli/cli/commands/doctor.py:8-16,252-275,528-534` +- Modify: `src/evalshift_cli/cli/commands/capture.py:244-246` (docstring only) +- Modify: `tests/unit/test_doctor.py:810-821` (+ any exit-code test: `grep -n 'captures.store\|missing_extra' tests/unit/test_doctor.py`) + +- [ ] **Step 1: Write the failing test** + +Replace `test_missing_extra_fails` in `tests/unit/test_doctor.py` with: + +```python + def test_missing_library_warns_without_failing( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + # A run on a committed suite never touches the bucket (the GitHub Action installs the + # bare package), so a missing client library must not fail doctor -- or compare, which + # exits on any doctor failure. The capture commands that do need it exit 1 themselves. + self._config(tmp_path, "gs://b/p") + + def _raise(parsed: object) -> object: + raise RemoteStoreUnavailable( + "gs:// captures store needs google-cloud-storage, which is not installed", + hint="install it: pip install google-cloud-storage", + ) + + monkeypatch.setattr(doctor_module, "open_store", _raise) + results = run_checks(cwd=tmp_path, env=_empty_env()) + row = _by_name(results, "captures.store") + assert row.status == "warn" + assert "pip install google-cloud-storage" in row.detail + assert not any(r.status == "fail" for r in results) +``` + +If the grep finds a command-level test asserting `exit_code == 1` for this case, change it to `== 0`. + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `pytest tests/unit/test_doctor.py -q --no-cov -k missing_library` +Expected: FAIL (`'fail' == 'warn'`). + +- [ ] **Step 3: Implement** + +`doctor.py` lines 252-275: change the status to `"warn"` and the docstring to: + +```python + """One row for ``captures.store`` when the config names one: library installed, bucket listable. + + No config, an invalid config (the ``evalshift.yaml`` row already reports that) or no + ``captures.store`` produce no row. A missing client library is a ``warn`` naming the package + to install: a run on a committed suite never touches the bucket (CI installs the bare + package, and ``compare`` exits on any doctor failure), and the capture commands that do + need it exit 1 themselves. A listing that raises is a ``warn`` too: doctor runs on laptops + without cloud credentials, and that must not fail the command. + """ +``` + +Lines 8-16 (module docstring): drop the `, or its ``captures.store`` needs a client extra that isn't installed` clause so exit 1 is `an ``evalshift.yaml`` exists in the cwd but doesn't validate`. Line 532: `an install hint such as ``pip install google-cloud-storage`` or a user-written store URI`. `capture.py:245`: `A missing client library is always a hard error here: the user asked for a store this command cannot read.` + +- [ ] **Step 4: Run the tests** + +Run: `pytest tests/unit/test_doctor.py tests/unit/test_compare_command.py -q --no-cov && mypy --strict src/evalshift_cli && ruff check .` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/evalshift_cli/cli/commands/doctor.py src/evalshift_cli/cli/commands/capture.py tests/unit/test_doctor.py +git commit -m "fix(doctor): a missing store library is a warning, not a failure + +It failed compare -- and so every GitHub Action run -- for a config that +names captures.store, although a run on a committed suite never reads the +bucket. The capture commands that need the library still exit 1. + +Co-Authored-By: Claude Fable 5.1 " +``` + +### Task 9: CLI docs, docs-currency tripwire and changelog + +**Files:** +- Modify: `DOCS.md:220,872`, `llms-full.txt:155-158,176-180,322-326,740-745,1308-1311`, `docs/configuration.md:1044-1046`, `docs/sdk.md:124-125`, `CHANGELOG.md:8`, `tests/unit/test_docs_currency.py` (append) + +- [ ] **Step 1: Write the failing tripwire** + +Append to `tests/unit/test_docs_currency.py`: + +```python +#: Install-story phrases retired on 2026-10-09: every message and doc names the package to +#: install (`pip install boto3`), never a pip extra, and `doctor` no longer fails over it. +RETIRED_INSTALL_TERMS: tuple[str, ...] = ( + "evalshift[s3]", + "evalshift[gcs]", + "evalshift[azure]", + "client extra", + "optional dependency", + "whose extra is missing", +) + +_INSTALL_COPY_FILES: tuple[str, ...] = (*PROSE_FILES, "docs/sdk.md") + + +@pytest.mark.parametrize("term", RETIRED_INSTALL_TERMS) +def test_copy_names_packages_not_extras(term: str) -> None: + offenders = [name for name in _INSTALL_COPY_FILES if term in _whitespace_normalized(name)] + assert offenders == [], f"{term!r} still appears in {offenders}" +``` + +Run: `pytest tests/unit/test_docs_currency.py -q --no-cov -k packages_not_extras` — expected: FAIL naming `DOCS.md`, `llms-full.txt`, `docs/configuration.md`, `docs/sdk.md`. + +- [ ] **Step 2: Rewrite the sites** + +`DOCS.md:220`: delete ` — or when its `captures.store` needs a client extra that is not installed` and change ``fail` (exit 1) naming the `pip install "evalshift[]"` when the client extra is missing` to ``warn` naming the `pip install ` to run when the client library is missing (the capture commands that need it exit 1 themselves)`. + +`DOCS.md:872`: `Exit 1 only on an invalid existing config.` and `(`warn` naming the package to install when the client library is missing, `warn` when the bucket cannot be listed)`. + +`llms-full.txt:155-158`: `Env/config check. Exit 1 ONLY when an existing evalshift.yaml fails validation (the row shows the problem count + "run `evalshift validate` for details"; validate prints each problem); missing API keys and a captures.store whose client library is missing are soft warnings (exit 0).` + +`llms-full.txt:176-180`: `captures.store: warn if the client library is missing (detail names "pip install "; exit 0 -- compare and CI are not failed over a bucket a committed-suite run never reads), warn if the bucket cannot be listed (exit 0), ok " reachable"; no row without a store (or when evalshift.yaml is invalid -- its own row reports that).` + +`llms-full.txt:322-326`: `Missing library -> exit 1 naming pip install boto3 | google-cloud-storage | azure-storage-blob azure-identity (az:// needs both; one command installs them).` + +`llms-full.txt:740-745`: replace `Extras: evalshift[s3] (boto3; AWS_ENDPOINT_URL covers MinIO/R2/B2), [gcs] (google-cloud-storage), [azure] (azure-storage-blob + azure-identity).` with `Client libraries are plain packages: pip install boto3 (s3://; AWS_ENDPOINT_URL covers MinIO/R2/B2), google-cloud-storage (gs://), azure-storage-blob azure-identity (az://).` + +`llms-full.txt:1308-1311`: `doctor exits 1 only on invalid existing config;`. + +`docs/configuration.md:1044-1046` Install column: `pip install boto3`, `pip install google-cloud-storage`, `pip install azure-storage-blob azure-identity`. + +`docs/sdk.md:124-125`: `and install the provider's client library (`pip install boto3`, `google-cloud-storage` or `azure-storage-blob azure-identity`); `list` and `sync` then fetch new captures first.` + +Then `git grep -n -e '\[s3\]' -e '\[gcs\]' -e '\[azure\]' -e 'client extra' -e 'optional dependency' -- README.md DOCS.md llms-full.txt docs/ src/` and fix anything the list above missed (the `CHANGELOG.md` 1.3.0 entry stays as history). + +- [ ] **Step 3: CHANGELOG.md** + +Under `## [Unreleased]`: + +```markdown +### Fixed + +- `doctor`'s `captures.store` row no longer fails the command — and with it `compare` and + every GitHub Action run — when the store's client library is not installed; it warns, naming + the package to install. A run on a committed suite never reads the bucket; `capture fetch`, + `list` and `sync`, which do, still exit 1. + +### Changed + +- Every store message names the package to install instead of a pip extra: `pip install boto3`, + `google-cloud-storage`, `azure-storage-blob azure-identity` (both Azure packages in one + command). The `[s3]` / `[gcs]` / `[azure]` extras still exist but are no longer documented. +``` + +- [ ] **Step 4: Verify and commit** + +Run: `pytest tests/unit/test_docs_currency.py -q --no-cov && pre-commit run --all-files` — all green. + +```bash +git add DOCS.md llms-full.txt docs/configuration.md docs/sdk.md CHANGELOG.md tests/unit/test_docs_currency.py +git commit -m "docs: install the store library as a plain package; doctor warns, never fails, over it + +Co-Authored-By: Claude Fable 5.1 " +``` + +Push and open the PR against `main` (`gh pr create -R evalshift/evalshift-cli`), body ending with `🤖 Generated with [Claude Code](https://claude.com/claude-code)`. + +--- + +# Part C — release and the website + +### Task 10: Release SDK 0.6.0 and CLI 1.3.1 (after both PRs merge) + +Follow the existing `chore(release): X` pattern, one PR per repo: + +- **SDK 0.6.0**: `pyproject.toml:7`, `src/evalshift/__init__.py:32`, `llms-full.txt:4` (`version: 0.6.0`) and `:298` (`__version__ = "0.6.0"`), `DOCS.md:5` and `:1142`, `llms.txt:7`; move the `Unreleased` block under `## [0.6.0] - `. `uv run pytest` green. Merge, tag `v0.6.0`, confirm PyPI shows 0.6.0. +- **CLI 1.3.1**: `pyproject.toml:7`, `DOCS.md:15`, `llms-full.txt:4`; changelog header `## [1.3.1] - `; `pip install -e ".[dev]"` into `.venv` so `test_reference_header_version_matches_the_package` sees 1.3.1; `make ci` green. Merge, tag `v1.3.1`. The action's `bump-cli-pin.yml` opens the pin PR on its own; merge it. + +### Task 11: Website follow-up (after Task 10) + +In `/home/lukas/repos/evalshift/evalshift-client` on a branch from `main`: + +- `src/pages/docs/pages/SdkConfig.tsx`: `SINK_ENV_CODE`'s first line → `pip install boto3 # s3:// — or google-cloud-storage (gs://), azure-storage-blob azure-identity (az://)`; the `EVALSHIFT_SINK` env bullet's `Needs the matching extra; an invalid value or a missing extra logs one warning and falls back to local disk.` → `Needs the provider's client library. An unusable value raises SinkConfigurationError the first time your code asks the SDK to capture — at startup — while capture is on; never on bare import.`; the `ObjectStoreSink` table's `Extra` column → `Install` with the three `pip install` strings; the paragraph starting `If EVALSHIFT_SINK is invalid, its extra is missing` → the DOCS.md wording from Task 6 Step 2 (the paragraph beginning "If `EVALSHIFT_SINK` is invalid, its library is missing"). +- `src/pages/docs/pages/Captures.tsx`: the `Extras.` bullet → `Client library. Install the provider's own package — pip install boto3, google-cloud-storage, or azure-storage-blob azure-identity; a missing one exits 1 naming it.` +- `src/pages/docs/pages/Configuration.tsx`: URI table `Install` column → the three `pip install` strings. +- `src/pages/docs/pages/CliCommands.tsx` `doctor` row and `src/pages/docs/pages/Cli.tsx` doctor bullet: `fail`/`exit 1` for the missing library → `warn`; exit 1 only on an invalid config. +- `src/pages/docs/pages/Changelog.tsx`: add `1.3.1` and `0.6.0` lines; `src/lib/version.ts`: `CLI_VERSION = "1.3.1"`, `SDK_VERSION = "0.6.0"`; `HostedSetup.tsx` pin `1.3.1`. +- `npm run sync:llms`; `npm run lint && npm run typecheck && npx vitest run && npm run build` (the `scripts/llmsFullSync.test.ts` tripwire must pass). Commit, push, PR. diff --git a/docs/superpowers/specs/2026-10-09-sink-dependency-loud-failure-design.md b/docs/superpowers/specs/2026-10-09-sink-dependency-loud-failure-design.md index 75fae4da..bdcdb889 100644 --- a/docs/superpowers/specs/2026-10-09-sink-dependency-loud-failure-design.md +++ b/docs/superpowers/specs/2026-10-09-sink-dependency-loud-failure-design.md @@ -79,11 +79,14 @@ Rule: a sink the user asked for is never silently replaced by local disk. ### The explicit-sink path -`S3Store(...)`, `GCSStore(...)`, `AzureBlobStore(...)` import their client lazily. On -`ImportError` they raise `MissingStoreDependencyError` (renamed from `MissingExtraError`, same -base `ImportError`, same module `evalshift.stores.uri`) whose text names the pip package(s), not -the extra. `open_store()` raises the same. No alias for the old name: 0.5.0 shipped today and the -rename is listed under *Changed* in the changelog. +`S3Store(...)`, `GCSStore(...)`, `AzureBlobStore(...)` still import their client lazily on the +first put, but a store constructed **without** an injected `client` now checks that its library +is importable (`importlib.util.find_spec`, no import, no network) and raises +`MissingStoreDependencyError` at construction — in user code, at startup — rather than failing +its first background upload. The error is renamed from `MissingExtraError` (same base +`ImportError`, same module `evalshift.stores.uri`) and its text names the pip package(s), not the +extra. `open_store()` and the new `require_store_modules(scheme)` raise the same. No alias for +the old name: 0.5.0 shipped today and the rename is listed under *Changed* in the changelog. ### Messages @@ -93,7 +96,7 @@ parser's quoted-URI message cannot appear as a chained cause. | Case | `SinkConfigurationError` text | |---|---| -| library missing | `EVALSHIFT_SINK names an s3:// store, but boto3 is not installed. Run: pip install boto3 — or unset EVALSHIFT_SINK to capture to local disk.` | +| library missing | `EVALSHIFT_SINK points at an object store (s3://), but boto3 is not installed. Run: pip install boto3, or unset EVALSHIFT_SINK to capture to local disk.` | | malformed URI | `EVALSHIFT_SINK is not a valid store URI. Accepted forms: s3:///, gs:///, az:////. Credentials never go in the URI.` | | store failed to open | `EVALSHIFT_SINK could not be opened (). Unset it to capture to local disk.` | From bdc4007c98f373ceb2bd71f4c1ebfedb911461ce Mon Sep 17 00:00:00 2001 From: Lukas Babaliauskas Date: Fri, 9 Oct 2026 23:18:06 +0200 Subject: [PATCH 4/8] fix(captures): a missing store library is reported as the package to install Co-Authored-By: Claude Opus 5.5 --- src/evalshift_cli/captures/remote.py | 10 ++++----- src/evalshift_cli/captures/store_uri.py | 29 ++++++++++++++++++++++++- tests/unit/test_captures_store_uri.py | 6 +++++ tests/unit/test_captures_stores.py | 15 ++++++++----- tests/unit/test_cli_capture_remote.py | 12 +++++----- 5 files changed, 55 insertions(+), 17 deletions(-) diff --git a/src/evalshift_cli/captures/remote.py b/src/evalshift_cli/captures/remote.py index 4f3b7ba9..6d225df0 100644 --- a/src/evalshift_cli/captures/remote.py +++ b/src/evalshift_cli/captures/remote.py @@ -39,7 +39,7 @@ from rich.text import Text from evalshift_cli.captures.reader import captures_root, toolsets_root -from evalshift_cli.captures.store_uri import StoreURI +from evalshift_cli.captures.store_uri import StoreURI, package_for_module #: Largest object a fetch will download. Captures are kilobytes; this is a safety stop. MAX_OBJECT_BYTES = 32 * 1024 * 1024 @@ -162,14 +162,14 @@ def open_store(parsed: StoreURI) -> RemoteStore: The scheme's adapter; its client is built on the first ``list`` or ``get``. Raises: - RemoteStoreUnavailable: when a module the scheme's extra provides is not installed. - The summary names the first missing module; the hint names the pip extra. + RemoteStoreUnavailable: when a client module the scheme needs is not installed. The summary names the package that provides the first missing module; the hint names the pip command that installs everything the scheme needs. """ for module in _REQUIRED_MODULES[parsed.scheme]: if not _installed(module): raise RemoteStoreUnavailable( - f"{parsed.scheme}:// captures store needs the optional dependency {module!r}", - hint=f'install it with: pip install "evalshift[{parsed.extra}]"', + f"{parsed.scheme}:// captures store needs {package_for_module(module)}, " + "which is not installed", + hint=f"install it: pip install {parsed.packages}", ) if parsed.scheme == "s3": from evalshift_cli.captures.stores.s3 import S3Store diff --git a/src/evalshift_cli/captures/store_uri.py b/src/evalshift_cli/captures/store_uri.py index 3de1b480..fd1d03ae 100644 --- a/src/evalshift_cli/captures/store_uri.py +++ b/src/evalshift_cli/captures/store_uri.py @@ -29,6 +29,28 @@ _EXTRA_FOR_SCHEME: dict[str, str] = {"s3": "s3", "gs": "gcs", "az": "azure"} +#: The pip distribution that provides each client module. Shared verbatim with the SDK. +_PACKAGE_FOR_MODULE: dict[str, str] = { + "boto3": "boto3", + "google.cloud.storage": "google-cloud-storage", + "azure.storage.blob": "azure-storage-blob", + "azure.identity": "azure-identity", +} + +#: The ``pip install`` argument that installs everything a scheme needs, in one command. +#: Named in every message instead of the extras: a user who installed the CLI the normal way +#: has no reason to know what an extra is. Shared verbatim with the SDK. +_PACKAGES_FOR_SCHEME: dict[str, str] = { + "s3": "boto3", + "gs": "google-cloud-storage", + "az": "azure-storage-blob azure-identity", +} + + +def package_for_module(module: str) -> str: + """The pip distribution that provides ``module`` (one of the client modules above).""" + return _PACKAGE_FOR_MODULE[module] + @dataclass(frozen=True, slots=True) class StoreURI: @@ -51,6 +73,11 @@ def extra(self) -> str: """The pip extra (``evalshift[]``) that installs this scheme's client.""" return _EXTRA_FOR_SCHEME[self.scheme] + @property + def packages(self) -> str: + """The ``pip install`` argument that installs this scheme's client library (or libraries).""" + return _PACKAGES_FOR_SCHEME[self.scheme] + def parse_store_uri(uri: str) -> StoreURI: """Parse ``uri`` against the grammar above. @@ -94,4 +121,4 @@ def parse_store_uri(uri: str) -> StoreURI: return StoreURI(scheme, parts.netloc, None, path) -__all__ = ["STORE_URI_FORMS", "Scheme", "StoreURI", "parse_store_uri"] +__all__ = ["STORE_URI_FORMS", "Scheme", "StoreURI", "package_for_module", "parse_store_uri"] diff --git a/tests/unit/test_captures_store_uri.py b/tests/unit/test_captures_store_uri.py index 05a1e47f..1e78d04f 100644 --- a/tests/unit/test_captures_store_uri.py +++ b/tests/unit/test_captures_store_uri.py @@ -104,3 +104,9 @@ def test_grammar_rejection_messages() -> None: assert str(no_container.value) == ( "Azure store URI names no container; expected az:////" ) + + +def test_packages_names_every_library_the_scheme_needs() -> None: + assert parse_store_uri("s3://b/p").packages == "boto3" + assert parse_store_uri("gs://b/p").packages == "google-cloud-storage" + assert parse_store_uri("az://a/c/p").packages == "azure-storage-blob azure-identity" diff --git a/tests/unit/test_captures_stores.py b/tests/unit/test_captures_stores.py index 2d34e68b..a32b5587 100644 --- a/tests/unit/test_captures_stores.py +++ b/tests/unit/test_captures_stores.py @@ -204,16 +204,21 @@ def test_open_store_dispatches_without_building_a_client() -> None: assert isinstance(open_store(parse_store_uri("az://a/c/p")), AzureBlobStore) -def test_open_store_missing_extra_names_pip_extra(monkeypatch: pytest.MonkeyPatch) -> None: +def test_open_store_missing_library_names_the_package(monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setattr(remote_module, "_installed", lambda module: False) with pytest.raises(RemoteStoreUnavailable) as info: open_store(parse_store_uri("gs://b/p")) - assert info.value.hint == 'install it with: pip install "evalshift[gcs]"' + assert info.value.summary == ( + "gs:// captures store needs google-cloud-storage, which is not installed" + ) + assert info.value.hint == "install it: pip install google-cloud-storage" -def test_open_store_azure_needs_identity_too(monkeypatch: pytest.MonkeyPatch) -> None: +def test_open_store_azure_names_both_packages_when_identity_is_missing( + monkeypatch: pytest.MonkeyPatch, +) -> None: monkeypatch.setattr(remote_module, "_installed", lambda module: module != "azure.identity") with pytest.raises(RemoteStoreUnavailable) as info: open_store(parse_store_uri("az://a/c/p")) - assert "'azure.identity'" in info.value.summary - assert info.value.hint == 'install it with: pip install "evalshift[azure]"' + assert info.value.summary == "az:// captures store needs azure-identity, which is not installed" + assert info.value.hint == "install it: pip install azure-storage-blob azure-identity" diff --git a/tests/unit/test_cli_capture_remote.py b/tests/unit/test_cli_capture_remote.py index 00e2eb02..1e95c8dd 100644 --- a/tests/unit/test_cli_capture_remote.py +++ b/tests/unit/test_cli_capture_remote.py @@ -320,25 +320,25 @@ def test_list_json_output_is_not_polluted_by_fetch_line( @pytest.fixture -def missing_extra(monkeypatch: pytest.MonkeyPatch) -> None: +def missing_library(monkeypatch: pytest.MonkeyPatch) -> None: """Make every store URI fail to open as if its pip extra were not installed.""" def _open(parsed: StoreURI) -> RemoteStore: raise RemoteStoreUnavailable( - "s3:// captures store needs the optional dependency 'boto3'", - hint='install it with: pip install "evalshift[s3]"', + "s3:// captures store needs boto3, which is not installed", + hint="install it: pip install boto3", ) monkeypatch.setattr(capture_module, "open_store", _open) @pytest.mark.parametrize("command", ["fetch", "sync", "list"]) -def test_missing_extra_exits_1_naming_the_pip_extra( - tmp_path: Path, missing_extra: None, command: str +def test_missing_library_exits_1_naming_the_package( + tmp_path: Path, missing_library: None, command: str ) -> None: config = _write_config(tmp_path / "evalshift.yaml") result = _invoke([command, "--config", str(config)], tmp_path) assert result.exit_code == 1 assert "Remote capture store" in result.stdout - assert 'pip install "evalshift[s3]"' in result.stdout + assert "pip install boto3" in result.stdout assert not (tmp_path / "suites").exists() From 6a966a9fa9a79d0b207f1b905807b154510c95cc Mon Sep 17 00:00:00 2001 From: Lukas Babaliauskas Date: Fri, 9 Oct 2026 23:25:16 +0200 Subject: [PATCH 5/8] fix(doctor): a missing store library is a warning, not a failure It failed compare -- and so every GitHub Action run -- for a config that names captures.store, although a run on a committed suite never reads the bucket. The capture commands that need the library still exit 1. Co-Authored-By: Claude Opus 5.5 --- src/evalshift_cli/cli/commands/capture.py | 4 ++-- src/evalshift_cli/cli/commands/doctor.py | 19 ++++++++++--------- tests/unit/test_doctor.py | 20 ++++++++++++++------ 3 files changed, 26 insertions(+), 17 deletions(-) diff --git a/src/evalshift_cli/cli/commands/capture.py b/src/evalshift_cli/cli/commands/capture.py index 7c6994bb..3d3d8a5f 100644 --- a/src/evalshift_cli/cli/commands/capture.py +++ b/src/evalshift_cli/cli/commands/capture.py @@ -242,8 +242,8 @@ def _configured_store(config_path: Path, *, console: Console, strict: bool) -> R one line, no traceback) when ``strict`` (``capture fetch`` has nothing else to do). Otherwise ``sync`` and ``list`` keep working on the local mirror the way they did before ``captures.store`` existed: an invalid or unreadable file is skipped, with a warning only when it has a top-level ``captures`` key (a - config that never asked for a store must not change their output). A missing client extra - is always a hard error: the user asked for a store it cannot reach. + config that never asked for a store must not change their output). A missing client library + is always a hard error here: the user asked for a store this command cannot read. """ if not config_path.exists(): return None diff --git a/src/evalshift_cli/cli/commands/doctor.py b/src/evalshift_cli/cli/commands/doctor.py index f53057ab..9b0b072d 100644 --- a/src/evalshift_cli/cli/commands/doctor.py +++ b/src/evalshift_cli/cli/commands/doctor.py @@ -8,8 +8,7 @@ * **0** — every check passed, or any failures were merely informational (e.g. an unset API key, or no config in this directory yet). * **1** — at least one **hard** failure was reported (currently: an - ``evalshift.yaml`` exists in the cwd but doesn't validate, or its - ``captures.store`` needs a client extra that isn't installed). + ``evalshift.yaml`` exists in the cwd but doesn't validate). Soft failures (missing API keys, no config yet) are surfaced visually with a yellow ``✗`` so users see them, but they never fail the command — this @@ -250,12 +249,14 @@ def _config_check(cwd: Path) -> CheckResult: def _captures_store_check(cwd: Path) -> list[CheckResult]: - """One row for ``captures.store`` when the config names one: extra installed, bucket listable. + """One row for ``captures.store`` when the config names one: library installed, bucket listable. No config, an invalid config (the ``evalshift.yaml`` row already reports that) or no - ``captures.store`` produce no row. A missing client extra is a ``fail`` -- it is a local, - deterministic problem with a one-line fix. A listing that raises is a ``warn``: doctor runs - on laptops without cloud credentials, and that must not fail the command. + ``captures.store`` produce no row. A missing client library is a ``warn`` naming the package + to install: a run on a committed suite never touches the bucket (CI installs the bare + package, and ``compare`` exits on any doctor failure), and the capture commands that do + need it exit 1 themselves. A listing that raises is a ``warn`` too: doctor runs on laptops + without cloud credentials, and that must not fail the command. """ cfg_path = cwd / CONFIG_FILENAME if not cfg_path.exists(): @@ -271,7 +272,7 @@ def _captures_store_check(cwd: Path) -> list[CheckResult]: store = open_store(parse_store_uri(uri)) except RemoteStoreError as exc: detail = exc.summary if exc.hint is None else f"{exc.summary} — {exc.hint}" - return [CheckResult(name="captures.store", status="fail", detail=detail)] + return [CheckResult(name="captures.store", status="warn", detail=detail)] try: next(iter(store.list("captures/")), None) except Exception as exc: @@ -528,8 +529,8 @@ def source_conformance_check(records: Sequence[EvalRecord]) -> CheckResult | Non def render_results(results: list[CheckResult], console: Console) -> None: """Render the check results as a Rich table. - Names and details are plain text, escaped so a bracketed value -- an install hint such as - ``pip install "evalshift[gcs]"`` or a user-written store URI -- is shown, not parsed as markup. + Names and details are plain text, escaped so an install hint such as ``pip install google-cloud-storage`` + or a user-written store URI is shown, not parsed as markup. """ table = Table(show_header=False, box=None, padding=(0, 1)) table.add_column("status", no_wrap=True) diff --git a/tests/unit/test_doctor.py b/tests/unit/test_doctor.py index f415aff4..f31fc4b2 100644 --- a/tests/unit/test_doctor.py +++ b/tests/unit/test_doctor.py @@ -807,18 +807,26 @@ def test_reachable_store_is_ok(self, tmp_path: Path, monkeypatch: pytest.MonkeyP assert row.status == "ok" assert "s3://acme-evals/p" in row.detail - def test_missing_extra_fails(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + def test_missing_library_warns_without_failing( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + # A run on a committed suite never touches the bucket (the GitHub Action installs the + # bare package), so a missing client library must not fail doctor -- or compare, which + # exits on any doctor failure. The capture commands that do need it exit 1 themselves. self._config(tmp_path, "gs://b/p") def _raise(parsed: object) -> object: raise RemoteStoreUnavailable( - "gs:// needs google-cloud-storage", hint='pip install "evalshift[gcs]"' + "gs:// captures store needs google-cloud-storage, which is not installed", + hint="install it: pip install google-cloud-storage", ) monkeypatch.setattr(doctor_module, "open_store", _raise) - row = _by_name(run_checks(cwd=tmp_path, env=_empty_env()), "captures.store") - assert row.status == "fail" - assert 'pip install "evalshift[gcs]"' in row.detail + results = run_checks(cwd=tmp_path, env=_empty_env()) + row = _by_name(results, "captures.store") + assert row.status == "warn" + assert "pip install google-cloud-storage" in row.detail + assert not any(r.status == "fail" for r in results) def test_unreachable_store_warns(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: from tests.unit.fake_remote_store import FakeRemoteStore @@ -852,5 +860,5 @@ def _raise(parsed: object) -> object: monkeypatch.setattr(doctor_module, "open_store", _raise) monkeypatch.chdir(tmp_path) result = runner.invoke(app, ["doctor"], env={"COLUMNS": "200"}) - assert result.exit_code == 1 + assert result.exit_code == 0 assert 'pip install "evalshift[gcs]"' in result.stdout From 694ce046caa576092aa78f16f77e4beb96bff89a Mon Sep 17 00:00:00 2001 From: Lukas Babaliauskas Date: Fri, 9 Oct 2026 23:35:01 +0200 Subject: [PATCH 6/8] docs: install the store library as a plain package; doctor warns, never fails, over it Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 13 +++++++++++++ DOCS.md | 6 +++--- docs/configuration.md | 6 +++--- docs/sdk.md | 6 +++--- llms-full.txt | 29 +++++++++++++++-------------- tests/unit/test_docs_currency.py | 20 ++++++++++++++++++++ 6 files changed, 57 insertions(+), 23 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 05274762..e13de8cb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Fixed + +- `doctor`'s `captures.store` row no longer fails the command — and with it `compare` and + every GitHub Action run — when the store's client library is not installed; it warns, naming + the package to install. A run on a committed suite never reads the bucket; `capture fetch`, + `list` and `sync`, which do, still exit 1. + +### Changed + +- Every store message names the package to install instead of a pip extra: `pip install boto3`, + `google-cloud-storage`, `azure-storage-blob azure-identity` (both Azure packages in one + command). The `[s3]` / `[gcs]` / `[azure]` extras still exist but are no longer documented. + ## [1.3.0] - 2026-10-09 ### Added diff --git a/DOCS.md b/DOCS.md index d1ec88e3..cc0fed06 100644 --- a/DOCS.md +++ b/DOCS.md @@ -173,7 +173,7 @@ captures: store: s3://acme-evals/support-agent ``` -and `capture sync` and `capture list` fetch what is new before they read the local directory — captures are immutable and toolsets content-addressed, so only objects missing locally are downloaded, captures already promoted are skipped (so `capture clean` never causes a re-download), and each file is written atomically. Bucket keys are untrusted: one not shaped like the SDK's layout — a suite directory containing `:`, `/`, `\`, NUL or `..`, or starting or ending with `.` or a space, included — is counted as malformed and never written (a key with a trailing newline included), and no key can place a file outside `.evalshift/`; folder-marker keys ending in `/` (the S3 console's "Create folder", ADLS directories) are skipped without being counted. One summary line reports it (`fetched 12 capture(s) and 1 toolset(s) from s3://…`; with `capture list --json` it goes to stderr, so stdout stays pure JSON). `--since 24h` limits a fetch to recently written captures, `--offline` skips it, and `capture fetch` does only that step. `promote`, `diff` and `clean` stay local. Install the matching extra — `evalshift[s3]`, `[gcs]` or `[azure]` (azure-storage-blob plus azure-identity) — and authenticate with the provider's normal chain (an OIDC role in CI, `aws sso login` / `gcloud auth application-default login` / `az login` locally); credentials never go in the URI, and a value containing `@` or `?` fails to load; no rejected value is echoed back, at most its scheme. A missing extra exits 1 naming the `pip install "evalshift[]"` to run; a list or download failure exits 1, so `sync` promotes nothing from a partial fetch, with a hint naming the provider's credential chain (`aws sso login` / `AWS_PROFILE` / the CI OIDC role, `gcloud auth application-default login`, `az login`); a local write failure names the file and asks you to check free disk space and permissions instead. If `evalshift.yaml` cannot be read or does not validate, `sync` and `list` skip the fetch and work on the local directory as before, warning only when the file has a top-level `captures:` key. Full field reference: [`captures`](docs/configuration.md#captures). +and `capture sync` and `capture list` fetch what is new before they read the local directory — captures are immutable and toolsets content-addressed, so only objects missing locally are downloaded, captures already promoted are skipped (so `capture clean` never causes a re-download), and each file is written atomically. Bucket keys are untrusted: one not shaped like the SDK's layout — a suite directory containing `:`, `/`, `\`, NUL or `..`, or starting or ending with `.` or a space, included — is counted as malformed and never written (a key with a trailing newline included), and no key can place a file outside `.evalshift/`; folder-marker keys ending in `/` (the S3 console's "Create folder", ADLS directories) are skipped without being counted. One summary line reports it (`fetched 12 capture(s) and 1 toolset(s) from s3://…`; with `capture list --json` it goes to stderr, so stdout stays pure JSON). `--since 24h` limits a fetch to recently written captures, `--offline` skips it, and `capture fetch` does only that step. `promote`, `diff` and `clean` stay local. Install the matching client library — `pip install boto3`, `google-cloud-storage` or `azure-storage-blob azure-identity` — and authenticate with the provider's normal chain (an OIDC role in CI, `aws sso login` / `gcloud auth application-default login` / `az login` locally); credentials never go in the URI, and a value containing `@` or `?` fails to load; no rejected value is echoed back, at most its scheme. A missing client library exits 1 naming the `pip install ` to run; a list or download failure exits 1, so `sync` promotes nothing from a partial fetch, with a hint naming the provider's credential chain (`aws sso login` / `AWS_PROFILE` / the CI OIDC role, `gcloud auth application-default login`, `az login`); a local write failure names the file and asks you to check free disk space and permissions instead. If `evalshift.yaml` cannot be read or does not validate, `sync` and `list` skip the fetch and work on the local directory as before, warning only when the file has a top-level `captures:` key. Full field reference: [`captures`](docs/configuration.md#captures). **`capture sync`** is the workhorse. Per suite it: @@ -217,7 +217,7 @@ init → doctor → run → evaluate → analy .json (if policy) ``` -- **`doctor`** validates local config and shows which provider keys are visible. Exit 1 only when an existing `evalshift.yaml` fails validation — the row gives the problem count and points at `evalshift validate`, which prints each problem — or when its `captures.store` needs a client extra that is not installed; missing keys are soft warnings. Its second row, `evalshift-sdk`, reports the SDK version the `evalshift` import name resolves to in this environment (`warn` when the SDK is missing or shadowed by an older CLI's leftover files; never a failure). It also reports the toolset each configured suite carries (or the flat `golden.jsonl`) and flags a suite whose examples carry more than one distinct toolset — legal (each example dispatches its own), but also the shape a wiring mistake takes. When a workflow under `.github/workflows/` uses the GitHub Action it adds a `ci pin` row: `ok` (`pinned to `) when CI installs this CLI version, `warn` when the pin is older, absent, or newer than the local CLI (see [Pin drift](#pin-drift)). When the config sets `captures.store`, a `captures.store` row reports it: `fail` (exit 1) naming the `pip install "evalshift[]"` when the client extra is missing, `warn` when the bucket cannot be listed (no cloud credentials on this machine, say), `ok` (` reachable`) otherwise; no row without a store. When the config wires an `llm_judge` evaluator and names both `defaults.source_model` and `target_model`, a `judge family` row warns for every `judge_model` that resolves to the same provider as an arm (self-preference bias; `ok` "from a third family" otherwise, no row when either arm is unset) — advisory, never a failure; `validate` prints the same line and the report repeats it above the verdict (see [`evaluators.llm_judge`](docs/configuration.md#evaluatorsllm_judge)). +- **`doctor`** validates local config and shows which provider keys are visible. Exit 1 only when an existing `evalshift.yaml` fails validation — the row gives the problem count and points at `evalshift validate`, which prints each problem; missing keys are soft warnings. Its second row, `evalshift-sdk`, reports the SDK version the `evalshift` import name resolves to in this environment (`warn` when the SDK is missing or shadowed by an older CLI's leftover files; never a failure). It also reports the toolset each configured suite carries (or the flat `golden.jsonl`) and flags a suite whose examples carry more than one distinct toolset — legal (each example dispatches its own), but also the shape a wiring mistake takes. When a workflow under `.github/workflows/` uses the GitHub Action it adds a `ci pin` row: `ok` (`pinned to `) when CI installs this CLI version, `warn` when the pin is older, absent, or newer than the local CLI (see [Pin drift](#pin-drift)). When the config sets `captures.store`, a `captures.store` row reports it: `warn` naming the `pip install ` to run when the client library is missing (the capture commands that need it exit 1 themselves), `warn` when the bucket cannot be listed (no cloud credentials on this machine, say), `ok` (` reachable`) otherwise; no row without a store. When the config wires an `llm_judge` evaluator and names both `defaults.source_model` and `target_model`, a `judge family` row warns for every `judge_model` that resolves to the same provider as an arm (self-preference bias; `ok` "from a third family" otherwise, no row when either arm is unset) — advisory, never a failure; `validate` prints the same line and the report repeats it above the verdict (see [`evaluators.llm_judge`](docs/configuration.md#evaluatorsllm_judge)). - **`run`** parses prompts, validates every example against every prompt, estimates cost, then dispatches `(prompt × example × {source, target})` calls through an async orchestrator under a concurrency semaphore. Responses are cached — per call, and per replayed round for tool-calling examples (see [Response cache](#response-cache)); progress is checkpointed every 50 completions. - **`evaluate`** scores each (source, target) pair with the configured evaluators, one `EvalRecord` per pair × evaluator. Scoring runs under the same `defaults.concurrency` semaphore as `run`, and the embedding/judge calls it makes go through the same response cache. - **`analyze`** runs paired statistics per `(prompt, evaluator, slice)`, applies Benjamini–Hochberg FDR correction, classifies severities, and — when a `migration_policy` is configured — computes a pass/fail verdict. @@ -869,7 +869,7 @@ Common conventions: `-c/--config` defaults to `./evalshift.yaml`; run artefacts `-f/--force` · `-d/--directory ` · `--ci` · `--wire-agents/--no-wire-agents` (default on) · `--provider gemini|openai|anthropic|deepseek` · `--profile model-upgrade|cost-reduction|local-model|quantization|provider-switch` (default `model-upgrade`) Without `--ci`, warns after writing when an existing workflow under `.github/workflows/` pins an older or newer CLI than this one, or none at all (see [Pin drift](#pin-drift)); `init --ci` writes the pin itself and does not warn about the file it just wrote. -**`evalshift doctor`** — environment/config check. Exit 1 only on an invalid existing config or a `captures.store` whose client extra is missing. Row 2, `evalshift-sdk`, confirms `import evalshift` is the SDK (`warn` when missing or shadowed, never a failure). Reports the toolset each configured suite carries and flags a suite whose examples carry more than one distinct toolset. The suite-side checks cover every suite in the config's `suites:` block, falling back to `./golden.jsonl` when none are wired. Adds a `ci pin` row when a workflow uses the GitHub Action (`warn` on pin drift, never a failure) and a `judge family` row when an `llm_judge` judge shares a provider with a configured arm (`warn`, never a failure). Adds a `captures.store` row when the config names a store (`fail` on a missing extra, `warn` when the bucket cannot be listed). +**`evalshift doctor`** — environment/config check. Exit 1 only on an invalid existing config. Row 2, `evalshift-sdk`, confirms `import evalshift` is the SDK (`warn` when missing or shadowed, never a failure). Reports the toolset each configured suite carries and flags a suite whose examples carry more than one distinct toolset. The suite-side checks cover every suite in the config's `suites:` block, falling back to `./golden.jsonl` when none are wired. Adds a `ci pin` row when a workflow uses the GitHub Action (`warn` on pin drift, never a failure) and a `judge family` row when an `llm_judge` judge shares a provider with a configured arm (`warn`, never a failure). Adds a `captures.store` row when the config names a store (`warn` naming the package to install when the client library is missing, `warn` when the bucket cannot be listed). **`evalshift run`** — paired evaluation run (costs money — calls real models). `-f/--from ` · `-t/--to ` · `-c/--config` · `-s/--suite ` · `--suite-name ` · `--resume` · `-y/--yes` diff --git a/docs/configuration.md b/docs/configuration.md index 24c30006..50e8fb8c 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1041,9 +1041,9 @@ Accepted URI forms — the same grammar the SDK uses: | Form | Store | Install | Credentials | | --- | --- | --- | --- | -| `s3:///` | Amazon S3; MinIO, Cloudflare R2, Backblaze B2 via boto3's `AWS_ENDPOINT_URL` | `pip install "evalshift[s3]"` | IAM role in CI (OIDC), `aws sso login` locally | -| `gs:///` | Google Cloud Storage | `pip install "evalshift[gcs]"` | Workload Identity in CI, `gcloud auth application-default login` locally | -| `az:////` | Azure Blob Storage | `pip install "evalshift[azure]"` | Managed Identity / federated credential in CI, `az login` locally | +| `s3:///` | Amazon S3; MinIO, Cloudflare R2, Backblaze B2 via boto3's `AWS_ENDPOINT_URL` | `pip install boto3` | IAM role in CI (OIDC), `aws sso login` locally | +| `gs:///` | Google Cloud Storage | `pip install google-cloud-storage` | Workload Identity in CI, `gcloud auth application-default login` locally | +| `az:////` | Azure Blob Storage | `pip install azure-storage-blob azure-identity` | Managed Identity / federated credential in CI, `az login` locally | `` is optional. Credentials never go in the URI: a value containing `@` or `?` fails to load, naming the accepted forms. A rejected value is never echoed back (it may be a pasted diff --git a/docs/sdk.md b/docs/sdk.md index 60cd3bc1..3ee33e1a 100644 --- a/docs/sdk.md +++ b/docs/sdk.md @@ -121,9 +121,9 @@ lifecycle rule; `EVALSHIFT_MAX_CAPTURES` applies to local disk only. Run the CLI from the directory holding `.evalshift/` (or set `EVALSHIFT_DIR`). If the captures live in a bucket, name it once in `evalshift.yaml` -(`captures: {store: s3://acme-evals/support-agent}`) and install `evalshift[s3]` (or `[gcs]` / -`[azure]`); `list` and `sync` then fetch new captures first. `evalshift capture fetch` does -only that step. +(`captures: {store: s3://acme-evals/support-agent}`) and install the provider's client library +(`pip install boto3`, `google-cloud-storage` or `azure-storage-blob azure-identity`); `list` and +`sync` then fetch new captures first. `evalshift capture fetch` does only that step. ```bash evalshift capture list # what was recorded (--json for machine output) diff --git a/llms-full.txt b/llms-full.txt index 1fc511e7..af6e9d34 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -153,9 +153,9 @@ evalshift init [-f/--force] [-d/--directory DIR] [--ci] [--wire-agents/--no-wire quantization: .02/0/.97/.005/.02/0/.00/.20 | provider-switch: .03/0/.95/.01/.03/0/.20/.40 evalshift doctor Env/config check. Exit 1 ONLY when an existing evalshift.yaml fails validation (the row shows - the problem count + "run `evalshift validate` for details"; validate prints each problem) or - its captures.store needs a client extra that is not installed; missing API keys are soft - warnings (exit 0). + the problem count + "run `evalshift validate` for details"; validate prints each problem); + missing API keys and a captures.store whose client library is missing are soft warnings + (exit 0). Row 2 `evalshift-sdk`: ok (" (import name `evalshift`)"); warn (exit 0) when the SDK is not installed, fails to import, or the name resolves to something else (an older CLI's leftover files, a local evalshift/ directory). @@ -175,9 +175,10 @@ evalshift doctor when none overlaps; no row when either arm is unset or no llm_judge is configured. Provider "other" (id the registry cannot place) never matches. Same wording from `validate`; the report repeats it (see Judge family below). - captures.store: fail if the extra is missing (detail names pip install "evalshift[]"; - exit 1), warn if the bucket cannot be listed (exit 0), ok " reachable"; no row without - a store (or when evalshift.yaml is invalid -- its own row reports that). + captures.store: warn if the client library is missing (detail names "pip install "; + exit 0 -- compare and CI are not failed over a bucket a committed-suite run never reads), warn + if the bucket cannot be listed (exit 0), ok " reachable"; no row without a store (or when + evalshift.yaml is invalid -- its own row reports that). evalshift run [-f/--from MODEL] [-t/--to MODEL] [-c CONFIG] [-s/--suite FILE] [--suite-name NAME] [--resume] [-y/--yes] Paired run over (prompt x example x {source,target}). --from/--to override @@ -319,10 +320,10 @@ evalshift capture fetch [--suite S] [--since 30m|24h|7d|ISO] [-c CONFIG] naming the scheme's credential chain (s3: aws sso login / AWS_PROFILE / the CI OIDC role; gs: gcloud auth application-default login / Workload Identity; az: az login / managed identity) and --offline on sync/list. A local write failure exits 1 naming the file, with a - hint to check free disk space and permissions on its directory. Missing extra -> exit 1 naming - pip install "evalshift[s3|gcs|azure]" (az:// needs azure-storage-blob AND azure-identity, - both in [azure]). No captures.store, or no evalshift.yaml -> exit 1; an invalid - evalshift.yaml -> its load error, exit 1; an unreadable or non-UTF-8 one -> a one-line + hint to check free disk space and permissions on its directory. Missing library -> exit 1 + naming pip install boto3 | google-cloud-storage | azure-storage-blob azure-identity (az:// + needs both; one command installs them). No captures.store, or no evalshift.yaml -> exit 1; + an invalid evalshift.yaml -> its load error, exit 1; an unreadable or non-UTF-8 one -> a one-line "could not read " error, exit 1. Has no --offline flag. evalshift capture promote CAPTURE_ID [--as CASE_ID] [--suite S] [--input-var NAME=input] [--tag T]... [--strict-args] [--names-only] [--tool-count] [--rounds first(default)|all] @@ -739,9 +740,9 @@ captures: store: s3:/// | gs:/// | az://// -- optional block; store validated at load (bad scheme, missing bucket/container, '@' or '?' -> load error naming the forms; no error echoes the value, at most its scheme); prefix optional, - trailing slash stripped; default none = local disk only. Extras: evalshift[s3] (boto3; - AWS_ENDPOINT_URL covers MinIO/R2/B2), [gcs] (google-cloud-storage), [azure] - (azure-storage-blob + azure-identity). Credentials: each provider's default chain only. + trailing slash stripped; default none = local disk only. Client libraries are plain packages: + pip install boto3 (s3://; AWS_ENDPOINT_URL covers MinIO/R2/B2), google-cloud-storage (gs://), + azure-storage-blob azure-identity (az://). Credentials: each provider's default chain only. Model ids: built-in registry provides aliases/metadata but NEVER gates — LiteLLM is the call-time authority; any LiteLLM-supported model works. Unknown ids pass through with provider @@ -1307,7 +1308,7 @@ content leaves the machine. - Report is a single self-contained HTML file: no external assets, works offline. - Run pruning never touches an in-progress run or the run just finished. - Exit codes: 0 success; 1 handled errors and tripped CI gates; doctor exits 1 only on invalid - existing config or a captures.store whose client extra is missing; init exits 2 on unknown + existing config; init exits 2 on unknown --provider; Typer usage errors exit 2. ## Minimal examples diff --git a/tests/unit/test_docs_currency.py b/tests/unit/test_docs_currency.py index fe0afa7e..e034f277 100644 --- a/tests/unit/test_docs_currency.py +++ b/tests/unit/test_docs_currency.py @@ -253,3 +253,23 @@ def test_scaffolded_agent_guide_names_every_capture_subcommand(name: str) -> Non ) assert f"`{name}`" in row, f"EVALSHIFT.md's capture row does not name `{name}`" + + +#: Install-story phrases retired on 2026-10-09: every message and doc names the package to +#: install (`pip install boto3`), never a pip extra, and `doctor` no longer fails over it. +RETIRED_INSTALL_TERMS: tuple[str, ...] = ( + "evalshift[s3]", + "evalshift[gcs]", + "evalshift[azure]", + "client extra", + "optional dependency", + "whose extra is missing", +) + +_INSTALL_COPY_FILES: tuple[str, ...] = (*PROSE_FILES, "docs/sdk.md") + + +@pytest.mark.parametrize("term", RETIRED_INSTALL_TERMS) +def test_copy_names_packages_not_extras(term: str) -> None: + offenders = [name for name in _INSTALL_COPY_FILES if term in _whitespace_normalized(name)] + assert offenders == [], f"{term!r} still appears in {offenders}" From bdfe13c17b44d81b5d8660260a765fe7e41923bd Mon Sep 17 00:00:00 2001 From: Lukas Babaliauskas Date: Fri, 9 Oct 2026 23:56:29 +0200 Subject: [PATCH 7/8] chore: wrap a long docstring; the install tripwire covers two more docs pages - captures/remote.py: open_store()'s Raises: line for RemoteStoreUnavailable was 231 characters; wrap it to the file's ~100-column style. - test_docs_currency.py: _INSTALL_COPY_FILES was missing docs/getting-started.md and docs/agents.md. Both exist and currently contain none of RETIRED_INSTALL_TERMS; add them so a regression there is caught. Co-Authored-By: Claude Opus 5.5 --- src/evalshift_cli/captures/remote.py | 4 +++- tests/unit/test_docs_currency.py | 7 ++++++- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/src/evalshift_cli/captures/remote.py b/src/evalshift_cli/captures/remote.py index 6d225df0..3ffd20ed 100644 --- a/src/evalshift_cli/captures/remote.py +++ b/src/evalshift_cli/captures/remote.py @@ -162,7 +162,9 @@ def open_store(parsed: StoreURI) -> RemoteStore: The scheme's adapter; its client is built on the first ``list`` or ``get``. Raises: - RemoteStoreUnavailable: when a client module the scheme needs is not installed. The summary names the package that provides the first missing module; the hint names the pip command that installs everything the scheme needs. + RemoteStoreUnavailable: when a client module the scheme needs is not installed. The + summary names the package that provides the first missing module; the hint names + the pip command that installs everything the scheme needs. """ for module in _REQUIRED_MODULES[parsed.scheme]: if not _installed(module): diff --git a/tests/unit/test_docs_currency.py b/tests/unit/test_docs_currency.py index e034f277..08357d26 100644 --- a/tests/unit/test_docs_currency.py +++ b/tests/unit/test_docs_currency.py @@ -266,7 +266,12 @@ def test_scaffolded_agent_guide_names_every_capture_subcommand(name: str) -> Non "whose extra is missing", ) -_INSTALL_COPY_FILES: tuple[str, ...] = (*PROSE_FILES, "docs/sdk.md") +_INSTALL_COPY_FILES: tuple[str, ...] = ( + *PROSE_FILES, + "docs/sdk.md", + "docs/getting-started.md", + "docs/agents.md", +) @pytest.mark.parametrize("term", RETIRED_INSTALL_TERMS) From 41b2492369d83916ed0251443c7c89786204e42a Mon Sep 17 00:00:00 2001 From: Lukas Babaliauskas Date: Sat, 10 Oct 2026 23:56:42 +0200 Subject: [PATCH 8/8] docs: the SDK's store variable is EVALSHIFT_CAPTURE_STORE (EVALSHIFT_SINK on SDK 0.5.0) Matches the SDK rename in evalshift-sdk#15, so the bucket has one name on both sides: EVALSHIFT_CAPTURE_STORE in the agent's environment and captures.store in evalshift.yaml. Version notes say which SDK reads which name. Also drops a leftover 'optional extras' phrase in sdk.md. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 3 +++ DOCS.md | 2 +- docs/configuration.md | 4 ++-- docs/getting-started.md | 7 ++++--- docs/sdk.md | 11 ++++++----- llms-full.txt | 3 ++- src/evalshift_cli/config/models.py | 2 +- 7 files changed, 19 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e13de8cb..5cd18ce1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- Docs: the SDK's object-store variable is `EVALSHIFT_CAPTURE_STORE` from SDK 0.6.0, matching + `captures.store` here; SDK 0.5.0 reads the same value from `EVALSHIFT_SINK`, which keeps + working on newer SDKs too. - Every store message names the package to install instead of a pip extra: `pip install boto3`, `google-cloud-storage`, `azure-storage-blob azure-identity` (both Azure packages in one command). The `[s3]` / `[gcs]` / `[azure]` extras still exist but are no longer documented. diff --git a/DOCS.md b/DOCS.md index cc0fed06..bdb5ea6a 100644 --- a/DOCS.md +++ b/DOCS.md @@ -166,7 +166,7 @@ evalshift capture diff cap_ab12 cap_cd34 # compare two tool traces evalshift capture clean # delete promoted capture files + sweep orphaned toolsets ``` -**Captures that live in a bucket.** Production hosts on Fargate, Lambda or Kubernetes lose their disk when they stop, so the SDK can ship captures to an object store you own instead (`EVALSHIFT_SINK=s3:///`, `gs://…` or `az:////…` in the agent's environment, SDK 0.5.0+). Name the same store once in `evalshift.yaml`: +**Captures that live in a bucket.** Production hosts on Fargate, Lambda or Kubernetes lose their disk when they stop, so the SDK can ship captures to an object store you own instead (`EVALSHIFT_CAPTURE_STORE=s3:///`, `gs://…` or `az:////…` in the agent's environment, SDK 0.6.0+; SDK 0.5.0 reads the same value from `EVALSHIFT_SINK`). Name the same store once in `evalshift.yaml`: ```yaml captures: diff --git a/docs/configuration.md b/docs/configuration.md index 50e8fb8c..74f8bbc3 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1021,8 +1021,8 @@ entirely for the default: captures are read from `.evalshift/captures/` (or `EVA on local disk and nothing here is needed. Production agents on Fargate, Lambda or Kubernetes lose their disk when they stop, so the -SDK can ship captures to an object store you own instead (`EVALSHIFT_SINK=` in the -agent's environment — see [Capture SDK](sdk.md)). Name the same store here and the CLI +SDK can ship captures to an object store you own instead (`EVALSHIFT_CAPTURE_STORE=` in the +agent's environment, `EVALSHIFT_SINK` on SDK 0.5.0 — see [Capture SDK](sdk.md)). Name the same store here and the CLI pulls new captures down before it promotes them, with no flags to remember: ```yaml diff --git a/docs/getting-started.md b/docs/getting-started.md index 092b3dbe..765df499 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -124,9 +124,10 @@ EVALSHIFT_CAPTURE=1 python your_agent.py # writes .evalshift/captures/ ``` Running on Fargate, Lambda or Kubernetes, where the disk does not outlive the task? Add -`EVALSHIFT_SINK=s3:///` (or `gs://…` / `az://…`) next to it and put the same -URI under `captures: {store: …}` in `evalshift.yaml`; step 6 then fetches from the bucket -first. See [`captures`](configuration.md#captures). +`EVALSHIFT_CAPTURE_STORE=s3:///` (or `gs://…` / `az://…`; SDK 0.6.0+, or +`EVALSHIFT_SINK` on SDK 0.5.0) next to it and put the same URI under `captures: {store: …}` in +`evalshift.yaml`; step 6 then fetches from the bucket first. See +[`captures`](configuration.md#captures). Captures are off unless `EVALSHIFT_CAPTURE=1` is set, so the decorator can stay in production code. If the agent calls OpenAI, Anthropic or Google GenAI diff --git a/docs/sdk.md b/docs/sdk.md index 3ee33e1a..de1152c9 100644 --- a/docs/sdk.md +++ b/docs/sdk.md @@ -14,7 +14,7 @@ dense LLM reference at . ``` your agent (evalshift-sdk) → .evalshift/captures//cap_.json - → or, EVALSHIFT_SINK=s3://… : /captures//cap_.json + → or, EVALSHIFT_CAPTURE_STORE=s3://… : /captures//cap_.json → evalshift capture sync (fetches from captures.store first) → .evalshift/suites//golden.jsonl + evalshift.yaml ``` @@ -30,9 +30,10 @@ your agent (evalshift-sdk) → .evalshift/captures//cap_.json `evalshift-sdk` alone. - **Off by default.** Nothing is recorded unless `EVALSHIFT_CAPTURE=1` is set, so the instrumentation is safe to leave in production code permanently. -- **No network by default.** The SDK writes local files unless `EVALSHIFT_SINK` names a bucket - you own (`s3://`, `gs://`, `az://`, SDK 0.5.0+), and then it writes only there — never to - EvalShift. Python 3.10+, stdlib-only; the cloud clients are optional extras. +- **No network by default.** The SDK writes local files unless `EVALSHIFT_CAPTURE_STORE` names a + bucket you own (`s3://`, `gs://`, `az://`; SDK 0.6.0+, or `EVALSHIFT_SINK` on SDK 0.5.0), and + then it writes only there — never to EvalShift. Python 3.10+, stdlib-only; a bucket's client + library is a separate package you install only if you use one. ## 1. Instrument the agent @@ -110,7 +111,7 @@ the SDK, in your agent's process. ```bash # on a host whose disk does not outlive it (Fargate, Lambda, pods): -EVALSHIFT_CAPTURE=1 EVALSHIFT_SINK=s3://acme-evals/support-agent python your_agent.py +EVALSHIFT_CAPTURE=1 EVALSHIFT_CAPTURE_STORE=s3://acme-evals/support-agent python your_agent.py ``` The SDK ships each capture (and its toolset sidecar) to the bucket on a background thread, diff --git a/llms-full.txt b/llms-full.txt index af6e9d34..7d068da2 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -31,7 +31,8 @@ Data flow: SDK captures -> CLI runs and bundles -> server stores/diffs -> web ap Companion package: evalshift-sdk (separate PyPI package, in-process capture SDK) records production agent runs as JSON captures under .evalshift/captures//cap_.json; or, -with EVALSHIFT_SINK, under the same layout in the user's bucket; +with EVALSHIFT_CAPTURE_STORE (EVALSHIFT_SINK on SDK 0.5.0), under the same layout in the +user's bucket; this CLI promotes them into golden suites (`evalshift capture sync`). That file layout is the only interface between SDK and CLI — they never import or call each other. The CLI imports as `evalshift_cli` and depends on evalshift-sdk, so one environment holds diff --git a/src/evalshift_cli/config/models.py b/src/evalshift_cli/config/models.py index 6b0fe877..3953c5cb 100644 --- a/src/evalshift_cli/config/models.py +++ b/src/evalshift_cli/config/models.py @@ -821,7 +821,7 @@ class CapturesConfig(_StrictModel): By default captures are read from ``/captures/`` on this machine and nothing here is needed. Production hosts whose disk does not outlive them (Fargate, Lambda, pods) can - ship captures to an object store instead (``EVALSHIFT_SINK`` on the SDK side); naming the + ship captures to an object store instead (``EVALSHIFT_CAPTURE_STORE`` on the SDK side); naming the same store here makes ``capture sync`` and ``capture list`` mirror new captures into the local directory first, with no flags.