Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Fixed

- `doctor`'s `captures.store` row no longer fails the command — and with it `compare` and
every GitHub Action run — when the store's client library is not installed; it warns, naming
the package to install. A run on a committed suite never reads the bucket; `capture fetch`,
`list` and `sync`, which do, still exit 1.
- A suspended account is named as such. When the server refuses a token with
`account_suspended`, the CLI prints its message — "Your account has been suspended.
Contact support@evalshift.dev." `evalshift login` with a stored token for that account
exits 1 with it instead of calling the token no longer valid and starting a browser
sign-in, and push auto-create shows it without the token-access hint. Service account
keys are not affected by a user's suspension.

### Changed

- Docs: the SDK's object-store variable is `EVALSHIFT_CAPTURE_STORE` from SDK 0.6.0, matching
`captures.store` here; SDK 0.5.0 reads the same value from `EVALSHIFT_SINK`, which keeps
working on newer SDKs too.
- Every store message names the package to install instead of a pip extra: `pip install boto3`,
`google-cloud-storage`, `azure-storage-blob azure-identity` (both Azure packages in one
command). The `[s3]` / `[gcs]` / `[azure]` extras still exist but are no longer documented.

## [1.3.0] - 2026-10-09

### Added
Expand Down
8 changes: 4 additions & 4 deletions DOCS.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,14 +166,14 @@ evalshift capture diff cap_ab12 cap_cd34 # compare two tool traces
evalshift capture clean # delete promoted capture files + sweep orphaned toolsets
```

**Captures that live in a bucket.** Production hosts on Fargate, Lambda or Kubernetes lose their disk when they stop, so the SDK can ship captures to an object store you own instead (`EVALSHIFT_SINK=s3://<bucket>/<prefix>`, `gs://…` or `az://<account>/<container>/…` in the agent's environment, SDK 0.5.0+). Name the same store once in `evalshift.yaml`:
**Captures that live in a bucket.** Production hosts on Fargate, Lambda or Kubernetes lose their disk when they stop, so the SDK can ship captures to an object store you own instead (`EVALSHIFT_CAPTURE_STORE=s3://<bucket>/<prefix>`, `gs://…` or `az://<account>/<container>/…` in the agent's environment, SDK 0.6.0+; SDK 0.5.0 reads the same value from `EVALSHIFT_SINK`). Name the same store once in `evalshift.yaml`:

```yaml
captures:
store: s3://acme-evals/support-agent
```

and `capture sync` and `capture list` fetch what is new before they read the local directory — captures are immutable and toolsets content-addressed, so only objects missing locally are downloaded, captures already promoted are skipped (so `capture clean` never causes a re-download), and each file is written atomically. Bucket keys are untrusted: one not shaped like the SDK's layout — a suite directory containing `:`, `/`, `\`, NUL or `..`, or starting or ending with `.` or a space, included — is counted as malformed and never written (a key with a trailing newline included), and no key can place a file outside `.evalshift/`; folder-marker keys ending in `/` (the S3 console's "Create folder", ADLS directories) are skipped without being counted. One summary line reports it (`fetched 12 capture(s) and 1 toolset(s) from s3://…`; with `capture list --json` it goes to stderr, so stdout stays pure JSON). `--since 24h` limits a fetch to recently written captures, `--offline` skips it, and `capture fetch` does only that step. `promote`, `diff` and `clean` stay local. Install the matching extra — `evalshift[s3]`, `[gcs]` or `[azure]` (azure-storage-blob plus azure-identity) — and authenticate with the provider's normal chain (an OIDC role in CI, `aws sso login` / `gcloud auth application-default login` / `az login` locally); credentials never go in the URI, and a value containing `@` or `?` fails to load; no rejected value is echoed back, at most its scheme. A missing extra exits 1 naming the `pip install "evalshift[<extra>]"` to run; a list or download failure exits 1, so `sync` promotes nothing from a partial fetch, with a hint naming the provider's credential chain (`aws sso login` / `AWS_PROFILE` / the CI OIDC role, `gcloud auth application-default login`, `az login`); a local write failure names the file and asks you to check free disk space and permissions instead. If `evalshift.yaml` cannot be read or does not validate, `sync` and `list` skip the fetch and work on the local directory as before, warning only when the file has a top-level `captures:` key. Full field reference: [`captures`](docs/configuration.md#captures).
and `capture sync` and `capture list` fetch what is new before they read the local directory — captures are immutable and toolsets content-addressed, so only objects missing locally are downloaded, captures already promoted are skipped (so `capture clean` never causes a re-download), and each file is written atomically. Bucket keys are untrusted: one not shaped like the SDK's layout — a suite directory containing `:`, `/`, `\`, NUL or `..`, or starting or ending with `.` or a space, included — is counted as malformed and never written (a key with a trailing newline included), and no key can place a file outside `.evalshift/`; folder-marker keys ending in `/` (the S3 console's "Create folder", ADLS directories) are skipped without being counted. One summary line reports it (`fetched 12 capture(s) and 1 toolset(s) from s3://…`; with `capture list --json` it goes to stderr, so stdout stays pure JSON). `--since 24h` limits a fetch to recently written captures, `--offline` skips it, and `capture fetch` does only that step. `promote`, `diff` and `clean` stay local. Install the matching client library — `pip install boto3`, `google-cloud-storage` or `azure-storage-blob azure-identity` — and authenticate with the provider's normal chain (an OIDC role in CI, `aws sso login` / `gcloud auth application-default login` / `az login` locally); credentials never go in the URI, and a value containing `@` or `?` fails to load; no rejected value is echoed back, at most its scheme. A missing client library exits 1 naming the `pip install <package>` to run; a list or download failure exits 1, so `sync` promotes nothing from a partial fetch, with a hint naming the provider's credential chain (`aws sso login` / `AWS_PROFILE` / the CI OIDC role, `gcloud auth application-default login`, `az login`); a local write failure names the file and asks you to check free disk space and permissions instead. If `evalshift.yaml` cannot be read or does not validate, `sync` and `list` skip the fetch and work on the local directory as before, warning only when the file has a top-level `captures:` key. Full field reference: [`captures`](docs/configuration.md#captures).

**`capture sync`** is the workhorse. Per suite it:

Expand Down Expand Up @@ -217,7 +217,7 @@ init → doctor → run → evaluate → analy
.json (if policy)
```

- **`doctor`** validates local config and shows which provider keys are visible. Exit 1 only when an existing `evalshift.yaml` fails validation — the row gives the problem count and points at `evalshift validate`, which prints each problem — or when its `captures.store` needs a client extra that is not installed; missing keys are soft warnings. Its second row, `evalshift-sdk`, reports the SDK version the `evalshift` import name resolves to in this environment (`warn` when the SDK is missing or shadowed by an older CLI's leftover files; never a failure). It also reports the toolset each configured suite carries (or the flat `golden.jsonl`) and flags a suite whose examples carry more than one distinct toolset — legal (each example dispatches its own), but also the shape a wiring mistake takes. When a workflow under `.github/workflows/` uses the GitHub Action it adds a `ci pin` row: `ok` (`pinned to <v>`) when CI installs this CLI version, `warn` when the pin is older, absent, or newer than the local CLI (see [Pin drift](#pin-drift)). When the config sets `captures.store`, a `captures.store` row reports it: `fail` (exit 1) naming the `pip install "evalshift[<extra>]"` when the client extra is missing, `warn` when the bucket cannot be listed (no cloud credentials on this machine, say), `ok` (`<uri> reachable`) otherwise; no row without a store. When the config wires an `llm_judge` evaluator and names both `defaults.source_model` and `target_model`, a `judge family` row warns for every `judge_model` that resolves to the same provider as an arm (self-preference bias; `ok` "from a third family" otherwise, no row when either arm is unset) — advisory, never a failure; `validate` prints the same line and the report repeats it above the verdict (see [`evaluators.llm_judge`](docs/configuration.md#evaluatorsllm_judge)).
- **`doctor`** validates local config and shows which provider keys are visible. Exit 1 only when an existing `evalshift.yaml` fails validation — the row gives the problem count and points at `evalshift validate`, which prints each problem; missing keys are soft warnings. Its second row, `evalshift-sdk`, reports the SDK version the `evalshift` import name resolves to in this environment (`warn` when the SDK is missing or shadowed by an older CLI's leftover files; never a failure). It also reports the toolset each configured suite carries (or the flat `golden.jsonl`) and flags a suite whose examples carry more than one distinct toolset — legal (each example dispatches its own), but also the shape a wiring mistake takes. When a workflow under `.github/workflows/` uses the GitHub Action it adds a `ci pin` row: `ok` (`pinned to <v>`) when CI installs this CLI version, `warn` when the pin is older, absent, or newer than the local CLI (see [Pin drift](#pin-drift)). When the config sets `captures.store`, a `captures.store` row reports it: `warn` naming the `pip install <package>` to run when the client library is missing (the capture commands that need it exit 1 themselves), `warn` when the bucket cannot be listed (no cloud credentials on this machine, say), `ok` (`<uri> reachable`) otherwise; no row without a store. When the config wires an `llm_judge` evaluator and names both `defaults.source_model` and `target_model`, a `judge family` row warns for every `judge_model` that resolves to the same provider as an arm (self-preference bias; `ok` "from a third family" otherwise, no row when either arm is unset) — advisory, never a failure; `validate` prints the same line and the report repeats it above the verdict (see [`evaluators.llm_judge`](docs/configuration.md#evaluatorsllm_judge)).
- **`run`** parses prompts, validates every example against every prompt, estimates cost, then dispatches `(prompt × example × {source, target})` calls through an async orchestrator under a concurrency semaphore. Responses are cached — per call, and per replayed round for tool-calling examples (see [Response cache](#response-cache)); progress is checkpointed every 50 completions.
- **`evaluate`** scores each (source, target) pair with the configured evaluators, one `EvalRecord` per pair × evaluator. Scoring runs under the same `defaults.concurrency` semaphore as `run`, and the embedding/judge calls it makes go through the same response cache.
- **`analyze`** runs paired statistics per `(prompt, evaluator, slice)`, applies Benjamini–Hochberg FDR correction, classifies severities, and — when a `migration_policy` is configured — computes a pass/fail verdict.
Expand Down Expand Up @@ -869,7 +869,7 @@ Common conventions: `-c/--config` defaults to `./evalshift.yaml`; run artefacts
`-f/--force` · `-d/--directory <dir>` · `--ci` · `--wire-agents/--no-wire-agents` (default on) · `--provider gemini|openai|anthropic|deepseek` · `--profile model-upgrade|cost-reduction|local-model|quantization|provider-switch` (default `model-upgrade`)
Without `--ci`, warns after writing when an existing workflow under `.github/workflows/` pins an older or newer CLI than this one, or none at all (see [Pin drift](#pin-drift)); `init --ci` writes the pin itself and does not warn about the file it just wrote.

**`evalshift doctor`** — environment/config check. Exit 1 only on an invalid existing config or a `captures.store` whose client extra is missing. Row 2, `evalshift-sdk`, confirms `import evalshift` is the SDK (`warn` when missing or shadowed, never a failure). Reports the toolset each configured suite carries and flags a suite whose examples carry more than one distinct toolset. The suite-side checks cover every suite in the config's `suites:` block, falling back to `./golden.jsonl` when none are wired. Adds a `ci pin` row when a workflow uses the GitHub Action (`warn` on pin drift, never a failure) and a `judge family` row when an `llm_judge` judge shares a provider with a configured arm (`warn`, never a failure). Adds a `captures.store` row when the config names a store (`fail` on a missing extra, `warn` when the bucket cannot be listed).
**`evalshift doctor`** — environment/config check. Exit 1 only on an invalid existing config. Row 2, `evalshift-sdk`, confirms `import evalshift` is the SDK (`warn` when missing or shadowed, never a failure). Reports the toolset each configured suite carries and flags a suite whose examples carry more than one distinct toolset. The suite-side checks cover every suite in the config's `suites:` block, falling back to `./golden.jsonl` when none are wired. Adds a `ci pin` row when a workflow uses the GitHub Action (`warn` on pin drift, never a failure) and a `judge family` row when an `llm_judge` judge shares a provider with a configured arm (`warn`, never a failure). Adds a `captures.store` row when the config names a store (`warn` naming the package to install when the client library is missing, `warn` when the bucket cannot be listed).

**`evalshift run`** — paired evaluation run (costs money — calls real models).
`-f/--from <model>` · `-t/--to <model>` · `-c/--config` · `-s/--suite <file>` · `--suite-name <name>` · `--resume` · `-y/--yes`
Expand Down
10 changes: 5 additions & 5 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -1021,8 +1021,8 @@ entirely for the default: captures are read from `.evalshift/captures/` (or `EVA
on local disk and nothing here is needed.

Production agents on Fargate, Lambda or Kubernetes lose their disk when they stop, so the
SDK can ship captures to an object store you own instead (`EVALSHIFT_SINK=<uri>` in the
agent's environment — see [Capture SDK](sdk.md)). Name the same store here and the CLI
SDK can ship captures to an object store you own instead (`EVALSHIFT_CAPTURE_STORE=<uri>` in the
agent's environment, `EVALSHIFT_SINK` on SDK 0.5.0 — see [Capture SDK](sdk.md)). Name the same store here and the CLI
pulls new captures down before it promotes them, with no flags to remember:

```yaml
Expand All @@ -1041,9 +1041,9 @@ Accepted URI forms — the same grammar the SDK uses:

| Form | Store | Install | Credentials |
| --- | --- | --- | --- |
| `s3://<bucket>/<prefix>` | Amazon S3; MinIO, Cloudflare R2, Backblaze B2 via boto3's `AWS_ENDPOINT_URL` | `pip install "evalshift[s3]"` | IAM role in CI (OIDC), `aws sso login` locally |
| `gs://<bucket>/<prefix>` | Google Cloud Storage | `pip install "evalshift[gcs]"` | Workload Identity in CI, `gcloud auth application-default login` locally |
| `az://<account>/<container>/<prefix>` | Azure Blob Storage | `pip install "evalshift[azure]"` | Managed Identity / federated credential in CI, `az login` locally |
| `s3://<bucket>/<prefix>` | Amazon S3; MinIO, Cloudflare R2, Backblaze B2 via boto3's `AWS_ENDPOINT_URL` | `pip install boto3` | IAM role in CI (OIDC), `aws sso login` locally |
| `gs://<bucket>/<prefix>` | Google Cloud Storage | `pip install google-cloud-storage` | Workload Identity in CI, `gcloud auth application-default login` locally |
| `az://<account>/<container>/<prefix>` | Azure Blob Storage | `pip install azure-storage-blob azure-identity` | Managed Identity / federated credential in CI, `az login` locally |

`<prefix>` is optional. Credentials never go in the URI: a value containing `@` or `?` fails
to load, naming the accepted forms. A rejected value is never echoed back (it may be a pasted
Expand Down
7 changes: 4 additions & 3 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,9 +124,10 @@ EVALSHIFT_CAPTURE=1 python your_agent.py # writes .evalshift/captures/
```

Running on Fargate, Lambda or Kubernetes, where the disk does not outlive the task? Add
`EVALSHIFT_SINK=s3://<bucket>/<prefix>` (or `gs://…` / `az://…`) next to it and put the same
URI under `captures: {store: …}` in `evalshift.yaml`; step 6 then fetches from the bucket
first. See [`captures`](configuration.md#captures).
`EVALSHIFT_CAPTURE_STORE=s3://<bucket>/<prefix>` (or `gs://…` / `az://…`; SDK 0.6.0+, or
`EVALSHIFT_SINK` on SDK 0.5.0) next to it and put the same URI under `captures: {store: …}` in
`evalshift.yaml`; step 6 then fetches from the bucket first. See
[`captures`](configuration.md#captures).

Captures are off unless `EVALSHIFT_CAPTURE=1` is set, so the decorator can
stay in production code. If the agent calls OpenAI, Anthropic or Google GenAI
Expand Down
17 changes: 9 additions & 8 deletions docs/sdk.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ dense LLM reference at <https://www.evalshift.dev/sdk-llms-full.txt>.

```
your agent (evalshift-sdk) → .evalshift/captures/<suite>/cap_<hex>.json
→ or, EVALSHIFT_SINK=s3://… : <prefix>/captures/<suite>/cap_<hex>.json
→ or, EVALSHIFT_CAPTURE_STORE=s3://… : <prefix>/captures/<suite>/cap_<hex>.json
→ evalshift capture sync (fetches from captures.store first)
→ .evalshift/suites/<suite>/golden.jsonl + evalshift.yaml
```
Expand All @@ -30,9 +30,10 @@ your agent (evalshift-sdk) → .evalshift/captures/<suite>/cap_<hex>.json
`evalshift-sdk` alone.
- **Off by default.** Nothing is recorded unless `EVALSHIFT_CAPTURE=1` is set,
so the instrumentation is safe to leave in production code permanently.
- **No network by default.** The SDK writes local files unless `EVALSHIFT_SINK` names a bucket
you own (`s3://`, `gs://`, `az://`, SDK 0.5.0+), and then it writes only there — never to
EvalShift. Python 3.10+, stdlib-only; the cloud clients are optional extras.
- **No network by default.** The SDK writes local files unless `EVALSHIFT_CAPTURE_STORE` names a
bucket you own (`s3://`, `gs://`, `az://`; SDK 0.6.0+, or `EVALSHIFT_SINK` on SDK 0.5.0), and
then it writes only there — never to EvalShift. Python 3.10+, stdlib-only; a bucket's client
library is a separate package you install only if you use one.

## 1. Instrument the agent

Expand Down Expand Up @@ -110,7 +111,7 @@ the SDK, in your agent's process.

```bash
# on a host whose disk does not outlive it (Fargate, Lambda, pods):
EVALSHIFT_CAPTURE=1 EVALSHIFT_SINK=s3://acme-evals/support-agent python your_agent.py
EVALSHIFT_CAPTURE=1 EVALSHIFT_CAPTURE_STORE=s3://acme-evals/support-agent python your_agent.py
```

The SDK ships each capture (and its toolset sidecar) to the bucket on a background thread,
Expand All @@ -121,9 +122,9 @@ lifecycle rule; `EVALSHIFT_MAX_CAPTURES` applies to local disk only.

Run the CLI from the directory holding `.evalshift/` (or set `EVALSHIFT_DIR`).
If the captures live in a bucket, name it once in `evalshift.yaml`
(`captures: {store: s3://acme-evals/support-agent}`) and install `evalshift[s3]` (or `[gcs]` /
`[azure]`); `list` and `sync` then fetch new captures first. `evalshift capture fetch` does
only that step.
(`captures: {store: s3://acme-evals/support-agent}`) and install the provider's client library
(`pip install boto3`, `google-cloud-storage` or `azure-storage-blob azure-identity`); `list` and
`sync` then fetch new captures first. `evalshift capture fetch` does only that step.

```bash
evalshift capture list # what was recorded (--json for machine output)
Expand Down
Loading
Loading