babaliauskas · Oct 10, 2026 · Oct 9, 2026 · Oct 9, 2026 · Oct 9, 2026 · Oct 9, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,8 +7,49 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- `evalshift init` writes the `semantic` evaluator active for every provider.
+  Anthropic and DeepSeek have no embeddings endpoint, so the scaffold borrows
+  OpenAI's or Gemini's — whichever key you already have — and `init --ci`
+  wires that key as a second secret.
+- `compare`, `run` and `evaluate` check every judge and embedding model's API
+  key before the first call. An advisory evaluator without one is skipped with
+  a line naming the env var to export; a blocking one stops the run, and so
+  does a config whose every evaluator would be skipped. A `semantic` that is a
+  gate only because a blocking `tool_arguments` entry uses its embedder is
+  reported as such, naming that entry. The skip
+  is recorded in `state.json` (`skipped_evaluators`) and repeated in the
+  verdict's recommendations, the HTML report and the bundle.
+- `doctor` gains an `evaluator keys` row.
+- When nothing gates the verdict, the recommendation now names the advisory
+  judge and says, from its own sample size, whether it is ready for
+  `blocking: true` (20 pairs per prompt) or how many examples it still needs.
+  A judge that measured nothing (every call errored) is said to have measured
+  nothing rather than told to collect more examples.
+
+### Changed
+
+- A `semantic`/`llm_judge` entry without `blocking: false` is a gate (the
+  library default). If its model has no API key, `compare`, `run`, `evaluate`
+  and `doctor` now exit 1 before any call — previously the run completed with
+  an unmeasured axis, landed on `inconclusive` and passed `--policy-gate`, so
+  CI that was green can turn red on upgrade. The bare `text-embedding-3-small`
+  default is checked for the first time. Export the key, or set
+  `blocking: false` to have it skipped with a warning.
+- Docs: the SDK's object-store variable is `EVALSHIFT_CAPTURE_STORE` from SDK 0.6.0, matching
+  `captures.store` here; SDK 0.5.0 reads the same value from `EVALSHIFT_SINK`, which keeps
+  working on newer SDKs too.
+- Every store message names the package to install instead of a pip extra: `pip install boto3`,
+  `google-cloud-storage`, `azure-storage-blob azure-identity` (both Azure packages in one
+  command). The `[s3]` / `[gcs]` / `[azure]` extras still exist but are no longer documented.
+
 ### Fixed
 
+- A judge or embedding model with no API key no longer fails silently on every
+  pair: it is skipped (advisory) or refused (blocking) before any call.
+- The bare `text-embedding-3-small` default now resolves to OpenAI, so its key
+  is checked.
 - `doctor`'s `captures.store` row no longer fails the command — and with it `compare` and
   every GitHub Action run — when the store's client library is not installed; it warns, naming
   the package to install. A run on a committed suite never reads the bucket; `capture fetch`,
@@ -20,15 +61,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   sign-in, and push auto-create shows it without the token-access hint. Service account
   keys are not affected by a user's suspension.
 
-### Changed
-
-- Docs: the SDK's object-store variable is `EVALSHIFT_CAPTURE_STORE` from SDK 0.6.0, matching
-  `captures.store` here; SDK 0.5.0 reads the same value from `EVALSHIFT_SINK`, which keeps
-  working on newer SDKs too.
-- Every store message names the package to install instead of a pip extra: `pip install boto3`,
-  `google-cloud-storage`, `azure-storage-blob azure-identity` (both Azure packages in one
-  command). The `[s3]` / `[gcs]` / `[azure]` extras still exist but are no longer documented.
-
 ## [1.3.0] - 2026-10-09
 
 ### Added

diff --git a/DOCS.md b/DOCS.md
diff --git a/docs/configuration.md b/docs/configuration.md
@@ -269,7 +269,9 @@ Two behaviours to know:
   fresh-`init` state), the verdict is `inconclusive` — unless
   `max_cost_increase` or `max_latency_increase` is breached, which
   still `fail`s: those are computed from the run's calls, not from
-  evaluator records.
+  evaluator records. The recommendations then say whether each advisory
+  judge has enough pairs to promote (see
+  [`evaluators.llm_judge`](#evaluatorsllm_judge)).
 * **The four rate budgets are Wilson-CI-aware.**
   `max_overall_regression_rate`, `min_equivalence_rate`,
   `max_tool_argument_drift` and `max_tool_divergence` are each a
@@ -487,7 +489,7 @@ A single object (not a list).
 
 | Field             | Type   | Default                  | Description |
 | ----------------- | ------ | ------------------------ | ----------- |
-| `embedding_model` | string | `text-embedding-3-small` | LiteLLM-compatible embedding model id. Use a Gemini one (e.g. `gemini/gemini-embedding-001`) if you don't have an OpenAI key. |
+| `embedding_model` | string | `text-embedding-3-small` | LiteLLM-compatible embedding model id. The bare default resolves to OpenAI (`OPENAI_API_KEY`). Use a Gemini one (e.g. `gemini/gemini-embedding-001`) if you don't have an OpenAI key. |
 | `min_similarity`  | float  | `0.9`                    | Cosine similarity (0–1) below which the target is flagged as a semantic regression. Minor rewording/formatting typically scores ~0.98, so the default 0.9 avoids false flags; set to `1.0` to flag any deviation from byte-identical. Also governs whether semantic drift counts toward the [`migration_policy`](#migration_policy) regression/equivalence gates. |
 
 The semantic evaluator scores the **target's similarity to the source**:
@@ -496,6 +498,34 @@ negative `delta` means the target drifted from the source's meaning.
 `blocking` defaults to `true` in the library but `init` writes `false` —
 see [`blocking`](#blocking-every-evaluator) for why.
 
+`init` writes this block active for every provider. Gemini and OpenAI
+projects use their own embedding model. Anthropic and DeepSeek have no
+embeddings endpoint, so their scaffold borrows one, checking the keys you
+already have: `openai/text-embedding-3-small` when `OPENAI_API_KEY` is set,
+else `gemini/gemini-embedding-001` when `GEMINI_API_KEY` or `GOOGLE_API_KEY`
+is, else `openai/text-embedding-3-small` anyway. A comment above the block
+names the key it needs, and `init --ci` wires that key into the workflow as
+a second, optional secret.
+
+**No API key for `embedding_model`.** `compare` and `run` check it before
+the first model call. An advisory block (`blocking: false`, as scaffolded)
+is skipped — no embedding calls — with
+`⚠ semantic skipped: no API key for openai/text-embedding-3-small — export OPENAI_API_KEY to enable it.`;
+the skip is recorded in `state.json` (`skipped_evaluators`) and repeated in
+the verdict's recommendations. A blocking one stops the run with exit 1
+instead. So does a config whose every evaluator would be skipped — nothing
+would score the run, so it is refused before any call rather than after the
+arms are paid for. `evaluate` applies the same rule.
+
+**Interaction with `tool_arguments`.** [`tool_arguments`](#evaluatorstool_arguments)
+borrows this block's embedder, so skipping it changes how arguments score:
+
+| `tool_arguments` uses | Without the embedder | A keyless `semantic` is treated as |
+| --- | --- | --- |
+| the `semantic` strategy (in `strategies` or as `default_strategy`) on a **blocking** entry | that gate would fall back to `exact` | **blocking** — the run stops, naming the entry when this block says `blocking: false` (``semantic is blocking because tool_arguments `args` uses the semantic strategy — export OPENAI_API_KEY or change that strategy.``) |
+| the `semantic` strategy on an advisory entry | `exact` | this block's own `blocking`; the warning says so |
+| `auto` (the default) | free text graded by `difflib` ratio | this block's own `blocking`; the warning says so |
+
 ### `evaluators.tool_selection`
 
 A list. Each entry has:
@@ -547,7 +577,10 @@ deprecated, and a config still carrying one fails validation.
 `strategies` keys are **field names matched across every tool**, so pick names
 that mean the same thing throughout your toolset. The `semantic` strategy needs
 a configured [`evaluators.semantic`](#evaluatorssemantic) to borrow an embedding
-model (and its cache) from; without one it degrades to `exact`.
+model (and its cache) from; without one it degrades to `exact`. A `semantic`
+block whose model has no API key is skipped when advisory — unless a blocking
+entry here uses the `semantic` strategy, in which case the missing key stops
+the run — see [`evaluators.semantic`](#evaluatorssemantic).
 
 #### The `auto` strategy ladder
 
@@ -803,6 +836,17 @@ bias) and produces strict-JSON `{"winner": "A"|"B"|"tie", "reason":
 error preserved. `blocking` defaults to `true` in the library but `init`
 writes `false` — see [`blocking`](#blocking-every-evaluator) for why.
 
+**When to promote it.** While nothing gates the verdict (the fresh-`init`
+state), the recommendations name each advisory judge and read its readiness
+from its own smallest per-prompt `n`: at 20 pairs on every prompt it says
+the judge is ready for `blocking: true`; below, it names the thinnest prompt
+and its count. Nothing is printed once any evaluator gates.
+
+**No API key for `judge_model`.** Checked before the first model call, as
+for [`semantic`](#evaluatorssemantic): an advisory judge is skipped with a
+`⚠ llm_judge.<criterion_name> skipped: …` line naming the env var to export,
+and a blocking one stops `compare` / `run` with exit 1.
+
 ## Slices
 
 Slices come from the suite; there is nothing to configure. Every distinct

diff --git a/docs/faq.md b/docs/faq.md
@@ -76,9 +76,11 @@ DeepSeek project. Three things differ from other providers:
   The recording holds no DeepSeek reasoning to pass back. DeepSeek requires
   the field on any request with tools, where an empty chain may degrade
   multi-turn answer quality, and ignores it otherwise.
-- **No embeddings.** DeepSeek has no embedding endpoint. The `semantic`
-  evaluator needs an OpenAI or Gemini embedding model and its key, which is
-  why the DeepSeek scaffold ships it commented out.
+- **No embeddings.** DeepSeek has no embedding endpoint, so the `semantic`
+  evaluator borrows an OpenAI or Gemini embedding model and needs that
+  provider's key. The DeepSeek scaffold picks whichever key you already
+  have (OpenAI first) and writes `semantic` active; without the key,
+  `compare` skips it with a line naming the env var to export.
 
 DeepSeek served by another host (self-hosted open weights, or a cloud region
 of your choice) goes through that host's LiteLLM prefix (`hosted_vllm/`,
@@ -175,7 +177,10 @@ Three common causes, all by design:
 
 1. **Every configured evaluator is advisory** (`blocking: false` — the
    fresh `evalshift init` state), so nothing gates quality. Promote
-   evaluators to blocking as your suite grows. The cost and latency
+   evaluators to blocking as your suite grows. The recommendations say
+   when: each advisory judge is named with its smallest per-prompt
+   sample — ready to gate at 20 pairs on every prompt, otherwise which
+   prompt is short and by how much. The cost and latency
    budgets still apply — they read the run's calls, so a breach there
    reports `fail`, not `inconclusive`.
 2. **A rate budget was breached but the 95% Wilson interval can't
@@ -224,6 +229,9 @@ semantic score measures drift from the *source* output, not correctness,
 so a right answer in different words reads as a regression, and on the
 small suites a fresh capture starts with that noise (and judge noise)
 would dominate the verdict. Use a judge criterion for correctness.
+The flag also decides what a missing API key does: an advisory `semantic`
+or judge whose model has no key is skipped with a warning naming the env
+var, while a blocking one stops `compare` before the first call.
 
 The library default is not flipped to match because that would silently
 turn a failing migration into a passing one for every existing config that

diff --git a/docs/getting-started.md b/docs/getting-started.md
@@ -71,10 +71,11 @@ The default `model-upgrade` profile scaffolds a `migration_policy` block
 that powers the verdict in `analyze`, `compare`, and `report`.
 
 This writes a single, minimal, capture-first `evalshift.yaml`: a
-passthrough `replay` prompt, an advisory LLM-judge evaluator and — for the
-Gemini and OpenAI scaffolds — an advisory semantic evaluator (the Anthropic
-and DeepSeek scaffolds write it commented out, since neither provider has an
-embedding endpoint), an empty managed `suites:` block for `capture sync` to fill, and the migration
+passthrough `replay` prompt, an advisory LLM-judge evaluator and an
+advisory semantic evaluator (Anthropic and DeepSeek have no embeddings
+endpoint, so their scaffolds borrow OpenAI's or Gemini's, whichever key you
+already have; without it, `compare` skips `semantic` and names the key to
+export), an empty managed `suites:` block for `capture sync` to fill, and the migration
 policy. `init` refuses to clobber an existing `evalshift.yaml`; pass
 `--force` to overwrite, or `--directory my-eval/` to scaffold into a
 different folder.
@@ -90,8 +91,9 @@ You'll see a short table:
 * Green ✓ — check passes.
 * Yellow ✗ — informational warning (e.g. an unset API key, or no
   `evalshift.yaml` here yet). Doctor still exits 0.
-* Red ✗ — hard failure (e.g. an `evalshift.yaml` that doesn't validate;
-  run `evalshift validate` to see each problem). Doctor exits 1.
+* Red ✗ — hard failure (e.g. an `evalshift.yaml` that doesn't validate —
+  run `evalshift validate` to see each problem — or a blocking judge or
+  embedding model with no API key). Doctor exits 1.
 
 The second row, `evalshift-sdk`, confirms that `import evalshift` in this
 environment is the capture SDK — yellow when it is missing or shadowed by an

diff --git a/docs/github-action.md b/docs/github-action.md
@@ -42,8 +42,13 @@ checklist) with three jobs:
   writer wins that status).
 
 The workflow keys off `${{ secrets.<PROVIDER>_API_KEY }}` for the provider
-chosen at `init` time; add further keys under the eval job's `env:` if your
-judge or embedding models live in another family. Suites must be committed
+chosen at `init` time. An Anthropic or DeepSeek project borrows its
+embedding model from OpenAI or Gemini, so `init --ci` also wires that
+provider's key (`OPENAI_API_KEY` or `GEMINI_API_KEY`) as a second, optional
+secret: left unset it arrives empty, and the run skips the advisory
+`semantic` evaluator with a warning instead of failing. Add further keys
+under the eval job's `env:` if you point a judge or embedding model at
+another family. Suites must be committed
 for CI to see them — keep runtime data ignored and un-ignore just the suites:
 
 ```gitignore
@@ -72,6 +77,8 @@ your project.
   owner-only), so create the project once in the web app and set
   `create-project: false`.
 - Provider API keys used by the source, target, judge, or embedding models.
+  A missing key for a *blocking* judge or embedding model fails the run before
+  any call; an advisory one is skipped with a warning.
 
 Do not hard-code tokens in workflow YAML. Keep them in GitHub encrypted secrets
 — repository or, better for production repos, environment secrets. Never expose