feat: judge and semantic on by default; name the missing evaluator key

[babaliauskas]

Summary

llm_judge and semantic are the evaluators that say whether the new model's prose is as good as the old one's. A fresh project got the least out of them, for three reasons:

1. semantic shipped commented out for Anthropic and DeepSeek, which have no embeddings endpoint. The YAML comment was the only explanation.
2. A missing judge or embedding key failed silently. compare checked keys only for the two arms being compared. A keyless judge raised on every pair; being advisory, nothing gated on it and nothing was printed, so the user paid for retries and never found out.
3. The "promote" advice was generic. Every all-advisory run ended inconclusive with the same "Set blocking: true on at least one trusted evaluator…" line, with no evaluator named and no "is my suite big enough yet".

This PR fixes all three:

• init turns everything on. semantic is active for every provider. Anthropic and DeepSeek borrow an embedder: OpenAI's if OPENAI_API_KEY is set, else Gemini's if GEMINI_API_KEY/GOOGLE_API_KEY is set, else OpenAI's anyway. Next steps name a missing borrowed key, and init --ci wires it as a second secret.
• Keys are checked before the first call. compare and run check every judge and embedding model up front, and evaluate applies the same rule:
  • Advisory, no key: ⚠ <label> skipped: no API key for <model> — export <VAR> to enable it. The run continues without it, making no calls and no retries.
  • Blocking, no key: exit 1 with the env var named.
  • Every evaluator would be skipped: exit 1 before any spend.
  • semantic forced blocking by a blocking tool_arguments entry (one using the semantic strategy): it counts as a gate, and the message names that entry.
• Skips are recorded. state.json gains skipped_evaluators. Each skip becomes a line in the verdict's recommendations, which reach the terminal, the HTML report, migration_decision.json and the hosted bundle. Local and hosted output match when a migration_policy is configured.
• Promotion advice uses the run's own numbers. When nothing gates, an advisory judge is compared against 20 pairs per prompt (MIN_N_RELIABLE), and the smallest prompt decides:
  • At or above 20: "The equivalence judge scored at least 24 pairs on every prompt — enough to gate…".
  • Below 20: "…summarize has 8. Collect more examples…".
  • A judge that measured nothing is not told to collect more.
• doctor gains an evaluator keys row: fail for blocking, warn for advisory. compare's doctor stage ignores this row because its own check is scoped to the suite being run, so a keyless judge in suite B never fails compare --suite-name A.
• Smaller changes:
  • One shared missing_api_keys helper replaces three copies; an empty-string env var counts as unset.
  • Bare text-embedding-* ids resolve to OpenAI, so the config default's key is checked.
  • Missing-key errors are titled "Missing API key", not "Invalid config".

Also in this branch: the design spec and implementation plan (docs/superpowers/specs/2026-10-09-evaluators-on-by-default-design.md, docs/superpowers/plans/2026-10-09-evaluators-on-by-default.md), plus DOCS.md, llms-full.txt, docs/configuration.md, docs/faq.md, docs/getting-started.md, docs/github-action.md and the CHANGELOG under Unreleased (Added / Changed / Fixed).

Upgrade impact

semantic/llm_judge default to blocking: true in the library. A hand-written config that doesn't say blocking: false and whose judge or embedder has no key used to finish as inconclusive (exit 0, passing --policy-gate). Now compare, run, evaluate and doctor exit 1 before any call. The CHANGELOG ### Changed entry and the DOCS.md upgrade paragraph say so. I'd ship this as a minor bump, not a patch.

Known, deliberately left

• When several tool_arguments entries force semantic blocking, only the first is named. A rerun names the next.
• When semantic is blocking by its own flag and forced, the hint suggests only blocking: false. Exporting the key fixes it either way.
• doctor doesn't detect the "every evaluator would be skipped" case; it warns and exits 0, while compare/run exit 1.
• Possible overlap with fix: doctor no longer fails CI over a missing store library; messages name the package #44: both touch doctor.py, compare.py's doctor stage and the CHANGELOG Unreleased section. Whichever merges second will need a small rebase.

Test plan

• pytest: 2833 passed, coverage 94.84% (make ci)
• ruff check, ruff format --check clean
• mypy --strict src/evalshift_cli clean
• pre-commit run --all-files clean
• CHANGELOG.md updated under ## [Unreleased]
• Docs updated (DOCS.md, llms-full.txt, docs/ pages)

Each of the 8 tasks was test-driven and reviewed separately. The whole branch then had a final review that probed an empty CI secret, an upgraded hand-written config, a re-run of evaluate after exporting the key (stale skips are cleared), and multi-suite compare. Its findings were fixed and re-reviewed.

🤖 Generated with Claude Code