Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion DOCS.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ EvalShift is four pieces. Each is released and documented independently; each ow
| **GitHub Action** | `evalshift/evalshift-action@v0` | Runs the pipeline on PRs, pushes the run, maintains one PR comment, sets the `evalshift/regression` status. | [docs/github-action.md](docs/github-action.md), [action repo](https://github.com/evalshift/evalshift-action) | <https://www.evalshift.dev/ci-llms-full.txt> |
| **Hosted server** | service — API `https://api.evalshift.dev`, web app `https://evalshift.dev` | Stores pushed run bundles, diffs runs across branches, serves the web app, drives PR comments and gating. | [docs/hosted.md](docs/hosted.md) | covered by the CLI reference (`push`/`bundle` contract) |

Data flow is one-directional: **SDK captures → CLI runs and bundles → server stores and diffs → web app displays.** The SDK and CLI never call each other — the interface is files under `.evalshift/captures/`. The CLI (import `evalshift_cli`) depends on the SDK (import `evalshift`), so one environment holds both.
Data flow is one-directional: **SDK captures → CLI runs and bundles → server stores and diffs → web app displays.** The SDK and CLI never call each other — the interface is the capture layout under `.evalshift/captures/`, on disk or mirrored from a bucket you own (`captures.store`). The CLI (import `evalshift_cli`) depends on the SDK (import `evalshift`), so one environment holds both.

The CLI reference is generated from [llms-full.txt](llms-full.txt) at this repo's root — edit that file when CLI behaviour changes. The SDK and Action references are owned by their own repos; the copies served from `evalshift.dev` are synced from there.

Expand Down
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,9 @@ Four pieces, released and documented independently:
| **GitHub Action** — `evalshift/evalshift-action@v0` | Runs the pipeline on pull requests, pushes the run, posts one PR comment, sets the `evalshift/regression` status. | [docs/github-action.md](docs/github-action.md) |
| **Hosted server** — `api.evalshift.dev`, web app at `evalshift.dev` | Optional. Stores pushed run bundles, diffs them across branches, drives PR comments and gating. | [docs/hosted.md](docs/hosted.md) |

The SDK and the CLI never call each other — the interface is files under
`.evalshift/captures/`, so either works without the other. The CLI (import
The SDK and the CLI never call each other — the interface is the capture layout
under `.evalshift/captures/`, on disk or mirrored from a bucket you own
(`captures.store`), so either works without the other. The CLI (import
package `evalshift_cli`) depends on the SDK (import name `evalshift`), so one
environment holds both: `pip install evalshift` brings the SDK with it, and a
production agent that only records captures installs `evalshift-sdk` alone.
Expand Down
2 changes: 1 addition & 1 deletion src/evalshift_cli/cli/commands/_agents.py
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@
| `evalshift report` | Render the single-file HTML report. |
| `evalshift compare` | Run doctor -> run -> evaluate -> analyze -> report end to end on one suite (formerly `all`). |
| `evalshift inspect` | Inspect a run / artifact. |
| `evalshift capture ...` | `list` / `promote` / `clean` / `diff` / `sync` captures into suites. |
| `evalshift capture ...` | `list` / `fetch` / `promote` / `clean` / `diff` / `sync` captures into suites. |
| `evalshift diff case` | Diff a single case between runs. |
| `evalshift replay case` | Replay a single case. |
| `evalshift traces import` | Import external traces. |
Expand Down
26 changes: 26 additions & 0 deletions tests/unit/test_docs_currency.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@
import pytest

import evalshift_cli
from evalshift_cli.cli.commands._agents import AGENT_INSTRUCTIONS
from evalshift_cli.cli.commands.capture import capture_app
from evalshift_cli.cli.commands.init import PROVIDERS
from evalshift_cli.models.registry import PROVIDER_ENV_VARS

Expand Down Expand Up @@ -227,3 +229,27 @@ def _whitespace_normalized(name: str) -> str:
def test_copy_does_not_describe_the_retired_hosted_plans(term: str) -> None:
offenders = [name for name in _PLAN_COPY_FILES if term in _whitespace_normalized(name)]
assert offenders == [], f"{term!r} still appears in {offenders}"


def _registered_capture_subcommands() -> list[str]:
"""Every subcommand `evalshift capture --help` lists, straight from the Typer group."""
return sorted(str(cmd.name) for cmd in capture_app.registered_commands)


@pytest.mark.parametrize("name", _registered_capture_subcommands())
def test_scaffolded_agent_guide_names_every_capture_subcommand(name: str) -> None:
"""`init` writes EVALSHIFT.md into user repos, and its `evalshift capture ...`
row is the one place a coding agent learns which subcommands exist.

`capture fetch` shipped in 1.3.0 without being added to that row, so every
scaffolded guide told agents the group had five subcommands. The row is
matched by its leading cell so a bare word like `list` elsewhere in the
guide cannot satisfy the check.
"""
row = next(
line
for line in AGENT_INSTRUCTIONS.splitlines()
if line.startswith("| `evalshift capture ...`")
)

assert f"`{name}`" in row, f"EVALSHIFT.md's capture row does not name `{name}`"
Loading