diff --git a/CHANGELOG.md b/CHANGELOG.md index 0527476..9f95eab 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Fixed + +- A suspended account is named as such. When the server refuses a token with + `account_suspended`, the CLI prints its message — "Your account has been suspended. + Contact support@evalshift.dev." `evalshift login` with a stored token for that account + exits 1 with it instead of calling the token no longer valid and starting a browser + sign-in, and push auto-create shows it without the token-access hint. Service account + keys are not affected by a user's suspension. + ## [1.3.0] - 2026-10-09 ### Added diff --git a/DOCS.md b/DOCS.md index d1ec88e..bd58f90 100644 --- a/DOCS.md +++ b/DOCS.md @@ -794,7 +794,7 @@ evalshift logout ``` - Credentials live in `~/.evalshift/credentials` (owner-only permissions). Precedence: CLI flags (`--host`/`--token`) > env (`EVALSHIFT_HOST`/`EVALSHIFT_TOKEN`) > credentials file. `--no-browser` prints the approval URL for remote shells. -- Re-running `login` (without `--token`) while the stored token for that host still works reuses it — no new token is minted — and prints `already logged in as `. To switch accounts, run `evalshift logout` first. +- Re-running `login` (without `--token`) while the stored token for that host still works reuses it — no new token is minted — and prints `already logged in as `. To switch accounts, run `evalshift logout` first. A suspended account gets the server's message instead (`Your account has been suspended. Contact support@evalshift.dev.`, exit 1) — signing in again cannot fix it, and a suspended account cannot approve a fresh browser login, which then waits until `--timeout`. Service account keys in the org keep working. - **`login` issues a personal token — don't use one in CI.** A personal token belongs to you and stops working when your membership does, which is correct on a workstation and fatal in a pipeline. For CI, mint a **service account key** in the web app (Settings → API tokens → Service accounts), scope it to the permissions the job needs, and pass it as `EVALSHIFT_TOKEN` from an encrypted secret instead of running `login` on the runner. Service accounts are org-owned, so the key survives the person who created it leaving. - Projects are `org/project` slugs — from `--project`, or the `project:` key in config. Missing projects are auto-created when permissions allow (`--no-create-project` disables; project-scoped tokens can't auto-create). - `evalshift bundle ` builds the upload artefact without uploading; `push --bundle ` uploads a prebuilt one. `run_bundle.json.gz` carries the manifest, per-example rows (inputs, both outputs, per-evaluator scores and the cost/latency deltas), each example's `traces` — the replay's own tool-call traces, one stream per model side with the ordered tool calls (names, arguments, call ids), round markers, any final text and refusal messages (no `model_call` events and no tool results; a stream over 256 KB keeps its leading events and is flagged `truncated`; imported agent traces from `traces import` stay local and are not uploaded) — the aggregate, `analysis`, the policy `decision` (whose `policy` field is the resolved `migration_policy` this run's verdict was computed under, or `null` when none is configured — see [Migration policy and CI gating](#migration-policy-and-ci-gating)), a run-level `economics` rollup (per-role calls, tokens, cost, latency), `methodology_notes`, the [insights](#run-insights) narrative, the evaluator config and the dataset snapshot. **`report.html` is not uploaded** — it is still written to the run directory for local viewing, and the hosted app renders the run from the data instead. Bundle bytes are deterministic: the same run always compresses identically. Pushes are idempotent on run id. On GitHub Actions, git metadata (`GITHUB_SHA`, branch refs) is baked into the bundle so the server can pair PR runs with base-branch baselines; elsewhere it comes from git, and `bundle` fails with `could not determine a valid 40-character git SHA; run inside git or set GITHUB_SHA` outside a git checkout. `push ` builds a bundle only when `run_bundle.json.gz` is missing and otherwise uploads the existing one as-is. diff --git a/docs/hosted.md b/docs/hosted.md index 0a839a5..1509271 100644 --- a/docs/hosted.md +++ b/docs/hosted.md @@ -25,6 +25,12 @@ Re-running `login` while the stored token for that host still works reuses it instead of minting a new one, and prints `already logged in as `. To switch accounts, run `evalshift logout` first, then `evalshift login`. +If your account has been suspended, `login` with its stored token prints the +server's message — `Your account has been suspended. Contact support@evalshift.dev.` — +and exits 1 instead of starting a new browser sign-in. A suspended account cannot +approve a CLI login either, so a fresh `login` on a machine with no stored token +waits until `--timeout` runs out. See [Troubleshooting](#troubleshooting). + You can still paste an existing hosted API token manually: ```bash @@ -373,5 +379,6 @@ credential file locally and repository secrets in CI. | `could not determine a valid 40-character git SHA` | `bundle` (or `push ` building a bundle) ran outside a git checkout, or in one with no commits, and `GITHUB_SHA` is unset. | Run inside a git repository with at least one commit, or set `GITHUB_SHA`. | | `project was not found` | The project does not exist and auto-create is disabled or not allowed. | Ask an owner to create it, use an org-scoped owner token, or enable auto-create. | | `cannot auto-create at ` | The message names the host it talked to and the server's status. Most often the host is not the one you meant: with no `--host` and no `EVALSHIFT_HOST`, an unset credentials file falls back to `https://api.evalshift.dev`, where your org does not exist. | Run `evalshift whoami` and check the host it prints. If it is wrong, `evalshift login --host `. If the host is right and the status is 403, the token lacks org access — see [Project auto-create](#project-auto-create). | +| `Your account has been suspended` | An EvalShift admin suspended the account this token belongs to. Its browser sessions and personal tokens are refused; service-account keys in your org keep working. | Contact support@evalshift.dev. Running `evalshift login` again does not help. | | `this run needs a paid plan` | The org's plan does not cover this push, its 30-day trial ended without a subscription, or the subscription has stopped paying. | Open the upgrade URL printed with the message; for the monthly quota you can also wait for the reset and push the same run id again. See [Plan limits](#plan-limits). | | `this run carries no migration policy` warning | No `migration_policy` is configured in `evalshift.yaml`, so the bundle has no `decision.policy`. | Add `migration_policy` to `evalshift.yaml` (see [Configuration](configuration.md#migration_policy)). Until then the gate has only whatever old web-app policy the project still has; with none, it reports `inconclusive` and never blocks the pull request unless the GitHub Action sets `require-policy: true`. | diff --git a/llms-full.txt b/llms-full.txt index 1fc511e..aa377b5 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -220,7 +220,8 @@ evalshift login [--token es_...] [--host URL] [--no-browser] [--timeout SECS=900 Device-code browser flow, or --token (verified via GET /me). Credentials stored at ~/.evalshift/credentials (owner-only perms). Without --token, a still-valid stored token for the host is reused (no new token minted; "already logged in as "); a 401/403 falls - through to the browser flow. To switch accounts: `evalshift logout`, then `login`. + through to the browser flow, except a suspended account (403 account_suspended), which exits 1 + with the server's message. To switch accounts: `evalshift logout`, then `login`. Issues a PERSONAL token: tied to your membership, dies with it. Correct for a workstation, wrong for CI. For CI mint a SERVICE ACCOUNT KEY in the hosted web app (Settings -> API tokens -> Service accounts): org-owned machine identity, never owner-equivalent (role is @@ -1494,5 +1495,10 @@ Hosted 401 -> check evalshift whoami; precedence flags > env > credentials file; start with es_; a key past its rotation grace window authenticates as nobody. Hosted 403 `Permission denied: ` -> the token authenticated but its scopes (or its principal's role) don't cover that permission. Widen the scope or mint a key that holds it. +Hosted 403 "Your account has been suspended" (code account_suspended) -> the user behind a +personal token was suspended; whoami/push print that message and exit 1. login with a stored +token does too, instead of starting a browser sign-in; a fresh login waits until --timeout (a +suspended user cannot approve it). Re-login cannot fix it: contact support@evalshift.dev. +Service account keys are unaffected. Owner-only keys a service account can never hold: project:create (auto-create -> use --no-create-project on a pre-created project). diff --git a/src/evalshift_cli/cli/commands/login.py b/src/evalshift_cli/cli/commands/login.py index 3997c7f..b533a0f 100644 --- a/src/evalshift_cli/cli/commands/login.py +++ b/src/evalshift_cli/cli/commands/login.py @@ -10,7 +10,12 @@ import typer from rich.console import Console -from evalshift_cli.hosted.client import HostedClient, HostedError, HostedHTTPError +from evalshift_cli.hosted.client import ( + HostedAccountSuspendedError, + HostedClient, + HostedError, + HostedHTTPError, +) from evalshift_cli.hosted.credentials import ( CredentialsError, is_insecure_host, @@ -96,8 +101,8 @@ def _reuse_stored_credentials(*, host: str, console: Console) -> bool: Every browser approval mints a fresh personal token on the server, and the CLI can only hold one at a time, so re-running ``login`` with a working credential would strand the previous token. A rejected token (revoked, or for another account) falls - through to the browser flow; any other failure is reported as-is because the browser - flow would hit the same wall. + through to the browser flow. A suspended account is reported and stops here. Any other + failure is reported as-is because the browser flow would hit the same wall. """ try: stored = load_credentials() @@ -107,6 +112,10 @@ def _reuse_stored_credentials(*, host: str, console: Console) -> bool: return False try: me = HostedClient(host=host, token=stored.token).me() + except HostedAccountSuspendedError as exc: + # A 403 like the rejected-token case below, but a fresh sign-in cannot fix it. + console.print(f"[red]✗[/red] {exc}") + raise typer.Exit(code=1) from exc except HostedHTTPError as exc: if exc.status_code in (401, 403): console.print("[yellow]![/yellow] stored token is no longer valid; signing in again") diff --git a/src/evalshift_cli/hosted/client.py b/src/evalshift_cli/hosted/client.py index 51deb24..2864620 100644 --- a/src/evalshift_cli/hosted/client.py +++ b/src/evalshift_cli/hosted/client.py @@ -37,6 +37,17 @@ def __init__( super().__init__(message) +ACCOUNT_SUSPENDED = "account_suspended" + + +class HostedAccountSuspendedError(HostedHTTPError): + """The account behind the token is suspended (HTTP 403, code ``account_suspended``). + + The message is the server's sentence and names who to contact, so callers print it as-is. + It is not an expired token: signing in again cannot help. + """ + + class HostedClient: """Typed-ish wrapper around the hosted API endpoints used by the CLI.""" @@ -163,7 +174,10 @@ def _request(self, method: str, path: str, **kwargs: Any) -> Any: ) from exc if response.status_code >= 400: code, message, details = _extract_error(response) - raise HostedHTTPError(response.status_code, message, code=code, details=details) + error_type = ( + HostedAccountSuspendedError if code == ACCOUNT_SUSPENDED else HostedHTTPError + ) + raise error_type(response.status_code, message, code=code, details=details) if not response.content: return None return response.json() diff --git a/src/evalshift_cli/hosted/push.py b/src/evalshift_cli/hosted/push.py index 0aa467a..c432d6e 100644 --- a/src/evalshift_cli/hosted/push.py +++ b/src/evalshift_cli/hosted/push.py @@ -25,6 +25,7 @@ validate_bundle, ) from evalshift_cli.hosted.client import ( + HostedAccountSuspendedError, HostedClient, HostedError, HostedHTTPError, @@ -515,6 +516,9 @@ def _auto_create_project(client: HostedClient, *, project_slug: str) -> None: raise PushError(f"invalid project slug {project_slug!r}; expected org/project") from exc try: projects = client.list_projects(org_slug) + except HostedAccountSuspendedError as exc: + # The server's sentence already says what to do; an access hint would mislead. + raise PushError(str(exc)) from exc except HostedHTTPError as exc: raise PushError( f"cannot auto-create {project_slug!r} at {client.host}: " @@ -528,6 +532,9 @@ def _auto_create_project(client: HostedClient, *, project_slug: str) -> None: return try: client.create_project(org_slug, slug=project, name=_name_from_slug(project)) + except HostedAccountSuspendedError as exc: + # The server's sentence already says what to do; an access hint would mislead. + raise PushError(str(exc)) from exc except HostedHTTPError as exc: if exc.status_code == _PAYMENT_REQUIRED: # An expired org is read-only: creating a project is refused with the same 402 a diff --git a/tests/unit/test_hosted_cli.py b/tests/unit/test_hosted_cli.py index 3fccf7b..210cd72 100644 --- a/tests/unit/test_hosted_cli.py +++ b/tests/unit/test_hosted_cli.py @@ -14,7 +14,7 @@ from evalshift_cli.cli.main import app from evalshift_cli.hosted.bundle import BUNDLE_FILENAME, BundleError, build_bundle -from evalshift_cli.hosted.client import HostedNetworkError +from evalshift_cli.hosted.client import HostedAccountSuspendedError, HostedNetworkError from evalshift_cli.hosted.credentials import ( CredentialsError, load_credentials, @@ -37,6 +37,8 @@ runner = CliRunner() +SUSPENDED_MESSAGE = "Your account has been suspended. Contact support@evalshift.dev." + def _read_bundle(path: Path) -> dict[str, Any]: with gzip.open(path, "rt", encoding="utf-8") as fh: @@ -384,6 +386,30 @@ def test_login_reauthenticates_when_stored_token_is_rejected( assert stored.token == "es_device_plaintext" +def test_login_reports_a_suspended_account_instead_of_signing_in_again( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + credentials_path = tmp_path / "credentials" + monkeypatch.setenv("EVALSHIFT_CREDENTIALS_PATH", str(credentials_path)) + save_credentials("https://api.evalshift.test", "es_suspended", path=credentials_path) + fake_client, calls = _device_flow_client( + me_for_stored=HostedAccountSuspendedError(403, SUSPENDED_MESSAGE, code="account_suspended"), + ) + monkeypatch.setattr("evalshift_cli.cli.commands.login.HostedClient", fake_client) + monkeypatch.setattr( + "evalshift_cli.cli.commands.login.webbrowser.open", + lambda _url: pytest.fail("a suspended account must not start a browser sign-in"), + ) + + result = runner.invoke(app, ["login", "--host", "https://api.evalshift.test"]) + + assert result.exit_code == 1 + assert calls == [] + assert SUSPENDED_MESSAGE in result.output + assert "no longer valid" not in result.output + + def test_login_ignores_stored_credentials_for_a_different_host( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, @@ -1047,6 +1073,28 @@ def test_auto_create_failure_names_the_host_and_the_servers_answer( assert "forbidden: org not visible to this token" in text +def test_auto_create_repeats_a_suspension_without_the_permission_hint( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + bundle_path = _build_bundle_for_push(tmp_path, monkeypatch) + fake = _fake_client( + responses=[{"raise_404": True}], + list_projects_error=HostedAccountSuspendedError( + 403, SUSPENDED_MESSAGE, code="account_suspended" + ), + host="http://localhost:8080", + ) + monkeypatch.setenv("EVALSHIFT_HOST", "http://localhost:8080") + monkeypatch.setenv("EVALSHIFT_TOKEN", "es_secret") + monkeypatch.setattr("evalshift_cli.hosted.push.HostedClient", lambda **_: fake) + + with pytest.raises(PushError) as excinfo: + _push(bundle_path, tmp_path, create_project=True) + + assert str(excinfo.value) == SUSPENDED_MESSAGE + + def test_create_project_failure_names_the_host_and_the_servers_answer( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, diff --git a/tests/unit/test_hosted_client.py b/tests/unit/test_hosted_client.py index 6bbf919..7315983 100644 --- a/tests/unit/test_hosted_client.py +++ b/tests/unit/test_hosted_client.py @@ -9,7 +9,7 @@ import httpx import pytest -from evalshift_cli.hosted.client import HostedClient, HostedHTTPError +from evalshift_cli.hosted.client import HostedAccountSuspendedError, HostedClient, HostedHTTPError from tests.conftest import FakeHostedClient @@ -345,3 +345,41 @@ def test_fake_hosted_client_mirrors_hosted_client() -> None: assert inspect.signature(getattr(FakeHostedClient, name)) == inspect.signature(real), ( f"FakeHostedClient.{name}() no longer matches HostedClient.{name}()" ) + + +SUSPENDED_MESSAGE = "Your account has been suspended. Contact support@evalshift.dev." + + +def test_account_suspended_maps_to_its_own_error(monkeypatch: pytest.MonkeyPatch) -> None: + client = _client_with_response( + monkeypatch, + httpx.Response( + 403, + json={ + "error": { + "code": "account_suspended", + "message": SUSPENDED_MESSAGE, + "details": {"support_email": "support@evalshift.dev"}, + } + }, + ), + ) + + with pytest.raises(HostedAccountSuspendedError) as excinfo: + client.me() + + assert isinstance(excinfo.value, HostedHTTPError) + assert excinfo.value.status_code == 403 + assert str(excinfo.value) == SUSPENDED_MESSAGE + + +def test_other_403s_stay_plain_http_errors(monkeypatch: pytest.MonkeyPatch) -> None: + client = _client_with_response( + monkeypatch, + httpx.Response(403, json={"error": {"code": "forbidden", "message": "nope"}}), + ) + + with pytest.raises(HostedHTTPError) as excinfo: + client.me() + + assert not isinstance(excinfo.value, HostedAccountSuspendedError)