From 52a2aeac1a38866805a33f489741daa68062c8e5 Mon Sep 17 00:00:00 2001 From: Basil Zafar Date: Thu, 15 Oct 2026 17:54:30 +0530 Subject: [PATCH] docs: reconcile published release and local regression workflow --- CHANGELOG.md | 28 ++++++++++++++- README.md | 10 ++++-- docs/ARCHITECTURE.md | 12 ++++--- docs/CLI.md | 6 ++++ docs/DECISION_LOG.md | 5 +++ docs/DESIGN.md | 5 ++- docs/GOOD_FIRST_ISSUES.md | 9 +++-- docs/PRODUCTION_READINESS.md | 25 ++++++++++---- docs/PUBLISHING.md | 15 +++++--- examples/openai_support_agent/README.md | 46 ++++++++++++++++++------- 10 files changed, 126 insertions(+), 35 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4e72da8..07b5ddb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,8 +7,33 @@ Published on PyPI as **`mutiny-ai`** (+ [`mutiny-core`](https://pypi.org/project ## [Unreleased] +These entries describe changes after tag `v0.2.0`. PyPI's latest release for +`mutiny-ai`, `mutiny-core`, and `mutiny-openai-agents` is **0.2.0** (checked +2026-10-15); that wheel set does not include the post-tag fixes below. Use a +source checkout for these changes until maintainers publish another release. + +### Added + +- `mutiny --version` / `-V` from installed metadata (#26, PR #52). +- String `contains`, `startswith`, and `endswith` policy constraints (#23, PR #51). +- Clear PASS/FAIL/SKIPPED replay summary (#39, PR #53). + ### Fixed +- Coerce numeric-looking string tool arguments for `gt`/`gte`/`lt`/`lte` + constraints and numeric boundary fitness (#42, commit `bd6a37b`, PR #50). + Currency-formatted numeric arguments are also recognized (PR #83). These + behaviors are git-only relative to PyPI 0.2.0. +- Pass Hosted token and API URL through the compose web service (commit + `75dfbdd`, PR #48). +- Preserve matched condition evidence (PR #84), successive saved regressions + (PR #89), and nonzero exit status for unexpected skipped replay cases (PR #88). +- Bound adapter steps and campaign wall time (PR #96), preserve correct agent + imports across project loads (PR #119), and free the ingest execution slot + after interruption (PR #98). +- Require explicit opt-in for remote Hosted uploads (PR #94), reject mismatched + ingest hashes (PR #93), and fix SSE/SQLite concurrency gaps (PR #92). + - Use an unversioned workspace root and dependency groups (`uv sync --group dev`, replacing `--extra dev`) instead of advertising a separate product version or duplicate CLI entrypoint. Optional LLM dependencies use `--group llm`. @@ -91,7 +116,8 @@ Published on PyPI as **`mutiny-ai`** (+ [`mutiny-core`](https://pypi.org/project ### Notes -- PyPI may still show **`0.1.0`** until this version is published — do not treat editable/`uv sync` success as PyPI readiness. +- Published on PyPI on 2026-09-10. Source changes after `v0.2.0` are listed under + Unreleased; editable/`uv sync` success does not prove parity with published wheels. - Hosted UI/API remain optional secondary surfaces; primary path needs no Hosted process. ## [0.1.0] — 2026-08-09 diff --git a/README.md b/README.md index 909cece..8166e92 100644 --- a/README.md +++ b/README.md @@ -493,7 +493,13 @@ Full graph: [github.com/CodewithJha/mutiny/graphs/contributors](https://github.c ## FAQ **Is Mutiny on PyPI?** -Yes — install with `pip install mutiny-ai` ([PyPI](https://pypi.org/project/mutiny-ai/)). CLI command is **`mutiny`**. Do not use bare `pip install mutiny` or `mutiny-sdk` — those are other projects. Until `0.2.0` is uploaded, the index may still serve `0.1.0`; Local CLI does not require Hosted or a monorepo checkout. +Yes. The latest published release is **0.2.0** (verified 2026-10-15). Install with +`pip install mutiny-ai` ([PyPI](https://pypi.org/project/mutiny-ai/)); the CLI command +is **`mutiny`**. Do not use bare `pip install mutiny` or `mutiny-sdk`, which are +other projects. Local CLI needs neither Hosted nor a monorepo checkout. +Current git includes fixes absent from the 0.2.0 wheels, including numeric-string +policy coercion (#42); see [Unreleased](./CHANGELOG.md#unreleased) before relying +on source behavior in a PyPI installation. **How do I run on Windows?** Create a venv, then activate with PowerShell `.\.venv\Scripts\Activate.ps1` (or cmd `activate.bat`). Prefer WSL for contributor/`uv` workflows. Short FAQ: [SUPPORT.md § Windows](./SUPPORT.md#windows). @@ -525,7 +531,7 @@ No. Authorized testing only — local / in-process / localhost. Not an open-inte - **Policy expressiveness** is deliberately narrow (deterministic operators on tool calls) — not NL policies judged by an LLM. - **Hosted** is optional and secondary: observe-only for customer runs; non-loopback requires Bearer auth; in-process rate limits when auth is configured; multi-tenant cloud / SSO are not current scope. - Sample and reference demo agents use **mock tools**; they are not production payment or email systems. -- See [CHANGELOG.md](./CHANGELOG.md) for `0.2.0` Local CLI RC notes and what is explicitly out of scope. +- See [CHANGELOG.md](./CHANGELOG.md) for published `0.2.0` notes, post-tag source changes, and what is explicitly out of scope. Honest scope detail: [`docs/ROADMAP.md`](./docs/ROADMAP.md) § Current scope. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index b42ce21..3987758 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -135,19 +135,23 @@ Mutiny (behavioral fuzz-testing engine) Persistence is **not** Core’s job. The API or CLI persists what Core returns. -### CLI / init surfaces (intended package ownership — docs; implement per IMPLEMENTATION_PLAN) +### CLI surfaces (`packages/mutiny_cli`, published as `mutiny-ai`) -**Owns (planned)** +**Owns (implemented)** - `mutiny init` — generate `.mutiny/adapter.py`, `policy.yaml`, `mutiny.yaml` in the customer project - `mutiny run` — load adapter + policy + config; drive **local** Core campaign by default; write regressions; `--hosted` = local + Hosted ingest sync (M-PR8C) -- `mutiny test` — regression replay (P1) +- `mutiny test` — replay saved regressions, report PASS/FAIL/SKIPPED and return an exit status +- `mutiny --version` — report installed CLI distribution metadata +- `mutiny db backup` / `restore` — Hosted SQLite operator commands requiring + the source/workspace `mutiny-api` package; not included by `pip install mutiny-ai` **Must not contain** - Reimplemented policy evaluation, fitness, or mutation logic -Placement may be `integrations/cli` or a published `mutiny` package entrypoint; Core remains the engine. +The `mutiny` console entrypoint belongs to the `mutiny-ai` distribution. Core +remains the engine; optional Hosted sync uploads redacted local results. ### OpenAI Agents SDK adapter (Adapter #1) diff --git a/docs/CLI.md b/docs/CLI.md index 65dc997..cd2a6c0 100644 --- a/docs/CLI.md +++ b/docs/CLI.md @@ -7,6 +7,12 @@ by `pip install mutiny-ai`; local `init`, `run`, and `test` do not require it. Console script from the **`mutiny-ai`** PyPI package (`pip install mutiny-ai`). Source of truth for flags: `mutiny --help` and `mutiny --help`. +The usage snapshot below reflects current main (2026-10-15), including `db` +and `--version`. Some flags and summary improvements landed after PyPI 0.2.0; +see [Unreleased](../CHANGELOG.md#unreleased). For an offline regression that +starts with an expected FAIL and passes after enabling the sample's existing +enforcement switch, follow the [sample walkthrough](../examples/openai_support_agent/README.md#local-fail-to-pass-walkthrough-recommended). + ```text usage: mutiny [-h] [--version] {db,init,run,test} ... diff --git a/docs/DECISION_LOG.md b/docs/DECISION_LOG.md index 0d9dfeb..a91aa6f 100644 --- a/docs/DECISION_LOG.md +++ b/docs/DECISION_LOG.md @@ -208,6 +208,11 @@ Add new ADRs at the bottom. Do not rewrite history; supersede with a new ADR. ## ADR-015 — Inngest-inspired Hosted visual language +**Status (2026-10-15): Superseded for current visual direction by root +[DESIGN.md](../DESIGN.md).** The implemented UI uses its purple-black palette, +Rubik UI and IBM Plex Mono data typography. The decision below records the +earlier design; new work follows the root canon and current CSS tokens. + **Problem:** Hosted UI needed a coherent developer-tool aesthetic for Persuade (landing) and Operate (campaign/exploit/policies/tests) without inventing a one-off brand system mid-hackathon. **Decision:** Pin [Inngest](https://www.inngest.com) v1 as visual canon for Mutiny Hosted — charcoal canvas (`#1c1c1c`), frost primary CTAs that hover to salmon (`#fb5536`), uppercase outline display headlines, floating salmon micro-dots, mono ■ code accordion. CircularXX is licensed to Inngest only; Mutiny uses Plus Jakarta Sans + IBM Plex Mono as stand-ins. Documented in root `DESIGN.md` / `PRODUCT.md`. diff --git a/docs/DESIGN.md b/docs/DESIGN.md index 9364aab..2a27042 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -4,4 +4,7 @@ > Canonical copy: [DESIGN.md](../DESIGN.md) at repo root. -High-fidelity [Inngest](https://www.inngest.com) v1 language for Mutiny Hosted: charcoal canvas `#1c1c1c`, frost CTAs → salmon `#fb5536` hover, uppercase outline display, floating salmon dots, ■ mono accordion. +Use the root [DESIGN.md](../DESIGN.md) for all current visual decisions and +`apps/web/src/app/globals.css` for implemented tokens. This file is a pointer, +not a second visual specification. The former Inngest palette is historical; +see the supersession note on [ADR-015](./DECISION_LOG.md#adr-015--inngest-inspired-hosted-visual-language). diff --git a/docs/GOOD_FIRST_ISSUES.md b/docs/GOOD_FIRST_ISSUES.md index 39c592d..29e6cf8 100644 --- a/docs/GOOD_FIRST_ISSUES.md +++ b/docs/GOOD_FIRST_ISSUES.md @@ -12,6 +12,11 @@ Maintainer catalog of **small, realistic** tasks for new contributors. Open GitH ## Open on GitHub (labeled) +Catalog status checked 2026-10-15. Sections below also contain shipped tasks +and future refresh ideas; verify the linked issue before claiming work. +`mutiny --version` (#26, PR #52), replay summary counts (#39, PR #53), and string +policy operators (#23, PR #51) have shipped on main after `v0.2.0`. + **Shipped:** [#6](https://github.com/CodewithJha/mutiny/issues/6) init `--no-hosted` hint (PR [#13](https://github.com/CodewithJha/mutiny/pull/13) by [@kkkhs](https://github.com/kkkhs)) · [#5](https://github.com/CodewithJha/mutiny/issues/5) policy evaluator edge cases (PR [#15](https://github.com/CodewithJha/mutiny/pull/15) by [@Mr-Neutr0n](https://github.com/Mr-Neutr0n)) · [#2](https://github.com/CodewithJha/mutiny/issues/2) CLI flag docs (`docs/CLI.md`) · [#3](https://github.com/CodewithJha/mutiny/issues/3) real screenshots / demo GIF. **Next picks (claim in the issue comments first):** @@ -73,11 +78,11 @@ Maintainer catalog of **small, realistic** tasks for new contributors. Open GitH | # | Title | Difficulty | Acceptance | Affected files | Labels | |---|---|---|---|---|---| | 21 | Strengthen `mutiny init` next-step hint | XS | **Done** — PR [#13](https://github.com/CodewithJha/mutiny/pull/13) / GH [#6](https://github.com/CodewithJha/mutiny/issues/6) (`--no-hosted` + init test asserts) | `packages/mutiny_cli/`, init tests | `cli` | -| 22 | Friendlier `mutiny test` counts | XS | Clear PASS/FAIL/SKIPPED summary line | CLI test command | `cli` | +| 22 | Friendlier `mutiny test` counts | XS | **Shipped on main** (#39, PR #53): PASS/FAIL/SKIPPED summary | CLI test command | `cli` | | 23 | Missing `policy.yaml` warning | S | Actionable path in error | `packages/mutiny_cli/` run | `cli` | | 24 | `dev.sh` npm install hint | XS | Tip already exists; verify on clean tree / harden if missable (GH [#9](https://github.com/CodewithJha/mutiny/issues/9)) | `scripts/dev.sh`, `docs/COLD_START.md` | `cli` `docs` | | 25 | Sample README always `uv run mutiny` | XS | Docs-only alignment | `examples/**/README.md` | `docs` `examples` `cli` | -| 26 | `mutiny --version` | S | Print package version from metadata | `packages/mutiny_cli/` | `cli` | +| 26 | `mutiny --version` | S | **Shipped on main** (#26, PR #52): version from metadata | `packages/mutiny_cli/` | `cli` | ## Examples / policy packs (27–32) diff --git a/docs/PRODUCTION_READINESS.md b/docs/PRODUCTION_READINESS.md index 7af7a25..eba138e 100644 --- a/docs/PRODUCTION_READINESS.md +++ b/docs/PRODUCTION_READINESS.md @@ -11,6 +11,12 @@ **Related:** [ARCHITECTURE.md](./ARCHITECTURE.md) · [SYSTEM_DESIGN.md](./SYSTEM_DESIGN.md) · [DECISION_LOG.md](./DECISION_LOG.md) · [ROADMAP.md](./ROADMAP.md) · [IMPLEMENTATION_PLAN.md](./IMPLEMENTATION_PLAN.md) · [SECURITY.md](../SECURITY.md) +**Release reconciliation (2026-10-15):** The audit above is historical. The latest +published CLI/Core/adapter release is 0.2.0 (2026-09-10), tagged `v0.2.0`. +Current source contains additional Unreleased changes; see [CHANGELOG](../CHANGELOG.md). +Target A evidence below distinguishes implemented CI gates from checks awaiting +a fresh clean-PyPI verification. Target B remains incomplete. + **Conflict rule:** Product intent → PRD. Hard boundaries → ARCHITECTURE. Runtime contracts → SYSTEM_DESIGN. Sequencing → IMPLEMENTATION_PLAN. Decisions → DECISION_LOG. **Ship / no-ship readiness and Hosted vs Local CLI gates → this document.** If an ADR conflicts with readiness findings, propose a superseding ADR — do not silently rewrite history. --- @@ -35,14 +41,14 @@ It is **not** production-ready as a **public or multi-tenant Hosted** service. W |---|---| | Core kernel | `packages/mutiny_core` — policy, campaign, fitness, mutate, minimize, regress; no FastAPI/React/SQL | | Adapter #1 | `packages/mutiny_openai_agents` — `OpenAIAgentsAdapter`, file loader for `.mutiny/adapter.py` | -| CLI | `packages/mutiny_cli` — `mutiny init` / `run` / `test`; PyPI name `mutiny-ai` @ `0.1.0` | +| CLI | `packages/mutiny_cli` — `mutiny init` / `run` / `test`; PyPI `mutiny-ai` @ `0.2.0`; post-tag fixes remain Unreleased | | Sample project | `examples/openai_support_agent/` with offline scripted model path | | Hosted API | `apps/api` — campaigns, SSE, minimize, regressions, projects, policy routes (customer FS/exec removed), **ingest `/api/ingest/v1/*`** | | Hosted UI | `apps/web` — talks HTTP/SSE to API only (ADR-014) | | Demo harness | `apps/demo_agent` — `in_process_demo` still supported by supervisor | | CI | `.github/workflows/ci.yml` — unit + integration + reliability (3.11/3.12), CLI smoke, web build, package build | | Publish | `.github/workflows/publish.yml` + `docs/PUBLISHING.md` | -| Tag | `v0.1.0` | +| Tag | `v0.2.0` (published 2026-09-10; registry checked 2026-10-15) | ### What docs claim vs code (material drift) @@ -526,15 +532,20 @@ Scheduled after Target A gate; do not block 0.2.0. ### Measurable — Production Local CLI (Target A) -- [ ] `pip install mutiny-ai==` on clean venv; `mutiny --help` works -- [ ] Sample: `mutiny run` finds or honestly reports no violation without crashing -- [ ] Regression save + `mutiny test` FAIL→PASS path documented and tested offline -- [ ] No Hosted required for primary path; Hosted explicitly opt-in +- [ ] Fresh clean-PyPI `pip install mutiny-ai==0.2.0` + `mutiny --help` recheck: + publication verified 2026-10-15; this docs pass did not repeat isolated install +- [x] Sample `mutiny run` completes locally and saves a violation (workspace + smoke verified 2026-10-14; isolated wheel path is gated by release CI) +- [x] Regression FAIL to PASS procedure documented in the sample README; + `scripts/verify_release_artifacts.sh` gates exit 1 before the enforcement + switch and exit 0 after it on isolated built wheels (reviewed 2026-10-15) +- [x] No Hosted required; CLI defaults local and Hosted sync is opt-in - [x] Secret redaction tests green - [x] Seeds/mutators policy-general (M-PR6 / ADR-020); refund packs remain demo/harness helpers - [x] CI: unit + offline integration green on PR - [x] README/SECURITY/IMPLEMENTATION_PLAN match code (P1-7 docs reconcile) -- [ ] Version **≥ 0.2.0** tagged with CHANGELOG +- [x] `v0.2.0` tagged with CHANGELOG; all three 0.2.0 distributions published + 2026-09-10 (registry/tag verified 2026-10-15) ### Measurable — Production Hosted (Target B) diff --git a/docs/PUBLISHING.md b/docs/PUBLISHING.md index 9f9ce0a..e68f549 100644 --- a/docs/PUBLISHING.md +++ b/docs/PUBLISHING.md @@ -4,13 +4,13 @@ End users install **`mutiny-ai`** (CLI command stays `mutiny`). That meta/CLI wh | PyPI name | Path | Role | Status | |---|---|---|---| -| [`mutiny-core`](https://pypi.org/project/mutiny-core/) | `packages/mutiny_core` | Policy oracle / campaign kernel | **0.2.0 RC** (git); PyPI may still show `0.1.0` until upload | -| [`mutiny-openai-agents`](https://pypi.org/project/mutiny-openai-agents/) | `packages/mutiny_openai_agents` | Adapter #1 | **0.2.0 RC** (git); PyPI may still show `0.1.0` until upload | -| [`mutiny-ai`](https://pypi.org/project/mutiny-ai/) | `packages/mutiny_cli` | Console script `mutiny` | **0.2.0 RC** (git); PyPI may still show `0.1.0` until upload | +| [`mutiny-core`](https://pypi.org/project/mutiny-core/) | `packages/mutiny_core` | Policy oracle / campaign kernel | **0.2.0 published**, 2026-09-10 | +| [`mutiny-openai-agents`](https://pypi.org/project/mutiny-openai-agents/) | `packages/mutiny_openai_agents` | Adapter #1 | **0.2.0 published**, 2026-09-10 | +| [`mutiny-ai`](https://pypi.org/project/mutiny-ai/) | `packages/mutiny_cli` | Console script `mutiny` | **0.2.0 published**, 2026-09-10 | > **Why not `pip install mutiny`?** The name [`mutiny`](https://pypi.org/project/mutiny/) is already taken (unrelated Revolt API wrapper). [`mutiny-sdk`](https://pypi.org/project/mutiny-sdk/) is also taken. We publish as **`mutiny-ai`**. -Do **not** publish the repo-root `name = "mutiny"` workspace project — it is not a user wheel. +The repo root is an unversioned uv workspace, not a publishable user wheel. End users: `pip install mutiny-ai` then `mutiny init` / `mutiny run` / `mutiny test` (Hosted not required). @@ -18,7 +18,12 @@ End users: `pip install mutiny-ai` then `mutiny init` / `mutiny run` / `mutiny t ## Release contract (required before upload) -Until **`0.2.0`** is uploaded, PyPI **`0.1.0` may lag the git tree**. Editable/`uv sync` success does **not** prove release integrity — checkout imports hide broken wheels. +PyPI's latest version for all three packages is **0.2.0** (verified 2026-10-15). +The `v0.2.0` tag and wheels predate numeric-string policy coercion (#42, +`bd6a37b`) and other fixes listed in [Unreleased](../CHANGELOG.md#unreleased). +The published core wheel's `_as_number` accepts numbers but not strings. Source +behavior requires a source install until a later release is published; do not +overwrite 0.2.0 or infer wheel parity from editable/`uv sync` success. Before every publish: diff --git a/examples/openai_support_agent/README.md b/examples/openai_support_agent/README.md index 4ca7ff8..d7d3a92 100644 --- a/examples/openai_support_agent/README.md +++ b/examples/openai_support_agent/README.md @@ -39,34 +39,54 @@ Offline / CI (default when no key): export MUTINY_SAMPLE_OFFLINE=1 ``` -## Mutiny (recommended) +## Local FAIL to PASS walkthrough (recommended) From the Mutiny repo after `uv sync --group dev`, always prefer `uv run mutiny …` so the workspace CLI is on your PATH. ```bash -# Terminal A — Hosted API + UI (from Mutiny repo root) -./scripts/dev.sh - -# Terminal B — this sample project -cd examples/openai_support_agent -uv run mutiny init # once -uv run mutiny run --hosted # explicit Hosted opt-in +# From this sample directory; no Hosted process or API key required +uv run mutiny init +uv run mutiny run +uv run mutiny test ``` -Then open the printed dashboard URL (`/campaign/`). +The campaign saves a verified violation under `.mutiny/tests/`. The first +`mutiny test` is expected to report **FAIL** for `refund_limit` and exit **1**: +the sample intentionally accepts fake approval codes. This confirms the +regression reproduces; it is not a replay error. + +Apply the sample's existing policy enforcement switch in `.mutiny/adapter.py`: + +```diff +-def create_adapter(*, enforce_refund_policy: bool = False) -> OpenAIAgentsAdapter: ++def create_adapter(*, enforce_refund_policy: bool = True) -> OpenAIAgentsAdapter: +``` -Local-only (default): +Keep the saved regression and replay it: ```bash -uv run mutiny run -uv run mutiny test # after regressions are saved under .mutiny/tests/ +uv run mutiny test ``` +The refund regression now reports **PASS**, with exit **0**. The switch makes the +sample enforce refund limits instead of trusting approval strings. Production +agents need equivalent checks in their own trusted tool implementation. To +repeat the vulnerable demonstration, change the default back to `False`. +The release artifact gate uses this same switch in an isolated sample copy +([script](../../scripts/verify_release_artifacts.sh)). + +## Optional Hosted lineage + +Start `./scripts/dev.sh` from the repository root in another terminal, then run +`uv run mutiny run --hosted` from this sample and open the printed dashboard URL. +Execution stays local; Hosted receives redacted lineage through ingest. Hosted +does not load or execute this project's adapter. + ## Notes - Tools are **mocks** — no payment rails or real email. -- Hosted loads this directory as `project_path` (`.mutiny/adapter.py`). +- Local CLI loads `.mutiny/adapter.py`; Hosted stores `project_path` as opaque metadata. - This folder is the **sample / reference** agent. Backup fixture demos live under [`examples/demo/`](../demo/) (harness, not the product). - Cold start: [`docs/COLD_START.md`](../../docs/COLD_START.md). - Authorized testing only.