Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 27 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,33 @@ Published on PyPI as **`mutiny-ai`** (+ [`mutiny-core`](https://pypi.org/project

## [Unreleased]

These entries describe changes after tag `v0.2.0`. PyPI's latest release for
`mutiny-ai`, `mutiny-core`, and `mutiny-openai-agents` is **0.2.0** (checked
2026-10-15); that wheel set does not include the post-tag fixes below. Use a
source checkout for these changes until maintainers publish another release.

### Added

- `mutiny --version` / `-V` from installed metadata (#26, PR #52).
- String `contains`, `startswith`, and `endswith` policy constraints (#23, PR #51).
- Clear PASS/FAIL/SKIPPED replay summary (#39, PR #53).

### Fixed

- Coerce numeric-looking string tool arguments for `gt`/`gte`/`lt`/`lte`
constraints and numeric boundary fitness (#42, commit `bd6a37b`, PR #50).
Currency-formatted numeric arguments are also recognized (PR #83). These
behaviors are git-only relative to PyPI 0.2.0.
- Pass Hosted token and API URL through the compose web service (commit
`75dfbdd`, PR #48).
- Preserve matched condition evidence (PR #84), successive saved regressions
(PR #89), and nonzero exit status for unexpected skipped replay cases (PR #88).
- Bound adapter steps and campaign wall time (PR #96), preserve correct agent
imports across project loads (PR #119), and free the ingest execution slot
after interruption (PR #98).
- Require explicit opt-in for remote Hosted uploads (PR #94), reject mismatched
ingest hashes (PR #93), and fix SSE/SQLite concurrency gaps (PR #92).

- Use an unversioned workspace root and dependency groups (`uv sync --group dev`,
replacing `--extra dev`) instead of advertising a separate product version or
duplicate CLI entrypoint. Optional LLM dependencies use `--group llm`.
Expand Down Expand Up @@ -91,7 +116,8 @@ Published on PyPI as **`mutiny-ai`** (+ [`mutiny-core`](https://pypi.org/project

### Notes

- PyPI may still show **`0.1.0`** until this version is published — do not treat editable/`uv sync` success as PyPI readiness.
- Published on PyPI on 2026-09-10. Source changes after `v0.2.0` are listed under
Unreleased; editable/`uv sync` success does not prove parity with published wheels.
- Hosted UI/API remain optional secondary surfaces; primary path needs no Hosted process.

## [0.1.0] — 2026-08-09
Expand Down
10 changes: 8 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -493,7 +493,13 @@ Full graph: [github.com/CodewithJha/mutiny/graphs/contributors](https://github.c
## FAQ

**Is Mutiny on PyPI?**
Yes — install with `pip install mutiny-ai` ([PyPI](https://pypi.org/project/mutiny-ai/)). CLI command is **`mutiny`**. Do not use bare `pip install mutiny` or `mutiny-sdk` — those are other projects. Until `0.2.0` is uploaded, the index may still serve `0.1.0`; Local CLI does not require Hosted or a monorepo checkout.
Yes. The latest published release is **0.2.0** (verified 2026-10-15). Install with
`pip install mutiny-ai` ([PyPI](https://pypi.org/project/mutiny-ai/)); the CLI command
is **`mutiny`**. Do not use bare `pip install mutiny` or `mutiny-sdk`, which are
other projects. Local CLI needs neither Hosted nor a monorepo checkout.
Current git includes fixes absent from the 0.2.0 wheels, including numeric-string
policy coercion (#42); see [Unreleased](./CHANGELOG.md#unreleased) before relying
on source behavior in a PyPI installation.

**How do I run on Windows?**
Create a venv, then activate with PowerShell `.\.venv\Scripts\Activate.ps1` (or cmd `activate.bat`). Prefer WSL for contributor/`uv` workflows. Short FAQ: [SUPPORT.md § Windows](./SUPPORT.md#windows).
Expand Down Expand Up @@ -525,7 +531,7 @@ No. Authorized testing only — local / in-process / localhost. Not an open-inte
- **Policy expressiveness** is deliberately narrow (deterministic operators on tool calls) — not NL policies judged by an LLM.
- **Hosted** is optional and secondary: observe-only for customer runs; non-loopback requires Bearer auth; in-process rate limits when auth is configured; multi-tenant cloud / SSO are not current scope.
- Sample and reference demo agents use **mock tools**; they are not production payment or email systems.
- See [CHANGELOG.md](./CHANGELOG.md) for `0.2.0` Local CLI RC notes and what is explicitly out of scope.
- See [CHANGELOG.md](./CHANGELOG.md) for published `0.2.0` notes, post-tag source changes, and what is explicitly out of scope.

Honest scope detail: [`docs/ROADMAP.md`](./docs/ROADMAP.md) § Current scope.

Expand Down
12 changes: 8 additions & 4 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,19 +135,23 @@ Mutiny (behavioral fuzz-testing engine)

Persistence is **not** Core’s job. The API or CLI persists what Core returns.

### CLI / init surfaces (intended package ownership — docs; implement per IMPLEMENTATION_PLAN)
### CLI surfaces (`packages/mutiny_cli`, published as `mutiny-ai`)

**Owns (planned)**
**Owns (implemented)**

- `mutiny init` — generate `.mutiny/adapter.py`, `policy.yaml`, `mutiny.yaml` in the customer project
- `mutiny run` — load adapter + policy + config; drive **local** Core campaign by default; write regressions; `--hosted` = local + Hosted ingest sync (M-PR8C)
- `mutiny test` — regression replay (P1)
- `mutiny test` — replay saved regressions, report PASS/FAIL/SKIPPED and return an exit status
- `mutiny --version` — report installed CLI distribution metadata
- `mutiny db backup` / `restore` — Hosted SQLite operator commands requiring
the source/workspace `mutiny-api` package; not included by `pip install mutiny-ai`

**Must not contain**

- Reimplemented policy evaluation, fitness, or mutation logic

Placement may be `integrations/cli` or a published `mutiny` package entrypoint; Core remains the engine.
The `mutiny` console entrypoint belongs to the `mutiny-ai` distribution. Core
remains the engine; optional Hosted sync uploads redacted local results.

### OpenAI Agents SDK adapter (Adapter #1)

Expand Down
6 changes: 6 additions & 0 deletions docs/CLI.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,12 @@ by `pip install mutiny-ai`; local `init`, `run`, and `test` do not require it.
Console script from the **`mutiny-ai`** PyPI package (`pip install mutiny-ai`).
Source of truth for flags: `mutiny --help` and `mutiny <cmd> --help`.

The usage snapshot below reflects current main (2026-10-15), including `db`
and `--version`. Some flags and summary improvements landed after PyPI 0.2.0;
see [Unreleased](../CHANGELOG.md#unreleased). For an offline regression that
starts with an expected FAIL and passes after enabling the sample's existing
enforcement switch, follow the [sample walkthrough](../examples/openai_support_agent/README.md#local-fail-to-pass-walkthrough-recommended).

```text
usage: mutiny [-h] [--version] {db,init,run,test} ...

Expand Down
5 changes: 5 additions & 0 deletions docs/DECISION_LOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,11 @@ Add new ADRs at the bottom. Do not rewrite history; supersede with a new ADR.

## ADR-015 — Inngest-inspired Hosted visual language

**Status (2026-10-15): Superseded for current visual direction by root
[DESIGN.md](../DESIGN.md).** The implemented UI uses its purple-black palette,
Rubik UI and IBM Plex Mono data typography. The decision below records the
earlier design; new work follows the root canon and current CSS tokens.

**Problem:** Hosted UI needed a coherent developer-tool aesthetic for Persuade (landing) and Operate (campaign/exploit/policies/tests) without inventing a one-off brand system mid-hackathon.

**Decision:** Pin [Inngest](https://www.inngest.com) v1 as visual canon for Mutiny Hosted — charcoal canvas (`#1c1c1c`), frost primary CTAs that hover to salmon (`#fb5536`), uppercase outline display headlines, floating salmon micro-dots, mono ■ code accordion. CircularXX is licensed to Inngest only; Mutiny uses Plus Jakarta Sans + IBM Plex Mono as stand-ins. Documented in root `DESIGN.md` / `PRODUCT.md`.
Expand Down
5 changes: 4 additions & 1 deletion docs/DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,7 @@

> Canonical copy: [DESIGN.md](../DESIGN.md) at repo root.

High-fidelity [Inngest](https://www.inngest.com) v1 language for Mutiny Hosted: charcoal canvas `#1c1c1c`, frost CTAs → salmon `#fb5536` hover, uppercase outline display, floating salmon dots, ■ mono accordion.
Use the root [DESIGN.md](../DESIGN.md) for all current visual decisions and
`apps/web/src/app/globals.css` for implemented tokens. This file is a pointer,
not a second visual specification. The former Inngest palette is historical;
see the supersession note on [ADR-015](./DECISION_LOG.md#adr-015--inngest-inspired-hosted-visual-language).
9 changes: 7 additions & 2 deletions docs/GOOD_FIRST_ISSUES.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,11 @@ Maintainer catalog of **small, realistic** tasks for new contributors. Open GitH

## Open on GitHub (labeled)

Catalog status checked 2026-10-15. Sections below also contain shipped tasks
and future refresh ideas; verify the linked issue before claiming work.
`mutiny --version` (#26, PR #52), replay summary counts (#39, PR #53), and string
policy operators (#23, PR #51) have shipped on main after `v0.2.0`.

**Shipped:** [#6](https://github.com/CodewithJha/mutiny/issues/6) init `--no-hosted` hint (PR [#13](https://github.com/CodewithJha/mutiny/pull/13) by [@kkkhs](https://github.com/kkkhs)) · [#5](https://github.com/CodewithJha/mutiny/issues/5) policy evaluator edge cases (PR [#15](https://github.com/CodewithJha/mutiny/pull/15) by [@Mr-Neutr0n](https://github.com/Mr-Neutr0n)) · [#2](https://github.com/CodewithJha/mutiny/issues/2) CLI flag docs (`docs/CLI.md`) · [#3](https://github.com/CodewithJha/mutiny/issues/3) real screenshots / demo GIF.

**Next picks (claim in the issue comments first):**
Expand Down Expand Up @@ -73,11 +78,11 @@ Maintainer catalog of **small, realistic** tasks for new contributors. Open GitH
| # | Title | Difficulty | Acceptance | Affected files | Labels |
|---|---|---|---|---|---|
| 21 | Strengthen `mutiny init` next-step hint | XS | **Done** — PR [#13](https://github.com/CodewithJha/mutiny/pull/13) / GH [#6](https://github.com/CodewithJha/mutiny/issues/6) (`--no-hosted` + init test asserts) | `packages/mutiny_cli/`, init tests | `cli` |
| 22 | Friendlier `mutiny test` counts | XS | Clear PASS/FAIL/SKIPPED summary line | CLI test command | `cli` |
| 22 | Friendlier `mutiny test` counts | XS | **Shipped on main** (#39, PR #53): PASS/FAIL/SKIPPED summary | CLI test command | `cli` |
| 23 | Missing `policy.yaml` warning | S | Actionable path in error | `packages/mutiny_cli/` run | `cli` |
| 24 | `dev.sh` npm install hint | XS | Tip already exists; verify on clean tree / harden if missable (GH [#9](https://github.com/CodewithJha/mutiny/issues/9)) | `scripts/dev.sh`, `docs/COLD_START.md` | `cli` `docs` |
| 25 | Sample README always `uv run mutiny` | XS | Docs-only alignment | `examples/**/README.md` | `docs` `examples` `cli` |
| 26 | `mutiny --version` | S | Print package version from metadata | `packages/mutiny_cli/` | `cli` |
| 26 | `mutiny --version` | S | **Shipped on main** (#26, PR #52): version from metadata | `packages/mutiny_cli/` | `cli` |

## Examples / policy packs (27–32)

Expand Down
25 changes: 18 additions & 7 deletions docs/PRODUCTION_READINESS.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,12 @@

**Related:** [ARCHITECTURE.md](./ARCHITECTURE.md) · [SYSTEM_DESIGN.md](./SYSTEM_DESIGN.md) · [DECISION_LOG.md](./DECISION_LOG.md) · [ROADMAP.md](./ROADMAP.md) · [IMPLEMENTATION_PLAN.md](./IMPLEMENTATION_PLAN.md) · [SECURITY.md](../SECURITY.md)

**Release reconciliation (2026-10-15):** The audit above is historical. The latest
published CLI/Core/adapter release is 0.2.0 (2026-09-10), tagged `v0.2.0`.
Current source contains additional Unreleased changes; see [CHANGELOG](../CHANGELOG.md).
Target A evidence below distinguishes implemented CI gates from checks awaiting
a fresh clean-PyPI verification. Target B remains incomplete.

**Conflict rule:** Product intent → PRD. Hard boundaries → ARCHITECTURE. Runtime contracts → SYSTEM_DESIGN. Sequencing → IMPLEMENTATION_PLAN. Decisions → DECISION_LOG. **Ship / no-ship readiness and Hosted vs Local CLI gates → this document.** If an ADR conflicts with readiness findings, propose a superseding ADR — do not silently rewrite history.

---
Expand All @@ -35,14 +41,14 @@ It is **not** production-ready as a **public or multi-tenant Hosted** service. W
|---|---|
| Core kernel | `packages/mutiny_core` — policy, campaign, fitness, mutate, minimize, regress; no FastAPI/React/SQL |
| Adapter #1 | `packages/mutiny_openai_agents` — `OpenAIAgentsAdapter`, file loader for `.mutiny/adapter.py` |
| CLI | `packages/mutiny_cli` — `mutiny init` / `run` / `test`; PyPI name `mutiny-ai` @ `0.1.0` |
| CLI | `packages/mutiny_cli` — `mutiny init` / `run` / `test`; PyPI `mutiny-ai` @ `0.2.0`; post-tag fixes remain Unreleased |
| Sample project | `examples/openai_support_agent/` with offline scripted model path |
| Hosted API | `apps/api` — campaigns, SSE, minimize, regressions, projects, policy routes (customer FS/exec removed), **ingest `/api/ingest/v1/*`** |
| Hosted UI | `apps/web` — talks HTTP/SSE to API only (ADR-014) |
| Demo harness | `apps/demo_agent` — `in_process_demo` still supported by supervisor |
| CI | `.github/workflows/ci.yml` — unit + integration + reliability (3.11/3.12), CLI smoke, web build, package build |
| Publish | `.github/workflows/publish.yml` + `docs/PUBLISHING.md` |
| Tag | `v0.1.0` |
| Tag | `v0.2.0` (published 2026-09-10; registry checked 2026-10-15) |

### What docs claim vs code (material drift)

Expand Down Expand Up @@ -526,15 +532,20 @@ Scheduled after Target A gate; do not block 0.2.0.

### Measurable — Production Local CLI (Target A)

- [ ] `pip install mutiny-ai==<ver>` on clean venv; `mutiny --help` works
- [ ] Sample: `mutiny run` finds or honestly reports no violation without crashing
- [ ] Regression save + `mutiny test` FAIL→PASS path documented and tested offline
- [ ] No Hosted required for primary path; Hosted explicitly opt-in
- [ ] Fresh clean-PyPI `pip install mutiny-ai==0.2.0` + `mutiny --help` recheck:
publication verified 2026-10-15; this docs pass did not repeat isolated install
- [x] Sample `mutiny run` completes locally and saves a violation (workspace
smoke verified 2026-10-14; isolated wheel path is gated by release CI)
- [x] Regression FAIL to PASS procedure documented in the sample README;
`scripts/verify_release_artifacts.sh` gates exit 1 before the enforcement
switch and exit 0 after it on isolated built wheels (reviewed 2026-10-15)
- [x] No Hosted required; CLI defaults local and Hosted sync is opt-in
- [x] Secret redaction tests green
- [x] Seeds/mutators policy-general (M-PR6 / ADR-020); refund packs remain demo/harness helpers
- [x] CI: unit + offline integration green on PR
- [x] README/SECURITY/IMPLEMENTATION_PLAN match code (P1-7 docs reconcile)
- [ ] Version **≥ 0.2.0** tagged with CHANGELOG
- [x] `v0.2.0` tagged with CHANGELOG; all three 0.2.0 distributions published
2026-09-10 (registry/tag verified 2026-10-15)

### Measurable — Production Hosted (Target B)

Expand Down
15 changes: 10 additions & 5 deletions docs/PUBLISHING.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,21 +4,26 @@ End users install **`mutiny-ai`** (CLI command stays `mutiny`). That meta/CLI wh

| PyPI name | Path | Role | Status |
|---|---|---|---|
| [`mutiny-core`](https://pypi.org/project/mutiny-core/) | `packages/mutiny_core` | Policy oracle / campaign kernel | **0.2.0 RC** (git); PyPI may still show `0.1.0` until upload |
| [`mutiny-openai-agents`](https://pypi.org/project/mutiny-openai-agents/) | `packages/mutiny_openai_agents` | Adapter #1 | **0.2.0 RC** (git); PyPI may still show `0.1.0` until upload |
| [`mutiny-ai`](https://pypi.org/project/mutiny-ai/) | `packages/mutiny_cli` | Console script `mutiny` | **0.2.0 RC** (git); PyPI may still show `0.1.0` until upload |
| [`mutiny-core`](https://pypi.org/project/mutiny-core/) | `packages/mutiny_core` | Policy oracle / campaign kernel | **0.2.0 published**, 2026-09-10 |
| [`mutiny-openai-agents`](https://pypi.org/project/mutiny-openai-agents/) | `packages/mutiny_openai_agents` | Adapter #1 | **0.2.0 published**, 2026-09-10 |
| [`mutiny-ai`](https://pypi.org/project/mutiny-ai/) | `packages/mutiny_cli` | Console script `mutiny` | **0.2.0 published**, 2026-09-10 |

> **Why not `pip install mutiny`?** The name [`mutiny`](https://pypi.org/project/mutiny/) is already taken (unrelated Revolt API wrapper). [`mutiny-sdk`](https://pypi.org/project/mutiny-sdk/) is also taken. We publish as **`mutiny-ai`**.

Do **not** publish the repo-root `name = "mutiny"` workspace project — it is not a user wheel.
The repo root is an unversioned uv workspace, not a publishable user wheel.

End users: `pip install mutiny-ai` then `mutiny init` / `mutiny run` / `mutiny test` (Hosted not required).

---

## Release contract (required before upload)

Until **`0.2.0`** is uploaded, PyPI **`0.1.0` may lag the git tree**. Editable/`uv sync` success does **not** prove release integrity — checkout imports hide broken wheels.
PyPI's latest version for all three packages is **0.2.0** (verified 2026-10-15).
The `v0.2.0` tag and wheels predate numeric-string policy coercion (#42,
`bd6a37b`) and other fixes listed in [Unreleased](../CHANGELOG.md#unreleased).
The published core wheel's `_as_number` accepts numbers but not strings. Source
behavior requires a source install until a later release is published; do not
overwrite 0.2.0 or infer wheel parity from editable/`uv sync` success.

Before every publish:

Expand Down
46 changes: 33 additions & 13 deletions examples/openai_support_agent/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,34 +39,54 @@ Offline / CI (default when no key):
export MUTINY_SAMPLE_OFFLINE=1
```

## Mutiny (recommended)
## Local FAIL to PASS walkthrough (recommended)

From the Mutiny repo after `uv sync --group dev`, always prefer `uv run mutiny …`
so the workspace CLI is on your PATH.

```bash
# Terminal A — Hosted API + UI (from Mutiny repo root)
./scripts/dev.sh

# Terminal B — this sample project
cd examples/openai_support_agent
uv run mutiny init # once
uv run mutiny run --hosted # explicit Hosted opt-in
# From this sample directory; no Hosted process or API key required
uv run mutiny init
uv run mutiny run
uv run mutiny test
```

Then open the printed dashboard URL (`/campaign/<id>`).
The campaign saves a verified violation under `.mutiny/tests/`. The first
`mutiny test` is expected to report **FAIL** for `refund_limit` and exit **1**:
the sample intentionally accepts fake approval codes. This confirms the
regression reproduces; it is not a replay error.

Apply the sample's existing policy enforcement switch in `.mutiny/adapter.py`:

```diff
-def create_adapter(*, enforce_refund_policy: bool = False) -> OpenAIAgentsAdapter:
+def create_adapter(*, enforce_refund_policy: bool = True) -> OpenAIAgentsAdapter:
```

Local-only (default):
Keep the saved regression and replay it:

```bash
uv run mutiny run
uv run mutiny test # after regressions are saved under .mutiny/tests/
uv run mutiny test
```

The refund regression now reports **PASS**, with exit **0**. The switch makes the
sample enforce refund limits instead of trusting approval strings. Production
agents need equivalent checks in their own trusted tool implementation. To
repeat the vulnerable demonstration, change the default back to `False`.
The release artifact gate uses this same switch in an isolated sample copy
([script](../../scripts/verify_release_artifacts.sh)).

## Optional Hosted lineage

Start `./scripts/dev.sh` from the repository root in another terminal, then run
`uv run mutiny run --hosted` from this sample and open the printed dashboard URL.
Execution stays local; Hosted receives redacted lineage through ingest. Hosted
does not load or execute this project's adapter.

## Notes

- Tools are **mocks** — no payment rails or real email.
- Hosted loads this directory as `project_path` (`.mutiny/adapter.py`).
- Local CLI loads `.mutiny/adapter.py`; Hosted stores `project_path` as opaque metadata.
- This folder is the **sample / reference** agent. Backup fixture demos live under [`examples/demo/`](../demo/) (harness, not the product).
- Cold start: [`docs/COLD_START.md`](../../docs/COLD_START.md).
- Authorized testing only.
Loading