honest-tests is an agent skill + CLI that stops Claude Code, Codex, Cursor and friends from writing fake tests — and proves which of your existing tests are fake by mutating your code and watching them stay green.
npx @mintwork/honest-tests audit . # find tests that can't fail
npx @mintwork/honest-tests mutate src/pricing.ts # prove it: break the code, see who noticesAsk an agent to "add tests" and you get a wall of green checkmarks. Look closer:
it('handles multiple items', () => {
const result = subtotal(items);
expect(result).toBe(result); // compares a value to itself
});
it('applies a percentage discount', () => {
applyDiscount(100, 10); // no assertion at all
});
it('computes the discounted price', () => {
try {
expect(applyDiscount(100, 20)).toBe(70); // WRONG (it's 80) — and still passes
} catch {}
});
it('gives members free shipping', () => {
const qualifies = vi.fn().mockReturnValue(true);
expect(qualifies(10, true)).toBe(true); // tests the mock it just configured
});Every one of these passes. Coverage goes up. Nothing is verified.
$ npx @mintwork/honest-tests audit test/pricing.test.ts
test/pricing.test.ts
16:5 error Compares `result` to itself — this can never fail. tautological-assertion
│ expect(result).toBe(result);
25:1 error Test "applies a percentage discount" has no assertions — it can only fail by crashing. no-assertions
│ it('applies a percentage discount', () => {
39:7 error Assertion is inside a try/catch that swallows errors — when it fails, the catch eats the AssertionError and the test passes. swallowed-assertion
│ expect(applyDiscount(100, 20)).toBe(70); // wrong! but the catch hides it
53:5 error Asserts on what mock `qualifies` returns (configured on line 52) — this verifies the mock, not your code. asserts-on-mock
│ expect(qualifies(10, true)).toBe(true);
... 5 more
7 of 10 tests can't fail (or don't test what they claim).
1 file, 10 tests: 7 errors, 2 warnings. Run with --verbose for fixes.
Then the killer demo — mutation testing, built in. honest-tests flips operators, booleans, constants and return values in your source (on a temp copy), runs your tests, and shows every change nobody noticed:
$ npx @mintwork/honest-tests mutate src/pricing.ts
22 mutants survived — your tests don't actually check these lines:
src/pricing.ts:15 if (percent < 0 || percent > 100) {
✗ condition `percent < 0 || percent > 100` → `false`
✗ `<` → `<=`
✗ `0` → `1`
✗ `||` → `&&`
✗ `>` → `>=`
✗ `100` → `101`
src/pricing.ts:22 return isMember || total >= 50;
✗ `>=` → `>`
✗ `50` → `51`
src/pricing.ts:27 return 0;
✗ return value replaced with `undefined` (no test executes this line)
✗ `0` → `1` (no test executes this line)
...
Mutation score: 15.4% (4 killed, 0 timed out, 22 survived of 26; baseline 0.5s)
it('takes the percentage off the amount', () => {
expect(applyDiscount(200, 25)).toBe(150);
});
it('accepts the 0% and 100% boundaries', () => {
expect(applyDiscount(200, 0)).toBe(200);
expect(applyDiscount(200, 100)).toBe(0);
});
it('rejects percentages outside 0..100', () => {
expect(() => applyDiscount(200, -1)).toThrow(RangeError);
expect(() => applyDiscount(200, 101)).toThrow(RangeError);
});
it('starts free shipping exactly at 50 for non-members', () => {
expect(qualifiesForFreeShipping(49.99, false)).toBe(false);
expect(qualifiesForFreeShipping(50, false)).toBe(true);
});$ npx @mintwork/honest-tests audit test && npx @mintwork/honest-tests mutate src/pricing.ts
✓ 12 tests in 2 files: no fake tests found.
Mutation score: 100% (26 killed, 0 timed out, 0 survived of 26; baseline 0.5s)
✓ Every mutation was caught. These tests actually test something.
Both projects live in fixtures/ — run npm run demo:bad / npm run demo:good yourself
(npm run demo:gif re-renders the GIF above from the real CLI output).
/plugin marketplace add mintwork/honest-tests
/plugin install honest-tests@honest-tests
Or drop just the skill into a repo: npx @mintwork/honest-tests init claude (writes .claude/skills/honest-tests/).
Then ask: "write tests for src/cart.ts" or "audit my tests" — or use the slash commands
/honest-tests:audit [path] and /honest-tests:mutate <file>.
npx @mintwork/honest-tests init codex # appends an honest-tests section to AGENTS.md (idempotent)npx @mintwork/honest-tests init cursor # writes .cursor/rules/honest-tests.mdc (auto-attaches to test files)npx @mintwork/honest-tests audit . --json # machine-readable findings
npx @mintwork/honest-tests audit . --sarif > ht.sarif # SARIF 2.1.0 for GitHub code scanning
npx @mintwork/honest-tests audit . --changed origin/main # only test files this branch changed
npx @mintwork/honest-tests mutate src/x.ts --min-score 80 # fail CI below 80% mutation score--changed <ref> audits the test files changed since the merge base with <ref>, including
uncommitted and untracked files, and reports how many tests the change adds or modifies and how many
of those can't fail (changed in the JSON; findings in changed tests carry change: "added" | "modified").
In CI, check out with enough history for the merge base (actions/checkout with fetch-depth: 0).
--sarif writes SARIF 2.1.0 (findings, plus surviving mutants as surviving-mutant warnings when
mutation runs); upload it with github/codeql-action/upload-sarif, and run it from the repository root
so paths line up. On GitHub, the Action below wraps all of this.
Comment on every pull request with what its tests are worth:
# .github/workflows/honest-tests.yml
name: Honest tests
on: pull_request
permissions:
contents: read
pull-requests: write # create/update the PR comment
jobs:
audit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
with:
fetch-depth: 0 # the audit diffs against the merge base
- uses: mintwork/honest-tests@v0The PR gets one comment, updated on every push:
This PR adds 12 tests — 4 can't fail.
Severity Change Test Finding Location error added applies a discount Assertion is inside a try/catch that swallows errors … swallowed-assertiontest/pricing.test.ts:39
Findings in untouched tests of the same files are listed in a collapsed section; they never fail the step. A PR that changes no test files gets no new comment. The same report goes to the job summary, and the findings that fail the step are also annotated on the diff.
- What it does: runs
audit --changedagainst the PR's base commit (or the previous commit onpush). It builds the CLI from the action's own ref (Node 22+; it sets up Node 24 if the runner has an older one). It only parses test files: it never installs your dependencies or runs your code or tests. - Permissions:
contents: readandpull-requests: writefor the comment. Withsarif-fileandgithub/codeql-action/upload-sarif, addsecurity-events: write. - Forks: pull requests from forks get a read-only token, so the comment is skipped with a warning;
the job summary, annotations and
fail-onstill work. - History: use
fetch-depth: 0. With a shallow checkout the action fetches the missing history itself, which is slower.
| Input | Default | |
|---|---|---|
base |
PR base commit / previous commit | Ref or SHA to compare against |
paths |
. |
Paths to audit, comma- or newline-separated, relative to working-directory |
ignore |
Globs to ignore, e.g. fixtures/** |
|
working-directory |
. |
Directory to run in (a package in a monorepo) |
fail-on |
error |
error, warning or never; only findings in tests the PR adds or modifies count |
comment |
true |
Create or update the PR comment |
sarif-file |
Also write SARIF to this path, for upload-sarif |
|
github-token |
github.token |
Token for the comment |
Outputs: tests-added, tests-modified, cant-fail-added, cant-fail-modified (tests with an
error-level finding), findings, report (path of the JSON report) and sarif-file.
@v0 follows the latest 0.x release; pin @v0.2.0 or a commit SHA for a fixed version. Code
scanning, monorepo and other setups: examples/.
The skill (Agent Skills format) teaches agents one rule — a test is honest only if it fails when the behavior it names is broken — plus two workflows:
- Writing tests: name the bug each test catches, derive expected values independently, mock only
boundaries, cover boundaries and error paths, then self-audit with
audit+mutatebefore saying "done". Hard rules forbid the classic agent moves: weakening assertions to get green, changing the expected value to match output, mocking the unit under test,try/catcharound asserts. - Auditing tests, in phases, like a security audit:
- Detect —
honest-tests audit --jsonplus a manual pass for smells static analysis can't see. - Verify — every finding is opened and marked confirmed / false-positive / accepted; key findings are proven with surviving mutants.
- Report — "N of M tests can't fail", confirmed findings, surviving mutants, next steps.
- Fix — only when asked; re-run and show the before/after mutation score.
- Detect —
References the agent loads on demand: test smell catalog, writing honest tests, audit procedure, mutation testing.
- JavaScript / TypeScript (
.js,.jsx,.ts,.tsx,.mjs,.cjs,.mts,.cts): every rule, plus mutation testing. - Python (pytest and unittest): every rule except three (see Rules); no mutation testing yet.
Needs
python33.9+.
| Runner | Recognized | mutate runs |
|---|---|---|
| vitest, jest | *.test.*, *.spec.*, __tests__/, vi/jest mocks |
npm test -- <related files> |
| Mocha | it/describe/context/specify, chai, this.skip(), done callbacks, test/*.js |
mocha <related files> (nyc/c8 wrappers dropped) |
| node:test | test/t.test subtests, t.assert.*, { skip/todo/only }, t.skip(), t.plan(), mock.fn/t.mock.method, tap/tspl-style t.equal, test/**, test-*.js |
node --test <related files> |
| Bun | bun:test expect, mock(), spyOn(), mock.module, test.if/skipIf/todo |
bun test ./<related files> |
| Deno | Deno.test (all call shapes), t.step, @std/assert, @std/expect, assertSnapshot, *_test.ts |
deno test <related files> (from the deno.json test task, or plain deno test) |
| pytest, unittest | test_*.py, *_test.py, assert, pytest.raises, self.assert*, Mock/patch/mocker/monkeypatch, skip markers |
not yet (audit only) |
| Rule | Severity | Catches |
|---|---|---|
no-assertions |
error | Test body never asserts — passes unless it crashes |
tautological-assertion |
error | expect(x).toBe(x), expect(f(1)).toBe(f(1)), expect(true).toBe(true), toHaveLength(list.length) |
asserts-on-mock |
error | Asserting on the return value of a mock the test configured (vi.fn().mockReturnValue, spyOn, vi.mocked, .mock.results) |
asserts-on-own-literal |
error | Asserting on a literal the test just built: const p = { service: 's3' }; expect(p.service).toBe('s3') — common in runtime "type tests" |
never-failing-assertion |
error | expect(x).toBeTruthy (never called), bare expect(x), un-awaited .resolves/.rejects, typeof x "defined", length >= 0, {} truthy |
swallowed-assertion |
error | Assertion inside try whose catch doesn't rethrow |
catch-only-assertion |
error | Assertions only in catch — passes when nothing throws (warning when the test also asserts outside the catch) |
mocked-unit-under-test |
error | vi.mock('./cart') inside cart.test.ts; spying on and replacing the unit's own exports (warning for partial mocks) |
focused-test |
error | .only, fit, fdescribe left in |
excessive-mocking |
warning | 5+ module mocks in one file |
snapshot-only |
warning | The only assertion is a snapshot |
conditional-assertion |
warning | Assertion behind if / && / ternary |
reimplemented-algorithm |
warning | Expected value recomputed from the call's own inputs: expect(total(p, q)).toBe(p * q * 1.2) |
oracle-copied-from-output |
warning | Expected float carries rounding noise (2.9000000000000004), so it was copied from the code's output |
skipped-test |
warning | .skip, xit, xdescribe |
weak-called-assertion |
info | Bare toHaveBeenCalled() on a shared mock whose arguments other tests check (warning when the test checks nothing else) |
Every rule except reimplemented-algorithm, oracle-copied-from-output and weak-called-assertion
(JS/TS only for now) also runs on pytest files (test_*.py, *_test.py): assert statements,
pytest.raises, unittest self.assert*, Mock.assert_called_*, @pytest.mark.skip, patch /
mocker.patch / monkeypatch.setattr and syrupy snapshots map onto the same model. Python-only catches
include assert (x == 1, "msg") (a tuple is always truthy) and assert m.called_once_with(1) (not a
Mock assertion; it returns a truthy child mock). Files are parsed by Python's own ast module, so
python3 (3.9+) must be on PATH (or set HONEST_TESTS_PYTHON); the audited code is never imported or
run. Mutation testing is JS/TS only for now.
npx @mintwork/honest-tests rules prints the list. Suppress a deliberate case with a reason:
// honest-tests-disable-next-line no-assertions -- smoke test: module must import cleanly
it('loads', () => { require('../src/plugin'); });(honest-tests-disable-line <rule> and file-level /* honest-tests-disable <rule> */ also work.)
| Operator | Example |
|---|---|
| binary | >= → >, < → <=, === → !==, + → -, * → / |
| logical | && ↔ || |
| assignment | += ↔ -= |
| boolean | true ↔ false |
| negation | !x → x |
| number | 50 → 51, 0 → 1 |
| condition | if (c) → if (true) / if (false) |
| return | return x → return undefined |
Safe by design: the project is copied to a temp dir (node_modules symlinked), mutants are written
only to the copy and restored after every run, and the copy is removed at the end — even on Ctrl-C. A red
baseline aborts the run. Only test files that import the target — directly or through other local
modules — are run (vitest, jest, mocha, node --test, bun test, deno test), once and with coverage
off. Deno projects without a package.json are found by their deno.json(c); if the tests need
permissions the test task doesn't grant, pass --test-cmd. Imports through path aliases (@/lib/x)
aren't followed yet; pass --test-cmd if your tests reach the target that way.
Mutation testing covers JS/TS source files (any of the runners above); Python is next.
Fast: mutants run in parallel, each worker in its own copy (default: half the cores, at most 4; the fixture's 26 mutants take ~4s with 4 workers instead of ~10s with one, on a 12-core laptop). Before trusting parallel runs, the baseline runs in every copy at once; if that fails (fixed ports, a shared database), it falls back to one worker, so a suite that isn't parallel-safe can't inflate the score. With 2+ related test files, a coverage map runs each test file once against a probed copy of the target, and each mutant then runs only the test files that execute its statement; mutants no test executes are reported without a run.
--test-cmd "<cmd>" custom command; {files} expands to related test files
--max-mutants <n> cap (default 100) --timeout <ms> per-run timeout
--min-score <0-100> exit 1 below this score --root <dir> project root
--concurrency <n> parallel workers --coverage-map auto|on|off (default auto)
honest-tests audit [paths...] [--json | --sarif] [-v] [--ignore <glob>] [--rule <id>] [--disable <id>]
[--changed <ref>] [--fail-on error|warning|never] [--mutate <file> ...]
honest-tests mutate <file> [--json | --sarif] [--test-cmd <cmd>] [--max-mutants <n>] [--min-score <n>]
[--timeout <ms>] [--root <dir>] [--concurrency <n>] [--coverage-map auto|on|off]
honest-tests rules [--json]
honest-tests init <claude|codex|cursor> [--dir <repo>]
Exit codes: 0 clean, 1 findings at/above --fail-on (default error) or score below --min-score,
2 usage/runtime error. JSON output schema is documented in
audit-procedure.md.
honest-tests audits its own suite in CI (npm run dogfood → zero findings) and mutation-checks its own
rules. While building v0.1, mutate src/rules/asserts-on-mock.ts scored 50% against our first rule
tests — the tool found the gaps in the tests for the tool. After adding the missing negative cases it
reached 84%. The rule has grown since, and in v0.2.0 it's at 80% (45 of 56 mutants killed), while
never-failing-assertion.ts is at 100% (43 of 43). The remaining survivors are mostly near-equivalent
mutants such as early-exit optimizations.
- Go and Rust adapters
- Mutation testing for Python files
-
reimplemented-algorithm,oracle-copied-from-outputandweak-called-assertionfor Python - Per-test-case coverage mapping (today the map is per test file)
Shipped features are in the CHANGELOG.
PRs welcome — especially new rules with a bad/good fixture and Python mutation support. See CONTRIBUTING.md.
