assertkit

AI-assisted QA, with the human kept in the loop.

Two commands and a demo app:

• assertkit generate <spec.md> — turn acceptance criteria into a Playwright spec plus a review-notes file listing every assumption the model made. It doesn't run the tests and doesn't commit them. You review, fix, and own what lands.
• assertkit triage <report.json> — read a Playwright JSON report and classify each failure as real-regression, flake, environment, or assertion-too-strict, with a rationale and a next step. Runs in CI and writes to the job summary.

Plus docs/testing-ai-output.md + tests/ai-output.spec.js: how to write tests for non-deterministic LLM output without them being flaky or useless.

Why it's built this way

Generated tests are a draft, not a deliverable. Every path here assumes a person checks the work: the generator emits assumptions alongside the code, the triage output is advice not action, and the demo app ships with a hand-written baseline suite that generated specs are held against.

Quick start

npm install
npx playwright install chromium

npm run test:unit      # the toolkit's own tests — API mocked, no key needed
npm run test:e2e       # baseline + generated Playwright specs against the demo app
npm run demo           # serve the demo app at http://localhost:4321

Use the CLI:

export ANTHROPIC_API_KEY=sk-...
node bin/assertkit.js generate docs/example-spec.md          # writes tests/generated/<slug>.spec.js + .NOTES.md
npx playwright test --reporter=json > results.json
node bin/assertkit.js triage results.json                    # markdown to stdout; exit 1 on a real regression

ASSERTKIT_MODEL overrides the model (default claude-opus-5; try claude-sonnet-5).

ANTHROPIC_API_KEY

The two CLI commands and tests/ai-output.spec.js call the Anthropic API, so they need a key:

• Locally: export ANTHROPIC_API_KEY=sk-ant-... (or copy .env.example to .env).
• In CI: add it as a repo secret — Settings → Secrets and variables → Actions → New repository secret, name ANTHROPIC_API_KEY. With it set, tests/ai-output.spec.js runs and failed CI runs get an automatic assertkit triage summary. Without it, those tests skip and everything else still passes — no key is needed for npm run test:unit (the client is mocked) or npm run test:e2e.

Layout

Path	What
bin/assertkit.js	CLI: arg parsing + dispatch only
src/generate.js, src/triage.js	the two orchestrations
src/lib/report.js	Playwright JSON report parser (pure)
src/lib/anthropic.js	client + model resolution + one ask() helper
src/lib/json.js	tolerant JSON extraction from model text
src/prompts/	prompt builders (pure, unit-tested)
demo-app/	TaskBoard — the system under test — + its static server
demo-app/tests/baseline/	hand-written Playwright specs
tests/generated/	reviewed assertkit generate output, committed
tests/ai-output.spec.js	non-determinism testing patterns
test/	the toolkit's node:test unit suite (API mocked)
docs/DESIGN.md	the design this repo was built to

Roadmap

Self-healing locators · visual regression · non-Playwright frameworks · multi-model comparison for generate. Not in v1 on purpose.

License

MIT