test: make integration suites assert deterministic outcomes#5
Merged
Conversation
--json-file-name ./output/results.json failed with ENOENT because writeJSONFile never created intermediate directories — masked until now by an integration test that only asserted the announcement message. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The integration tests wrapped nearly every CLI invocation in try/catch
and accepted failures matching broad regexes (one alternation was
literally /error/i), plus tautological expect(true) assertions — a
fully broken command still passed the suite. The excuses are gone: the
mock API (Prism + auth shim) is booted deterministically by the test
runner with readiness polling and an isolated config dir.
Every scenario was probed against the live mock API and now asserts
its real outcome unconditionally:
- happy paths assert parsed JSON shapes / exact output markers; a dead
or missing mock API now fails 61 tests instead of zero
- failure paths use a runExpectingFailure helper (replacing expect.fail
inside try/catch, which chai's own AssertionError could satisfy) and
assert the precise error message and exit code
- file-writing tests run in temp cwds, verify the written JSON content,
and no longer pollute the repo working tree
- shared constants live in test/integration/helpers.ts: mock URL (env
override), API key, CLI path, and a dead-API URL on port 9 (discard)
replacing the assumed-unbound localhost:9999
- stale oclif-era regex alternations and a broken \\{ escape removed
Net -914 lines. Also surfaced two real behaviors: junit report
downloads succeed against the mock (now asserted), and --json-file-name
with intermediate directories was broken (fixed in the previous commit).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #3, knocking out the first deferred item: the integration suites could not fail. Nearly every CLI invocation was wrapped in try/catch with fallbacks accepting any output matching broad regexes (one alternation was literally
/error/i), plusexpect(true).to.be.truetautologies — a fully broken command still went green.Since #3 the mock API boots deterministically (readiness polling, isolated
DCD_CONFIG_DIR), so every scenario was probed against the live mock API and now asserts its actual outcome unconditionally:runExpectingFailurehelper (replacingexpect.failinside try/catch, which chai's own AssertionError could satisfy) and assert exact error messages and exit codes.test/integration/helpers.ts(mock URL with env override, API key, CLI path, dead-API URL on port 9 replacing the assumed-unboundlocalhost:9999).\\{escape removed.Net −914 lines of test code while getting strictly stronger.
Bug found by the tightening
--json-file-name ./output/results.jsonfailed with ENOENT —writeJSONFilenever created intermediate directories, masked by the old test asserting only the announcement message. Fixed in the first commit; the test now verifies the written file's content.Also pinned two real behaviors as tests: junit report downloads succeed against the mock, and
status --jsonreports errors in-band (exit 0,attempts: 1on client errors).Test plan
pnpm test: 122/122 passingMOCK_API_URL=http://127.0.0.1:9 npx mocha ...→ 61 failures (previously this scenario passed silently)pnpm lint: 0 errorspnpm buildclean🤖 Generated with Claude Code