Repository navigation
test: test harness, testscript CLI tests, coverage floor and nightly smoke test - #11
Open
RogerCecilpot wants to merge 4 commits into
Open
RogerCecilpot wants to merge 4 commits into
RogerCecilpot wants to merge 4 commits into
Conversation
- cmdutil.Factory gains API (and Client()), so commands get the facade from the factory and tests can inject the gomock client. - The redacting logger moves to internal/logging so internal/api tests can use it without an import cycle now that cmdutil imports api; cmdutil keeps the old names. - cli.ExecuteWith runs the CLI in-process with a given factory and extra commands, for the test harness. Refs devlyft/devlyft#250 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- internal/testutil/golden: golden-file assertions with -update. - internal/testutil/apitest: the contract-checking fake API (moved out of the api tests), reusable by any test. - internal/testutil/clitest: runs the CLI in-process with buffered streams, an optional terminal and a gomock API client. - The golden tests move to package cli_test on top of clitest, unchanged, plus cases for a command using the injected client. - testscript runs the real binary for version, completion, help and usage errors, with `exitcode` and `fakeapi` script commands. - make cover-check (and CI on Linux) fails below 80% statement coverage of internal/, counted across packages, excluding generated code, mocks and the harness. - govulncheck covers shipped code only: kin-openapi, pinned by oapi-codegen v2.5.1, is now in a non-test package of the harness. Refs devlyft/devlyft#250 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test/smoke (build tag smoke) builds the binary and reads the test organization with an org API token; a create-and-delete deploy test runs only with DEVLYFT_SMOKE_WRITE=1. The Smoke workflow runs nightly, by hand or from another workflow in this repo, on a hosted runner with secrets from the protected `smoke` environment, and never for pull requests. docs/testing.md describes the four layers, updating goldens, running the smoke test and the one-off setup. Refs devlyft/devlyft#250 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs devlyft/devlyft#250 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RogerCecilpot
force-pushed
the
spike/261-credential-storage
branch
from
October 25, 2026 13:32
82ef0d4 to
d0d5548
Compare
RogerCecilpot
changed the base branch from
spike/261-credential-storage
to
feat/245-token-auth
October 25, 2026 13:32
RogerCecilpot
force-pushed
the
feat/250-test-harness
branch
from
October 25, 2026 13:32
7951503 to
290cbaa
Compare
This was referenced Oct 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes devlyft/devlyft#250. Stacked on #6 (top of the current chain); it will be restacked onto the #245 branch.
What
Turns the existing pieces (gomock facade mock from #236, contract tests, golden files from #246) into a documented harness, and adds what was missing. See
docs/testing.md.internal/testutil/golden:golden.Assertwith-update, shared by any package.internal/testutil/apitest: the kin-openapi contract-checking fake API, moved out ofinternal/api's tests. Every request must match the vendored specs; every canned 2xx response must match the schema.internal/api'sTestContractnow uses it.internal/testutil/clitest:clitest.Run(t, &clitest.Options{Args, TTY, Stdin, Env, API, Commands, Context})runs the whole CLI in-process and returns stdout, stderr and exit code;Transcript()is the golden format.cmdutil.Factory.API/Client(), so commands get the facade from the factory and tests injectmock.NewMockClient. To avoid an import cycle (api tests → cmdutil → api) the redacting logger moved tointernal/logging;cmdutilkeeps the old names, soroot.gois unchanged apart from a newcli.ExecuteWithfor the harness.cli_teston top ofclitest; all existing golden files are byte-for-byte unchanged. Four new cases cover a command using the injected gomock client (table, JSON, 404 with request ID, no client configured).internal/cli/testdata/script/*.txtar) runs the real binary as a subprocess: version, completion, help, usage errors and JSON errors, with two script commands:exitcode N prog ...andfakeapi(serves$WORK/api, setsDEVLYFT_API_URL; first real use comes with the list commands in #247).make cover-check(run on Linux in CI instead of the plain test step) fails below 80% statement coverage ofinternal/, counted across packages with-coverpkg, excluding generated clients, mocks and the harness.hack/covercheckdoes the merging. Currently 83.7% of 1015 statements.test/smoke, build tagsmoke): builds the binary and runsversion; reads the test org and its projects through the facade.TestDeployLifecyclecreates a build and a throwaway deploy (smoke-<unix time>) and deletes the deploy in cleanup, only withDEVLYFT_SMOKE_WRITE=1, off by default and not enabled in the workflow. There is no build-delete endpoint, so each write run leaves one build behind..github/workflows/smoke.yml: nightly (03:17 UTC),workflow_dispatchandworkflow_call; hosted runner,environment: smoke,permissions: contents: read,if: github.repository == 'devlyft/cli', never onpull_request, actions pinned to the same SHAs aschecks.yml. It fails loudly if the secrets are missing.golangci-lintnow lints with thesmoketag so the smoke tests are linted too.govulncheck scope (please look)
Moving the contract validator into a non-test package made govulncheck report two kin-openapi advisories (GO-2026-6275, GO-2026-6112; fixed in v0.142.0/v0.144.0). kin-openapi can't be upgraded: oapi-codegen v2.5.1 (the platform's pinned version) fails to build against v0.144.0 and later. The affected calls are only in the test harness, against our own fixtures, which is the same exposure as before when the code lived in
_test.gofiles (which govulncheck skips). Somake vulnnow scans everything exceptinternal/testutil/, and thechecks / securityjob runsmake vulninstead ofgovulncheck-action(check name unchanged). Revisit when the platform moves to a newer oapi-codegen.Setup you need to do (not done by this PR)
smoke, restricted to themainbranch (required reviewers optional).DEVLYFT_SMOKE_API_TOKENandDEVLYFT_SMOKE_ORG(the org UUID).release.yml(uses: ./.github/workflows/smoke.yml,secrets: inherit) to run on each release.Verified locally
make lint(0 issues),make test(-race),make cover-check(83.7%),make vuln(no vulnerabilities in shipped code),make generate(no diff), actionlint, andgo test -tags smoke ./test/smoke/skips cleanly without credentials.🤖 Generated with Claude Code