Skip to content

test: test harness, testscript CLI tests, coverage floor and nightly smoke test - #11

Open
RogerCecilpot wants to merge 4 commits into
feat/245-token-authfrom
feat/250-test-harness
Open

RogerCecilpot wants to merge 4 commits into
feat/245-token-authfrom
feat/250-test-harness

Conversation

@RogerCecilpot

Copy link
Copy Markdown

Closes devlyft/devlyft#250. Stacked on #6 (top of the current chain); it will be restacked onto the #245 branch.

What

Turns the existing pieces (gomock facade mock from #236, contract tests, golden files from #246) into a documented harness, and adds what was missing. See docs/testing.md.

  • internal/testutil/golden: golden.Assert with -update, shared by any package.
  • internal/testutil/apitest: the kin-openapi contract-checking fake API, moved out of internal/api's tests. Every request must match the vendored specs; every canned 2xx response must match the schema. internal/api's TestContract now uses it.
  • internal/testutil/clitest: clitest.Run(t, &clitest.Options{Args, TTY, Stdin, Env, API, Commands, Context}) runs the whole CLI in-process and returns stdout, stderr and exit code; Transcript() is the golden format.
  • Factory carries the API client: cmdutil.Factory.API / Client(), so commands get the facade from the factory and tests inject mock.NewMockClient. To avoid an import cycle (api tests → cmdutil → api) the redacting logger moved to internal/logging; cmdutil keeps the old names, so root.go is unchanged apart from a new cli.ExecuteWith for the harness.
  • Golden tests moved to package cli_test on top of clitest; all existing golden files are byte-for-byte unchanged. Four new cases cover a command using the injected gomock client (table, JSON, 404 with request ID, no client configured).
  • testscript (internal/cli/testdata/script/*.txtar) runs the real binary as a subprocess: version, completion, help, usage errors and JSON errors, with two script commands: exitcode N prog ... and fakeapi (serves $WORK/api, sets DEVLYFT_API_URL; first real use comes with the list commands in #247).
  • Coverage floor: make cover-check (run on Linux in CI instead of the plain test step) fails below 80% statement coverage of internal/, counted across packages with -coverpkg, excluding generated clients, mocks and the harness. hack/covercheck does the merging. Currently 83.7% of 1015 statements.
  • Smoke test (test/smoke, build tag smoke): builds the binary and runs version; reads the test org and its projects through the facade. TestDeployLifecycle creates a build and a throwaway deploy (smoke-<unix time>) and deletes the deploy in cleanup, only with DEVLYFT_SMOKE_WRITE=1, off by default and not enabled in the workflow. There is no build-delete endpoint, so each write run leaves one build behind.
  • .github/workflows/smoke.yml: nightly (03:17 UTC), workflow_dispatch and workflow_call; hosted runner, environment: smoke, permissions: contents: read, if: github.repository == 'devlyft/cli', never on pull_request, actions pinned to the same SHAs as checks.yml. It fails loudly if the secrets are missing.
  • golangci-lint now lints with the smoke tag so the smoke tests are linted too.

govulncheck scope (please look)

Moving the contract validator into a non-test package made govulncheck report two kin-openapi advisories (GO-2026-6275, GO-2026-6112; fixed in v0.142.0/v0.144.0). kin-openapi can't be upgraded: oapi-codegen v2.5.1 (the platform's pinned version) fails to build against v0.144.0 and later. The affected calls are only in the test harness, against our own fixtures, which is the same exposure as before when the code lived in _test.go files (which govulncheck skips). So make vuln now scans everything except internal/testutil/, and the checks / security job runs make vuln instead of govulncheck-action (check name unchanged). Revisit when the platform moves to a newer oapi-codegen.

Setup you need to do (not done by this PR)

  1. Create a dedicated test organization with one project and environment.
  2. Create an org API token for it (short expiry, rotation reminder).
  3. In devlyft/cli → Settings → Environments, create smoke, restricted to the main branch (required reviewers optional).
  4. Add environment secrets DEVLYFT_SMOKE_API_TOKEN and DEVLYFT_SMOKE_ORG (the org UUID).
  5. Run Actions → Smoke → Run workflow once to confirm. Until then the nightly run fails with a clear error.
  6. Optional: call it from release.yml (uses: ./.github/workflows/smoke.yml, secrets: inherit) to run on each release.

Verified locally

make lint (0 issues), make test (-race), make cover-check (83.7%), make vuln (no vulnerabilities in shipped code), make generate (no diff), actionlint, and go test -tags smoke ./test/smoke/ skips cleanly without credentials.

🤖 Generated with Claude Code

RogerCecilpot and others added 4 commits October 25, 2026 13:30
- cmdutil.Factory gains API (and Client()), so commands get the facade
  from the factory and tests can inject the gomock client.
- The redacting logger moves to internal/logging so internal/api tests
  can use it without an import cycle now that cmdutil imports api;
  cmdutil keeps the old names.
- cli.ExecuteWith runs the CLI in-process with a given factory and extra
  commands, for the test harness.

Refs devlyft/devlyft#250

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- internal/testutil/golden: golden-file assertions with -update.
- internal/testutil/apitest: the contract-checking fake API (moved out of
  the api tests), reusable by any test.
- internal/testutil/clitest: runs the CLI in-process with buffered
  streams, an optional terminal and a gomock API client.
- The golden tests move to package cli_test on top of clitest, unchanged,
  plus cases for a command using the injected client.
- testscript runs the real binary for version, completion, help and
  usage errors, with `exitcode` and `fakeapi` script commands.
- make cover-check (and CI on Linux) fails below 80% statement coverage
  of internal/, counted across packages, excluding generated code,
  mocks and the harness.
- govulncheck covers shipped code only: kin-openapi, pinned by
  oapi-codegen v2.5.1, is now in a non-test package of the harness.

Refs devlyft/devlyft#250

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test/smoke (build tag smoke) builds the binary and reads the test
organization with an org API token; a create-and-delete deploy test runs
only with DEVLYFT_SMOKE_WRITE=1. The Smoke workflow runs nightly, by hand
or from another workflow in this repo, on a hosted runner with secrets
from the protected `smoke` environment, and never for pull requests.

docs/testing.md describes the four layers, updating goldens, running the
smoke test and the one-off setup.

Refs devlyft/devlyft#250

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs devlyft/devlyft#250

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@RogerCecilpot
RogerCecilpot force-pushed the spike/261-credential-storage branch from 82ef0d4 to d0d5548 Compare October 25, 2026 13:32
@RogerCecilpot
RogerCecilpot changed the base branch from spike/261-credential-storage to feat/245-token-auth October 25, 2026 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant