From 712d8e2b0469dc0f5bd62042834b0d87a091a0f9 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 10 Aug 2026 03:58:09 +0000 Subject: [PATCH 1/2] fix(spec,cli): govern the QA testing domain and enforce TestSuiteSchema at the `os test` load site (#6247) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #6247 filed `packages/spec/src/qa/testing.zod.ts` as declared-but-inert on a grep that scanned only `*Schema` identifiers. Every consumer here reads the TYPE names (`QA.TestSuite`, `QA.TestStep`, `QA.TestAction`), so the search matched nothing and a complete execution chain read as zero consumers: core's `TestRunner` + `HttpTestAdapter` (whose `action.type` switch labels ARE the `TestActionTypeSchema` values), published through `export * as QA`, driven by the shipped, documented CLI command `os test`. The 2026-08-07 retire ruling rested on that reading and was withdrawn on 2026-08-08 in favour of enforce. The real gap was narrower and genuine: the type was the contract and the schema had no `parse` site anywhere, so `os test` loaded suites with `JSON.parse(content) as QA.TestSuite` beside the author's own `// Should validate with Zod`. - `packages/spec/liveness/qa.json` — seed the ledger, governed via the same `SPEC_ONLY_SCHEMAS` override as `query`/`webhook`/`validation` (a QA suite is an authored file, not stack metadata). 4 live rows with file:line evidence into the runner, 5 dead recorded honestly; step/action/assertion keys sit below the one-level walk and are measured in the notes rather than fanned into rows the gate would not check. No `authorWarn` anywhere, deliberately: the lint walks stack collections and a QA suite belongs to no stack, so the flag would be a silent no-op inside the mechanism built to catch silent no-ops. - `packages/cli/src/commands/test.ts` — `loadTestSuite()` parses with `TestSuiteSchema.safeParse` at the load boundary and refuses a bad suite there, naming the file, listing the issues and quoting the expected shape. A refusal counts as one failed suite instead of killing the run. - pin `packages/cli/test/qa-suite-schema-load.test.ts` — the three shapes the cast admitted (missing `scenarios` → TypeError inside the runner; misspelled `steps` → scenario reports PASSED having executed nothing; bad `action.type` → dies mid-run after earlier steps wrote records) are now refused at load. Closes #6247 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Up7rAGwREEy754haLKVtZH --- .changeset/qa-testing-liveness-enforce.md | 47 ++++++++ packages/cli/src/commands/test.ts | 74 +++++++++++- .../cli/test/qa-suite-schema-load.test.ts | 113 ++++++++++++++++++ packages/spec/liveness/README.md | 10 +- packages/spec/liveness/qa.json | 68 +++++++++++ .../spec/scripts/liveness/check-liveness.mts | 17 ++- 6 files changed, 322 insertions(+), 7 deletions(-) create mode 100644 .changeset/qa-testing-liveness-enforce.md create mode 100644 packages/cli/test/qa-suite-schema-load.test.ts create mode 100644 packages/spec/liveness/qa.json diff --git a/.changeset/qa-testing-liveness-enforce.md b/.changeset/qa-testing-liveness-enforce.md new file mode 100644 index 0000000000..9a4d869308 --- /dev/null +++ b/.changeset/qa-testing-liveness-enforce.md @@ -0,0 +1,47 @@ +--- +"@objectstack/spec": minor +"@objectstack/cli": patch +--- + +fix(spec,cli): govern the QA testing domain and enforce `TestSuiteSchema` at the `os test` load site (#6247) + +`packages/spec/src/qa/testing.zod.ts` declares the Quality Protocol — test +suites, scenarios, steps, actions, assertions — and had no liveness ledger, so +the ADR-0049 enforce-or-remove machinery had never looked at it. #6247 filed it +as **declared-but-inert on zero runtime consumers**, and that reading was wrong: +the grep behind it matched only `*Schema` identifiers, while every consumer here +reads the **type** names (`QA.TestSuite`, `QA.TestStep`, `QA.TestAction`). What +it missed is a complete execution chain — core's `TestRunner` and +`HttpTestAdapter` (whose `action.type` switch labels *are* the +`TestActionTypeSchema` values), published through `export * as QA`, driven by the +shipped, documented CLI command `os test`. The retire ruling that followed from +the bad reading was withdrawn; this is the enforce leg. + +**The real gap was narrower and genuine.** The type was the contract and the +schema had no `parse` site anywhere in the platform: `os test` loaded suites with +`JSON.parse(content) as QA.TestSuite`, next to the schema author's own +`// Should validate with Zod`. A type assertion checks nothing at runtime, so a +malformed suite failed late and in the wrong place — a missing `scenarios` +TypeError'd inside the runner with no idea which file it came from, a misspelled +`steps` key reported the scenario **passed** having executed nothing, and a bad +`action.type` died in the HTTP adapter's `default:` branch mid-run, after earlier +steps had already written records. `os test` now parses at the load boundary and +refuses a bad suite there, naming the file, listing the issues and quoting the +expected shape; a refusal counts as one failed suite rather than killing the run. +Valid suites load and execute unchanged. + +**`packages/spec/liveness/qa.json`** seeds the ledger, governed through the same +`SPEC_ONLY_SCHEMAS` override as `query`/`webhook`/`validation` — a QA suite is a +file an author writes, not stack metadata, so there is no registry to fold it +onto and the override *is* its governance. Four live rows (`scenarios.id`, +`.setup`, `.steps`, `.teardown`) carry `file:line` evidence into the runner; +step, action and assertion keys sit below the gate's one-level walk and their +measurements are recorded in the notes rather than fanned into rows the gate +would not check. Five dead rows are recorded honestly, two of which go onto the +enforce-or-remove worklist: `scenarios.tags` advertises filtering that `os test` +has no flag to express, and `scenarios.requires` declares param/plugin +preconditions nothing checks, so a suite naming an absent plugin runs anyway and +fails later as an unexplained HTTP error. None is marked `authorWarn`, and the +omission is deliberate — the author-side lint walks stack collections, a QA suite +belongs to no stack, and a warn flag that can never be emitted would be a silent +no-op inside the mechanism built to catch them. diff --git a/packages/cli/src/commands/test.ts b/packages/cli/src/commands/test.ts index 8001908123..662652e409 100644 --- a/packages/cli/src/commands/test.ts +++ b/packages/cli/src/commands/test.ts @@ -6,6 +6,7 @@ import path from 'path'; import fs from 'fs'; import { QA as CoreQA } from '@objectstack/core'; import * as QA from '@objectstack/spec/qa'; +import type { ZodError } from 'zod'; /** * Resolve a glob-like pattern to matching file paths. @@ -52,6 +53,59 @@ function resolveGlob(pattern: string): string[] { .filter(fullPath => fs.statSync(fullPath).isFile()); } +/** The suite shape, quoted back at an author whose file did not match it. */ +const SUITE_SHAPE = + '{ "name": string, "scenarios": [ { "id", "name", "steps": [ { "name", "action": { "type", "target" } } ] } ] }'; + +/** + * Load and VALIDATE one Quality Protocol suite file. + * + * This used to be `JSON.parse(content) as QA.TestSuite`, carrying the schema + * author's own `// Should validate with Zod`. The cast is the declared≠enforced + * gap ADR-0049 names (#6247): `TestSuiteSchema` was declared, shipped and + * documented, and had no `parse` site anywhere in the platform — the TYPE was + * the contract the runner read, and a type assertion checks nothing at runtime. + * What a bad file did instead of being refused: a missing `scenarios` TypeError'd + * inside `TestRunner.runSuite` with no idea which file it came from; a misspelled + * `steps` reported the scenario PASSED having executed nothing; a bad + * `action.type` reached the HTTP adapter's `default:` branch mid-run, after the + * earlier steps had already written records. + * + * So the parse happens HERE, at the boundary where the file name is still in + * hand, and the error names the file, lists the issues and prescribes the shape. + * Throws rather than exiting, so the caller keeps ownership of the run tally — + * one broken suite is a failed suite, not a dead command. + */ +export function loadTestSuite(file: string): QA.TestSuite { + const content = fs.readFileSync(file, 'utf-8'); + + let doc: unknown; + try { + doc = JSON.parse(content); + } catch (e) { + // A bare `Unexpected end of JSON input` names nothing; with a glob expanding + // to a dozen files that is a message you have to bisect by hand. + throw new Error( + `${file} is not valid JSON: ${e instanceof Error ? e.message : String(e)}\n` + + ` A Quality Protocol suite is a JSON document shaped ${SUITE_SHAPE}`, + ); + } + + const result = QA.TestSuiteSchema.safeParse(doc); + if (!result.success) { + const issues = (result.error as ZodError).issues + .map((issue) => ` ✗ ${issue.path.join('.') || '(root)'}: ${issue.message}`) + .join('\n'); + throw new Error( + `${file} is not a valid Quality Protocol suite (TestSuiteSchema):\n${issues}\n` + + ` Expected shape: ${SUITE_SHAPE}\n` + + ` Reference: content/docs/references/qa/testing.mdx`, + ); + } + + return result.data as QA.TestSuite; +} + export default class Test extends Command { static override description = 'Run Quality Protocol test scenarios against a running server'; @@ -93,12 +147,22 @@ export default class Test extends Command { for (const file of testFiles) { console.log(`\n📄 Running suite: ${chalk.bold(path.basename(file))}`); + + // Load and validate FIRST, and report a refusal on its own terms: a file + // the schema rejects never had a chance to run, so folding it into the + // run-failure branch below would report it as if the server had said no. + let suite: QA.TestSuite; + try { + suite = loadTestSuite(file); + } catch (e) { + console.error(chalk.red(e instanceof Error ? e.message : String(e))); + totalFailed++; // Count suite failure + continue; + } + try { - const content = fs.readFileSync(file, 'utf-8'); - const suite = JSON.parse(content) as QA.TestSuite; // Should validate with Zod - const results = await runner.runSuite(suite); - + for (const result of results) { const icon = result.passed ? '✅' : '❌'; console.log(` ${icon} Scenario: ${result.scenarioId} (${result.duration}ms)`); @@ -117,7 +181,7 @@ export default class Test extends Command { } } } catch (e) { - console.error(chalk.red(`Failed to load or run suite ${file}: ${e}`)); + console.error(chalk.red(`Failed to run suite ${file}: ${e}`)); totalFailed++; // Count suite failure } } diff --git a/packages/cli/test/qa-suite-schema-load.test.ts b/packages/cli/test/qa-suite-schema-load.test.ts new file mode 100644 index 0000000000..04aaec0692 --- /dev/null +++ b/packages/cli/test/qa-suite-schema-load.test.ts @@ -0,0 +1,113 @@ +// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * PIN (#6247) — `os test` REFUSES a malformed `qa/*.test.json` at LOAD time. + * + * The load site used to be `JSON.parse(content) as QA.TestSuite`, with the + * schema author's own `// Should validate with Zod` sitting next to it. A type + * assertion is not a check: the cast made every JSON document on disk a + * `TestSuite` as far as the compiler was concerned, and the first thing that + * noticed otherwise was the runner, several layers down and with no idea which + * file it came from. The three failure shapes that produced: + * + * - `{}` (no `scenarios`) → `TestRunner.runSuite` iterates `undefined` and + * throws a TypeError attributed to the runner, not the file; + * - `{ scenarios: [] }` with a typo'd key → the suite reports SUCCESS having + * executed nothing, which is the dangerous one: a broken suite that passes + * is indistinguishable from a green one in CI; + * - a bad `action.type` → survives the load, survives the runner, and dies in + * the HTTP adapter's `default:` branch mid-run, after the preceding steps + * have already written records. + * + * An AI-authored suite hits all three routinely, which is exactly the + * declared≠enforced gap ADR-0049 names: `TestSuiteSchema` was declared, shipped, + * documented — and had no `parse` site anywhere in the platform. + * + * So the pin is on the LOAD boundary, not on the runner: a document that + * `TestSuiteSchema` rejects must never reach `runSuite`, and the refusal must + * name the file and the issues. Valid suites must still load unchanged. + */ + +import { describe, it, expect } from 'vitest'; +import { mkdtempSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { loadTestSuite } from '../src/commands/test'; + +const dir = mkdtempSync(join(tmpdir(), 'os-qa-suite-')); + +function suiteFile(name: string, body: string): string { + const file = join(dir, name); + writeFileSync(file, body, 'utf-8'); + return file; +} + +const VALID_SUITE = { + name: 'crm smoke', + scenarios: [ + { + id: 'create-account', + name: 'Create an account', + steps: [ + { + name: 'create', + action: { type: 'create_record', target: 'accounts', payload: { name: 'Acme' } }, + assertions: [{ field: 'id', operator: 'not_null', expectedValue: null }], + }, + ], + }, + ], +}; + +describe('os test — suite load is schema-checked (#6247)', () => { + it('loads a valid suite unchanged', () => { + const file = suiteFile('valid.test.json', JSON.stringify(VALID_SUITE)); + const suite = loadTestSuite(file); + expect(suite.name).toBe('crm smoke'); + expect(suite.scenarios).toHaveLength(1); + expect(suite.scenarios[0].steps[0].action.type).toBe('create_record'); + }); + + it('refuses a suite with no `scenarios` — the shape that TypeErrors inside the runner', () => { + const file = suiteFile('no-scenarios.test.json', JSON.stringify({ name: 'empty' })); + expect(() => loadTestSuite(file)).toThrow(/no-scenarios\.test\.json/); + expect(() => loadTestSuite(file)).toThrow(/scenarios/); + }); + + it('refuses a misspelled step key — the shape that SILENTLY passes', () => { + // `stepz` instead of `steps`: the cast let this through and the scenario + // reported PASSED having run nothing at all. + const file = suiteFile( + 'typo.test.json', + JSON.stringify({ name: 's', scenarios: [{ id: 'a', name: 'A', stepz: [] }] }), + ); + expect(() => loadTestSuite(file)).toThrow(/typo\.test\.json/); + }); + + it('refuses an unknown action type — the shape that dies mid-run after writes', () => { + const bad = structuredClone(VALID_SUITE) as Record; + bad.scenarios[0].steps[0].action.type = 'summon_record'; + const file = suiteFile('bad-action.test.json', JSON.stringify(bad)); + expect(() => loadTestSuite(file)).toThrow(/bad-action\.test\.json/); + }); + + it('names the offending file and the issues, and prescribes the shape', () => { + const file = suiteFile('broken.test.json', JSON.stringify({ scenarios: 'not an array' })); + let message = ''; + try { + loadTestSuite(file); + } catch (e) { + message = e instanceof Error ? e.message : String(e); + } + expect(message).toContain('broken.test.json'); + expect(message).toContain('TestSuiteSchema'); + // Self-prescribing: the author is told what a suite looks like, not just + // that theirs is wrong. + expect(message).toContain('scenarios'); + }); + + it('refuses invalid JSON with the file named, rather than a bare SyntaxError', () => { + const file = suiteFile('not-json.test.json', '{ "name": '); + expect(() => loadTestSuite(file)).toThrow(/not-json\.test\.json/); + }); +}); diff --git a/packages/spec/liveness/README.md b/packages/spec/liveness/README.md index 704bd80c97..1006ee74a0 100644 --- a/packages/spec/liveness/README.md +++ b/packages/spec/liveness/README.md @@ -645,7 +645,14 @@ The governed set is `GOVERNED` at the top of `check-liveness.mts`. To add a type RecordDetailView had been gating the History tab on it the whole time (#2707). 4. Add the type to `GOVERNED`; confirm the gate is green. -## Current state — 27 governed types (complete registry coverage) +## Current state — 30 governed types (complete registry coverage) + +> The table below carries 28 of the 30. `api` and `capability` are governed +> (they are in `GOVERNED`, they have ledgers, the gate counts them) and were +> added without a row here — the table fell behind its own registry, which is +> the shape this file keeps warning about one level down. Filed rather than +> back-filled from a guess: writing two Notes cells for changes somebody else +> measured is exactly the fabrication the drill section forbids. **The counting method for this table is the gate's own report** — `check-liveness.mts --json`, `types..byStatus` — decided in #4488 after @@ -698,6 +705,7 @@ for t, v in r['types'].items(): | mapping | 14 | – | 0 | – | seeded 2026-08-01 (#4488) at 8/11 live; **0 dead since #4509** retired the three that were not. The import half (#2611) is loudly enforced — unsupported transforms/formats are 400s, `mode`/`upsertKey` default the request, the wizard picker renders `label`. RETIRED 17.0.0: `extractQuery` (authorWarn — "for export only" promised an export path no exporter implements) + `errorPolicy`/`batchSize`, which were dead AND **unwarnable** (schema defaults materialize at parse, so presence ≠ authored — `_authorWarnSkipped`, the non-boolean instance of the default(true) rule). That unwarnability is why they went out in the 17.0.0 window rather than after a deprecation cycle: removal was the only channel that could ever reach the author. Rows DELETED, not tombstoned — MappingSchema is strict, so the keys left the walked shape | | seed | 5 | – | 0 | – | seeded 2026-08-01 (#4488). Fully live via SeedLoaderService on both doors (boot/per-org replay + runtime-draft publish). `records` is the z.record walk boundary: the keys an author writes are the target object's fields, governed by that object's own definitions — recorded in the entry, not silently skipped | | translation | 17 | – | 2 | – | seeded 2026-08-01 (#4488) — after fixing the walker: the registered schema is a z.preprocess pipe (#3778 retired-dialect guard) whose transform side the unwrap always took, so the type was literally unwalkable. 10 of 11 groups live across spec resolvers, REST localization, objectui client resolvers and plugin-audit (whose composed-key `t()` calls make `messages` easy to mis-verify as dead). Dead 1 = `validationMessages` (authorWarn): nothing resolves it, and #3778's own legacy-key migration table steers `errors:` authors into it — a shipped false signpost, the capabilities.readOnly shape | **#4667**: `validationMessages` REMOVED (row deleted) — removed from the shared translationDataShape(), so it retired at BOTH doors at once, closing the item-only asymmetry #3778's original guard had. #3778's own `errors` guidance was rewritten in the same change: it had been steering authors INTO this dead group. | +| qa | 4 | – | 5 | – | seeded 2026-08-10 (#6247) — **not a metadata type**: `TestSuiteSchema` is the FILE surface of the shipped `os test` command (`qa/*.test.json`), governed through the same `SPEC_ONLY_SCHEMAS` override as `query`/`webhook`/`validation`. It is in the table as the clearest worked example of a **false `dead` measurement**: #6247 reported the whole domain declared-but-inert on a grep that scanned only `*Schema` identifiers, and every consumer here reads the **type** names (`QA.TestSuite`, `QA.TestStep`, `QA.TestAction`) — so an entire execution chain (core's `TestRunner` + `HttpTestAdapter`, published via `export * as QA`, driven by a documented CLI command) read as zero consumers, and a retire ruling was issued on it before being withdrawn. The `evidenceScope` table one section up says no amount of specifier matching is sufficient for a negative claim; this is the same lesson for **identifier** matching. What was really wrong was narrower and real: the type was the contract and the schema had no `parse` site, so the CLI's `JSON.parse(content) as QA.TestSuite` cast admitted anything — ENFORCED in the same change (`TestSuiteSchema.safeParse` at the load site, pinned). Dead 5 = `name` (the file name is the suite identity; the CLI prints `path.basename`), `scenarios.name` (describe() says "for test reports"; every report carries `scenarioId` instead), `scenarios.description` (docs-shaped, kept), and the two on the enforce-or-remove worklist — `scenarios.tags` promises filtering that `os test`'s two flags cannot express, and `scenarios.requires` declares param/plugin preconditions nothing checks, so a suite naming a missing plugin runs anyway and fails as an unexplained HTTP error. Neither carries `authorWarn` and the omission is deliberate (`_authorWarnSkipped`): the lint walks stack **collections**, a QA suite is a loose file in no stack, so a warn flag here would emit nothing — a silent no-op inside the mechanism built to catch silent no-ops | | validation | 15 | 0 | 3 | 0 | seeded 2026-08-01 (#4488). The ADR-0020 carrier: the evaluator honors active/events/priority/severity/type/condition/message (the zod header's "only reads type/condition/…" prose is STALE — trust the ledger). Dead 3 = label/description/tags, declared governance metadata, kept unmarked. Union walk boundary recorded: only base + `script` keys walked; per-variant keys are governed by the evaluator's tests, not ledger rows. **No longer a registered metadata kind** — #4509 retired it under ADR-0088 (a standalone rule had no object-binding key and every variant is `.strict()`, so it bound to nothing and gated no write; a state machine authored that way saved cleanly and did nothing). The rule VOCABULARY is untouched and fully live via `object.validations[]`, so the ledger keeps governing it through the gate's spec-only override, alongside `webhook` and `query`. The contrast with the two bridges in the same batch is the point: enforce-or-remove picked ENFORCE where the feature existed and only the wiring was missing, and REMOVE where the shape itself could not carry the feature | The `dead` set across types is the enforce-or-remove worklist (ADR-0049); every diff --git a/packages/spec/liveness/qa.json b/packages/spec/liveness/qa.json new file mode 100644 index 0000000000..2b054a3027 --- /dev/null +++ b/packages/spec/liveness/qa.json @@ -0,0 +1,68 @@ +{ + "type": "qa", + "_note": "TestSuiteSchema (packages/spec/src/qa/testing.zod.ts) — the Quality Protocol file surface: an author writes `qa/*.test.json`, `os test` loads it, and core's TestRunner executes it. Seeded 2026-08-10 (#6247), the ENFORCE leg of an enforce-or-remove call that was ruled the other way first and then withdrawn, which is the lesson worth keeping. #6247 filed this domain as declared-but-inert on a grep that scanned only `*Schema` identifiers; every consumer here reads the TYPE names (`QA.TestSuite`, `QA.TestScenario`, `QA.TestStep`, `QA.TestAction`, `QA.TestAssertion`), so the search matched nothing and a complete execution chain read as zero consumers. The 2026-08-07 retire ruling rested on that reading and was WITHDRAWN on 2026-08-08 (issue comment 5225532429) once the sweep's pre-flight gate falsified it. A schema with no `parse` site is not the same finding as a schema with no consumer, and only the first one was true. The measured chain, by layer: the RUNNER (packages/core/src/qa/runner.ts) reads suite.scenarios, scenario.id/setup/steps/teardown and step.name/action/capture/assertions; the ADAPTER (packages/core/src/qa/http-adapter.ts) switches on action.type — its case labels ARE the TestActionTypeSchema values — and reads target/payload/user; both are published through packages/core/src/index.ts:25 (`export * as QA`); the driving entry point is the shipped oclif command `os test` (packages/cli/src/commands/test.ts), documented at content/docs/deployment/cli.mdx:987,1012-1020 and packages/cli/README.md:104. WALK BOUNDARY, recorded rather than silently skipped: the gate classifies one level and this file drills `scenarios` one more, so the verdicts here cover the suite and scenario levels only. Step / action / assertion keys sit BELOW the walk; they were measured in the same pass and their verdicts are recorded in the `setup`/`steps`/`teardown` notes instead of being fanned out into rows the gate would not check. AUTHOR-WARN CHANNEL: none exists for this type, and no entry is marked `authorWarn` for that reason (`_authorWarnSkipped`). The CLI lint (packages/lint/src/lint-liveness-properties.ts) walks stack COLLECTIONS — `stack.flows`, `stack.views`, … — and a QA suite is not part of a stack at all; it is a loose JSON file `os test` globs off disk. Marking an entry `authorWarn` here would produce a warning nothing can emit, which is the same silent no-op this ledger exists to catch, so the dead entries below carry their correction in `note` and the load-site parse (below) is what actually reaches the author. LOAD-SITE ENFORCEMENT: the same change that seeded this file replaced the CLI's `JSON.parse(content) as QA.TestSuite` cast — the schema author's own `// Should validate with Zod` TODO — with a real `TestSuiteSchema.safeParse`, so a malformed suite is named at load time instead of reaching the runner as a lie about its own shape. That is what makes the `live` rows below enforced rather than merely read. No `qa` property is a bound HIGH_RISK class in proof-registry.mts, so no entry carries a `proof`; none is invented to look thorough.", + "props": { + "name": { + "status": "dead", + "verifiedAt": "2026-08-10", + "evidenceScope": "in-repo", + "note": "Parsed and never read. `TestRunner.runSuite` (packages/core/src/qa/runner.ts:25-31) touches only `suite.scenarios`, and the one caller prints `path.basename(file)` as the suite heading (packages/cli/src/commands/test.ts:95) — so renaming a suite changes nothing an operator sees. Kept, not retired: it is display-shaped metadata whose describe() promises no capability ('Test suite name'), and it is the natural title the moment anything reports per-suite results. The honest reading is that the FILE NAME is the suite identity today. Recorded so the next reader does not have to re-derive that, and so that wiring it up counts as making a dead key live rather than as a no-op refactor." + }, + "scenarios": { + "children": { + "id": { + "status": "live", + "evidence": "packages/core/src/qa/runner.ts:101 (TestResult.scenarioId), :47 (the same id on the setup-failure path)", + "verifiedAt": "2026-08-10", + "evidenceScope": "in-repo", + "note": "The scenario identity in every result the CLI prints (packages/cli/src/commands/test.ts:104) and the ONLY human-readable handle a failing run gives you — `name` is not printed anywhere. Not deduplicated: two scenarios may declare the same id and both run, so an id collision shows up as two indistinguishable result lines rather than an error." + }, + "name": { + "status": "dead", + "verifiedAt": "2026-08-10", + "evidenceScope": "in-repo", + "note": "Its describe() says 'Scenario name for test reports' and no report carries it: `TestResult` (packages/core/src/qa/runner.ts:6-12) has `scenarioId` and no name field, and the CLI prints `result.scenarioId` (packages/cli/src/commands/test.ts:104). So an author who writes a careful human-readable `name` and a terse `id` gets the terse one in every failure message. Mildly misleading rather than benign — the describe() names an output that does not exist — but it is a display key with an obvious enforcement route (print it beside the id), which is why the note carries the correction instead of a retirement." + }, + "description": { + "status": "dead", + "verifiedAt": "2026-08-10", + "evidenceScope": "in-repo", + "note": "Docs-shaped annotation, no consumer — the same disposition `flow.description` and `hook.label`/`description` carry and for the same reason: an author is not misled by a field that only claims to describe. Exempt from enforce-or-remove (ADR-0033)." + }, + "tags": { + "status": "dead", + "verifiedAt": "2026-08-10", + "evidenceScope": "in-repo", + "note": "The sharpest row in this file. Its describe() promises 'Tags for filtering and categorization (e.g. \"critical\", \"regression\", \"crm\")' and NOTHING filters on it: `os test` has exactly two flags, `--url` and `--token` (packages/cli/src/commands/test.ts:62-65), the runner never reads `scenario.tags`, and the only selection the command offers is the file glob. So `os test --tags critical` is not a narrower run, it is an unknown-flag error, and a suite tagged `regression` runs on every invocation. This is the entry that would carry `authorWarn` if the type had a channel for one (see `_authorWarnSkipped` in the file note): an author tagging scenarios is buying a filter that does not exist. Enforce-or-remove worklist — the enforce route is a `--tags` filter in the command, the remove route drops the key; either is a decision, not a cleanup." + }, + "setup": { + "status": "live", + "evidence": "packages/core/src/qa/runner.ts:41-54 (run before the main steps; a throw here aborts the scenario with 'Setup failed' and no step results)", + "verifiedAt": "2026-08-10", + "evidenceScope": "in-repo", + "note": "Array of TestStep — one level below the walk, measured in the same pass and recorded here rather than fanned into rows the gate would not check. Step keys: `name` LIVE (runner.ts:67, :76 — the label on every step result), `action` LIVE (:112, the only thing actually executed), `capture` LIVE (:118-121, writes result paths into the scenario variable context that `{{var}}` interpolation reads at :136), `assertions` LIVE (:125-128), `description` DEAD (docs-shaped, unread, kept). Action keys: `type` LIVE (http-adapter.ts:21 switch), `target` LIVE (:23-33), `payload` LIVE (:23-35), `user` LIVE (:17-18 — emitted as the `X-Run-As` header, so impersonation is only as real as the server's handling of that header). Assertion keys: `field` LIVE (runner.ts:160), `operator` LIVE (:164), `expectedValue` LIVE (:161). Two VALUE-level gaps, both loud rather than silent, and neither of them a key verdict (the api.json `type` precedent): `run_script` is in TestActionTypeSchema with no adapter branch and throws 'Unsupported action type', and the `not_contains`/`gt`/`gte`/`lt`/`lte`/`error` operators throw 'Unknown assertion operator'. The one genuinely silent path is `contains` against an actual that is neither array nor string (runner.ts:171-177), which falls through and PASSES — filed separately." + }, + "steps": { + "status": "live", + "evidence": "packages/core/src/qa/runner.ts:62-83 (the main sequence; stops at the first failing step)", + "verifiedAt": "2026-08-10", + "evidenceScope": "in-repo", + "note": "The required member — a scenario with an empty `steps` array parses, runs nothing, and reports passed. Step/action/assertion sub-keys are recorded on the `setup` entry above; all three arrays are the same TestStep surface executed by the same `runStep`." + }, + "teardown": { + "status": "live", + "evidence": "packages/core/src/qa/runner.ts:86-98 (runs even after a failed step, and does not mask an existing failure)", + "verifiedAt": "2026-08-10", + "evidenceScope": "in-repo", + "note": "Same TestStep surface as `setup`/`steps`. Worth knowing: a teardown throw only turns the scenario red when the scenario had otherwise PASSED (:92-95), so cleanup failures behind a real failure are swallowed on purpose." + }, + "requires": { + "status": "dead", + "verifiedAt": "2026-08-10", + "evidenceScope": "in-repo", + "note": "Declared environment preconditions — `requires.params` (environment variables) and `requires.plugins` (plugins that must be loaded) — that nothing checks. Neither the runner nor the CLI reads `scenario.requires`; there is no skip path and no precondition failure in the code at all, so a suite that declares `plugins: ['plugin-sharing']` runs unchanged against a server without it and fails later as an unexplained HTTP error. That is the misleading direction — the author reads it as a guard and gets none — so it belongs on the enforce-or-remove worklist beside `tags`, not in the docs-shaped bucket with `description`. Enforce route: check requirements before running and report the scenario as skipped-with-reason. Remove route: drop the block, since a precondition that is never checked is worse than an absent one." + } + } + } + } +} diff --git a/packages/spec/scripts/liveness/check-liveness.mts b/packages/spec/scripts/liveness/check-liveness.mts index 20dd444d1f..eccb6ba964 100644 --- a/packages/spec/scripts/liveness/check-liveness.mts +++ b/packages/spec/scripts/liveness/check-liveness.mts @@ -98,6 +98,7 @@ import { getMetadataTypeSchema, listMetadataTypeSchemaTypes } from '../../src/ke import { WebhookSchema } from '../../src/automation/webhook.zod'; import { QuerySchema } from '../../src/data/query.zod'; import { ValidationRuleSchema } from '../../src/data/validation.zod'; +import { TestSuiteSchema } from '../../src/qa/testing.zod'; import { BOUND_PROOF_PATHS, HIGH_RISK_CLASSES, @@ -145,7 +146,7 @@ const ledgerRoot = ledgerRootArg // Governed metadata types, rolled out highest-frequency / highest-risk first. // (`query` is not a metadata type — see SPEC_ONLY_SCHEMAS below.) -const GOVERNED = ['object', 'field', 'flow', 'action', 'hook', 'permission', 'position', 'agent', 'tool', 'skill', 'dataset', 'page', 'view', 'report', 'dashboard', 'webhook', 'query', 'datasource', 'app', 'book', 'doc', 'email_template', 'job', 'mapping', 'seed', 'translation', 'validation', 'api', 'capability']; +const GOVERNED = ['object', 'field', 'flow', 'action', 'hook', 'permission', 'position', 'agent', 'tool', 'skill', 'dataset', 'page', 'view', 'report', 'dashboard', 'webhook', 'query', 'datasource', 'app', 'book', 'doc', 'email_template', 'job', 'mapping', 'seed', 'translation', 'validation', 'api', 'capability', 'qa']; // Registered metadata types that are NOT yet governed — the coverage ratchet. // @@ -209,10 +210,24 @@ const PENDING_GOVERNANCE: Record = { // and update — so the ledger must keep governing `ValidationRuleSchema`; it is // the kind, not the schema, that went away. Governing it here is what stops the // retirement from quietly un-governing a live surface. +// +// `qa` is not a metadata type either — `TestSuiteSchema` is the FILE surface of +// the shipped `os test` command: an author writes `qa/*.test.json`, the CLI +// loads it, and `TestRunner`/`HttpTestAdapter` execute it. #6247 filed it as +// declared-but-inert on a grep that scanned only `*Schema` identifiers; the +// consumers read the TYPE names (`QA.TestSuite`, `QA.TestStep`, `QA.TestAction`), +// so the grep missed an entire execution chain and the 2026-08-07 retire ruling +// was withdrawn on 2026-08-08 in favour of enforce. Governing it here is the +// half of that ruling that keeps the surface honest going forward: the runner +// reads a real, measured subset of the declared keys, and the ones nothing reads +// (`suite.name`, `scenario.tags`, `scenario.requires`) are now recorded as such +// instead of being invisible. Like `query`, there is no registry to fold it back +// onto — the override IS its governance. const SPEC_ONLY_SCHEMAS: Record = { webhook: WebhookSchema, query: QuerySchema, validation: ValidationRuleSchema, + qa: TestSuiteSchema, }; // ADR-0010 provenance/lock overlay fields — system-stamped, on every type; auto-live. From 799b991c5b9c1613fa0f4cb448d76826eb273e2b Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 10 Aug 2026 04:02:06 +0000 Subject: [PATCH 2/2] docs(cli): state that `os test` validates each suite before running it (#6247) The load-site parse is user-visible behaviour: a suite that does not match `TestSuiteSchema` is now refused before it runs, named, and counted as one failed suite while the rest of the glob continues. The `os test` section listed the flags and said nothing about what happens to a bad file. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Up7rAGwREEy754haLKVtZH --- content/docs/deployment/cli.mdx | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/content/docs/deployment/cli.mdx b/content/docs/deployment/cli.mdx index 35269cd3d7..d74ed0e53e 100644 --- a/content/docs/deployment/cli.mdx +++ b/content/docs/deployment/cli.mdx @@ -1020,6 +1020,12 @@ os test --url http://localhost:4000 # Custom server URL os test --token my-api-key # With authentication ``` +Each file is validated against `TestSuiteSchema` **before it runs**. A suite that +does not match is refused at load time, naming the file and every offending path, +and counts as one failed suite — the rest of the glob still runs. This is what +stops a malformed suite from reporting success: a misspelled `steps` key used to +produce a scenario that passed having executed nothing. + #### `os doctor` Checks your development environment and reports issues: