fix(spec,cli): govern the QA testing domain and enforce TestSuiteSchema at the os test load site (#6247)

.changeset/qa-testing-liveness-enforce.md

@@ -0,0 +1,47 @@
+---
+"@objectstack/spec": minor
+"@objectstack/cli": patch
+---
+
+fix(spec,cli): govern the QA testing domain and enforce `TestSuiteSchema` at the `os test` load site (#6247)
+
+`packages/spec/src/qa/testing.zod.ts` declares the Quality Protocol — test
+suites, scenarios, steps, actions, assertions — and had no liveness ledger, so
+the ADR-0049 enforce-or-remove machinery had never looked at it. #6247 filed it
+as **declared-but-inert on zero runtime consumers**, and that reading was wrong:
+the grep behind it matched only `*Schema` identifiers, while every consumer here
+reads the **type** names (`QA.TestSuite`, `QA.TestStep`, `QA.TestAction`). What
+it missed is a complete execution chain — core's `TestRunner` and
+`HttpTestAdapter` (whose `action.type` switch labels *are* the
+`TestActionTypeSchema` values), published through `export * as QA`, driven by the
+shipped, documented CLI command `os test`. The retire ruling that followed from
+the bad reading was withdrawn; this is the enforce leg.
+
+**The real gap was narrower and genuine.** The type was the contract and the
+schema had no `parse` site anywhere in the platform: `os test` loaded suites with
+`JSON.parse(content) as QA.TestSuite`, next to the schema author's own
+`// Should validate with Zod`. A type assertion checks nothing at runtime, so a
+malformed suite failed late and in the wrong place — a missing `scenarios`
+TypeError'd inside the runner with no idea which file it came from, a misspelled
+`steps` key reported the scenario **passed** having executed nothing, and a bad
+`action.type` died in the HTTP adapter's `default:` branch mid-run, after earlier
+steps had already written records. `os test` now parses at the load boundary and
+refuses a bad suite there, naming the file, listing the issues and quoting the
+expected shape; a refusal counts as one failed suite rather than killing the run.
+Valid suites load and execute unchanged.
+
+**`packages/spec/liveness/qa.json`** seeds the ledger, governed through the same
+`SPEC_ONLY_SCHEMAS` override as `query`/`webhook`/`validation` — a QA suite is a
+file an author writes, not stack metadata, so there is no registry to fold it
+onto and the override *is* its governance. Four live rows (`scenarios.id`,
+`.setup`, `.steps`, `.teardown`) carry `file:line` evidence into the runner;
+step, action and assertion keys sit below the gate's one-level walk and their
+measurements are recorded in the notes rather than fanned into rows the gate
+would not check. Five dead rows are recorded honestly, two of which go onto the
+enforce-or-remove worklist: `scenarios.tags` advertises filtering that `os test`
+has no flag to express, and `scenarios.requires` declares param/plugin
+preconditions nothing checks, so a suite naming an absent plugin runs anyway and
+fails later as an unexplained HTTP error. None is marked `authorWarn`, and the
+omission is deliberate — the author-side lint walks stack collections, a QA suite
+belongs to no stack, and a warn flag that can never be emitted would be a silent
+no-op inside the mechanism built to catch them.

content/docs/deployment/cli.mdx

@@ -1020,6 +1020,12 @@ os test --url http://localhost:4000    # Custom server URL
 os test --token my-api-key            # With authentication
 ```
 
+Each file is validated against `TestSuiteSchema` **before it runs**. A suite that
+does not match is refused at load time, naming the file and every offending path,
+and counts as one failed suite — the rest of the glob still runs. This is what
+stops a malformed suite from reporting success: a misspelled `steps` key used to
+produce a scenario that passed having executed nothing.
+
 #### `os doctor`
 
 Checks your development environment and reports issues:

packages/cli/src/commands/test.ts

@@ -6,6 +6,7 @@ import path from 'path';
 import fs from 'fs';
 import { QA as CoreQA } from '@objectstack/core';
 import * as QA from '@objectstack/spec/qa';
+import type { ZodError } from 'zod';
 
 /**
  * Resolve a glob-like pattern to matching file paths.
@@ -52,6 +53,59 @@ function resolveGlob(pattern: string): string[] {
     .filter(fullPath => fs.statSync(fullPath).isFile());
 }
 
+/** The suite shape, quoted back at an author whose file did not match it. */
+const SUITE_SHAPE =
+  '{ "name": string, "scenarios": [ { "id", "name", "steps": [ { "name", "action": { "type", "target" } } ] } ] }';
+
+/**
+ * Load and VALIDATE one Quality Protocol suite file.
+ *
+ * This used to be `JSON.parse(content) as QA.TestSuite`, carrying the schema
+ * author's own `// Should validate with Zod`. The cast is the declared≠enforced
+ * gap ADR-0049 names (#6247): `TestSuiteSchema` was declared, shipped and
+ * documented, and had no `parse` site anywhere in the platform — the TYPE was
+ * the contract the runner read, and a type assertion checks nothing at runtime.
+ * What a bad file did instead of being refused: a missing `scenarios` TypeError'd
+ * inside `TestRunner.runSuite` with no idea which file it came from; a misspelled
+ * `steps` reported the scenario PASSED having executed nothing; a bad
+ * `action.type` reached the HTTP adapter's `default:` branch mid-run, after the
+ * earlier steps had already written records.
+ *
+ * So the parse happens HERE, at the boundary where the file name is still in
+ * hand, and the error names the file, lists the issues and prescribes the shape.
+ * Throws rather than exiting, so the caller keeps ownership of the run tally —
+ * one broken suite is a failed suite, not a dead command.
+ */
+export function loadTestSuite(file: string): QA.TestSuite {
+  const content = fs.readFileSync(file, 'utf-8');
+
+  let doc: unknown;
+  try {
+    doc = JSON.parse(content);
+  } catch (e) {
+    // A bare `Unexpected end of JSON input` names nothing; with a glob expanding
+    // to a dozen files that is a message you have to bisect by hand.
+    throw new Error(
+      `${file} is not valid JSON: ${e instanceof Error ? e.message : String(e)}\n` +
+        `  A Quality Protocol suite is a JSON document shaped ${SUITE_SHAPE}`,
+    );
+  }
+
+  const result = QA.TestSuiteSchema.safeParse(doc);
+  if (!result.success) {
+    const issues = (result.error as ZodError).issues
+      .map((issue) => `  ✗ ${issue.path.join('.') || '(root)'}: ${issue.message}`)
+      .join('\n');
+    throw new Error(
+      `${file} is not a valid Quality Protocol suite (TestSuiteSchema):\n${issues}\n` +
+        `  Expected shape: ${SUITE_SHAPE}\n` +
+        `  Reference: content/docs/references/qa/testing.mdx`,
+    );
+  }
+
+  return result.data as QA.TestSuite;
+}
+
 export default class Test extends Command {
   static override description = 'Run Quality Protocol test scenarios against a running server';
 
@@ -93,12 +147,22 @@ export default class Test extends Command {
 
     for (const file of testFiles) {
         console.log(`\n📄 Running suite: ${chalk.bold(path.basename(file))}`);
+
+        // Load and validate FIRST, and report a refusal on its own terms: a file
+        // the schema rejects never had a chance to run, so folding it into the
+        // run-failure branch below would report it as if the server had said no.
+        let suite: QA.TestSuite;
+        try {
+            suite = loadTestSuite(file);
+        } catch (e) {
+            console.error(chalk.red(e instanceof Error ? e.message : String(e)));
+            totalFailed++; // Count suite failure
+            continue;
+        }
+
         try {
-            const content = fs.readFileSync(file, 'utf-8');
-            const suite = JSON.parse(content) as QA.TestSuite; // Should validate with Zod
-
             const results = await runner.runSuite(suite);
-            
+
             for (const result of results) {
                 const icon = result.passed ? '✅' : '❌';
                 console.log(`  ${icon} Scenario: ${result.scenarioId} (${result.duration}ms)`);
@@ -117,7 +181,7 @@ export default class Test extends Command {
                 }
             }
         } catch (e) {
-            console.error(chalk.red(`Failed to load or run suite ${file}: ${e}`));
+            console.error(chalk.red(`Failed to run suite ${file}: ${e}`));
             totalFailed++; // Count suite failure
         }
     }

packages/cli/test/qa-suite-schema-load.test.ts

@@ -0,0 +1,113 @@
+// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * PIN (#6247) — `os test` REFUSES a malformed `qa/*.test.json` at LOAD time.
+ *
+ * The load site used to be `JSON.parse(content) as QA.TestSuite`, with the
+ * schema author's own `// Should validate with Zod` sitting next to it. A type
+ * assertion is not a check: the cast made every JSON document on disk a
+ * `TestSuite` as far as the compiler was concerned, and the first thing that
+ * noticed otherwise was the runner, several layers down and with no idea which
+ * file it came from. The three failure shapes that produced:
+ *
+ *   - `{}` (no `scenarios`) → `TestRunner.runSuite` iterates `undefined` and
+ *     throws a TypeError attributed to the runner, not the file;
+ *   - `{ scenarios: [] }` with a typo'd key → the suite reports SUCCESS having
+ *     executed nothing, which is the dangerous one: a broken suite that passes
+ *     is indistinguishable from a green one in CI;
+ *   - a bad `action.type` → survives the load, survives the runner, and dies in
+ *     the HTTP adapter's `default:` branch mid-run, after the preceding steps
+ *     have already written records.
+ *
+ * An AI-authored suite hits all three routinely, which is exactly the
+ * declared≠enforced gap ADR-0049 names: `TestSuiteSchema` was declared, shipped,
+ * documented — and had no `parse` site anywhere in the platform.
+ *
+ * So the pin is on the LOAD boundary, not on the runner: a document that
+ * `TestSuiteSchema` rejects must never reach `runSuite`, and the refusal must
+ * name the file and the issues. Valid suites must still load unchanged.
+ */
+
+import { describe, it, expect } from 'vitest';
+import { mkdtempSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { loadTestSuite } from '../src/commands/test';
+
+const dir = mkdtempSync(join(tmpdir(), 'os-qa-suite-'));
+
+function suiteFile(name: string, body: string): string {
+  const file = join(dir, name);
+  writeFileSync(file, body, 'utf-8');
+  return file;
+}
+
+const VALID_SUITE = {
+  name: 'crm smoke',
+  scenarios: [
+    {
+      id: 'create-account',
+      name: 'Create an account',
+      steps: [
+        {
+          name: 'create',
+          action: { type: 'create_record', target: 'accounts', payload: { name: 'Acme' } },
+          assertions: [{ field: 'id', operator: 'not_null', expectedValue: null }],
+        },
+      ],
+    },
+  ],
+};
+
+describe('os test — suite load is schema-checked (#6247)', () => {
+  it('loads a valid suite unchanged', () => {
+    const file = suiteFile('valid.test.json', JSON.stringify(VALID_SUITE));
+    const suite = loadTestSuite(file);
+    expect(suite.name).toBe('crm smoke');
+    expect(suite.scenarios).toHaveLength(1);
+    expect(suite.scenarios[0].steps[0].action.type).toBe('create_record');
+  });
+
+  it('refuses a suite with no `scenarios` — the shape that TypeErrors inside the runner', () => {
+    const file = suiteFile('no-scenarios.test.json', JSON.stringify({ name: 'empty' }));
+    expect(() => loadTestSuite(file)).toThrow(/no-scenarios\.test\.json/);
+    expect(() => loadTestSuite(file)).toThrow(/scenarios/);
+  });
+
+  it('refuses a misspelled step key — the shape that SILENTLY passes', () => {
+    // `stepz` instead of `steps`: the cast let this through and the scenario
+    // reported PASSED having run nothing at all.
+    const file = suiteFile(
+      'typo.test.json',
+      JSON.stringify({ name: 's', scenarios: [{ id: 'a', name: 'A', stepz: [] }] }),
+    );
+    expect(() => loadTestSuite(file)).toThrow(/typo\.test\.json/);
+  });
+
+  it('refuses an unknown action type — the shape that dies mid-run after writes', () => {
+    const bad = structuredClone(VALID_SUITE) as Record<string, any>;
+    bad.scenarios[0].steps[0].action.type = 'summon_record';
+    const file = suiteFile('bad-action.test.json', JSON.stringify(bad));
+    expect(() => loadTestSuite(file)).toThrow(/bad-action\.test\.json/);
+  });
+
+  it('names the offending file and the issues, and prescribes the shape', () => {
+    const file = suiteFile('broken.test.json', JSON.stringify({ scenarios: 'not an array' }));
+    let message = '';
+    try {
+      loadTestSuite(file);
+    } catch (e) {
+      message = e instanceof Error ? e.message : String(e);
+    }
+    expect(message).toContain('broken.test.json');
+    expect(message).toContain('TestSuiteSchema');
+    // Self-prescribing: the author is told what a suite looks like, not just
+    // that theirs is wrong.
+    expect(message).toContain('scenarios');
+  });
+
+  it('refuses invalid JSON with the file named, rather than a bare SyntaxError', () => {
+    const file = suiteFile('not-json.test.json', '{ "name": ');
+    expect(() => loadTestSuite(file)).toThrow(/not-json\.test\.json/);
+  });
+});

packages/spec/liveness/README.md

@@ -645,7 +645,14 @@ The governed set is `GOVERNED` at the top of `check-liveness.mts`. To add a type
    RecordDetailView had been gating the History tab on it the whole time (#2707).
 4. Add the type to `GOVERNED`; confirm the gate is green.
 
-## Current state — 27 governed types (complete registry coverage)
+## Current state — 30 governed types (complete registry coverage)
+
+> The table below carries 28 of the 30. `api` and `capability` are governed
+> (they are in `GOVERNED`, they have ledgers, the gate counts them) and were
+> added without a row here — the table fell behind its own registry, which is
+> the shape this file keeps warning about one level down. Filed rather than
+> back-filled from a guess: writing two Notes cells for changes somebody else
+> measured is exactly the fabrication the drill section forbids.
 
 **The counting method for this table is the gate's own report** —
 `check-liveness.mts --json`, `types.<type>.byStatus` — decided in #4488 after
@@ -698,6 +705,7 @@ for t, v in r['types'].items():
 | mapping | 14 | – | 0 | – | seeded 2026-08-01 (#4488) at 8/11 live; **0 dead since #4509** retired the three that were not. The import half (#2611) is loudly enforced — unsupported transforms/formats are 400s, `mode`/`upsertKey` default the request, the wizard picker renders `label`. RETIRED 17.0.0: `extractQuery` (authorWarn — "for export only" promised an export path no exporter implements) + `errorPolicy`/`batchSize`, which were dead AND **unwarnable** (schema defaults materialize at parse, so presence ≠ authored — `_authorWarnSkipped`, the non-boolean instance of the default(true) rule). That unwarnability is why they went out in the 17.0.0 window rather than after a deprecation cycle: removal was the only channel that could ever reach the author. Rows DELETED, not tombstoned — MappingSchema is strict, so the keys left the walked shape |
 | seed | 5 | – | 0 | – | seeded 2026-08-01 (#4488). Fully live via SeedLoaderService on both doors (boot/per-org replay + runtime-draft publish). `records` is the z.record walk boundary: the keys an author writes are the target object's fields, governed by that object's own definitions — recorded in the entry, not silently skipped |
 | translation | 17 | – | 2 | – | seeded 2026-08-01 (#4488) — after fixing the walker: the registered schema is a z.preprocess pipe (#3778 retired-dialect guard) whose transform side the unwrap always took, so the type was literally unwalkable. 10 of 11 groups live across spec resolvers, REST localization, objectui client resolvers and plugin-audit (whose composed-key `t()` calls make `messages` easy to mis-verify as dead). Dead 1 = `validationMessages` (authorWarn): nothing resolves it, and #3778's own legacy-key migration table steers `errors:` authors into it — a shipped false signpost, the capabilities.readOnly shape | **#4667**: `validationMessages` REMOVED (row deleted) — removed from the shared translationDataShape(), so it retired at BOTH doors at once, closing the item-only asymmetry #3778's original guard had. #3778's own `errors` guidance was rewritten in the same change: it had been steering authors INTO this dead group. |
+| qa | 4 | – | 5 | – | seeded 2026-08-10 (#6247) — **not a metadata type**: `TestSuiteSchema` is the FILE surface of the shipped `os test` command (`qa/*.test.json`), governed through the same `SPEC_ONLY_SCHEMAS` override as `query`/`webhook`/`validation`. It is in the table as the clearest worked example of a **false `dead` measurement**: #6247 reported the whole domain declared-but-inert on a grep that scanned only `*Schema` identifiers, and every consumer here reads the **type** names (`QA.TestSuite`, `QA.TestStep`, `QA.TestAction`) — so an entire execution chain (core's `TestRunner` + `HttpTestAdapter`, published via `export * as QA`, driven by a documented CLI command) read as zero consumers, and a retire ruling was issued on it before being withdrawn. The `evidenceScope` table one section up says no amount of specifier matching is sufficient for a negative claim; this is the same lesson for **identifier** matching. What was really wrong was narrower and real: the type was the contract and the schema had no `parse` site, so the CLI's `JSON.parse(content) as QA.TestSuite` cast admitted anything — ENFORCED in the same change (`TestSuiteSchema.safeParse` at the load site, pinned). Dead 5 = `name` (the file name is the suite identity; the CLI prints `path.basename`), `scenarios.name` (describe() says "for test reports"; every report carries `scenarioId` instead), `scenarios.description` (docs-shaped, kept), and the two on the enforce-or-remove worklist — `scenarios.tags` promises filtering that `os test`'s two flags cannot express, and `scenarios.requires` declares param/plugin preconditions nothing checks, so a suite naming a missing plugin runs anyway and fails as an unexplained HTTP error. Neither carries `authorWarn` and the omission is deliberate (`_authorWarnSkipped`): the lint walks stack **collections**, a QA suite is a loose file in no stack, so a warn flag here would emit nothing — a silent no-op inside the mechanism built to catch silent no-ops |
 | validation | 15 | 0 | 3 | 0 | seeded 2026-08-01 (#4488). The ADR-0020 carrier: the evaluator honors active/events/priority/severity/type/condition/message (the zod header's "only reads type/condition/…" prose is STALE — trust the ledger). Dead 3 = label/description/tags, declared governance metadata, kept unmarked. Union walk boundary recorded: only base + `script` keys walked; per-variant keys are governed by the evaluator's tests, not ledger rows. **No longer a registered metadata kind** — #4509 retired it under ADR-0088 (a standalone rule had no object-binding key and every variant is `.strict()`, so it bound to nothing and gated no write; a state machine authored that way saved cleanly and did nothing). The rule VOCABULARY is untouched and fully live via `object.validations[]`, so the ledger keeps governing it through the gate's spec-only override, alongside `webhook` and `query`. The contrast with the two bridges in the same batch is the point: enforce-or-remove picked ENFORCE where the feature existed and only the wiring was missing, and REMOVE where the shape itself could not carry the feature |
 
 The `dead` set across types is the enforce-or-remove worklist (ADR-0049); every