fix(core): fail the QA contains assertion on a non-evaluable actual instead of silently passing (#7256)

[os-zhuang]

Closes #7256. Follows #6247 (PR #7255), which enforced TestSuiteSchema at the os test load site and is what makes this engine's verdicts load-bearing.

The defect

TestRunner.assert (packages/core/src/qa/runner.ts) handled contains for the two shapes it can evaluate and had no else:

case 'contains':
   if (Array.isArray(actual)) { … }
   else if (typeof actual === 'string') { … }
   break;   // ← everything else falls out here, throwing nothing

undefined (the common case: a typo'd field path, or a response shape that moved), null, a number, a boolean or an object fell straight out of the switch and the assertion reported PASSED. A scenario asserting { field: "body.data.items", operator: "contains", expectedValue: "acme" } against a response with no body.data.items at all reported ✅ — the assertion that was supposed to be the test is the thing that silently disappears, and CI believes the green.

contains was the only path in this engine that could decide "no comparison applies here" and report success. Every other unhandled shape already fails loud: an operator with no branch throws Unknown assertion operator, an action type with no adapter branch throws Unsupported action type in HttpAdapter, and equals/not_equals/is_null/not_null compare unconditionally. This closes the asymmetry rather than adding a new posture.

The fix

An assertion the engine cannot evaluate is a failed assertion (issue option 1). The message is written for the author who has to act on it — it names the field, the operator and the runtime type the path actually resolved to, then says which of the two things is wrong:

Assertion failed: body.data.items cannot be evaluated by 'contains' — expected an
array or a string at that path, got undefined. The path resolved to nothing — the
field is absent from the result, or the path is misspelled. Use 'is_null' if
asserting absence is what you meant.

• undefined / null → the path did not resolve, so the fixture (the field path, or the response shape it was written against) is the suspect.
• number / boolean / object → the path resolved fine, so the assertion picked an operator that does not apply: "contains tests array membership and string substrings only. Use equals to compare a scalar, or point the field at the array or string you meant to look inside."

null and arrays are named as themselves rather than typeof's object, because those are exactly the two shapes a contains author needs told apart from a plain record.

Option 2 from the issue (String(actual).includes(…)) is not taken: it would make undefined "contain" the substring "defi", trading one false green for a stranger one.

The os test section of content/docs/deployment/cli.mdx already explained what stops a malformed suite from reporting success; it now also states the assertion-engine half of the same rule.

Blast radius — measured, not assumed

Zero in-tree cases were passing vacuously. os test is the runner's only consumer in the repository, and there are no Quality Protocol suite documents to run.

Run A — the default glob on the loud build, from the repo root and each of the three example apps: No test files found matching: qa/*.test.json, exit 0 in all four. None of the apps has a qa/ directory; all three run vitest run for pnpm test.

Run B — the repo-wide sweep. Every *.test.json in the tree (node_modules excluded) is three tsconfig.test.json files; fed to os test explicitly, each is refused at the load site by #6247's TestSuiteSchema parse. The only JSON with a top-level scenarios key is packages/spec/liveness/qa.json, the ledger. packages/qa/{dogfood,downstream-contract,http-conformance} are vitest suites that never touch TestRunner. No workflow invokes os test.

Run C — positive control, so "zero suites" is not an untested claim about the loud path. Same fixture and the real HttpTestAdapter, against a stand-in server returning {"ok":true,"data":{"total":0},"name":"acme corp","tags":["acme","globex"]}:

scenario	base f3f855ac	this PR
contains on data.items — path absent	✅ (asserting nothing)	❌ got undefined
contains on body.data.items — the documented body.* convention	✅ (asserting nothing)	❌ got undefined
contains on data.total — a number	✅ (asserting nothing)	❌ got number
contains on tags — a real array	✅	✅
contains on name — a real string	✅	✅
suite verdict	SUCCESS: All 5 scenarios passed.	FAILED: 3 scenarios failed. 2 passed.

Exactly the three vacuous scenarios flip; both evaluable controls stay green.

Nothing in-repo relied on silent-pass as "field may be absent" semantics, so there was no semantics decision to escalate and no fixture to repair. Downstream suites are what will see red, and every case they see is a test that was never running.

Two adjacent defects found while measuring, filed rather than folded (different files):

• finding: os test with a ** glob OOMs — resolveGlob readdir-recurses the whole tree including node_modules #7363 — os test '**/*.test.json' OOMs; resolveGlob readdir-recurses node_modules.
• finding: the QA assertion field path is documented as body.*, but HttpTestAdapter returns the parsed body with no body wrapper — every documented path resolves to undefined #7365 — TestAssertionSchema.field documents body.* paths, but HttpTestAdapter returns the parsed body with no body wrapper, so every documented path resolves to undefined. That is the body.data.items row above; this PR makes it visible rather than causing it.

Sibling-operator sweep

Same file, per the card. contains was the only type-switched assertion, and the only one carrying this defect:

operator	shape	verdict
equals, not_equals	unconditional compare	no hole
is_null, not_null	unconditional compare	no hole
contains	type-switched, no else	the defect — fixed here
not_contains, gt, gte, lt, lte, error	declared in TestAssertionTypeSchema, no branch in the runner	already refused loudly at default: — annoying but honest, not this defect

The six declared-but-unimplemented operators are left as they are: implementing them is a feature decision, not this cleanup. They are pinned in the new tests so a later implementation is deliberate rather than accidental.

Tests

New packages/core/src/qa/runner.test.ts, 17 cases driven through the public runSuite surface:

• each non-evaluable shape (undefined at the leaf, undefined for the whole response, null, number, boolean, object) fails, and the message names the type and the right suspect;
• the two evaluable shapes keep today's behaviour in both directions — array match passes / array miss fails with its existing message, string match passes / string miss fails — plus the empty-array and empty-string edges;
• the sibling operators are unchanged, including is_null still passing on a missing path.

Reverse-verified: reverting runner.ts alone fails 7 of the 17 and passes the other 10, so the behaviour pins constrain the fix without over-constraining the unchanged paths. Full @objectstack/core suite green (744 tests / 30 files).

Changeset

@objectstack/core: minor, read off scripts/check-changeset-no-major.mjs: every publishable package is in the Changesets fixed group, so one major promotes the whole ~70-package stack; during the launch window breaking changes ship as minor under pre-1.0 semantics, and the gate enforces it. This is a breaking behaviour change of published @objectstack/core (vacuous green → red), so it is not a patch. No ADR-0087 disposition trio is required — check-adr-0087-registration confirms the diff adds no declared-breaking changeset, and packages/spec is untouched.

Gates run locally

70 workflow gates enumerated fresh (64 check:* in lint.yml + the 3 in spec-liveness-check.yml + check:console-sha, check:objectui-pin-fresh, check:adr-links). Run in the worktree on a fully built workspace:

• 49/49 pass in the lint.yml quality-gates batch (including lint, all changeset gates, check:wildcard-fallthrough, check:published-files, check:type-check-coverage).
• 15/15 pass in the @objectstack/spec batch (tsc --noEmit, check:generated, check:docs, check:liveness, check:api-surface, check:exported-any, check:dual-source-exports, …).
• 4/4 pass in the typecheck batch (workspace packages, check:type-check-debt, example apps, downstream-contract), plus @objectstack/driver-sql tests and @objectstack/spec analyze.

Two gates went red on first run and both reproduce on an untouched worktree at the base commit f3f855ac, so neither is this diff:

• check:i18n-coverage — a build-prerequisite gate; it needs the whole workspace built. Green after pnpm build (12 configs, 660 baselined untranslated strings, none new).
• check:platform-checklist — coverage.json · qa: UNCLASSIFIED, because fix(spec,cli): govern the QA testing domain and enforce TestSuiteSchema at the os test load site (#6247) #7255 seeded packages/spec/liveness/qa.json and the checklist has no qa row yet. Already filed as finding: check:platform-checklist is red on main — the new qa liveness ledger is neither mapped nor waived in coverage.json #7347; the gate runs on a manual cadence, not in CI.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
objectstack	Ignored		Aug 10, 2026 8:01am

[github-actions]

📓 Docs Drift Check

This PR changes 1 package(s): @objectstack/core.

19 hand-written doc(s) reference the affected code and may need an implementation-accuracy re-verification:

• content/docs/ai/actions-as-tools.mdx (via @objectstack/core)
• content/docs/ai/knowledge-rag.mdx (via @objectstack/core)
• content/docs/ai/natural-language-queries.mdx (via @objectstack/core)
• content/docs/automation/webhooks.mdx (via @objectstack/core)
• content/docs/concepts/north-star.mdx (via packages/core)
• content/docs/deployment/migration-from-objectql.mdx (via @objectstack/core)
• content/docs/kernel/contracts/index.mdx (via @objectstack/core)
• content/docs/kernel/runtime-services/examples.mdx (via @objectstack/core)
• content/docs/kernel/services-checklist.mdx (via @objectstack/core)
• content/docs/kernel/services.mdx (via @objectstack/core)
• content/docs/permissions/authentication.mdx (via @objectstack/core)
• content/docs/permissions/authorization.mdx (via packages/core)
• content/docs/plugins/anatomy.mdx (via @objectstack/core)
• content/docs/plugins/development.mdx (via @objectstack/core)
• content/docs/plugins/index.mdx (via @objectstack/core)
• content/docs/plugins/packages.mdx (via @objectstack/core)
• content/docs/protocol/kernel/index.mdx (via @objectstack/core)
• content/docs/protocol/kernel/lifecycle.mdx (via @objectstack/core)
• content/docs/protocol/kernel/plugin-spec.mdx (via @objectstack/core)

⛔ 4 release-owned page(s) also reference the affected code. These are read-only:

• content/docs/releases/implementation-status.mdx (via @objectstack/core)
• content/docs/releases/v12.mdx (via @objectstack/core)
• content/docs/releases/v15.mdx (via @objectstack/core)
• content/docs/releases/v17.mdx (via @objectstack/core)

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

Advisory only. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs origin/main → pass the list as args.docs.