diff --git a/.github/prompts/check-surface.prompt.md b/.github/prompts/check-surface.prompt.md new file mode 100644 index 00000000000..c68101431d6 --- /dev/null +++ b/.github/prompts/check-surface.prompt.md @@ -0,0 +1,104 @@ +--- +description: "Orchestrate surface checks: one parallel deterministic batch, then AI for the rest" +mode: "agent" +--- + +You are the **orchestrator** for surface checks on an emitter's generated +SDK — properties (naming, visibility/access, shape) the wire tests can't see. +You verify every check in **two parallel phases**, not one-by-one: + +1. **Deterministic phase (parallel, no AI).** One batch call runs every check + whose category has a routine — concurrently. +2. **AI phase (only the leftovers).** You verify the few checks with no + deterministic routine; dispatch them in parallel if there are several. + +Read: + +- `packages/${input:emitter}/eng/scripts/client-criteria/context.md` — Emitter + facts (language, generated-root, flavors, checks-doc path) and concept prose. +- The **checks doc** — the `checks-doc` Markdown table named in `context.md` + (default `surface-checks.json`): the batch runner reads it for deterministic + checks and you read its rows' `doc` prose for the AI ones. Use this path unless the caller + explicitly overrides it. +- `packages/${input:emitter}/eng/scripts/client-criteria/verifiers.json` — the + routing table, **authored once per emitter (by you or a human) — never + generated by the emitter build.** A category listed here is decided without + AI, either by a `check` (a declarative pattern the runner matches) or by a `test` + (defer to an existing emitter test that already asserts this surface, e.g. a + Java reflection test). A category not listed is verified by AI. Even with **no** + verifiers.json at all you can still verify every check from `context.md` + + the checks doc — verifiers.json only moves work from AI to code. + +Inputs: + +- Emitter: ${input:emitter:package folder, e.g. http-client-python} +- Flavor: ${input:flavor:one of the flavors in the emitter's context.md} +- Scenario: ${input:scenario:optional substring filter on the check id} + +## Phase 1 — deterministic batch (parallel) + +Run the shared runner **once**. It is a single language-agnostic tool +(`tsp-spector verify-surface-checks`) that runs every check whose category has a +verifier — in parallel, no AI — and prints `{"results": [...], "needs_ai": [...]}`: + +``` +npx tsp-spector verify-surface-checks \ + --checks \ + --verifiers packages/${input:emitter}/eng/scripts/client-criteria/verifiers.json \ + --generated-root \ + --flavor ${input:flavor} --language +``` + +Take `results` as-is — each is a verdict decided without AI: `how: deterministic` +(a declarative pattern matched the generated code), `how: test`, +`status: deferred` (a real emitter test already asserts this surface; the +`evidence` cites it), or `how: not-applicable`, `status: not-applicable` (the +emitter declared this category N/A for its language in `verifiers.json`). Do not +re-check or second-guess them. + +## Phase 2 — AI verification (the residual, in parallel) + +For each item in `needs_ai`, verify it yourself: find the target in the generated +package under `//` (folder matching the scenario), judge +against its `verify` text and context.md, and cite the exact file + line. Mark +`how: ai`. **If there are several, dispatch them as parallel subagents** (one per +item) rather than serially, then collect the verdicts. + +## Phase 3 — merge + +Combine `results` + your AI verdicts into one table, filtered by the scenario +input if given: + +`id | category | result (PASS/FAIL/N/A/DEFERRED) | how (deterministic/test/ai) | evidence` + +Render `status: deferred` rows as `DEFERRED` with `how: test` — they are covered +by the referenced test rather than checked here. Render `status: not-applicable` +rows as `N/A` with `how: not-applicable` — the category has no surface in this +language. + +End with a summary: pass/fail counts, and how many were code vs test vs AI. + +## Phase 4 — propose promotions (optional; only write when asked) + +For every item you had to verify with **AI** in Phase 2, consider whether it +could be decided without AI next time, and if so propose a `verifiers.json` +entry for its category. Two forms: + +- **`check`** — the emitter's `context.md` describes the concept concretely + enough to be a mechanical file check. Propose a declarative pattern entry + `{"": {"files": ["…"], "find": "…{target}…", "expect": "…"}}` + (placeholders/casing/expect as documented in `verifiers.json`). +- **`test`** — an existing test in the emitter repo already asserts this surface + (e.g. a reflection/`assert`-based test). Propose + `{"": {"test": "", "note": "…"}}` so the + check defers to that test instead of AI. + +Print the proposed JSON entries under a **"Suggested verifiers.json additions"** +heading with a one-line rationale each. verifiers.json is authored, rarely +changes, and is never generated by the emitter — so **only write the file if the +caller explicitly asks**; otherwise leave the suggestions for review. A verify +run is read-only by default. + +The point: the bulk is decided without AI — a deterministic routine or an existing test — +and AI is spent only on the checks that actually need judgment, shrinking further +as verifiers.json is filled in. diff --git a/packages/http-client-python/eng/scripts/client-criteria/context.md b/packages/http-client-python/eng/scripts/client-criteria/context.md new file mode 100644 index 00000000000..a854aa7292e --- /dev/null +++ b/packages/http-client-python/eng/scripts/client-criteria/context.md @@ -0,0 +1,75 @@ +# Client surface checks — context: http-client-python + +## Emitter facts + +_The shared `/check-surface` prompt and `tsp-spector verify-surface-checks` read these._ + +- **language:** python _(selects which per-language client name applies when a check is language-scoped)_ +- **generated-root:** `packages/http-client-python/tests/generated/` +- **flavors:** `azure`, `unbranded` +- **checks-doc:** `packages/http-specs/surface-checks.json` _(generated from `@surfaceDoc`, ships with the specs)_ + + + +### What differs by flavor (branded = azure, unbranded) + +Only the **generated-root** subfolder and the **import namespace** (`azure.core` +vs `corehttp`). Symbol locations and every concept description below are +flavor-invariant. + +## What each check looks like + +What each verifiable concept looks like in generated Python. Authored once, +reused across every scenario that uses the concept. Prefer concrete, checkable +verifiers so the shared deterministic runner can decide **without calling AI** — +see `verifiers.json` for the machine-readable declarative form (file globs + +regex patterns + casing) consumed by `tsp-spector verify-surface-checks`. + +### access: internal + +- **enum** → the class is NOT re-exported from `models/__init__.py`; it may also + be defined with a leading `_`. +- **model** → absent from the package's public `__init__.py` exports. +- **operation / method** → prefixed `_`, or not present on the public client. + +### access: public + +- present in the public `__init__.py` exports, no leading underscore. + +### naming + +- the generated identifier equals the `expected` value from `@surfaceDoc` + **recast to Python's idiomatic casing for the subject's `kind`** + (case-sensitive): `enum`/`model`/`type` → `PascalCase`, + `property`/`parameter`/`operation` → `snake_case`, enum value → + `UPPER_SNAKE`. The per-kind casing map lives in `verifiers.json` + (`{expected:byKind}`); the `@surfaceDoc` carries the subject's `kind`. +- **exact per-language names:** when a name can't be derived by casing (a + reserved-word escape, an acronym, a hard rename), the spec author supplies a + `scope → value` dict on `@surfaceDoc` (e.g. `#{ python: "io_thing" }`). Those + checks carry a `scope` column; the engine only runs the ones scoped to + `python` and matches `{expected}` **verbatim**. The same `naming` routine + handles both — no `verifiers.json` change. + +### flatten + +- the nested model's fields appear directly as attributes on the parent class; + `expected` names the promoted property (blank = assert inlined). + +### client-location + +- an operation group is a class under `operations/_operations.py`; a method is a + public `def` on it; the **root client** is the main `*Client` class. An + operation is "on" a client/group when its snake_case name is a method of the + matching class. The `expected` value names the client/group where the method + should appear. + +### hierarchy + +- a client subtype is expressed as the base in `class (, …)` + in `models/_models.py`; inherited members are attributes present on the + subclass. + +> When an assertion has no concrete verifier here (and none in verifiers.json), +> `/check-surface` may fall back to judgment — but add a verifier when +> you can, to keep AI calls rare. diff --git a/packages/http-client-python/eng/scripts/client-criteria/verifiers.json b/packages/http-client-python/eng/scripts/client-criteria/verifiers.json new file mode 100644 index 00000000000..ed474786304 --- /dev/null +++ b/packages/http-client-python/eng/scripts/client-criteria/verifiers.json @@ -0,0 +1,25 @@ +{ + "_comment": "Declarative routing table for the shared `tsp-spector verify-surface-checks` engine. AUTHORED once per emitter — NOT generated. The engine substitutes placeholders into `find`/`scope`: {target}=the subject name (from @surfaceDoc), {expected}=the expected client-surface output. Add `:casing` to recase, e.g. {expected:snake}, {target:pascal}. Use `:byKind` to apply per-kind casing from the `casing` map (keyed by `details.kind`). `expect`: present (default), absent, {absentWhen:}, or {whenExpected:{:present|absent}}. SCOPE: when a check is scoped to a language, the engine only runs it for the matching language and matches {expected} VERBATIM (casing modifiers are ignored).", + "naming": { + "files": ["**/*.py"], + "find": "\\b{expected:byKind}\\b", + "casing": { + "type": "pascal", + "model": "pascal", + "enum": "pascal", + "property": "snake", + "parameter": "snake", + "operation": "snake", + "enumvalue": "upper_snake" + } + }, + "access": { + "files": ["**/models/__init__.py"], + "find": "\\b{target}\\b", + "expect": { "whenExpected": { "internal": "absent", "public": "present" } } + }, + "hierarchy": { + "files": ["**/models/_models.py"], + "find": "class\\s+{target}\\s*\\([^)]*\\b{expected}\\b" + } +} diff --git a/packages/http-specs/specs/payload/pageable/main.tsp b/packages/http-specs/specs/payload/pageable/main.tsp index db89029037c..e25ba2c733b 100644 --- a/packages/http-specs/specs/payload/pageable/main.tsp +++ b/packages/http-specs/specs/payload/pageable/main.tsp @@ -51,6 +51,11 @@ namespace ServerDrivenPagination { } ``` """) + @surfaceDoc(#{ + category: "paging", + expected: "Pet", + doc: "The paginated operation yields `Pet` elements.", + }) @route("/link") @list op link(): { @@ -518,6 +523,11 @@ namespace PageSize { } ``` """) + @surfaceDoc(#{ + category: "paging", + expected: "Pet", + doc: "Single-page pageable operation yielding `Pet` elements, no continuation.", + }) @route("/without-continuation") @list op listWithoutContinuation(): { @@ -556,6 +566,11 @@ namespace PageSize { } ``` """) + @surfaceDoc(#{ + category: "paging", + expected: "Pet", + doc: "Page-size-driven pageable operation yielding `Pet` elements.", + }) @route("/list") @list op listWithPageSize(@pageSize @query pageSize?: int32): { diff --git a/packages/http-specs/specs/serialization/encoded-name/json/main.tsp b/packages/http-specs/specs/serialization/encoded-name/json/main.tsp index 8d1d0e006f9..9c880129dd4 100644 --- a/packages/http-specs/specs/serialization/encoded-name/json/main.tsp +++ b/packages/http-specs/specs/serialization/encoded-name/json/main.tsp @@ -26,6 +26,13 @@ namespace Property { {"wireName": true} ``` """) + @surfaceDoc(#{ + category: "naming", + subject: "JsonEncodedNameModel.defaultName", + expected: "defaultName", + kind: "property", + doc: "The client property is named `defaultName` even though the wire name is `wireName`.", + }) @post op send(@bodyRoot body: JsonEncodedNameModel): NoContentResponse; diff --git a/packages/http-specs/specs/type/model/inheritance/not-discriminated/main.tsp b/packages/http-specs/specs/type/model/inheritance/not-discriminated/main.tsp index 69c186445be..730b8bcf69f 100644 --- a/packages/http-specs/specs/type/model/inheritance/not-discriminated/main.tsp +++ b/packages/http-specs/specs/type/model/inheritance/not-discriminated/main.tsp @@ -43,6 +43,12 @@ op postValid(@body input: Siamese): NoContentResponse; {"name": "abc", "age": 32, "smart": true} ``` """) +@surfaceDoc(#{ + category: "hierarchy", + subject: "Siamese", + expected: "Cat", + doc: "The `Siamese` model extends `Cat` as its base type on the client surface.", +}) @route("/valid") @get op getValid(): Siamese; diff --git a/packages/http-specs/specs/type/model/inheritance/single-discriminator/main.tsp b/packages/http-specs/specs/type/model/inheritance/single-discriminator/main.tsp index 519b9576535..316bd3464a4 100644 --- a/packages/http-specs/specs/type/model/inheritance/single-discriminator/main.tsp +++ b/packages/http-specs/specs/type/model/inheritance/single-discriminator/main.tsp @@ -65,6 +65,12 @@ model Fish { {"wingspan": 1, "kind": "sparrow"} ``` """) +@surfaceDoc(#{ + category: "hierarchy", + subject: "Sparrow", + expected: "Bird", + doc: "The `Sparrow` model extends `Bird` as its base type on the client surface.", +}) @get op getModel(): Bird; diff --git a/packages/http-specs/surface-checks.json b/packages/http-specs/surface-checks.json new file mode 100644 index 00000000000..523bbc116a9 --- /dev/null +++ b/packages/http-specs/surface-checks.json @@ -0,0 +1,65 @@ +{ + "items": [ + { + "id": "Payload_Pageable_PageSize_listWithoutContinuation_paging", + "scenario": "Payload_Pageable_PageSize_listWithoutContinuation", + "category": "paging", + "target": "listWithoutContinuation", + "details": { + "expected": "Pet" + }, + "doc": "Single-page pageable operation yielding `Pet` elements, no continuation." + }, + { + "id": "Payload_Pageable_PageSize_listWithPageSize_paging", + "scenario": "Payload_Pageable_PageSize_listWithPageSize", + "category": "paging", + "target": "listWithPageSize", + "details": { + "expected": "Pet" + }, + "doc": "Page-size-driven pageable operation yielding `Pet` elements." + }, + { + "id": "Payload_Pageable_ServerDrivenPagination_link_paging", + "scenario": "Payload_Pageable_ServerDrivenPagination_link", + "category": "paging", + "target": "link", + "details": { + "expected": "Pet" + }, + "doc": "The paginated operation yields `Pet` elements." + }, + { + "id": "Serialization_EncodedName_Json_Property_send_naming", + "scenario": "Serialization_EncodedName_Json_Property_send", + "category": "naming", + "target": "JsonEncodedNameModel.defaultName", + "details": { + "expected": "defaultName", + "kind": "property" + }, + "doc": "The client property is named `defaultName` even though the wire name is `wireName`." + }, + { + "id": "Type_Model_Inheritance_NotDiscriminated_getValid_hierarchy", + "scenario": "Type_Model_Inheritance_NotDiscriminated_getValid", + "category": "hierarchy", + "target": "Siamese", + "details": { + "expected": "Cat" + }, + "doc": "The `Siamese` model extends `Cat` as its base type on the client surface." + }, + { + "id": "Type_Model_Inheritance_SingleDiscriminator_getModel_hierarchy", + "scenario": "Type_Model_Inheritance_SingleDiscriminator_getModel", + "category": "hierarchy", + "target": "Sparrow", + "details": { + "expected": "Bird" + }, + "doc": "The `Sparrow` model extends `Bird` as its base type on the client surface." + } + ] +} diff --git a/packages/spector/docs/surface-checks.md b/packages/spector/docs/surface-checks.md new file mode 100644 index 00000000000..61d51a57031 --- /dev/null +++ b/packages/spector/docs/surface-checks.md @@ -0,0 +1,161 @@ +# Client surface checks for generated SDKs + +_Verify properties of generated SDKs that the wire tests can't observe — +described once in the spec, precomputed centrally, checked per emitter. Mostly +deterministic; AI only for the fuzzy residual._ + +Full design write-up: https://gist.github.com/iscai-msft/8be1b3ecf016bfd3bfbf3a7f6857d78c + +## What this is + +Spector verifies emitter parity on the **wire**. A large class of behavior never +reaches the wire — whether an internal type is actually hidden, a property was +renamed, a method relocated, a model flattened. Today those are checked by hand, +per language. + +**Client surface checks** close that gap. On a spec element that already carries +`@scenarioDoc`, you add a `@surfaceDoc` — an explicit, language-agnostic +assertion about the generated client surface: + +```typespec +@surfaceDoc(#{ + category: "naming", + expected: "defaultName", + subject: "Widget", // optional — defaults to target's name + kind: "property", // optional — enables per-kind casing in verifiers + doc: "Explanation...", // optional +}) +``` + +The `category` names what kind of surface property is asserted, `expected` is the +expected client-facing output, and `subject` identifies what's being checked +(defaults to the decorated element's name when omitted). `expected` is normally a +single canonical string; when a name can't be derived by casing, it can be a +`scope → value` dict (e.g. `#{ python: "io_thing", csharp: "IOThing" }`) whose +values are matched **verbatim** for their language. + +Nothing is inferred from other decorators, so a check is deterministic to read +and grounded in a real scenario. A precompute step collects every `@surfaceDoc` +into a shared **checks doc**. Each emitter supplies a **context.md** (what those +concepts look like in its output) and a declarative **verifiers.json** (file +globs + regex patterns per category). A single shared, language-agnostic runner +(`tsp-spector verify-surface-checks`) decides every check whose category has a +verifier; everything else goes to the `/check-surface` orchestrator prompt (AI). + +Whether a check is deterministic or AI is decided by whether the language +supplied a verifier for its category — not by the scenario. Add a verifier and +the same check is promoted from AI to deterministic; the spec assertion never +changes. A category may instead **defer to an existing emitter test** that +already asserts the surface (`{ "test": "…" }`) — no pattern, no AI. + +## The pieces + +| Piece | Lives in | AI? | +| ------------------------------------------------------ | -------------------------------------- | ------------------------ | +| `@surfaceDoc` decorator | spector (requires `@scenarioDoc`) | no | +| checks doc (precomputed Markdown table) | generated by spector, ships with specs | no | +| `context.md` + `verifiers.json` (concept descriptions) | each emitter repo (authored) | no (authored) | +| `verify-surface-checks` (shared runner) | spector `tsp-spector` CLI (all langs) | no | +| `/check-surface` orchestrator | monorepo `.github/prompts/` | AI only for the residual | + +Work is pushed left so AI is the exception: (1) the `@surfaceDoc` manifest +walk is deterministic; (2) concept descriptions are authored once per language, +~a dozen entries reused across all scenarios; (3) the shared runner decides every +item with a verifier (a declarative pattern, or a deferral to an existing test); +(4) AI handles only the residual, cached on +`(item id + generated-file hash + model version)` and scoped to changed packages. + +## Categories + +A check's `category` says what kind of surface property is asserted; `subject` +and `expected` say about-what and to-what. `SurfaceCategory` is an **extensible** +union — the named members are the canonical, tight set, but any string is +accepted. + +| category | `subject` is… | `expected` is… | +| ---------------- | ---------------------------- | ------------------------------------------------------- | +| `naming` | renamed type/member/op/param | expected client identifier (verbatim or with `:casing`) | +| `access` | type/operation | `"internal"` or `"public"` | +| `clientLocation` | operation | client/group name it should be surfaced on | +| `hierarchy` | derived model | base type's client name | +| `flatten` | the flattened property | promoted property name(s); blank = just inlined | +| `paging` | operation | item/element type the pager yields | +| `other` | anything | free-form; verified against the prose (AI) | + +The precompute step derives a category-agnostic set of placeholders for the +verifier engine: + +| placeholder | derived from | +| ------------ | --------------------------------------- | +| `{target}` | the `subject` string (or target's name) | +| `{expected}` | the `expected` value | +| `{kind}` | the symbol kind (e.g. `property`) | + +Any other key in `details` is also accessible as `{key}` (e.g. `{internal}`). +Use the `:byKind` modifier (e.g. `{expected:byKind}`) with a `casing` map in +the verifier to apply per-kind conventions (Python: property → snake_case, +model → PascalCase, etc.). + +### Adding a category + +The category is just a routing key, so a new one is expandable without touching +core spector: + +1. Optionally add a member to the `SurfaceCategory` union in spector for + autocomplete/discoverability (a one-line, backward-compatible change). +2. In each emitter that wants it deterministic, add a `verifiers.json` entry + (using `{target}`/`{expected}` + casing modifiers) and a `context.md` blurb. + +An emitter that adds **nothing** for a category still works: it falls back to +AI-verifying the prose — exactly how `other` behaves. So promoting a check from +`other` to a specific category (a central spec edit) never requires any emitter +to change; each opts into determinism at its own pace. + +One call (Copilot Chat agent mode or CLI): + +``` +/check-surface → emitter: http-client-python flavor: azure scenario: (blank = all) +``` + +The prompt is an **orchestrator**: in one agent session it runs the shared +`tsp-spector verify-surface-checks` once (every category with a verifier, decided +without AI), then verifies only the residual itself (AI). It prints one merged +table marking each row `deterministic`, `test`, `not-applicable`, or `ai`. + +## Onboarding a language + +One required file: **context.md** under +`packages//eng/scripts/client-criteria/` — Emitter facts (language, +generated-root, flavors, checks-doc path) plus a **concept description** per +category, written concretely enough to execute as a file check. Optionally add +**verifiers.json** — a declarative table mapping each category to file globs + +regex patterns (or a deferral to an existing test) for a zero-AI path in CI, +consumed by the shared `tsp-spector verify-surface-checks` runner. A new emitter +writes no assertions, no per-scenario criteria, and **no runner code** — the +decorator, checks doc, runner, and orchestrator are all shared; only config +differs per language. + +### Marking a category not-applicable + +When a surface concept has no equivalent in a language (e.g. a language with no +class inheritance can't have a `hierarchy` surface), the emitter marks that +**category** N/A in its `verifiers.json` — mirroring how wire scenarios are +marked N/A via `unsupportedScenarios`. It's emitter-owned, needs no pattern, and +never reaches AI; every check in the category resolves to `not-applicable`: + +```json +{ + "hierarchy": { "na": true, "note": "no class inheritance in this language" } +} +``` + +## Resolved + +- **Where the precomputed checks doc ships:** alongside the specs, exactly like + the scenario summary. `packages/http-specs` runs + `tsp-spector generate-surface-checks ./specs` (wired into `regen-docs`) to + produce a committed `surface-checks.json`, the same way + `generate-scenarios-summary` produces `spec-summary.md`. +- **How items mark N/A for a language:** per **category** in the emitter's + `verifiers.json` (`{ "na": true }`), mirroring wire `unsupportedScenarios`; + the runner reports `not-applicable` (see above). diff --git a/packages/spector/generated-defs/TypeSpec.Spector.ts b/packages/spector/generated-defs/TypeSpec.Spector.ts index 62c13be7be7..8ae119704b4 100644 --- a/packages/spector/generated-defs/TypeSpec.Spector.ts +++ b/packages/spector/generated-defs/TypeSpec.Spector.ts @@ -1,11 +1,14 @@ -import type { - DecoratorContext, - DecoratorValidatorCallbacks, - Interface, - Model, - Namespace, - Operation, - Type, +import { + type DecoratorContext, + type DecoratorValidatorCallbacks, + getAutoDecoratorValue, + type Interface, + type Model, + type Namespace, + type Operation, + type Program, + setAutoDecorator, + type Type, } from "@typespec/compiler"; /** @@ -45,3 +48,48 @@ export type TypeSpecSpectorDecorators = { scenario: ScenarioDecorator; scenarioDoc: ScenarioDocDecorator; }; + +export function getSurfaceDoc( + program: Program, + target: Namespace | Interface | Operation, +): + | { + readonly category: + | string + | "naming" + | "access" + | "client-location" + | "hierarchy" + | "flatten" + | "paging" + | "other"; + readonly expected: string | Record; + readonly subject?: string; + readonly kind?: string; + readonly doc?: string; + } + | undefined { + return getAutoDecoratorValue(program, "TypeSpec.Spector.surfaceDoc", target)?.["check"] as any; +} + +export function setSurfaceDoc( + program: Program, + target: Namespace | Interface | Operation, + check: { + readonly category: + | string + | "naming" + | "access" + | "client-location" + | "hierarchy" + | "flatten" + | "paging" + | "other"; + readonly expected: string | Record; + readonly subject?: string; + readonly kind?: string; + readonly doc?: string; + }, +): void { + setAutoDecorator(program, "TypeSpec.Spector.surfaceDoc", target, { check: check }); +} diff --git a/packages/spector/lib/main.tsp b/packages/spector/lib/main.tsp index f22ab2d0aa4..b940002a5f7 100644 --- a/packages/spector/lib/main.tsp +++ b/packages/spector/lib/main.tsp @@ -35,3 +35,92 @@ extern dec scenarioDoc( doc: valueof string, formatArgs?: Model ); + +/** + * The category of a client-surface check: what kind of generated-surface + * property is being asserted. This is an **extensible** union — the named + * members are the canonical, tight set every emitter understands, but any + * string is accepted so a new category can be introduced without changing this + * core library. A category an emitter hasn't onboarded a verifier for simply + * falls back to being verified against the prose (AI), exactly like `other`. + * + * Each category fixes what `subject` and `expected` mean: + * + * | category | `subject` is… | `expected` is… | + * | ---------------- | ---------------------------- | ------------------------------------------------ | + * | `naming` | renamed type/member/op/param | canonical client identifier (recast per language) | + * | `access` | type/operation | `"internal"` or `"public"` | + * | `clientLocation` | operation | client/group name it should be surfaced on | + * | `hierarchy` | derived model | base type's client name | + * | `flatten` | the flattened property | promoted property name(s); blank = just inlined | + * | `paging` | operation | item/element type the pager yields | + * | `other` | anything | free-form; verified against the prose (AI) | + */ +union SurfaceCategory { + string, + + /** Client-facing identifier of `subject` (recast to the language's idiomatic casing). */ + naming: "naming", + + /** Public vs internal visibility of `subject` on the client surface. */ + access: "access", + + /** Which client or operation group an operation is surfaced on. */ + clientLocation: "client-location", + + /** Base type `subject` extends on the client surface. */ + hierarchy: "hierarchy", + + /** Nested model properties inlined onto the parent model. */ + flatten: "flatten", + + /** `subject` surfaces as a pageable collection of an item type. */ + paging: "paging", + + /** No mechanical rule — verified against the prose (AI residual). */ + other: "other", +} + +/** + * Describe one expected property of the **generated SDK surface** — something a + * wire test can't see, such as a client rename, an access change, an operation + * relocated to another client, or a reshaped inheritance hierarchy. Mirrors + * `@scenarioDoc`, but for the surface instead of the wire. + * + * `@surfaceDoc` may only be applied to an element that also carries + * `@scenarioDoc`, so every surface check is grounded in a documented scenario. + * Like `@scenarioDoc`, each target gets exactly one `@surfaceDoc`. + * + * ```tsp + * @surfaceDoc(#{ + * category: "naming", + * expected: "widgetName", + * subject: "Widget.defaultName", + * kind: "property", + * }) + * ``` + * + * When `subject` is omitted it defaults to the decorated target's name. + * When `kind` is provided, emitter verifiers can apply per-kind conventions + * (e.g. Python recases properties to snake_case, types to PascalCase). + * + * `expected` is a single canonical, language-agnostic string by default, recast + * by each emitter. For exact per-language names, pass a `scope → value` dict: + * + * ```tsp + * @surfaceDoc(#{ + * category: "naming", + * expected: #{ python: "io_thing", csharp: "IOThing" }, + * }) + * ``` + */ +auto dec surfaceDoc( + target: Namespace | Interface | Operation, + check: valueof { + category: SurfaceCategory, + expected: string | Record, + subject?: string, + kind?: string, + doc?: string, + } +); diff --git a/packages/spector/src/actions/generate-surface-checks.ts b/packages/spector/src/actions/generate-surface-checks.ts new file mode 100644 index 00000000000..e65f64ac2bf --- /dev/null +++ b/packages/spector/src/actions/generate-surface-checks.ts @@ -0,0 +1,29 @@ +import { writeFile } from "fs/promises"; +import pc from "picocolors"; +import { + computeSurfaceChecksManifest, + createSurfaceChecksSummary, +} from "../coverage/surface-checks-manifest.js"; +import { logger } from "../logger.js"; + +export interface GenerateSurfaceChecksConfig { + scenariosPath: string; + outputFile: string; +} + +export async function generateSurfaceChecks({ + scenariosPath, + outputFile, +}: GenerateSurfaceChecksConfig) { + logger.info(`Computing surface checks manifest for ${scenariosPath}`); + const [manifest, diagnostics] = await computeSurfaceChecksManifest(scenariosPath); + if (manifest === undefined || diagnostics.length > 0) { + process.exit(-1); + } + + const summary = createSurfaceChecksSummary(manifest); + await writeFile(outputFile, summary); + logger.info( + `${pc.green("✓")} Surface checks doc generated at ${outputFile} (${manifest.items.length} checks).`, + ); +} diff --git a/packages/spector/src/actions/verify-surface-checks.ts b/packages/spector/src/actions/verify-surface-checks.ts new file mode 100644 index 00000000000..524e10bba30 --- /dev/null +++ b/packages/spector/src/actions/verify-surface-checks.ts @@ -0,0 +1,450 @@ +import { glob, readFile, readdir } from "fs/promises"; +import { join } from "path"; + +/** + * Shared, language-agnostic runner for client-surface checks. + * + * The wire-format Spector tests can't observe surface properties (naming, + * visibility, shape, hierarchy). `@surfaceDoc` describes them once in the spec; + * `generate-surface-checks` precomputes a checks doc; each emitter authors a + * **declarative** `verifiers.json` that tells this engine, per category, which + * files to look in and what pattern proves the surface is correct. Nothing here + * is language-specific: Python/Java/C#/... all share this engine and differ only + * in their `verifiers.json` (globs + regex templates + casing) and generated-root. + * + * A category may instead defer to an existing emitter test + * (`{ "test": "…" }`) — reported `deferred`/`test`, no pattern and no AI. A + * category with no verifier at all is left for the AI orchestrator (`needs_ai`). + */ + +// --------------------------------------------------------------------------- +// Checks doc +// --------------------------------------------------------------------------- + +export interface CheckItem { + id: string; + scenario?: string; + category: string; + target: string; + /** + * Language scope, e.g. `python` or `!java`. Set only for verbatim per-language + * checks; empty means all languages (idiomatic recast). + */ + scope?: string; + details: Record; + doc: string; +} + +/** Parse the checks doc (JSON format). */ +export function parseChecksDoc(text: string): CheckItem[] { + const data = JSON.parse(text); + return Array.isArray(data) ? data : data.items; +} + +/** + * Decide whether a scoped check applies to the active language, using the same + * scope grammar as the client-generator decorators: comma-separated language + * names, `!name` to exclude, and `!(a,b)` to exclude a group. An empty/undefined + * scope applies to every language. + */ +export function scopeApplies(scope: string | undefined, language: string): boolean { + const s = scope?.trim(); + if (!s) return true; + const group = s.match(/^!\((.*)\)$/); + if (group) { + const excluded = group[1].split(",").map((x) => x.trim()); + return !excluded.includes(language); + } + const positives: string[] = []; + const negatives: string[] = []; + for (const part of s + .split(",") + .map((x) => x.trim()) + .filter(Boolean)) { + if (part.startsWith("!")) { + negatives.push(part.slice(1).trim()); + } else { + positives.push(part); + } + } + if (positives.length > 0) return positives.includes(language); + if (negatives.length > 0) return !negatives.includes(language); + return true; +} + +type Expect = + | "present" + | "absent" + | { absentWhen: string } + | { whenExpected: Record }; + +/** A pattern rule: search `files` for `find`; PASS per `expect`. */ +export interface Rule { + /** Glob(s) relative to the package dir. Default `**\/*`. */ + files?: string | string[]; + /** Optional regex template isolating blocks (e.g. a class body); `find` runs within each. */ + scope?: string; + /** Regex template with `{var[:casing]}` placeholders. */ + find: string; + /** `present` (default), `absent`, `{ absentWhen: }`, or `{ whenExpected: { : present|absent } }`. */ + expect?: Expect; + /** Per-kind casing map for the `{var:byKind}` modifier; keyed by `details.kind`. */ + casing?: Record; + /** A second rule AND-ed with this one (e.g. present here AND absent there). */ + also?: Rule; +} + +/** Defer to an existing emitter test that already asserts this surface. */ +export interface TestVerifier { + test: string; + note?: string; +} + +/** + * Declare a whole category not-applicable for this language — the surface concept + * has no equivalent in the emitter's output (e.g. a language with no method + * overloading). Mirrors how wire scenarios are marked N/A via + * `unsupportedScenarios`: emitter-owned, no pattern, no AI. Every check in the + * category resolves to `not-applicable`. + */ +export interface NaVerifier { + na: true; + note?: string; +} + +export type Verifier = Rule | TestVerifier | NaVerifier; + +function isTestVerifier(v: Verifier): v is TestVerifier { + return "test" in v && !!v.test && !("find" in v); +} + +function isNaVerifier(v: Verifier): v is NaVerifier { + return "na" in v && v.na === true; +} + +// --------------------------------------------------------------------------- +// Casing (language-idiomatic recasting, driven by verifiers.json) +// --------------------------------------------------------------------------- + +function splitWords(name: string): string[] { + let s = name.replace(/[_\-\s]+/g, " "); + s = s.replace(/([a-z0-9])([A-Z])/g, "$1 $2"); + s = s.replace(/([A-Z]+)([A-Z][a-z])/g, "$1 $2"); + return s.split(/\s+/).filter(Boolean); +} + +function applyCasing(name: string, casing: string): string { + const w = splitWords(name); + const cap = (x: string) => x.charAt(0).toUpperCase() + x.slice(1).toLowerCase(); + switch (casing) { + case "pascal": + return w.map(cap).join(""); + case "camel": + return w.length ? w[0].toLowerCase() + w.slice(1).map(cap).join("") : name; + case "snake": + return w.map((x) => x.toLowerCase()).join("_"); + case "upper_snake": + return w.map((x) => x.toUpperCase()).join("_"); + default: + return name; + } +} + +// --------------------------------------------------------------------------- +// Template + rule engine +// --------------------------------------------------------------------------- + +function escapeRegExp(s: string): string { + return s.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); +} + +/** + * Resolve a `{var[:casing]}` placeholder to a concrete value. + * `target` → item.target (the subject string); `expected` → details.expected; + * anything else → a `details` field. `:casing` applies explicit casing (snake, + * pascal, etc.). A scoped check carries a verbatim per-language value — never + * recase it. + */ +function resolveVar( + name: string, + mod: string | undefined, + item: CheckItem, + _language: string, + casingMap: Record | undefined, +): string | undefined { + let val: string | undefined; + if (name === "target") { + val = item.target; + } else if (name === "expected") { + const d = item.details?.["expected"]; + val = d == null ? undefined : String(d); + } else { + const d = item.details?.[name]; + val = d == null ? undefined : String(d); + } + if (val == null) return undefined; + // A scoped check carries a verbatim per-language value — never recase it. + if (mod && !item.scope) { + if (mod === "byKind") { + const kind = item.details?.["kind"]; + if (kind && casingMap) { + const casingName = casingMap[String(kind)]; + if (casingName) { + val = applyCasing(val, casingName); + } + } + } else { + val = applyCasing(val, mod); + } + } + return val; +} + +/** Compile a regex template; returns null if any placeholder is unresolvable (N/A). */ +function buildRegex( + template: string, + item: CheckItem, + language: string, + casingMap: Record | undefined, +): RegExp | null { + let missing = false; + const source = template.replace(/\{(\w+)(?::(\w+))?\}/g, (_m, v: string, mod?: string) => { + const resolved = resolveVar(v, mod, item, language, casingMap); + if (resolved == null) { + missing = true; + return ""; + } + return escapeRegExp(resolved); + }); + return missing ? null : new RegExp(source); +} + +async function readGlobbed(pkgDir: string, files: string | string[] | undefined): Promise { + const globs = Array.isArray(files) ? files : [files ?? "**/*"]; + const seen = new Set(); + for (const g of globs) { + for await (const f of glob(g, { cwd: pkgDir })) seen.add(f as string); + } + const texts: string[] = []; + for (const rel of [...seen].sort()) { + try { + texts.push(await readFile(join(pkgDir, rel), "utf8")); + } catch { + // directories / unreadable entries are skipped + } + } + return texts.join("\n"); +} + +function resolveExpect(expect: Expect | undefined, item: CheckItem): "present" | "absent" { + if (!expect) return "present"; + if (typeof expect === "string") return expect; + if ("whenExpected" in expect) { + return expect.whenExpected[String(item.details?.expected)] ?? "present"; + } + return item.details?.[expect.absentWhen] ? "absent" : "present"; +} + +interface RuleOutcome { + status: "pass" | "fail"; + evidence: string; +} + +/** Run one pattern rule. Returns null when it doesn't apply (missing placeholder → N/A). */ +async function runRule( + rule: Rule, + item: CheckItem, + pkgDir: string, + language: string, +): Promise { + const findRe = buildRegex(rule.find, item, language, rule.casing); + if (findRe === null) return null; + + const text = await readGlobbed(pkgDir, rule.files); + let haystacks: string[] = [text]; + if (rule.scope) { + const scopeRe = buildRegex(rule.scope, item, language, rule.casing); + if (scopeRe === null) return null; + haystacks = [...text.matchAll(new RegExp(scopeRe.source, "g"))].map((m) => m[0]); + } + + const found = haystacks.some((h) => findRe.test(h)); + const expect = resolveExpect(rule.expect, item); + const ok = expect === "present" ? found : !found; + const evidence = + `/${findRe.source}/ ${found ? "found" : "not found"}` + + (rule.scope ? " in scope" : "") + + ` (wanted ${expect})`; + return { status: ok ? "pass" : "fail", evidence }; +} + +export interface VerifyResult { + id: string; + category: string; + status: string; + evidence: string; + how: string; +} + +async function evalVerifier( + v: Verifier, + item: CheckItem, + pkgDir: string, + language: string, +): Promise<{ status: string; evidence: string; how: string }> { + const primary = await runRule(v as Rule, item, pkgDir, language); + if (primary === null) { + return { + status: "na", + how: "deterministic", + evidence: `no applicable rule (missing detail) for '${item.category}'`, + }; + } + let { status, evidence } = primary; + const also = (v as Rule).also; + if (also) { + const second = await runRule(also, item, pkgDir, language); + if (second) { + if (second.status === "fail") status = "fail"; + evidence += ` | also: ${second.evidence}`; + } + } + return { status, evidence, how: "deterministic" }; +} + +// --------------------------------------------------------------------------- +// Package resolution + orchestration +// --------------------------------------------------------------------------- + +/** Find the generated package dir under / whose name contains the scenario. */ +async function findPackage( + root: string, + flavor: string, + scenario: string, +): Promise { + const base = join(root, flavor); + let entries; + try { + entries = await readdir(base, { withFileTypes: true }); + } catch { + return undefined; + } + const hit = entries + .filter((e) => e.isDirectory()) + .map((e) => e.name) + .sort() + .find((n) => scenario && n.toLowerCase().includes(scenario.toLowerCase())); + return hit ? join(base, hit) : undefined; +} + +export interface VerifySurfaceChecksConfig { + checksPath: string; + verifiersPath: string; + generatedRoot: string; + flavor: string; + language: string; + /** Optional case-insensitive substring filter on the check id. */ + scenario?: string; +} + +export interface VerifySurfaceChecksOutput { + results: VerifyResult[]; + needs_ai: Array<{ + id: string; + category: string; + target: string; + doc: string; + details: Record; + }>; +} + +/** + * Run every check whose category has a verifier (in parallel, no AI) and collect + * the residual (no verifier) into `needs_ai` for the orchestrator. + */ +export async function verifySurfaceChecks( + config: VerifySurfaceChecksConfig, +): Promise { + const { checksPath, verifiersPath, generatedRoot, flavor, language, scenario } = config; + const items = parseChecksDoc(await readFile(checksPath, "utf8")).filter( + (it) => + (!scenario || it.id.toLowerCase().includes(scenario.toLowerCase())) && + scopeApplies(it.scope, language), + ); + const rawVerifiers = JSON.parse(await readFile(verifiersPath, "utf8")) as Record; + const verifiers: Record = {}; + for (const [k, val] of Object.entries(rawVerifiers)) { + if (!k.startsWith("_")) verifiers[k] = val as Verifier; + } + + const results: VerifyResult[] = []; + const needsAi: VerifySurfaceChecksOutput["needs_ai"] = []; + + await Promise.all( + items.map(async (item) => { + const v = verifiers[item.category]; + if (!v) { + needsAi.push({ + id: item.id, + category: item.category, + target: item.target, + doc: item.doc, + details: item.details, + }); + return; + } + // Emitter-declared not-applicable: the concept has no surface in this + // language. Mirrors wire `unsupportedScenarios`; no package or pattern. + if (isNaVerifier(v)) { + const note = v.note ? ` — ${v.note}` : ""; + results.push({ + id: item.id, + category: item.category, + status: "not-applicable", + how: "not-applicable", + evidence: `category '${item.category}' not applicable for ${language}${note}`, + }); + return; + } + // Defer to an existing test: no package or pattern needed. + if (isTestVerifier(v)) { + const note = v.note ? ` — ${v.note}` : ""; + results.push({ + id: item.id, + category: item.category, + status: "deferred", + how: "test", + evidence: `covered by existing test: ${v.test}${note}`, + }); + return; + } + const pkg = await findPackage(generatedRoot, flavor, item.scenario ?? ""); + if (!pkg) { + results.push({ + id: item.id, + category: item.category, + status: "error", + how: "deterministic", + evidence: `package for scenario '${item.scenario}' not found`, + }); + return; + } + try { + const r = await evalVerifier(v, item, pkg, language); + results.push({ id: item.id, category: item.category, ...r }); + } catch (e) { + results.push({ + id: item.id, + category: item.category, + status: "error", + how: "deterministic", + evidence: `${(e as Error).name}: ${(e as Error).message}`, + }); + } + }), + ); + + results.sort((a, b) => a.id.localeCompare(b.id)); + needsAi.sort((a, b) => a.id.localeCompare(b.id)); + return { results, needs_ai: needsAi }; +} diff --git a/packages/spector/src/cli/cli.ts b/packages/spector/src/cli/cli.ts index 99684a2ff2e..4e8648a2226 100644 --- a/packages/spector/src/cli/cli.ts +++ b/packages/spector/src/cli/cli.ts @@ -2,12 +2,14 @@ import { join, resolve } from "path"; import yargs from "yargs"; import { checkCoverage } from "../actions/check-coverage.js"; import { generateScenarioSummary } from "../actions/generate-scenario-summary.js"; +import { generateSurfaceChecks } from "../actions/generate-surface-checks.js"; import { validateScenarios } from "../actions/index.js"; import { serve, startInBackground, stop } from "../actions/serve.js"; import { serverTest } from "../actions/server-test.js"; import { uploadCoverageReport } from "../actions/upload-coverage-report.js"; import { uploadScenarioManifest } from "../actions/upload-scenario-manifest.js"; import { validateMockApis } from "../actions/validate-mock-apis.js"; +import { verifySurfaceChecks } from "../actions/verify-surface-checks.js"; import { logger } from "../logger.js"; import { getCommit } from "../utils/misc-utils.js"; @@ -90,6 +92,80 @@ async function main() { } }, ) + .command( + "generate-surface-checks ", + "Precompute the surface-checks doc (JSON, from @surfaceDoc) for all specs.", + (cmd) => { + return cmd + .positional("scenariosPaths", { + description: "Path(s) to the scenarios", + type: "string", + array: true, + demandOption: true, + }) + .option("outputFile", { + type: "string", + description: "Path to the generated surface-checks doc (JSON).", + default: join(process.cwd(), "surface-checks.json"), + }); + }, + async (args) => { + for (const scenariosPath of args.scenariosPaths) { + await generateSurfaceChecks({ + scenariosPath: resolve(process.cwd(), scenariosPath), + outputFile: resolve(process.cwd(), args.outputFile), + }); + } + }, + ) + .command( + "verify-surface-checks", + "Run the deterministic client-surface checks (shared, language-agnostic engine).", + (cmd) => { + return cmd + .option("checks", { + type: "string", + demandOption: true, + description: "Path to the checks doc (Markdown table from @surfaceDoc, or JSON).", + }) + .option("verifiers", { + type: "string", + demandOption: true, + description: "Path to the emitter's declarative verifiers.json routing table.", + }) + .option("generated-root", { + type: "string", + demandOption: true, + description: "Root of the generated SDKs; packages live under //.", + }) + .option("flavor", { + type: "string", + default: "azure", + description: "Flavor subfolder under generated-root (e.g. azure, unbranded).", + }) + .option("language", { + type: "string", + demandOption: true, + description: "Emitter language (selects per-language client names).", + }) + .option("scenario", { + type: "string", + description: "Optional case-insensitive substring filter on the check id.", + }); + }, + async (args) => { + const output = await verifySurfaceChecks({ + checksPath: resolve(process.cwd(), args.checks), + verifiersPath: resolve(process.cwd(), args.verifiers), + generatedRoot: resolve(process.cwd(), args["generated-root"]), + flavor: args.flavor, + language: args.language, + scenario: args.scenario, + }); + // eslint-disable-next-line no-console + console.log(JSON.stringify(output, null, 2)); + }, + ) .command("server", "Server management", (cmd) => { cmd .command( diff --git a/packages/spector/src/coverage/surface-checks-manifest.ts b/packages/spector/src/coverage/surface-checks-manifest.ts new file mode 100644 index 00000000000..82fcf8afe07 --- /dev/null +++ b/packages/spector/src/coverage/surface-checks-manifest.ts @@ -0,0 +1,190 @@ +import { getSourceLocation, normalizePath } from "@typespec/compiler"; +import { relative } from "path"; +import pc from "picocolors"; +import type { SurfaceDetails, SurfaceDoc, SurfaceDocTarget } from "../lib/decorators.js"; +import { buildSurfaceDetails } from "../lib/decorators.js"; +import { logger } from "../logger.js"; +import { findScenarioSpecFiles } from "../scenarios-resolver.js"; +import { importSpecExpect, importTypeSpec } from "../spec-utils/index.js"; +import { createDiagnosticReporter, Diagnostic, getSourceLocationStr } from "../utils/index.js"; +import { getCommit, getPackageJson } from "../utils/misc-utils.js"; + +/** One language-neutral entry of the surface-checks doc — what to check, for every emitter. */ +export interface SurfaceCheckItem { + /** + * Stable id, `_` (with a `_n` suffix to disambiguate + * several same-category checks on one scenario), e.g. `listWithPageSize_paging`. + */ + id: string; + /** The enclosing `@scenario` name this surface check belongs to. */ + scenario: string | undefined; + /** The kind of surface assertion (routes the check to a verifier). */ + category: string; + /** The subject symbol's own name, matched against the generated code. */ + target: string; + /** + * Language scope, e.g. `python` or `!java`. Set only for verbatim per-language + * checks (from a `scope → value` dict); empty means all languages (recast). + */ + scope?: string; + /** Language-agnostic description of the expected surface. */ + doc: string; + /** Where the `@surfaceDoc` lives in the spec. */ + location: SurfaceCheckLocation; + /** Generic detail fields (`expected`, `kind`, `origin`); absent for prose-only checks. */ + details?: SurfaceDetails; +} + +export interface SurfaceCheckLocation { + path: string; + start: { line: number; character: number }; + end: { line: number; character: number }; +} + +export interface SurfaceChecksManifest { + version: string; + commit: string; + items: SurfaceCheckItem[]; +} + +export async function computeSurfaceChecksManifest( + scenariosPath: string, +): Promise<[SurfaceChecksManifest | undefined, readonly Diagnostic[]]> { + const [surfaceDocs, diagnostics] = await loadSurfaceDocs(scenariosPath); + if (diagnostics.length > 0) { + return [undefined, diagnostics]; + } + + const commit = getCommit(scenariosPath); + const pkg = await getPackageJson(scenariosPath); + return [createSurfaceChecksManifest(scenariosPath, pkg?.version, commit, surfaceDocs), []]; +} + +export function createSurfaceChecksManifest( + scenariosPath: string, + version: string | undefined, + commit: string, + surfaceDocs: SurfaceDoc[], +): SurfaceChecksManifest { + const items: SurfaceCheckItem[] = []; + const usedIds = new Map(); + for (const surfaceDoc of surfaceDocs) { + const location = getCheckLocation(scenariosPath, surfaceDoc.target); + const details = buildSurfaceDetails(surfaceDoc); + const scopeSuffix = surfaceDoc.scope ? `_${surfaceDoc.scope.replace(/[^\w]+/g, "-")}` : ""; + const idBase = surfaceDoc.scenario ?? surfaceDoc.subject; + items.push({ + id: uniqueId(usedIds, `${idBase}_${surfaceDoc.category}${scopeSuffix}`), + scenario: surfaceDoc.scenario, + category: surfaceDoc.category, + target: surfaceDoc.subject, + ...(surfaceDoc.scope ? { scope: surfaceDoc.scope } : {}), + doc: surfaceDoc.doc, + location, + ...(Object.keys(details).length > 0 ? { details } : {}), + }); + } + items.sort((a, b) => a.id.localeCompare(b.id)); + return { version: version ?? "?", commit, items }; +} + +/** + * Render a {@link SurfaceChecksManifest} as a JSON document consumed by the + * shared `verify-surface-checks` runner. Each item includes id, scenario, + * category, target, scope, details, and doc. + */ +export function createSurfaceChecksSummary(manifest: SurfaceChecksManifest): string { + const items = manifest.items.map((item) => ({ + id: item.id, + scenario: item.scenario ?? "", + category: item.category, + target: item.target, + ...(item.scope ? { scope: item.scope } : {}), + details: item.details ?? {}, + doc: item.doc, + })); + return JSON.stringify({ items }, null, 2) + "\n"; +} + +function uniqueId(used: Map, base: string): string { + const count = used.get(base) ?? 0; + used.set(base, count + 1); + return count === 0 ? base : `${base}_${count + 1}`; +} + +function getCheckLocation( + scenariosPath: string, + target: SurfaceDoc["target"], +): SurfaceCheckLocation { + const tspLocation = getSourceLocation(target); + return { + path: normalizePath(relative(scenariosPath, tspLocation.file.path)), + start: tspLocation.file.getLineAndCharacterOfPosition(tspLocation.pos), + end: tspLocation.file.getLineAndCharacterOfPosition(tspLocation.end), + }; +} + +export async function loadSurfaceDocs( + scenariosPath: string, +): Promise<[SurfaceDoc[], readonly Diagnostic[]]> { + const scenarioFiles = await findScenarioSpecFiles(scenariosPath); + const typespecCompiler = await importTypeSpec(scenariosPath); + const specExpect = await importSpecExpect(scenariosPath); + const diagnostics = createDiagnosticReporter(); + + const surfaceDocs: SurfaceDoc[] = []; + for (const { name, specFilePath } of scenarioFiles) { + logger.debug(`Collecting surface docs from "${specFilePath}"`); + const program = await typespecCompiler.compile(typespecCompiler.NodeHost, specFilePath, { + additionalImports: ["@typespec/spector"], + noEmit: true, + warningAsError: true, + }); + + const programDiagnostics = program.diagnostics.filter( + (d) => + !( + d.code === "@azure-tools/typespec-azure-core/casing-style" && + typeof d.target === "object" && + "kind" in d.target && + d.target.kind === "Namespace" && + d.target.name === "DPG" + ), + ); + + if (programDiagnostics.length > 0) { + for (const item of programDiagnostics) { + const sourceLocation = typespecCompiler.getSourceLocation(item.target); + diagnostics.reportDiagnostic({ + message: `${item.message}: ${sourceLocation && getSourceLocationStr(sourceLocation)}`, + }); + } + diagnostics.reportDiagnostic({ + message: `${pc.red("✘")} Scenario ${name} is invalid.`, + }); + continue; + } + + // `@surfaceDoc` must sit on an element that also carries `@scenarioDoc`, so + // every surface check is grounded in a documented scenario. Enforced here + // (not via a compiler `$onValidate` hook) so plain spec compilation does not + // trigger spector's other, currently-dormant scenario validations. + const missingScenarioDoc = specExpect.listSurfaceDocsMissingScenarioDoc(program); + if (missingScenarioDoc.length > 0) { + for (const target of missingScenarioDoc) { + const sourceLocation = typespecCompiler.getSourceLocation(target); + diagnostics.reportDiagnostic({ + message: `@surfaceDoc may only be applied to an element that also has @scenarioDoc, so every surface check is grounded in a documented scenario.: ${sourceLocation && getSourceLocationStr(sourceLocation)}`, + }); + } + diagnostics.reportDiagnostic({ + message: `${pc.red("✘")} Scenario ${name} is invalid.`, + }); + continue; + } + + surfaceDocs.push(...specExpect.listSurfaceDocs(program)); + } + + return [surfaceDocs, diagnostics.diagnostics]; +} diff --git a/packages/spector/src/lib/decorators.ts b/packages/spector/src/lib/decorators.ts index 4b61d2cd261..7ae8dbb750c 100644 --- a/packages/spector/src/lib/decorators.ts +++ b/packages/spector/src/lib/decorators.ts @@ -1,6 +1,8 @@ import { $service, Enum, + getAutoDecoratorTargets, + getAutoDecoratorValue, getNamespaceFullName, getTypeName, Interface, @@ -221,3 +223,197 @@ export function getScenarioName( } return resolveScenarioName(target, name); } + +/** + * The element `@surfaceDoc` is applied to. To keep surface checks grounded in a + * real scenario, this must be a `@scenario`/`@scenarioDoc` element (enforced in + * `loadSurfaceDocs` while the manifest is built), so the union matches + * `@scenarioDoc`'s target. + */ +export type SurfaceDocTarget = Namespace | Interface | Operation; + +/** + * The generic, category-agnostic fields the shared runner substitutes into an + * emitter's `verifiers.json` (as `{expected}`, `{kind}`, `{origin}`). They are + * derived the same way for every category, so a new category needs no core + * change — only a `verifiers.json` entry (or the AI fallback). + */ +export interface SurfaceDetails { + /** The author's `expected` client-surface output for this check. */ + expected?: string; + /** The symbol kind (e.g. "property", "model", "enum", "operation"). */ + kind?: string; +} + +/** A resolved `@surfaceDoc` annotation. */ +export interface SurfaceDoc { + /** + * The resolved name of the enclosing `@scenario` this check belongs to (named + * the way `@scenario`s are, e.g. `Payload_Pageable_PageSize_listWithPageSize`). + * Every surface check is grounded in — and identified by — its scenario, so + * multiple checks on one scenario share this name. `undefined` only if the + * annotated element has no enclosing `@scenario` (which `loadSurfaceDocs` + * rejects, since `@surfaceDoc` requires `@scenarioDoc`). + */ + scenario: string | undefined; + /** The annotated element (a scenario namespace/interface/operation). */ + target: SurfaceDocTarget; + /** The subject of the check — defaults to the target's name when omitted. */ + subject: string; + /** The kind of surface assertion (routes the check to a verifier). */ + category: string; + /** The expected client-surface output for this category. */ + expected: string; + /** The symbol kind (e.g. "property", "model", "enum", "operation") — used by verifiers for per-kind conventions. */ + kind?: string; + /** + * The language scope this check applies to, e.g. `"python"`, `"python,csharp"`, + * or `"!java"`. Set only when `expected` came from a `scope → value` dict; in + * that case the value is matched **verbatim**. Unset = all languages (recast). + */ + scope?: string; + /** Natural-language description (author-supplied, or synthesized for fallback). */ + doc: string; +} + +/** A short prose fallback so the AI path always has something to verify against. */ +function synthesizeDoc(category: string, subject: string, expected: string): string { + return `${category}: ${subject} → ${expected}`; +} + +/** + * Build a scenario-style name by walking up an element's containers and joining + * their names, stopping at the (unnamed) global or the `_Specs_` root — the same + * convention `@scenario` uses. + */ +function getEnclosingScenarioName(program: Program, target: SurfaceDocTarget): string | undefined { + let current: SurfaceDocTarget | Namespace | Interface | undefined = target; + while (current) { + if ( + current.kind === "Namespace" || + current.kind === "Interface" || + current.kind === "Operation" + ) { + const name = getScenarioName(program, current); + if (name) { + return name; + } + } + if (current.kind === "Namespace" || current.kind === "Interface") { + current = current.namespace; + } else if (current.kind === "Operation") { + current = current.interface ?? current.namespace; + } else { + break; + } + } + return undefined; +} + +/** The FQN used by the auto dec for `@surfaceDoc`. */ +const SURFACE_DOC_FQN = "TypeSpec.Spector.surfaceDoc"; + +/** + * What the auto dec stores for `@surfaceDoc`. Single valueof model param `check`: + * `{ check: { category, expected, subject?, doc? } }`. + */ +interface StoredSurfaceDoc { + check: { + category: string; + expected: string | Record; + subject?: string; + kind?: string; + doc?: string; + }; +} + +/** + * Collect every `@surfaceDoc` in the program into a list of language-agnostic + * surface docs. Analogous to {@link listScenarios}, but for the generated + * surface instead of the wire. Each entry records the author-supplied category, + * subject, and expected output verbatim — nothing is inferred from other + * decorators. Feeds the `surface-checks.md` checks doc. + */ +export function listSurfaceDocs(program: Program): SurfaceDoc[] { + const targets = getAutoDecoratorTargets(program, SURFACE_DOC_FQN); + const result: SurfaceDoc[] = []; + for (const [target] of targets) { + const stored = getAutoDecoratorValue(program, SURFACE_DOC_FQN, target) as + StoredSurfaceDoc | undefined; + if (!stored) continue; + const docTarget = target as SurfaceDocTarget; + const scenario = getEnclosingScenarioName(program, docTarget); + const { category, expected, subject, kind, doc } = stored.check; + const resolvedSubject = subject ?? docTarget.name ?? ""; + for (const { expected: exp, scope } of expandExpected(expected)) { + result.push({ + scenario, + target: docTarget, + subject: resolvedSubject, + category, + expected: exp, + kind, + scope, + doc: doc ?? synthesizeDoc(category, resolvedSubject, exp), + }); + } + } + return result.sort( + (a, b) => + (a.scenario ?? "").localeCompare(b.scenario ?? "") || + a.category.localeCompare(b.category) || + (a.scope ?? "").localeCompare(b.scope ?? ""), + ); +} + +/** + * Surface docs whose annotated target does not also carry `@scenarioDoc`. + * + * `@surfaceDoc` must sit on an element that also has `@scenarioDoc` so every + * surface check is grounded in a documented scenario. This is enforced while + * the surface-checks manifest is built (see `loadSurfaceDocs`) rather than as a + * compiler `$onValidate` hook, so it never activates spector's other, + * currently-dormant scenario validations for consumers that only compile specs. + */ +export function listSurfaceDocsMissingScenarioDoc(program: Program): SurfaceDocTarget[] { + const targets = getAutoDecoratorTargets(program, SURFACE_DOC_FQN); + const result: SurfaceDocTarget[] = []; + for (const [target] of targets) { + const docTarget = target as SurfaceDocTarget; + if (getScenarioDoc(program, docTarget) === undefined) { + result.push(docTarget); + } + } + return result; +} + +/** + * Normalize an author's `expected` into one entry per check. A bare string is a + * single, unscoped (idiomatically recast) check; a `scope → value` dict yields + * one verbatim check per scope key. + */ +function expandExpected( + expected: string | Record, +): { expected: string; scope?: string }[] { + if (typeof expected === "string") { + return [{ expected }]; + } + return Object.entries(expected).map(([scope, value]) => ({ expected: value, scope })); +} + +/** + * Build the generic, category-agnostic detail fields the shared runner + * substitutes into `verifiers.json`. Derived the same way for every category so + * a new category needs no core change: `expected` verbatim, plus the subject's + * `kind` (for casing) and `origin` (its declaring container). + */ +export function buildSurfaceDetails(doc: SurfaceDoc): SurfaceDetails { + const details: SurfaceDetails = {}; + if (doc.expected !== "") { + details.expected = doc.expected; + } + if (doc.kind) { + details.kind = doc.kind; + } + return details; +} diff --git a/packages/spector/src/lib/index.ts b/packages/spector/src/lib/index.ts index 0e7f1ce808c..f9ff027404e 100644 --- a/packages/spector/src/lib/index.ts +++ b/packages/spector/src/lib/index.ts @@ -4,12 +4,18 @@ export { $scenarioService, Scenario, ScenarioEndpoint, + SurfaceDetails, + SurfaceDoc, + SurfaceDocTarget, + buildSurfaceDetails, getScenarioDoc, getScenarioEndpoints, getScenarioName, isScenario, listScenarioIn, listScenarios, + listSurfaceDocs, + listSurfaceDocsMissingScenarioDoc, } from "./decorators.js"; export { $lib, reportDiagnostic } from "./lib.js"; export { $onValidate } from "./validate.js"; diff --git a/packages/spector/test/surface-docs.test.ts b/packages/spector/test/surface-docs.test.ts new file mode 100644 index 00000000000..7e684820f18 --- /dev/null +++ b/packages/spector/test/surface-docs.test.ts @@ -0,0 +1,300 @@ +import { resolvePath } from "@typespec/compiler"; +import { createTester } from "@typespec/compiler/testing"; +import { describe, expect, it } from "vitest"; +import { + createSurfaceChecksManifest, + createSurfaceChecksSummary, + type SurfaceCheckItem, +} from "../src/coverage/surface-checks-manifest.js"; +import { + listSurfaceDocs, + listSurfaceDocsMissingScenarioDoc, + type SurfaceDoc, +} from "../src/lib/decorators.js"; + +const Tester = createTester(resolvePath(import.meta.dirname, ".."), { + libraries: ["@typespec/spector"], +}) + .importLibraries() + .using("Spector"); + +/** Compile `code` and return its single resolved surface doc. */ +async function docOf(code: string): Promise { + const { program } = await Tester.compile(code); + const docs = listSurfaceDocs(program); + expect(docs).toHaveLength(1); + return docs[0]; +} + +/** Compile `code` and return the surface-checks items keyed by id. */ +async function manifestItems(code: string): Promise> { + const { program } = await Tester.compile(code); + const manifest = createSurfaceChecksManifest(".", "1.0.0", "abc123", listSurfaceDocs(program)); + return Object.fromEntries(manifest.items.map((i) => [i.id, i])); +} + +describe("@surfaceDoc", () => { + // --- authoring ---------------------------------------------------------- + + it("records the explicit category, subject, expected and prose", async () => { + const doc = await docOf(` + enum ServerExtensibleEnum { + value1, + value2, + } + + @scenario + @scenarioDoc("Return the kind.") + @surfaceDoc(#{ category: "naming", expected: "ClientExtensibleEnum", doc: "Exposed to clients as ClientExtensibleEnum.", subject: "ServerExtensibleEnum" }) + op getKind(): ServerExtensibleEnum; + `); + expect(doc.category).toBe("naming"); + expect(doc.subject).toBe("ServerExtensibleEnum"); + expect(doc.expected).toBe("ClientExtensibleEnum"); + expect(doc.doc).toBe("Exposed to clients as ClientExtensibleEnum."); + }); + + it("resolves the scenario name from the annotated element", async () => { + const doc = await docOf(` + enum ServerExtensibleEnum { + value1, + value2, + } + + @scenario + @scenarioDoc("Return the kind.") + @surfaceDoc(#{ category: "naming", expected: "ClientExtensibleEnum", subject: "ServerExtensibleEnum" }) + op getKind(): ServerExtensibleEnum; + `); + expect(doc.scenario).toBe("getKind"); + }); + + it("synthesizes prose when `doc` is omitted (so the AI fallback always has one)", async () => { + const doc = await docOf(` + model Widget { + id: string; + } + + @scenario + @scenarioDoc("Get a widget.") + @surfaceDoc(#{ category: "access", expected: "internal", subject: "Widget" }) + op get(): Widget; + `); + expect(doc.doc).toContain("access"); + expect(doc.doc).toContain("Widget"); + expect(doc.doc).toContain("internal"); + }); + + it("accepts any category string (extensible), so new categories need no core change", async () => { + const doc = await docOf(` + model Widget { + id: string; + } + + @scenario + @scenarioDoc("Get a widget.") + @surfaceDoc(#{ category: "brand-new-category", expected: "whatever", subject: "Widget" }) + op get(): Widget; + `); + expect(doc.category).toBe("brand-new-category"); + }); + + // --- validation: must be grounded in a scenario doc --------------------- + + it("flags a surface doc whose target has no @scenarioDoc", async () => { + const { program } = await Tester.compile( + ` + model Widget { + id: string; + } + + @scenario + @surfaceDoc(#{ category: "access", expected: "internal", subject: "Widget" }) + op get(): Widget; + `, + ); + const missing = listSurfaceDocsMissingScenarioDoc(program); + expect(missing).toHaveLength(1); + }); + + it("does not flag a surface doc whose target also has @scenarioDoc", async () => { + const { program } = await Tester.compile( + ` + model Widget { + id: string; + } + + @scenario + @scenarioDoc("Get a widget.") + @surfaceDoc(#{ category: "access", expected: "internal", subject: "Widget" }) + op get(): Widget; + `, + ); + expect(listSurfaceDocsMissingScenarioDoc(program)).toHaveLength(0); + }); + + // --- manifest: generic, category-agnostic details ----------------------- + + it("derives {expected, kind} for a naming check", async () => { + const items = await manifestItems(` + enum ServerExtensibleEnum { + value1, + value2, + } + + @scenario + @scenarioDoc("Return the kind.") + @surfaceDoc(#{ category: "naming", expected: "ClientExtensibleEnum", doc: "Exposed as ClientExtensibleEnum.", subject: "ServerExtensibleEnum" }) + op getKind(): ServerExtensibleEnum; + `); + const item = items["getKind_naming"]; + expect(item.id).toBe("getKind_naming"); + expect(item.category).toBe("naming"); + expect(item.target).toBe("ServerExtensibleEnum"); + expect(item.doc).toContain("Exposed as ClientExtensibleEnum"); + expect(item.details).toEqual({ expected: "ClientExtensibleEnum" }); + // A location the verifier can point back to for reporting. + expect(typeof item.location.path).toBe("string"); + expect(item.location.start.line).toBeGreaterThanOrEqual(0); + expect(item.location.end.line).toBeGreaterThanOrEqual(item.location.start.line); + }); + + it("derives {expected} for a subject inside a container", async () => { + const items = await manifestItems(` + model Widget { + id: string; + } + + @scenario + @scenarioDoc("Get a widget.") + @surfaceDoc(#{ category: "naming", expected: "identifier", doc: "Renamed to identifier.", subject: "Widget.id" }) + op get(): Widget; + `); + const item = items["get_naming"]; + expect(item.category).toBe("naming"); + expect(item.target).toBe("Widget.id"); + expect(item.details).toEqual({ expected: "identifier" }); + }); + + it("omits `expected` from details when it is blank", async () => { + const items = await manifestItems(` + @scenario + @scenarioDoc("List items.") + @surfaceDoc(#{ category: "paging", expected: "", subject: "listItems" }) + op listItems(): string[]; + `); + const item = items["listItems_paging"]; + expect(item.category).toBe("paging"); + expect(item.details?.expected).toBeUndefined(); + }); + + it("keeps the prose for a prose-only `other` check", async () => { + const items = await manifestItems(` + model Widget { + id: string; + } + + @scenario + @scenarioDoc("Get a widget.") + @surfaceDoc(#{ category: "other", expected: "strongly typed", doc: "The response body is a strongly typed model.", subject: "Widget" }) + op get(): Widget; + `); + const item = items["get_other"]; + expect(item.category).toBe("other"); + expect(item.doc).toContain("strongly typed model"); + }); + + // --- rendering ---------------------------------------------------------- + + it("renders a JSON checks doc that carries every routable field", async () => { + const { program } = await Tester.compile( + ` + model Widget { + id: string; + } + + @scenario + @scenarioDoc("Get a widget.") + @surfaceDoc(#{ category: "naming", expected: "WidgetInternal", doc: "Hidden | renamed to WidgetInternal.", subject: "Widget" }) + op get(): Widget; + `, + ); + const manifest = createSurfaceChecksManifest(".", "1.0.0", "abc123", listSurfaceDocs(program)); + const json = createSurfaceChecksSummary(manifest); + const data = JSON.parse(json); + + // The rendered doc is idempotent: no volatile version/commit provenance. + expect(json).not.toContain("commit:"); + // Structured items with all routable fields. + expect(data.items).toHaveLength(1); + const item = data.items[0]; + expect(item).toMatchObject({ + category: "naming", + target: "Widget", + details: { expected: "WidgetInternal" }, + doc: "Hidden | renamed to WidgetInternal.", + }); + expect(item.id).toBeDefined(); + expect(item.scenario).toBeDefined(); + }); + + // --- per-language exact names (scope → value dict) ---------------------- + + it("expands a `scope → value` dict into one verbatim check per scope", async () => { + const { program } = await Tester.compile( + ` + model IOThing { + id: string; + } + + @scenario + @scenarioDoc("Get the thing.") + @surfaceDoc(#{ category: "naming", expected: #{ python: "io_thing", csharp: "IOThing" }, subject: "IOThing" }) + op get(): IOThing; + `, + ); + const docs = listSurfaceDocs(program); + expect(docs).toHaveLength(2); + const byScope = Object.fromEntries(docs.map((d) => [d.scope, d])); + expect(byScope["python"].expected).toBe("io_thing"); + expect(byScope["csharp"].expected).toBe("IOThing"); + expect(byScope["python"].category).toBe("naming"); + }); + + it("leaves scope unset for a single canonical (recast) value", async () => { + const doc = await docOf(` + model Widget { + id: string; + } + + @scenario + @scenarioDoc("Get a widget.") + @surfaceDoc(#{ category: "naming", expected: "Widget", subject: "Widget" }) + op get(): Widget; + `); + expect(doc.scope).toBeUndefined(); + }); + + it("includes the scope in the manifest id and renders a scope column", async () => { + const { program } = await Tester.compile( + ` + model IOThing { + id: string; + } + + @scenario + @scenarioDoc("Get the thing.") + @surfaceDoc(#{ category: "naming", expected: #{ python: "io_thing" }, subject: "IOThing" }) + op get(): IOThing; + `, + ); + const manifest = createSurfaceChecksManifest(".", "1.0.0", "abc123", listSurfaceDocs(program)); + const item = manifest.items.find((i) => i.id === "get_naming_python"); + expect(item).toBeDefined(); + expect(item!.scope).toBe("python"); + expect(item!.details).toEqual({ expected: "io_thing" }); + const md = await createSurfaceChecksSummary(manifest); + expect(md).toContain("io_thing"); + expect(md).toContain("python"); + }); +}); diff --git a/packages/spector/test/verify-surface-checks.test.ts b/packages/spector/test/verify-surface-checks.test.ts new file mode 100644 index 00000000000..f2c0346de0e --- /dev/null +++ b/packages/spector/test/verify-surface-checks.test.ts @@ -0,0 +1,324 @@ +import { mkdir, mkdtemp, writeFile } from "fs/promises"; +import { tmpdir } from "os"; +import { join } from "path"; +import { beforeAll, describe, expect, it } from "vitest"; +import { + parseChecksDoc, + verifySurfaceChecks, + type VerifySurfaceChecksOutput, +} from "../src/actions/verify-surface-checks.js"; + +// A tiny generated "Python SDK" fixture under /azure/surfacedemo/, plus a +// declarative verifiers.json, exercising every routine kind of the shared engine. +let root: string; +let verifiersPath: string; + +const verifiers = { + access: { + files: ["**/models/__init__.py"], + find: "\\b{target}\\b", + expect: { absentWhen: "internal" }, + }, + naming: { + files: ["**/*.py"], + find: "\\b{expected:byKind}\\b", + casing: { enum: "pascal", property: "snake" }, + }, + hierarchy: { + files: ["**/models/_models.py"], + find: "class\\s+{target}\\s*\\([^)]*\\b{expected}\\b[^)]*\\)", + }, +}; + +beforeAll(async () => { + root = await mkdtemp(join(tmpdir(), "surface-verify-")); + const pkg = join(root, "azure", "surfacedemo"); + await mkdir(join(pkg, "models"), { recursive: true }); + await mkdir(join(pkg, "operations"), { recursive: true }); + + await writeFile( + join(pkg, "models", "__init__.py"), + `from ._enums import ClientExtensibleEnum\n__all__ = ["ClientExtensibleEnum"]\n`, + ); + await writeFile( + join(pkg, "models", "_enums.py"), + `class ClientExtensibleEnum(str, Enum):\n ONE = "one"\n`, + ); + await writeFile( + join(pkg, "models", "_models.py"), + `class Animal:\n pass\n\nclass Pet(Animal):\n pass\n\nclass Dog(Pet, discriminator="dog"):\n pass\n`, + ); + await writeFile( + join(pkg, "operations", "_operations.py"), + `class WidgetsOperations:\n def get_widget(self):\n pass\n\nclass GadgetsOperations:\n def list_gadgets(self):\n pass\n`, + ); + await writeFile(join(pkg, "models", "_scoped.py"), `class IOThing:\n pass\n`); + await writeFile(join(pkg, "models", "_props.py"), `class SomeModel:\n default_name: bool\n`); + + verifiersPath = join(root, "verifiers.json"); + await writeFile(verifiersPath, JSON.stringify(verifiers)); +}); + +async function run(items: object[], scenario?: string): Promise { + const checksPath = join(root, "checks.json"); + await writeFile(checksPath, JSON.stringify({ items })); + return verifySurfaceChecks({ + checksPath, + verifiersPath, + generatedRoot: root, + flavor: "azure", + language: "python", + scenario, + }); +} + +describe("verify-surface-checks (shared engine)", () => { + it("naming: idiomatic casing via {expected:byKind} passes on the recast identifier", async () => { + const out = await run([ + { + id: "n1", + scenario: "surfacedemo", + category: "naming", + target: "ServerExtensibleEnum", + details: { expected: "ClientExtensibleEnum", kind: "enum" }, + doc: "", + }, + ]); + expect(out.results[0]).toMatchObject({ status: "pass", how: "deterministic" }); + }); + + it("naming: wrong expected name fails", async () => { + const out = await run([ + { + id: "n2", + scenario: "surfacedemo", + category: "naming", + target: "X", + details: { expected: "TotallyMadeUp", kind: "enum" }, + doc: "", + }, + ]); + expect(out.results[0].status).toBe("fail"); + }); + + it("naming: byKind recases a property to snake_case", async () => { + const out = await run([ + { + id: "n3", + scenario: "surfacedemo", + category: "naming", + target: "SomeModel.defaultName", + details: { expected: "defaultName", kind: "property" }, + doc: "", + }, + ]); + // expected "defaultName" + kind "property" → snake → "default_name" → found in _props.py + expect(out.results[0]).toMatchObject({ status: "pass", how: "deterministic" }); + }); + + it("access: public export present passes; internal expectation flips it", async () => { + const pass = await run([ + { + id: "a1", + scenario: "surfacedemo", + category: "access", + target: "ClientExtensibleEnum", + details: {}, + doc: "", + }, + ]); + expect(pass.results[0].status).toBe("pass"); + const fail = await run([ + { + id: "a2", + scenario: "surfacedemo", + category: "access", + target: "ClientExtensibleEnum", + details: { internal: true }, + doc: "", + }, + ]); + expect(fail.results[0].status).toBe("fail"); // present but wanted internal (absent) + }); + + it("hierarchy: subtype base pattern passes for Dog(Pet), fails for Dog(Animal)", async () => { + const good = await run([ + { + id: "h1", + scenario: "surfacedemo", + category: "hierarchy", + target: "Dog", + details: { expected: "Pet" }, + doc: "", + }, + ]); + expect(good.results[0].status).toBe("pass"); + const bad = await run([ + { + id: "h2", + scenario: "surfacedemo", + category: "hierarchy", + target: "Dog", + details: { expected: "Animal" }, + doc: "", + }, + ]); + expect(bad.results[0].status).toBe("fail"); + }); + + it("client-location: no verifier for this category → needs_ai", async () => { + const out = await run([ + { + id: "c1", + scenario: "surfacedemo", + category: "client-location", + target: "getWidget", + details: { expected: "Widgets" }, + doc: "operation should live on the Widgets client", + }, + ]); + expect(out.results).toHaveLength(0); + expect(out.needs_ai[0]).toMatchObject({ id: "c1", category: "client-location" }); + }); + + it("scope: a check scoped to another language is skipped entirely", async () => { + const out = await run([ + { + id: "s1", + scenario: "surfacedemo", + category: "naming", + target: "IOThing", + scope: "csharp", + details: { expected: "IOThing", kind: "enum" }, + doc: "", + }, + ]); + expect(out.results).toHaveLength(0); + expect(out.needs_ai).toHaveLength(0); + }); + + it("scope: a scoped value is matched verbatim, bypassing byKind casing", async () => { + const out = await run([ + { + id: "s2", + scenario: "surfacedemo", + category: "naming", + target: "IOThing", + scope: "python", + details: { expected: "IOThing", kind: "enum" }, + doc: "", + }, + ]); + expect(out.results[0]).toMatchObject({ status: "pass", how: "deterministic" }); + }); + + it("no verifier for a category → needs_ai", async () => { + const out = await run([ + { + id: "u1", + scenario: "surfacedemo", + category: "flatten", + target: "Foo", + details: {}, + doc: "prose", + }, + ]); + expect(out.results).toHaveLength(0); + expect(out.needs_ai[0]).toMatchObject({ id: "u1", category: "flatten", doc: "prose" }); + }); + + it("test-deferral verifier → deferred/test without touching the package", async () => { + const p = join(root, "verifiers.test.json"); + await writeFile( + p, + JSON.stringify({ + hierarchy: { test: "path/To/Test.java#L1-L2", note: "reflection asserts base" }, + }), + ); + const checksPath = join(root, "checks.json"); + await writeFile( + checksPath, + JSON.stringify({ + items: [ + { + id: "t1", + scenario: "surfacedemo", + category: "hierarchy", + target: "Dog", + details: { expected: "Pet" }, + doc: "", + }, + ], + }), + ); + const out = await verifySurfaceChecks({ + checksPath, + verifiersPath: p, + generatedRoot: root, + flavor: "azure", + language: "python", + }); + expect(out.results[0]).toMatchObject({ status: "deferred", how: "test" }); + expect(out.results[0].evidence).toContain("path/To/Test.java#L1-L2"); + }); + + it("na verifier → not-applicable without touching the package", async () => { + const p = join(root, "verifiers.na.json"); + await writeFile( + p, + JSON.stringify({ + hierarchy: { na: true, note: "no class inheritance in this language" }, + }), + ); + const checksPath = join(root, "checks.json"); + await writeFile( + checksPath, + JSON.stringify({ + items: [ + { + id: "na1", + scenario: "surfacedemo", + category: "hierarchy", + target: "Dog", + details: { expected: "Pet" }, + doc: "", + }, + ], + }), + ); + const out = await verifySurfaceChecks({ + checksPath, + verifiersPath: p, + generatedRoot: root, + flavor: "azure", + language: "python", + }); + expect(out.results[0]).toMatchObject({ status: "not-applicable", how: "not-applicable" }); + expect(out.results[0].evidence).toContain("no class inheritance in this language"); + expect(out.needs_ai).toHaveLength(0); + }); + + it("parseChecksDoc reads a JSON checks doc", () => { + const json = JSON.stringify({ + items: [ + { + id: "x1", + scenario: "S", + category: "access", + target: "Widget", + details: { internal: true }, + doc: "Hidden | renamed.", + }, + ], + }); + const items = parseChecksDoc(json); + expect(items[0]).toMatchObject({ + id: "x1", + scenario: "S", + category: "access", + target: "Widget", + details: { internal: true }, + doc: "Hidden | renamed.", + }); + }); +}); diff --git a/packages/spector/tspconfig.yaml b/packages/spector/tspconfig.yaml new file mode 100644 index 00000000000..d4912c747a4 --- /dev/null +++ b/packages/spector/tspconfig.yaml @@ -0,0 +1,3 @@ +kind: project +features: + - auto-decorators