Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
104 changes: 104 additions & 0 deletions .github/prompts/check-surface.prompt.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
---
description: "Orchestrate surface checks: one parallel deterministic batch, then AI for the rest"
mode: "agent"
---

You are the **orchestrator** for surface checks on an emitter's generated
SDK — properties (naming, visibility/access, shape) the wire tests can't see.
You verify every check in **two parallel phases**, not one-by-one:

1. **Deterministic phase (parallel, no AI).** One batch call runs every check
whose category has a routine — concurrently.
2. **AI phase (only the leftovers).** You verify the few checks with no
deterministic routine; dispatch them in parallel if there are several.

Read:

- `packages/${input:emitter}/eng/scripts/client-criteria/context.md` — Emitter
facts (language, generated-root, flavors, checks-doc path) and concept prose.
- The **checks doc** — the `checks-doc` Markdown table named in `context.md`
(default `surface-checks.json`): the batch runner reads it for deterministic
checks and you read its rows' `doc` prose for the AI ones. Use this path unless the caller
explicitly overrides it.
- `packages/${input:emitter}/eng/scripts/client-criteria/verifiers.json` — the
routing table, **authored once per emitter (by you or a human) — never
generated by the emitter build.** A category listed here is decided without
AI, either by a `check` (a declarative pattern the runner matches) or by a `test`
(defer to an existing emitter test that already asserts this surface, e.g. a
Java reflection test). A category not listed is verified by AI. Even with **no**
verifiers.json at all you can still verify every check from `context.md` +
the checks doc — verifiers.json only moves work from AI to code.

Inputs:

- Emitter: ${input:emitter:package folder, e.g. http-client-python}
- Flavor: ${input:flavor:one of the flavors in the emitter's context.md}
- Scenario: ${input:scenario:optional substring filter on the check id}

## Phase 1 — deterministic batch (parallel)

Run the shared runner **once**. It is a single language-agnostic tool
(`tsp-spector verify-surface-checks`) that runs every check whose category has a
verifier — in parallel, no AI — and prints `{"results": [...], "needs_ai": [...]}`:

```
npx tsp-spector verify-surface-checks \
--checks <checks-doc path from context.md> \
--verifiers packages/${input:emitter}/eng/scripts/client-criteria/verifiers.json \
--generated-root <generated-root from context.md> \
--flavor ${input:flavor} --language <language from context.md>
```

Take `results` as-is — each is a verdict decided without AI: `how: deterministic`
(a declarative pattern matched the generated code), `how: test`,
`status: deferred` (a real emitter test already asserts this surface; the
`evidence` cites it), or `how: not-applicable`, `status: not-applicable` (the
emitter declared this category N/A for its language in `verifiers.json`). Do not
re-check or second-guess them.

## Phase 2 — AI verification (the residual, in parallel)

For each item in `needs_ai`, verify it yourself: find the target in the generated
package under `<generated-root>/<flavor>/` (folder matching the scenario), judge
against its `verify` text and context.md, and cite the exact file + line. Mark
`how: ai`. **If there are several, dispatch them as parallel subagents** (one per
item) rather than serially, then collect the verdicts.

## Phase 3 — merge

Combine `results` + your AI verdicts into one table, filtered by the scenario
input if given:

`id | category | result (PASS/FAIL/N/A/DEFERRED) | how (deterministic/test/ai) | evidence`

Render `status: deferred` rows as `DEFERRED` with `how: test` — they are covered
by the referenced test rather than checked here. Render `status: not-applicable`
rows as `N/A` with `how: not-applicable` — the category has no surface in this
language.

End with a summary: pass/fail counts, and how many were code vs test vs AI.

## Phase 4 — propose promotions (optional; only write when asked)

For every item you had to verify with **AI** in Phase 2, consider whether it
could be decided without AI next time, and if so propose a `verifiers.json`
entry for its category. Two forms:

- **`check`** — the emitter's `context.md` describes the concept concretely
enough to be a mechanical file check. Propose a declarative pattern entry
`{"<category>": {"files": ["…"], "find": "…{target}…", "expect": "…"}}`
(placeholders/casing/expect as documented in `verifiers.json`).
- **`test`** — an existing test in the emitter repo already asserts this surface
(e.g. a reflection/`assert`-based test). Propose
`{"<category>": {"test": "<path/to/Test.ext#Lx-Ly>", "note": "…"}}` so the
check defers to that test instead of AI.

Print the proposed JSON entries under a **"Suggested verifiers.json additions"**
heading with a one-line rationale each. verifiers.json is authored, rarely
changes, and is never generated by the emitter — so **only write the file if the
caller explicitly asks**; otherwise leave the suggestions for review. A verify
run is read-only by default.

The point: the bulk is decided without AI — a deterministic routine or an existing test —
and AI is spent only on the checks that actually need judgment, shrinking further
as verifiers.json is filled in.
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# Client surface checks — context: http-client-python

## Emitter facts

_The shared `/check-surface` prompt and `tsp-spector verify-surface-checks` read these._

- **language:** python _(selects which per-language client name applies when a check is language-scoped)_
- **generated-root:** `packages/http-client-python/tests/generated/<flavor>`
- **flavors:** `azure`, `unbranded`
- **checks-doc:** `packages/http-specs/surface-checks.json` _(generated from `@surfaceDoc`, ships with the specs)_

<!-- Markdown table generated from `@surfaceDoc`; both human- and machine-readable (parsed by the shared runner). -->

### What differs by flavor (branded = azure, unbranded)

Only the **generated-root** subfolder and the **import namespace** (`azure.core`
vs `corehttp`). Symbol locations and every concept description below are
flavor-invariant.

## What each check looks like

What each verifiable concept looks like in generated Python. Authored once,
reused across every scenario that uses the concept. Prefer concrete, checkable
verifiers so the shared deterministic runner can decide **without calling AI** —
see `verifiers.json` for the machine-readable declarative form (file globs +
regex patterns + casing) consumed by `tsp-spector verify-surface-checks`.

### access: internal

- **enum** → the class is NOT re-exported from `models/__init__.py`; it may also
be defined with a leading `_`.
- **model** → absent from the package's public `__init__.py` exports.
- **operation / method** → prefixed `_`, or not present on the public client.

### access: public

- present in the public `__init__.py` exports, no leading underscore.

### naming

- the generated identifier equals the `expected` value from `@surfaceDoc`
**recast to Python's idiomatic casing for the subject's `kind`**
(case-sensitive): `enum`/`model`/`type` → `PascalCase`,
`property`/`parameter`/`operation` → `snake_case`, enum value →
`UPPER_SNAKE`. The per-kind casing map lives in `verifiers.json`
(`{expected:byKind}`); the `@surfaceDoc` carries the subject's `kind`.
- **exact per-language names:** when a name can't be derived by casing (a
reserved-word escape, an acronym, a hard rename), the spec author supplies a
`scope → value` dict on `@surfaceDoc` (e.g. `#{ python: "io_thing" }`). Those
checks carry a `scope` column; the engine only runs the ones scoped to
`python` and matches `{expected}` **verbatim**. The same `naming` routine
handles both — no `verifiers.json` change.

### flatten

- the nested model's fields appear directly as attributes on the parent class;
`expected` names the promoted property (blank = assert inlined).

### client-location

- an operation group is a class under `operations/_operations.py`; a method is a
public `def` on it; the **root client** is the main `*Client` class. An
operation is "on" a client/group when its snake_case name is a method of the
matching class. The `expected` value names the client/group where the method
should appear.

### hierarchy

- a client subtype is expressed as the base in `class <target>(<expected>, …)`
in `models/_models.py`; inherited members are attributes present on the
subclass.

> When an assertion has no concrete verifier here (and none in verifiers.json),
> `/check-surface` may fall back to judgment — but add a verifier when
> you can, to keep AI calls rare.
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
{
"_comment": "Declarative routing table for the shared `tsp-spector verify-surface-checks` engine. AUTHORED once per emitter — NOT generated. The engine substitutes placeholders into `find`/`scope`: {target}=the subject name (from @surfaceDoc), {expected}=the expected client-surface output. Add `:casing` to recase, e.g. {expected:snake}, {target:pascal}. Use `:byKind` to apply per-kind casing from the `casing` map (keyed by `details.kind`). `expect`: present (default), absent, {absentWhen:<detailKey>}, or {whenExpected:{<value>:present|absent}}. SCOPE: when a check is scoped to a language, the engine only runs it for the matching language and matches {expected} VERBATIM (casing modifiers are ignored).",
"naming": {
"files": ["**/*.py"],
"find": "\\b{expected:byKind}\\b",
"casing": {
"type": "pascal",
"model": "pascal",
"enum": "pascal",
"property": "snake",
"parameter": "snake",
"operation": "snake",
"enumvalue": "upper_snake"
}
},
"access": {
"files": ["**/models/__init__.py"],
"find": "\\b{target}\\b",
"expect": { "whenExpected": { "internal": "absent", "public": "present" } }
},
"hierarchy": {
"files": ["**/models/_models.py"],
"find": "class\\s+{target}\\s*\\([^)]*\\b{expected}\\b"
}
}
15 changes: 15 additions & 0 deletions packages/http-specs/specs/payload/pageable/main.tsp
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,11 @@ namespace ServerDrivenPagination {
}
```
""")
@surfaceDoc(#{
category: "paging",
expected: "Pet",
doc: "The paginated operation yields `Pet` elements.",
})
@route("/link")
@list
op link(): {
Expand Down Expand Up @@ -518,6 +523,11 @@ namespace PageSize {
}
```
""")
@surfaceDoc(#{
category: "paging",
expected: "Pet",
doc: "Single-page pageable operation yielding `Pet` elements, no continuation.",
})
@route("/without-continuation")
@list
op listWithoutContinuation(): {
Expand Down Expand Up @@ -556,6 +566,11 @@ namespace PageSize {
}
```
""")
@surfaceDoc(#{
category: "paging",
expected: "Pet",
doc: "Page-size-driven pageable operation yielding `Pet` elements.",
})
@route("/list")
@list
op listWithPageSize(@pageSize @query pageSize?: int32): {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,13 @@ namespace Property {
{"wireName": true}
```
""")
@surfaceDoc(#{
category: "naming",
subject: "JsonEncodedNameModel.defaultName",
expected: "defaultName",
kind: "property",
doc: "The client property is named `defaultName` even though the wire name is `wireName`.",
})
@post
op send(@bodyRoot body: JsonEncodedNameModel): NoContentResponse;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,12 @@ op postValid(@body input: Siamese): NoContentResponse;
{"name": "abc", "age": 32, "smart": true}
```
""")
@surfaceDoc(#{
category: "hierarchy",
subject: "Siamese",
expected: "Cat",
doc: "The `Siamese` model extends `Cat` as its base type on the client surface.",
})
@route("/valid")
@get
op getValid(): Siamese;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,12 @@ model Fish {
{"wingspan": 1, "kind": "sparrow"}
```
""")
@surfaceDoc(#{
category: "hierarchy",
subject: "Sparrow",
expected: "Bird",
doc: "The `Sparrow` model extends `Bird` as its base type on the client surface.",
})
@get
op getModel(): Bird;

Expand Down
65 changes: 65 additions & 0 deletions packages/http-specs/surface-checks.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
{
"items": [
{
"id": "Payload_Pageable_PageSize_listWithoutContinuation_paging",
"scenario": "Payload_Pageable_PageSize_listWithoutContinuation",
"category": "paging",
"target": "listWithoutContinuation",
"details": {
"expected": "Pet"
},
"doc": "Single-page pageable operation yielding `Pet` elements, no continuation."
},
{
"id": "Payload_Pageable_PageSize_listWithPageSize_paging",
"scenario": "Payload_Pageable_PageSize_listWithPageSize",
"category": "paging",
"target": "listWithPageSize",
"details": {
"expected": "Pet"
},
"doc": "Page-size-driven pageable operation yielding `Pet` elements."
},
{
"id": "Payload_Pageable_ServerDrivenPagination_link_paging",
"scenario": "Payload_Pageable_ServerDrivenPagination_link",
"category": "paging",
"target": "link",
"details": {
"expected": "Pet"
},
"doc": "The paginated operation yields `Pet` elements."
},
{
"id": "Serialization_EncodedName_Json_Property_send_naming",
"scenario": "Serialization_EncodedName_Json_Property_send",
"category": "naming",
"target": "JsonEncodedNameModel.defaultName",
"details": {
"expected": "defaultName",
"kind": "property"
},
"doc": "The client property is named `defaultName` even though the wire name is `wireName`."
},
{
"id": "Type_Model_Inheritance_NotDiscriminated_getValid_hierarchy",
"scenario": "Type_Model_Inheritance_NotDiscriminated_getValid",
"category": "hierarchy",
"target": "Siamese",
"details": {
"expected": "Cat"
},
"doc": "The `Siamese` model extends `Cat` as its base type on the client surface."
},
{
"id": "Type_Model_Inheritance_SingleDiscriminator_getModel_hierarchy",
"scenario": "Type_Model_Inheritance_SingleDiscriminator_getModel",
"category": "hierarchy",
"target": "Sparrow",
"details": {
"expected": "Bird"
},
"doc": "The `Sparrow` model extends `Bird` as its base type on the client surface."
}
]
}
Loading