From 7198facafc32aef02d04d82a91daa8d9857cc578 Mon Sep 17 00:00:00 2001 From: Claude Fable 5 Date: Sun, 9 Aug 2026 07:46:29 +0000 Subject: [PATCH] =?UTF-8?q?refactor(spec,objectql)!:=20retire=20`Aggregati?= =?UTF-8?q?onNode.distinct`=20=E2=80=94=20one=20face=20honoured=20it,=20fi?= =?UTF-8?q?ve=20ignored=20it=20(#6815,=20ADR-0049)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `AggregationNode.distinct` was read by exactly one of the six faces that consume an `aggregations[]` entry: objectql's in-memory fallback deduplicated before applying the function, while driver-sql, driver-turso, driver-mongodb, driver-memory and service-analytics' AGGREGATE_SQL all ignored it. So `{ function: 'sum', field: 'amount', distinct: true }` answered a deduplicated sum on the fallback path and an ordinary sum on every SQL datasource — one query, two plausible numbers, chosen by which backend served it. Removed per the maintainer ruling of 2026-08-09: tombstoned with retiredKey() (the schema is non-strict, so a bare deletion would silently strip what callers still send), registered as RETIRED_KEYS_BY_MAJOR[17] 'data/AggregationNode:distinct' plus the D3 semantic migration 'aggregation-node-distinct-retired'. No D2 conversion — QueryAST is a request surface with no stored source. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PiRUoQkTSBBmpyXBY3cVn2 --- .../aggregation-node-distinct-retired.md | 71 ++++++++++++ content/docs/kernel/contracts/data-engine.mdx | 7 +- .../docs/protocol/objectql/query-syntax.mdx | 15 ++- content/docs/references/api/contract.mdx | 2 +- content/docs/references/data/data-engine.mdx | 2 +- content/docs/references/data/query.mdx | 4 +- docs/protocol-upgrade-guide.md | 5 + .../src/in-memory-aggregation.test.ts | 69 +++++++++++ .../objectql/src/in-memory-aggregation.ts | 39 ++++--- .../request-schema-gate.conformance.test.ts | 8 ++ packages/spec/authorable-surface/data.json | 2 +- packages/spec/liveness/README.md | 2 +- packages/spec/liveness/query.json | 8 +- packages/spec/spec-changes.json | 14 +++ packages/spec/src/data/query.test.ts | 107 ++++++++++++++++-- packages/spec/src/data/query.zod.ts | 41 ++++++- packages/spec/src/migrations/registry.ts | 100 ++++++++++++++++ 17 files changed, 456 insertions(+), 40 deletions(-) create mode 100644 .changeset/aggregation-node-distinct-retired.md diff --git a/.changeset/aggregation-node-distinct-retired.md b/.changeset/aggregation-node-distinct-retired.md new file mode 100644 index 0000000000..78855678c1 --- /dev/null +++ b/.changeset/aggregation-node-distinct-retired.md @@ -0,0 +1,71 @@ +--- +'@objectstack/spec': major +'@objectstack/objectql': major +--- + +refactor(spec,objectql)!: retire `AggregationNode.distinct` — one face honoured it, five ignored it, and the same query answered two plausible numbers (#6815, ADR-0049) + + + +**FROM → TO:** `{ function: 'count', field: 'x', distinct: true, alias: 'a' }` → +`{ function: 'count_distinct', field: 'x', alias: 'a' }` — the deduplicating spelling +every backend computes, lowered to `COUNT(DISTINCT x)` on both SQL faces since #6409. +`{ function: 'sum' | 'avg' | 'min' | 'max', …, distinct: true }` → delete the key; there is +no replacement, because no SQL backend ever computed `SUM(DISTINCT …)` here and the +in-memory fallback was the only thing that did. `distinct: false` → delete the key; it +selected the behaviour that is now the only behaviour. + +`AggregationNode.distinct` was read by exactly ONE of the six faces that consume an +`aggregations[]` entry. `objectql`'s in-memory fallback (`in-memory-aggregation.ts`) +deduplicated the values before applying the function; `SqlDriver.aggregate`, the Turso +`RemoteTransport.aggregate`, `driver-mongodb`'s `buildAggregationStage`, `driver-memory`'s +`computeAggregate` and `service-analytics`' `AGGREGATE_SQL` all ignored it. So +`{ function: 'sum', field: 'amount', distinct: true }` returned a deduplicated sum when the +engine fell back in memory and an ordinary sum on every SQL datasource — one query, two +numbers, chosen by which backend answered. The engine picks that path per query (a driver +without native aggregation, a non-UTC date bucket, a partial SQL driver), so the number +could move under a dashboard with nothing changing in the query. + +That is the divergence class #6203 and #5907 each closed on the aggregate axis, still open +on this key, and it is worse to leave: both answers are plausible NUMBERS rather than a +refusal, so nothing surfaced it. It survived the #4286 sweep of this same schema because +that sweep asked which members no executor reads — the wrong question for a key whose +defect is *which* executor reads it. + +REMOVE rather than ENFORCE, per the maintainer ruling of 2026-08-09: `count_distinct` +already covers the only deduplicating spelling with measured demand and took ADR-0049's +enforce leg in #6409, while `SUM(DISTINCT …)` / `AVG(DISTINCT …)` are near-universally a +modelling mistake and would have to be lowered across five faces — two of them frozen under +#5499 — to buy it. + +The retirement kit: + +- **Tombstone, not deletion** (`retiredKey()`): `AggregationNodeSchema` is not `.strict()`, + so a plain delete would let existing queries parse clean and lose the key in silence + (#3733, ADR-0104) — trading a divergent flag for an ignored one. Authoring it is now a + `tsc` error at the call site and a parse error carrying the prescription. One tombstone + covers every aggregation door: `QuerySchema.aggregations` and + `EngineAggregateOptionsSchema.aggregations` both reuse that one schema by reference. +- **ADR-0087 D3 `SemanticMigration`** (`aggregation-node-distinct-retired`) plus the exact + `RETIRED_KEYS_BY_MAJOR[17]` entry `data/AggregationNode:distinct`. No D2 conversion, + deliberately: `QueryAST` is a request surface — the client SDK builder's output and the + `POST /data/:object/query` body — never stored in stack metadata, so there is no source + for `os migrate meta` to rewrite. That is the disposition every other `data.query.*` + retirement in this major already takes (#4286). +- `objectql`'s in-memory fallback loses its `collectValues` dedupe limb — the whole runtime + cost of the removal. **The observable numbers change on that one path, and that is the + point:** a `sum`/`avg` that used to be deduplicated there now answers what every SQL face + has always answered for the same query. Verify against the SQL answer, not against the + pre-upgrade fallback answer — the two disagreed. +- Measured blast radius inside the fallback, narrower than the key suggests: only `sum` and + `avg` ever changed answer. `count` returned from its own branch before reaching the + dedupe, `count_distinct` fed the values into a `Set` (dedupe-then-`Set` is `Set`), and + dedupe does not move `min`/`max`. +- `POST /api/v1/data/:object/query` answers `400 VALIDATION_FAILED` with a `fields[]` entry + at `aggregations..distinct` instead of serving a number — the #3899 entry validation + descending into the array, pinned in the REST request-schema conformance gate. +- Liveness ledger (`query.json` `aggregations.children.distinct` → `dead`, README counts), + generated baselines (`authorable-surface/data.json` gains `[RETIRED]`), + `spec-changes.json`, the upgrade guide and the reference docs regenerated. + +`count_distinct` is untouched and remains the live deduplicating spelling. diff --git a/content/docs/kernel/contracts/data-engine.mdx b/content/docs/kernel/contracts/data-engine.mdx index 9a4bab0ef6..3ac963dea1 100644 --- a/content/docs/kernel/contracts/data-engine.mdx +++ b/content/docs/kernel/contracts/data-engine.mdx @@ -357,11 +357,16 @@ interface AggregationNode { function: 'count' | 'sum' | 'avg' | 'min' | 'max' | 'count_distinct'; field?: string; // Field to aggregate (optional for COUNT(*)) alias: string; // Result column alias - distinct?: boolean; // Apply DISTINCT before aggregation filter?: FilterCondition; // Per-aggregation FILTER WHERE } ``` +`distinct?: boolean` was **removed** from `AggregationNode` in protocol 17 (#6815, +ADR-0049). Only the engine's in-memory fallback ever honoured it — every SQL face +ignored it — so the same query answered a deduplicated `sum` or an ordinary one +depending on which backend served it. For a deduplicated count use the +`count_distinct` function, which every face computes. + --- ## Optional Capabilities diff --git a/content/docs/protocol/objectql/query-syntax.mdx b/content/docs/protocol/objectql/query-syntax.mdx index 638ecfb9eb..9578b585ea 100644 --- a/content/docs/protocol/objectql/query-syntax.mdx +++ b/content/docs/protocol/objectql/query-syntax.mdx @@ -104,6 +104,19 @@ a `where` predicate on the sort key — §7), and `distinct` (unique values via effect was suppressing the REST list count, which is truthful again). **Enforced**: `having` (§5). The experimental flags above are tracked in the liveness ledger (`packages/spec/liveness/query.json`). + +One member of `AggregationNode` was settled separately, in #6815: the +per-aggregation **`distinct`** flag is **removed** on the same terms. It escaped the +#4286 sweep because that sweep asked which members no executor reads and this one had +a reader — one out of six. The engine's in-memory fallback deduplicated before +applying the function, while `driver-sql`, `driver-turso`, `driver-mongodb`, +`driver-memory` and the analytics SQL builder all ignored it, so +`{ function: 'sum', field: 'amount', distinct: true }` answered a deduplicated sum on +the fallback path and an ordinary sum on every SQL datasource — one query, two +plausible numbers, chosen by which backend served it. The live deduplicating spelling +is the **`count_distinct` function** (`COUNT(DISTINCT field)` on both SQL faces since +#6409); `SUM(DISTINCT …)` / `AVG(DISTINCT …)` have no replacement, because no backend +ever computed them here. ### Key Types @@ -120,9 +133,9 @@ interface AggregationNode { function: 'count' | 'sum' | 'avg' | 'min' | 'max' | 'count_distinct'; field?: string; // optional for COUNT(*) alias: string; // result column alias - distinct?: boolean; // DISTINCT before aggregation — in-memory path only filter?: FilterCondition; // [EXPERIMENTAL — not enforced] FILTER WHERE clause — never applied } +// `distinct?: boolean` was REMOVED in protocol 17 (#6815) — see the callout below. // FieldNode — one entry of the select list. A field name, optionally dotted to // reach through a relationship ('owner.name'). Related *records* come from diff --git a/content/docs/references/api/contract.mdx b/content/docs/references/api/contract.mdx index 037aeaa013..c595678de2 100644 --- a/content/docs/references/api/contract.mdx +++ b/content/docs/references/api/contract.mdx @@ -412,7 +412,7 @@ const result = ApiErrorSchema.parse(data); | **top** | `number` | optional | Alias for limit (OData compatibility) | | **cursor** | `never` | optional | [REMOVED] `query.cursor` was removed in @objectstack/spec 17 (#4286, ADR-0049) — no driver ever implemented keyset pagination, so the cursor was accepted and ignored and every page came back identical (a caller looping "until hasMore is false" never terminates). Delete the key; `QueryBuilder.cursor()` was removed with it. Express the keyset as an ordinary `where` predicate on your sort key — `where: { created_at: { $gt: last.created_at } }` with the matching `orderBy` — which every driver executes with canonicalised comparands. A first-class cursor, if ever built, will be a response-minted opaque token, not this caller-built record. | | **joins** | `never` | optional | [REMOVED] `query.joins` was removed in @objectstack/spec 17 (#4286, ADR-0049) — no engine or driver ever read it: a query carrying `joins` behaved exactly as if the key were absent, while its name squatted on the reserved REST parameter set. Delete the key. Related records are read through `expand` — `expand: { owner: { object: 'user', fields: ['name'] } }` — which the engine resolves via batch $in queries, and a single related column is a dotted `fields` path (`fields: ['owner.name']`). | -| **aggregations** | `{ function: Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct'>; field?: string; alias: string; distinct?: boolean; … }[]` | optional | Aggregation functions | +| **aggregations** | `{ function: Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct'>; field?: string; alias: string; filter?: any }[]` | optional | Aggregation functions | | **groupBy** | `(string \| { field: string; dateGranularity?: Enum<'day' \| 'week' \| 'month' \| 'quarter' \| 'year'>; alias?: string })[]` | optional | GROUP BY targets (strings or `{field, dateGranularity?}` objects for date bucketing) | | **having** | `any` | optional | HAVING — filter over the AGGREGATED rows (aggregation aliases + groupBy projections); applied engine-side after aggregation | | **windowFunctions** | `never` | optional | [REMOVED] `query.windowFunctions` was removed in @objectstack/spec 17 (#4286, ADR-0049) — `find()` never applied it: no engine or driver read the key on the query path, so every OVER clause it declared was silently dropped. Delete the key. Window functions are a SQL-driver capability behind `SqlDriver.findWithWindowFunctions(object, query)` (embedder-level; not on the `IDataDriver` contract or the REST surface); request-level analytics are `aggregations` + `groupBy`. | diff --git a/content/docs/references/data/data-engine.mdx b/content/docs/references/data/data-engine.mdx index ec45b95e01..5c8a97bb9a 100644 --- a/content/docs/references/data/data-engine.mdx +++ b/content/docs/references/data/data-engine.mdx @@ -467,7 +467,7 @@ QueryAST-aligned options for DataEngine.aggregate operations | **context** | `{ userId?: string; actor?: string; attributedUserId?: string; email?: string; … }` | optional | | | **where** | `Record \| any` | optional | | | **groupBy** | `string[]` | optional | | -| **aggregations** | `{ function: Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct'>; field?: string; alias: string; distinct?: boolean; … }[]` | optional | | +| **aggregations** | `{ function: Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct'>; field?: string; alias: string; filter?: any }[]` | optional | | | **having** | `any` | optional | HAVING — filter over the aggregated rows (aggregation aliases + groupBy projections); applied engine-side after aggregation | | **timezone** | `string` | optional | | diff --git a/content/docs/references/data/query.mdx b/content/docs/references/data/query.mdx index c56b17ce3f..08477bfe7c 100644 --- a/content/docs/references/data/query.mdx +++ b/content/docs/references/data/query.mdx @@ -48,7 +48,7 @@ const result = AggregationFunction.parse(data); | **function** | `Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct'>` | ✅ | Aggregation function | | **field** | `string` | optional | Field to aggregate (optional for COUNT(*)) | | **alias** | `string` | ✅ | Result column alias | -| **distinct** | `boolean` | optional | Apply DISTINCT before aggregation | +| **distinct** | `never` | optional | [REMOVED] `query.aggregations[].distinct` was removed in @objectstack/spec 17 (#6815, ADR-0049) — exactly ONE of the six faces that read an aggregation honoured it. The objectql in-memory fallback deduplicated the values before applying the function, while `driver-sql`, `driver-turso`, `driver-mongodb`, `driver-memory` and the service-analytics SQL builder all ignored it — so `{ function: 'sum', field: 'amount', distinct: true }` answered a DEDUPLICATED sum when the engine fell back in memory and an ordinary sum on every SQL datasource: one query, two numbers, chosen by which backend happened to serve it. Both answers are plausible, so nothing surfaced the divergence. Delete the key. For a deduplicated COUNT the live spelling is the `count_distinct` aggregation function, which every SQL face compiles to `COUNT(DISTINCT field)` (#6409) and the in-memory fallback computes identically. `SUM(DISTINCT …)` / `AVG(DISTINCT …)` get no replacement: no backend ever computed them here, and a per-row measure that needs deduplicating is a modelling problem to fix in the data, not a flag on the read. | | **filter** | `any` | optional | [EXPERIMENTAL — not enforced] Per-aggregation filter (SQL FILTER (WHERE …)). Neither the SQL builders nor the in-memory fallback applies it (#4286); filter the whole query with `where` instead. | @@ -132,7 +132,7 @@ Type: `string` | **top** | `number` | optional | Alias for limit (OData compatibility) | | **cursor** | `never` | optional | [REMOVED] `query.cursor` was removed in @objectstack/spec 17 (#4286, ADR-0049) — no driver ever implemented keyset pagination, so the cursor was accepted and ignored and every page came back identical (a caller looping "until hasMore is false" never terminates). Delete the key; `QueryBuilder.cursor()` was removed with it. Express the keyset as an ordinary `where` predicate on your sort key — `where: { created_at: { $gt: last.created_at } }` with the matching `orderBy` — which every driver executes with canonicalised comparands. A first-class cursor, if ever built, will be a response-minted opaque token, not this caller-built record. | | **joins** | `never` | optional | [REMOVED] `query.joins` was removed in @objectstack/spec 17 (#4286, ADR-0049) — no engine or driver ever read it: a query carrying `joins` behaved exactly as if the key were absent, while its name squatted on the reserved REST parameter set. Delete the key. Related records are read through `expand` — `expand: { owner: { object: 'user', fields: ['name'] } }` — which the engine resolves via batch $in queries, and a single related column is a dotted `fields` path (`fields: ['owner.name']`). | -| **aggregations** | `{ function: Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct'>; field?: string; alias: string; distinct?: boolean; … }[]` | optional | Aggregation functions | +| **aggregations** | `{ function: Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct'>; field?: string; alias: string; filter?: any }[]` | optional | Aggregation functions | | **groupBy** | `(string \| { field: string; dateGranularity?: Enum<'day' \| 'week' \| 'month' \| 'quarter' \| 'year'>; alias?: string })[]` | optional | GROUP BY targets (strings or `{field, dateGranularity?}` objects for date bucketing) | | **having** | `any` | optional | HAVING — filter over the AGGREGATED rows (aggregation aliases + groupBy projections); applied engine-side after aggregation | | **windowFunctions** | `never` | optional | [REMOVED] `query.windowFunctions` was removed in @objectstack/spec 17 (#4286, ADR-0049) — `find()` never applied it: no engine or driver read the key on the query path, so every OVER clause it declared was silently dropped. Delete the key. Window functions are a SQL-driver capability behind `SqlDriver.findWithWindowFunctions(object, query)` (embedder-level; not on the `IDataDriver` contract or the REST surface); request-level analytics are `aggregations` + `groupBy`. | diff --git a/docs/protocol-upgrade-guide.md b/docs/protocol-upgrade-guide.md index 8595b3734f..addae3ef97 100644 --- a/docs/protocol-upgrade-guide.md +++ b/docs/protocol-upgrade-guide.md @@ -222,6 +222,8 @@ That count turned out to be incomplete, and #6776 finishes it: five more keys th Finally it narrows the aggregation vocabulary: `array_agg` and `string_agg` leave `AggregationFunction` (#6188, ADR-0049). The enum declared eight functions and the SQL family compiles five — `SqlDriver.mapAggregateFunc` and the Turso `RemoteTransport.aggregate` each lower `count`/`sum`/`avg`/`min`/`max` and route the rest to one refusal — so three were declared-but-unenforced against the backends this platform targets. What makes these two worse than an ordinary inert declaration is that another package had to carry a denylist for them: `service-analytics` subtracted `array_agg` and `string_agg` by name in `UNSUPPORTED_AGGREGATES`, because without that subtraction they reached the Cube strategy's `default` and returned `COUNT(*)` — a row count in place of the requested value, with no error and no log. The maintainer SPLIT the three rather than retiring them as a block (2026-08-07), and the split is the point: `count_distinct` STAYS and takes the enforce leg — one portable lowering (`COUNT(DISTINCT x)`), a dashboard staple, already lowered by `service-analytics` — with its SQL implementation following on its own card, so that declaration leads its implementation by decision rather than by drift. These two take the remove leg: display conveniences with no measured pull, and `string_agg` never had one shape to lower to (the delimiter is a second argument in PostgreSQL, a `SEPARATOR` clause in MySQL, a differently named function in SQL Server). This is an enum VALUE, not a key, so — as with `crypto.hash` above — there is no `retiredKey()` tombstone: the enum error map carries the prescription, keyed on the received value so only the two spellings that used to be legal are told they "were removed". Of the two authoring surfaces only one is stored metadata: the conversion rewrites `dataset.measures[].aggregate`, dropping the measure outright (a measure with neither `aggregate` nor `derived` fails the dataset's own refinement, so stripping just the key would emit an item that cannot parse) plus any derived measure the drop strands, with a notice each. Nothing is lost: `compileDataset` refused both by name already, so such a measure never produced a number. `QueryAST.aggregations[].function` is a request surface with no stored source — one semantic TODO below. The mongodb and in-memory backends that implemented these two are inside the #5499 freeze and are untouched; their code is simply no longer reachable through a spec-valid request. +The same aggregation node loses one more member, and it is the sharper class of the two: `aggregations[].distinct` is removed (#6815, ADR-0049, maintainer ruling 2026-08-09). The functions above were declared and UNLOWERED — a caller on a SQL datasource got a refusal. This flag was declared and lowered by exactly ONE of the six faces that read an aggregation: the engine's in-memory fallback deduplicated the values before applying the function, while `SqlDriver.aggregate`, the Turso `RemoteTransport.aggregate`, `driver-mongodb`'s `buildAggregationStage`, `driver-memory`'s `computeAggregate` and service-analytics' `AGGREGATE_SQL` all ignored it. So the same query answered a deduplicated `sum` on the fallback path and an ordinary `sum` on every SQL datasource, with the engine choosing between the two per query — by driver, by a non-UTC date bucket, by whether the driver aggregates natively at all. That is the divergence class #6203 and #5907 each closed on this axis, still open on this key, and it is worse to sit on because the wrong answer is a PLAUSIBLE NUMBER rather than a refusal: no error, no log, nothing for a dashboard author to notice. It survived the #4286 sweep of this very schema because that sweep asked which members no executor reads, and this one had a reader — the wrong question for a key whose defect is WHICH executor reads it. Remove rather than enforce, per the ruling: `count_distinct` (which just took the enforce leg above, and whose SQL lowering #6409 landed) already covers the only deduplicating spelling with measured demand, while `SUM(DISTINCT …)` / `AVG(DISTINCT …)` are near-universally a modelling mistake and would have to be lowered across five faces, two of them frozen under #5499, to buy it. The blast radius inside the fallback is narrower than the key suggests and was measured rather than assumed: only `sum` and `avg` ever changed answer — `count` returned from its own branch before reaching the dedupe, `count_distinct` fed a Set, and dedupe does not move `min`/`max`. `AggregationNodeSchema` is non-strict, so the key is `retiredKey()`-tombstoned rather than bare-deleted: a plain deletion would have made zod silently STRIP what callers still send, trading a divergent flag for an ignored one (#3733, ADR-0104). One tombstone covers every aggregation door, because `QuerySchema.aggregations` and `EngineAggregateOptionsSchema.aggregations` reuse that one schema by reference. No conversion: a request surface with no stored source — one semantic TODO below, the disposition every other `data.query.*` retirement in this major already takes. + One entry in this step is not a removal at all but a SECURE-DEFAULT FLIP, the shape protocol 12 last used for `api.requireAuth`: an omitted `ActionDescriptor.resumeAuthority` resolves to `'service'` instead of `'any'`, so a pausing node type that never states who may continue its pauses is refused on the generic resume route rather than open to it (#5561, ADR-0044's 2026-07-28 amendment). Nothing is removed and no metadata shape changes — the field has been optional since step one of the same issue — so tsc reports nothing and only the MEANING of silence moved. That is exactly why it needs a ledger entry: a third-party plugin author has no compile error to discover it with, and the one-line prescription (declare `resumeAuthority` on the descriptor) has to arrive before a user meets a run that will not continue. The same descriptor loses a key in this step, and the pairing is the point (#6748, ADR-0049). `ActionDescriptor.isAsync` and `ActionDescriptor.supportsPause` were two spellings of one capability — "this node type can suspend the run" — and #6667 split them by evidence rather than by preference: `supportsPause` took the ENFORCE leg (the engine now refuses a suspension the descriptor never declared, at the one seam every suspension passes through), and `isAsync` takes the REMOVE leg, because a fresh three-repo measurement found zero readers and no consumer it could grow into. What makes the duplicate worse than an ordinary inert key is that five shipped descriptors WROTE it, so the platform itself modelled a declaration that decided nothing — and a plugin author copying `screen` (which declared BOTH) had no way to tell which of the two the runtime honoured. It is tombstoned rather than deleted, so the answer arrives as a rejection carrying the fix; and because a descriptor lives in executor TypeScript rather than in stored metadata, its prescription is a semantic entry below rather than a conversion `os migrate meta` could replay. @@ -327,6 +329,9 @@ The same descriptor loses a key in this step, and the pairing is the point (#674 - **`query-array-string-agg-retired`** — `data.query.aggregations[].function ('array_agg' / 'string_agg')` → an ordinary `fields` query, shaped in the caller — or a stored field that materialises the roll-up. For a deduplicated COUNT the live spelling is unchanged: `count_distinct` stays declared - Why not automatic: The stored half of this retirement is a conversion (`dataset-measure-array-string-agg-removed`); this entry is the REQUEST half. `QueryAST` is never stored in stack metadata — it is the client SDK builder's output and the `POST /data/:object/query` body — so there is no source for the chain to rewrite and callers move their own queries. Both values were declared-but-unlowered on the SQL family: `SqlDriver.mapAggregateFunc` and the Turso `RemoteTransport.aggregate` compile five functions and refuse the rest, so a caller following the schema against a SQL datasource got a refusal, not an array. They did run on `driver-mongodb` and on the engine's in-memory fallback, which is what makes this the one narrowing in the batch that removes reachable behaviour: an aggregation that worked on one backend and failed on another is exactly the unpredictability the ruling ended, and #5499 has both of those backends frozen. `count_distinct` was deliberately NOT retired with them (maintainer, 2026-08-07) — it takes ADR-0049's enforce leg, and its SQL lowering is a separate drivers-side card. ADR-0049, #6188. - Done when: No caller sends `array_agg` or `string_agg` in `aggregations[].function`; list-style roll-ups are assembled by the caller from an ordinary `fields` query, or materialised as a stored field. A query still carrying either value fails to parse with the removal prescription naming it, and authoring it is a `tsc` error at the call site; `count_distinct` continues to parse and is unaffected. +- **`aggregation-node-distinct-retired`** — `data.query.aggregations[].distinct` → the `count_distinct` aggregation FUNCTION for a deduplicated count — the one deduplicating spelling every face computes, lowered to `COUNT(DISTINCT field)` on both SQL faces since #6409. `SUM(DISTINCT …)` / `AVG(DISTINCT …)` get no replacement: no backend ever computed them here, and a per-row measure that needs deduplicating before summing is a modelling problem to fix in the data + - Why not automatic: A DIVERGENCE, not an inert declaration — which is why it outlived the #4286 sweep that dispositioned every other `data.query.*` member. That sweep asked which keys no executor reads; this one HAD an executor, exactly one out of six. The engine's in-memory fallback (`objectql/src/in-memory-aggregation.ts`) deduplicated the values before applying the function, while `SqlDriver.aggregate`, the Turso `RemoteTransport.aggregate`, `driver-mongodb`'s `buildAggregationStage`, `driver-memory`'s `computeAggregate` and service-analytics' `AGGREGATE_SQL` all ignored the key. So `{ function: 'sum', field: 'amount', distinct: true }` answered a deduplicated sum when the engine fell back in memory and an ordinary sum on every SQL datasource: one query, two numbers, chosen by which backend happened to serve it — and unlike the #6203 / #5907 divergences closed on the same axis, the wrong answer here is a plausible NUMBER rather than a refusal, so nothing surfaced it to the author. Measured blast radius inside the fallback: `sum` and `avg` only — `count` returned from its own branch before reaching the dedupe, `count_distinct` fed the values into a Set (dedupe-then-Set is Set), and dedupe does not move `min`/`max`. ENFORCE was weighed and rejected (maintainer ruling 2026-08-09): `count_distinct` already covers the only spelling anyone has measured demand for, and lowering `SUM(DISTINCT …)` across five faces — two of them frozen under #5499 — buys a shape that is near-universally a modelling mistake. A REQUEST surface — `QueryAST` is the client SDK builder's output and the `POST /data/:object/query` body, never stored in stack metadata — so there is no source for the chain to rewrite and callers move their own queries: the #4286 disposition for `joins`/`cursor`/`distinct`/`windowFunctions`, applied verbatim one level down. ADR-0049, #6815. + - Done when: No caller sends `distinct` inside an `aggregations[]` entry, on the wire or through the SDK; a deduplicated count is written as `{ function: 'count_distinct', field }` and reads the same number on every backend. A query still carrying the key fails to parse with the removal prescription — including through `EngineAggregateOptionsSchema`, which reuses `AggregationNodeSchema` by reference — and `POST /api/v1/data/:object/query` answers `400 VALIDATION_FAILED` with a `fields[]` entry at `aggregations..distinct` instead of serving a number. Authoring it is a `tsc` error at the call site. ⚠️ The observable NUMBERS change on exactly one path and that is the point of the change: a `sum`/`avg` that used to be deduplicated by the in-memory fallback now answers what every SQL face has always answered for the same query. Verify against the SQL answer, not against the pre-upgrade fallback answer — the two disagreed, which is why the key is gone. - **`workflow-service-slot-retired`** — `CoreServiceName 'workflow' / IWorkflowService / WorkflowProtocol / discovery routes.workflow / RestApiRouteCategory workflow` → the live mechanisms the slot only ever pointed at: `state_machine` validation rules for record state machines, approval flow nodes on the approvals runtime (ADR-0019) for approvals, lifecycle hooks + `record_change` flows (service-automation) for record-triggered automation - Why not automatic: The workflow slot was declared end to end and implemented nowhere: no code in either repository ever registered or resolved it (ADR-0115 Evidence 5 — the only touches were plugin-dev's retired stub probe and the generic discovery walk), no implementation of any WorkflowProtocol method ever existed, and no host ever mounted `/api/v1/workflow` (the pre-#3586 DEFAULT_DISPATCHER_ROUTES listed it among routes that never existed). Every part of it was ADR-0078's silently-inert declaration: a CoreServiceName nothing filled, a contract nothing implemented, a protocol nothing served, a discovery route field no builder could truthfully populate. These are TS/API surfaces and a discovery RESPONSE field — never stored in stack metadata, so there is no source for the chain to rewrite; consumers of the deleted types move their imports themselves. ADR-0049 / ADR-0078, #4451. - Done when: No import of IWorkflowService, WorkflowProtocol or the Get/WorkflowState/Config/Transition types resolves; no code calls getService('workflow') or reads discovery `routes.workflow` / `services.workflow`; record state machines, approvals and record-triggered automation go through the replacement mechanisms. Discovery output on a default boot is unchanged (the slot was always reported unavailable; now it is simply absent). diff --git a/packages/objectql/src/in-memory-aggregation.test.ts b/packages/objectql/src/in-memory-aggregation.test.ts index 8cdb74a834..cb7b5d210d 100644 --- a/packages/objectql/src/in-memory-aggregation.test.ts +++ b/packages/objectql/src/in-memory-aggregation.test.ts @@ -94,6 +94,75 @@ describe('applyInMemoryAggregation', () => { expect(east!.owner_str).toBeNull(); }); + // #6815 — the per-aggregation `distinct` flag, retired under ADR-0049. This + // module was the ONLY reader of it in the repo: it deduplicated the values + // before applying the function while `driver-sql`, `driver-turso`, + // `driver-mongodb`, `driver-memory` and the service-analytics SQL builder + // all ignored the key. So the engine's choice of path — a driver without + // native aggregation, a non-UTC date bucket (#1982), a partial SQL driver — + // silently decided WHICH NUMBER a report showed, and both numbers looked + // right. `AggregationNodeSchema` tombstones the key, so a parsed query can + // no longer carry it; this pins the behaviour for the unparsed shapes that + // reach the fallback directly (an engine-options bag, a driver forwarding a + // raw body), which is where a resurrected dedupe limb would hide. + // + // The dedupe limb had NO test of its own before this one — the flag was + // honoured here for its whole life without a case pinning the divergence. + // The blast radius, measured rather than assumed — it is NARROWER than + // "every aggregation": `count` returned from its own branch before ever + // calling `collectValues`, so the flag never reached it; `count_distinct` + // fed the values into a `Set` (dedupe-then-Set is Set); and dedupe does not + // move `min`/`max`. Exactly TWO functions changed answer — `sum` and `avg` — + // and this dataset makes both of them move. The pin covers all six anyway: + // the guarantee is "no function reads the key", not "two functions stopped". + const DUPES = [ + { region: 'East', amount: 100, owner: 'alice' }, + { region: 'East', amount: 100, owner: 'alice' }, + { region: 'East', amount: 400, owner: 'bob' }, + ]; + + it('ignores a `distinct` flag on an aggregation — uniform with every SQL face', () => { + // Cast: `distinct` is exactly the key `AggregationNode` no longer has, + // which is what this test exists to pin. + const out = applyInMemoryAggregation(DUPES, { + groupBy: ['region'], + aggregations: [ + { function: 'sum', field: 'amount', alias: 'total', distinct: true }, + { function: 'avg', field: 'amount', alias: 'avg_amount', distinct: true }, + { function: 'min', field: 'amount', alias: 'lo', distinct: true }, + { function: 'max', field: 'amount', alias: 'hi', distinct: true }, + { function: 'count', field: 'amount', alias: 'n', distinct: true }, + { function: 'count_distinct', field: 'amount', alias: 'uniques', distinct: true }, + ] as never, + }); + const east = out.find((r) => r.region === 'East')!; + // The two that moved. `SUM(amount)` over 100/100/400 is 600 on every SQL + // face; the removed limb collapsed the pair and answered 500. + expect(east.total).toBe(600); + // `AVG(amount)` is 600/3 = 200; the removed limb answered 500/2 = 250. + expect(east.avg_amount).toBe(200); + // The four that never moved, pinned so a reader can see the real radius. + expect(east.lo).toBe(100); + expect(east.hi).toBe(400); + expect(east.n).toBe(3); + // …and the live deduplicating spelling still deduplicates, in its own arm, + // with or without the retired key present. + expect(east.uniques).toBe(2); + }); + + it('`count_distinct` is unaffected by the removal — dedupe lives in its own arm', () => { + const out = applyInMemoryAggregation(DUPES, { + groupBy: ['region'], + aggregations: [ + { function: 'count_distinct', field: 'owner', alias: 'owners' }, + { function: 'count_distinct', field: 'amount', alias: 'amounts' }, + ], + }); + const east = out.find((r) => r.region === 'East')!; + expect(east.owners).toBe(2); + expect(east.amounts).toBe(2); + }); + // #3839 — this used to be the literal string `'(null)'`, which the pushed-down // SQL path never produced (a NULL group column stays SQL NULL). The engine // picks between the two paths per query, so the bucket key's TYPE changed diff --git a/packages/objectql/src/in-memory-aggregation.ts b/packages/objectql/src/in-memory-aggregation.ts index c5d267049f..c3483092e7 100644 --- a/packages/objectql/src/in-memory-aggregation.ts +++ b/packages/objectql/src/in-memory-aggregation.ts @@ -20,8 +20,18 @@ // are deleted rather than left unreachable — a `switch` case on a value // the enum no longer has does not type-check, and dead arms are how a // retired vocabulary comes back by accident. -// * `distinct: true` on aggregations (collapse duplicates before applying -// the function) +// * `distinct: true` on aggregations is GONE (#6815, ADR-0049). This module +// was its only reader in the whole repo: it collapsed duplicates before +// applying the function while `driver-sql`, `driver-turso`, +// `driver-mongodb`, `driver-memory` and the service-analytics SQL builder +// all ignored the key, so a `sum` with `distinct: true` answered a +// deduplicated total here and an ordinary total on every SQL datasource — +// the same divergence class as `array_agg`/`string_agg` above, except the +// wrong answer was a plausible NUMBER rather than a refusal, so nothing +// surfaced it. The key is tombstoned on `AggregationNodeSchema` and the +// dedupe limb is deleted rather than left unreachable, for the same reason +// the retired function arms were. `count_distinct` is unaffected: it +// deduplicates inside its own arm, and every SQL face compiles it (#6409). // * `filter: FilterCondition` on aggregations is **not** evaluated here — // the engine routes filtered aggregations through the driver where // possible; the in-memory fallback ignores the per-aggregation filter and @@ -170,7 +180,7 @@ function aggregateBucket(rows: any[], aggregations: AggregationNode[]): Record r?.[field]); - const seen = new Set(); - const out: any[] = []; - for (const r of rows) { - const v = r?.[field]; - if (seen.has(v)) continue; - seen.add(v); - out.push(v); - } - return out; +/** + * Every row's value for `field`, in row order, duplicates included. + * + * The `distinct` parameter is gone with `AggregationNode.distinct` (#6815): + * this was the repo's only reader of that flag, and honouring it here while + * five other faces ignored it is what made one query answer two numbers. Each + * function arm decides for itself what to do with duplicates — `count_distinct` + * folds them into a `Set`, every other arm counts them, which is what + * `SUM(x)` / `AVG(x)` mean on every SQL face. + */ +function collectValues(rows: any[], field: string): any[] { + return rows.map((r) => r?.[field]); } function toNumber(v: any): number { diff --git a/packages/rest/src/request-schema-gate.conformance.test.ts b/packages/rest/src/request-schema-gate.conformance.test.ts index 5d18cd58a3..e8b50e9650 100644 --- a/packages/rest/src/request-schema-gate.conformance.test.ts +++ b/packages/rest/src/request-schema-gate.conformance.test.ts @@ -109,6 +109,14 @@ const CASES: GateCase[] = [ [{ limit: 5 }], // array body { limit: 'ten' }, // mistyped clause — used to be forwarded as-is { joins: [] }, // retired key (#4286) — the tombstone must be audible + // #6815 — the per-aggregation `distinct` flag, retired under ADR-0049. + // It is the one retired key on this route that is NOT top-level: it sits + // inside an `aggregations[]` entry, so the tombstone is only audible if + // entry validation descends into the array. Before the retirement this + // body parsed clean and answered a DEDUPLICATED sum on an in-memory + // fallback and an ordinary sum on every SQL datasource — one query, two + // numbers, both plausible. + { aggregations: [{ function: 'sum', field: 'amount', alias: 'total', distinct: true }] }, ], valid: { where: { status: 'active' }, limit: 5 }, spy: 'findData', diff --git a/packages/spec/authorable-surface/data.json b/packages/spec/authorable-surface/data.json index 98317abc7a..bb215f37b6 100644 --- a/packages/spec/authorable-surface/data.json +++ b/packages/spec/authorable-surface/data.json @@ -17,7 +17,7 @@ "data/AddressValue:state", "data/AddressValue:street", "data/AggregationNode:alias", - "data/AggregationNode:distinct", + "data/AggregationNode:distinct [RETIRED]", "data/AggregationNode:field", "data/AggregationNode:filter", "data/AggregationNode:function", diff --git a/packages/spec/liveness/README.md b/packages/spec/liveness/README.md index df1129ff97..670ff56591 100644 --- a/packages/spec/liveness/README.md +++ b/packages/spec/liveness/README.md @@ -587,7 +587,7 @@ for t, v in r['types'].items(): | report | 21 | 0 | 0 | – | dataset-bound (ADR-0021); the aria/performance LEDGER entries were stale — the keys left the schema in the report-liveness close-out; deleted 2026-07-30 as hygiene. Audit-era `chart` DEAD superseded (framework#1890 / #3441) | | dashboard | 33 | 0 | 8 | – | ADR-0021 dataset widgets (#3251; DashboardWidgetSchema `.strict()`); `aria`/`performance` (and widget `performance` + PerformanceConfigSchema) REMOVED 2026-07-30 (#3896 close-out sweep — no renderer applied any of them); audit-era `globalFilters`/`dateRange` DEAD superseded (framework#2501) | **#4956**: `widgets` DRILLED — the row jumps 20 → 41 classified because all 22 widget-level keys enter the count at once. They had never been classified at all: the entry carried one blanket `live` plus a `note` asserting they were classified "in the DashboardWidgetSchema subtree", and no such subtree existed in any of the 28 ledger files. That gap, not any evidence, is what carried `widgets[].responsive` through the #3896 sweep that removed both its sibling `widgets[].performance` and its literal namesake `view.responsive` — `view` is drilled, so `list.responsive` got asked and went out. New dead 6 = `responsive` (retired #4876/#4995, tombstone keeps the row) + `colorVariant` + `actionUrl`/`actionType`/`actionIcon` + `aria`. The action trio is the sharpest: no renderer draws a per-widget action button at all (every `actionUrl` read in DashboardRenderer is scoped to `header.actions[]`), yet `validate-dashboard-action-refs.ts` enforces reference integrity on it and its docblock calls it "the per-widget button" — a lint guarding an affordance that does not exist. `requiresService` is the counter-example worth remembering: dead by every objectui measurement, and LIVE server-side (`filterDashboardForUser`, ADR-0057 D10) — judging a widget key from the renderer repo alone would have retired an enforced gate. `compareTo` is `live` on ONE path only (inline object-provider charts); on the ADR-0021 dataset path the string arms are dropped and `{ offset }` throws in the executor | -| query | 16 | 1 | 4 | – | **not a metadata type** — the REQUEST surface (`QuerySchema`: client SDK QueryBuilder output; the `POST /data/:object/query` body), governed via `SPEC_ONLY_SCHEMAS` (#4286). The gate's one-level walk resolves 1 experimental; the 7 marker-experimental search affordances sit one level deeper, below the walk — resolved from `[EXPERIMENTAL — not enforced]` describe markers, not ledger entries (search `fuzzy`/`operator`/`boost`/`minScore`/`language`/`highlight` + `aggregations[].filter` — declared engine affordances no executor receives). The #4286 sweep closed out same-release: `having` ENFORCED 2026-07-31 (engine-side post-aggregation filter, both paths; was finding 1); dead 4 = the tombstoned removals `joins`/`windowFunctions`/`cursor`/`distinct` — REMOVED 2026-07-31 (retiredKey keeps each in the walked shape so the rows stay; protocol-17 semantic migrations; the JoinNode + WindowFunctionNode clusters and the `QueryBuilder.cursor()`/`.distinct()` producers deleted with their keys; `distinct`'s mis-wired REST count suppression deleted too — finding 2) | +| query | 15 | 1 | 5 | 0 | **not a metadata type** — the REQUEST surface (`QuerySchema`: client SDK QueryBuilder output; the `POST /data/:object/query` body), governed via `SPEC_ONLY_SCHEMAS` (#4286). The gate's one-level walk resolves 1 experimental; the 7 marker-experimental search affordances sit one level deeper, below the walk — resolved from `[EXPERIMENTAL — not enforced]` describe markers, not ledger entries (search `fuzzy`/`operator`/`boost`/`minScore`/`language`/`highlight` + `aggregations[].filter` — declared engine affordances no executor receives). The #4286 sweep closed out same-release: `having` ENFORCED 2026-07-31 (engine-side post-aggregation filter, both paths; was finding 1); dead 4 = the tombstoned removals `joins`/`windowFunctions`/`cursor`/`distinct` — REMOVED 2026-07-31 (retiredKey keeps each in the walked shape so the rows stay; protocol-17 semantic migrations; the JoinNode + WindowFunctionNode clusters and the `QueryBuilder.cursor()`/`.distinct()` producers deleted with their keys; `distinct`'s mis-wired REST count suppression deleted too — finding 2). **#6815** adds the 5th dead: `aggregations[].distinct` REMOVED 2026-08-09 (live → dead, `-1` live). It is the one member of this ledger the #4286 sweep could not have caught with the question it asked — that sweep looked for keys NO executor reads, and this one had a reader: the objectql in-memory fallback deduplicated before applying the function while all five other faces (driver-sql, driver-turso, driver-mongodb, driver-memory, service-analytics' `AGGREGATE_SQL`) ignored it, so one query answered two plausible NUMBERS depending on which backend served it. The lesson for the next audit is the question, not the key: a per-key `live` verdict is only as good as the count of faces it was measured across, and this row's 2026-07-31 evidence (`in-memory-aggregation.ts:167,204-206`) was TRUE and still the wrong verdict. `count_distinct` is the surviving spelling (enforce leg, #6409) | | datasource | 30 | 0 | 0 | 0 | seeded 2026-08-01 (#4487) — the **highest dead ratio of any governed type** (20 of 43), and it was ungoverned until now, which is not a coincidence: #4410/#4465/#4481 found six inert keys here by hand, two security-shaped (`schemaMode` left an external DB constructible as `managed` with DDL ungated; `ssl` configured nothing while looking configured). Dead set = `capabilities.*` (all 11 — the engine gates pushdown on the runtime driver's `supports.*` object, a non-overlapping vocabulary), `healthCheck.*` (3 — nothing schedules a datasource probe; the 20 `healthCheck` hits in the repo all belong to the PLUGIN health monitor and other surfaces), `retryPolicy.*` (4 — `retryPolicy` IS enforced on `hook` and `job`, which is what makes this one read alive; the shapes differ), `external.label`, `external.requirePermission`. **`capabilities.readOnly` is the one to know**: it reads as a safety switch, gates nothing, and two shipped prescriptions pointed authors at it until #4487 — `external.allowWrites: false` is the enforced write gate. `config` is a `z.record`, so its per-driver keys sit outside the walk (recorded in the entry's note, not silently skipped) **批 A CLOSED 2026-08-02 (#4583)**: the `capabilities` block — 11 flags, every one dead and authorWarn'd — was REMOVED rather than bridged; pushdown comes from the runtime driver's own `supports.*`, so there was nothing to connect it to. Its rows are deleted (strict-removal route), which is why dead falls 20 → 9. `readOnly` was the reason the audit was worth doing: it read as a safety switch, gated nothing, and had already been MOVED twice toward somewhere it might be enforced (#4410, #4465) — the shipped CRM example called a datasource a read replica on the strength of it while the datasource took writes. Removing it does NOT hand the author a working alternative: `external.allowWrites` only gates FEDERATED datasources, so a managed one has no read-only gate at all (#4584). Remaining 9 = healthCheck ×3 + retryPolicy ×4 + external ×2, batches B/C/D of #4583 **BATCHES B/C/D CLOSED 2026-08-02 — datasource now has ZERO dead properties**, down from the 20 it was seeded with (the highest dead ratio of any governed type). `retryPolicy` ×4 and `healthCheck` ×3 went as whole blocks, `external.label` / `external.requirePermission` as keys. None was bridgeable: each already had a different LIVE mechanism doing the job — the boot policy, the driver handle's on-demand `ping()`/`checkHealth()`, the top-level `label`, and ordinary permission sets + RLS. The `retryPolicy` rejection deliberately refuses to offer a rename: `hook`/`job` retryPolicy ARE enforced but spell the delay `backoffMs`, and that inconsistency is itself the evidence nothing read the datasource one (#4488's sharpest trap) | | webhook | 11 | 0 | 0 | – | **not a registered metadata type** — governed via the gate's spec-only schema override (`SPEC_ONLY_SCHEMAS`), not `getMetadataTypeSchema`; folding it onto the registry is the #3490 reassessment. This row once read 0/1/16 ("the ENTIRE authoring surface is dead", #3461) and both halves of that were CLOSED same-quarter: #3489 built the materializer bridge (authored `webhooks:` entries now land as `sys_webhook` dispatcher rows) and #3494 pruned the aspirational props outright — so the surviving surface is fully live. Kept in the table as the worked example that a dead verdict is a worklist entry, not a tombstone: enforce-or-remove resolved this one by ENFORCING | | app | 45 | – | 9 | – | seeded 2026-08-01 (#4488). Dead 9 = the seven #4142 `retiredKey` tombstones (version/aria/objects/apis/sharing/embed/mobileNavigation — rows stay while the tombstones hold the keys in the walked shape) + `homePageId` (#4667 tombstone — the landing IS the first nav item; root landing follows `isDefault` routing) + `areas.description` (benign, docs-shaped, kept and not warned). RETIRED 17.0.0 (#4509, rows deleted — the selector schema is strict): selector `includeAll` (deliberately DISOBEYED, not merely unread — selectors are mandatory-scope and an "All" row would clear the scope, leaking system metadata through Studio's package filter; STUDIO_APP authored it against a renderer that ignored it) and `placement` (no renderer read it; "topbar" placed nothing). Nav walk covers the union's `object` variant; other variants hand-verified live, and the `actionDef` dispatch gap closed in #4509 | **#4651**: the **fail-open area gates** `areas.visible` / `areas.requiredPermissions` — this ledger's most important app finding — are REMOVED, rows DELETED (strict removal; retained rows would report ORPHAN). They were not merely unread: `filterAppForUser` never reads `item.areas` at all and the shell renders every area, so a "hidden" or permission-gated area was served to everyone, while the identically named per-ITEM and per-APP keys ARE enforced. Route B (remove) over route A (enforce): enforcing needs semantics decided first (does filtering an area remove its items everywhere? does the server bind `user` for area CEL?), which the 17.0.0 window could not hold. Boundary unchanged and still recorded on `areas.navigation`: per-item gating inside an area is shell-side only. **#4667**: `homePageId` TOMBSTONED (row stays — retiredKey keeps it in the walked shape) and `areas.order` row DELETED (strict removal); `areas.order` read alive because the per-ITEM `order` really is sorted (NavigationRenderer.tsx:1154) while no renderer ever sorted areas. | diff --git a/packages/spec/liveness/query.json b/packages/spec/liveness/query.json index ac10cf02c8..98daa82bc1 100644 --- a/packages/spec/liveness/query.json +++ b/packages/spec/liveness/query.json @@ -1,6 +1,6 @@ { "type": "query", - "_note": "QuerySchema (BaseQuerySchema + expand) — packages/spec/src/data/query.zod.ts. NOT a registered metadata type: QueryAST is the REQUEST surface (the client SDK QueryBuilder's output; the POST /data/:object/query body), governed via the gate's SPEC_ONLY_SCHEMAS override like webhook (#4286). It is authorable by every API caller yet never stored as stack metadata, which is why no other gate reached it: this ledger read the metadata-type registry, check:authorable-surface walks the same registry, and check:exported-any is a type-resolution gate. Seeded 2026-07-31 from the #4286 sweep; closed out same day — joins/windowFunctions/cursor/distinct REMOVED (tombstoned; protocol-17 semantic migrations; the cursor/distinct SDK producers deleted with their keys), having ENFORCED engine-side, search sub-flags + aggregations.filter marked experimental at the schema. The main executor evidence below: objectql engine find/aggregate and the drivers' find paths.", + "_note": "QuerySchema (BaseQuerySchema + expand) — packages/spec/src/data/query.zod.ts. NOT a registered metadata type: QueryAST is the REQUEST surface (the client SDK QueryBuilder's output; the POST /data/:object/query body), governed via the gate's SPEC_ONLY_SCHEMAS override like webhook (#4286). It is authorable by every API caller yet never stored as stack metadata, which is why no other gate reached it: this ledger read the metadata-type registry, check:authorable-surface walks the same registry, and check:exported-any is a type-resolution gate. Seeded 2026-07-31 from the #4286 sweep; closed out same day — joins/windowFunctions/cursor/distinct REMOVED (tombstoned; protocol-17 semantic migrations; the cursor/distinct SDK producers deleted with their keys), having ENFORCED engine-side, search sub-flags + aggregations.filter marked experimental at the schema. Re-opened 2026-08-09 for ONE key the #4286 sweep did not reach: `aggregations[].distinct` (#6815) — dispositioned REMOVED. #4286 swept the query's own members plus `aggregations.filter`; the per-aggregation `distinct` was neither a top-level key nor an unread one, so it passed both filters of that sweep while diverging by BACKEND rather than by declaration. The main executor evidence below: objectql engine find/aggregate and the drivers' find paths.", "props": { "object": { "status": "live", @@ -78,9 +78,9 @@ "evidence": "result column naming in both the SQL builders and packages/objectql/src/in-memory-aggregation.ts" }, "distinct": { - "status": "live", - "verifiedAt": "2026-07-31", - "evidence": "packages/objectql/src/in-memory-aggregation.ts:167,204-206 (dedupe before applying the function)" + "status": "dead", + "verifiedAt": "2026-08-09", + "note": "REMOVED 2026-08-09 (#6815, ADR-0049) — tombstoned at the schema (retiredKey carries the prescription; authoring it is a tsc error and a parse error). The 2026-07-31 verdict above was `live` on TRUE evidence and the wrong question: the flag DID have a reader, and one reader out of six faces is what made it a defect. `objectql`'s in-memory fallback deduplicated before applying the function while driver-sql, driver-turso, driver-mongodb, driver-memory and service-analytics' AGGREGATE_SQL all ignored the key, so `{ function: 'sum', field: 'amount', distinct: true }` answered a deduplicated sum on the fallback path and an ordinary sum on every SQL datasource — one query, two plausible NUMBERS, chosen by which backend served it, with no refusal to surface the split. Measured blast radius inside the fallback: `sum` and `avg` only — `count` returned from its own branch before reaching collectValues, `count_distinct` fed a Set (idempotent), and dedupe does not move min/max. No conversion strips it: QueryAST is a request surface, never stored in stack metadata, so the removal is the protocol-17 semantic migration `aggregation-node-distinct-retired` (the #4286 disposition for the query-level keys, verbatim). The entry stays because retiredKey keeps the key in the walked shape (the rls.priority precedent). The live deduplicating spelling is the `count_distinct` FUNCTION, which took ADR-0049's enforce leg in #6409 and is lowered on both SQL faces; `SUM(DISTINCT …)` / `AVG(DISTINCT …)` get no replacement — no backend ever computed them here." } } }, diff --git a/packages/spec/spec-changes.json b/packages/spec/spec-changes.json index 4cfb52686a..eea56630e2 100644 --- a/packages/spec/spec-changes.json +++ b/packages/spec/spec-changes.json @@ -550,6 +550,13 @@ "toMajor": 17, "rationale": "The stored half of this retirement is a conversion (`dataset-measure-array-string-agg-removed`); this entry is the REQUEST half. `QueryAST` is never stored in stack metadata — it is the client SDK builder's output and the `POST /data/:object/query` body — so there is no source for the chain to rewrite and callers move their own queries. Both values were declared-but-unlowered on the SQL family: `SqlDriver.mapAggregateFunc` and the Turso `RemoteTransport.aggregate` compile five functions and refuse the rest, so a caller following the schema against a SQL datasource got a refusal, not an array. They did run on `driver-mongodb` and on the engine's in-memory fallback, which is what makes this the one narrowing in the batch that removes reachable behaviour: an aggregation that worked on one backend and failed on another is exactly the unpredictability the ruling ended, and #5499 has both of those backends frozen. `count_distinct` was deliberately NOT retired with them (maintainer, 2026-08-07) — it takes ADR-0049's enforce leg, and its SQL lowering is a separate drivers-side card. ADR-0049, #6188." }, + { + "surface": "data.query.aggregations[].distinct", + "replacement": "the `count_distinct` aggregation FUNCTION for a deduplicated count — the one deduplicating spelling every face computes, lowered to `COUNT(DISTINCT field)` on both SQL faces since #6409. `SUM(DISTINCT …)` / `AVG(DISTINCT …)` get no replacement: no backend ever computed them here, and a per-row measure that needs deduplicating before summing is a modelling problem to fix in the data", + "migrationId": "aggregation-node-distinct-retired", + "toMajor": 17, + "rationale": "A DIVERGENCE, not an inert declaration — which is why it outlived the #4286 sweep that dispositioned every other `data.query.*` member. That sweep asked which keys no executor reads; this one HAD an executor, exactly one out of six. The engine's in-memory fallback (`objectql/src/in-memory-aggregation.ts`) deduplicated the values before applying the function, while `SqlDriver.aggregate`, the Turso `RemoteTransport.aggregate`, `driver-mongodb`'s `buildAggregationStage`, `driver-memory`'s `computeAggregate` and service-analytics' `AGGREGATE_SQL` all ignored the key. So `{ function: 'sum', field: 'amount', distinct: true }` answered a deduplicated sum when the engine fell back in memory and an ordinary sum on every SQL datasource: one query, two numbers, chosen by which backend happened to serve it — and unlike the #6203 / #5907 divergences closed on the same axis, the wrong answer here is a plausible NUMBER rather than a refusal, so nothing surfaced it to the author. Measured blast radius inside the fallback: `sum` and `avg` only — `count` returned from its own branch before reaching the dedupe, `count_distinct` fed the values into a Set (dedupe-then-Set is Set), and dedupe does not move `min`/`max`. ENFORCE was weighed and rejected (maintainer ruling 2026-08-09): `count_distinct` already covers the only spelling anyone has measured demand for, and lowering `SUM(DISTINCT …)` across five faces — two of them frozen under #5499 — buys a shape that is near-universally a modelling mistake. A REQUEST surface — `QueryAST` is the client SDK builder's output and the `POST /data/:object/query` body, never stored in stack metadata — so there is no source for the chain to rewrite and callers move their own queries: the #4286 disposition for `joins`/`cursor`/`distinct`/`windowFunctions`, applied verbatim one level down. ADR-0049, #6815." + }, { "surface": "CoreServiceName 'workflow' / IWorkflowService / WorkflowProtocol / discovery routes.workflow / RestApiRouteCategory workflow", "replacement": "the live mechanisms the slot only ever pointed at: `state_machine` validation rules for record state machines, approval flow nodes on the approvals runtime (ADR-0019) for approvals, lifecycle hooks + `record_change` flows (service-automation) for record-triggered automation", @@ -1347,6 +1354,13 @@ "toMajor": 17, "rationale": "The stored half of this retirement is a conversion (`dataset-measure-array-string-agg-removed`); this entry is the REQUEST half. `QueryAST` is never stored in stack metadata — it is the client SDK builder's output and the `POST /data/:object/query` body — so there is no source for the chain to rewrite and callers move their own queries. Both values were declared-but-unlowered on the SQL family: `SqlDriver.mapAggregateFunc` and the Turso `RemoteTransport.aggregate` compile five functions and refuse the rest, so a caller following the schema against a SQL datasource got a refusal, not an array. They did run on `driver-mongodb` and on the engine's in-memory fallback, which is what makes this the one narrowing in the batch that removes reachable behaviour: an aggregation that worked on one backend and failed on another is exactly the unpredictability the ruling ended, and #5499 has both of those backends frozen. `count_distinct` was deliberately NOT retired with them (maintainer, 2026-08-07) — it takes ADR-0049's enforce leg, and its SQL lowering is a separate drivers-side card. ADR-0049, #6188." }, + { + "surface": "data.query.aggregations[].distinct", + "replacement": "the `count_distinct` aggregation FUNCTION for a deduplicated count — the one deduplicating spelling every face computes, lowered to `COUNT(DISTINCT field)` on both SQL faces since #6409. `SUM(DISTINCT …)` / `AVG(DISTINCT …)` get no replacement: no backend ever computed them here, and a per-row measure that needs deduplicating before summing is a modelling problem to fix in the data", + "migrationId": "aggregation-node-distinct-retired", + "toMajor": 17, + "rationale": "A DIVERGENCE, not an inert declaration — which is why it outlived the #4286 sweep that dispositioned every other `data.query.*` member. That sweep asked which keys no executor reads; this one HAD an executor, exactly one out of six. The engine's in-memory fallback (`objectql/src/in-memory-aggregation.ts`) deduplicated the values before applying the function, while `SqlDriver.aggregate`, the Turso `RemoteTransport.aggregate`, `driver-mongodb`'s `buildAggregationStage`, `driver-memory`'s `computeAggregate` and service-analytics' `AGGREGATE_SQL` all ignored the key. So `{ function: 'sum', field: 'amount', distinct: true }` answered a deduplicated sum when the engine fell back in memory and an ordinary sum on every SQL datasource: one query, two numbers, chosen by which backend happened to serve it — and unlike the #6203 / #5907 divergences closed on the same axis, the wrong answer here is a plausible NUMBER rather than a refusal, so nothing surfaced it to the author. Measured blast radius inside the fallback: `sum` and `avg` only — `count` returned from its own branch before reaching the dedupe, `count_distinct` fed the values into a Set (dedupe-then-Set is Set), and dedupe does not move `min`/`max`. ENFORCE was weighed and rejected (maintainer ruling 2026-08-09): `count_distinct` already covers the only spelling anyone has measured demand for, and lowering `SUM(DISTINCT …)` across five faces — two of them frozen under #5499 — buys a shape that is near-universally a modelling mistake. A REQUEST surface — `QueryAST` is the client SDK builder's output and the `POST /data/:object/query` body, never stored in stack metadata — so there is no source for the chain to rewrite and callers move their own queries: the #4286 disposition for `joins`/`cursor`/`distinct`/`windowFunctions`, applied verbatim one level down. ADR-0049, #6815." + }, { "surface": "CoreServiceName 'workflow' / IWorkflowService / WorkflowProtocol / discovery routes.workflow / RestApiRouteCategory workflow", "replacement": "the live mechanisms the slot only ever pointed at: `state_machine` validation rules for record state machines, approval flow nodes on the approvals runtime (ADR-0019) for approvals, lifecycle hooks + `record_change` flows (service-automation) for record-triggered automation", diff --git a/packages/spec/src/data/query.test.ts b/packages/spec/src/data/query.test.ts index 5db3458b3a..a8f72ba15b 100644 --- a/packages/spec/src/data/query.test.ts +++ b/packages/spec/src/data/query.test.ts @@ -4,8 +4,10 @@ import { FieldNodeSchema, SortNodeSchema, AggregationFunction, + AggregationNodeSchema, type QueryAST, } from './query.zod'; +import { EngineAggregateOptionsSchema } from './data-engine.zod'; describe('AggregationFunction', () => { it('should accept valid aggregation functions', () => { @@ -327,15 +329,17 @@ describe('QuerySchema - Aggregations', () => { expect(() => QuerySchema.parse(query)).not.toThrow(); }); - it('should accept aggregation with distinct flag', () => { - const query: QueryAST = { + it('rejects the retired per-aggregation `distinct` flag (#6815)', () => { + // Was `should accept aggregation with distinct flag`. `count` + `distinct` + // is the exact shape `count_distinct` already spells portably, and it is + // the shape the removal is loudest about: the two answered the same number + // in the in-memory fallback and different numbers on every SQL face. + expect(() => QuerySchema.parse({ object: 'order', aggregations: [ { function: 'count', field: 'customer_id', distinct: true, alias: 'unique_customers' }, ], - }; - - expect(() => QuerySchema.parse(query)).not.toThrow(); + })).toThrow(/aggregations\[\]\.distinct.*removed.*count_distinct/s); }); // ============================================================================ @@ -679,12 +683,83 @@ describe('QueryAST.distinct — REMOVED (#4286)', () => { .toThrow(/query\.distinct.*removed/s); }); - it('per-aggregation `distinct` is a DIFFERENT, live member and still parses', () => { + // This case used to read `per-aggregation distinct is a DIFFERENT, live + // member and still parses` — true when #4286 swept the request surface, and + // the reason `AggregationNode.distinct` outlived that sweep: #4286 + // dispositioned `QueryAST.distinct` (this file's other tombstone) and + // `AggregationNode.filter` (marked EXPERIMENTAL), and the per-aggregation + // flag was neither. It IS still a different member — with the same verdict + // one level down, reached separately in #6815. + it('per-aggregation `distinct` is a DIFFERENT member with its OWN prescription (#6815)', () => { expect(() => QuerySchema.parse({ object: 'order', aggregations: [{ function: 'count', field: 'customer_id', distinct: true, alias: 'unique_customers' }], + })).toThrow(/aggregations\[\]\.distinct.*removed/s); + // Not the query-level message: two keys, two prescriptions, two live + // replacements. Mixing them would send an author to `groupBy` for a + // problem `count_distinct` solves. + expect(() => QuerySchema.parse({ + object: 'order', + aggregations: [{ function: 'count', field: 'customer_id', distinct: true, alias: 'unique_customers' }], + })).not.toThrow(/`query\.distinct` was removed/); + }); +}); + +describe('AggregationNode.distinct — REMOVED (#6815, ADR-0049)', () => { + it('refuses either value — `false` was as divergent as `true`', () => { + // `distinct: false` selected the SQL faces' behaviour on the in-memory + // path, so it was never inert: it was one of the two answers. The + // tombstone refuses the KEY, not a value. + for (const value of [true, false]) { + expect(() => AggregationNodeSchema.parse( + { function: 'sum', field: 'amount', alias: 'total', distinct: value }, + )).toThrow(/aggregations\[\]\.distinct.*removed/s); + } + }); + + it('the prescription names the six faces and the live spelling', () => { + const message = AggregationNodeSchema.safeParse( + { function: 'sum', field: 'amount', alias: 'total', distinct: true }, + ).error!.issues[0]!.message; + // The FROM → TO an upgrading author needs, and the measurement that + // justifies it: one honouring face, five ignoring ones. + expect(message).toContain('@objectstack/spec 17'); + expect(message).toContain('count_distinct'); + for (const face of ['driver-sql', 'driver-turso', 'driver-mongodb', 'driver-memory']) { + expect(message, `the prescription must name the ${face} face`).toContain(face); + } + }); + + it('a legal aggregation parses and carries NO `distinct` property (non-strict strip path)', () => { + // `AggregationNodeSchema` is `z.object()`, not `.strict()`. A bare + // deletion would therefore have made zod SILENTLY STRIP a caller's + // `distinct` — replacing a divergent flag with an ignored one, which is + // the #3733 / ADR-0104 shape. The tombstone is what makes the removal + // audible; this pins that the surviving members are untouched by it. + const parsed = AggregationNodeSchema.parse({ function: 'sum', field: 'amount', alias: 'total' }); + expect(parsed).not.toHaveProperty('distinct'); + expect(parsed).toEqual({ function: 'sum', field: 'amount', alias: 'total' }); + }); + + it('`count_distinct` — the live deduplicating spelling — is untouched', () => { + // The removal deletes the flag, never the capability: `count_distinct` + // took ADR-0049's ENFORCE leg in #6409 and every SQL face compiles it. + expect(() => QuerySchema.parse({ + object: 'order', + aggregations: [{ function: 'count_distinct', field: 'customer_id', alias: 'unique_customers' }], })).not.toThrow(); }); + + it('reaches `EngineAggregateOptionsSchema` too — one schema, every aggregation door', () => { + // `EngineAggregateOptionsSchema.aggregations` reuses this schema BY + // REFERENCE (data-engine.zod.ts), so the engine-options door inherits the + // tombstone without restating it. Pinned because the query-level + // `cursor`/`distinct` pair had to be re-declared there by hand, and a + // reader could reasonably expect the same here. + expect(() => EngineAggregateOptionsSchema.parse({ + aggregations: [{ function: 'sum', field: 'amount', alias: 'total', distinct: true }], + })).toThrow(/aggregations\[\]\.distinct.*removed/s); + }); }); describe('QuerySchema - Complex Queries', () => { @@ -822,17 +897,25 @@ describe('QuerySchema - Edge Cases and Null Handling', () => { expect(() => QuerySchema.parse(query)).not.toThrow(); }); - it('should handle optional distinct flag in aggregation', () => { - const query: QueryAST = { + it('refuses a mixed aggregation list the moment ONE entry carries `distinct` (#6815)', () => { + // Was `should handle optional distinct flag in aggregation`. The mixed + // list is the interesting shape: the second entry is untouched by the + // retirement, so the refusal has to come from the first ENTRY rather than + // from the array, and it has to point at that entry's index. + const result = QuerySchema.safeParse({ object: 'order', aggregations: [ { function: 'count', field: 'customer_id', alias: 'unique_customers', distinct: true }, - { function: 'sum', field: 'amount', alias: 'total_amount' }, // distinct undefined + { function: 'sum', field: 'amount', alias: 'total_amount' }, // never carried the key ], groupBy: ['region'], - }; - - expect(() => QuerySchema.parse(query)).not.toThrow(); + }); + expect(result.success).toBe(false); + const issue = result.error!.issues.find((i) => i.path.join('.') === 'aggregations.0.distinct'); + expect(issue, JSON.stringify(result.error!.issues)).toBeDefined(); + expect(issue!.message).toMatch(/aggregations\[\]\.distinct.*removed/s); + // Only the offending entry is reported — the sibling `sum` is legal. + expect(result.error!.issues).toHaveLength(1); }); it('should reject invalid object type', () => { diff --git a/packages/spec/src/data/query.zod.ts b/packages/spec/src/data/query.zod.ts index 24cd0307d9..4e5551de3a 100644 --- a/packages/spec/src/data/query.zod.ts +++ b/packages/spec/src/data/query.zod.ts @@ -197,10 +197,42 @@ export const GroupByNodeSchema = lazySchema(() => z.union([ }), ])); +/** + * The prescription for the per-aggregation `distinct` flag removed in #6815. + * + * Not exported, unlike {@link QUERY_CURSOR_REMOVED} / {@link + * QUERY_DISTINCT_REMOVED}: those two are re-declared on + * `EngineQueryOptionsSchema` and `HttpFindQueryParamsSchema` and need one + * string at two rejection sites, while this key lives on exactly one schema — + * `AggregationNodeSchema`, which `QuerySchema.aggregations` and + * `EngineAggregateOptionsSchema.aggregations` both reuse by reference, so both + * inherit the tombstone without restating it. + * + * No `os migrate meta` step is named: `QueryAST` is a REQUEST surface (the + * client SDK builder's output and the `POST /data/:object/query` body), never + * stored in stack metadata, so there is no source for a conversion to rewrite + * — the #4286 disposition, verbatim. The ADR-0087 registration is the + * protocol-17 semantic migration `aggregation-node-distinct-retired`. + */ +const AGGREGATION_DISTINCT_REMOVED = + '`query.aggregations[].distinct` was removed in @objectstack/spec 17 (#6815, ADR-0049) — ' + + 'exactly ONE of the six faces that read an aggregation honoured it. The objectql in-memory ' + + 'fallback deduplicated the values before applying the function, while `driver-sql`, ' + + '`driver-turso`, `driver-mongodb`, `driver-memory` and the service-analytics SQL builder ' + + "all ignored it — so `{ function: 'sum', field: 'amount', distinct: true }` answered a " + + 'DEDUPLICATED sum when the engine fell back in memory and an ordinary sum on every SQL ' + + 'datasource: one query, two numbers, chosen by which backend happened to serve it. Both ' + + 'answers are plausible, so nothing surfaced the divergence. Delete the key. For a ' + + 'deduplicated COUNT the live spelling is the `count_distinct` aggregation function, which ' + + 'every SQL face compiles to `COUNT(DISTINCT field)` (#6409) and the in-memory fallback ' + + 'computes identically. `SUM(DISTINCT …)` / `AVG(DISTINCT …)` get no replacement: no ' + + 'backend ever computed them here, and a per-row measure that needs deduplicating is a ' + + 'modelling problem to fix in the data, not a flag on the read.'; + /** * Aggregation Node * Represents an aggregated field with function. - * + * * Aggregations summarize data across groups of rows (GROUP BY). * Used with `groupBy` to create analytical queries. * @@ -231,7 +263,12 @@ export const AggregationNodeSchema = lazySchema(() => z.object({ function: AggregationFunction.describe('Aggregation function'), field: z.string().optional().describe('Field to aggregate (optional for COUNT(*))'), alias: z.string().describe('Result column alias'), - distinct: z.boolean().optional().describe('Apply DISTINCT before aggregation'), + /** + * Per-aggregation DISTINCT — REMOVED (#6815, ADR-0049). One face honoured it + * and five ignored it; `count_distinct` is the one deduplicating spelling + * every face computes. See {@link AGGREGATION_DISTINCT_REMOVED}. + */ + distinct: retiredKey(AGGREGATION_DISTINCT_REMOVED), filter: FilterConditionSchema.optional().describe('[EXPERIMENTAL — not enforced] Per-aggregation filter (SQL FILTER (WHERE …)). Neither the SQL builders nor the in-memory fallback applies it (#4286); filter the whole query with `where` instead.'), })); diff --git a/packages/spec/src/migrations/registry.ts b/packages/spec/src/migrations/registry.ts index 9bbc73981e..9c8c6c873b 100644 --- a/packages/spec/src/migrations/registry.ts +++ b/packages/spec/src/migrations/registry.ts @@ -1147,6 +1147,37 @@ const step17: MigrationStep = { + 'surface with no stored source — one semantic TODO below. The mongodb and in-memory ' + 'backends that implemented these two are inside the #5499 freeze and are untouched; their ' + 'code is simply no longer reachable through a spec-valid request.\n\n' + + 'The same aggregation node loses one more member, and it is the sharper class of the two: ' + + '`aggregations[].distinct` is removed (#6815, ADR-0049, maintainer ruling 2026-08-09). ' + + 'The functions above were declared and UNLOWERED — a caller on a SQL datasource got a ' + + 'refusal. This flag was declared and lowered by exactly ONE of the six faces that read an ' + + 'aggregation: the engine\'s in-memory fallback deduplicated the values before applying the ' + + 'function, while `SqlDriver.aggregate`, the Turso `RemoteTransport.aggregate`, ' + + '`driver-mongodb`\'s `buildAggregationStage`, `driver-memory`\'s `computeAggregate` and ' + + 'service-analytics\' `AGGREGATE_SQL` all ignored it. So the same query answered a ' + + 'deduplicated `sum` on the fallback path and an ordinary `sum` on every SQL datasource, ' + + 'with the engine choosing between the two per query — by driver, by a non-UTC date bucket, ' + + 'by whether the driver aggregates natively at all. That is the divergence class #6203 and ' + + '#5907 each closed on this axis, still open on this key, and it is worse to sit on because ' + + 'the wrong answer is a PLAUSIBLE NUMBER rather than a refusal: no error, no log, nothing ' + + 'for a dashboard author to notice. It survived the #4286 sweep of this very schema because ' + + 'that sweep asked which members no executor reads, and this one had a reader — the wrong ' + + 'question for a key whose defect is WHICH executor reads it. Remove rather than enforce, ' + + 'per the ruling: `count_distinct` (which just took the enforce leg above, and whose SQL ' + + 'lowering #6409 landed) already covers the only deduplicating spelling with measured ' + + 'demand, while `SUM(DISTINCT …)` / `AVG(DISTINCT …)` are near-universally a modelling ' + + 'mistake and would have to be lowered across five faces, two of them frozen under #5499, ' + + 'to buy it. The blast radius inside the fallback is narrower than the key suggests and was ' + + 'measured rather than assumed: only `sum` and `avg` ever changed answer — `count` returned ' + + 'from its own branch before reaching the dedupe, `count_distinct` fed a Set, and dedupe ' + + 'does not move `min`/`max`. `AggregationNodeSchema` is non-strict, so the key is ' + + '`retiredKey()`-tombstoned rather than bare-deleted: a plain deletion would have made zod ' + + 'silently STRIP what callers still send, trading a divergent flag for an ignored one ' + + '(#3733, ADR-0104). One tombstone covers every aggregation door, because ' + + '`QuerySchema.aggregations` and `EngineAggregateOptionsSchema.aggregations` reuse that one ' + + 'schema by reference. No conversion: a request surface with no stored source — one ' + + 'semantic TODO below, the disposition every other `data.query.*` retirement in this major ' + + 'already takes.\n\n' + 'One entry in this step is not a removal at all but a SECURE-DEFAULT FLIP, the shape ' + "protocol 12 last used for `api.requireAuth`: an omitted `ActionDescriptor.resumeAuthority` " + "resolves to `'service'` instead of `'any'`, so a pausing node type that never states who " @@ -1524,6 +1555,54 @@ const step17: MigrationStep = { + 'removal prescription naming it, and authoring it is a `tsc` error at the call site; ' + '`count_distinct` continues to parse and is unaffected.', }, + { + id: 'aggregation-node-distinct-retired', + surface: 'data.query.aggregations[].distinct', + replacement: + 'the `count_distinct` aggregation FUNCTION for a deduplicated count — the one ' + + 'deduplicating spelling every face computes, lowered to `COUNT(DISTINCT field)` on ' + + 'both SQL faces since #6409. `SUM(DISTINCT …)` / `AVG(DISTINCT …)` get no ' + + 'replacement: no backend ever computed them here, and a per-row measure that needs ' + + 'deduplicating before summing is a modelling problem to fix in the data', + reason: + 'A DIVERGENCE, not an inert declaration — which is why it outlived the #4286 sweep ' + + 'that dispositioned every other `data.query.*` member. That sweep asked which keys ' + + 'no executor reads; this one HAD an executor, exactly one out of six. The engine\'s ' + + 'in-memory fallback (`objectql/src/in-memory-aggregation.ts`) deduplicated the ' + + 'values before applying the function, while `SqlDriver.aggregate`, the Turso ' + + '`RemoteTransport.aggregate`, `driver-mongodb`\'s `buildAggregationStage`, ' + + '`driver-memory`\'s `computeAggregate` and service-analytics\' `AGGREGATE_SQL` all ' + + 'ignored the key. So `{ function: \'sum\', field: \'amount\', distinct: true }` ' + + 'answered a deduplicated sum when the engine fell back in memory and an ordinary sum ' + + 'on every SQL datasource: one query, two numbers, chosen by which backend happened ' + + 'to serve it — and unlike the #6203 / #5907 divergences closed on the same axis, the ' + + 'wrong answer here is a plausible NUMBER rather than a refusal, so nothing surfaced ' + + 'it to the author. Measured blast radius inside the fallback: `sum` and `avg` only — ' + + '`count` returned from its own branch before reaching the dedupe, `count_distinct` ' + + 'fed the values into a Set (dedupe-then-Set is Set), and dedupe does not move ' + + '`min`/`max`. ENFORCE was weighed and rejected (maintainer ruling 2026-08-09): ' + + '`count_distinct` already covers the only spelling anyone has measured demand for, ' + + 'and lowering `SUM(DISTINCT …)` across five faces — two of them frozen under #5499 — ' + + 'buys a shape that is near-universally a modelling mistake. A REQUEST surface — ' + + '`QueryAST` is the client SDK builder\'s output and the `POST /data/:object/query` ' + + 'body, never stored in stack metadata — so there is no source for the chain to ' + + 'rewrite and callers move their own queries: the #4286 disposition for ' + + '`joins`/`cursor`/`distinct`/`windowFunctions`, applied verbatim one level down. ' + + 'ADR-0049, #6815.', + acceptanceCriteria: + 'No caller sends `distinct` inside an `aggregations[]` entry, on the wire or through ' + + 'the SDK; a deduplicated count is written as `{ function: \'count_distinct\', field }` ' + + 'and reads the same number on every backend. A query still carrying the key fails to ' + + 'parse with the removal prescription — including through ' + + '`EngineAggregateOptionsSchema`, which reuses `AggregationNodeSchema` by reference — ' + + 'and `POST /api/v1/data/:object/query` answers `400 VALIDATION_FAILED` with a ' + + '`fields[]` entry at `aggregations..distinct` instead of serving a number. ' + + 'Authoring it is a `tsc` error at the call site. ⚠️ The observable NUMBERS change on ' + + 'exactly one path and that is the point of the change: a `sum`/`avg` that used to be ' + + 'deduplicated by the in-memory fallback now answers what every SQL face has always ' + + 'answered for the same query. Verify against the SQL answer, not against the ' + + 'pre-upgrade fallback answer — the two disagreed, which is why the key is gone.', + }, { id: 'workflow-service-slot-retired', surface: @@ -3140,6 +3219,27 @@ export const RETIRED_KEYS_BY_MAJOR: Readonly> // callers rather than a stack conversion"). 'api/ListNotificationsRequest:cursor', 'api/ListNotificationsResponse:cursor', + // #6815 — the per-aggregation DISTINCT flag, retired under ADR-0049 by + // maintainer ruling 2026-08-09. ONE key, and one entry, because + // `AggregationNodeSchema` is reused BY REFERENCE rather than `.extend()`ed: + // `QuerySchema.aggregations` and `EngineAggregateOptionsSchema. + // aggregations` are both `z.array(AggregationNodeSchema)`, so the walked + // shape has a single `data/AggregationNode` def and the baseline marks one + // line `[RETIRED]`. Contrast the `shared/FieldMapping:transform` trio at + // the top of this list, where two `.extend()`s copied the property into + // three walked shapes and each needed its own registration. + // + // Registered here but NOT in `src/conversions/registry.ts`, for the same + // reason as the notification pair above: `QueryAST` is a REQUEST surface — + // the client SDK builder's output and the `POST /data/:object/query` body + // — never stored in stack metadata, so there is no authored source or + // `sys_metadata` row for a D2 conversion to rewrite. The prescription + // reaches consumers as the D3 semantic entry + // `aggregation-node-distinct-retired` plus this tombstone, which is the + // disposition every other `data.query.*` retirement in this major already + // takes (`query-joins-retired` / `query-cursor-retired` / + // `query-distinct-retired` / `query-window-functions-retired`, #4286). + 'data/AggregationNode:distinct', ], };