refactor(spec,objectql)!: retire AggregationNode.distinct — one face honoured it, five ignored it, and the same query answered two plausible numbers (#6815, ADR-0049)

.changeset/aggregation-node-distinct-retired.md

@@ -0,0 +1,71 @@
+---
+'@objectstack/spec': major
+'@objectstack/objectql': major
+---
+
+refactor(spec,objectql)!: retire `AggregationNode.distinct` — one face honoured it, five ignored it, and the same query answered two plausible numbers (#6815, ADR-0049)
+
+<!-- adr-0087: registered aggregation-node-distinct-retired -->
+
+**FROM → TO:** `{ function: 'count', field: 'x', distinct: true, alias: 'a' }` →
+`{ function: 'count_distinct', field: 'x', alias: 'a' }` — the deduplicating spelling
+every backend computes, lowered to `COUNT(DISTINCT x)` on both SQL faces since #6409.
+`{ function: 'sum' | 'avg' | 'min' | 'max', …, distinct: true }` → delete the key; there is
+no replacement, because no SQL backend ever computed `SUM(DISTINCT …)` here and the
+in-memory fallback was the only thing that did. `distinct: false` → delete the key; it
+selected the behaviour that is now the only behaviour.
+
+`AggregationNode.distinct` was read by exactly ONE of the six faces that consume an
+`aggregations[]` entry. `objectql`'s in-memory fallback (`in-memory-aggregation.ts`)
+deduplicated the values before applying the function; `SqlDriver.aggregate`, the Turso
+`RemoteTransport.aggregate`, `driver-mongodb`'s `buildAggregationStage`, `driver-memory`'s
+`computeAggregate` and `service-analytics`' `AGGREGATE_SQL` all ignored it. So
+`{ function: 'sum', field: 'amount', distinct: true }` returned a deduplicated sum when the
+engine fell back in memory and an ordinary sum on every SQL datasource — one query, two
+numbers, chosen by which backend answered. The engine picks that path per query (a driver
+without native aggregation, a non-UTC date bucket, a partial SQL driver), so the number
+could move under a dashboard with nothing changing in the query.
+
+That is the divergence class #6203 and #5907 each closed on the aggregate axis, still open
+on this key, and it is worse to leave: both answers are plausible NUMBERS rather than a
+refusal, so nothing surfaced it. It survived the #4286 sweep of this same schema because
+that sweep asked which members no executor reads — the wrong question for a key whose
+defect is *which* executor reads it.
+
+REMOVE rather than ENFORCE, per the maintainer ruling of 2026-08-09: `count_distinct`
+already covers the only deduplicating spelling with measured demand and took ADR-0049's
+enforce leg in #6409, while `SUM(DISTINCT …)` / `AVG(DISTINCT …)` are near-universally a
+modelling mistake and would have to be lowered across five faces — two of them frozen under
+#5499 — to buy it.
+
+The retirement kit:
+
+- **Tombstone, not deletion** (`retiredKey()`): `AggregationNodeSchema` is not `.strict()`,
+  so a plain delete would let existing queries parse clean and lose the key in silence
+  (#3733, ADR-0104) — trading a divergent flag for an ignored one. Authoring it is now a
+  `tsc` error at the call site and a parse error carrying the prescription. One tombstone
+  covers every aggregation door: `QuerySchema.aggregations` and
+  `EngineAggregateOptionsSchema.aggregations` both reuse that one schema by reference.
+- **ADR-0087 D3 `SemanticMigration`** (`aggregation-node-distinct-retired`) plus the exact
+  `RETIRED_KEYS_BY_MAJOR[17]` entry `data/AggregationNode:distinct`. No D2 conversion,
+  deliberately: `QueryAST` is a request surface — the client SDK builder's output and the
+  `POST /data/:object/query` body — never stored in stack metadata, so there is no source
+  for `os migrate meta` to rewrite. That is the disposition every other `data.query.*`
+  retirement in this major already takes (#4286).
+- `objectql`'s in-memory fallback loses its `collectValues` dedupe limb — the whole runtime
+  cost of the removal. **The observable numbers change on that one path, and that is the
+  point:** a `sum`/`avg` that used to be deduplicated there now answers what every SQL face
+  has always answered for the same query. Verify against the SQL answer, not against the
+  pre-upgrade fallback answer — the two disagreed.
+- Measured blast radius inside the fallback, narrower than the key suggests: only `sum` and
+  `avg` ever changed answer. `count` returned from its own branch before reaching the
+  dedupe, `count_distinct` fed the values into a `Set` (dedupe-then-`Set` is `Set`), and
+  dedupe does not move `min`/`max`.
+- `POST /api/v1/data/:object/query` answers `400 VALIDATION_FAILED` with a `fields[]` entry
+  at `aggregations.<i>.distinct` instead of serving a number — the #3899 entry validation
+  descending into the array, pinned in the REST request-schema conformance gate.
+- Liveness ledger (`query.json` `aggregations.children.distinct` → `dead`, README counts),
+  generated baselines (`authorable-surface/data.json` gains `[RETIRED]`),
+  `spec-changes.json`, the upgrade guide and the reference docs regenerated.
+
+`count_distinct` is untouched and remains the live deduplicating spelling.

content/docs/kernel/contracts/data-engine.mdx

@@ -357,11 +357,16 @@ interface AggregationNode {
   function: 'count' | 'sum' | 'avg' | 'min' | 'max' | 'count_distinct';
   field?: string;           // Field to aggregate (optional for COUNT(*))
   alias: string;            // Result column alias
-  distinct?: boolean;       // Apply DISTINCT before aggregation
   filter?: FilterCondition; // Per-aggregation FILTER WHERE
 }
 ```
 
+`distinct?: boolean` was **removed** from `AggregationNode` in protocol 17 (#6815,
+ADR-0049). Only the engine's in-memory fallback ever honoured it — every SQL face
+ignored it — so the same query answered a deduplicated `sum` or an ordinary one
+depending on which backend served it. For a deduplicated count use the
+`count_distinct` function, which every face computes.
+
 ---
 
 ## Optional Capabilities

content/docs/protocol/objectql/query-syntax.mdx

@@ -104,6 +104,19 @@ a `where` predicate on the sort key — §7), and `distinct` (unique values via
 effect was suppressing the REST list count, which is truthful again). **Enforced**:
 `having` (§5). The experimental flags above are tracked in the liveness ledger
 (`packages/spec/liveness/query.json`).
+
+One member of `AggregationNode` was settled separately, in #6815: the
+per-aggregation **`distinct`** flag is **removed** on the same terms. It escaped the
+#4286 sweep because that sweep asked which members no executor reads and this one had
+a reader — one out of six. The engine's in-memory fallback deduplicated before
+applying the function, while `driver-sql`, `driver-turso`, `driver-mongodb`,
+`driver-memory` and the analytics SQL builder all ignored it, so
+`{ function: 'sum', field: 'amount', distinct: true }` answered a deduplicated sum on
+the fallback path and an ordinary sum on every SQL datasource — one query, two
+plausible numbers, chosen by which backend served it. The live deduplicating spelling
+is the **`count_distinct` function** (`COUNT(DISTINCT field)` on both SQL faces since
+#6409); `SUM(DISTINCT …)` / `AVG(DISTINCT …)` have no replacement, because no backend
+ever computed them here.
 </Callout>
 
 ### Key Types
@@ -120,9 +133,9 @@ interface AggregationNode {
   function: 'count' | 'sum' | 'avg' | 'min' | 'max' | 'count_distinct';
   field?: string;        // optional for COUNT(*)
   alias: string;         // result column alias
-  distinct?: boolean;    // DISTINCT before aggregation — in-memory path only
   filter?: FilterCondition; // [EXPERIMENTAL — not enforced] FILTER WHERE clause — never applied
 }
+// `distinct?: boolean` was REMOVED in protocol 17 (#6815) — see the callout above.
 
 // FieldNode — one entry of the select list. A field name, optionally dotted to
 // reach through a relationship ('owner.name'). Related *records* come from

content/docs/references/api/contract.mdx

@@ -412,7 +412,7 @@ const result = ApiErrorSchema.parse(data);
 | **top** | `number` | optional | Alias for limit (OData compatibility) |
 | **cursor** | `never` | optional | [REMOVED] `query.cursor` was removed in @objectstack/spec 17 (#4286, ADR-0049) — no driver ever implemented keyset pagination, so the cursor was accepted and ignored and every page came back identical (a caller looping "until hasMore is false" never terminates). Delete the key; `QueryBuilder.cursor()` was removed with it. Express the keyset as an ordinary `where` predicate on your sort key — `where: { created_at: { $gt: last.created_at } }` with the matching `orderBy` — which every driver executes with canonicalised comparands. A first-class cursor, if ever built, will be a response-minted opaque token, not this caller-built record. |
 | **joins** | `never` | optional | [REMOVED] `query.joins` was removed in @objectstack/spec 17 (#4286, ADR-0049) — no engine or driver ever read it: a query carrying `joins` behaved exactly as if the key were absent, while its name squatted on the reserved REST parameter set. Delete the key. Related records are read through `expand` — `expand: { owner: { object: 'user', fields: ['name'] } }` — which the engine resolves via batch $in queries, and a single related column is a dotted `fields` path (`fields: ['owner.name']`). |
-| **aggregations** | `{ function: Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct'>; field?: string; alias: string; distinct?: boolean; … }[]` | optional | Aggregation functions |
+| **aggregations** | `{ function: Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct'>; field?: string; alias: string; filter?: any }[]` | optional | Aggregation functions |
 | **groupBy** | `(string \| { field: string; dateGranularity?: Enum<'day' \| 'week' \| 'month' \| 'quarter' \| 'year'>; alias?: string })[]` | optional | GROUP BY targets (strings or `{field, dateGranularity?}` objects for date bucketing) |
 | **having** | `any` | optional | HAVING — filter over the AGGREGATED rows (aggregation aliases + groupBy projections); applied engine-side after aggregation |
 | **windowFunctions** | `never` | optional | [REMOVED] `query.windowFunctions` was removed in @objectstack/spec 17 (#4286, ADR-0049) — `find()` never applied it: no engine or driver read the key on the query path, so every OVER clause it declared was silently dropped. Delete the key. Window functions are a SQL-driver capability behind `SqlDriver.findWithWindowFunctions(object, query)` (embedder-level; not on the `IDataDriver` contract or the REST surface); request-level analytics are `aggregations` + `groupBy`. |

content/docs/references/data/data-engine.mdx

@@ -467,7 +467,7 @@ QueryAST-aligned options for DataEngine.aggregate operations
 | **context** | `{ userId?: string; actor?: string; attributedUserId?: string; email?: string; … }` | optional |  |
 | **where** | `Record<string, any> \| any` | optional |  |
 | **groupBy** | `string[]` | optional |  |
-| **aggregations** | `{ function: Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct'>; field?: string; alias: string; distinct?: boolean; … }[]` | optional |  |
+| **aggregations** | `{ function: Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct'>; field?: string; alias: string; filter?: any }[]` | optional |  |
 | **having** | `any` | optional | HAVING — filter over the aggregated rows (aggregation aliases + groupBy projections); applied engine-side after aggregation |
 | **timezone** | `string` | optional |  |
 

content/docs/references/data/query.mdx

@@ -48,7 +48,7 @@ const result = AggregationFunction.parse(data);
 | **function** | `Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct'>` | ✅ | Aggregation function |
 | **field** | `string` | optional | Field to aggregate (optional for COUNT(*)) |
 | **alias** | `string` | ✅ | Result column alias |
-| **distinct** | `boolean` | optional | Apply DISTINCT before aggregation |
+| **distinct** | `never` | optional | [REMOVED] `query.aggregations[].distinct` was removed in @objectstack/spec 17 (#6815, ADR-0049) — exactly ONE of the six faces that read an aggregation honoured it. The objectql in-memory fallback deduplicated the values before applying the function, while `driver-sql`, `driver-turso`, `driver-mongodb`, `driver-memory` and the service-analytics SQL builder all ignored it — so `{ function: 'sum', field: 'amount', distinct: true }` answered a DEDUPLICATED sum when the engine fell back in memory and an ordinary sum on every SQL datasource: one query, two numbers, chosen by which backend happened to serve it. Both answers are plausible, so nothing surfaced the divergence. Delete the key. For a deduplicated COUNT the live spelling is the `count_distinct` aggregation function, which every SQL face compiles to `COUNT(DISTINCT field)` (#6409) and the in-memory fallback computes identically. `SUM(DISTINCT …)` / `AVG(DISTINCT …)` get no replacement: no backend ever computed them here, and a per-row measure that needs deduplicating is a modelling problem to fix in the data, not a flag on the read. |
 | **filter** | `any` | optional | [EXPERIMENTAL — not enforced] Per-aggregation filter (SQL FILTER (WHERE …)). Neither the SQL builders nor the in-memory fallback applies it (#4286); filter the whole query with `where` instead. |
 
 
@@ -132,7 +132,7 @@ Type: `string`
 | **top** | `number` | optional | Alias for limit (OData compatibility) |
 | **cursor** | `never` | optional | [REMOVED] `query.cursor` was removed in @objectstack/spec 17 (#4286, ADR-0049) — no driver ever implemented keyset pagination, so the cursor was accepted and ignored and every page came back identical (a caller looping "until hasMore is false" never terminates). Delete the key; `QueryBuilder.cursor()` was removed with it. Express the keyset as an ordinary `where` predicate on your sort key — `where: { created_at: { $gt: last.created_at } }` with the matching `orderBy` — which every driver executes with canonicalised comparands. A first-class cursor, if ever built, will be a response-minted opaque token, not this caller-built record. |
 | **joins** | `never` | optional | [REMOVED] `query.joins` was removed in @objectstack/spec 17 (#4286, ADR-0049) — no engine or driver ever read it: a query carrying `joins` behaved exactly as if the key were absent, while its name squatted on the reserved REST parameter set. Delete the key. Related records are read through `expand` — `expand: { owner: { object: 'user', fields: ['name'] } }` — which the engine resolves via batch $in queries, and a single related column is a dotted `fields` path (`fields: ['owner.name']`). |
-| **aggregations** | `{ function: Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct'>; field?: string; alias: string; distinct?: boolean; … }[]` | optional | Aggregation functions |
+| **aggregations** | `{ function: Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct'>; field?: string; alias: string; filter?: any }[]` | optional | Aggregation functions |
 | **groupBy** | `(string \| { field: string; dateGranularity?: Enum<'day' \| 'week' \| 'month' \| 'quarter' \| 'year'>; alias?: string })[]` | optional | GROUP BY targets (strings or `{field, dateGranularity?}` objects for date bucketing) |
 | **having** | `any` | optional | HAVING — filter over the AGGREGATED rows (aggregation aliases + groupBy projections); applied engine-side after aggregation |
 | **windowFunctions** | `never` | optional | [REMOVED] `query.windowFunctions` was removed in @objectstack/spec 17 (#4286, ADR-0049) — `find()` never applied it: no engine or driver read the key on the query path, so every OVER clause it declared was silently dropped. Delete the key. Window functions are a SQL-driver capability behind `SqlDriver.findWithWindowFunctions(object, query)` (embedder-level; not on the `IDataDriver` contract or the REST surface); request-level analytics are `aggregations` + `groupBy`. |