docs(spec): fields stops prescribing a dotted path no driver resolves (#7601)

.changeset/dotted-fields-prose-corrected.md

@@ -0,0 +1,40 @@
+---
+"@objectstack/spec": patch
+---
+
+**docs(spec): `fields` stops prescribing a dotted path no driver resolves (#7601)**
+
+Six in-repo surfaces offered `fields: ['owner.name']` as the supported way to read
+one related column. No driver ever implemented it — measured on a real `SqlDriver`,
+a dotted projection is byte-identical to no projection at all, because Knex renders
+`"account"."name"` against a table that was never joined and the #3821 recovery
+ladder retries `select('*')`. Since #7532 those surfaces are additionally
+contradicted by a `400 INVALID_FIELD` at the ingress gate
+(`assertProjectionFieldsExist`). The migration tooling was the sharpest case: both
+protocol-17 upgrade prescriptions routed authors off `query.joins` and off the
+retired `{ field, fields, alias }` form directly into the refused spelling.
+
+This aligns the declaration to the enforcement. The normative `fields` `.describe()`
+now names `expand` as the sanctioned mechanism for related data — its nested
+`QueryAST` both filters (`where`) and selects (`fields`) the related record's
+columns — and carries the sharp edge that was pinned but never documented: **the
+projection must retain the foreign-key column.** `fields: ['title']` with
+`expand: 'project_id'` resolves nothing, because the relation is carried by that
+key; adding `'project_id'` makes it work. Where the value is wanted on the queried
+object itself, the honest remedy is to denormalise it onto that object (a stored
+field, written when the source changes) — the same remedy the sort axis prescribes
+(#6924). Both retirement prescriptions and the two tombstone rejection messages now
+say the same thing, and the JSON Schema artifacts and reference docs regenerate from
+the source.
+
+**No schema change.** `FieldNodeSchema` stays `z.string()`: the refusal of dotted
+projections is a *semantic* verdict, made at the ingress gate where the field map is
+available to judge against — not a *shape* check. Narrowing the type would duplicate
+that gate and refuse the registry-less internal callers the ingress deliberately
+tolerates. Every input that parsed before this change parses byte-identically after
+it, and the type/runtime pins that assert so are kept and renamed
+(`fieldNodeDottedNotNarrowed`) so they read as the non-narrowing guard they are
+rather than as an endorsement of a feature that does not exist.
+
+Prose, prescriptions and generated artifacts only — no wire, stored-data or
+validation behaviour changes.

.changeset/query-field-node-object-form-removed.md

@@ -40,7 +40,7 @@ Directive #12: one capability, one contract.
 | :--- | :--- |
 | `fields: [{ field: 'owner', fields: ['name'] }]` | `expand: { owner: { object: 'user', fields: ['name'] } }` |
 | `fields: [{ field: 'owner' }]` | `fields: ['owner']` |
-| `fields: [{ field: 'owner', fields: ['name'] }]`, one column only | `fields: ['owner.name']` (dotted path) |
+| `fields: [{ field: 'owner', fields: ['name'] }]`, one column only | the same `expand`, keeping the FK in your own projection (`fields: ['title', 'owner_id']`) — **not** a dotted `fields` path, which no driver resolves and the ingress refuses (#7532) |
 | `fields: [{ field: 'total', alias: 't' }]` | `aggregations` / `windowFunctions` — they carry the live `alias` |
 
 The one-line fix: **a `fields[]` entry is a string.** Move nested selection to

content/docs/kernel/contracts/data-engine.mdx

@@ -105,7 +105,8 @@ Defined by `EngineQueryOptionsSchema` in `@objectstack/spec`:
 ```typescript
 interface EngineQueryOptions {
   where?: FilterCondition;                   // WHERE clause — MongoDB-style $op
-  fields?: FieldNode[];                      // SELECT — field names ('name', 'owner.name')
+  fields?: FieldNode[];                      // SELECT — the object's OWN column names; related
+                                             // data comes from `expand`, not a dotted path (#7532)
   orderBy?: SortNode[];                      // ORDER BY
   limit?: number;                            // LIMIT
   offset?: number;                           // OFFSET

content/docs/protocol/objectql/index.mdx

@@ -135,7 +135,12 @@ const query: QueryAST = {
     industry: 'tech',
     annual_revenue: { $gt: 1000000 }
   },
-  fields: ['company_name', 'industry', 'owner.name'],
+  // `fields` names the object's OWN columns — a dotted path ('owner.name') is
+  // refused by the ingress (400 INVALID_FIELD, #7532). Related data comes from
+  // `expand`, which resolves THROUGH the foreign key, so `owner_id` has to stay
+  // in the projection.
+  fields: ['company_name', 'industry', 'owner_id'],
+  expand: { owner_id: { object: 'user', fields: ['name'] } },
   orderBy: [{ field: 'created_at', order: 'desc' }],
   limit: 10
 };
@@ -144,35 +149,27 @@ const query: QueryAST = {
 **Runtime compilation to different databases:**
 
 ```sql
--- PostgreSQL (with JOIN)
-SELECT c.company_name, c.industry, u.name AS "owner.name"
-FROM customer c
-LEFT JOIN user u ON c.owner_id = u.id
-WHERE c.industry = 'tech' AND c.annual_revenue > 1000000
-ORDER BY c.created_at DESC
+-- PostgreSQL
+SELECT company_name, industry, owner_id
+FROM customer
+WHERE industry = 'tech' AND annual_revenue > 1000000
+ORDER BY created_at DESC
 LIMIT 10;
+
+-- …then `expand` is a second, batched read on the related object — driver-agnostic,
+-- not a JOIN the driver renders:
+SELECT id, name FROM "user" WHERE id IN (…the owner_ids of the page above);
 ```
 
 ```javascript
 // MongoDB
-db.customer.aggregate([
-  {
-    $match: {
-      industry: 'tech',
-      annual_revenue: { $gt: 1000000 }
-    }
-  },
-  {
-    $lookup: {
-      from: 'user',
-      localField: 'owner_id',
-      foreignField: '_id',
-      as: 'owner'
-    }
-  },
-  { $sort: { created_at: -1 } },
-  { $limit: 10 }
-]);
+db.customer.find(
+  { industry: 'tech', annual_revenue: { $gt: 1000000 } },
+  { company_name: 1, industry: 1, owner_id: 1 }
+).sort({ created_at: -1 }).limit(10);
+
+// …then the same batched expand read, spelled $in:
+db.user.find({ _id: { $in: [/* the owner_ids of the page above */] } }, { name: 1 });
 ```
 
 ### 4. Validation: Business Rules as Data

content/docs/protocol/objectql/query-syntax.mdx

@@ -137,9 +137,12 @@ interface AggregationNode {
 }
 // `distinct?: boolean` was REMOVED in protocol 17 (#6815) — see the callout above.
 
-// FieldNode — one entry of the select list. A field name, optionally dotted to
-// reach through a relationship ('owner.name'). Related *records* come from
-// `expand`, not from inside this list.
+// FieldNode — one entry of the select list. One of the queried object's OWN
+// column names. The type is `string`, so a dotted path ('owner.name') still
+// PARSES, but it resolves nothing: no driver ever implemented dotted
+// projection, and the ingress refuses it (400 INVALID_FIELD, #7532). Related
+// data — whole records and single related columns alike — comes from `expand`,
+// not from inside this list.
 //
 // The `{ field, fields, alias }` nested-select member this union used to carry
 // was REMOVED in protocol 17 (#4196): nothing produced it and nothing read
@@ -1003,9 +1006,30 @@ as a single-table query. The key is tombstoned — authoring it is a `tsc` error
 query that still carries it (even as an empty array) fails to parse with the upgrade
 prescription. The `JoinNode` / `JoinType` / `JoinStrategy` exports left with it.
 
-Use `expand` (§4) for relationship loading — the live spelling for related records —
-a dotted `fields` path (`'owner.name'`) for a single related column, or two queries
-joined in application code.
+Use `expand` (§4) for relationship loading — the live spelling for related records,
+and for single related columns too, since its nested `QueryAST` both filters (`where`)
+and selects (`fields`) the related record's columns. Otherwise, two queries joined in
+application code.
+
+**A dotted `fields` path is not the alternative.** `'owner.name'` still *parses* —
+`FieldNode` is `string`, a shape check — but no driver ever resolved one, and the
+ingress refuses it with `400 INVALID_FIELD` (#7532). Where the value is wanted on the
+queried object itself, denormalise it onto that object (a stored field, written when
+the source changes) — the same remedy the sort axis prescribes (#6924).
+
+<Callout type="warn">
+**`expand` needs the foreign key in the projection.** The relation is carried by the
+FK column, so a narrowed projection that projects it away leaves expansion nothing to
+resolve:
+
+```typescript
+{ fields: ['title'],                expand: { project_id: { object: 'project' } } }
+// -> nothing to resolve; no related record comes back
+
+{ fields: ['title', 'project_id'],  expand: { project_id: { object: 'project' } } }
+// -> works
+```
+</Callout>
 
 ### Window Functions — removed from the request surface (#4286)
 

content/docs/references/api/contract.mdx

@@ -404,7 +404,7 @@ const result = ApiErrorSchema.parse(data);
 | Property | Type | Required | Description |
 | :--- | :--- | :--- | :--- |
 | **object** | `string` | ✅ | Object name (e.g. account) |
-| **fields** | `string[]` | optional | Fields to retrieve — field names, optionally dotted to reach through a relationship (`owner.name`). Related *records* are selected with `expand`, not from inside this list. |
+| **fields** | `string[]` | optional | Fields to retrieve — names of the queried object's OWN columns. A dotted path (`owner.name`) is not a projection: no driver resolves one, and the ingress refuses it with `400 INVALID_FIELD` (#7532). Related data is read with `expand`, whose nested QueryAST both filters (`where`) and selects (`fields`) the related record's columns. The projection must RETAIN the foreign-key column: `fields: ['title']` with `expand: 'project_id'` resolves nothing, because the relation is carried by that key — add `'project_id'` and it works. Where the value is wanted on the queried object itself, denormalise it onto that object (a stored field, written when the source changes), the same remedy the sort axis prescribes (#6924). |
 | **where** | `any` | optional | Filtering criteria (WHERE) |
 | **search** | `string \| { query: string; fields?: string[]; fuzzy: boolean; operator: Enum<'and' \| 'or'>; … }` | optional | Full-text search — the query text (canonical, ADR-0061 D1), or a structured FullTextSearch configuration |
 | **searchFields** | `string[]` | optional | Narrow the search to these fields (server-intersected with the allowed searchable set — can only narrow, never widen; ADR-0061 D1) |
@@ -413,7 +413,7 @@ const result = ApiErrorSchema.parse(data);
 | **offset** | `number` | optional | Records to skip (OFFSET) |
 | **top** | `number` | optional | Alias for limit (OData compatibility) |
 | **cursor** | `never` | optional | [REMOVED] `query.cursor` was removed in @objectstack/spec 17 (#4286, ADR-0049) — no driver ever implemented keyset pagination, so the cursor was accepted and ignored and every page came back identical (a caller looping "until hasMore is false" never terminates). Delete the key; `QueryBuilder.cursor()` was removed with it. Express the keyset as an ordinary `where` predicate on your sort key — `where: { created_at: { $gt: last.created_at } }` with the matching `orderBy` — which every driver executes with canonicalised comparands. A first-class cursor, if ever built, will be a response-minted opaque token, not this caller-built record. |
-| **joins** | `never` | optional | [REMOVED] `query.joins` was removed in @objectstack/spec 17 (#4286, ADR-0049) — no engine or driver ever read it: a query carrying `joins` behaved exactly as if the key were absent, while its name squatted on the reserved REST parameter set. Delete the key. Related records are read through `expand` — `expand: { owner: { object: 'user', fields: ['name'] } }` — which the engine resolves via batch $in queries, and a single related column is a dotted `fields` path (`fields: ['owner.name']`). |
+| **joins** | `never` | optional | [REMOVED] `query.joins` was removed in @objectstack/spec 17 (#4286, ADR-0049) — no engine or driver ever read it: a query carrying `joins` behaved exactly as if the key were absent, while its name squatted on the reserved REST parameter set. Delete the key. Related records are read through `expand` — `expand: { owner_id: { object: 'user', fields: ['name'] } }` — which the engine resolves via batch $in queries, and whose nested query selects the related record's own columns. Keep the foreign key in your own projection (`fields: ['title', 'owner_id']`): the relation is carried by that column, so projecting it away leaves expansion nothing to resolve. A dotted `fields` path is NOT a replacement — no driver ever resolved one and the ingress refuses it (`400 INVALID_FIELD`, #7532). |
 | **aggregations** | `{ function: Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct'>; field?: string; alias: string; filter?: any }[]` | optional | Aggregation functions |
 | **groupBy** | `(string \| { field: string; dateGranularity?: Enum<'day' \| 'week' \| 'month' \| 'quarter' \| 'year'>; alias?: string })[]` | optional | GROUP BY targets (strings or `{field, dateGranularity?}` objects for date bucketing) |
 | **having** | `any` | optional | HAVING — filter over the AGGREGATED rows (aggregation aliases + groupBy projections); applied engine-side after aggregation |

content/docs/references/data/query.mdx

@@ -122,7 +122,7 @@ Type: `string`
 | Property | Type | Required | Description |
 | :--- | :--- | :--- | :--- |
 | **object** | `string` | ✅ | Object name (e.g. account) |
-| **fields** | `string[]` | optional | Fields to retrieve — field names, optionally dotted to reach through a relationship (`owner.name`). Related *records* are selected with `expand`, not from inside this list. |
+| **fields** | `string[]` | optional | Fields to retrieve — names of the queried object's OWN columns. A dotted path (`owner.name`) is not a projection: no driver resolves one, and the ingress refuses it with `400 INVALID_FIELD` (#7532). Related data is read with `expand`, whose nested QueryAST both filters (`where`) and selects (`fields`) the related record's columns. The projection must RETAIN the foreign-key column: `fields: ['title']` with `expand: 'project_id'` resolves nothing, because the relation is carried by that key — add `'project_id'` and it works. Where the value is wanted on the queried object itself, denormalise it onto that object (a stored field, written when the source changes), the same remedy the sort axis prescribes (#6924). |
 | **where** | `any` | optional | Filtering criteria (WHERE) |
 | **search** | `string \| { query: string; fields?: string[]; fuzzy: boolean; operator: Enum<'and' \| 'or'>; … }` | optional | Full-text search — the query text (canonical, ADR-0061 D1), or a structured FullTextSearch configuration |
 | **searchFields** | `string[]` | optional | Narrow the search to these fields (server-intersected with the allowed searchable set — can only narrow, never widen; ADR-0061 D1) |
@@ -131,7 +131,7 @@ Type: `string`
 | **offset** | `number` | optional | Records to skip (OFFSET) |
 | **top** | `number` | optional | Alias for limit (OData compatibility) |
 | **cursor** | `never` | optional | [REMOVED] `query.cursor` was removed in @objectstack/spec 17 (#4286, ADR-0049) — no driver ever implemented keyset pagination, so the cursor was accepted and ignored and every page came back identical (a caller looping "until hasMore is false" never terminates). Delete the key; `QueryBuilder.cursor()` was removed with it. Express the keyset as an ordinary `where` predicate on your sort key — `where: { created_at: { $gt: last.created_at } }` with the matching `orderBy` — which every driver executes with canonicalised comparands. A first-class cursor, if ever built, will be a response-minted opaque token, not this caller-built record. |
-| **joins** | `never` | optional | [REMOVED] `query.joins` was removed in @objectstack/spec 17 (#4286, ADR-0049) — no engine or driver ever read it: a query carrying `joins` behaved exactly as if the key were absent, while its name squatted on the reserved REST parameter set. Delete the key. Related records are read through `expand` — `expand: { owner: { object: 'user', fields: ['name'] } }` — which the engine resolves via batch $in queries, and a single related column is a dotted `fields` path (`fields: ['owner.name']`). |
+| **joins** | `never` | optional | [REMOVED] `query.joins` was removed in @objectstack/spec 17 (#4286, ADR-0049) — no engine or driver ever read it: a query carrying `joins` behaved exactly as if the key were absent, while its name squatted on the reserved REST parameter set. Delete the key. Related records are read through `expand` — `expand: { owner_id: { object: 'user', fields: ['name'] } }` — which the engine resolves via batch $in queries, and whose nested query selects the related record's own columns. Keep the foreign key in your own projection (`fields: ['title', 'owner_id']`): the relation is carried by that column, so projecting it away leaves expansion nothing to resolve. A dotted `fields` path is NOT a replacement — no driver ever resolved one and the ingress refuses it (`400 INVALID_FIELD`, #7532). |
 | **aggregations** | `{ function: Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct'>; field?: string; alias: string; filter?: any }[]` | optional | Aggregation functions |
 | **groupBy** | `(string \| { field: string; dateGranularity?: Enum<'day' \| 'week' \| 'month' \| 'quarter' \| 'year'>; alias?: string })[]` | optional | GROUP BY targets (strings or `{field, dateGranularity?}` objects for date bucketing) |
 | **having** | `any` | optional | HAVING — filter over the AGGREGATED rows (aggregation aliases + groupBy projections); applied engine-side after aggregation |