diff --git a/.changeset/filter-list-field-reference-removed.md b/.changeset/filter-list-field-reference-removed.md new file mode 100644 index 0000000000..f7520b056d --- /dev/null +++ b/.changeset/filter-list-field-reference-removed.md @@ -0,0 +1,88 @@ +--- +"@objectstack/spec": major +--- + +fix(spec): remove `FieldReferenceSchema` from the `$between` endpoints and rule `$in`/`$nin` members out (#7596) + +The filter protocol declared a comparand form that **no backend has ever +implemented**, in every LIST position: both `$between` endpoints carried +`FieldReferenceSchema`, and `$in` / `$nin` were `z.array(z.any())`, which admits +a reference too. ADR-0049's enforce-or-remove shape at a declared position — +resolved by removal (maintainer ruling 2026-08-11). + +Why nothing implemented it, and why that is structural rather than an oversight: + +- **The in-memory evaluator cannot see a list member.** `matches-filter.ts` + `resolveValue` reads `$field` only off a NON-array object + (`!Array.isArray(raw) && '$field' in raw`), and `evalOp` resolves the whole + comparand and never its members. So `$in` / `$nin` compared the raw reference + OBJECT with `looseEq` against a stored value — never equal — and a `$between` + endpoint became an ordering bound that is an object. +- **Both failures are SILENT on that path.** `{ amount: { $in: [{ $field: + 'budget' }] } }` matched nothing and reported nothing; the `$nin` direction + lost an EXCLUSION the author wrote, which widens a scope rather than emptying + it. On an RLS `check` that is a denied write, or an over-broad read, with no + diagnostic anywhere. +- **Both SQL faces already refused these positions loudly** (`INVALID_FILTER` / + 400, naming the field, the operator and the member index — #5041 installed the + refusal and #5222 deliberately kept it: with no correct in-memory semantics + there is nothing for SQL to be conformance-equivalent TO). + +So the declaration was honoured by nobody and refused by two backends. It now +refuses at the schema door as well, with a message an author can act on: + +``` +A { "$field": … } reference is not a valid $in member at index 1. No evaluation +path resolves a field reference inside a list: the in-memory evaluator +(matchesFilter) leaves the list unresolved and compares the raw reference +OBJECT, so it silently matches nothing, and both SQL drivers refuse the position +with INVALID_FILTER / 400. Write a literal value here, or move the reference to +a scalar comparison operator ($eq/$ne/$gt/$gte/$lt/$lte), whose WHOLE comparand a +{ $field } reference may be. Ruled 2026-08-11 on #7596: declared = enforced +(ADR-0049). +``` + +The message deliberately does NOT repeat the SQL drivers' second escape hatch +("or evaluate the rule in memory"): at a list position the memory path is the +one that answers with a wrong row set instead of an error. + +**Unchanged, deliberately:** the four ordering slots and the two equality slots +still take a `{ $field }` reference as their WHOLE comparand. That is #5222's +shipped capability, and it is also what this refusal prescribes — a +column-to-column range is written as +`{ $gte: { $field: 'a' }, $lte: { $field: 'b' } }`, which every face already +answers. The evaluator (`matches-filter.ts`) is not touched. + +**Member types are otherwise untouched.** `$in` / `$nin` members stay `z.any()`: +a membership list is genuinely heterogeneous and this schema is +field-AGNOSTIC — it never sees which column the list applies to, so narrowing +the member type would refuse working filters. One shape is removed, as a check +rather than as a type change. + +## Upgrading + +An authored filter carrying a `{ $field }` reference in an `$in` / `$nin` list +or a `$between` endpoint now fails validation instead of parsing. It never +produced a correct answer on any backend, so no behaviour that worked stops +working: rewrite it as a scalar comparison, per the message above. + + + +**ADR-0087 conversion: not required**, and the reason is not blast radius alone. + +- **No key is retired.** `$in`, `$nin` and `$between` all remain, with their + arity and their other member types intact. A conversion layer converts old + SHAPES to new ones; here there is no new shape to convert to. +- **No lossless transform exists (D2's own requirement).** A `{ $field }` + endpoint has no literal equivalent — the value is a column, unknown at + conversion time — and rewriting `{ $between: [ref, ref] }` into + `{ $gte: ref, $lte: ref }` would not PRESERVE behaviour, it would invent it: + the removed shape evaluated to "matches nothing" in memory and to a 400 on + both SQL faces, so a conversion producing rows would change every existing + answer. +- **Blast radius measured, not asserted: zero.** A whole-repo sweep for + `$between` / `$in` / `$nin` carrying `$field` (`*.ts`, `*.tsx`, `*.md`, + `*.mdx`, `*.json`, `*.yml`) found no template, seed, fixture, example app or + stored metadata using the shape. Every hit was a test or a corpus entry + pinning the REFUSAL, plus historical changelog prose. Nothing to convert, and + no consumer can carry a working dependency on a shape that answered nothing. diff --git a/content/docs/references/data/filter.mdx b/content/docs/references/data/filter.mdx index f63978951d..8d5ad2bcae 100644 --- a/content/docs/references/data/filter.mdx +++ b/content/docs/references/data/filter.mdx @@ -119,8 +119,8 @@ Type: `[FilterArray](#filterarray)[]` | Property | Type | Required | Description | | :--- | :--- | :--- | :--- | -| **$in** | `any[]` | optional | | -| **$nin** | `any[]` | optional | | +| **$in** | `any[]` | optional | Membership list. Members are literal values of any type the column stores. A `{ $field }` reference is NOT a member shape: no backend resolves one inside a list (#7596) — put it in a scalar comparison ($eq/$ne/$gt/$gte/$lt/$lte) instead. | +| **$nin** | `any[]` | optional | Membership list. Members are literal values of any type the column stores. A `{ $field }` reference is NOT a member shape: no backend resolves one inside a list (#7596) — put it in a scalar comparison ($eq/$ne/$gt/$gte/$lt/$lte) instead. | --- diff --git a/packages/drivers/driver-sql/src/cross-field-conformance-cases.ts b/packages/drivers/driver-sql/src/cross-field-conformance-cases.ts index 7b293d45fb..9b2e647b0d 100644 --- a/packages/drivers/driver-sql/src/cross-field-conformance-cases.ts +++ b/packages/drivers/driver-sql/src/cross-field-conformance-cases.ts @@ -441,9 +441,20 @@ export const CROSS_FIELD_REFUSALS: readonly CrossFieldRefusalCase[] = [ // `resolveValue` returns an array unchanged, so `$in`/`$nin` compare against // the raw reference OBJECT (never equal to a stored value) and `$between` // orders against it. There is therefore no correct in-memory semantics for - // SQL to be equivalent TO — refusing is the only answer that is not a guess, - // and the spec's declaration of `FieldReferenceSchema` in the `$between` - // endpoints is filed as its own finding rather than being resolved here. + // SQL to be equivalent TO — refusing is the only answer that is not a guess. + // + // #7596 closed the other half: the spec no longer DECLARES these positions. + // `FieldReferenceSchema` is out of both `$between` endpoint unions and + // `$in`/`$nin` rule the member out by name (maintainer ruling 2026-08-11, + // ADR-0049 declared = enforced), so an authored filter is now refused at the + // schema door with a message naming the scalar-comparison alternative. + // + // These four cases stay, verbatim and unweakened. The schema door is not on + // every path to a driver — `find()` takes a `where` object that no face + // re-validates against `FieldOperatorsSchema` (see #7596's report), and a + // permission filter assembled in code never passes it at all. A driver that + // trusted the declaration would answer these shapes with a silent zero-row + // result, which is exactly what #5041 found here in the first place. { name: 'a $field member of an $in list is refused', filter: { amount: { $in: [{ $field: 'budget' }, 1] } }, diff --git a/packages/objectql/src/engine-filter-array-lowering.test.ts b/packages/objectql/src/engine-filter-array-lowering.test.ts index 9d64bd56c1..97114931c8 100644 --- a/packages/objectql/src/engine-filter-array-lowering.test.ts +++ b/packages/objectql/src/engine-filter-array-lowering.test.ts @@ -500,8 +500,12 @@ describe('Door 2 lowers FilterArray to FilterCondition before the driver (#5158) }); it('the gate does not re-judge list MEMBERS — that is #5234, on another face', async () => { - // A `$field` reference and a plain object are both legitimate members here; - // this gate asks only whether the comparand is a list at all. + // This gate asks only whether the comparand is a LIST at all; WHAT the + // members are is somebody else's judgement. The `$field` member is not a + // legal one — #7596 ruled it out of the spec's `$in`/`$nin` declaration and + // both SQL drivers refuse it by index — and it is used here precisely + // because it is judged elsewhere: the gate must pass it through untouched + // rather than grow a second opinion about members. const where = { stage: { $in: [{ $field: 'other' }, 'won'] } }; await engine.find('deal', { where }); expect(lastWhere()).toEqual(where); diff --git a/packages/spec/src/data/filter.test.ts b/packages/spec/src/data/filter.test.ts index 5ae584f202..e2fc01458c 100644 --- a/packages/spec/src/data/filter.test.ts +++ b/packages/spec/src/data/filter.test.ts @@ -169,6 +169,69 @@ describe('SetOperatorSchema', () => { expect(() => SetOperatorSchema.parse({ $in: [1, 2, 3] })).not.toThrow(); expect(() => SetOperatorSchema.parse({ $in: ['a', 'b', 'c'] })).not.toThrow(); }); + + // ========================================================================== + // #7596 — a `{ $field }` MEMBER is ruled out, by name and with an + // actionable message. + // + // `$in` / `$nin` were `z.array(z.any())`, so a reference parsed here while no + // backend resolved it: `matches-filter.ts` `resolveValue` returns an array + // unchanged, so the raw reference OBJECT was `looseEq`-compared and matched + // nothing (and the `$nin` direction lost an exclusion), while both SQL faces + // refused the position loudly. Maintainer ruling 2026-08-11: REMOVE — + // declared = enforced (ADR-0049). + // + // These assert the MESSAGE, not just the verdict: the whole point of refusing + // at the schema door rather than leaving the position undeclared is that the + // author is told which position works instead. A refusal that cannot go red + // on a missing prescription is not coverage of this ruling. + // ========================================================================== + + describe('$field members are refused (#7596)', () => { + it('refuses a $field member of $in, naming the index and the alternative', () => { + const result = SetOperatorSchema.safeParse({ $in: ['won', { $field: 'budget' }] }); + expect(result.success).toBe(false); + const issue = result.error?.issues[0]; + expect(issue?.path).toEqual(['$in', 1]); + expect(issue?.message).toContain('$in member at index 1'); + expect(issue?.message).toContain('$eq/$ne/$gt/$gte/$lt/$lte'); + expect(issue?.message).toContain('#7596'); + }); + + it('refuses a $field member of $nin — the direction that WIDENS a scope', () => { + const result = SetOperatorSchema.safeParse({ $nin: [{ $field: 'budget' }] }); + expect(result.success).toBe(false); + const issue = result.error?.issues[0]; + expect(issue?.path).toEqual(['$nin', 0]); + expect(issue?.message).toContain('$nin member at index 0'); + }); + + it('reports every offending member, not only the first', () => { + const result = SetOperatorSchema.safeParse({ + $in: [{ $field: 'a' }, 'won', { $field: 'b' }], + }); + expect(result.success).toBe(false); + expect(result.error?.issues.map(i => i.path)).toEqual([['$in', 0], ['$in', 2]]); + }); + + it('leaves every other member shape open — the list is field-AGNOSTIC', () => { + // The check removes ONE shape. A plain object member is still a legal + // membership value (a JSON column stores documents), and narrowing the + // member type would refuse working filters this schema cannot judge. + expect(SetOperatorSchema.safeParse({ $in: [{ nested: 1 }, null, 3, new Date()] }).success) + .toBe(true); + expect(SetOperatorSchema.safeParse({ $in: [] }).success).toBe(true); + }); + + it('is matched by the enforced copy — FieldOperatorsSchema', () => { + expect(FieldOperatorsSchema.safeParse({ $in: [{ $field: 'budget' }] }).success).toBe(false); + expect(FieldOperatorsSchema.safeParse({ $nin: [{ $field: 'budget' }] }).success).toBe(false); + // Positive control: the same lists without the reference still parse, so + // the red above is the member and not the surrounding shape. + expect(FieldOperatorsSchema.safeParse({ $in: [2, 1] }).success).toBe(true); + expect(FieldOperatorsSchema.safeParse({ $nin: [2, 1] }).success).toBe(true); + }); + }); }); describe('RangeOperatorSchema', () => { @@ -241,18 +304,15 @@ describe('RangeOperatorSchema', () => { $between: [new Date('2026-01-01'), '2026-12-31'], }).success).toBe(true); expect(RangeOperatorSchema.safeParse({ - $between: ['2026-01-01', { $field: 'contract.end_date' }], + $between: ['2026-01-01', new Date('2026-12-31')], }).success).toBe(true); }); - it('still accepts numbers, Dates and field references — widening is additive', () => { + it('still accepts numbers and Dates — widening is additive', () => { expect(RangeOperatorSchema.safeParse({ $between: [18, 65] }).success).toBe(true); expect(RangeOperatorSchema.safeParse({ $between: [new Date('2024-01-01'), new Date('2024-12-31')], }).success).toBe(true); - expect(RangeOperatorSchema.safeParse({ - $between: [{ $field: 'a.min' }, { $field: 'a.max' }], - }).success).toBe(true); }); /** @@ -295,6 +355,128 @@ describe('RangeOperatorSchema', () => { }).success).toBe(true); }); }); + + // ========================================================================== + // #7596 — a `{ $field }` ENDPOINT is ruled out, in both endpoint unions and + // in both copies of the schema. + // + // Both endpoints declared `FieldReferenceSchema` and no backend ever resolved + // one: `matches-filter.ts` `resolveValue` returns an array unchanged, so the + // raw reference OBJECT became an ordering bound, and both SQL faces refused + // the position with `INVALID_FILTER` / 400. Maintainer ruling 2026-08-11: + // REMOVE — declared = enforced (ADR-0049). + // + // The tests above this block asserted the ACCEPTANCE of exactly these shapes + // (#6571 pinned `['2026-01-01', { $field: 'contract.end_date' }]` and + // `[{ $field: 'a.min' }, { $field: 'a.max' }]`); they are flipped here rather + // than deleted, so the removal is pinned in the same place the declaration + // was. + // ========================================================================== + + describe('$field endpoints are refused (#7596)', () => { + it('refuses a $field LOWER bound, naming index 0 and the alternative', () => { + const result = RangeOperatorSchema.safeParse({ + $between: [{ $field: 'a.min' }, '2026-12-31'], + }); + expect(result.success).toBe(false); + const issue = result.error?.issues[0]; + expect(issue?.path).toEqual(['$between', 0]); + expect(issue?.message).toContain('$between endpoint at index 0'); + expect(issue?.message).toContain('$eq/$ne/$gt/$gte/$lt/$lte'); + expect(issue?.message).toContain('#7596'); + }); + + it('refuses a $field UPPER bound, naming index 1', () => { + const result = RangeOperatorSchema.safeParse({ + $between: ['2026-01-01', { $field: 'contract.end_date' }], + }); + expect(result.success).toBe(false); + expect(result.error?.issues[0]?.path).toEqual(['$between', 1]); + expect(result.error?.issues[0]?.message).toContain('$between endpoint at index 1'); + }); + + it('refuses a range whose BOTH endpoints are references', () => { + const result = RangeOperatorSchema.safeParse({ + $between: [{ $field: 'a.min' }, { $field: 'a.max' }], + }); + expect(result.success).toBe(false); + expect(result.error?.issues.map(i => i.path)).toEqual([['$between', 0], ['$between', 1]]); + }); + + /** + * The endpoint shapes that were ALREADY invalid keep zod's own wording and + * its `invalid_union` verdict — the `$field` message replaces the generic + * text for one shape and for no other. Without this the refusal could be + * passing by blanketing every rejection with one message. + */ + it('does not repaint the refusals that were already there', () => { + const boolMax = RangeOperatorSchema.safeParse({ $between: ['2026-01-01', true] }); + expect(boolMax.success).toBe(false); + expect(boolMax.error?.issues[0]?.path).toEqual(['$between', 1]); + expect(boolMax.error?.issues[0]?.message).not.toContain('#7596'); + + // An object that is NOT a reference is refused as it always was: this + // check reads the SHAPE, and `{ nope: 1 }` never carried a `$field` key. + const objectMin = RangeOperatorSchema.safeParse({ $between: [{ nope: 1 }, '2026-12-31'] }); + expect(objectMin.success).toBe(false); + expect(objectMin.error?.issues[0]?.message).not.toContain('#7596'); + }); + + it('is matched by the enforced copy — FieldOperatorsSchema', () => { + expect(FieldOperatorsSchema.safeParse({ $between: [{ $field: 'a' }, 100] }).success) + .toBe(false); + expect(FieldOperatorsSchema.safeParse({ $between: [0, { $field: 'a' }] }).success) + .toBe(false); + // Positive control: the same range with literal bounds still parses. + expect(FieldOperatorsSchema.safeParse({ $between: [0, 100] }).success).toBe(true); + }); + + /** + * ## `NormalizedFilterSchema` cannot go red on this, and that is NOT this + * ruling's doing — measured, and pinned so the next reader does not mistake + * the green for enforcement. + * + * A `$and` member is `z.union([z.record(z.string(), FieldOperatorsSchema), + * NormalizedFilterSchema])`. When the record branch rejects a field + * condition, the SECOND branch is a non-strict `z.object({ $and, $or, $not + * })` with every key optional — which accepts any object whatsoever. So the + * whole-filter face admits every field-condition shape, and the control + * below shows it does so for a comparand nobody has ever declared valid. + * The enforcement that this ruling moves lives one level down, in + * `FieldOperatorsSchema`, which the tests above assert directly. Filed + * separately as its own finding; asserting a red here would have been a + * fabricated pin. + */ + it('the whole-filter face is loose about field conditions — pre-existing, control included', () => { + const withReference = NormalizedFilterSchema.safeParse({ + $and: [{ amount: { $between: [{ $field: 'budget' }, 100] } }], + }); + const alreadyInvalidComparand = NormalizedFilterSchema.safeParse({ + $and: [{ close_date: { $null: 'not-a-boolean' } }], + }); + // Both green, for the same structural reason — the second has nothing to + // do with #7596 and was green before it. + expect(withReference.success).toBe(true); + expect(alreadyInvalidComparand.success).toBe(true); + // And the level that DOES judge comparands rejects both. + expect(FieldOperatorsSchema.safeParse({ $between: [{ $field: 'budget' }, 100] }).success) + .toBe(false); + expect(FieldOperatorsSchema.safeParse({ $null: 'not-a-boolean' }).success).toBe(false); + }); + + /** + * The capability this ruling does NOT touch: #5222's cross-field comparison + * is the SCALAR comparand, and it is also what the refusal above prescribes. + * If this went red the refusal message would be sending authors nowhere. + */ + it('leaves the four ORDERING slots taking a reference — #5222, and the prescribed alternative', () => { + expect(ComparisonOperatorSchema.safeParse({ $gt: { $field: 'budget' } }).success).toBe(true); + expect(FieldOperatorsSchema.safeParse({ + $gte: { $field: 'a.min' }, $lte: { $field: 'a.max' }, + }).success).toBe(true); + expect(FieldOperatorsSchema.safeParse({ $eq: { $field: 'budget' } }).success).toBe(true); + }); + }); }); // ============================================================================ diff --git a/packages/spec/src/data/filter.zod.ts b/packages/spec/src/data/filter.zod.ts index 1e99376886..6fabd0698e 100644 --- a/packages/spec/src/data/filter.zod.ts +++ b/packages/spec/src/data/filter.zod.ts @@ -41,9 +41,11 @@ import { z } from 'zod'; * uniform across evaluation paths, and a producer must know which path its * filter will run on: * - * - **In-memory evaluation — supported.** `matchesFilter` - * (`@objectstack/formula`, `matches-filter.ts`) resolves the reference - * against the record, dot-paths included. + * - **In-memory evaluation — supported, in SCALAR positions only.** + * `matchesFilter` (`@objectstack/formula`, `matches-filter.ts`) resolves the + * reference against the record, dot-paths included, when the reference is the + * WHOLE comparand. It does **not** descend into a list — see the LIST + * positions carve-out below. * - **SQL push-down — refused, loudly.** `@objectstack/driver-sql` (and * `driver-sqlite-wasm`, which inherits its filter compiler) does not compile * a field reference to a column-to-column comparison. Rather than bind the @@ -59,8 +61,37 @@ import { z } from 'zod'; * the two open semantic questions ride with it — dot-path relation references, * and the validation boundary for the referenced column name. * + * ## LIST positions are NOT part of this declaration (#7596, ruled 2026-08-11) + * + * A reference may be the whole comparand of a scalar comparison. It may **not** + * be a member of an `$in` / `$nin` list, nor an endpoint of a `$between` range. + * Those positions were declared here — both `$between` endpoints carried + * `FieldReferenceSchema`, and `$in` / `$nin` admitted anything — and **no** + * backend ever resolved them: + * + * - **The memory evaluator returns a list unresolved, structurally.** + * `matches-filter.ts` `resolveValue` reads `$field` only off a non-array + * object (`!Array.isArray(raw) && '$field' in raw`), and `evalOp` resolves the + * whole comparand and never its members. So `$in` / `$nin` compare the raw + * reference OBJECT with `looseEq` against a stored value — never equal — and + * a `$between` endpoint becomes an ordering bound that is an object. Both fail + * SILENTLY: the `$in` direction matches nothing, and the `$nin` direction + * loses an exclusion the author wrote, which on an RLS `check` widens a scope. + * - **Both SQL faces already refuse them**, by name and by index (#5041 installed + * the refusal, #5222 deliberately kept it: there is no correct in-memory + * semantics for SQL to be conformance-equivalent TO). + * + * So the declaration was honoured by nobody and refused by two backends — + * ADR-0049's enforce-or-remove shape at a declared position. The maintainer + * ruled REMOVE rather than implement: member resolution has zero measured + * consumers, and per-member OR-expansion carries NULL and type-affinity + * questions #5222 declined to guess at. The positions now refuse at the SCHEMA + * door too, with a message naming the working alternative — see + * {@link SetOperatorSchema} and {@link RangeOperatorSchema}. + * * @see https://github.com/objectstack-ai/objectstack/issues/5041 (refusal) * @see https://github.com/objectstack-ai/objectstack/issues/5222 (SQL support) + * @see https://github.com/objectstack-ai/objectstack/issues/7596 (list positions removed) */ import { lazySchema } from '../shared/lazy-schema'; export const FieldReferenceSchema = lazySchema(() => z.object({ @@ -214,15 +245,104 @@ export const ComparisonOperatorSchema = lazySchema(() => z.object({ // 3.2 Set & Range Operators // ============================================================================ +/** + * [#7596] Is `value` shaped like a {@link FieldReferenceSchema} reference? + * + * SHAPE only, and deliberately so — the referenced name is not consulted. The + * point is to recognise what the author WROTE so the refusal can name it, which + * has to happen for any `{ $field: … }`, not only for one whose referent would + * have resolved. It mirrors `matches-filter.ts` `resolveValue`'s own test + * (a non-array object carrying a `$field` key) so that "the shape the evaluator + * would have looked for" and "the shape refused here" cannot drift apart. + */ +function isFieldReferenceShape(value: unknown): boolean { + return !!value && typeof value === 'object' && !Array.isArray(value) && '$field' in value; +} + +/** + * [#7596] The author-facing refusal for a `{ $field }` reference in a LIST + * position — every `$in` / `$nin` member, and both `$between` endpoints. + * + * One builder for all four positions because it is one ruling; `position` + * carries the only part that differs, and it names the INDEX for the reason + * `crossFieldComparisonError` (`@objectstack/driver-sql`) names it: the index is + * the only thing distinguishing the bad member from its legitimate neighbours. + * + * ## Why the message says what it says + * + * The schema door and the driver door answer the same author, so they must not + * contradict each other. The SQL drivers' wording ends with two escapes — + * "compare against a literal value here, or evaluate the rule in memory + * (matchesFilter)" — and the SECOND one is not available at a list position: + * the memory evaluator does not resolve list members either, it just fails + * silently instead of loudly. Repeating it here would send an author to the one + * path whose answer is a wrong row set rather than an error. So this message + * keeps the literal-value escape, replaces the in-memory escape with the + * position that genuinely works (the reference as the WHOLE comparand of a + * scalar comparison, which #5222 compiles), and states the ruling that removed + * the position so the change is attributable from the error alone. + */ +function listPositionFieldReferenceMessage(position: string): string { + return ( + `A { "$field": … } reference is not a valid ${position}. No evaluation path resolves a field ` + + 'reference inside a list: the in-memory evaluator (matchesFilter) leaves the list ' + + 'unresolved and compares the raw reference OBJECT, so it silently matches nothing, and ' + + 'both SQL drivers refuse the position with INVALID_FILTER / 400. Write a literal value ' + + 'here, or move the reference to a scalar comparison operator ' + + '($eq/$ne/$gt/$gte/$lt/$lte), whose WHOLE comparand a { $field } reference may be. ' + + 'Ruled 2026-08-11 on #7596: declared = enforced (ADR-0049).' + ); +} + +/** + * [#7596] `$in` / `$nin`, with the `{ $field }` member ruled out by name. + * + * The members stay `z.any()`: a set-membership list is genuinely heterogeneous + * (a `lookup` id, an ISO day, a number), and this schema is field-AGNOSTIC — it + * never sees which column the list applies to, so narrowing the member type + * would refuse working filters, exactly the finding `RangeOperatorSchema`'s + * "why a BARE string" section records for the sibling slot. What IS removable + * is the one shape no backend implements, so it is removed as a check rather + * than as a type change: everything else keeps parsing, `{ $field }` is refused + * with the message it needs, and the generated JSON Schema still describes the + * list as the open one it is. + */ +const setMembershipSchema = (op: '$in' | '$nin') => + z.array(z.any()).superRefine((members, ctx) => { + members.forEach((member, index) => { + if (!isFieldReferenceShape(member)) return; + ctx.addIssue({ + code: 'custom', + path: [index], + message: listPositionFieldReferenceMessage(`${op} member at index ${index}`), + }); + }); + }); + +/** The `describe()` both `$in` and `$nin` carry, stating the one ruled-out member shape. */ +const SET_MEMBER_DESCRIPTION = + 'Membership list. Members are literal values of any type the column stores. A ' + + '{ $field } reference is NOT a member shape: no backend resolves one inside a list ' + + '(#7596) — put it in a scalar comparison ($eq/$ne/$gt/$gte/$lt/$lte) instead.'; + /** * Set operators for membership checks. + * + * ## A `{ $field }` member is refused (#7596, ruled 2026-08-11) + * + * `$in` / `$nin` admit any member type EXCEPT a {@link FieldReferenceSchema} + * reference, which no evaluation path resolves — the reasoning is recorded on + * `FieldReferenceSchema` itself, and the refusal wording on + * {@link listPositionFieldReferenceMessage}. The `$nin` direction is the one + * that mattered: an unresolved member drops an EXCLUSION the author wrote, + * which widens the result set rather than emptying it. */ export const SetOperatorSchema = lazySchema(() => z.object({ /** In list - SQL: IN (?, ?, ?) | MongoDB: $in */ - $in: z.array(z.any()).optional(), - + $in: setMembershipSchema('$in').optional().describe(SET_MEMBER_DESCRIPTION), + /** Not in list - SQL: NOT IN (...) | MongoDB: $nin */ - $nin: z.array(z.any()).optional(), + $nin: setMembershipSchema('$nin').optional().describe(SET_MEMBER_DESCRIPTION), })); /** @@ -234,9 +354,11 @@ export const SetOperatorSchema = lazySchema(() => z.object({ * sentence is in {@link RangeOperatorSchema}'s docblock. */ const RANGE_ENDPOINT_DESCRIPTION = - 'Closed interval [min, max]. Each endpoint is a number, a Date, a string, or ' - + 'a { $field } reference — the SAME union the ordering comparisons take, ' - + 'because a range IS its two ordering bounds. STRING is the form the ' + 'Closed interval [min, max]. Each endpoint is a number, a Date, or a string. ' + + 'A { $field } reference is NOT an endpoint shape: no backend resolves one ' + + 'inside a list (#7596) — put it in a scalar comparison ' + + '($gt/$gte/$lt/$lte), which does compile to a column-to-column bound. ' + + 'STRING is the form the ' + 'platform itself produces: the date-macro resolver walks INTO arrays, so ' + '{ $between: ["{current_year_start}", "{current_year_end}"] } resolves to ' + 'two strings. The guaranteed spellings are an ISO calendar day ' @@ -253,7 +375,26 @@ const RANGE_ENDPOINT_DESCRIPTION = * Range operator for interval checks (closed interval). * SQL: BETWEEN ? AND ? | MongoDB: $gte AND $lte * - * Supported endpoint types: **Number, Date, ISO/clock STRING, FieldReference**. + * Supported endpoint types: **Number, Date, ISO/clock STRING**. + * + * ## A `{ $field }` endpoint is refused (#7596, ruled 2026-08-11) + * + * Both endpoint unions carried `FieldReferenceSchema` until this ruling, and no + * backend ever resolved one: `matches-filter.ts` leaves a list unresolved and + * orders against the raw reference OBJECT, while both SQL faces refuse the + * position loudly. ADR-0049's enforce-or-remove shape, resolved by REMOVAL — + * the reasoning is recorded on {@link FieldReferenceSchema}, the wording on + * `listPositionFieldReferenceMessage`. + * + * The reference stays legal in the four ORDERING slots + * ({@link ComparisonOperatorSchema}) — that is #5222's shipped capability, and + * it is also the alternative this refusal prescribes. "A range IS its two + * ordering bounds" therefore stops being true of the COMPARAND union at exactly + * one member: `$gt` takes a reference because a scalar comparison compiles to a + * column-to-column bound; a `$between` endpoint does not, because nothing + * resolves a member of a list. An author wanting a column-to-column range + * writes the two bounds separately (`{ $gte: { $field: 'a' }, $lte: { $field: 'b' } }`), + * which every face already answers. * * ## Why `string` is in BOTH endpoint unions (#6571) * @@ -313,12 +454,36 @@ const RANGE_ENDPOINT_DESCRIPTION = * either — an inverted `[max, min]` range is a well-formed filter that matches * nothing, at every backend. */ +/** + * [#7596] One `$between` endpoint, with the `{ $field }` shape ruled out. + * + * ## Why the refusal rides on the union's `error` and not on a `superRefine` + * + * Measured on zod 4.4.3: a check attached to the TUPLE does not run once an + * ELEMENT has failed, so a tuple-level refinement could never see the endpoint + * it was written to explain — the author would get zod's bare + * `invalid_union` / "Invalid input" and nothing else. The union's own `error` + * callback runs exactly when the union rejects and sees the offending input, so + * it replaces that generic text with {@link listPositionFieldReferenceMessage} + * for this one shape and returns `undefined` for every other rejection, leaving + * zod's default wording — and, importantly, the issue's `code` and `path` — + * untouched for the endpoint shapes that were already invalid. + * + * `index` is baked in per endpoint rather than read from the issue: at the time + * the union reports, the path is still relative to the union itself and the + * tuple has not yet prefixed the position. + */ +const rangeEndpointSchema = (index: 0 | 1) => + z.union([z.number(), z.date(), z.string()], { + error: (issue) => + isFieldReferenceShape(issue.input) + ? listPositionFieldReferenceMessage(`$between endpoint at index ${index}`) + : undefined, + }); + export const RangeOperatorSchema = lazySchema(() => z.object({ /** Between (inclusive) - takes [min, max] array */ - $between: z.tuple([ - z.union([z.number(), z.date(), z.string(), FieldReferenceSchema]), - z.union([z.number(), z.date(), z.string(), FieldReferenceSchema]) - ]).optional() + $between: z.tuple([rangeEndpointSchema(0), rangeEndpointSchema(1)]).optional() .describe(`Between (inclusive). ${RANGE_ENDPOINT_DESCRIPTION}`), })); @@ -807,22 +972,25 @@ export const FieldOperatorsSchema = lazySchema(() => z.object({ $lt: z.union([z.number(), z.date(), z.string(), FieldReferenceSchema]).optional(), $lte: z.union([z.number(), z.date(), z.string(), FieldReferenceSchema]).optional(), - // Set & Range - $in: z.array(z.any()).optional(), - $nin: z.array(z.any()).optional(), + // Set. Members are open (`z.any()`) EXCEPT the one shape no backend resolves: + // a `{ $field }` reference, ruled out by name in #7596. Built from the same + // `setMembershipSchema` factory the documentation copy uses — the two copies + // share the code rather than a description of it, so this pair cannot drift. + $in: setMembershipSchema('$in').optional().describe(SET_MEMBER_DESCRIPTION), + $nin: setMembershipSchema('$nin').optional().describe(SET_MEMBER_DESCRIPTION), // Range. `string` is in BOTH endpoint unions for the reason // {@link RangeOperatorSchema} gives at length (#6571): the date-macro resolver // walks into arrays, so a token range resolves to two ISO/clock STRINGS, and - // this package's own `temporal-conformance.ts` corpus spells that shape. This - // copy is the ENFORCED one — `NormalizedFilterSchema` validates against it and - // the exported `FieldOperators` is inferred from it — so it must not drift - // from the documentation copy above. #5685 landed the sibling ordering slots - // in the documentation copy first and left the reachable surface still - // rejecting the platform's own output; both spellings move together. - $between: z.tuple([ - z.union([z.number(), z.date(), z.string(), FieldReferenceSchema]), - z.union([z.number(), z.date(), z.string(), FieldReferenceSchema]) - ]).optional(), + // this package's own `temporal-conformance.ts` corpus spells that shape. + // `FieldReferenceSchema` is NOT in them, for the reason the same docblock + // gives (#7596): nothing resolves a reference inside a list. This copy is the + // ENFORCED one — `NormalizedFilterSchema` validates against it and the + // exported `FieldOperators` is inferred from it — so it must not drift from + // the documentation copy above. #5685 landed the sibling ordering slots in the + // documentation copy first and left the reachable surface still rejecting the + // platform's own output; both spellings move together, which is why the + // endpoint is one shared `rangeEndpointSchema` factory here too. + $between: z.tuple([rangeEndpointSchema(0), rangeEndpointSchema(1)]).optional(), // String-specific. Case-SENSITIVE, except `$icontains` which folds ASCII case // only — see {@link StringOperatorSchema} for the contract and its boundary.