fix(spec,lint): a formula field in searchableFields is refused loudly (#6674)

.changeset/searchable-fields-formula-refused.md

@@ -0,0 +1,68 @@
+---
+"@objectstack/metadata-protocol": patch
+"@objectstack/spec": patch
+"@objectstack/lint": patch
+---
+
+fix(spec,lint): a virtual `formula` field in `searchableFields` is refused loudly, not admitted verbatim (#6674)
+
+#4254 closed the fail-open on the unknown-name axis: a `$searchFields` entry the
+engine would not scan is `400 INVALID_FIELD`, never a silently widened search.
+The same shape survived one axis over, on names that are perfectly real.
+
+The declared branch of `resolveSearchFieldResolution` filtered entries by
+EXISTENCE only, so a `formula` field declared in `searchableFields` entered the
+allowed set — and the ingress gate, which reads that same set, accepted it for
+exactly that reason. Measured on `origin/main`:
+
+```
+AUTO:          {"allowed":["name","project_name"],"source":"auto"}                formula excluded
+DECL-FORMULA:  {"allowed":["name","project_name_formula"],"source":"declared"}    admitted verbatim
+?search=Apollo&searchFields=project_name_formula  ->  200, 0 rows                 silent
+```
+
+Zero rows is the defect. A formula value is computed on read and no driver
+materializes a column for it (`driver-sql` `fieldHasColumn`, driver-turso's
+"Virtual — no column"), so the `$contains` the engine expands `$search` into has
+nothing to scan: 0 rows on driver-memory (the property is absent from the stored
+row) and 0 rows WITH NO ERROR on driver-sql/better-sqlite3. The declaration read
+as search coverage and delivered none.
+
+- **`@objectstack/spec` — the deciding face.** The declared branch now filters on
+  existence AND scannability: an entry naming a virtual field is not admitted.
+  New exports `SEARCH_VIRTUAL_TYPES` (exactly `formula`, pinned) and
+  `isVirtualSearchField` — one judgment, so the resolution, the gate and the
+  linter cannot drift about which types have a column. The resolution itself
+  stays non-throwing: it is consulted on every search by internal callers that
+  never pass an ingress, which is why #4254 put the loudness at the ingress.
+- **`@objectstack/metadata-protocol` — `400 INVALID_FIELD` with its own reason.**
+  Split out before the declared/auto branch, because both of those messages are
+  wrong for it: "outside the declared set" is false when the entry IS in the
+  list, and the auto-default's "declare `searchableFields` to choose the
+  searchable set" would instruct the author to write the declaration being
+  refused. The new message names the field, its type, that the value is computed
+  on read and never stored, and the fix (mirror onto a stored text field).
+- **`@objectstack/lint` — a build error at authoring time**, on the object's own
+  `searchableFields` as well as a view's narrowing, under the existing
+  `searchable-field-unsearchable` rule (no new rule id). This narrows the
+  canonical surface, which #4830 had deliberately left existence-only.
+
+The carve-out that made canonical existence-only is deliberately KEPT and pinned
+by controls in all three packages: the dividing line is STORAGE, not search
+quality. A `json` or `lookup` column declared in `searchableFields` is still the
+author's choice and still executed — a `$contains` over the stored JSON text or
+the stored foreign key. Narrow and rarely useful, but a scan that CAN match, so
+it is neither a 400 nor a finding. Only "there is no column at all" is refused.
+
+**Compatibility.** A corpus sweep of this repo plus `objectui` and `cloud` found
+ZERO authored `searchableFields` naming a formula-typed field, so nothing in the
+tree changes verdict. For an already-published object that does carry one:
+loading is unaffected (no schema-parse change — `searchableFields` is still
+`z.array(z.string())`, this is a resolution and enforcement rule); a plain
+`?search=` keeps returning the SAME rows, because the dropped entry matched none
+of them; only a request that NAMES the formula field flips from `200` with no
+rows to `400 INVALID_FIELD` — including objectui's list search, which echoes the
+declaration verbatim. An object whose `searchableFields` is ENTIRELY formula
+entries filters to empty and falls through to the auto-default, exactly as an
+all-stale declaration has since #4254; the linter reports the declaration rather
+than leaving that swap silent.

content/docs/data-modeling/schema-design.mdx

@@ -140,9 +140,13 @@ by the auto-default anyway; declare the set explicitly when you want to pin it.
 `$contains` predicate against one has nothing to scan (the SQL driver would emit
 a `WHERE` over a column that does not exist). A CEL formula also only reads this
 record's own fields (`record.<field>`), so it cannot fetch the related title in
-the first place. Nothing catches the mistake for you — `searchableFields` admits
-any field the object declares, so a formula entry passes both lint and the
-ingress gate and then just never matches.
+the first place.
+
+Since #6674 the mistake is **refused rather than silent**: a `formula` entry in
+any `searchableFields` — the object's own set included — is an `os validate`
+error (`searchable-field-unsearchable`), and a request naming one is `400
+INVALID_FIELD`. It used to clear both and then match nothing, which read as
+search coverage and delivered none.
 </Callout>
 
 **Keeping the mirror fresh.** A mirror is denormalized data, only as current as

content/docs/references/data/object.mdx

@@ -137,7 +137,7 @@ const result = ApiMethod.parse(data);
 | **highlightFields** | `string[]` | optional | [ADR-0085] Ordered most-important fields; first entry wins where only one fits. Drives default columns, cards, previews, detail highlight strip. Renamed from compactLayout. |
 | **stageField** | `string \| false` | optional | [ADR-0085] Lifecycle stage field (linear/ordered), or false to declare the status field non-linear and suppress stage heuristics. Absent = heuristic detection allowed. |
 | **listViews** | `Record<string, { name?: string; label?: string \| Record<string, string>; type?: Enum<'grid' \| 'kanban' \| 'gallery' \| 'calendar' \| 'timeline' \| 'gantt' \| 'map' \| 'chart' \| 'tree'>; data?: object \| … +3 more; … }>` | optional | Built-in named list views (segmented tabs) shipped with the object schema — "views" mode, dropdown userFilters allowed, no page-only tabs (ADR-0047) |
-| **searchableFields** | `string[]` | optional | Fields the `$search` query matches against (ADR-0061). Canonical default for the record picker, list quick-search and global search; views may narrow it. When unset, search auto-defaults to the name/title field plus short-text fields. |
+| **searchableFields** | `string[]` | optional | Fields the `$search` query matches against (ADR-0061). Canonical default for the record picker, list quick-search and global search; views may narrow it. When unset, search auto-defaults to the name/title field plus short-text fields. Entries must name a STORED column: a virtual `formula` field is computed on read and materializes no column, so searching it can never match and it is refused (#6674) — mirror the value onto a stored text field and declare that. |
 | **enable** | `{ trackHistory?: boolean; searchable?: boolean; apiEnabled?: boolean; apiMethods?: Enum<'get' \| 'list' \| 'create' \| 'update' \| 'delete' \| 'bulk'>[]; … }` | optional | Enabled system features modules |
 | **sharingModel** | `Enum<'private' \| 'public_read' \| 'public_read_write' \| 'controlled_by_parent'>` | optional | Org-Wide Default record visibility (OWD) for INTERNAL users. Canonical four only (legacy aliases removed, ADR-0090 D4): private (owner-only) \| public_read (everyone reads, owner writes) \| public_read_write (everyone reads+writes) \| controlled_by_parent (derived from the master record). A CUSTOM object that omits this resolves to private at runtime (ADR-0090 D1). |
 | **externalSharingModel** | `Enum<'private' \| 'public_read' \| 'public_read_write' \| 'controlled_by_parent'>` | optional | [ADR-0090 D11] OWD for external (portal/partner) principals. Defaults to private; must be &lt;= sharingModel in openness. |

content/docs/ui/views.mdx

@@ -160,6 +160,7 @@ all.
 | key omitted, or `searchableFields: []` | clean | scans the object's full allowed set |
 | a renamed / mistyped column, or a dotted path | `searchable-field-unknown` | `400 INVALID_FIELD` |
 | a real column outside the allowed set | `searchable-field-unsearchable` | `400 INVALID_FIELD` |
+| a virtual `formula` column — no stored column to scan (#6674) | `searchable-field-unsearchable` | `400 INVALID_FIELD` |
 
 Both diagnostics are **errors**, not warnings — `os validate` fails the build.
 The object's own set, and the stored-mirror prescription, are covered under

packages/lint/src/validate-searchable-fields.test.ts

@@ -647,3 +647,113 @@ describe('validateSearchableFields — objectstack-ui SKILL.md parity (#6675)',
     expect(refused[0].message).toContain("of type 'lookup', which 'search' cannot scan");
   });
 });
+
+/**
+ * [#6674] A virtual `formula` entry — the one check that runs on the object's
+ * OWN set as well as on a view's narrowing.
+ *
+ * The card's shape: the entry names a real field, so the existence check passes
+ * it; the runtime's declared branch admitted it verbatim; and the search then
+ * matched nothing, because a formula value is computed on read and no driver
+ * materializes a column for it (0 rows on driver-memory, 0 rows WITH NO ERROR on
+ * driver-sql). Declared coverage, zero delivery — the fail-open #4254 closed on
+ * the unknown-name axis, surviving on the known-but-virtual one.
+ */
+describe('[#6674] validateSearchableFields — a virtual formula entry', () => {
+  const accountFields = {
+    name: { type: 'text' },
+    billing_email: { type: 'email' },
+    payload: { type: 'json' },
+    account_id: { type: 'lookup', reference: 'crm_account' },
+    display_label: { type: 'formula', expression: "record.name + ' · x'" },
+  };
+
+  it("flags it on the OBJECT's own searchableFields — previously clean", () => {
+    const findings = validateSearchableFields({
+      objects: [
+        {
+          name: 'crm_account',
+          fields: accountFields,
+          searchableFields: ['name', 'display_label'],
+        },
+      ],
+    });
+
+    expect(findings).toHaveLength(1);
+    expect(findings[0].rule).toBe(SEARCHABLE_FIELD_UNSEARCHABLE);
+    expect(findings[0].severity).toBe('error');
+    expect(findings[0].path).toBe('objects[0].searchableFields[1]');
+    expect(findings[0].where).toBe('object "crm_account"');
+    expect(findings[0].message).toContain("is a virtual 'formula' field");
+    expect(findings[0].message).toContain('computed on read and never stored');
+    // The fix is a STORED mirror — the same prescription #6673 put on the
+    // neighbouring hints, and the only one that can work here.
+    expect(findings[0].hint).toContain('stored text field');
+    expect(findings[0].hint).toContain('400 INVALID_FIELD');
+  });
+
+  it('flags it on a list view narrowing too', () => {
+    const findings = validateSearchableFields({
+      objects: [
+        {
+          name: 'crm_account',
+          fields: accountFields,
+          listViews: { all: { type: 'grid', searchableFields: ['name', 'display_label'] } },
+        },
+      ],
+    });
+
+    expect(findings).toHaveLength(1);
+    expect(findings[0].rule).toBe(SEARCHABLE_FIELD_UNSEARCHABLE);
+    expect(findings[0].path).toBe('objects[0].listViews.all.searchableFields[1]');
+    expect(findings[0].message).toContain("is a virtual 'formula' field");
+  });
+
+  it('CONTROL — a json or lookup entry on the OBJECT\'s own set stays clean', () => {
+    // The carve-out #4830 wrote down, deliberately preserved: the runtime's
+    // declared branch executes those (a `$contains` over the stored JSON text /
+    // the stored foreign key). Narrow and rarely useful, but a scan that CAN
+    // match — flagging it would reject metadata the runtime accepts (ADR-0072
+    // D1). If this control ever goes red, #6674 has quietly become "the declared
+    // branch is type-filtered", which it is not.
+    expect(
+      validateSearchableFields({
+        objects: [
+          {
+            name: 'crm_account',
+            fields: accountFields,
+            searchableFields: ['name', 'payload', 'account_id'],
+          },
+        ],
+      }),
+    ).toEqual([]);
+  });
+
+  it('CONTROL — a stale entry on the object\'s own set keeps the #4254 message', () => {
+    const findings = validateSearchableFields({
+      objects: [
+        { name: 'crm_account', fields: accountFields, searchableFields: ['name', 'gone'] },
+      ],
+    });
+
+    expect(findings).toHaveLength(1);
+    expect(findings[0].rule).toBe(SEARCHABLE_FIELD_UNKNOWN);
+    expect(findings[0].message).toContain('is not a field on object');
+  });
+
+  it('an ALL-virtual declaration is reported, not silently swapped for the auto-default', () => {
+    // The degenerate case: at runtime the declaration filters to empty and
+    // resolution falls through to the auto-default, so the object silently
+    // searches a set the author never wrote. The build error is what stops that
+    // being invisible.
+    const findings = validateSearchableFields({
+      objects: [
+        { name: 'crm_account', fields: accountFields, searchableFields: ['display_label'] },
+      ],
+    });
+
+    expect(findings).toHaveLength(1);
+    expect(findings[0].path).toBe('objects[0].searchableFields[0]');
+    expect(findings[0].message).toContain("is a virtual 'formula' field");
+  });
+});

packages/lint/src/validate-searchable-fields.ts

@@ -60,12 +60,25 @@
  *    linter, gate and engine cannot drift apart (the same one-source move
  *    #4254 made between gate and engine).
  *
- * The OBJECT's own `searchableFields` stays existence-only: the runtime's
- * declared branch filters by existence, never by type, so a json or lookup
- * column declared THERE is a choice the engine executes (a `$contains` over
- * the raw column), not a 400. Flagging it would reject metadata the runtime
- * accepts — the false finding that makes authors stop trusting the linter
- * (ADR-0072 D1).
+ * 3. VIRTUALITY, on EVERY surface — the object's own set included
+ *    (`searchable-field-unsearchable`, #6674): a `formula` entry names a real
+ *    field, so check 1 passes it, and the runtime's declared branch admitted
+ *    it verbatim — but the value is computed on read with no stored column, so
+ *    the scan looks at nothing. Measured 0 rows on driver-memory and 0 rows
+ *    WITH NO ERROR on driver-sql. Since #6674 the spec resolution drops such
+ *    an entry and both enforcement faces refuse it by name.
+ *
+ * The OBJECT's own `searchableFields` stays existence-only OTHERWISE, and the
+ * dividing line is STORAGE, not search quality: the runtime's declared branch
+ * filters by existence and scannability, never by type taste, so a json or
+ * lookup column declared THERE is a choice the engine executes (a `$contains`
+ * over the stored JSON text / the stored foreign key) — narrow, rarely useful,
+ * but a scan that CAN match, so it is neither a 400 nor a finding. Flagging
+ * those would reject metadata the runtime accepts — the false finding that
+ * makes authors stop trusting the linter (ADR-0072 D1). A virtual entry is the
+ * opposite case: no column exists on any driver, so admitting it is the
+ * fail-open #4254 closed one axis over, surviving on the known-but-virtual
+ * axis.
  *
  * Three skips keep false positives near zero (ADR-0072 D1):
  *
@@ -93,6 +106,7 @@
 
 import {
   resolveSearchFieldResolution,
+  isVirtualSearchField,
   SEARCHABLE_TEXTUAL_TYPES,
   SEARCHABLE_ENUM_TYPES,
   SEARCH_AUTO_EXCLUDED_FIELDS,
@@ -124,8 +138,9 @@ export interface SearchableFieldFinding {
  * Which runtime judgment applies to the declaration being checked:
  *
  * - `'canonical'` — the object's own `searchableFields`. The runtime honors
- *   any entry that exists (existence-filtered, never type-filtered), so only
- *   existence is checked.
+ *   any entry that exists and has a stored column to scan (existence- and
+ *   scannability-filtered, never type-filtered), so existence and virtuality
+ *   are checked and nothing else (#6674).
  * - `'narrowing'` — a list view's `searchableFields` (metadata or react
  *   surface). Clients echo it as the `$searchFields` override, which the
  *   #4254 ingress gate intersects with the object's allowed set — entries the
@@ -354,6 +369,35 @@ export function checkSearchableFieldList(
       continue;
     }
 
+    // ── [#6674] Virtual entries — EVERY surface, canonical included ──
+    //
+    // The one check that is not view-level, because the runtime's declared
+    // branch no longer executes it: a `formula` value is computed on read and
+    // has no stored column, so the entry can never match wherever it is
+    // declared. Judged by the spec's own predicate, so linter and runtime
+    // cannot disagree about which types have a column.
+    if (isVirtualSearchField(target.fields[name])) {
+      const vtype = target.fields[name]?.type;
+      findings.push({
+        severity: 'error',
+        rule: SEARCHABLE_FIELD_UNSEARCHABLE,
+        where,
+        path: `${path}[${i}]`,
+        message:
+          `${subject} entry "${name}" on object "${objectName}" is a virtual ` +
+          `'${vtype}' field: its value is computed on read and never stored, so no ` +
+          `driver materializes a column for 'search' to scan and the entry can never ` +
+          `match. It reads as search coverage and delivers none — the runtime used to ` +
+          `admit it verbatim because the declaration named it (#6674).`,
+        hint:
+          `Mirror the computed value onto a stored text field on "${objectName}" and ` +
+          `declare that instead, or drop "${name}". At runtime the ingress gate now ` +
+          `refuses this entry with 400 INVALID_FIELD, the same answer a stale entry ` +
+          `gets (#4254).`,
+      });
+      continue;
+    }
+
     // ── Runtime admissibility (#4830) — view-level narrowings only ──
     if (!resolution || resolution.allowed.has(name)) continue;
     // ③ System column outside the allowed set: its runtime metadata is