fix(objectql,driver-mongodb): declare the tenant index in indexes[], so a registry-backed object stops reporting itself invalid (#6810)

.changeset/tenant-index-declared-in-indexes.md

@@ -0,0 +1,62 @@
+---
+"@objectstack/objectql": patch
+"@objectstack/driver-mongodb": patch
+---
+
+fix(objectql,driver-mongodb): declare the tenant index in `indexes[]`, so a registry-backed object stops reporting itself invalid (#6810)
+
+`applySystemFields` provisioned the injected `organization_id` column with
+`indexed: opts.multiTenant`. `indexed` is **not a `FieldSchema` key** — #2377 /
+ADR-0049 removed it because a field-level index flag built no index — and
+`FieldSchema` is a `strictObject`, so a field carrying it is rejected **by
+name**, with a purpose-written message.
+
+`registerObject` runs `applySystemFields` *before* storing and
+`getItem('object', …)` serves that post-injection document, so the key travelled
+all the way out to `/meta`, where `decorateMetadataItem` re-parsed the served
+body and stamped the verdict on it. Measured on every registry-backed object, in
+**both** tenancy modes, at **both** read exits:
+
+```
+_diagnostics: { valid: false,
+  errors: [{ path: 'fields.organization_id', code: 'unrecognized_keys' }] }
+```
+
+`_diagnostics` is what Studio renders invalid-metadata banners from and what an
+AI author reads to judge a document it produced. So the platform was reporting a
+defect on its own column — one the author never wrote and could not fix — and
+making the verdict useless as a signal on those objects, because a real
+authoring error was indistinguishable from this one.
+
+**Two directions, both of them user-visible:**
+
+- **The false `valid: false` verdict is gone.** A tenancy-enabled object
+  registered through the real `SchemaRegistry` now reads back
+  `_diagnostics: { valid: true }` at both `/meta` exits, in both tenancy modes.
+  Nothing else about the served field changed — `type`, `reference`, and the
+  governance keys that decide who may write it are byte-identical.
+- **The tenant index moved from a field-level flag to `indexes[]`**, the one
+  surface an index is declared on in this system. On a multi-tenant stack the
+  object now declares `{ fields: ['organization_id'] }`; on a single-tenant
+  stack it declares **nothing** — the absence *is* what `indexed: false` used to
+  say, since nothing filters by organization on an unwalled stack.
+
+This is also the first time the intent is actually **enforced**. The sole reader
+of the old flag was one line in `driver-mongodb`; `driver-sql` — which every
+walled deployment runs — only ever materialized `indexes[]`, so the wall's
+hottest predicate ran unindexed no matter what the flag said. Expect the tenant
+index to now appear as ordinary index drift on existing SQL tables
+(`idx_<table>_organization_id`), created by `os migrate apply` or by the
+`autoMigrate: 'safe'` path in dev, like any other declared index.
+
+`driver-mongodb` reads declared `indexes[]` in place of the retired flag. The
+generated index name matches the field-level convention already in that file
+(`idx_<fields>` / `idx_<fields>_unique`), so a re-synced collection finds its
+existing `idx_organization_id` rather than building a second index under a new
+name. Declarations are materialized over their columns **verbatim** at every
+`unique` scope, `'organization'` included — the same call the driver's
+field-level `unique` documents, because it implements no row-level tenancy and
+refuses to boot into a multi-tenant deployment (#3724).
+
+No `FieldSchema` change: re-declaring `indexed` would restore exactly the
+declared-but-unenforced key #2377 removed.

packages/drivers/driver-mongodb/README.md

@@ -158,14 +158,20 @@ try {
 
 Schema sync creates collections and indexes:
 
+Field-level `unique` and lookup fields index themselves; everything else is
+declared in the object's `indexes[]` — the one surface an index is declared on
+(a field-level `indexed` flag is not a `FieldSchema` key and never built an
+index, #2377 / #6810).
+
 ```typescript
 await driver.syncSchema('account', {
   name: 'account',
   fields: {
     name: { type: 'string', unique: true },
-    email: { type: 'email', indexed: true },
+    email: { type: 'email' },
     company_id: { type: 'lookup', reference_to: 'company' },
   },
+  indexes: [{ fields: ['email'] }],
 });
 // Creates: idx_id_unique, idx_name_unique, idx_email, idx_company_id_lookup
 ```

packages/drivers/driver-mongodb/src/mongodb-driver.test.ts

@@ -343,9 +343,15 @@ describe.skipIf(!sharedMongod)('MongoDBDriver', () => {
         name: 'account',
         fields: {
           name: { type: 'string', unique: true },
-          email: { type: 'email', indexed: true },
+          email: { type: 'email' },
           company_id: { type: 'lookup', reference_to: 'company' },
         },
+        // [#6810] `email` used to carry a field-level `indexed: true` here. That
+        // was never a `FieldSchema` key (#2377 / ADR-0049); the index is
+        // declared in `indexes[]`, where every other index in this system is
+        // declared. Same resulting index name — the assertions are unchanged,
+        // which is the point.
+        indexes: [{ fields: ['email'] }],
       });
 
       const db = driver.getDb();

packages/drivers/driver-mongodb/src/mongodb-schema-declared-indexes.test.ts

@@ -0,0 +1,159 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+//
+// #6810 — `syncCollectionSchema` materializes the object's DECLARED `indexes[]`.
+//
+// This driver used to read a field-level `indexed` flag instead. That flag was
+// never a `FieldSchema` key (#2377 / ADR-0049 removed it, and `FieldSchema` is a
+// `strictObject` that rejects it by name), so its one remaining producer — the
+// kernel's `organization_id` injection — was stamping every registry-backed
+// object with a document the platform's own schema refused. The declaration
+// moved to `indexes[]`; this pins that the DDL outcome came with it, rather than
+// the fix quietly deleting an index the flag used to build.
+//
+// Driven against a fake `Db` on purpose: the `mongodb-memory-server` suite in
+// this package is OPT-IN (it downloads a real server binary, #5517), so a DDL
+// assertion parked there would not run on any ordinary CI lane — which is
+// exactly the lane that has to notice if this regresses.
+
+import { describe, it, expect } from 'vitest';
+import type { Db } from 'mongodb';
+import { syncCollectionSchema } from './mongodb-schema.js';
+
+interface CreatedIndex {
+  spec: Record<string, unknown>;
+  options: Record<string, unknown>;
+}
+
+/**
+ * The narrow slice of `Db` `syncCollectionSchema` touches, recording every
+ * `createIndex` call in order. Nothing is stubbed beyond that slice — the
+ * function under test runs verbatim.
+ */
+function fakeDb(existingCollections: string[] = []) {
+  const created: CreatedIndex[] = [];
+  const collectionsCreated: string[] = [];
+  const db = {
+    listCollections: ({ name }: { name: string }) => ({
+      toArray: async () => (existingCollections.includes(name) ? [{ name }] : []),
+    }),
+    createCollection: async (name: string) => {
+      collectionsCreated.push(name);
+    },
+    collection: () => ({
+      createIndex: async (spec: Record<string, unknown>, options: Record<string, unknown>) => {
+        created.push({ spec, options });
+      },
+    }),
+  } as unknown as Db;
+  return { db, created, collectionsCreated };
+}
+
+/** Every index name the sync asked MongoDB to create, core indexes included. */
+const names = (created: CreatedIndex[]) => created.map((c) => c.options.name);
+
+/** The one recorded creation for `name`, or `undefined`. */
+const byName = (created: CreatedIndex[], name: string) =>
+  created.find((c) => c.options.name === name);
+
+describe('#6810 — syncCollectionSchema materializes declared indexes[]', () => {
+  it('creates the kernel-declared tenant index, byte-identical to what the retired flag built', async () => {
+    // THE regression pin. Before #6810 this DDL came from
+    // `organization_id: { …, indexed: true }`; it now comes from the object's
+    // `indexes[]`. Same collection, same key spec, same index NAME — a
+    // re-synced deployment finds its index already present rather than
+    // building a second one under a new name.
+    const { db, created } = fakeDb();
+    await syncCollectionSchema(db, 'lead', {
+      name: 'lead',
+      fields: {
+        first_name: { type: 'text' },
+        organization_id: { type: 'lookup' },
+      },
+      indexes: [{ fields: ['organization_id'] }],
+    });
+
+    const tenant = byName(created, 'idx_organization_id');
+    expect(tenant).toBeDefined();
+    expect(tenant!.spec).toEqual({ organization_id: 1 });
+    // A plain lookup index, never a constraint.
+    expect(tenant!.options.unique).toBeUndefined();
+  });
+
+  it('declares no tenant index when the object declares none', async () => {
+    // The single-tenant half: `multiTenant: false` now declares NOTHING rather
+    // than a `false` flag, so absence has to stay absence here.
+    const { db, created } = fakeDb();
+    await syncCollectionSchema(db, 'lead', {
+      name: 'lead',
+      fields: { first_name: { type: 'text' }, organization_id: { type: 'lookup' } },
+    });
+
+    expect(names(created)).not.toContain('idx_organization_id');
+    // The core set is untouched by any of this.
+    expect(names(created)).toEqual(['idx_id_unique', 'idx_created_at', 'idx_updated_at']);
+  });
+
+  it('honours an explicit index name and a multi-column declaration', async () => {
+    const { db, created } = fakeDb();
+    await syncCollectionSchema(db, 'lead', {
+      name: 'lead',
+      fields: { organization_id: { type: 'lookup' }, code: { type: 'text' } },
+      indexes: [{ name: 'lead_scope_idx', fields: ['organization_id', 'code'] }],
+    });
+
+    const idx = byName(created, 'lead_scope_idx');
+    expect(idx).toBeDefined();
+    // Key order follows the declaration — a compound index is order-sensitive.
+    expect(Object.keys(idx!.spec)).toEqual(['organization_id', 'code']);
+  });
+
+  it('materializes a declared unique index, at every scope, over the columns VERBATIM', async () => {
+    // `'organization'` is NOT scoped up with a tenant key part here, and that is
+    // deliberate — the same call `FieldDef.unique` documents in the source.
+    // This driver implements no row-level tenancy and refuses to boot into a
+    // multi-tenant deployment (#3724), so a `(tenant, field)` index would
+    // advertise an isolation it does not deliver.
+    for (const scope of [true, 'global', 'organization'] as const) {
+      const { db, created } = fakeDb();
+      await syncCollectionSchema(db, 'lead', {
+        name: 'lead',
+        fields: { organization_id: { type: 'lookup' }, code: { type: 'text' } },
+        indexes: [{ fields: ['code'], unique: scope }],
+      });
+
+      const idx = byName(created, 'idx_code_unique');
+      expect(idx, String(scope)).toBeDefined();
+      expect(idx!.spec).toEqual({ code: 1 });
+      expect(idx!.options.unique).toBe(true);
+      expect(names(created)).not.toContain('idx_organization_id_code_unique');
+    }
+  });
+
+  it('a declared index and a field-level `unique` on the same column converge on ONE index', async () => {
+    // Why the generated name mirrors the field-level convention: two routes
+    // asking for the same constraint must land on the same name, or MongoDB
+    // sees two indexes over one key spec.
+    const { db, created } = fakeDb();
+    await syncCollectionSchema(db, 'lead', {
+      name: 'lead',
+      fields: { email: { type: 'email', unique: true } },
+      indexes: [{ fields: ['email'], unique: true }],
+    });
+
+    expect(names(created).filter((n) => n === 'idx_email_unique')).toHaveLength(2);
+    const [a, b] = created.filter((c) => c.options.name === 'idx_email_unique');
+    expect(a.spec).toEqual(b.spec);
+    expect(a.options).toEqual(b.options);
+  });
+
+  it('skips a declaration with no usable fields rather than emitting empty DDL', async () => {
+    const { db, created } = fakeDb();
+    await syncCollectionSchema(db, 'lead', {
+      name: 'lead',
+      fields: { code: { type: 'text' } },
+      indexes: [{ fields: [] }, { name: 'ghost' } as { name: string }],
+    });
+
+    expect(names(created)).toEqual(['idx_id_unique', 'idx_created_at', 'idx_updated_at']);
+  });
+});

packages/drivers/driver-mongodb/src/mongodb-schema.ts

@@ -33,18 +33,37 @@ interface FieldDef {
    * `SqlDriver.uniqueIndexesFromFields`.
    */
   unique?: boolean | 'global';
-  indexed?: boolean;
   required?: boolean;
   reference_to?: string;
   multiple?: boolean;
 }
 
+/**
+ * A declared object-level index — `IndexSchema` in @objectstack/spec, narrowed
+ * to what this driver materializes.
+ *
+ * [#6810] This is the ONE surface an index is declared on. The field-level
+ * `indexed` flag this driver used to read alongside it was never a
+ * `FieldSchema` key: #2377 / ADR-0049 removed it because a field-level index
+ * flag built no index, and `FieldSchema` — a `strictObject` — rejects it by
+ * name. The single producer still emitting it was the kernel's
+ * `organization_id` injection, which therefore stamped every registry-backed
+ * object with a document its own schema refused. That declaration moved to
+ * `indexes[]`; nothing else in the repo ever read the flag, so it is gone.
+ */
+interface IndexDef {
+  name?: string;
+  fields?: string[];
+  unique?: boolean | 'global' | 'organization';
+}
+
 /**
  * ObjectStack object definition (subset needed for schema sync).
  */
 interface ObjectDef {
   name: string;
   fields?: Record<string, FieldDef>;
+  indexes?: IndexDef[];
 }
 
 /**
@@ -53,8 +72,9 @@ interface ObjectDef {
  * - Creates the collection if it doesn't exist
  * - Creates a unique index on `id`
  * - Creates indexes on `created_at` and `updated_at`
- * - Creates indexes for fields marked `unique` or `indexed`
+ * - Creates indexes for fields marked `unique`
  * - Creates indexes on lookup (reference) fields
+ * - Creates the object's DECLARED `indexes[]` (#6810)
  */
 export async function syncCollectionSchema(
   db: Db,
@@ -84,11 +104,6 @@ export async function syncCollectionSchema(
           spec: { [fieldName]: 1 },
           options: { unique: true, sparse: true, name: `idx_${fieldName}_unique` },
         });
-      } else if (field.indexed) {
-        indexOps.push({
-          spec: { [fieldName]: 1 },
-          options: { name: `idx_${fieldName}` },
-        });
       }
 
       // Lookup + user (a lookup specialized to sys_user) fields get an index for
@@ -106,6 +121,37 @@ export async function syncCollectionSchema(
     }
   }
 
+  // Declared object-level indexes (#6810) — the surface `indexes[]`, which is
+  // where every other index in this system is declared and where the kernel now
+  // declares the tenant index on `organization_id`.
+  //
+  // The generated name is `idx_<fields>` / `idx_<fields>_unique`, matching the
+  // field-level convention above so the two routes converge on ONE index rather
+  // than racing to create two with different options on the same column set
+  // (Mongo index names are per-collection, so no table qualifier is needed —
+  // unlike `SqlDriver`'s `buildIndexName`, which is why neither driver takes a
+  // name from the declaration when the author left it out).
+  //
+  // Every `unique` scope materializes the columns VERBATIM, `'organization'`
+  // included. That is the same call `FieldDef.unique` documents above and for
+  // the same reason: this driver implements no row-level tenancy at all and
+  // refuses to boot into a multi-tenant deployment (#3724), so prepending a
+  // tenant key part would advertise an isolation it does not deliver.
+  for (const idx of schema.indexes ?? []) {
+    const fields = (idx.fields ?? []).filter((f) => typeof f === 'string' && f.length > 0);
+    if (fields.length === 0) continue;
+    const unique = Boolean(idx.unique);
+    const spec: Record<string, 1> = {};
+    for (const f of fields) spec[f] = 1;
+    indexOps.push({
+      spec: spec as IndexSpecification,
+      options: {
+        ...(unique ? { unique: true, sparse: true } : {}),
+        name: idx.name ?? `idx_${fields.join('_')}${unique ? '_unique' : ''}`,
+      },
+    });
+  }
+
   // Create indexes (idempotent — MongoDB ignores duplicates)
   for (const { spec, options } of indexOps) {
     try {