feat(objectql,spec): purge and empty the recycle bin, and delete a record for good (ADR-0148)

.changeset/30237-recycle-bin-purge-empty-permanent.md

@@ -0,0 +1,24 @@
+---
+'@objectstack/objectql': minor
+'@objectstack/spec': minor
+'@objectstack/runtime': minor
+---
+
+feat(objectql,spec,runtime): purge and empty the recycle bin, and delete a record for good (ADR-0148): `ObjectQL.purge(entryId)`, `ObjectQL.empty(object)` and `delete(object, { where: { id }, permanent: true })`, all authorized as the `purge` operation; hooks and app-authored bodies cannot pass `permanent`
+
+Clause-②: yes (widening)
+Metadata-protocol: added
+Skills-docs: follow-up #30847 — `skills/objectstack-api/SKILL.md` :268, :417 teach that there is no recycle bin; the skills lane owns that file.
+
+The recycle bin's removal side. All three acts are dispatched through the middleware chain as one new operation, `purge`, on the business object the rows belong to, so one grant will govern all three: `manageRecycleBin`, the recycle-bin manager's (ADR-0148 D5). **Until `@objectstack/plugin-security` maps `purge` to that grant, it refuses `purge` for every caller that is not the system**, the fail-closed rule it already applies to the destructive verbs. So in this release only the system, or a composition without the security plugin, can purge, empty or delete for good. Restoring, the two grants and the REST doors are not in this release.
+
+- **`@objectstack/spec`:** `EngineDeleteOptions` gains `permanent?: boolean`: delete for good, so the rows skip the recycle bin and cannot be restored. It is authorized as the `purge` operation, not `delete`, and is honoured on a by-id delete only.
+- **Hooks and app bodies cannot pass `permanent`:** app-authored code cannot erase a record past the recycle bin; its deletes always enter the bin, and the permanent delete stays with a principal holding `manageRecycleBin`. `HookDeleteOptions` (the bag `ctx.api.object(n).delete()` takes) omits `permanent`, and `@objectstack/runtime`'s sandbox refuses a hook, action or job body's `delete` carrying `permanent` (any value but `false`) with `400 VALIDATION_ERROR` (`param: 'permanent'`) before anything is removed, whatever envelope the body runs under, a job's system envelope included. `permanent: false` and an absent key are the ordinary delete.
+- **`@objectstack/objectql`, the permanent delete:** `delete(object, { where: { id }, permanent: true })` runs as the by-id delete does (its `beforeDelete` / `afterDelete` hooks, which read `permanent` off `input.options`, its cascade, its events), writes no recycle-bin entry, and is dispatched as `purge` with the row's id. These are refused `400 VALIDATION_ERROR` (`param: 'permanent'`) before any middleware or hook runs:
+  - `permanent: true` on a delete that is not by id, or that asks `multi: true`;
+  - `permanent: true` on a platform object (`sys_*`, `cloud_*`, `ai_*`), whose delete never enters the bin and is already permanent;
+  - a `permanent` that is not a boolean.
+  `permanent: false` is the ordinary delete.
+- **`@objectstack/objectql`, `purge(entryId, { context })`:** removes one recycle-bin entry and answers `true`. It answers `false` when no entry with that id is visible to the caller's organization, or when the composition registers no bin. The entry is read first, scoped to the caller's organization, for the object its rows belong to; the removal is then dispatched as `purge` on that object, with no id.
+- **`@objectstack/objectql`, `empty(object, { context })`:** removes every entry of one object that the caller's organization holds, and answers how many. It is one `purge` dispatch on `object`, so one authorization covers it. The object need not be registered any more, so an uninstalled package's entries can still be emptied. A platform object, or an empty name, is refused `400 VALIDATION_ERROR` (`param: 'object'`). It answers `0` when the composition registers no bin.
+- **Middleware authors:** `OperationContext.operation` gains `'purge'`, which never carries an `ast`. A middleware that switches on the operation exhaustively, or keys write handling on `'delete'`, sees the permanent delete as `'purge'`. `purge` advances the engine's write epoch, as `insert`, `update` and `delete` do (`WRITE_EPOCH_OPERATIONS`). `beforePurge` / `afterPurge` are not engine hook events: registering one warns that it never fires, as before.

content/docs/references/data/data-engine.mdx

@@ -656,6 +656,7 @@ QueryAST-aligned options for DataEngine.delete operations
 | **context** | `{ userId?: string; actor?: string; attributedUserId?: string; email?: string; … }` | optional |  |
 | **where** | `Record<string, any> \| any` | optional |  |
 | **multi** | `boolean` | optional (default: `false`) |  |
+| **permanent** | `boolean` | optional | Delete for good: the rows skip the recycle bin and cannot be restored. Authorized as the `purge` operation, not `delete`. By-id deletes only. |
 
 ### Nested Shape: `EngineDeleteOptions.context`
 

packages/objectql/src/engine-middleware-operation-vocabulary.test.ts

@@ -1,7 +1,8 @@
 // Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
 //
-// [#7809] The engine's MIDDLEWARE DISPATCH VOCABULARY is fixed, and it contains
-// no destructive lifecycle verb (`purge` / `transfer` / `restore`).
+// [#7809] The engine's MIDDLEWARE DISPATCH VOCABULARY is fixed. Of the
+// destructive lifecycle verbs (`purge` / `transfer` / `restore`) it contains
+// `purge` alone (#30237, the recycle bin), and `purge` never carries an AST.
 //
 // ── Why a pin here, in objectql, for a finding filed against plugin-security ──
 //
@@ -18,10 +19,14 @@
 // the same select-only object is scoped (#7665).
 //
 // The finding's own first question was whether those verbs can reach that path
-// at all. They cannot — not on the AST path, and not on ANY path:
+// at all. They cannot reach the AST path. `transfer` and `restore` reach no
+// path; `purge` reaches the by-id and object-level ones only:
 //
-//   1. `OperationContext['operation']` is the 7-member union pinned below.
-//      No destructive lifecycle verb is a member.
+//   1. `OperationContext['operation']` is the 8-member union pinned below.
+//      Of the destructive lifecycle verbs, `purge` alone is a member (#30237),
+//      and every dispatch of it is built without an `ast`: the permanent by-id
+//      delete (`permanent: true`, refused on any delete that is not by id),
+//      `purge(entryId)` and `empty(object)`.
 //   2. Middleware is invoked from exactly one place — the PRIVATE
 //      `executeWithMiddleware(ctx: OperationContext, …)`. There is no other
 //      caller, and no public seam that hands a middleware a context the engine
@@ -30,12 +35,13 @@
 //      `operation` is a member of that union by construction.
 //
 // So step 3 passing `opCtx.operation` raw is SAFE TODAY — not by accident, but
-// because the vocabulary cannot deliver a verb it fails to handle. This file
-// exists so that stays true by assertion rather than by luck: the day a recycle
-// bin (#3146, parked — `enable.trash` was retired in #2377/ADR-0049, which is
-// why `API_METHOD_DERIVATION` gives `restore`/`purge` a permanent
-// `flag: () => false`) makes one of these verbs dispatchable, this pin goes red
-// and names the two sites that must normalise first.
+// because the vocabulary cannot deliver a verb it fails to handle ON THAT PATH.
+// This file exists so that stays true by assertion rather than by luck. The
+// recycle bin (ADR-0148, #30237) made `purge` dispatchable, and kept it off the
+// AST path, which the AST-set lock below proves; the by-id permanent delete is
+// judged by step 2.7, which normalises `purge -> delete`. The day a destructive
+// verb reaches the AST path, this pin goes red and names the two sites that
+// must normalise first.
 //
 // ⚠️ Note the vocabularies are deliberately DIFFERENT and must not be conflated:
 // `ExplainOperationSchema` (spec/security/explain.zod.ts) is a 7-verb
@@ -119,6 +125,8 @@ const DISPATCHED_OPERATIONS = [
   'delete',
   'count',
   'aggregate',
+  // [#30237] The recycle bin's removal verb: the permanent delete, purge, empty.
+  'purge',
 ] as const;
 
 type Declared = (typeof DISPATCHED_OPERATIONS)[number];
@@ -180,11 +188,14 @@ function readDispatchedUnionFromEngineSource(): string[] {
 /**
  * The verbs #7809 is about. `plugin-security`'s step-2.7 gate normalises these
  * onto `update`/`delete` before deriving row scope; its step-3 AST path and the
- * ADR-0090 D10 delegator half do NOT. Keeping them out of the dispatch
- * vocabulary is what makes that difference unobservable.
+ * ADR-0090 D10 delegator half do NOT. Keeping them off the AST path is what
+ * makes that difference unobservable.
  */
 const DESTRUCTIVE_LIFECYCLE_VERBS = ['purge', 'transfer', 'restore'] as const;
 
+/** [#30237] The one destructive lifecycle verb the engine dispatches, never with an AST. */
+const DISPATCHED_DESTRUCTIVE_VERBS = ['purge'] as const;
+
 const NOTE_SCHEMA = {
   name: 'note',
   fields: {
@@ -202,6 +213,20 @@ const NOTE_SCHEMA = {
  */
 const ROW = { id: 'n1', title: 'x', owner: 'me' };
 
+/**
+ * [#30237] The recycle bin, registered so the probe reaches its three `purge`
+ * dispatches: an entry the driver always has, of a `note`.
+ */
+const BIN_SCHEMA = {
+  name: 'sys_recycle_bin',
+  fields: {
+    id: { type: 'text' },
+    object_name: { type: 'text' },
+    record_id: { type: 'text' },
+  },
+};
+const ENTRY = { id: 'e1', object_name: 'note', record_id: 'n1' };
+
 // `any` on the double itself, matching the sibling read-filter test: this is a
 // deliberately PARTIAL driver (the engine only reaches the members listed), and
 // annotating it `IDataDriver` would demand the whole surface for no assertion.
@@ -214,7 +239,7 @@ function makeDriver(): any {
     supports: {},
     connect: vi.fn().mockResolvedValue(undefined),
     disconnect: vi.fn().mockResolvedValue(undefined),
-    find: vi.fn(async () => [{ ...ROW }]),
+    find: vi.fn(async (object: string) => [object === 'sys_recycle_bin' ? { ...ENTRY } : { ...ROW }]),
     findOne: vi.fn(async () => ({ ...ROW })),
     count: vi.fn(async () => 1),
     aggregate: vi.fn(async () => []),
@@ -229,11 +254,13 @@ function makeDriver(): any {
 interface Seen {
   operation: string;
   hasAst: boolean;
+  /** A scalar `options.where.id`: the live row a by-id write names. */
+  hasId: boolean;
 }
 
 async function makeEngine() {
   vi.mocked((SchemaRegistry as any).getObject).mockImplementation((name: string) =>
-    name === 'note' ? NOTE_SCHEMA : undefined,
+    name === 'note' ? NOTE_SCHEMA : name === 'sys_recycle_bin' ? BIN_SCHEMA : undefined,
   );
   const ql = new ObjectQL();
   ql.registerDriver(makeDriver(), true);
@@ -244,7 +271,8 @@ async function makeEngine() {
   // precisely why that plugin can compare against verbs the union does not
   // contain without TypeScript ever objecting.
   ql.registerMiddleware(async (opCtx: any, next: () => Promise<void>) => {
-    seen.push({ operation: opCtx.operation, hasAst: !!opCtx.ast });
+    const id = opCtx.options?.where?.id;
+    seen.push({ operation: opCtx.operation, hasAst: !!opCtx.ast, hasId: typeof id === 'string' || typeof id === 'number' });
     await next();
   });
   return { ql, seen };
@@ -285,6 +313,10 @@ async function driveEveryPublicMethod(ql: ObjectQL): Promise<void> {
   await ql.update('note', { title: 'bulk' }, { multi: true, where: { owner: 'me' }, context: ctx } as any);
   await ql.delete('note', { where: { id: 'n1' }, context: ctx } as any);
   await ql.delete('note', { multi: true, where: { owner: 'me' }, context: ctx } as any);
+  // [#30237] The recycle bin's three `purge` dispatches, in this order.
+  await ql.delete('note', { where: { id: 'n1' }, permanent: true, context: ctx });
+  await ql.purge('e1', { context: ctx });
+  await ql.empty('note', { context: ctx });
 }
 
 describe('[#7809] engine middleware dispatch vocabulary', () => {
@@ -297,34 +329,56 @@ describe('[#7809] engine middleware dispatch vocabulary', () => {
     expect(fromSource.slice().sort()).toEqual([...DISPATCHED_OPERATIONS].sort());
   });
 
-  it('the engine union declares no destructive lifecycle verb', () => {
+  it('the engine union declares `purge` and no other destructive lifecycle verb', () => {
     const fromSource = readDispatchedUnionFromEngineSource();
     const destructive = fromSource.filter((op) =>
       (DESTRUCTIVE_LIFECYCLE_VERBS as readonly string[]).includes(op),
     );
     // Asserted against the SOURCE list, not against `DISPATCHED_OPERATIONS` —
     // checking this file's own array against this file's own array would prove
     // nothing.
-    expect(destructive).toEqual([]);
+    expect(destructive).toEqual([...DISPATCHED_DESTRUCTIVE_VERBS]);
   });
 
-  it('a real engine driven through every public method dispatches exactly the 7', async () => {
+  it('a real engine driven through every public method dispatches exactly the 8', async () => {
     const { ql, seen } = await makeEngine();
     await driveEveryPublicMethod(ql);
 
     const observed = [...new Set(seen.map((s) => s.operation))].sort();
     expect(observed).toEqual([...DISPATCHED_OPERATIONS].sort());
   });
 
-  it('no destructive lifecycle verb reaches a middleware, on any path', async () => {
+  it('of the destructive lifecycle verbs, only `purge` reaches a middleware, and never with an AST', async () => {
     const { ql, seen } = await makeEngine();
     await driveEveryPublicMethod(ql);
 
     expect(seen.length).toBeGreaterThan(0);
     const destructive = seen.filter((s) =>
       (DESTRUCTIVE_LIFECYCLE_VERBS as readonly string[]).includes(s.operation),
     );
-    expect(destructive).toEqual([]);
+    // [#30237] The permanent delete, `purge` and `empty`, in the order driven.
+    // Only the permanent delete names a live row: step 2.7 judges that row
+    // as a delete (`purge -> delete`); `purge` and `empty` are object-level.
+    expect(destructive).toEqual([
+      { operation: 'purge', hasAst: false, hasId: true },
+      { operation: 'purge', hasAst: false, hasId: false },
+      { operation: 'purge', hasAst: false, hasId: false },
+    ]);
+  });
+
+  it('[#30237] `permanent: true` off the by-id path is refused before any middleware runs', async () => {
+    const { ql, seen } = await makeEngine();
+    const ctx: ExecutionContext = { userId: 'u1', isSystem: true };
+
+    // A delete by filter: it would carry its `where` as an AST.
+    await expect(
+      ql.delete('note', { multi: true, where: { owner: 'me' }, permanent: true, context: ctx }),
+    ).rejects.toMatchObject({ code: 'VALIDATION_ERROR', status: 400 });
+    // A by-id delete asked as a predicate one: the bin never takes it.
+    await expect(
+      ql.delete('note', { multi: true, where: { id: 'n1' }, permanent: true, context: ctx }),
+    ).rejects.toMatchObject({ code: 'VALIDATION_ERROR', status: 400 });
+    expect(seen).toEqual([]);
   });
 
   it('every AST-carrying dispatch — the #7809 path — carries a handled operation', async () => {
@@ -355,10 +409,14 @@ describe('[#7809] engine middleware dispatch vocabulary', () => {
     const astOperations = [...new Set(astCarrying.map((s) => s.operation))].sort();
     expect(astOperations).toEqual(['aggregate', 'count', 'delete', 'find', 'findOne', 'update']);
 
-    // The by-id write branches, conversely, must NOT carry an ast.
+    // The by-id write branches, conversely, must NOT carry an ast; nor does
+    // any `purge` (#30237: the permanent delete, `purge`, `empty`).
     expect(seen.filter((s) => !s.hasAst).map((s) => s.operation).sort()).toEqual([
       'delete',
       'insert',
+      'purge',
+      'purge',
+      'purge',
       'update',
     ]);
   });

packages/objectql/src/engine-top-level.ts

@@ -137,34 +137,42 @@ const DISPATCHABLE_HOOK_EVENTS: ReadonlySet<string> = new Set([
  * [#17713] Every verb the engine can put on an {@link OperationContext}, as
  * DATA rather than as a second hand-written list.
  *
- * The type annotation is the weld: `Record<Union, true>` is exhaustive in both
- * directions, so a new engine method that widens `OperationContext['operation']`
+ * The type annotation is the weld: `Record<Union, boolean>` is exhaustive in
+ * both directions, so a new engine method that widens `OperationContext['operation']`
  * fails to compile here until the verb is added, and a verb dropped from the
- * union makes its entry an excess property. `pnpm typecheck` compiles this file
+ * union makes its entry an excess property. The value says whether the verb
+ * names a `before`/`after` hook pair. `pnpm typecheck` compiles this file
  * (it is `src`, not the test layer), so the weld is enforced by the package's
  * own type check and not only by a test.
  *
  * It exists so {@link UNDISPATCHED_LIFECYCLE_HOOK_EVENTS} below is DERIVED. A
  * hand-written list of "the events we refuse" would be this card's own defect
  * one layer up: it would go stale the moment a verb is added, and silently.
  */
-const ENGINE_OPERATION_VERBS: Record<OperationContext['operation'], true> = {
+const ENGINE_OPERATION_VERBS: Record<OperationContext['operation'], boolean> = {
   find: true,
   findOne: true,
   insert: true,
   update: true,
   delete: true,
   count: true,
   aggregate: true,
+  // [#30237] The recycle bin's removal verb names no hook pair. `beforePurge` /
+  // `afterPurge` stay outside the lifecycle namespace, where they were before
+  // the verb existed: inside it they would be refused at registration, a
+  // narrowing of `registerHook` this verb does not make. The permanent delete
+  // still fires `beforeDelete` / `afterDelete`.
+  purge: false,
 };
 
 /**
  * [#17713] The engine's own lifecycle-event NAMESPACE — `before`/`after` × every
- * verb in {@link ENGINE_OPERATION_VERBS}. Fourteen names, of which
- * {@link DISPATCHABLE_HOOK_EVENTS} is the eight the engine actually dispatches.
+ * verb in {@link ENGINE_OPERATION_VERBS} that names a hook pair. Fourteen names,
+ * of which {@link DISPATCHABLE_HOOK_EVENTS} is the eight the engine actually
+ * dispatches.
  */
 const ENGINE_LIFECYCLE_HOOK_EVENTS: ReadonlySet<string> = new Set(
-  Object.keys(ENGINE_OPERATION_VERBS).flatMap((verb) => {
+  Object.keys(ENGINE_OPERATION_VERBS).filter((verb) => ENGINE_OPERATION_VERBS[verb as OperationContext['operation']]).flatMap((verb) => {
     const suffix = `${verb.charAt(0).toUpperCase()}${verb.slice(1)}`;
     return [`before${suffix}`, `after${suffix}`];
   }),
@@ -321,7 +329,7 @@ const ENGINE_UPDATE_OPTION_KEYS: ReadonlySet<string> = new Set([
   ...ENGINE_DRIVER_PASSTHROUGH_KEYS,
 ]);
 const ENGINE_DELETE_OPTION_KEYS: ReadonlySet<string> = new Set([
-  'context', 'where', 'multi',
+  'context', 'where', 'multi', 'permanent',
   ...ENGINE_DRIVER_PASSTHROUGH_KEYS,
 ]);
 // `search` / `searchFields` are READ by `count` and `aggregate` through the
@@ -3659,7 +3667,16 @@ export interface FunctionRegistrationOptions {
  */
 export interface OperationContext {
   object: string;
-  operation: 'find' | 'findOne' | 'insert' | 'update' | 'delete' | 'count' | 'aggregate';
+  /**
+   * The verb the middleware chain authorizes. `purge` [#30237, ADR-0148] is the
+   * recycle bin's removal verb, dispatched on the business object the rows
+   * belong to: a by-id `delete` with `permanent: true` (its `where.id` the live
+   * row's), `purge(entryId)` and `empty(object)` (no id). It never carries an
+   * `ast`, so `@objectstack/plugin-security`'s AST path (step 3), which hands the
+   * verb to the RLS compiler raw, never sees it (#7809,
+   * `engine-middleware-operation-vocabulary.test.ts`).
+   */
+  operation: 'find' | 'findOne' | 'insert' | 'update' | 'delete' | 'count' | 'aggregate' | 'purge';
   ast?: QueryAST;
   data?: any;
   options?: any;