From f7488e935f642ea79596c948f0bdf21597240a5f Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Tue, 1 Sep 2026 12:17:18 +1000 Subject: [PATCH 01/11] fix(cli): make EQL reinstall dependency-safe --- .changeset/safe-eql-reinstall.md | 43 ++ ...ta-survives-disposable-schema-reinstall.md | 32 + .../2026-08-31-eql-safe-reinstall-design.md | 128 ++++ packages/cli/README.md | 16 +- packages/cli/src/__tests__/installer.test.ts | 275 +++++++- .../__tests__/reinstall.live.test.ts | 596 ++++++++++++++++++ packages/cli/src/installer/index.ts | 40 ++ packages/cli/src/installer/reinstall.ts | 558 ++++++++++++++++ .../tests/sqlx/tests/v3_uninstall_tests.rs | 197 +++++- skills/stash-cli/SKILL.md | 15 +- 10 files changed, 1873 insertions(+), 27 deletions(-) create mode 100644 .changeset/safe-eql-reinstall.md create mode 100644 docs/adr/0001-eql-data-survives-disposable-schema-reinstall.md create mode 100644 docs/superpowers/specs/2026-08-31-eql-safe-reinstall-design.md create mode 100644 packages/cli/src/installer/__tests__/reinstall.live.test.ts create mode 100644 packages/cli/src/installer/reinstall.ts diff --git a/.changeset/safe-eql-reinstall.md b/.changeset/safe-eql-reinstall.md new file mode 100644 index 000000000..bcea770c5 --- /dev/null +++ b/.changeset/safe-eql-reinstall.md @@ -0,0 +1,43 @@ +--- +"stash": patch +--- + +Preserve encrypted data and reconstruct functional indexes when reinstalling EQL v3, while refusing unsupported external dependencies before mutation. + +`stash eql install` and `stash eql upgrade` replace the disposable `eql_v3` and +`eql_v3_internal` schemas with `DROP SCHEMA … CASCADE`. Encrypted columns and +rows live outside those schemas and are never dropped, but anything depending on +EQL machinery goes with it. The installer now: + +- Takes a lifecycle lock, captures customer functional indexes that depend on + EQL, replaces the schemas, then rebuilds, analyzes and verifies those indexes + — all inside one transaction, so a rebuild failure restores the previous + installation rather than leaving a silently de-indexed database. +- Refuses before making any change when something it cannot reconstruct (a view, + a policy) depends on EQL, naming each object. +- Refuses before mutation when a functional index sits on a partitioned table. + Such an index cannot be reconstructed from its definition alone: the parent's + definition says `ON ONLY`, its per-partition children are separate objects, + and the parent stays invalid until every child is re-attached. Refusing names + the index instead of failing partway through a replacement. +- Classifies bundle-owned operators and casts independently of the session + `search_path`. Previously these were matched through `format_type()`, which + drops the schema qualification for types visible on the `search_path`, so a + connection with `eql_v3` on its path saw EQL's own operators as + customer-owned and refused on a healthy database. +- Waits up to five minutes for a concurrent EQL lifecycle operation instead of + blocking forever with no output, then fails with a message naming the cause + and the remedy. A queued install still waits and succeeds; only a lock nobody + will release now reports itself. +- Reports a bundle its expected-surface parser cannot model with the parser's + own message, naming the statement, instead of burying it in an install + failure that describes a rollback that never happened. + +Two limits worth knowing. Run a reinstall in a schema-migration maintenance +window: the advisory lock serializes cooperating `stash` commands, but an +ordinary PostgreSQL role cannot block unrelated sessions from creating or +dropping EQL-backed indexes, and an index created between capture and +replacement is dropped without being rebuilt. And these protections live in the +CLI, not in the SQL — a migration generated by `stash eql migration` is the raw +bundle, so re-applying one over a database that already carries EQL functional +indexes drops them with no rebuild. diff --git a/docs/adr/0001-eql-data-survives-disposable-schema-reinstall.md b/docs/adr/0001-eql-data-survives-disposable-schema-reinstall.md new file mode 100644 index 000000000..a9795de01 --- /dev/null +++ b/docs/adr/0001-eql-data-survives-disposable-schema-reinstall.md @@ -0,0 +1,32 @@ +--- +status: accepted +--- + +# Keep encrypted data durable and EQL schemas disposable + +EQL data-bearing domains live in `public` and must survive install, uninstall, +and reinstall, while the `eql_v3` and `eql_v3_internal` schemas remain disposable +and may be dropped with `CASCADE`. Search indexes are derived state: tooling must +capture, rebuild, and verify them around reinstall. Tooling must refuse before +mutation when it finds customer-owned dependencies such as policies, +constraints, or views that it cannot reconstruct safely. This follows the EQL +v2 persistence boundary and deliberately rejects brittle object-by-object +in-place upgrades and permanently versioned implementation schemas. + +Schema replacement, index reconstruction, and verification are one PostgreSQL +transaction. A failed reconstruction therefore restores the previous EQL +schemas and indexes instead of leaving a partially upgraded database. + +## Consequences + +- Losing an encrypted application column or stored encrypted value during any + EQL lifecycle operation is a correctness failure. +- Reinstall may incur an explicit, potentially expensive index rebuild. +- Index restoration failures are loud and actionable; they never degrade + silently to sequential scans. +- Changes that make an existing index definition invalid require operator + intervention rather than guessed migration semantics. +- Reinstall requires a schema-migration maintenance window: the advisory lock + serializes EQL lifecycle commands, but ordinary PostgreSQL roles cannot block + arbitrary application DDL globally. Do not create, alter, or drop EQL-backed + indexes while reinstall is running. diff --git a/docs/superpowers/specs/2026-08-31-eql-safe-reinstall-design.md b/docs/superpowers/specs/2026-08-31-eql-safe-reinstall-design.md new file mode 100644 index 000000000..3ce545ced --- /dev/null +++ b/docs/superpowers/specs/2026-08-31-eql-safe-reinstall-design.md @@ -0,0 +1,128 @@ +# EQL safe reinstall — durable data and reconstructed indexes + +Status: proposed +Date: 2026-08-31 +Issues: cipherstash/stack#959, cipherstash/stack#918 +ADR: `docs/adr/0001-eql-data-survives-disposable-schema-reinstall.md` + +## 1. Goal + +Make the existing drop-and-reinstall lifecycle safe without introducing +object-by-object upgrade scripts. Every encrypted application table, column, +domain type, and stored value must survive. Functional search indexes are +captured and rebuilt as derived state. Dependencies that cannot be reconstructed +mechanically stop the operation before the first destructive statement. + +## 2. Persistence boundary + +### Durable + +- Application tables and rows. +- Columns typed with any data-bearing `public.eql_v3_*` domain. +- The bytes stored in those columns. +- The data-bearing public domains themselves. + +### Disposable + +- `eql_v3` and `eql_v3_internal`. +- Query-operand domains, functions, operators, aggregates, and internal term + types owned by those schemas. + +### Reconstructable + +- Functional indexes whose complete definitions can be obtained with + `pg_get_indexdef()`. + +### Fail-closed + +- RLS policies, constraints, views, generated expressions, triggers, and any + other customer-owned object depending on disposable EQL machinery. +- Unknown dependency classes. + +## 3. EQL artifact requirements + +The installer and uninstaller may continue dropping the EQL-owned schemas with +`CASCADE`. They must never explicitly drop a `public.eql_v3_*` data-bearing +domain. Every data-bearing domain must be idempotently retained when it already +exists. + +The SQLx lifecycle suite must discover every installed data-bearing public EQL +domain from PostgreSQL's catalog. For each domain it must create an application +table, insert a real cipherstash-client-generated payload accepted by that +domain, uninstall, reinstall, and prove that the table, column type, row count, +and JSONB value are unchanged. + +## 4. CLI reinstall protocol + +`stash eql upgrade` and force-install use one protocol: + +The protocol runs inside a schema-migration maintenance window. Its advisory +lock serializes cooperating EQL lifecycle commands; it cannot serialize +arbitrary DDL issued by unrelated PostgreSQL sessions without superuser-only +event triggers. Application migrations must not run concurrently. + +1. Acquire an advisory lock preventing concurrent EQL lifecycle operations. +2. Discover every customer-owned object with a dependency path to + `eql_v3` or `eql_v3_internal`. +3. Partition dependencies into reconstructable functional indexes and + fail-closed objects. +4. If any fail-closed or unknown dependency exists, print an inventory and exit + before executing installer SQL. +5. Capture each index's identity and `pg_get_indexdef()` output, including + schema-qualified table and index names. +6. Begin one transaction, execute the shipped installer, and recreate captured + indexes before commit. Use the original definition by default; any + concurrent-rebuild mode must account explicitly for PostgreSQL's transaction + restrictions. +7. `ANALYZE` affected tables. +8. Verify every captured index exists, is valid and ready, and still has the + exact server-rendered definition captured before replacement. Query-level + engagement remains the responsibility of `stash eql validate`, which has + the application schema needed to construct representative predicates. +9. Commit only after verification, then release the advisory lock. + +If installer execution or index reconstruction fails, the transaction rolls +back to the previous schemas and indexes. The command exits non-zero and prints +the exact captured definition. It must never report a successful upgrade while +an index is absent or invalid. + +## 5. Dependency discovery + +Discovery follows `pg_depend` transitively from objects in the two disposable +schemas to customer-owned objects. It must not rely only on `pg_indexes`, because +that misses policies, views, constraints, generated expressions, and indirect +dependencies. + +The classifier is an allowlist: only ordinary functional indexes with a complete +server-rendered definition are automatically reconstructable. Every unrecognised +class is fail-closed. + +Uniqueness, predicates, included columns, tablespaces, +storage parameters, quoting, and non-`public` application schemas require test +coverage before their corresponding index form enters the allowlist. + +## 6. Acceptance criteria + +- The lifecycle test covers every installed data-bearing public EQL domain with + real encrypted fixtures and passes on every supported PostgreSQL version. +- Uninstall and reinstall preserve table OIDs, column identities, domain types, + row counts, and stored JSONB values. +- A reinstall with no external dependencies succeeds normally. +- A reinstall with supported functional indexes rebuilds and validates them. +- Unique, partial, expression, quoted-name, and non-public-schema + index cases are either proven safe or rejected before mutation. +- Partitioned indexes are rejected before mutation. Rebuilding their attachment + tree can exhaust PostgreSQL's default lock table inside the bundle transaction. +- A policy, constraint, view, generated column, trigger, or unknown dependency + aborts before schema drop and appears in the diagnostic inventory. +- Installer failure leaves the previous installation and indexes intact. +- Index reconstruction failure is non-zero, names the index and rolls back the + entire replacement. +- Re-running after a failed reconstruction is safe and deterministic. + +## 7. Explicit non-goals + +- Versioned object-by-object EQL upgrade scripts. +- Immutable per-release implementation schemas. +- Preserving functional index OIDs across reinstall. +- Automatically rewriting customer policies, constraints, or views. diff --git a/packages/cli/README.md b/packages/cli/README.md index 019601d54..22461ec7f 100644 --- a/packages/cli/README.md +++ b/packages/cli/README.md @@ -210,7 +210,19 @@ npx stash eql upgrade [options] | `--dry-run` | Show what would happen without making changes | | `--supabase` | Use Supabase-compatible upgrade | -The install SQL is idempotent and safe to re-run. If EQL is not installed, the command suggests running `npx stash eql install` instead. +The install SQL is safe to re-run: encrypted columns and rows live outside the +disposable EQL schemas and are never dropped. Before replacing those schemas, +the CLI takes a database lifecycle lock, captures dependent functional indexes, +then rebuilds and verifies their definitions in the same transaction. Any other external +dependency (for example a policy or view) is reported and the operation refuses +before changing the database. If index reconstruction fails, the transaction +restores the previous EQL schemas and indexes. It never reports a partially +indexed database as successfully upgraded. If EQL is not installed, +the command suggests running `npx stash eql install` instead. + +Run upgrade in a schema-migration maintenance window. Its advisory lock prevents +overlapping `stash` lifecycle commands, but unrelated sessions must not create, +alter, or drop EQL-backed indexes while replacement is running. --- @@ -305,6 +317,8 @@ Reads `databaseUrl` from `stash.config.ts`. Use `eql migration` to add the EQL v3 installation to your migration history instead of applying it directly. The install then ships to every environment through the same migrate step as the rest of your schema. +**The re-run protections are in the CLI, not in the emitted SQL.** `eql install` and `eql upgrade` take the lifecycle lock, capture dependent functional indexes, refuse on unsupported dependants, and rebuild afterwards — all of that lives in the stash installer. A generated migration is the raw bundle, so applying it through drizzle-kit, the Supabase CLI, or any other migration runner performs the `DROP SCHEMA ... CASCADE` with none of those steps. On a first install there is nothing to lose. Re-applying one over a database that already has EQL and functional indexes on EQL expressions drops those indexes and does not rebuild them; use `eql upgrade` for that, or recreate the indexes in the same migration. + ### Drizzle ```bash diff --git a/packages/cli/src/__tests__/installer.test.ts b/packages/cli/src/__tests__/installer.test.ts index 4b4191226..14fd17302 100644 --- a/packages/cli/src/__tests__/installer.test.ts +++ b/packages/cli/src/__tests__/installer.test.ts @@ -8,12 +8,43 @@ vi.mock('pg', () => ({ default: { Client: vi.fn(() => ({ connect: mockConnect, - query: mockQuery, + query: async (...args: unknown[]) => { + const result = await mockQuery(...args) + if ( + typeof args[0] === 'string' && + args[0].includes('pg_try_advisory_lock') && + result?.rows?.[0]?.acquired === undefined + ) { + return { ...result, rows: [{ acquired: true }] } + } + return result + }, end: mockEnd, })), }, })) +/** + * Lets one test make the bundle parser throw the way a bundle the parser has + * outgrown would ({@link assertEveryStatementModelled}). Through `vi.hoisted` + * because the factory below is hoisted above every other top-level binding; + * everything else keeps the real parse. + */ +const { parseFailure } = vi.hoisted(() => ({ + parseFailure: { error: null as Error | null }, +})) + +vi.mock('../installer/verify.js', async (importOriginal) => { + const actual = await importOriginal() + return { + ...actual, + parseExpectedSurface: (sql: string) => { + if (parseFailure.error) throw parseFailure.error + return actual.parseExpectedSurface(sql) + }, + } +}) + /** A full preflight row with every capability present. */ const CAPABLE_ROW = { role_name: 'postgres', @@ -30,7 +61,10 @@ const CAPABLE_ROW = { } describe('EQLInstaller', () => { - beforeEach(() => vi.clearAllMocks()) + beforeEach(() => { + vi.clearAllMocks() + parseFailure.error = null + }) afterEach(() => vi.restoreAllMocks()) it('reports a fully-capable superuser with no gaps', async () => { @@ -246,14 +280,245 @@ describe('EQLInstaller', () => { const sqlCall = mockQuery.mock.calls.find( ([sql]) => - typeof sql === 'string' && - !['BEGIN', 'COMMIT', 'ROLLBACK'].includes(sql), + typeof sql === 'string' && sql.includes('CREATE SCHEMA eql_v3'), ) expect(sqlCall?.[0]).toContain('eql_v3') expect(sqlCall?.[0]).not.toContain('CREATE SCHEMA eql_v2') expect(mockQuery).toHaveBeenCalledWith('COMMIT') }) + it('captures, rebuilds, and verifies functional indexes around reinstall', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + const indexDefinition = + 'CREATE INDEX users_email_idx ON app.users USING btree (eql_v3.eq_term(email))' + mockQuery.mockImplementation((sql: string) => { + if (sql.includes('stash_eql_lifecycle_dependencies')) { + return Promise.resolve({ + rows: [ + { + dependency_kind: 'index', + identity: 'app.users_email_idx', + definition: indexDefinition, + table_identity: 'app.users', + valid: true, + ready: true, + }, + ], + rowCount: 1, + }) + } + if (sql.includes('stash_eql_verify_rebuilt_indexes')) { + return Promise.resolve({ + rows: [ + { + identity: 'app.users_email_idx', + valid: true, + ready: true, + definition: indexDefinition, + }, + ], + rowCount: 1, + }) + } + return Promise.resolve({ rows: [], rowCount: 0 }) + }) + const { EQLInstaller } = await import('@/installer/index.ts') + const installer = new EQLInstaller({ databaseUrl: 'postgres://test' }) + + await installer.install() + + expect(mockQuery).toHaveBeenCalledWith( + 'SELECT pg_try_advisory_lock(hashtext($1)) AS acquired', + ['cipherstash.eql.lifecycle'], + ) + expect(mockQuery).toHaveBeenCalledWith('SET jit = off') + expect(mockQuery).toHaveBeenCalledWith(indexDefinition) + expect(mockQuery).toHaveBeenCalledWith('ANALYZE app.users') + const bundleCall = mockQuery.mock.calls.findIndex( + ([sql]) => + typeof sql === 'string' && sql.includes('CREATE SCHEMA eql_v3'), + ) + const rebuildCall = mockQuery.mock.calls.findIndex( + ([sql]) => sql === indexDefinition, + ) + expect(bundleCall).toBeGreaterThan(-1) + expect(rebuildCall).toBeGreaterThan(bundleCall) + expect(mockQuery).toHaveBeenCalledWith( + 'SELECT pg_advisory_unlock(hashtext($1))', + ['cipherstash.eql.lifecycle'], + ) + }) + + it('preserves the captured validity state when verifying rebuilt indexes', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + const indexDefinition = + 'CREATE INDEX users_email_idx ON app.users USING btree (eql_v3.eq_term(email))' + mockQuery.mockImplementation((sql: string) => { + if (sql.includes('stash_eql_lifecycle_dependencies')) { + return Promise.resolve({ + rows: [ + { + dependency_kind: 'index', + identity: 'app.users_email_idx', + definition: indexDefinition, + table_identity: 'app.users', + valid: false, + ready: false, + }, + ], + rowCount: 1, + }) + } + if (sql.includes('stash_eql_verify_rebuilt_indexes')) { + return Promise.resolve({ + rows: [ + { + identity: 'app.users_email_idx', + valid: false, + ready: false, + definition: indexDefinition, + }, + ], + rowCount: 1, + }) + } + return Promise.resolve({ rows: [], rowCount: 0 }) + }) + const { EQLInstaller } = await import('@/installer/index.ts') + + await expect( + new EQLInstaller({ databaseUrl: 'postgres://test' }).install(), + ).resolves.toEqual({ deferredGrantsSql: null }) + }) + + it('captures dependencies before the destructive install transaction', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + mockQuery.mockResolvedValue({ rows: [], rowCount: 0 }) + const { EQLInstaller } = await import('@/installer/index.ts') + + await new EQLInstaller({ databaseUrl: 'postgres://test' }).install() + + const statements = mockQuery.mock.calls.map(([sql]) => + typeof sql === 'string' ? sql : '', + ) + const begin = statements.indexOf('BEGIN') + const capture = statements.findIndex((sql) => + sql.includes('stash_eql_lifecycle_dependencies'), + ) + const bundle = statements.findIndex((sql) => + sql.includes('CREATE SCHEMA eql_v3'), + ) + expect(begin).toBeGreaterThan(-1) + expect(capture).toBeGreaterThan(-1) + expect(bundle).toBeGreaterThan(-1) + expect(capture).toBeLessThan(begin) + expect(begin).toBeLessThan(bundle) + }) + + it('reports a bundle the parser cannot model without a transaction narration', async () => { + parseFailure.error = new Error( + 'The EQL install SQL contains a statement the expected-surface parser does not model, at line 12: `CREATE PROCEDURE eql_v3.reindex()`.', + ) + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + mockQuery.mockResolvedValue({ rows: [], rowCount: 0 }) + const { EQLInstaller } = await import('@/installer/index.ts') + const installer = new EQLInstaller({ databaseUrl: 'postgres://test' }) + + const error: unknown = await installer.install().then( + () => null, + (thrown: unknown) => thrown, + ) + + expect(error).toBeInstanceOf(Error) + const message = (error as Error).message + // The parser's own message names the statement and the remedy. Wrapping it + // in the install's "nothing was applied / rolled back" narration buries + // that behind a database story about a transaction that never opened. + expect(message).toContain('does not model') + expect(message).toContain('CREATE PROCEDURE eql_v3.reindex()') + expect(message).not.toContain('Failed to install EQL') + expect(message).not.toContain('rolled back') + expect(mockQuery).not.toHaveBeenCalledWith('BEGIN') + // A bundle this CLI cannot read is a local defect, like a failed digest + // check: it must not reach the database at all. + expect(mockConnect).not.toHaveBeenCalled() + }) + + it('refuses before mutation when a dependency cannot be reconstructed', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + mockQuery.mockImplementation((sql: string) => { + if (sql.includes('stash_eql_lifecycle_dependencies')) { + return Promise.resolve({ + rows: [ + { + dependency_kind: 'unsafe', + identity: 'policy app.users_visible', + definition: null, + table_identity: null, + }, + ], + rowCount: 1, + }) + } + return Promise.resolve({ rows: [], rowCount: 0 }) + }) + const { EQLInstaller } = await import('@/installer/index.ts') + const installer = new EQLInstaller({ databaseUrl: 'postgres://test' }) + + await expect(installer.install()).rejects.toThrow( + /refused before making changes.*policy app\.users_visible/s, + ) + expect(mockQuery).not.toHaveBeenCalledWith('BEGIN') + expect( + mockQuery.mock.calls.some( + ([sql]) => + typeof sql === 'string' && sql.includes('CREATE SCHEMA eql_v3'), + ), + ).toBe(false) + }) + + it('rolls back schema replacement when index rebuild fails', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + const indexDefinition = + 'CREATE INDEX users_email_idx ON app.users (eql_v3.eq_term(email))' + mockQuery.mockImplementation((sql: string) => { + if (sql.includes('stash_eql_lifecycle_dependencies')) { + return Promise.resolve({ + rows: [ + { + dependency_kind: 'index', + identity: 'app.users_email_idx', + definition: indexDefinition, + table_identity: 'app.users', + valid: true, + ready: true, + }, + ], + rowCount: 1, + }) + } + if (sql === indexDefinition) { + return Promise.reject(new Error('disk full')) + } + return Promise.resolve({ rows: [], rowCount: 0 }) + }) + const { EQLInstaller } = await import('@/installer/index.ts') + + await expect( + new EQLInstaller({ databaseUrl: 'postgres://test' }).install(), + ).rejects.toThrow( + /transaction will restore.*Captured index SQL:\nCREATE INDEX users_email_idx/s, + ) + expect(mockQuery).toHaveBeenCalledWith('ROLLBACK') + expect(mockQuery).not.toHaveBeenCalledWith('COMMIT') + }) + it('grants both EQL v3 schemas to Supabase roles when the role is a member of postgres', async () => { mockConnect.mockResolvedValue(undefined) mockQuery.mockImplementation((sql: string) => { @@ -326,7 +591,7 @@ describe('EQLInstaller', () => { mockConnect.mockResolvedValue(undefined) mockEnd.mockResolvedValue(undefined) mockQuery.mockImplementation((sql: string) => { - if (!['BEGIN', 'COMMIT', 'ROLLBACK'].includes(sql)) { + if (sql.includes('CREATE SCHEMA eql_v3')) { return Promise.reject(new Error('permission denied')) } return Promise.resolve({ rows: [], rowCount: 0 }) diff --git a/packages/cli/src/installer/__tests__/reinstall.live.test.ts b/packages/cli/src/installer/__tests__/reinstall.live.test.ts new file mode 100644 index 000000000..5f921a816 --- /dev/null +++ b/packages/cli/src/installer/__tests__/reinstall.live.test.ts @@ -0,0 +1,596 @@ +/** + * Live-Postgres coverage for safe EQL schema replacement. + * + * The catalog dependency graph and pg_get_indexdef() are the public seam: a + * mock cannot prove PostgreSQL records an expression-index or policy dependency + * in the shape our classifier expects. Two of the checks below go further and + * are unreproducible anywhere else: `format_type()`'s search_path sensitivity + * and `ALTER INDEX … ATTACH PARTITION`'s effect on `indisvalid` are behaviours + * of the server, not properties of our SQL text. + */ + +import { afterAll, beforeEach, describe, expect, it } from 'vitest' +import { EQLInstaller, loadBundledEqlSql } from '../index.js' +import { + acquireLifecycleLock, + LIFECYCLE_DEPENDENCIES_SQL, + releaseLifecycleLock, +} from '../reinstall.js' +import { parseExpectedSurface } from '../verify.js' + +const DATABASE_URL = process.env.STASH_TEST_DATABASE_URL +const describeLive = DATABASE_URL ? describe : describe.skip + +async function queryOn( + url: string, + sql: string, + params: unknown[] = [], +): Promise { + const { default: pg } = await import('pg') + const client = new pg.Client({ connectionString: url }) + await client.connect() + try { + return (await client.query(sql, params)).rows as T[] + } finally { + await client.end().catch(() => undefined) + } +} + +async function query(sql: string): Promise { + return queryOn(DATABASE_URL ?? '', sql) +} + +/** + * The same database, reached on a connection whose `search_path` names the EQL + * schemas. Provisioned databases routinely carry this (`ALTER ROLE … SET + * search_path`) so applications can call `eq_term()` unqualified — and it is + * the one condition under which `format_type()` stops schema-qualifying EQL's + * own types. + */ +function withEqlSearchPath(url: string): string { + const parsed = new URL(url) + parsed.searchParams.set( + 'options', + '-c search_path=public,eql_v3,eql_v3_internal', + ) + return parsed.toString() +} + +/** + * Every index in the test schema, with the two things a partitioned rebuild + * can silently lose: its validity, and which partitioned index it is attached + * to. + */ +const INDEX_STATE_SQL = ` + SELECT pg_catalog.format('%I.%I', n.nspname, c.relname) AS identity, + c.relkind::text AS relkind, + i.indisvalid AS valid, + i.indisready AS ready, + pg_catalog.pg_get_indexdef(i.indexrelid) AS definition, + ( + SELECT pg_catalog.format('%I.%I', pn.nspname, pc.relname) + FROM pg_catalog.pg_inherits inh + JOIN pg_catalog.pg_class pc ON pc.oid = inh.inhparent + JOIN pg_catalog.pg_namespace pn ON pn.oid = pc.relnamespace + WHERE inh.inhrelid = c.oid + ) AS attached_to + FROM pg_catalog.pg_index i + JOIN pg_catalog.pg_class c ON c.oid = i.indexrelid + JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace + WHERE n.nspname = 'stash_reinstall_test' + ORDER BY identity +` + +describeLive('EQLInstaller safe reinstall — live Postgres', () => { + beforeEach(async () => { + await query('DROP EVENT TRIGGER IF EXISTS stash_reinstall_pause_drop') + await query('DROP SCHEMA IF EXISTS stash_reinstall_test CASCADE') + await new EQLInstaller({ databaseUrl: DATABASE_URL ?? '' }).install() + await query(` + CREATE SCHEMA IF NOT EXISTS stash_reinstall_test; + DROP TABLE IF EXISTS stash_reinstall_test.records CASCADE; + CREATE TABLE stash_reinstall_test.records ( + id integer PRIMARY KEY, + encrypted public.eql_v3_text_eq NOT NULL + ); + INSERT INTO stash_reinstall_test.records VALUES + (1, '{"v":3,"i":{},"c":"ciphertext","hm":"term"}'::jsonb); + `) + }, 180_000) + + afterAll(async () => { + await query('DROP SCHEMA IF EXISTS stash_reinstall_test CASCADE').catch( + () => undefined, + ) + }) + + it('preserves data and rebuilds a functional index', async () => { + await query(` + CREATE INDEX records_encrypted_idx + ON stash_reinstall_test.records (eql_v3.eq_term(encrypted)); + CREATE UNIQUE INDEX "Records encrypted complex" + ON stash_reinstall_test.records USING btree (eql_v3.eq_term(encrypted)) + INCLUDE (id) WITH (fillfactor = 80) WHERE id > 0; + `) + const identityBefore = await query<{ + table_oid: string + column_number: number + column_type_oid: string + value: unknown + }>(` + SELECT c.oid::text AS table_oid, + a.attnum AS column_number, + a.atttypid::text AS column_type_oid, + r.encrypted::jsonb AS value + FROM stash_reinstall_test.records r + JOIN pg_catalog.pg_class c + ON c.oid = 'stash_reinstall_test.records'::regclass + JOIN pg_catalog.pg_attribute a + ON a.attrelid = c.oid AND a.attname = 'encrypted' + WHERE r.id = 1 + `) + const definitionsBefore = await query<{ + identity: string + definition: string + }>(` + SELECT c.relname AS identity, pg_catalog.pg_get_indexdef(c.oid) AS definition + FROM pg_catalog.pg_class c + JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace + WHERE n.nspname = 'stash_reinstall_test' + AND c.relname IN ('records_encrypted_idx', 'Records encrypted complex') + ORDER BY c.relname + `) + await new EQLInstaller({ databaseUrl: DATABASE_URL ?? '' }).install() + + const rows = await query<{ + value: unknown + index_valid: boolean + index_ready: boolean + }>(` + SELECT r.encrypted::jsonb AS value, + i.indisvalid AS index_valid, + i.indisready AS index_ready + FROM stash_reinstall_test.records r + CROSS JOIN pg_catalog.pg_index i + WHERE r.id = 1 + AND i.indexrelid = 'stash_reinstall_test.records_encrypted_idx'::regclass + `) + expect(rows).toEqual([ + { + value: { v: 3, i: {}, c: 'ciphertext', hm: 'term' }, + index_valid: true, + index_ready: true, + }, + ]) + expect( + await query<{ + table_oid: string + column_number: number + column_type_oid: string + value: unknown + }>(` + SELECT c.oid::text AS table_oid, + a.attnum AS column_number, + a.atttypid::text AS column_type_oid, + r.encrypted::jsonb AS value + FROM stash_reinstall_test.records r + JOIN pg_catalog.pg_class c + ON c.oid = 'stash_reinstall_test.records'::regclass + JOIN pg_catalog.pg_attribute a + ON a.attrelid = c.oid AND a.attname = 'encrypted' + WHERE r.id = 1 + `), + ).toEqual(identityBefore) + expect( + await query<{ identity: string; definition: string }>(` + SELECT c.relname AS identity, pg_catalog.pg_get_indexdef(c.oid) AS definition + FROM pg_catalog.pg_class c + JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace + WHERE n.nspname = 'stash_reinstall_test' + AND c.relname IN ('records_encrypted_idx', 'Records encrypted complex') + ORDER BY c.relname + `), + ).toEqual(definitionsBefore) + expect( + await query<{ definition: string }>(` + SELECT pg_catalog.pg_get_indexdef( + 'stash_reinstall_test."Records encrypted complex"'::regclass + ) AS definition + `), + ).toEqual([ + { + definition: expect.stringMatching( + /CREATE UNIQUE INDEX.*INCLUDE \(id\).*fillfactor='80'.*WHERE \(id > 0\)/, + ), + }, + ]) + }, 180_000) + + /** + * `format_type()` omits the schema whenever the type is visible on the + * current search_path, while the identities parsed out of the bundle always + * carry the qualification the bundle wrote. Read the catalogue with + * `format_type()` and every bundle-owned operator and cast falls out of its + * ownership exemption the moment a connection names `eql_v3` — and the + * installer refuses on a healthy database, listing EQL's own operators as + * customer-owned objects. Only a live server shows this: the sensitivity is + * in `format_type()`, not in our SQL. + */ + it("exempts the bundle's own operators and casts when the EQL schemas are on the search_path", async () => { + const expected = parseExpectedSurface(loadBundledEqlSql()) + const rows = await queryOn<{ + dependency_kind: string + identity: string + }>(withEqlSearchPath(DATABASE_URL ?? ''), LIFECYCLE_DEPENDENCIES_SQL, [ + expected.operators, + expected.casts, + ]) + const unsafe = rows + .filter((row) => row.dependency_kind !== 'index') + .map((row) => row.identity) + + // Reported as a count plus a sample: the pre-fix failure is ~600 rows, and + // a bare array comparison buries the count that identifies the bug. + expect({ count: unsafe.length, sample: unsafe.slice(0, 3) }).toEqual({ + count: 0, + sample: [], + }) + }, 60_000) + + /** + * The negative control for the test above. An empty result proves nothing on + * its own — it is also what an EMPTY dependency graph looks like. Withhold + * exactly the operators the bundle declares with a `pg_catalog` operand + * (`text`, `text[]`, `jsonb`, `jsonpath`, `integer` — the parser spells those + * bare, with no schema) and they must reappear as `unsafe`. That is what + * pins the other half of the rule: qualifying every operand unconditionally + * would spell these `pg_catalog.text` and silently break their exemption in + * the direction this test, not the one above, can see. + */ + it('exempts operators with a bare pg_catalog operand, and only because of the match', async () => { + const expected = parseExpectedSurface(loadBundledEqlSql()) + const hasBareOperand = (identity: string) => + identity + .slice(identity.indexOf('(') + 1, -1) + .split(', ') + .some((operand) => operand !== 'none' && !operand.includes('.')) + const withheld = expected.operators.filter(hasBareOperand) + expect(withheld.length).toBeGreaterThan(0) + + const rows = await queryOn<{ dependency_kind: string; identity: string }>( + withEqlSearchPath(DATABASE_URL ?? ''), + LIFECYCLE_DEPENDENCIES_SQL, + [expected.operators.filter((o) => !hasBareOperand(o)), expected.casts], + ) + + expect(rows.filter((row) => row.dependency_kind !== 'index')).toHaveLength( + withheld.length, + ) + }, 60_000) + + it('installs over a connection whose search_path names the EQL schemas', async () => { + await query(` + CREATE INDEX records_encrypted_idx + ON stash_reinstall_test.records (eql_v3.eq_term(encrypted)); + `) + await expect( + new EQLInstaller({ + databaseUrl: withEqlSearchPath(DATABASE_URL ?? ''), + }).install(), + ).resolves.toEqual({ deferredGrantsSql: null }) + }, 180_000) + + /** + * A partitioned index is `relkind = 'I'`, its per-partition children are + * separate `relkind = 'i'` rows, and the parent only becomes `indisvalid` + * once every child has been ATTACHed. Recreating the captured definitions + * and stopping there leaves the parent invalid forever. + * + * Two levels deep on purpose: an intermediate partitioned index is both a + * parent and a child. `INDEX_STATE_SQL` proves refusal preserves every name, + * definition, validity flag and attachment rather than partially rebuilding + * the tree. + */ + it('refuses a partitioned index tree before mutation', async () => { + await query(` + CREATE TABLE stash_reinstall_test.partitioned_records ( + id integer, encrypted public.eql_v3_text_eq NOT NULL + ) PARTITION BY RANGE (id); + CREATE TABLE stash_reinstall_test.partitioned_records_a + PARTITION OF stash_reinstall_test.partitioned_records + FOR VALUES FROM (0) TO (100) PARTITION BY RANGE (id); + CREATE TABLE stash_reinstall_test.partitioned_records_a1 + PARTITION OF stash_reinstall_test.partitioned_records_a + FOR VALUES FROM (0) TO (50); + CREATE TABLE stash_reinstall_test.partitioned_records_b + PARTITION OF stash_reinstall_test.partitioned_records + FOR VALUES FROM (100) TO (200); + CREATE INDEX partitioned_encrypted_idx + ON stash_reinstall_test.partitioned_records (eql_v3.eq_term(encrypted)); + INSERT INTO stash_reinstall_test.partitioned_records VALUES + (1, '{"v":3,"i":{},"c":"ciphertext","hm":"term"}'::jsonb), + (150, '{"v":3,"i":{},"c":"ciphertext","hm":"term"}'::jsonb); + `) + const before = await query>(INDEX_STATE_SQL) + // The fixture only means anything if Postgres really built the tree. + expect( + before.filter((row) => row.relkind === 'I').map((row) => row.identity), + ).toEqual([ + 'stash_reinstall_test.partitioned_encrypted_idx', + 'stash_reinstall_test.partitioned_records_a_eq_term_idx', + ]) + + await expect( + new EQLInstaller({ databaseUrl: DATABASE_URL ?? '' }).install(), + ).rejects.toThrow(/reinstall refused.*partitioned_encrypted_idx/is) + + expect(await query>(INDEX_STATE_SQL)).toEqual( + before, + ) + expect( + await query<{ count: number }>( + 'SELECT count(*)::int AS count FROM stash_reinstall_test.partitioned_records', + ), + ).toEqual([{ count: 2 }]) + }, 180_000) + + /** + * `pg_advisory_lock` waits forever. A concurrent install, or a session that + * died holding the lock, then makes the command hang with no output — + * indistinguishable from a network stall. + */ + it('refuses instead of hanging when another session holds the lifecycle lock', async () => { + const { default: pg } = await import('pg') + const holder = new pg.Client({ connectionString: DATABASE_URL }) + const blocked = new pg.Client({ connectionString: DATABASE_URL }) + await holder.connect() + await blocked.connect() + try { + // A regression here is a HANG, not a failure — a blocking + // `pg_advisory_lock` never returns, the `finally` below never runs, and + // the leaked lock then blocks every later `install()` in this file. The + // timeout turns that into a failed assertion. It is inert once the + // acquire polls with `pg_try_advisory_lock`, which never waits. + await blocked.query("SET statement_timeout = '10s'") + await holder.query( + "SELECT pg_advisory_lock(hashtext('cipherstash.eql.lifecycle'))", + ) + + // An explicit short budget: the behaviour under test is "refuses rather + // than hangs", which does not depend on how long the wait is, and the + // production default is deliberately a minute. The default's own + // property — that an ordinary queued install still succeeds — is the + // next test. + await expect(acquireLifecycleLock(blocked, 1_000)).rejects.toThrow( + /another EQL lifecycle operation is in progress/i, + ) + // The failed acquire holds nothing, so the caller's unconditional + // release in `finally` has to be a no-op rather than an unlock of a lock + // this session never took. + await expect(releaseLifecycleLock(blocked)).resolves.toBeUndefined() + + await holder.query('SELECT pg_advisory_unlock_all()') + await expect(acquireLifecycleLock(blocked)).resolves.toBeUndefined() + await releaseLifecycleLock(blocked) + } finally { + await holder.query('SELECT pg_advisory_unlock_all()').catch(() => {}) + await blocked.end().catch(() => undefined) + await holder.end().catch(() => undefined) + } + }, 30_000) + + /** + * The other half of the lock change, and the one that bites. Bounding the + * wait converts "hangs forever" into a message — but bound it too tightly and + * ordinary queueing becomes a failure, because what holds the lock is a whole + * install (`DROP SCHEMA … CASCADE` plus ~3,000 objects, 10-30s here). A + * five-second budget looked generous and broke exactly this: two installs + * back to back, the second refused. The waiter must still win. + */ + it('queues a second concurrent install instead of refusing it', async () => { + const results = await Promise.all([ + new EQLInstaller({ databaseUrl: DATABASE_URL ?? '' }).install(), + new EQLInstaller({ databaseUrl: DATABASE_URL ?? '' }).install(), + ]) + expect(results).toEqual([ + { deferredGrantsSql: null }, + { deferredGrantsSql: null }, + ]) + }, 180_000) + + it('restores schemas, data, column identity, and indexes when install fails after DROP SCHEMA', async () => { + await query(`CREATE INDEX records_encrypted_idx + ON stash_reinstall_test.records (eql_v3.eq_term(encrypted))`) + const before = await query<{ + version: string + table_oid: string + column_number: number + column_type_oid: string + value: unknown + index_definition: string + }>(` + SELECT eql_v3.version() AS version, + c.oid::text AS table_oid, + a.attnum AS column_number, + a.atttypid::text AS column_type_oid, + r.encrypted::jsonb AS value, + pg_catalog.pg_get_indexdef('stash_reinstall_test.records_encrypted_idx'::regclass) + AS index_definition + FROM stash_reinstall_test.records r + JOIN pg_catalog.pg_class c ON c.oid = 'stash_reinstall_test.records'::regclass + JOIN pg_catalog.pg_attribute a + ON a.attrelid = c.oid AND a.attname = 'encrypted' + WHERE r.id = 1 + `) + await query(` + CREATE FUNCTION stash_reinstall_test.reject_schema_create() + RETURNS event_trigger LANGUAGE plpgsql AS + 'BEGIN RAISE EXCEPTION ''forced installer failure after DROP SCHEMA''; END'; + CREATE EVENT TRIGGER stash_reinstall_reject_create + ON ddl_command_start WHEN TAG IN ('CREATE SCHEMA') + EXECUTE FUNCTION stash_reinstall_test.reject_schema_create(); + `) + try { + await expect( + new EQLInstaller({ databaseUrl: DATABASE_URL ?? '' }).install(), + ).rejects.toThrow(/forced installer failure after DROP SCHEMA/) + } finally { + await query('DROP EVENT TRIGGER IF EXISTS stash_reinstall_reject_create') + } + expect( + await query<{ + version: string + table_oid: string + column_number: number + column_type_oid: string + value: unknown + index_definition: string + }>(` + SELECT eql_v3.version() AS version, + c.oid::text AS table_oid, + a.attnum AS column_number, + a.atttypid::text AS column_type_oid, + r.encrypted::jsonb AS value, + pg_catalog.pg_get_indexdef('stash_reinstall_test.records_encrypted_idx'::regclass) + AS index_definition + FROM stash_reinstall_test.records r + JOIN pg_catalog.pg_class c ON c.oid = 'stash_reinstall_test.records'::regclass + JOIN pg_catalog.pg_attribute a + ON a.attrelid = c.oid AND a.attname = 'encrypted' + WHERE r.id = 1 + `), + ).toEqual(before) + }, 180_000) + + it.each([ + { + name: 'RLS policy', + identity: 'encrypted_visible', + setup: ` + ALTER TABLE stash_reinstall_test.records ENABLE ROW LEVEL SECURITY; + CREATE POLICY encrypted_visible ON stash_reinstall_test.records + USING (eql_v3.eq_term(encrypted) IS NOT NULL); + `, + remains: `SELECT count(*)::int AS count FROM pg_catalog.pg_policy WHERE polname = 'encrypted_visible'`, + }, + { + name: 'view', + identity: 'encrypted_terms', + setup: `CREATE VIEW stash_reinstall_test.encrypted_terms AS + SELECT eql_v3.eq_term(encrypted) AS term FROM stash_reinstall_test.records`, + remains: `SELECT count(*)::int AS count FROM pg_catalog.pg_views + WHERE schemaname = 'stash_reinstall_test' AND viewname = 'encrypted_terms'`, + }, + { + name: 'check constraint', + identity: 'encrypted_has_term', + setup: `ALTER TABLE stash_reinstall_test.records + ADD CONSTRAINT encrypted_has_term CHECK (eql_v3.eq_term(encrypted) IS NOT NULL)`, + remains: `SELECT count(*)::int AS count FROM pg_catalog.pg_constraint + WHERE conname = 'encrypted_has_term'`, + }, + { + name: 'generated column', + identity: 'generated_term', + setup: `ALTER TABLE stash_reinstall_test.records ADD COLUMN generated_term text + GENERATED ALWAYS AS (eql_v3.eq_term(encrypted)::text) STORED`, + remains: `SELECT count(*)::int AS count FROM pg_catalog.pg_attribute + WHERE attrelid = 'stash_reinstall_test.records'::regclass + AND attname = 'generated_term' AND NOT attisdropped`, + }, + { + name: 'trigger predicate', + identity: 'encrypted_trigger', + setup: ` + CREATE FUNCTION stash_reinstall_test.noop_trigger() RETURNS trigger + LANGUAGE plpgsql AS 'BEGIN RETURN NEW; END'; + CREATE TRIGGER encrypted_trigger BEFORE UPDATE ON stash_reinstall_test.records + FOR EACH ROW WHEN (eql_v3.eq_term(NEW.encrypted) IS NOT NULL) + EXECUTE FUNCTION stash_reinstall_test.noop_trigger(); + `, + remains: `SELECT count(*)::int AS count FROM pg_catalog.pg_trigger + WHERE tgname = 'encrypted_trigger'`, + }, + { + name: 'customer operator', + identity: '===', + setup: `CREATE OPERATOR stash_reinstall_test.=== ( + FUNCTION = eql_v3.eq, + LEFTARG = public.eql_v3_text_eq, + RIGHTARG = public.eql_v3_text_eq + )`, + remains: `SELECT count(*)::int AS count FROM pg_catalog.pg_operator o + JOIN pg_catalog.pg_namespace n ON n.oid = o.oprnamespace + WHERE n.nspname = 'stash_reinstall_test' AND o.oprname = '==='`, + }, + { + name: 'operator duplicating a bundle signature in another schema', + identity: 'stash_reinstall_test.=(', + setup: `CREATE OPERATOR stash_reinstall_test.= ( + FUNCTION = eql_v3.eq, + LEFTARG = public.eql_v3_text_eq, + RIGHTARG = public.eql_v3_text_eq + )`, + remains: `SELECT count(*)::int AS count FROM pg_catalog.pg_operator o + JOIN pg_catalog.pg_namespace n ON n.oid = o.oprnamespace + WHERE n.nspname = 'stash_reinstall_test' AND o.oprname = '=' + AND o.oprleft = 'public.eql_v3_text_eq'::regtype + AND o.oprright = 'public.eql_v3_text_eq'::regtype`, + }, + ])( + 'refuses before mutation for a customer-owned $name', + async ({ setup, identity, remains }) => { + await query(setup) + const versionBefore = await query<{ version: string }>( + 'SELECT eql_v3.version() AS version', + ) + + await expect( + new EQLInstaller({ databaseUrl: DATABASE_URL ?? '' }).install(), + ).rejects.toThrow( + new RegExp( + `refused before making changes.*${identity.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}`, + 's', + ), + ) + + expect( + await query<{ version: string }>('SELECT eql_v3.version() AS version'), + ).toEqual(versionBefore) + expect(await query<{ count: number }>(remains)).toEqual([{ count: 1 }]) + }, + 180_000, + ) + + it('names every view in a dependent chain, not the rewrite rule', async () => { + // Only chain_1 references EQL. chain_2 and chain_3 reach it through the + // view above them, and CASCADE takes all three -- but a view's dependency + // on an EQL function is recorded against its _RETURN rule, and nothing in + // pg_depend leads back out of a rule to its view. Before the walk carried + // that edge, the refusal named `"_RETURN" on ...chain_1` and mentioned no + // view at all, understating what was about to be destroyed by two. + await query(`CREATE VIEW stash_reinstall_test.chain_1 AS + SELECT eql_v3.eq_term(encrypted) AS term FROM stash_reinstall_test.records`) + await query( + 'CREATE VIEW stash_reinstall_test.chain_2 AS SELECT term FROM stash_reinstall_test.chain_1', + ) + await query( + 'CREATE VIEW stash_reinstall_test.chain_3 AS SELECT term FROM stash_reinstall_test.chain_2', + ) + + const failure = await new EQLInstaller({ databaseUrl: DATABASE_URL ?? '' }) + .install() + .then(() => null) + .catch((error: unknown) => + error instanceof Error ? error.message : String(error), + ) + + expect(failure).toMatch(/refused before making changes/) + expect(failure).toContain('stash_reinstall_test.chain_1') + expect(failure).toContain('stash_reinstall_test.chain_2') + expect(failure).toContain('stash_reinstall_test.chain_3') + // The rule and the row/array types name the same casualties the views do. + expect(failure).not.toContain('_RETURN') + expect(failure).not.toContain('chain_1[]') + }, 180_000) +}) diff --git a/packages/cli/src/installer/index.ts b/packages/cli/src/installer/index.ts index 21ac65440..737ecba33 100644 --- a/packages/cli/src/installer/index.ts +++ b/packages/cli/src/installer/index.ts @@ -10,6 +10,12 @@ import { SUPABASE_IMMEDIATE_GRANTS_SQL_V3, SUPABASE_PERMISSIONS_SQL_V3, } from './grants.js' +import { + acquireLifecycleLock, + inspectReinstallDependencies, + rebuildIndexes, + releaseLifecycleLock, +} from './reinstall.js' export { DEFERRED_GRANTS_HEADER, @@ -432,6 +438,18 @@ export class EQLInstaller { // whole error, rather than a `detail` interpolated into the install // wrapper's transaction narration below. const bundledSql = loadBundledEqlSql() + // Parsed here for the same reason, and deliberately outside the try below: + // a parse failure is a bundle this CLI has outgrown, and the parser's + // message names the offending statement. Interpolated into the install + // wrapper it would be narrated as a rolled-back transaction that was never + // opened, pointing at the database instead of at the bundle. + // + // `parseExpectedSurface(bundledSql)` rather than `bundledExpectedSurface()`, + // which would re-read and re-digest the bundle we already hold. Dynamic to + // avoid verify.ts's intentional import of this module for the + // digest-checked bundle loader. + const { parseExpectedSurface } = await import('./verify.js') + const expected = parseExpectedSurface(bundledSql) const client = createPgClient(this.databaseUrl) try { await client.connect() @@ -444,13 +462,34 @@ export class EQLInstaller { } try { + await acquireLifecycleLock(client) try { + // This catalogue query has a very high estimated cost because its + // recursive walk carries every dependency edge. PostgreSQL otherwise + // JIT-compiles hundreds of tiny expressions (~880ms locally) for work + // that executes in ~35ms interpreted. This connection belongs solely + // to this install and is closed below, so the session setting cannot + // leak into application queries. + await client.query('SET jit = off') + const indexes = await inspectReinstallDependencies( + client, + expected.operators, + expected.casts, + ) + // Keep catalogue discovery outside the DDL transaction. The EQL + // bundle creates thousands of objects and already approaches + // max_locks_per_transaction; retaining the discovery query's catalogue + // locks across it can exhaust PostgreSQL's shared lock table. The + // documented maintenance window excludes unrelated application DDL + // between discovery and replacement. await client.query('BEGIN') await client.query(bundledSql) + await rebuildIndexes(client, indexes) await client.query('COMMIT') } catch (error) { await client.query('ROLLBACK').catch(() => {}) const detail = error instanceof Error ? error.message : String(error) + if (detail.startsWith('EQL reinstall refused')) throw error throw new Error( `Failed to install EQL: ${detail}. Nothing was applied — the install runs in a transaction and was rolled back.`, { cause: error }, @@ -469,6 +508,7 @@ export class EQLInstaller { ) } } finally { + await releaseLifecycleLock(client).catch(() => {}) await client.end() } } diff --git a/packages/cli/src/installer/reinstall.ts b/packages/cli/src/installer/reinstall.ts new file mode 100644 index 000000000..3bbc2c353 --- /dev/null +++ b/packages/cli/src/installer/reinstall.ts @@ -0,0 +1,558 @@ +import type pg from 'pg' + +const LIFECYCLE_LOCK = 'cipherstash.eql.lifecycle' + +/** + * How long to keep trying for the lifecycle lock before giving up. + * + * Sized off what actually holds it: the whole install, which is a + * `DROP SCHEMA … CASCADE` plus ~3,000 object creations and measures 10-30s + * against a local container — longer on a managed Postgres. A budget of a few + * seconds looks generous and is not; it converts ordinary queueing (two + * installs, a CI job overlapping a developer) into a hard failure on the second + * one. That regression is real and was caught by the live suite: with 5s, two + * live files installing back to back failed each other. + * + * So: long enough that anything a waiter would sensibly wait for still + * succeeds, and bounded so a lock nobody will release reports itself instead of + * looking like a stalled connection. + * + * The budget is sized off the INSTALL, which is what actually holds the lock: + * `DROP SCHEMA ... CASCADE` plus ~3,000 object creations, 10-30s against a + * local container and longer on managed Postgres. The dependency capture + * ({@link LIFECYCLE_DEPENDENCIES_SQL}) also runs under the lock but is not the + * cost. Its recursive shape inflates PostgreSQL's estimate enough to trigger + * JIT compilation of hundreds of expressions: measured on an idle local + * container with EQL installed, the query itself is ~35ms with JIT disabled + * versus ~880ms with JIT enabled. `install()` disables JIT on its dedicated + * connection before running it. + * + * Five minutes is deliberate margin over that, not a measurement: a waiter + * should never be refused for ordinary queueing, only for a holder that will + * never release. Earlier revisions of this comment justified the number with + * capture timings of 17-68s. Those were measured on a server concurrently + * running the test suite and are contention, not query cost; do not reinstate + * them as evidence. + */ +const LOCK_WAIT_MS = 300_000 +const LOCK_RETRY_INTERVAL_MS = 250 + +/** Connections currently holding the lifecycle lock through this module. */ +const lockHolders = new WeakSet() + +export interface ReinstallIndex { + identity: string + definition: string + tableIdentity: string + valid: boolean + ready: boolean +} + +interface DependencyRow { + dependency_kind?: unknown + identity?: unknown + definition?: unknown + table_identity?: unknown + valid?: unknown + ready?: unknown +} + +export const LIFECYCLE_DEPENDENCIES_SQL = ` +/* stash_eql_lifecycle_dependencies */ +WITH RECURSIVE +/* + * A type's name spelled the way \`parseExpectedSurface\` spells it, so the + * operator and cast identities below can be compared against the ones parsed + * out of the bundle. + * + * The rule is NOT \`format_type()\`. That function omits the schema whenever the + * type is visible on the current search_path, while the parser always emits the + * qualification the bundle wrote (\`eql_v3.query_text_eq\`, + * \`eql_v3_internal.ore_block_256\`). On a connection whose search_path names + * eql_v3 — routine on a provisioned database, so an application can call + * \`eq_term()\` unqualified — \`format_type()\` answers a bare \`query_text_eq\`, + * every bundle-owned operator and cast misses its ownership exemption, and the + * installer refuses on a healthy database listing EQL's own operators as + * customer-owned. (#918) + * + * It is not unconditional qualification either: the bundle declares operands in + * \`pg_catalog\` (\`text\`, \`text[]\`, \`jsonb\`, \`jsonpath\`, \`integer\`) and writes + * them bare, so \`pg_catalog.text\` would miss in the other direction. Hence: + * \`format_type()\` for pg_catalog — it also spells the SQL-standard multi-word + * names (\`double precision\`) and array suffixes the parser's alias map targets + * — and an explicit qualification for everything else. + */ +type_identity(oid, identity) AS ( + SELECT t.oid, + CASE + WHEN tn.nspname = 'pg_catalog' + THEN pg_catalog.format_type(t.oid, NULL) + -- An array of a non-catalog type: pg_type calls it \`_eql_v3_text_eq\`, + -- the bundle would write \`public.eql_v3_text_eq[]\`. + WHEN t.typcategory = 'A' AND et.oid IS NOT NULL + THEN pg_catalog.format('%I.%I[]', en.nspname, et.typname) + ELSE pg_catalog.format('%I.%I', tn.nspname, t.typname) + END + FROM pg_catalog.pg_type t + JOIN pg_catalog.pg_namespace tn ON tn.oid = t.typnamespace + LEFT JOIN pg_catalog.pg_type et + ON et.oid = t.typelem AND t.typcategory = 'A' + LEFT JOIN pg_catalog.pg_namespace en ON en.oid = et.typnamespace +), +eql_roots(classid, objid, objsubid) AS ( + SELECT 'pg_catalog.pg_namespace'::regclass, n.oid, 0 + FROM pg_catalog.pg_namespace n + WHERE n.nspname IN ('eql_v3', 'eql_v3_internal') + UNION ALL + SELECT 'pg_catalog.pg_proc'::regclass, p.oid, 0 + FROM pg_catalog.pg_proc p + JOIN pg_catalog.pg_namespace n ON n.oid = p.pronamespace + WHERE n.nspname IN ('eql_v3', 'eql_v3_internal') + UNION ALL + SELECT 'pg_catalog.pg_type'::regclass, t.oid, 0 + FROM pg_catalog.pg_type t + JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace + WHERE n.nspname IN ('eql_v3', 'eql_v3_internal') + UNION ALL + SELECT 'pg_catalog.pg_operator'::regclass, o.oid, 0 + FROM pg_catalog.pg_operator o + JOIN pg_catalog.pg_namespace n ON n.oid = o.oprnamespace + WHERE n.nspname IN ('eql_v3', 'eql_v3_internal') + UNION ALL + SELECT 'pg_catalog.pg_opclass'::regclass, o.oid, 0 + FROM pg_catalog.pg_opclass o + JOIN pg_catalog.pg_namespace n ON n.oid = o.opcnamespace + WHERE n.nspname IN ('eql_v3', 'eql_v3_internal') + UNION ALL + SELECT 'pg_catalog.pg_opfamily'::regclass, o.oid, 0 + FROM pg_catalog.pg_opfamily o + JOIN pg_catalog.pg_namespace n ON n.oid = o.opfnamespace + WHERE n.nspname IN ('eql_v3', 'eql_v3_internal') +), +/* + * Everything reachable from the bundle, transitively. The lateral is one + * recursive self-reference carrying two kinds of edge, because PostgreSQL + * permits only one. + * + * The first is the ordinary one: whatever depends on the object we are + * standing on. + * + * The second exists because a view's dependency on an EQL function is recorded + * against its rewrite RULE, not against the view, and nothing in pg_depend + * leads from the rule back out to the view — the rule's own edge to it points + * the wrong way. Walking edge one alone therefore stops at + * \`"_RETURN" on public.v1\` and never learns that v1 is a view, that v2 selects + * from v1, or that v3 selects from v2. The refusal was still correct (a rule is + * not a rebuildable index, so the install refused) but it named a rule where + * three views were at stake, and under-reported the blast radius to whoever had + * to decide what to do about it. + * + * Restricted to \`relkind IN ('v', 'm')\` deliberately. A view cannot outlive its + * _RETURN rule, so the view really is destroyed and really is the better name. + * A rule on an ordinary TABLE is different: dropping the rule leaves the table + * standing, so hopping there would name a table that survives, and the rule + * stays the honest answer. + */ +dependency_edges(refclassid, refobjid, refobjsubid, classid, objid, objsubid) AS ( + SELECT d.refclassid, d.refobjid, d.refobjsubid, d.classid, d.objid, d.objsubid + FROM pg_catalog.pg_depend d + UNION ALL + SELECT 'pg_catalog.pg_rewrite'::regclass, rewrite_rule.oid, 0, + 'pg_catalog.pg_class'::regclass, view_class.oid, 0 + FROM pg_catalog.pg_rewrite rewrite_rule + JOIN pg_catalog.pg_class view_class ON view_class.oid = rewrite_rule.ev_class + WHERE view_class.relkind IN ('v', 'm') +), +dependants(classid, objid, objsubid) AS ( + SELECT classid, objid, objsubid FROM eql_roots + UNION + SELECT e.classid, e.objid, e.objsubid + FROM dependency_edges e + JOIN dependants parent + ON e.refclassid = parent.classid + AND e.refobjid = parent.objid + AND (parent.objsubid = 0 OR e.refobjsubid = parent.objsubid) +), +external_dependants AS ( + SELECT d.classid, d.objid, d.objsubid + FROM dependants d + WHERE NOT EXISTS ( + SELECT 1 FROM eql_roots r + WHERE r.classid = d.classid + AND r.objid = d.objid + AND r.objsubid = d.objsubid + ) + -- Objects whose own namespace is disposable are bundle contents, even when + -- they were reached indirectly through an operator family or row type. + AND COALESCE( + (pg_catalog.pg_identify_object(d.classid, d.objid, d.objsubid)).schema, + '' + ) NOT IN ('eql_v3', 'eql_v3_internal') + -- A relation drags its composite row type and that type's array type along. + -- Both name the same casualty the relation already names (\`public.v1\`, + -- \`public.v1[]\`), so report the relation once and drop the two types. + AND NOT ( + d.classid = 'pg_catalog.pg_type'::regclass + AND EXISTS ( + SELECT 1 + FROM pg_catalog.pg_type shadow + LEFT JOIN pg_catalog.pg_type element ON element.oid = shadow.typelem + WHERE shadow.oid = d.objid + AND COALESCE(element.typrelid, shadow.typrelid) <> 0 + ) + ) + -- The view this rule belongs to is now in the walk and is the better name for + -- the same casualty, so report the view and drop the rule rather than both. + AND NOT ( + d.classid = 'pg_catalog.pg_rewrite'::regclass + AND EXISTS ( + SELECT 1 + FROM pg_catalog.pg_rewrite rewrite_rule + JOIN pg_catalog.pg_class view_class ON view_class.oid = rewrite_rule.ev_class + WHERE rewrite_rule.oid = d.objid + AND view_class.relkind IN ('v', 'm') + ) + ) + AND NOT ( + d.classid = 'pg_catalog.pg_constraint'::regclass + AND EXISTS ( + SELECT 1 + FROM pg_catalog.pg_constraint con + JOIN pg_catalog.pg_type typ ON typ.oid = con.contypid + JOIN pg_catalog.pg_namespace n ON n.oid = typ.typnamespace + WHERE con.oid = d.objid + AND ( + n.nspname IN ('eql_v3', 'eql_v3_internal') + OR ( + con.conname = 'eql_ore_unavailable' + AND n.nspname = 'public' + AND typ.typname LIKE 'eql\\_v3\\_%' ESCAPE '\\' + ) + ) + ) + ) + AND NOT ( + d.classid = 'pg_catalog.pg_proc'::regclass + AND EXISTS ( + SELECT 1 + FROM pg_catalog.pg_proc proc + JOIN pg_catalog.pg_namespace n ON n.oid = proc.pronamespace + WHERE proc.oid = d.objid + AND n.nspname IN ('eql_v3', 'eql_v3_internal') + ) + ) + AND NOT ( + d.classid = 'pg_catalog.pg_class'::regclass + AND EXISTS ( + SELECT 1 + FROM pg_catalog.pg_class class + JOIN pg_catalog.pg_namespace n ON n.oid = class.relnamespace + WHERE class.oid = d.objid + AND n.nspname IN ('eql_v3', 'eql_v3_internal') + ) + ) + -- Only exact operator identities parsed from the pinned bundle are owned by + -- EQL. A customer may legally give an EQL function a different operator + -- name, so implementation namespace alone is not an ownership marker. + AND NOT ( + d.classid = 'pg_catalog.pg_operator'::regclass + AND EXISTS ( + SELECT 1 + FROM pg_catalog.pg_operator operator + -- Prefix operators carry oprleft = 0, which matches no pg_type row; the + -- parser writes that operand as \`none\`. + LEFT JOIN type_identity left_type ON left_type.oid = operator.oprleft + LEFT JOIN type_identity right_type ON right_type.oid = operator.oprright + WHERE operator.oid = d.objid + AND pg_catalog.lower(operator.oprname) || ' (' || + COALESCE(left_type.identity, 'none') || ', ' || + COALESCE(right_type.identity, 'none') || ')' + = ANY($1::text[]) + -- Compared raw, NOT through lower(). Postgres operator names are drawn + -- from +-*/<>=~!@#%^&|? and cannot contain a letter, so lower() is a + -- no-op on the value -- but it is not a no-op on the plan: it makes + -- pg_operator_oprname_l_r_n_index (oprname, oprleft, oprright, + -- oprnamespace) unusable and turns this into a sequential scan. The + -- planner evaluates this count as a filter over the whole of + -- pg_operator before operator.oid = d.objid narrows anything, so + -- with lower() the guard is a self-join of pg_operator against itself: + -- ~3,900 rows squared, ~15M comparisons and ~9GB of buffer traffic on + -- a database with EQL installed, which is most of this query's cost. + -- Do not reintroduce it. + AND ( + SELECT pg_catalog.count(*) + FROM pg_catalog.pg_operator candidate + WHERE candidate.oprname = operator.oprname + AND candidate.oprleft = operator.oprleft + AND candidate.oprright = operator.oprright + ) = 1 + ) + ) + -- Likewise, these public-data-domain <-> query-domain casts are declarations + -- in the bundle, not application objects. + AND NOT ( + d.classid = 'pg_catalog.pg_cast'::regclass + AND EXISTS ( + SELECT 1 + FROM pg_catalog.pg_cast cast_row + JOIN type_identity source_type ON source_type.oid = cast_row.castsource + JOIN type_identity target_type ON target_type.oid = cast_row.casttarget + WHERE cast_row.oid = d.objid + AND source_type.identity || ' AS ' || target_type.identity + = ANY($2::text[]) + ) + ) + -- pg_amop/pg_amproc have no namespace of their own; their owning family does. + AND NOT ( + d.classid IN ( + 'pg_catalog.pg_amop'::regclass, + 'pg_catalog.pg_amproc'::regclass + ) + AND EXISTS ( + SELECT 1 + FROM pg_catalog.pg_opfamily family + JOIN pg_catalog.pg_namespace n ON n.oid = family.opfnamespace + WHERE family.oid = CASE + WHEN d.classid = 'pg_catalog.pg_amop'::regclass + THEN (SELECT amopfamily FROM pg_catalog.pg_amop WHERE oid = d.objid) + ELSE (SELECT amprocfamily FROM pg_catalog.pg_amproc WHERE oid = d.objid) + END + AND n.nspname IN ('eql_v3', 'eql_v3_internal') + ) + ) +) +/* + * \`i\` is an ordinary index; \`I\` is a PARTITIONED index — the parent template on + * a partitioned table. Both are rebuildable, but only together: the parent's + * \`pg_get_indexdef\` says \`ON ONLY\`, its per-partition children come back as + * separate \`i\` rows, and the parent stays \`indisvalid = false\` until every one + * of them is re-ATTACHed. \`parent_identity\` (from pg_inherits, which relates an + * attached child index to its parent) is what carries that edge across the + * DROP; see {@link rebuildIndexes}. + */ +SELECT DISTINCT + CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL THEN 'index' + ELSE 'unsafe' + END AS dependency_kind, + CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL + THEN pg_catalog.format('%I.%I', index_namespace.nspname, index_class.relname) + ELSE (pg_catalog.pg_identify_object(e.classid, e.objid, e.objsubid)).identity + END AS identity, + CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL THEN pg_catalog.pg_get_indexdef(e.objid) + END AS definition, + CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL + THEN pg_catalog.format('%I.%I', table_namespace.nspname, table_class.relname) + END AS table_identity, + CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL THEN index_meta.indisvalid + END AS valid, + CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL THEN index_meta.indisready + END AS ready +FROM external_dependants e +LEFT JOIN pg_catalog.pg_class index_class + ON e.classid = 'pg_catalog.pg_class'::regclass AND index_class.oid = e.objid +LEFT JOIN pg_catalog.pg_inherits index_partition + ON index_partition.inhrelid = index_class.oid +LEFT JOIN pg_catalog.pg_namespace index_namespace ON index_namespace.oid = index_class.relnamespace +LEFT JOIN pg_catalog.pg_index index_meta ON index_meta.indexrelid = index_class.oid +LEFT JOIN pg_catalog.pg_class table_class ON table_class.oid = index_meta.indrelid +LEFT JOIN pg_catalog.pg_namespace table_namespace ON table_namespace.oid = table_class.relnamespace +ORDER BY dependency_kind, identity +` + +const VERIFY_REBUILT_INDEXES_SQL = ` +/* stash_eql_verify_rebuilt_indexes */ +SELECT pg_catalog.format('%I.%I', n.nspname, c.relname) AS identity, + i.indisvalid AS valid, + i.indisready AS ready, + pg_catalog.pg_get_indexdef(i.indexrelid) AS definition +FROM pg_catalog.pg_index i +JOIN pg_catalog.pg_class c ON c.oid = i.indexrelid +JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace +WHERE pg_catalog.format('%I.%I', n.nspname, c.relname) = ANY($1::text[]) +` + +/** + * Take the installer's advisory lock, or say why not. + * + * `pg_advisory_lock` waits forever. A concurrent `stash eql install`, or a + * session that died holding the lock, then makes the command hang with no + * output and no timeout — indistinguishable from a network stall, and the one + * failure a user cannot diagnose. Polling `pg_try_advisory_lock` against a + * bounded budget keeps the ordinary case working — a queued install still wins + * the lock and runs — while turning the pathological one into a sentence. See + * {@link LOCK_WAIT_MS} for why the budget is a minute and not the handful of + * seconds it first was. (#959) + */ +export async function acquireLifecycleLock( + client: pg.ClientBase, + // Optional so `acquireLifecycleLock(client)` keeps meaning what it did. The + // budget is a policy, not a constant of nature, and the live suite drives it + // short to assert the refusal without waiting out the production default. + waitMs: number = LOCK_WAIT_MS, +) { + const deadline = Date.now() + waitMs + for (;;) { + const result = await client.query<{ acquired: boolean }>( + 'SELECT pg_try_advisory_lock(hashtext($1)) AS acquired', + [LIFECYCLE_LOCK], + ) + if (result.rows[0]?.acquired === true) { + lockHolders.add(client) + return + } + if (Date.now() >= deadline) { + throw new Error( + `Another EQL lifecycle operation is in progress on this database — it has held the installer's advisory lock for more than ${Math.round(waitMs / 1000)} seconds. Nothing was changed. Wait for the other \`stash eql install\`/\`eql upgrade\` to finish and re-run. If no other command is running, an earlier one may have died holding the lock: find its session in \`pg_stat_activity\` and close it, then retry.`, + ) + } + // Deliberately NOT unref'd: the retry is the only pending work between + // polls, and letting Node drop it would end the process mid-install. + await new Promise((resolve) => setTimeout(resolve, LOCK_RETRY_INTERVAL_MS)) + } +} + +/** + * Release the lock if this connection took it. The caller unlocks + * unconditionally in a `finally`, which includes the path where the acquire + * above refused — and `pg_advisory_unlock` on a lock a session never held + * returns false and raises a WARNING, i.e. noise on the one path that already + * has a clear message. + */ +export async function releaseLifecycleLock(client: pg.ClientBase) { + if (!lockHolders.has(client)) return + lockHolders.delete(client) + await client.query('SELECT pg_advisory_unlock(hashtext($1))', [ + LIFECYCLE_LOCK, + ]) +} + +export async function inspectReinstallDependencies( + client: pg.ClientBase, + bundleOperators: string[], + bundleCasts: string[], +): Promise { + const result = await client.query(LIFECYCLE_DEPENDENCIES_SQL, [ + bundleOperators, + bundleCasts, + ]) + const unsafe: string[] = [] + const indexes: ReinstallIndex[] = [] + for (const row of result.rows as DependencyRow[]) { + if (row.dependency_kind !== 'index') { + unsafe.push(String(row.identity ?? 'unknown database object')) + continue + } + if ( + typeof row.identity !== 'string' || + typeof row.definition !== 'string' || + typeof row.table_identity !== 'string' || + typeof row.valid !== 'boolean' || + typeof row.ready !== 'boolean' + ) { + unsafe.push( + String(row.identity ?? 'index with incomplete catalog metadata'), + ) + continue + } + indexes.push({ + identity: row.identity, + definition: row.definition, + tableIdentity: row.table_identity, + valid: row.valid, + ready: row.ready, + }) + } + if (unsafe.length > 0) { + throw new Error( + `EQL reinstall refused before making changes because customer-owned database objects depend on disposable EQL machinery and cannot be reconstructed safely:\n${unsafe.map((identity) => ` - ${identity}`).join('\n')}`, + ) + } + return indexes +} + +/** + * Recreate captured ordinary indexes and verify their exact catalog shape. + * + * `pg_get_indexdef` is itself search_path-sensitive: on a connection whose + * search_path names `eql_v3` the captured definition reads `eq_term(encrypted)` + * rather than `eql_v3.eq_term(encrypted)`. That is safe here and deliberately + * not "fixed" — the same session rebuilds it, and Postgres qualifies a name + * exactly when leaving it bare would resolve to something else, so the + * rebuilt expression binds to the same function the original did. + * + */ +export async function rebuildIndexes( + client: pg.ClientBase, + indexes: ReinstallIndex[], +) { + for (const index of indexes) { + try { + await client.query(index.definition) + } catch (error) { + const detail = error instanceof Error ? error.message : String(error) + throw new Error( + `EQL reinstall could not rebuild search index ${index.identity}: ${detail}\nThe transaction will restore the previous EQL installation and index. Captured index SQL:\n${index.definition}`, + { cause: error }, + ) + } + } + for (const tableIdentity of new Set( + indexes.map((index) => index.tableIdentity), + )) { + await client.query(`ANALYZE ${tableIdentity}`) + } + if (indexes.length === 0) return + const result = await client.query(VERIFY_REBUILT_INDEXES_SQL, [ + indexes.map((index) => index.identity), + ]) + const healthy = new Set( + result.rows + .filter((row) => { + const expected = indexes.find( + (index) => index.identity === row.identity, + ) + if (expected === undefined) return false + // The definition must match EXACTLY: it is the only evidence that the + // index which came back is the index that went away. + // + // Validity is compared against what was CAPTURED rather than against + // `true`, because an index can already be invalid going in and the + // reinstall does not promise to repair one it did not break — a + // partitioned parent created `ON ONLY` while its per-partition indexes + // are still being built, or an interrupted `CREATE INDEX CONCURRENTLY`. + // Demanding `true` there fails the whole install and the rollback says + // nothing about why. A VALID index coming back invalid is still caught; + // that is the regression this check exists for. + return ( + expected.definition === row.definition && + (row.valid === true || expected.valid === false) && + (row.ready === true || expected.ready === false) + ) + }) + .map((row) => String(row.identity)), + ) + const unhealthy = indexes.filter((index) => !healthy.has(index.identity)) + if (unhealthy.length > 0) { + throw new Error( + `EQL reinstall produced missing, invalid, or changed search indexes; the transaction will restore the previous installation:\n${unhealthy.map((index) => ` - ${index.identity}\n ${index.definition}`).join('\n')}`, + ) + } +} diff --git a/packages/eql/tests/sqlx/tests/v3_uninstall_tests.rs b/packages/eql/tests/sqlx/tests/v3_uninstall_tests.rs index aef936754..52f76cc12 100644 --- a/packages/eql/tests/sqlx/tests/v3_uninstall_tests.rs +++ b/packages/eql/tests/sqlx/tests/v3_uninstall_tests.rs @@ -189,7 +189,7 @@ async fn shipped_installer_can_run_over_existing_public_domains(pool: PgPool) -> Ok(()) } -#[sqlx::test] +#[sqlx::test(fixtures(path = "../fixtures", scripts("eql_v3_integer", "v3_ste_vec")))] async fn uninstaller_preserves_application_tables_with_public_domain_columns( pool: PgPool, ) -> Result<()> { @@ -199,10 +199,23 @@ async fn uninstaller_preserves_application_tables_with_public_domain_columns( "expected both eql_v3 schemas installed before uninstall" ); - let scalar_payload = r#"{"v":3,"i":{},"c":"scalar-42","hm":"hm-42"}"#; - // SteVec entries carry no `hm`: an op path entry / term-less entry. - let json_payload = r#"{"i":{},"v":3,"h":"kh","sv":[{"s":"age","c":"cipher-age","op":"ab"}]}"#; - let entry_payload = r#"{"s":"age","c":"cipher-age","op":"ab"}"#; + // These are generated by cipherstash-client for every test run. Preservation + // must be proved with real ciphertext and index terms, not JSON shaped by + // the test to happen to satisfy today's domain constraints. + let scalar_payload: serde_json::Value = + sqlx::query_scalar("SELECT payload FROM fixtures.eql_v3_integer ORDER BY id LIMIT 1") + .fetch_one(&pool) + .await?; + let json_payload: serde_json::Value = + sqlx::query_scalar("SELECT payload FROM fixtures.v3_ste_vec ORDER BY id LIMIT 1") + .fetch_one(&pool) + .await?; + let entry_payload = json_payload + .get("sv") + .and_then(serde_json::Value::as_array) + .and_then(|entries| entries.first()) + .cloned() + .expect("generated SteVec fixture must contain at least one entry"); sqlx::query( r#" @@ -230,9 +243,9 @@ async fn uninstaller_preserves_application_tables_with_public_domain_columns( ) "#, ) - .bind(scalar_payload) - .bind(json_payload) - .bind(entry_payload) + .bind(&scalar_payload) + .bind(&json_payload) + .bind(&entry_payload) .execute(&pool) .await?; @@ -311,18 +324,9 @@ async fn uninstaller_preserves_application_tables_with_public_domain_columns( ) .fetch_one(&pool) .await?; - assert_eq!( - values.0, - serde_json::from_str::(scalar_payload)? - ); - assert_eq!( - values.1, - serde_json::from_str::(json_payload)? - ); - assert_eq!( - values.2, - serde_json::from_str::(entry_payload)? - ); + assert_eq!(values.0, scalar_payload); + assert_eq!(values.1, json_payload); + assert_eq!(values.2, entry_payload); // The misuse table survives, but its query-operand column went down with // the eql_v3 schema (DROP SCHEMA ... CASCADE drops the domain, which @@ -352,5 +356,158 @@ async fn uninstaller_preserves_application_tables_with_public_domain_columns( "a column typed as an eql_v3 query-operand domain is dropped with the schema" ); + run_shipped_installer(&pool).await?; + + assert_eq!( + schema_count(&pool).await?, + 2, + "reinstall must recreate both EQL-owned schemas without replacing application data" + ); + let values_after_reinstall: (serde_json::Value, serde_json::Value, serde_json::Value) = + sqlx::query_as( + r#" + SELECT + scalar_value::jsonb, + doc_value::jsonb, + entry_value::jsonb + FROM public.eql_v3_uninstall_preserve + WHERE id = 1 + "#, + ) + .fetch_one(&pool) + .await?; + assert_eq!( + values_after_reinstall, values, + "install -> uninstall -> reinstall must preserve every stored byte" + ); + + Ok(()) +} + +#[sqlx::test(fixtures(path = "../fixtures", scripts("eql_v3_text", "v3_ste_vec")))] +async fn uninstall_and_reinstall_preserve_rows_for_every_public_eql_domain( + pool: PgPool, +) -> Result<()> { + let scalar_payload: serde_json::Value = + sqlx::query_scalar("SELECT payload FROM fixtures.eql_v3_text ORDER BY id LIMIT 1") + .fetch_one(&pool) + .await?; + let document_payload: serde_json::Value = + sqlx::query_scalar("SELECT payload FROM fixtures.v3_ste_vec ORDER BY id LIMIT 1") + .fetch_one(&pool) + .await?; + let entry_payload = document_payload + .get("sv") + .and_then(serde_json::Value::as_array) + .and_then(|entries| entries.first()) + .cloned() + .expect("generated SteVec fixture must contain at least one entry"); + + let domains: Vec = sqlx::query_scalar( + r#" + SELECT t.typname + FROM pg_catalog.pg_type t + JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace + WHERE n.nspname = 'public' + AND t.typtype = 'd' + AND t.typname LIKE 'eql_v3\_%' ESCAPE '\' + ORDER BY t.typname + "#, + ) + .fetch_all(&pool) + .await?; + assert!( + !domains.is_empty(), + "installer must expose public data domains" + ); + + let mut expected = Vec::with_capacity(domains.len()); + for (index, domain) in domains.iter().enumerate() { + // Names come from pg_type and are quoted before interpolation; values + // remain bind parameters. One independent table per domain makes a + // CASCADE-dropped column observable as missing data, not just a catalog + // discrepancy. + let quoted_domain = domain.replace('"', "\"\""); + let table = format!("eql_v3_preserve_{index}"); + let payload = match domain.as_str() { + "eql_v3_json_search" => &document_payload, + "eql_v3_json_entry" => &entry_payload, + _ => &scalar_payload, + }; + sqlx::raw_sql(&format!( + "CREATE TABLE public.{table} (id integer PRIMARY KEY, value public.\"{quoted_domain}\" NOT NULL)" + )) + .execute(&pool) + .await?; + sqlx::query(&format!( + "INSERT INTO public.{table} VALUES (1, $1::jsonb::public.\"{quoted_domain}\")" + )) + .bind(payload) + .execute(&pool) + .await?; + let identity: (i64, i16, i64, String) = sqlx::query_as(&format!( + r#" + SELECT c.oid::bigint, a.attnum, a.atttypid::bigint, a.attname::text + FROM pg_catalog.pg_class c + JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace + JOIN pg_catalog.pg_attribute a ON a.attrelid = c.oid + WHERE n.nspname = 'public' AND c.relname = '{table}' AND a.attname = 'value' + "# + )) + .fetch_one(&pool) + .await?; + expected.push((table, payload.clone(), identity)); + } + + run_shipped_uninstaller(&pool).await?; + for (table, payload, identity) in &expected { + let actual: serde_json::Value = sqlx::query_scalar(&format!( + "SELECT value::jsonb FROM public.{table} WHERE id = 1" + )) + .fetch_one(&pool) + .await?; + assert_eq!(&actual, payload, "uninstall changed data in {table}"); + let actual_identity: (i64, i16, i64, String) = sqlx::query_as(&format!( + r#" + SELECT c.oid::bigint, a.attnum, a.atttypid::bigint, a.attname::text + FROM pg_catalog.pg_class c + JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace + JOIN pg_catalog.pg_attribute a ON a.attrelid = c.oid + WHERE n.nspname = 'public' AND c.relname = '{table}' AND a.attname = 'value' + "# + )) + .fetch_one(&pool) + .await?; + assert_eq!( + &actual_identity, identity, + "uninstall changed column identity in {table}" + ); + } + + run_shipped_installer(&pool).await?; + for (table, payload, identity) in &expected { + let actual: serde_json::Value = sqlx::query_scalar(&format!( + "SELECT value::jsonb FROM public.{table} WHERE id = 1" + )) + .fetch_one(&pool) + .await?; + assert_eq!(&actual, payload, "reinstall changed data in {table}"); + let actual_identity: (i64, i16, i64, String) = sqlx::query_as(&format!( + r#" + SELECT c.oid::bigint, a.attnum, a.atttypid::bigint, a.attname::text + FROM pg_catalog.pg_class c + JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace + JOIN pg_catalog.pg_attribute a ON a.attrelid = c.oid + WHERE n.nspname = 'public' AND c.relname = '{table}' AND a.attname = 'value' + "# + )) + .fetch_one(&pool) + .await?; + assert_eq!( + &actual_identity, identity, + "reinstall changed column identity in {table}" + ); + } + Ok(()) } diff --git a/skills/stash-cli/SKILL.md b/skills/stash-cli/SKILL.md index ace23e94a..c40726cdd 100644 --- a/skills/stash-cli/SKILL.md +++ b/skills/stash-cli/SKILL.md @@ -419,6 +419,8 @@ Run it whenever query-time behaviour looks inconsistent with a "successful" inst Generates an **EQL v3 install migration**, instead of running SQL directly against the database (`eql install`). Migration-first is the preferred path: the install lands in your migration history and ships to every environment through the same migrate step as the rest of your schema. On Supabase it is the *only* durable path — `supabase db reset` replays the migrations directory, so a direct install is wiped by the next reset. v3 only — there is no `--eql-version` here. +**The re-run protections do not travel with the file.** The lifecycle lock, index capture, refusal on unsupported dependants, and rebuild all live in the `eql install`/`eql upgrade` code path, not in the bundle SQL. A generated migration is the raw bundle, so a migration runner applying it does the `DROP SCHEMA ... CASCADE` unprotected. That is fine on a first install. Re-applying it over a database that already carries EQL functional indexes drops them with no rebuild — reach for `eql upgrade` there, or recreate the indexes in the same migration. + ```bash stash eql migration --drizzle # Drizzle custom migration in drizzle/ stash eql migration --drizzle --supabase # also grant eql_v3 to anon/authenticated/service_role @@ -495,7 +497,18 @@ An applied migration carrying a statement the sweep would have skipped anyway #### `eql upgrade` -The install SQL is safe to re-run — columns and data survive — but it cascade-drops functional indexes that depend on `eql_v3`; recreate them afterward. `upgrade` is v3-only and accepts `--supabase`, `--dry-run`, and `--database-url`. +The install SQL is safe to re-run: encrypted columns and rows live outside the +disposable EQL schemas. `upgrade` serializes the lifecycle with a database +advisory lock, captures functional-index definitions, replaces the schemas, +then rebuilds, analyzes, and verifies those indexes in the same transaction. Unsupported external +dependencies (including policies and views) make it refuse before mutation. A +rebuild failure rolls back the schema replacement and restores the prior indexes; +never describe a failed reconstruction as a successful upgrade. `upgrade` is +v3-only and accepts `--supabase`, `--dry-run`, and `--database-url`. + +Run it in a schema-migration maintenance window. The advisory lock serializes +other `stash` lifecycle commands, not arbitrary DDL from unrelated sessions; +do not create, alter, or drop EQL-backed indexes concurrently. #### `eql status` From 14d23aa9cb7e9ff87f01a9a10ec6b358c541ab3b Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Tue, 1 Sep 2026 12:17:20 +1000 Subject: [PATCH 02/11] test(cli): upgrade a released EQL bundle with real ciphertext --- .github/workflows/tests.yml | 6 +- packages/cli/package.json | 2 + .../upgrade-encrypted-indexes.live.test.ts | 174 ++++++++++++++++++ pnpm-lock.yaml | 11 ++ .../lint-no-eql-registry-pins.test.mjs | 24 +++ scripts/lint-no-eql-registry-pins.mjs | 36 +++- 6 files changed, 241 insertions(+), 12 deletions(-) create mode 100644 packages/cli/src/installer/__tests__/upgrade-encrypted-indexes.live.test.ts diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index a1d915d7d..0f5e2af5d 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -333,8 +333,10 @@ jobs: # (`installer/__tests__/verify.live.test.ts`) would run in no CI # workflow at all: a routine `@cipherstash/eql` bump could then make # every `stash eql install` fail with phantom damage, on green CI. - # These suites need Postgres only, no CipherStash credentials; the - # verify suite installs EQL v3 into its own schemas, which coexists + # Most suites need Postgres only. The encrypted-index upgrade suite also + # loads this job's CipherStash credentials from packages/stack/.env and + # uses the binding built above to create genuine ciphertext. The verify + # suite installs EQL v3 into its own schemas, which coexists # with the image's pre-installed EQL v2 that the stack tests use. # They share that one database, so the CLI vitest config runs them # serially (the `live` project sets `fileParallelism: false` — diff --git a/packages/cli/package.json b/packages/cli/package.json index 879cc0f09..881f69626 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -75,6 +75,8 @@ } }, "devDependencies": { + "@cipherstash/eql-upgrade-baseline": "npm:@cipherstash/eql@3.0.2", + "@cipherstash/protect-ffi": "workspace:*", "@cipherstash/stack": "workspace:*", "@types/pg": "^8.23.1", "node-pty": "^1.1.0", diff --git a/packages/cli/src/installer/__tests__/upgrade-encrypted-indexes.live.test.ts b/packages/cli/src/installer/__tests__/upgrade-encrypted-indexes.live.test.ts new file mode 100644 index 000000000..26cdf39d3 --- /dev/null +++ b/packages/cli/src/installer/__tests__/upgrade-encrypted-indexes.live.test.ts @@ -0,0 +1,174 @@ +/** + * Credentialed upgrade coverage: a released EQL bundle owns the original + * database objects, protect-ffi writes genuine ciphertext, and the public CLI + * installer replaces that bundle without breaking the encrypted indexes. + */ + +import { readInstallSql as readBaselineInstallSql } from '@cipherstash/eql-upgrade-baseline/sql' +import { + decryptBulk, + type EncryptConfig, + type EncryptedPayload, + encryptBulk, + encryptQuery, + newClient, +} from '@cipherstash/protect-ffi' +import { config as loadEnv } from 'dotenv' +import pg from 'pg' +import { afterAll, beforeAll, describe, expect, it } from 'vitest' +import { EQLInstaller } from '../index.js' + +loadEnv({ + path: new URL('../../../../stack/.env', import.meta.url), + quiet: true, +}) + +const DATABASE_URL = process.env.STASH_TEST_DATABASE_URL +const hasCredentials = [ + 'CS_WORKSPACE_CRN', + 'CS_CLIENT_ID', + 'CS_CLIENT_KEY', + 'CS_CLIENT_ACCESS_KEY', +].every((name) => process.env[name]) +const describeLive = DATABASE_URL && hasCredentials ? describe : describe.skip + +const BASELINE_VERSION = '3.0.2' +const encryptConfig: EncryptConfig = { + v: 1, + tables: { + eql_upgrade_records: { + email: { cast_as: 'text', indexes: { unique: {} } }, + score: { cast_as: 'int', indexes: { ore: {} } }, + }, + }, +} + +describeLive('EQLInstaller upgrade — genuine encrypted indexes', () => { + const client = new pg.Client({ connectionString: DATABASE_URL }) + let protectClient: Awaited> + + beforeAll(async () => { + await client.connect() + await client.query('DROP TABLE IF EXISTS eql_upgrade_records') + await client.query(readBaselineInstallSql()) + expect( + ( + await client.query<{ version: string }>( + 'SELECT eql_v3.version() AS version', + ) + ).rows[0].version, + ).toBe(BASELINE_VERSION) + + protectClient = await newClient({ encryptConfig, eqlVersion: 3 }) + const rows = await encryptBulk(protectClient, { + plaintexts: [ + { + plaintext: 'alice@example.com', + column: 'email', + table: 'eql_upgrade_records', + }, + { plaintext: 10, column: 'score', table: 'eql_upgrade_records' }, + { + plaintext: 'bob@example.com', + column: 'email', + table: 'eql_upgrade_records', + }, + { plaintext: 20, column: 'score', table: 'eql_upgrade_records' }, + ], + }) + await client.query(` + CREATE TABLE eql_upgrade_records ( + id integer PRIMARY KEY, + email public.eql_v3_text_eq NOT NULL, + score public.eql_v3_integer_ord_ore NOT NULL + ); + CREATE INDEX eql_upgrade_email_idx + ON eql_upgrade_records (eql_v3.eq_term(email)); + CREATE INDEX eql_upgrade_score_idx + ON eql_upgrade_records (eql_v3.ord_term_ore(score)); + `) + await client.query( + `INSERT INTO eql_upgrade_records VALUES + (1, $1::jsonb, $2::jsonb), (2, $3::jsonb, $4::jsonb)`, + rows, + ) + }, 180_000) + + afterAll(async () => { + await client + .query('DROP TABLE IF EXISTS eql_upgrade_records') + .catch(() => undefined) + await client.end().catch(() => undefined) + }) + + it('upgrades a released installation while preserving usable encrypted indexes', async () => { + await new EQLInstaller({ databaseUrl: DATABASE_URL ?? '' }).install() + + const emailOperand = await encryptQuery(protectClient, { + plaintext: 'bob@example.com', + column: 'email', + table: 'eql_upgrade_records', + indexType: 'unique', + }) + const scoreOperand = await encryptQuery(protectClient, { + plaintext: 15, + column: 'score', + table: 'eql_upgrade_records', + indexType: 'ore', + }) + const result = await client.query<{ + email: EncryptedPayload + score: EncryptedPayload + }>( + `SELECT email::jsonb, score::jsonb FROM eql_upgrade_records + WHERE email = $1::jsonb::eql_v3.query_text_eq + AND score > $2::jsonb::eql_v3.query_integer_ord_ore`, + [emailOperand, scoreOperand], + ) + expect( + await decryptBulk(protectClient, { + ciphertexts: result.rows.flatMap(({ email, score }) => [ + { ciphertext: email }, + { ciphertext: score }, + ]), + }), + ).toEqual(['bob@example.com', 20]) + + const indexes = await client.query<{ relname: string; valid: boolean }>(` + SELECT c.relname, i.indisvalid AS valid + FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid + WHERE c.relname IN ('eql_upgrade_email_idx', 'eql_upgrade_score_idx') + ORDER BY c.relname + `) + expect(indexes.rows).toEqual([ + { relname: 'eql_upgrade_email_idx', valid: true }, + { relname: 'eql_upgrade_score_idx', valid: true }, + ]) + + await client.query('SET enable_seqscan = off') + try { + const emailPlan = await client.query<{ 'QUERY PLAN': string }>( + `EXPLAIN (COSTS OFF) + SELECT id FROM eql_upgrade_records + WHERE eql_v3.eq_term(email) = + eql_v3.eq_term($1::jsonb::eql_v3.query_text_eq)`, + [emailOperand], + ) + const scorePlan = await client.query<{ 'QUERY PLAN': string }>( + `EXPLAIN (COSTS OFF) + SELECT id FROM eql_upgrade_records + WHERE eql_v3.ord_term_ore(score) > + eql_v3.ord_term_ore($1::jsonb::eql_v3.query_integer_ord_ore)`, + [scoreOperand], + ) + expect( + emailPlan.rows.map((row) => row['QUERY PLAN']).join('\n'), + ).toContain('eql_upgrade_email_idx') + expect( + scorePlan.rows.map((row) => row['QUERY PLAN']).join('\n'), + ).toContain('eql_upgrade_score_idx') + } finally { + await client.query('RESET enable_seqscan') + } + }, 180_000) +}) diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 4ae3e15b7..5db220f73 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -248,6 +248,12 @@ importers: specifier: ^3.25.76 version: 3.25.76 devDependencies: + '@cipherstash/eql-upgrade-baseline': + specifier: npm:@cipherstash/eql@3.0.2 + version: '@cipherstash/eql@3.0.2' + '@cipherstash/protect-ffi': + specifier: workspace:* + version: link:../protect-ffi '@cipherstash/stack': specifier: workspace:* version: link:../stack @@ -1038,6 +1044,9 @@ packages: '@cipherstash/auth-win32-x64-msvc': optional: true + '@cipherstash/eql@3.0.2': + resolution: {integrity: sha512-E85o0aoOqgCW6RReLtJ0YLh/ExRlmDJo7LlJGpWPoMTVaw+CW8o11DJ4oJIF1vFtuxSVxNULuPzzBuVmpTvvcA==} + '@clack/core@1.4.3': resolution: {integrity: sha512-/kr3UWNtdJfxZtPgDqUOmG2pvwlmcLGheex5yiZKdwbzZJxhV+HMNR9QNmyY5cGwTNV6LrR7Jtp+KjhUAP1qBQ==} engines: {node: '>= 20.12.0'} @@ -3990,6 +3999,8 @@ snapshots: '@cipherstash/auth-linux-x64-musl': 0.44.0 '@cipherstash/auth-win32-x64-msvc': 0.44.0 + '@cipherstash/eql@3.0.2': {} + '@clack/core@1.4.3': dependencies: fast-wrap-ansi: 0.2.0 diff --git a/scripts/__tests__/lint-no-eql-registry-pins.test.mjs b/scripts/__tests__/lint-no-eql-registry-pins.test.mjs index a248f333a..04796b8bb 100644 --- a/scripts/__tests__/lint-no-eql-registry-pins.test.mjs +++ b/scripts/__tests__/lint-no-eql-registry-pins.test.mjs @@ -802,6 +802,30 @@ describe('the scan reports what it finds', () => { 'b/package.json :: @cipherstash/eql', ]) }) + + it('exempts a test alias without exempting the runtime dependency beside it', () => { + const root = tree({ + 'a/package.json': JSON.stringify({ + dependencies: { '@cipherstash/eql': '3.0.4' }, + devDependencies: { + '@cipherstash/eql-upgrade-baseline': 'npm:@cipherstash/eql@3.0.2', + }, + }), + }) + const result = lint({ + root, + expected: [], + exemptions: new Map([ + ['a/package.json :: @cipherstash/eql-upgrade-baseline', 'test fixture'], + ]), + }) + expect(result.exempted.map((entry) => entry.key)).toEqual([ + '@cipherstash/eql-upgrade-baseline', + ]) + expect(result.offenders.map((entry) => entry.key)).toEqual([ + '@cipherstash/eql', + ]) + }) }) describe('the linter fails when its own configuration goes stale', () => { diff --git a/scripts/lint-no-eql-registry-pins.mjs b/scripts/lint-no-eql-registry-pins.mjs index 830993e28..62408b632 100644 --- a/scripts/lint-no-eql-registry-pins.mjs +++ b/scripts/lint-no-eql-registry-pins.mjs @@ -167,7 +167,7 @@ export const EXPECTED_SOURCES = [WORKSPACE_FILE] /** * Declarations allowed to name a registry version, each with the reason. * - * EMPTY, and that is the goal state rather than an oversight. Every entry is a + * Keep this list as short as possible. Every entry is a * place the two halves of EQL can drift apart again, and the reason is what a * later reader needs in order to decide whether it is still true. * @@ -179,9 +179,19 @@ export const EXPECTED_SOURCES = [WORKSPACE_FILE] * DECLARES `@cipherstash/eql` and an existence-based check would have gone on * passing over a standing permission nothing needed. * - * Adding one back means writing the reason down here. Prefer not to. + * The remaining entry is an immutable test fixture rather than a runtime + * dependency. Adding another means writing the reason down here. Prefer not + * to. */ -export const EXEMPT_DECLARATIONS = new Map([]) +export const EXEMPT_DECLARATIONS = new Map([ + [ + 'packages/cli/package.json :: @cipherstash/eql-upgrade-baseline', + 'Test-only immutable upgrade origin: the credentialed live installer test ' + + 'must install a real previously released bundle before the workspace ' + + 'installer upgrades it. Runtime `@cipherstash/eql` and the payload-emitting ' + + 'Rust remain workspace-linked; this alias is never packed for consumers.', + ], +]) /** Files this scan reads, by name. */ const SCANNED_FILES = new Set(['Cargo.toml', 'package.json', WORKSPACE_FILE]) @@ -458,9 +468,9 @@ function collectNpmEntries(file, table, entries, found) { if (named || NPM_ALIAS.test(scalar)) { found.push({ file, - // Always the package, never the key it was found under: both - // hand-maintained lists are keyed ` :: `, so an - // alias filed under its alias name could never be exempted. + // Expected declarations stay keyed by the real package. Exemptions + // use the alias key when present so they cannot excuse a runtime + // declaration of the same package in the same manifest. dependency: NPM_DEPENDENCY, table, key, @@ -614,6 +624,12 @@ export function scanTree(root) { export const declarationId = (declaration) => `${declaration.file} :: ${declaration.dependency}` +/** A renamed npm dependency can be exempted without exempting its runtime twin. */ +const exemptionId = (declaration) => + declaration.form === 'alias' && declaration.key !== declaration.dependency + ? `${declaration.file} :: ${declaration.key}` + : declarationId(declaration) + /** * The whole check, as data. Separated from the reporting below so the tests can * drive every branch — including the two exit-2 ones — by passing a different @@ -629,15 +645,15 @@ export function lint({ const ids = declarations.map(declarationId) const registryPinned = declarations .filter((d) => !d.inTree) - .map(declarationId) + .map(exemptionId) return { declarations, ids, offenders: declarations.filter( - (d) => !d.inTree && !exemptions.has(declarationId(d)), + (d) => !d.inTree && !exemptions.has(exemptionId(d)), ), exempted: declarations.filter( - (d) => !d.inTree && exemptions.has(declarationId(d)), + (d) => !d.inTree && exemptions.has(exemptionId(d)), ), // An exemption that is not excusing anything, and an exemption whose reason // was emptied out. Both are the configuration going stale, and both must @@ -745,7 +761,7 @@ export function report(result) { if (result.offenders.length === 0) { const suffix = result.exempted.length ? ` (${result.exempted.length} exempt: ${result.exempted - .map(declarationId) + .map(exemptionId) .join(', ')})` : '' return { code: 0, out: `Every EQL dependency resolves in-tree${suffix}.` } From 23eedf16e49b9d797187394f0211fb228b37155c Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Tue, 1 Sep 2026 12:49:03 +1000 Subject: [PATCH 03/11] refactor(cli): deepen EQL index restoration --- packages/cli/src/__tests__/installer.test.ts | 57 +++++- .../__tests__/reinstall.live.test.ts | 37 ++-- ...ts => derived-search-index-restoration.ts} | 192 ++++++++++++------ packages/cli/src/installer/index.ts | 91 +++------ 4 files changed, 231 insertions(+), 146 deletions(-) rename packages/cli/src/installer/{reinstall.ts => derived-search-index-restoration.ts} (81%) diff --git a/packages/cli/src/__tests__/installer.test.ts b/packages/cli/src/__tests__/installer.test.ts index 14fd17302..1e096d5b3 100644 --- a/packages/cli/src/__tests__/installer.test.ts +++ b/packages/cli/src/__tests__/installer.test.ts @@ -12,7 +12,7 @@ vi.mock('pg', () => ({ const result = await mockQuery(...args) if ( typeof args[0] === 'string' && - args[0].includes('pg_try_advisory_lock') && + args[0].includes('pg_try_advisory') && result?.rows?.[0]?.acquired === undefined ) { return { ...result, rows: [{ acquired: true }] } @@ -303,6 +303,8 @@ describe('EQLInstaller', () => { table_identity: 'app.users', valid: true, ready: true, + clustered: false, + cluster_sql: null, }, ], rowCount: 1, @@ -315,6 +317,7 @@ describe('EQLInstaller', () => { identity: 'app.users_email_idx', valid: true, ready: true, + clustered: false, definition: indexDefinition, }, ], @@ -329,10 +332,10 @@ describe('EQLInstaller', () => { await installer.install() expect(mockQuery).toHaveBeenCalledWith( - 'SELECT pg_try_advisory_lock(hashtext($1)) AS acquired', + 'SELECT pg_catalog.pg_try_advisory_xact_lock(pg_catalog.hashtext($1)) AS acquired', ['cipherstash.eql.lifecycle'], ) - expect(mockQuery).toHaveBeenCalledWith('SET jit = off') + expect(mockQuery).toHaveBeenCalledWith('SET LOCAL jit = off') expect(mockQuery).toHaveBeenCalledWith(indexDefinition) expect(mockQuery).toHaveBeenCalledWith('ANALYZE app.users') const bundleCall = mockQuery.mock.calls.findIndex( @@ -344,10 +347,14 @@ describe('EQLInstaller', () => { ) expect(bundleCall).toBeGreaterThan(-1) expect(rebuildCall).toBeGreaterThan(bundleCall) - expect(mockQuery).toHaveBeenCalledWith( - 'SELECT pg_advisory_unlock(hashtext($1))', - ['cipherstash.eql.lifecycle'], + const beginCall = mockQuery.mock.calls.findIndex(([sql]) => sql === 'BEGIN') + const lockCall = mockQuery.mock.calls.findIndex( + ([sql]) => + sql === + 'SELECT pg_catalog.pg_try_advisory_xact_lock(pg_catalog.hashtext($1)) AS acquired', ) + expect(beginCall).toBeGreaterThan(-1) + expect(lockCall).toBeGreaterThan(beginCall) }) it('preserves the captured validity state when verifying rebuilt indexes', async () => { @@ -366,6 +373,8 @@ describe('EQLInstaller', () => { table_identity: 'app.users', valid: false, ready: false, + clustered: true, + cluster_sql: 'ALTER TABLE app.users CLUSTER ON users_email_idx', }, ], rowCount: 1, @@ -378,6 +387,7 @@ describe('EQLInstaller', () => { identity: 'app.users_email_idx', valid: false, ready: false, + clustered: true, definition: indexDefinition, }, ], @@ -391,9 +401,12 @@ describe('EQLInstaller', () => { await expect( new EQLInstaller({ databaseUrl: 'postgres://test' }).install(), ).resolves.toEqual({ deferredGrantsSql: null }) + expect(mockQuery).toHaveBeenCalledWith( + 'ALTER TABLE app.users CLUSTER ON users_email_idx', + ) }) - it('captures dependencies before the destructive install transaction', async () => { + it('captures dependencies before destructive SQL in the protected transaction', async () => { mockConnect.mockResolvedValue(undefined) mockEnd.mockResolvedValue(undefined) mockQuery.mockResolvedValue({ rows: [], rowCount: 0 }) @@ -414,8 +427,28 @@ describe('EQLInstaller', () => { expect(begin).toBeGreaterThan(-1) expect(capture).toBeGreaterThan(-1) expect(bundle).toBeGreaterThan(-1) - expect(capture).toBeLessThan(begin) - expect(begin).toBeLessThan(bundle) + expect(begin).toBeLessThan(capture) + expect(capture).toBeLessThan(bundle) + }) + + it('opens a transaction before setup failures use rollback narration', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + mockQuery.mockImplementation((sql: string) => { + if (sql === 'SET LOCAL jit = off') { + return Promise.reject(new Error('setting unavailable')) + } + return Promise.resolve({ rows: [], rowCount: 0 }) + }) + const { EQLInstaller } = await import('@/installer/index.ts') + + await expect( + new EQLInstaller({ databaseUrl: 'postgres://test' }).install(), + ).rejects.toThrow( + /Failed to install EQL: setting unavailable.*rolled back/s, + ) + expect(mockQuery).toHaveBeenCalledWith('BEGIN') + expect(mockQuery).toHaveBeenCalledWith('ROLLBACK') }) it('reports a bundle the parser cannot model without a transaction narration', async () => { @@ -443,6 +476,7 @@ describe('EQLInstaller', () => { expect(message).not.toContain('Failed to install EQL') expect(message).not.toContain('rolled back') expect(mockQuery).not.toHaveBeenCalledWith('BEGIN') + expect(mockQuery).not.toHaveBeenCalledWith('ROLLBACK') // A bundle this CLI cannot read is a local defect, like a failed digest // check: it must not reach the database at all. expect(mockConnect).not.toHaveBeenCalled() @@ -473,7 +507,8 @@ describe('EQLInstaller', () => { await expect(installer.install()).rejects.toThrow( /refused before making changes.*policy app\.users_visible/s, ) - expect(mockQuery).not.toHaveBeenCalledWith('BEGIN') + expect(mockQuery).toHaveBeenCalledWith('BEGIN') + expect(mockQuery).toHaveBeenCalledWith('ROLLBACK') expect( mockQuery.mock.calls.some( ([sql]) => @@ -498,6 +533,8 @@ describe('EQLInstaller', () => { table_identity: 'app.users', valid: true, ready: true, + clustered: false, + cluster_sql: null, }, ], rowCount: 1, diff --git a/packages/cli/src/installer/__tests__/reinstall.live.test.ts b/packages/cli/src/installer/__tests__/reinstall.live.test.ts index 5f921a816..6d9e0b7e4 100644 --- a/packages/cli/src/installer/__tests__/reinstall.live.test.ts +++ b/packages/cli/src/installer/__tests__/reinstall.live.test.ts @@ -10,14 +10,13 @@ */ import { afterAll, beforeEach, describe, expect, it } from 'vitest' +import { derivedSearchIndexRestorationTestSeam } from '../derived-search-index-restoration.js' import { EQLInstaller, loadBundledEqlSql } from '../index.js' -import { - acquireLifecycleLock, - LIFECYCLE_DEPENDENCIES_SQL, - releaseLifecycleLock, -} from '../reinstall.js' import { parseExpectedSurface } from '../verify.js' +const { acquireLifecycleLock, lifecycleDependenciesSql } = + derivedSearchIndexRestorationTestSeam + const DATABASE_URL = process.env.STASH_TEST_DATABASE_URL const describeLive = DATABASE_URL ? describe : describe.skip @@ -221,7 +220,7 @@ describeLive('EQLInstaller safe reinstall — live Postgres', () => { const rows = await queryOn<{ dependency_kind: string identity: string - }>(withEqlSearchPath(DATABASE_URL ?? ''), LIFECYCLE_DEPENDENCIES_SQL, [ + }>(withEqlSearchPath(DATABASE_URL ?? ''), lifecycleDependenciesSql, [ expected.operators, expected.casts, ]) @@ -259,7 +258,7 @@ describeLive('EQLInstaller safe reinstall — live Postgres', () => { const rows = await queryOn<{ dependency_kind: string; identity: string }>( withEqlSearchPath(DATABASE_URL ?? ''), - LIFECYCLE_DEPENDENCIES_SQL, + lifecycleDependenciesSql, [expected.operators.filter((o) => !hasBareOperand(o)), expected.casts], ) @@ -335,7 +334,7 @@ describeLive('EQLInstaller safe reinstall — live Postgres', () => { }, 180_000) /** - * `pg_advisory_lock` waits forever. A concurrent install, or a session that + * `pg_advisory_xact_lock` waits forever. A concurrent install, or a session that * died holding the lock, then makes the command hang with no output — * indistinguishable from a network stall. */ @@ -347,14 +346,16 @@ describeLive('EQLInstaller safe reinstall — live Postgres', () => { await blocked.connect() try { // A regression here is a HANG, not a failure — a blocking - // `pg_advisory_lock` never returns, the `finally` below never runs, and + // `pg_advisory_xact_lock` never returns, the `finally` below never runs, and // the leaked lock then blocks every later `install()` in this file. The // timeout turns that into a failed assertion. It is inert once the - // acquire polls with `pg_try_advisory_lock`, which never waits. + // acquire polls with `pg_try_advisory_xact_lock`, which never waits. await blocked.query("SET statement_timeout = '10s'") + await holder.query('BEGIN') await holder.query( - "SELECT pg_advisory_lock(hashtext('cipherstash.eql.lifecycle'))", + "SELECT pg_catalog.pg_advisory_xact_lock(pg_catalog.hashtext('cipherstash.eql.lifecycle'))", ) + await blocked.query('BEGIN') // An explicit short budget: the behaviour under test is "refuses rather // than hangs", which does not depend on how long the wait is, and the @@ -364,16 +365,14 @@ describeLive('EQLInstaller safe reinstall — live Postgres', () => { await expect(acquireLifecycleLock(blocked, 1_000)).rejects.toThrow( /another EQL lifecycle operation is in progress/i, ) - // The failed acquire holds nothing, so the caller's unconditional - // release in `finally` has to be a no-op rather than an unlock of a lock - // this session never took. - await expect(releaseLifecycleLock(blocked)).resolves.toBeUndefined() - - await holder.query('SELECT pg_advisory_unlock_all()') + await blocked.query('ROLLBACK') + await holder.query('COMMIT') + await blocked.query('BEGIN') await expect(acquireLifecycleLock(blocked)).resolves.toBeUndefined() - await releaseLifecycleLock(blocked) + await blocked.query('COMMIT') } finally { - await holder.query('SELECT pg_advisory_unlock_all()').catch(() => {}) + await blocked.query('ROLLBACK').catch(() => {}) + await holder.query('ROLLBACK').catch(() => {}) await blocked.end().catch(() => undefined) await holder.end().catch(() => undefined) } diff --git a/packages/cli/src/installer/reinstall.ts b/packages/cli/src/installer/derived-search-index-restoration.ts similarity index 81% rename from packages/cli/src/installer/reinstall.ts rename to packages/cli/src/installer/derived-search-index-restoration.ts index 3bbc2c353..678b10ecf 100644 --- a/packages/cli/src/installer/reinstall.ts +++ b/packages/cli/src/installer/derived-search-index-restoration.ts @@ -1,22 +1,12 @@ import type pg from 'pg' +import { createPgClient, TlsVerificationError } from '@/db/client.js' + const LIFECYCLE_LOCK = 'cipherstash.eql.lifecycle' /** * How long to keep trying for the lifecycle lock before giving up. * - * Sized off what actually holds it: the whole install, which is a - * `DROP SCHEMA … CASCADE` plus ~3,000 object creations and measures 10-30s - * against a local container — longer on a managed Postgres. A budget of a few - * seconds looks generous and is not; it converts ordinary queueing (two - * installs, a CI job overlapping a developer) into a hard failure on the second - * one. That regression is real and was caught by the live suite: with 5s, two - * live files installing back to back failed each other. - * - * So: long enough that anything a waiter would sensibly wait for still - * succeeds, and bounded so a lock nobody will release reports itself instead of - * looking like a stalled connection. - * * The budget is sized off the INSTALL, which is what actually holds the lock: * `DROP SCHEMA ... CASCADE` plus ~3,000 object creations, 10-30s against a * local container and longer on managed Postgres. The dependency capture @@ -24,8 +14,7 @@ const LIFECYCLE_LOCK = 'cipherstash.eql.lifecycle' * cost. Its recursive shape inflates PostgreSQL's estimate enough to trigger * JIT compilation of hundreds of expressions: measured on an idle local * container with EQL installed, the query itself is ~35ms with JIT disabled - * versus ~880ms with JIT enabled. `install()` disables JIT on its dedicated - * connection before running it. + * versus ~880ms with JIT enabled. `install()` disables JIT for its transaction. * * Five minutes is deliberate margin over that, not a measurement: a waiter * should never be refused for ordinary queueing, only for a holder that will @@ -37,8 +26,25 @@ const LIFECYCLE_LOCK = 'cipherstash.eql.lifecycle' const LOCK_WAIT_MS = 300_000 const LOCK_RETRY_INTERVAL_MS = 250 -/** Connections currently holding the lifecycle lock through this module. */ -const lockHolders = new WeakSet() +export class EqlReinstallRefusalError extends Error {} + +export class EqlReinstallConnectionError extends Error {} + +export class DerivedSearchIndexReconstructionError extends Error {} + +export class DerivedSearchIndexVerificationError extends Error {} + +export interface RestorationSummary { + restoredIndexes: number + analyzedTables: number +} + +interface RestoreAroundEqlReplacementOptions { + databaseUrl: string + bundledSql: string + bundleOperators: string[] + bundleCasts: string[] +} export interface ReinstallIndex { identity: string @@ -46,6 +52,8 @@ export interface ReinstallIndex { tableIdentity: string valid: boolean ready: boolean + clustered: boolean + clusterSql: string | null } interface DependencyRow { @@ -55,9 +63,11 @@ interface DependencyRow { table_identity?: unknown valid?: unknown ready?: unknown + clustered?: unknown + cluster_sql?: unknown } -export const LIFECYCLE_DEPENDENCIES_SQL = ` +const LIFECYCLE_DEPENDENCIES_SQL = ` /* stash_eql_lifecycle_dependencies */ WITH RECURSIVE /* @@ -322,13 +332,9 @@ external_dependants AS ( ) ) /* - * \`i\` is an ordinary index; \`I\` is a PARTITIONED index — the parent template on - * a partitioned table. Both are rebuildable, but only together: the parent's - * \`pg_get_indexdef\` says \`ON ONLY\`, its per-partition children come back as - * separate \`i\` rows, and the parent stays \`indisvalid = false\` until every one - * of them is re-ATTACHed. \`parent_identity\` (from pg_inherits, which relates an - * attached child index to its parent) is what carries that edge across the - * DROP; see {@link rebuildIndexes}. + * Only standalone ordinary indexes are reconstructed. Partitioned index parents + * (\`I\`) and attached child indexes are classified as unsafe because recreating + * their attachment graph requires metadata beyond \`pg_get_indexdef\`. */ SELECT DISTINCT CASE @@ -364,7 +370,24 @@ SELECT DISTINCT WHEN e.classid = 'pg_catalog.pg_class'::regclass AND index_class.relkind = 'i' AND index_partition.inhrelid IS NULL THEN index_meta.indisready - END AS ready + END AS ready, + CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL THEN index_meta.indisclustered + END AS clustered, + CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL + AND index_meta.indisclustered + THEN pg_catalog.format( + 'ALTER TABLE %I.%I CLUSTER ON %I', + table_namespace.nspname, + table_class.relname, + index_class.relname + ) + END AS cluster_sql FROM external_dependants e LEFT JOIN pg_catalog.pg_class index_class ON e.classid = 'pg_catalog.pg_class'::regclass AND index_class.oid = e.objid @@ -382,6 +405,7 @@ const VERIFY_REBUILT_INDEXES_SQL = ` SELECT pg_catalog.format('%I.%I', n.nspname, c.relname) AS identity, i.indisvalid AS valid, i.indisready AS ready, + i.indisclustered AS clustered, pg_catalog.pg_get_indexdef(i.indexrelid) AS definition FROM pg_catalog.pg_index i JOIN pg_catalog.pg_class c ON c.oid = i.indexrelid @@ -392,16 +416,16 @@ WHERE pg_catalog.format('%I.%I', n.nspname, c.relname) = ANY($1::text[]) /** * Take the installer's advisory lock, or say why not. * - * `pg_advisory_lock` waits forever. A concurrent `stash eql install`, or a + * `pg_advisory_xact_lock` waits forever. A concurrent `stash eql install`, or a * session that died holding the lock, then makes the command hang with no * output and no timeout — indistinguishable from a network stall, and the one - * failure a user cannot diagnose. Polling `pg_try_advisory_lock` against a + * failure a user cannot diagnose. Polling `pg_try_advisory_xact_lock` against a * bounded budget keeps the ordinary case working — a queued install still wins * the lock and runs — while turning the pathological one into a sentence. See - * {@link LOCK_WAIT_MS} for why the budget is a minute and not the handful of + * {@link LOCK_WAIT_MS} for why the five-minute budget is not the handful of * seconds it first was. (#959) */ -export async function acquireLifecycleLock( +async function acquireLifecycleLock( client: pg.ClientBase, // Optional so `acquireLifecycleLock(client)` keeps meaning what it did. The // budget is a policy, not a constant of nature, and the live suite drives it @@ -411,11 +435,10 @@ export async function acquireLifecycleLock( const deadline = Date.now() + waitMs for (;;) { const result = await client.query<{ acquired: boolean }>( - 'SELECT pg_try_advisory_lock(hashtext($1)) AS acquired', + 'SELECT pg_catalog.pg_try_advisory_xact_lock(pg_catalog.hashtext($1)) AS acquired', [LIFECYCLE_LOCK], ) if (result.rows[0]?.acquired === true) { - lockHolders.add(client) return } if (Date.now() >= deadline) { @@ -429,22 +452,7 @@ export async function acquireLifecycleLock( } } -/** - * Release the lock if this connection took it. The caller unlocks - * unconditionally in a `finally`, which includes the path where the acquire - * above refused — and `pg_advisory_unlock` on a lock a session never held - * returns false and raises a WARNING, i.e. noise on the one path that already - * has a clear message. - */ -export async function releaseLifecycleLock(client: pg.ClientBase) { - if (!lockHolders.has(client)) return - lockHolders.delete(client) - await client.query('SELECT pg_advisory_unlock(hashtext($1))', [ - LIFECYCLE_LOCK, - ]) -} - -export async function inspectReinstallDependencies( +async function inspectReinstallDependencies( client: pg.ClientBase, bundleOperators: string[], bundleCasts: string[], @@ -465,7 +473,10 @@ export async function inspectReinstallDependencies( typeof row.definition !== 'string' || typeof row.table_identity !== 'string' || typeof row.valid !== 'boolean' || - typeof row.ready !== 'boolean' + typeof row.ready !== 'boolean' || + typeof row.clustered !== 'boolean' || + (row.cluster_sql !== null && typeof row.cluster_sql !== 'string') || + (row.clustered && typeof row.cluster_sql !== 'string') ) { unsafe.push( String(row.identity ?? 'index with incomplete catalog metadata'), @@ -478,10 +489,12 @@ export async function inspectReinstallDependencies( tableIdentity: row.table_identity, valid: row.valid, ready: row.ready, + clustered: row.clustered, + clusterSql: row.cluster_sql, }) } if (unsafe.length > 0) { - throw new Error( + throw new EqlReinstallRefusalError( `EQL reinstall refused before making changes because customer-owned database objects depend on disposable EQL machinery and cannot be reconstructed safely:\n${unsafe.map((identity) => ` - ${identity}`).join('\n')}`, ) } @@ -499,7 +512,7 @@ export async function inspectReinstallDependencies( * rebuilt expression binds to the same function the original did. * */ -export async function rebuildIndexes( +async function rebuildIndexes( client: pg.ClientBase, indexes: ReinstallIndex[], ) { @@ -508,18 +521,22 @@ export async function rebuildIndexes( await client.query(index.definition) } catch (error) { const detail = error instanceof Error ? error.message : String(error) - throw new Error( + throw new DerivedSearchIndexReconstructionError( `EQL reinstall could not rebuild search index ${index.identity}: ${detail}\nThe transaction will restore the previous EQL installation and index. Captured index SQL:\n${index.definition}`, { cause: error }, ) } } - for (const tableIdentity of new Set( - indexes.map((index) => index.tableIdentity), - )) { + for (const index of indexes) { + if (index.clusterSql !== null) await client.query(index.clusterSql) + } + const tableIdentities = new Set(indexes.map((index) => index.tableIdentity)) + for (const tableIdentity of tableIdentities) { await client.query(`ANALYZE ${tableIdentity}`) } - if (indexes.length === 0) return + if (indexes.length === 0) { + return { restoredIndexes: 0, analyzedTables: 0 } + } const result = await client.query(VERIFY_REBUILT_INDEXES_SQL, [ indexes.map((index) => index.identity), ]) @@ -544,15 +561,76 @@ export async function rebuildIndexes( return ( expected.definition === row.definition && (row.valid === true || expected.valid === false) && - (row.ready === true || expected.ready === false) + (row.ready === true || expected.ready === false) && + row.clustered === expected.clustered ) }) .map((row) => String(row.identity)), ) const unhealthy = indexes.filter((index) => !healthy.has(index.identity)) if (unhealthy.length > 0) { - throw new Error( + throw new DerivedSearchIndexVerificationError( `EQL reinstall produced missing, invalid, or changed search indexes; the transaction will restore the previous installation:\n${unhealthy.map((index) => ` - ${index.identity}\n ${index.definition}`).join('\n')}`, ) } + return { + restoredIndexes: indexes.length, + analyzedTables: tableIdentities.size, + } +} + +/** + * Replace EQL machinery while preserving every reconstructable derived search + * index in one transaction. Catalog discovery, reconstruction details, and + * verification remain private so callers cannot execute the protocol out of + * order or retain stale captured state. + */ +export async function restoreDerivedSearchIndexesAroundEqlReplacement({ + databaseUrl, + bundledSql, + bundleOperators, + bundleCasts, +}: RestoreAroundEqlReplacementOptions): Promise { + const client = createPgClient(databaseUrl) + try { + await client.connect() + } catch (error) { + await client.end().catch(() => {}) + if (error instanceof TlsVerificationError) throw error + const detail = error instanceof Error ? error.message : String(error) + throw new EqlReinstallConnectionError( + `Failed to connect to database: ${detail}`, + { + cause: error, + }, + ) + } + + try { + await client.query('BEGIN') + try { + await acquireLifecycleLock(client) + await client.query('SET LOCAL jit = off') + const indexes = await inspectReinstallDependencies( + client, + bundleOperators, + bundleCasts, + ) + await client.query(bundledSql) + const summary = await rebuildIndexes(client, indexes) + await client.query('COMMIT') + return summary + } catch (error) { + await client.query('ROLLBACK').catch(() => {}) + throw error + } + } finally { + await client.end() + } +} + +/** @internal Direct catalog probes retained only for live PostgreSQL evidence. */ +export const derivedSearchIndexRestorationTestSeam = { + acquireLifecycleLock, + lifecycleDependenciesSql: LIFECYCLE_DEPENDENCIES_SQL, } diff --git a/packages/cli/src/installer/index.ts b/packages/cli/src/installer/index.ts index 737ecba33..cfc8690fe 100644 --- a/packages/cli/src/installer/index.ts +++ b/packages/cli/src/installer/index.ts @@ -2,6 +2,11 @@ import { readInstallSql } from '@cipherstash/eql/sql' import type pg from 'pg' import { createPgClient, TlsVerificationError } from '@/db/client.js' import { assertBundledEqlSqlDigest } from './bundle-digest.js' +import { + EqlReinstallConnectionError, + EqlReinstallRefusalError, + restoreDerivedSearchIndexesAroundEqlReplacement, +} from './derived-search-index-restoration.js' import { DEFERRED_GRANTS_HEADER, EQL_V3_INTERNAL_SCHEMA_NAME, @@ -10,12 +15,6 @@ import { SUPABASE_IMMEDIATE_GRANTS_SQL_V3, SUPABASE_PERMISSIONS_SQL_V3, } from './grants.js' -import { - acquireLifecycleLock, - inspectReinstallDependencies, - rebuildIndexes, - releaseLifecycleLock, -} from './reinstall.js' export { DEFERRED_GRANTS_HEADER, @@ -450,66 +449,38 @@ export class EQLInstaller { // digest-checked bundle loader. const { parseExpectedSurface } = await import('./verify.js') const expected = parseExpectedSurface(bundledSql) - const client = createPgClient(this.databaseUrl) try { - await client.connect() + await restoreDerivedSearchIndexesAroundEqlReplacement({ + databaseUrl: this.databaseUrl, + bundledSql, + bundleOperators: expected.operators, + bundleCasts: expected.casts, + }) } catch (error) { - if (error instanceof TlsVerificationError) throw error + if ( + error instanceof TlsVerificationError || + error instanceof EqlReinstallConnectionError || + error instanceof EqlReinstallRefusalError + ) { + throw error + } const detail = error instanceof Error ? error.message : String(error) - throw new Error(`Failed to connect to database: ${detail}`, { - cause: error, - }) + throw new Error( + `Failed to install EQL: ${detail}. Nothing was applied — the install runs in a transaction and was rolled back.`, + { cause: error }, + ) } - try { - await acquireLifecycleLock(client) - try { - // This catalogue query has a very high estimated cost because its - // recursive walk carries every dependency edge. PostgreSQL otherwise - // JIT-compiles hundreds of tiny expressions (~880ms locally) for work - // that executes in ~35ms interpreted. This connection belongs solely - // to this install and is closed below, so the session setting cannot - // leak into application queries. - await client.query('SET jit = off') - const indexes = await inspectReinstallDependencies( - client, - expected.operators, - expected.casts, - ) - // Keep catalogue discovery outside the DDL transaction. The EQL - // bundle creates thousands of objects and already approaches - // max_locks_per_transaction; retaining the discovery query's catalogue - // locks across it can exhaust PostgreSQL's shared lock table. The - // documented maintenance window excludes unrelated application DDL - // between discovery and replacement. - await client.query('BEGIN') - await client.query(bundledSql) - await rebuildIndexes(client, indexes) - await client.query('COMMIT') - } catch (error) { - await client.query('ROLLBACK').catch(() => {}) - const detail = error instanceof Error ? error.message : String(error) - if (detail.startsWith('EQL reinstall refused')) throw error - throw new Error( - `Failed to install EQL: ${detail}. Nothing was applied — the install runs in a transaction and was rolled back.`, - { cause: error }, - ) - } - - if (!options?.supabase) return { deferredGrantsSql: null } + if (!options?.supabase) return { deferredGrantsSql: null } - try { - return await this.runSupabaseGrants(client) - } catch (error) { - const detail = error instanceof Error ? error.message : String(error) - throw new Error( - `EQL v3 is installed, but granting the Supabase roles failed: ${detail}. The install itself was NOT rolled back — re-run \`stash eql install --force\` (or plain \`stash eql install\`, which re-applies the grants on an already-installed database).`, - { cause: error }, - ) - } - } finally { - await releaseLifecycleLock(client).catch(() => {}) - await client.end() + try { + return await this.applySupabaseGrants() + } catch (error) { + const detail = error instanceof Error ? error.message : String(error) + throw new Error( + `EQL v3 is installed, but granting the Supabase roles failed: ${detail}. The install itself was NOT rolled back — re-run \`stash eql install --force\` (or plain \`stash eql install\`, which re-applies the grants on an already-installed database).`, + { cause: error }, + ) } } From 05fa9927e79249689092271221f5a07da818515e Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Tue, 1 Sep 2026 12:53:04 +1000 Subject: [PATCH 04/11] refactor(cli): deepen verified EQL bundle --- packages/cli/src/__tests__/installer.test.ts | 4 +- .../installer/__tests__/bundle-digest.test.ts | 12 +++++ packages/cli/src/installer/eql-bundle.ts | 23 ++++++++++ packages/cli/src/installer/index.ts | 44 +++++-------------- packages/cli/src/installer/verify.ts | 25 ++++++++--- 5 files changed, 66 insertions(+), 42 deletions(-) create mode 100644 packages/cli/src/installer/eql-bundle.ts diff --git a/packages/cli/src/__tests__/installer.test.ts b/packages/cli/src/__tests__/installer.test.ts index 1e096d5b3..d0d5f559c 100644 --- a/packages/cli/src/__tests__/installer.test.ts +++ b/packages/cli/src/__tests__/installer.test.ts @@ -38,9 +38,9 @@ vi.mock('../installer/verify.js', async (importOriginal) => { const actual = await importOriginal() return { ...actual, - parseExpectedSurface: (sql: string) => { + loadVerifiedEqlBundle: () => { if (parseFailure.error) throw parseFailure.error - return actual.parseExpectedSurface(sql) + return actual.loadVerifiedEqlBundle() }, } }) diff --git a/packages/cli/src/installer/__tests__/bundle-digest.test.ts b/packages/cli/src/installer/__tests__/bundle-digest.test.ts index 4bf8477a7..430100699 100644 --- a/packages/cli/src/installer/__tests__/bundle-digest.test.ts +++ b/packages/cli/src/installer/__tests__/bundle-digest.test.ts @@ -59,6 +59,18 @@ describe('bundled EQL SQL digest verification', () => { ) }) + it('presents verified SQL and its derived surface as one artifact', async () => { + eqlSql.tampered = null + const { loadVerifiedEqlBundle } = await import('@/installer/verify.ts') + const { releaseManifest } = await import('@cipherstash/eql/sql') + + const bundle = loadVerifiedEqlBundle() + + expect(bundle.sql).toContain('CREATE SCHEMA eql_v3') + expect(bundle.expectedSurface.eqlVersion).toBe(releaseManifest.eqlVersion) + expect(bundle.expectedSurface.operators.length).toBeGreaterThan(2000) + }) + it('refuses SQL whose bytes do not hash to the manifest digest', async () => { eqlSql.tampered = TAMPERED const { loadBundledEqlSql } = await import('@/installer/index.ts') diff --git a/packages/cli/src/installer/eql-bundle.ts b/packages/cli/src/installer/eql-bundle.ts new file mode 100644 index 000000000..ead2fd6d6 --- /dev/null +++ b/packages/cli/src/installer/eql-bundle.ts @@ -0,0 +1,23 @@ +import { readInstallSql } from '@cipherstash/eql/sql' +import { assertBundledEqlSqlDigest } from './bundle-digest.js' + +/** Schemas in which the pinned EQL bundle can resolve pgcrypto safely. */ +export const SUPPORTED_PGCRYPTO_SCHEMAS = ['extensions', 'public'] + +/** + * Read the pinned EQL installer and prove its bytes match the resolved release. + * Keep this file free of installer and verifier imports: both consume the same + * artifact, and neither should become the other's dependency. + */ +export function loadBundledEqlSql(): string { + let sql: string + try { + sql = readInstallSql() + } catch (error) { + throw new Error( + 'Failed to read the EQL v3 install SQL from `@cipherstash/eql`. Reinstall dependencies (the package ships the bundle in `dist/sql/`).', + { cause: error }, + ) + } + return assertBundledEqlSqlDigest(sql) +} diff --git a/packages/cli/src/installer/index.ts b/packages/cli/src/installer/index.ts index cfc8690fe..22c94e176 100644 --- a/packages/cli/src/installer/index.ts +++ b/packages/cli/src/installer/index.ts @@ -1,12 +1,11 @@ -import { readInstallSql } from '@cipherstash/eql/sql' import type pg from 'pg' import { createPgClient, TlsVerificationError } from '@/db/client.js' -import { assertBundledEqlSqlDigest } from './bundle-digest.js' import { EqlReinstallConnectionError, EqlReinstallRefusalError, restoreDerivedSearchIndexesAroundEqlReplacement, } from './derived-search-index-restoration.js' +import { SUPPORTED_PGCRYPTO_SCHEMAS } from './eql-bundle.js' import { DEFERRED_GRANTS_HEADER, EQL_V3_INTERNAL_SCHEMA_NAME, @@ -15,6 +14,12 @@ import { SUPABASE_IMMEDIATE_GRANTS_SQL_V3, SUPABASE_PERMISSIONS_SQL_V3, } from './grants.js' +import { loadVerifiedEqlBundle } from './verify.js' + +export { + loadBundledEqlSql, + SUPPORTED_PGCRYPTO_SCHEMAS, +} from './eql-bundle.js' export { DEFERRED_GRANTS_HEADER, @@ -48,19 +53,6 @@ const EQL_V2_SCHEMA_NAME = 'eql_v2' * @throws if the bundle cannot be read, or if its bytes are not the ones the * resolved release attests to (see {@link assertBundledEqlSqlDigest}). */ -export function loadBundledEqlSql(): string { - let sql: string - try { - sql = readInstallSql() - } catch (error) { - throw new Error( - 'Failed to read the EQL v3 install SQL from `@cipherstash/eql`. Reinstall dependencies (the package ships the bundle in `dist/sql/`).', - { cause: error }, - ) - } - return assertBundledEqlSqlDigest(sql) -} - /** Supabase grants for the sole installable generation, EQL v3. */ export function supabaseGrantsFor(): string { return SUPABASE_PERMISSIONS_SQL_V3 @@ -177,8 +169,6 @@ const PREFLIGHT_SQL = ` ` /** The schemas the pinned bundle accepts `pgcrypto` in (its search_path). */ -export const SUPPORTED_PGCRYPTO_SCHEMAS = ['extensions', 'public'] - /** * Can this role create the ORE btree operator class? (#891) * @@ -436,25 +426,13 @@ export class EQLInstaller { // was attempted and rolled back". It also keeps the digest message the // whole error, rather than a `detail` interpolated into the install // wrapper's transaction narration below. - const bundledSql = loadBundledEqlSql() - // Parsed here for the same reason, and deliberately outside the try below: - // a parse failure is a bundle this CLI has outgrown, and the parser's - // message names the offending statement. Interpolated into the install - // wrapper it would be narrated as a rolled-back transaction that was never - // opened, pointing at the database instead of at the bundle. - // - // `parseExpectedSurface(bundledSql)` rather than `bundledExpectedSurface()`, - // which would re-read and re-digest the bundle we already hold. Dynamic to - // avoid verify.ts's intentional import of this module for the - // digest-checked bundle loader. - const { parseExpectedSurface } = await import('./verify.js') - const expected = parseExpectedSurface(bundledSql) + const bundle = loadVerifiedEqlBundle() try { await restoreDerivedSearchIndexesAroundEqlReplacement({ databaseUrl: this.databaseUrl, - bundledSql, - bundleOperators: expected.operators, - bundleCasts: expected.casts, + bundledSql: bundle.sql, + bundleOperators: bundle.expectedSurface.operators, + bundleCasts: bundle.expectedSurface.casts, }) } catch (error) { if ( diff --git a/packages/cli/src/installer/verify.ts b/packages/cli/src/installer/verify.ts index e0c69bd1d..e9f4dceaf 100644 --- a/packages/cli/src/installer/verify.ts +++ b/packages/cli/src/installer/verify.ts @@ -1,8 +1,8 @@ import { releaseManifest } from '@cipherstash/eql/sql' import type pg from 'pg' import { createPgClient, TlsVerificationError } from '@/db/client.js' +import { loadBundledEqlSql, SUPPORTED_PGCRYPTO_SCHEMAS } from './eql-bundle.js' import { EQL_V3_INTERNAL_SCHEMA_NAME, EQL_V3_SCHEMA_NAME } from './grants.js' -import { loadBundledEqlSql, SUPPORTED_PGCRYPTO_SCHEMAS } from './index.js' import { classifyOreState, describeOreState, @@ -61,6 +61,12 @@ export interface ExpectedSurface { oreDomains: string[] } +/** The digest-verified installer bytes and the complete surface derived from them. */ +export interface VerifiedEqlBundle { + sql: string + expectedSurface: ExpectedSurface +} + /** * How the ORE half of the install reads. Defined in `./ore.js`, which owns the * whole ORE model — the catalogue probe, the state machine, and the copy every @@ -353,6 +359,16 @@ export function parseExpectedSurface(sql: string): ExpectedSurface { } } +/** + * Load the one verified artifact consumed by installation, restoration, and + * verification. Completeness parsing happens here so callers cannot combine + * SQL bytes with metadata derived from a different bundle. + */ +export function loadVerifiedEqlBundle(): VerifiedEqlBundle { + const sql = loadBundledEqlSql() + return { sql, expectedSurface: parseExpectedSurface(sql) } +} + /** The expected surface of the pinned bundle this CLI installs. */ export function bundledExpectedSurface(): ExpectedSurface { // Through `loadBundledEqlSql()` rather than `readInstallSql()` directly, so @@ -360,12 +376,7 @@ export function bundledExpectedSurface(): ExpectedSurface { // database against this expectation — derived from an unverified bundle it // would answer a different question than the one asked, and could report a // healthy install as broken (or the reverse) from tampered bytes alone. - const sql = loadBundledEqlSql() - // Deliberately outside any try: a parse failure is a bundle the parser has - // outgrown ({@link assertEveryStatementModelled}), and its message names the - // statement. Wrapping it in "reinstall dependencies" would send whoever hits - // it to the one remedy that cannot help. - return parseExpectedSurface(sql) + return loadVerifiedEqlBundle().expectedSurface } // --------------------------------------------------------------------------- From 3e442a42b244c3f96e0e38ff4a1ea781a532b336 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Tue, 1 Sep 2026 12:56:14 +1000 Subject: [PATCH 05/11] refactor(cli): deepen installer test scenarios --- packages/cli/src/__tests__/installer.test.ts | 239 +++++------------- .../__tests__/reinstall.live.test.ts | 70 ++--- .../__tests__/restoration-scenarios.ts | 162 ++++++++++++ 3 files changed, 253 insertions(+), 218 deletions(-) create mode 100644 packages/cli/src/installer/__tests__/restoration-scenarios.ts diff --git a/packages/cli/src/__tests__/installer.test.ts b/packages/cli/src/__tests__/installer.test.ts index d0d5f559c..a85473bec 100644 --- a/packages/cli/src/__tests__/installer.test.ts +++ b/packages/cli/src/__tests__/installer.test.ts @@ -1,4 +1,8 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { + RecordingRestorationDatabase, + searchIndexRestorationScenario, +} from '../installer/__tests__/restoration-scenarios.js' const mockConnect = vi.fn() const mockQuery = vi.fn() @@ -290,156 +294,73 @@ describe('EQLInstaller', () => { it('captures, rebuilds, and verifies functional indexes around reinstall', async () => { mockConnect.mockResolvedValue(undefined) mockEnd.mockResolvedValue(undefined) - const indexDefinition = - 'CREATE INDEX users_email_idx ON app.users USING btree (eql_v3.eq_term(email))' - mockQuery.mockImplementation((sql: string) => { - if (sql.includes('stash_eql_lifecycle_dependencies')) { - return Promise.resolve({ - rows: [ - { - dependency_kind: 'index', - identity: 'app.users_email_idx', - definition: indexDefinition, - table_identity: 'app.users', - valid: true, - ready: true, - clustered: false, - cluster_sql: null, - }, - ], - rowCount: 1, - }) - } - if (sql.includes('stash_eql_verify_rebuilt_indexes')) { - return Promise.resolve({ - rows: [ - { - identity: 'app.users_email_idx', - valid: true, - ready: true, - clustered: false, - definition: indexDefinition, - }, - ], - rowCount: 1, - }) - } - return Promise.resolve({ rows: [], rowCount: 0 }) - }) + const database = new RecordingRestorationDatabase( + searchIndexRestorationScenario(), + ) + mockQuery.mockImplementation(database.query) const { EQLInstaller } = await import('@/installer/index.ts') const installer = new EQLInstaller({ databaseUrl: 'postgres://test' }) await installer.install() - expect(mockQuery).toHaveBeenCalledWith( - 'SELECT pg_catalog.pg_try_advisory_xact_lock(pg_catalog.hashtext($1)) AS acquired', - ['cipherstash.eql.lifecycle'], - ) - expect(mockQuery).toHaveBeenCalledWith('SET LOCAL jit = off') - expect(mockQuery).toHaveBeenCalledWith(indexDefinition) - expect(mockQuery).toHaveBeenCalledWith('ANALYZE app.users') - const bundleCall = mockQuery.mock.calls.findIndex( - ([sql]) => - typeof sql === 'string' && sql.includes('CREATE SCHEMA eql_v3'), - ) - const rebuildCall = mockQuery.mock.calls.findIndex( - ([sql]) => sql === indexDefinition, - ) - expect(bundleCall).toBeGreaterThan(-1) - expect(rebuildCall).toBeGreaterThan(bundleCall) - const beginCall = mockQuery.mock.calls.findIndex(([sql]) => sql === 'BEGIN') - const lockCall = mockQuery.mock.calls.findIndex( - ([sql]) => - sql === - 'SELECT pg_catalog.pg_try_advisory_xact_lock(pg_catalog.hashtext($1)) AS acquired', - ) - expect(beginCall).toBeGreaterThan(-1) - expect(lockCall).toBeGreaterThan(beginCall) + expect(database.events).toEqual([ + 'begin', + 'lock', + 'configure', + 'capture', + 'replace', + 'reconstruct', + 'analyze', + 'verify', + 'commit', + ]) }) it('preserves the captured validity state when verifying rebuilt indexes', async () => { mockConnect.mockResolvedValue(undefined) mockEnd.mockResolvedValue(undefined) - const indexDefinition = - 'CREATE INDEX users_email_idx ON app.users USING btree (eql_v3.eq_term(email))' - mockQuery.mockImplementation((sql: string) => { - if (sql.includes('stash_eql_lifecycle_dependencies')) { - return Promise.resolve({ - rows: [ - { - dependency_kind: 'index', - identity: 'app.users_email_idx', - definition: indexDefinition, - table_identity: 'app.users', - valid: false, - ready: false, - clustered: true, - cluster_sql: 'ALTER TABLE app.users CLUSTER ON users_email_idx', - }, - ], - rowCount: 1, - }) - } - if (sql.includes('stash_eql_verify_rebuilt_indexes')) { - return Promise.resolve({ - rows: [ - { - identity: 'app.users_email_idx', - valid: false, - ready: false, - clustered: true, - definition: indexDefinition, - }, - ], - rowCount: 1, - }) - } - return Promise.resolve({ rows: [], rowCount: 0 }) - }) + const database = new RecordingRestorationDatabase( + searchIndexRestorationScenario({ + valid: false, + ready: false, + clustered: true, + clusterSql: 'ALTER TABLE app.users CLUSTER ON users_email_idx', + }), + ) + mockQuery.mockImplementation(database.query) const { EQLInstaller } = await import('@/installer/index.ts') await expect( new EQLInstaller({ databaseUrl: 'postgres://test' }).install(), ).resolves.toEqual({ deferredGrantsSql: null }) - expect(mockQuery).toHaveBeenCalledWith( - 'ALTER TABLE app.users CLUSTER ON users_email_idx', - ) + expect(database.events).toContain('cluster') }) it('captures dependencies before destructive SQL in the protected transaction', async () => { mockConnect.mockResolvedValue(undefined) mockEnd.mockResolvedValue(undefined) - mockQuery.mockResolvedValue({ rows: [], rowCount: 0 }) + const database = new RecordingRestorationDatabase() + mockQuery.mockImplementation(database.query) const { EQLInstaller } = await import('@/installer/index.ts') await new EQLInstaller({ databaseUrl: 'postgres://test' }).install() - const statements = mockQuery.mock.calls.map(([sql]) => - typeof sql === 'string' ? sql : '', - ) - const begin = statements.indexOf('BEGIN') - const capture = statements.findIndex((sql) => - sql.includes('stash_eql_lifecycle_dependencies'), - ) - const bundle = statements.findIndex((sql) => - sql.includes('CREATE SCHEMA eql_v3'), - ) - expect(begin).toBeGreaterThan(-1) - expect(capture).toBeGreaterThan(-1) - expect(bundle).toBeGreaterThan(-1) - expect(begin).toBeLessThan(capture) - expect(capture).toBeLessThan(bundle) + expect(database.events.slice(0, 5)).toEqual([ + 'begin', + 'lock', + 'configure', + 'capture', + 'replace', + ]) }) it('opens a transaction before setup failures use rollback narration', async () => { mockConnect.mockResolvedValue(undefined) mockEnd.mockResolvedValue(undefined) - mockQuery.mockImplementation((sql: string) => { - if (sql === 'SET LOCAL jit = off') { - return Promise.reject(new Error('setting unavailable')) - } - return Promise.resolve({ rows: [], rowCount: 0 }) + const database = new RecordingRestorationDatabase(undefined, { + failConfigurationWith: new Error('setting unavailable'), }) + mockQuery.mockImplementation(database.query) const { EQLInstaller } = await import('@/installer/index.ts') await expect( @@ -447,8 +368,7 @@ describe('EQLInstaller', () => { ).rejects.toThrow( /Failed to install EQL: setting unavailable.*rolled back/s, ) - expect(mockQuery).toHaveBeenCalledWith('BEGIN') - expect(mockQuery).toHaveBeenCalledWith('ROLLBACK') + expect(database.events).toEqual(['begin', 'lock', 'configure', 'rollback']) }) it('reports a bundle the parser cannot model without a transaction narration', async () => { @@ -485,66 +405,36 @@ describe('EQLInstaller', () => { it('refuses before mutation when a dependency cannot be reconstructed', async () => { mockConnect.mockResolvedValue(undefined) mockEnd.mockResolvedValue(undefined) - mockQuery.mockImplementation((sql: string) => { - if (sql.includes('stash_eql_lifecycle_dependencies')) { - return Promise.resolve({ - rows: [ - { - dependency_kind: 'unsafe', - identity: 'policy app.users_visible', - definition: null, - table_identity: null, - }, - ], - rowCount: 1, - }) - } - return Promise.resolve({ rows: [], rowCount: 0 }) + const database = new RecordingRestorationDatabase(undefined, { + unsafeIdentity: 'policy app.users_visible', }) + mockQuery.mockImplementation(database.query) const { EQLInstaller } = await import('@/installer/index.ts') const installer = new EQLInstaller({ databaseUrl: 'postgres://test' }) await expect(installer.install()).rejects.toThrow( /refused before making changes.*policy app\.users_visible/s, ) - expect(mockQuery).toHaveBeenCalledWith('BEGIN') - expect(mockQuery).toHaveBeenCalledWith('ROLLBACK') - expect( - mockQuery.mock.calls.some( - ([sql]) => - typeof sql === 'string' && sql.includes('CREATE SCHEMA eql_v3'), - ), - ).toBe(false) + expect(database.events).toEqual([ + 'begin', + 'lock', + 'configure', + 'capture', + 'rollback', + ]) }) it('rolls back schema replacement when index rebuild fails', async () => { mockConnect.mockResolvedValue(undefined) mockEnd.mockResolvedValue(undefined) - const indexDefinition = - 'CREATE INDEX users_email_idx ON app.users (eql_v3.eq_term(email))' - mockQuery.mockImplementation((sql: string) => { - if (sql.includes('stash_eql_lifecycle_dependencies')) { - return Promise.resolve({ - rows: [ - { - dependency_kind: 'index', - identity: 'app.users_email_idx', - definition: indexDefinition, - table_identity: 'app.users', - valid: true, - ready: true, - clustered: false, - cluster_sql: null, - }, - ], - rowCount: 1, - }) - } - if (sql === indexDefinition) { - return Promise.reject(new Error('disk full')) - } - return Promise.resolve({ rows: [], rowCount: 0 }) + const scenario = searchIndexRestorationScenario({ + definition: + 'CREATE INDEX users_email_idx ON app.users (eql_v3.eq_term(email))', }) + const database = new RecordingRestorationDatabase(scenario, { + failReconstructionWith: new Error('disk full'), + }) + mockQuery.mockImplementation(database.query) const { EQLInstaller } = await import('@/installer/index.ts') await expect( @@ -552,8 +442,15 @@ describe('EQLInstaller', () => { ).rejects.toThrow( /transaction will restore.*Captured index SQL:\nCREATE INDEX users_email_idx/s, ) - expect(mockQuery).toHaveBeenCalledWith('ROLLBACK') - expect(mockQuery).not.toHaveBeenCalledWith('COMMIT') + expect(database.events).toEqual([ + 'begin', + 'lock', + 'configure', + 'capture', + 'replace', + 'reconstruct', + 'rollback', + ]) }) it('grants both EQL v3 schemas to Supabase roles when the role is a member of postgres', async () => { diff --git a/packages/cli/src/installer/__tests__/reinstall.live.test.ts b/packages/cli/src/installer/__tests__/reinstall.live.test.ts index 6d9e0b7e4..e94242e74 100644 --- a/packages/cli/src/installer/__tests__/reinstall.live.test.ts +++ b/packages/cli/src/installer/__tests__/reinstall.live.test.ts @@ -13,46 +13,19 @@ import { afterAll, beforeEach, describe, expect, it } from 'vitest' import { derivedSearchIndexRestorationTestSeam } from '../derived-search-index-restoration.js' import { EQLInstaller, loadBundledEqlSql } from '../index.js' import { parseExpectedSurface } from '../verify.js' +import { + LiveRestorationDatabase, + searchIndexRestorationScenario, +} from './restoration-scenarios.js' -const { acquireLifecycleLock, lifecycleDependenciesSql } = - derivedSearchIndexRestorationTestSeam +const { acquireLifecycleLock } = derivedSearchIndexRestorationTestSeam const DATABASE_URL = process.env.STASH_TEST_DATABASE_URL const describeLive = DATABASE_URL ? describe : describe.skip - -async function queryOn( - url: string, - sql: string, - params: unknown[] = [], -): Promise { - const { default: pg } = await import('pg') - const client = new pg.Client({ connectionString: url }) - await client.connect() - try { - return (await client.query(sql, params)).rows as T[] - } finally { - await client.end().catch(() => undefined) - } -} +const postgres = new LiveRestorationDatabase(DATABASE_URL ?? '') async function query(sql: string): Promise { - return queryOn(DATABASE_URL ?? '', sql) -} - -/** - * The same database, reached on a connection whose `search_path` names the EQL - * schemas. Provisioned databases routinely carry this (`ALTER ROLE … SET - * search_path`) so applications can call `eq_term()` unqualified — and it is - * the one condition under which `format_type()` stops schema-qualifying EQL's - * own types. - */ -function withEqlSearchPath(url: string): string { - const parsed = new URL(url) - parsed.searchParams.set( - 'options', - '-c search_path=public,eql_v3,eql_v3_internal', - ) - return parsed.toString() + return postgres.query(sql) } /** @@ -104,9 +77,14 @@ describeLive('EQLInstaller safe reinstall — live Postgres', () => { }) it('preserves data and rebuilds a functional index', async () => { + const scenario = searchIndexRestorationScenario({ + identity: 'stash_reinstall_test.records_encrypted_idx', + tableIdentity: 'stash_reinstall_test.records', + definition: + 'CREATE INDEX records_encrypted_idx ON stash_reinstall_test.records (eql_v3.eq_term(encrypted))', + }) await query(` - CREATE INDEX records_encrypted_idx - ON stash_reinstall_test.records (eql_v3.eq_term(encrypted)); + ${scenario.definition}; CREATE UNIQUE INDEX "Records encrypted complex" ON stash_reinstall_test.records USING btree (eql_v3.eq_term(encrypted)) INCLUDE (id) WITH (fillfactor = 80) WHERE id > 0; @@ -217,13 +195,10 @@ describeLive('EQLInstaller safe reinstall — live Postgres', () => { */ it("exempts the bundle's own operators and casts when the EQL schemas are on the search_path", async () => { const expected = parseExpectedSurface(loadBundledEqlSql()) - const rows = await queryOn<{ + const rows = await postgres.withEqlSearchPath().dependencyInventory<{ dependency_kind: string identity: string - }>(withEqlSearchPath(DATABASE_URL ?? ''), lifecycleDependenciesSql, [ - expected.operators, - expected.casts, - ]) + }>(expected.operators, expected.casts) const unsafe = rows .filter((row) => row.dependency_kind !== 'index') .map((row) => row.identity) @@ -256,11 +231,12 @@ describeLive('EQLInstaller safe reinstall — live Postgres', () => { const withheld = expected.operators.filter(hasBareOperand) expect(withheld.length).toBeGreaterThan(0) - const rows = await queryOn<{ dependency_kind: string; identity: string }>( - withEqlSearchPath(DATABASE_URL ?? ''), - lifecycleDependenciesSql, - [expected.operators.filter((o) => !hasBareOperand(o)), expected.casts], - ) + const rows = await postgres + .withEqlSearchPath() + .dependencyInventory<{ dependency_kind: string; identity: string }>( + expected.operators.filter((o) => !hasBareOperand(o)), + expected.casts, + ) expect(rows.filter((row) => row.dependency_kind !== 'index')).toHaveLength( withheld.length, @@ -274,7 +250,7 @@ describeLive('EQLInstaller safe reinstall — live Postgres', () => { `) await expect( new EQLInstaller({ - databaseUrl: withEqlSearchPath(DATABASE_URL ?? ''), + databaseUrl: postgres.withEqlSearchPath().url, }).install(), ).resolves.toEqual({ deferredGrantsSql: null }) }, 180_000) diff --git a/packages/cli/src/installer/__tests__/restoration-scenarios.ts b/packages/cli/src/installer/__tests__/restoration-scenarios.ts new file mode 100644 index 000000000..b8615f3ec --- /dev/null +++ b/packages/cli/src/installer/__tests__/restoration-scenarios.ts @@ -0,0 +1,162 @@ +export interface SearchIndexRestorationScenario { + identity: string + tableIdentity: string + definition: string + valid: boolean + ready: boolean + clustered: boolean + clusterSql: string | null +} + +export type RestorationEvent = + | 'begin' + | 'lock' + | 'configure' + | 'capture' + | 'replace' + | 'reconstruct' + | 'cluster' + | 'analyze' + | 'verify' + | 'commit' + | 'rollback' + +export function searchIndexRestorationScenario( + overrides: Partial = {}, +): SearchIndexRestorationScenario { + return { + identity: 'app.users_email_idx', + tableIdentity: 'app.users', + definition: + 'CREATE INDEX users_email_idx ON app.users USING btree (eql_v3.eq_term(email))', + valid: true, + ready: true, + clustered: false, + clusterSql: null, + ...overrides, + } +} + +/** Recording PostgreSQL adapter for deterministic restoration protocol tests. */ +export class RecordingRestorationDatabase { + readonly events: RestorationEvent[] = [] + + constructor( + private readonly scenario?: SearchIndexRestorationScenario, + private readonly options: { + unsafeIdentity?: string + failReconstructionWith?: Error + failConfigurationWith?: Error + } = {}, + ) {} + + query = async ( + sql: string, + ): Promise<{ rows: unknown[]; rowCount: number }> => { + const event = restorationEvent(sql, this.scenario) + if (event !== null) this.events.push(event) + + if (event === 'lock') return { rows: [{ acquired: true }], rowCount: 1 } + if (event === 'configure' && this.options.failConfigurationWith) { + throw this.options.failConfigurationWith + } + if (event === 'capture') { + if (this.options.unsafeIdentity) { + return { + rows: [ + { + dependency_kind: 'unsafe', + identity: this.options.unsafeIdentity, + }, + ], + rowCount: 1, + } + } + return this.scenario + ? { rows: [captureRow(this.scenario)], rowCount: 1 } + : { rows: [], rowCount: 0 } + } + if (event === 'reconstruct' && this.options.failReconstructionWith) { + throw this.options.failReconstructionWith + } + if (event === 'verify' && this.scenario) { + return { rows: [verificationRow(this.scenario)], rowCount: 1 } + } + return { rows: [], rowCount: 0 } + } +} + +/** Live PostgreSQL adapter for catalog behavior that a recording cannot prove. */ +export class LiveRestorationDatabase { + constructor(readonly url: string) {} + + withEqlSearchPath(): LiveRestorationDatabase { + const parsed = new URL(this.url) + parsed.searchParams.set( + 'options', + '-c search_path=public,eql_v3,eql_v3_internal', + ) + return new LiveRestorationDatabase(parsed.toString()) + } + + async query(sql: string, params: unknown[] = []): Promise { + const { default: pg } = await import('pg') + const client = new pg.Client({ connectionString: this.url }) + await client.connect() + try { + return (await client.query(sql, params)).rows as T[] + } finally { + await client.end().catch(() => undefined) + } + } + + dependencyInventory(operators: string[], casts: string[]): Promise { + return this.query( + derivedSearchIndexRestorationTestSeam.lifecycleDependenciesSql, + [operators, casts], + ) + } +} + +function captureRow(scenario: SearchIndexRestorationScenario) { + return { + dependency_kind: 'index', + identity: scenario.identity, + definition: scenario.definition, + table_identity: scenario.tableIdentity, + valid: scenario.valid, + ready: scenario.ready, + clustered: scenario.clustered, + cluster_sql: scenario.clusterSql, + } +} + +function verificationRow(scenario: SearchIndexRestorationScenario) { + return { + identity: scenario.identity, + definition: scenario.definition, + valid: scenario.valid, + ready: scenario.ready, + clustered: scenario.clustered, + } +} + +function restorationEvent( + sql: string, + scenario?: SearchIndexRestorationScenario, +): RestorationEvent | null { + if (sql === 'BEGIN') return 'begin' + if (sql.includes('pg_try_advisory_xact_lock')) return 'lock' + if (sql === 'SET LOCAL jit = off') return 'configure' + if (sql.includes('stash_eql_lifecycle_dependencies')) return 'capture' + if (sql.includes('CREATE SCHEMA eql_v3')) return 'replace' + if (scenario && sql === scenario.definition) return 'reconstruct' + if (scenario?.clusterSql && sql === scenario.clusterSql) return 'cluster' + if (sql.startsWith('ANALYZE ')) return 'analyze' + if (sql.includes('stash_eql_verify_rebuilt_indexes')) return 'verify' + if (sql === 'COMMIT') return 'commit' + if (sql === 'ROLLBACK') return 'rollback' + return null +} + +import { derivedSearchIndexRestorationTestSeam } from '../derived-search-index-restoration.js' From a9e51bba7389b2d6184f52646d238cf736e9f720 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Tue, 1 Sep 2026 14:16:34 +1000 Subject: [PATCH 06/11] refactor(cli): deepen EQL installation assessment --- packages/cli/src/__tests__/installer.test.ts | 64 +++- .../db/__tests__/install-verify-gate.test.ts | 47 ++- packages/cli/src/commands/db/install.ts | 23 +- packages/cli/src/commands/db/status.ts | 84 ++--- packages/cli/src/commands/db/upgrade.ts | 21 +- packages/cli/src/commands/eql/verify.ts | 11 +- packages/cli/src/installer/index.ts | 302 ++---------------- .../cli/src/installer/installation-state.ts | 269 ++++++++++++++++ packages/cli/src/installer/verify.ts | 10 +- packages/eql/CONTEXT.md | 7 + 10 files changed, 464 insertions(+), 374 deletions(-) create mode 100644 packages/cli/src/installer/installation-state.ts diff --git a/packages/cli/src/__tests__/installer.test.ts b/packages/cli/src/__tests__/installer.test.ts index a85473bec..4791fea77 100644 --- a/packages/cli/src/__tests__/installer.test.ts +++ b/packages/cli/src/__tests__/installer.test.ts @@ -250,25 +250,75 @@ describe('EQLInstaller', () => { const { EQLInstaller } = await import('@/installer/index.ts') const installer = new EQLInstaller({ databaseUrl: 'postgres://test' }) - mockQuery.mockResolvedValue({ rows: [{ found: 2 }], rowCount: 1 }) + mockQuery.mockImplementation((sql: string) => { + if (sql.includes("to_regnamespace('eql_v2')")) { + return Promise.resolve({ + rows: [ + { + eql_v2_present: false, + eql_v3_present: true, + eql_v3_internal_present: true, + }, + ], + rowCount: 1, + }) + } + if (sql.includes('eql_v3.version()')) { + return Promise.resolve({ rows: [{ version: '3.0.5' }], rowCount: 1 }) + } + return Promise.resolve({ + rows: [{ ore_opclass_present: true, poisoned_domains: 0 }], + rowCount: 1, + }) + }) await expect(installer.isInstalled()).resolves.toBe(true) - expect(mockQuery).toHaveBeenCalledWith(expect.any(String), [ - ['eql_v3', 'eql_v3_internal'], - ]) - mockQuery.mockResolvedValue({ rows: [{ found: 1 }], rowCount: 1 }) + mockQuery.mockImplementation((sql: string) => { + if (sql.includes("to_regnamespace('eql_v2')")) { + return Promise.resolve({ + rows: [ + { + eql_v2_present: false, + eql_v3_present: true, + eql_v3_internal_present: false, + }, + ], + rowCount: 1, + }) + } + return Promise.resolve({ rows: [], rowCount: 0 }) + }) await expect(installer.isInstalled()).resolves.toBe(false) }) it('retains read-only EQL v2 installation detection for status', async () => { mockConnect.mockResolvedValue(undefined) mockEnd.mockResolvedValue(undefined) - mockQuery.mockResolvedValue({ rows: [{ found: 1 }], rowCount: 1 }) + mockQuery.mockImplementation((sql: string) => { + if (sql.includes("to_regnamespace('eql_v2')")) { + return Promise.resolve({ + rows: [ + { + eql_v2_present: true, + eql_v3_present: false, + eql_v3_internal_present: false, + }, + ], + rowCount: 1, + }) + } + if (sql.includes('eql_v2.version()')) { + return Promise.resolve({ rows: [{ version: '2.3.1' }], rowCount: 1 }) + } + return Promise.resolve({ rows: [], rowCount: 0 }) + }) const { EQLInstaller } = await import('@/installer/index.ts') const installer = new EQLInstaller({ databaseUrl: 'postgres://test' }) await expect(installer.isInstalled({ eqlVersion: 2 })).resolves.toBe(true) - expect(mockQuery).toHaveBeenCalledWith(expect.any(String), [['eql_v2']]) + expect(mockQuery).toHaveBeenCalledWith( + expect.stringContaining("to_regnamespace('eql_v2')"), + ) }) it('installs only the pinned EQL v3 bundle', async () => { diff --git a/packages/cli/src/commands/db/__tests__/install-verify-gate.test.ts b/packages/cli/src/commands/db/__tests__/install-verify-gate.test.ts index c2871f80b..aff22e72c 100644 --- a/packages/cli/src/commands/db/__tests__/install-verify-gate.test.ts +++ b/packages/cli/src/commands/db/__tests__/install-verify-gate.test.ts @@ -41,11 +41,28 @@ vi.mock('@clack/prompts', () => ({ outro: clack.outro, })) -const verifier = vi.hoisted(() => ({ verifyEqlSurface: vi.fn() })) -vi.mock('@/installer/verify.js', () => ({ - verifyEqlSurface: verifier.verifyEqlSurface, +const assessment = vi.hoisted(() => ({ assessEqlInstallation: vi.fn() })) +vi.mock('@/installer/installation-state.js', () => ({ + assessEqlInstallation: assessment.assessEqlInstallation, })) +function assessed(report: VerifyReport) { + return { + v2: { status: 'absent' }, + v3: { status: 'installed', version: report.installedVersion ?? 'unknown' }, + ore: { status: 'absent' }, + surface: { + status: + report.status === 'version-mismatch' + ? 'not-comparable' + : report.ok + ? 'complete' + : 'damaged', + report, + }, + } +} + // Imported dynamically by the damage path for its findings renderer. const findingsReporter = vi.hoisted(() => ({ reportVerifyFindings: vi.fn() })) vi.mock('../../eql/verify.js', () => ({ @@ -77,21 +94,23 @@ describe('verifySurfaceOrExit', () => { }) it('returns without exiting on a complete surface', async () => { - verifier.verifyEqlSurface.mockResolvedValueOnce(report({})) + assessment.assessEqlInstallation.mockResolvedValueOnce(assessed(report({}))) await expect( verifySurfaceOrExit('postgres://db', spinner(), { remedy: 'r' }), ).resolves.toBeUndefined() }) it('exits 1 on damage, after reporting the findings and the remedy', async () => { - verifier.verifyEqlSurface.mockResolvedValueOnce( - report({ - status: 'incomplete', - ok: false, - findings: [ - { severity: 'damage', kind: 'operator', message: 'op missing' }, - ], - }), + assessment.assessEqlInstallation.mockResolvedValueOnce( + assessed( + report({ + status: 'incomplete', + ok: false, + findings: [ + { severity: 'damage', kind: 'operator', message: 'op missing' }, + ], + }), + ), ) const exit = vi .spyOn(process, 'exit') @@ -125,7 +144,7 @@ describe('verifySurfaceOrExit', () => { }, ], }) - verifier.verifyEqlSurface.mockResolvedValueOnce(mismatch) + assessment.assessEqlInstallation.mockResolvedValueOnce(assessed(mismatch)) const exit = vi.spyOn(process, 'exit').mockImplementation(() => { throw new Error('exit called') }) @@ -142,7 +161,7 @@ describe('verifySurfaceOrExit', () => { }) it('warns and continues when verification itself errors', async () => { - verifier.verifyEqlSurface.mockRejectedValueOnce( + assessment.assessEqlInstallation.mockRejectedValueOnce( new Error('connection terminated'), ) await expect( diff --git a/packages/cli/src/commands/db/install.ts b/packages/cli/src/commands/db/install.ts index cd8569ccc..c0a051fee 100644 --- a/packages/cli/src/commands/db/install.ts +++ b/packages/cli/src/commands/db/install.ts @@ -4,8 +4,9 @@ import { resolveDatabaseUrl } from '@/config/database-url.js' import { findConfigFile, loadStashConfig } from '@/config/index.js' import { createPgClient } from '@/db/client.js' import { EQLInstaller } from '@/installer/index.js' +import { assessEqlInstallation } from '@/installer/installation-state.js' import { describeOreState } from '@/installer/ore.js' -import { type VerifyReport, verifyEqlSurface } from '@/installer/verify.js' +import type { VerifyReport } from '@/installer/verify.js' import { messages } from '@/messages.js' import { detectPackageManager, runnerCommand } from '../init/utils.js' import { ensureEncryptionClient } from './client-scaffold.js' @@ -141,7 +142,14 @@ export async function installCommand( const installer = new EQLInstaller({ databaseUrl }) s.start('Checking database permissions...') - const permissions = await installer.preflight() + const installation = await assessEqlInstallation({ + databaseUrl, + includeCapabilities: true, + }) + if (installation.capabilities.status !== 'assessed') { + throw new Error('Database capabilities were not assessed') + } + const permissions = installation.capabilities.preflight if (!permissions.ok) { s.stop('Insufficient database permissions.') p.log.error('The connected database role is missing required permissions:') @@ -163,7 +171,7 @@ export async function installCommand( if (!options.force) { s.start('Checking if EQL is already installed...') - const installed = await installer.isInstalled() + const installed = installation.v3.status === 'installed' s.stop(installed ? 'EQL is already installed.' : 'EQL is not installed.') if (installed) { // Re-apply the grants even when the bundle is present: since the bundle @@ -243,7 +251,14 @@ export async function verifySurfaceOrExit( s.start('Verifying the installed EQL surface...') let report: VerifyReport try { - report = await verifyEqlSurface(databaseUrl) + const installation = await assessEqlInstallation({ + databaseUrl, + depth: 'exhaustive', + }) + if (installation.surface.status === 'not-requested') { + throw new Error('Exhaustive EQL assessment returned no surface result') + } + report = installation.surface.report } catch (err) { s.stop('Could not verify the installed EQL surface.') p.log.warn( diff --git a/packages/cli/src/commands/db/status.ts b/packages/cli/src/commands/db/status.ts index 1fc3b25f1..8bd6ae860 100644 --- a/packages/cli/src/commands/db/status.ts +++ b/packages/cli/src/commands/db/status.ts @@ -2,9 +2,8 @@ import * as p from '@clack/prompts' import { detectPackageManager, runnerCommand } from '@/commands/init/utils.js' import { loadStashConfig } from '@/config/index.js' import { createPgClient } from '@/db/client.js' -import { EQLInstaller } from '@/installer/index.js' +import { assessEqlInstallation } from '@/installer/installation-state.js' import { describeOreState } from '@/installer/ore.js' -import { readOreState } from '@/installer/verify.js' export async function statusCommand(options: { databaseUrl?: string } = {}) { const pm = detectPackageManager() @@ -16,29 +15,18 @@ export async function statusCommand(options: { databaseUrl?: string } = {}) { const config = await loadStashConfig({ databaseUrlFlag: options.databaseUrl }) s.stop('Configuration loaded.') - const installer = new EQLInstaller({ - databaseUrl: config.databaseUrl, - }) - // 1. Check EQL installation status and version — both generations, so a // v3-only database is not misreported as "not installed" (the v2 check // only looks for the eql_v2 schema). s.start('Checking EQL installation...') - let installedV2: boolean - let installedV3: boolean - let versionV2: string | null - let versionV3: string | null + let installation: Awaited> try { - installedV2 = await installer.isInstalled({ eqlVersion: 2 }) - installedV3 = await installer.isInstalled({ eqlVersion: 3 }) - versionV2 = installedV2 - ? await installer.getInstalledVersion({ eqlVersion: 2 }) - : null - versionV3 = installedV3 - ? await installer.getInstalledVersion({ eqlVersion: 3 }) - : null + installation = await assessEqlInstallation({ + databaseUrl: config.databaseUrl, + includeCapabilities: true, + }) } catch (error) { s.stop('Failed.') p.log.error( @@ -50,16 +38,18 @@ export async function statusCommand(options: { databaseUrl?: string } = {}) { process.exit(1) } + const installedV2 = installation.v2.status === 'installed' + const installedV3 = installation.v3.status === 'installed' if (installedV2 || installedV3) { s.stop('EQL is installed.') if (installedV2) { p.log.success( - `EQL v2 installed: yes (version: ${versionV2 ?? 'unknown'})`, + `EQL v2 installed: yes (version: ${installation.v2.status === 'installed' ? installation.v2.version : 'unknown'})`, ) } if (installedV3) { p.log.success( - `EQL v3 installed: yes (version: ${versionV3 ?? 'unknown'})`, + `EQL v3 installed: yes (version: ${installation.v3.status === 'installed' ? installation.v3.version : 'unknown'})`, ) } } else { @@ -75,7 +65,10 @@ export async function statusCommand(options: { databaseUrl?: string } = {}) { s.start('Checking database permissions...') try { - const permissions = await installer.preflight() + if (installation.capabilities.status !== 'assessed') { + throw new Error('Database capabilities were not assessed') + } + const permissions = installation.capabilities.preflight s.stop('Permissions checked.') if (permissions.ok) { @@ -103,41 +96,26 @@ export async function statusCommand(options: { databaseUrl?: string } = {}) { // half, and reads as 'fallback' on a database that has no EQL at all. if (installedV3) { s.start('Checking ORE operator class...') - const oreClient = createPgClient(config.databaseUrl) - try { - await oreClient.connect() - const ore = await readOreState(oreClient) - s.stop('ORE state checked.') - if (ore.comparable) { - const described = describeOreState(ore.state) - if (described.severity === 'damage') { - p.log.error(described.message) - } else { - p.log.info(described.message) - } + const ore = installation.ore + s.stop('ORE state checked.') + if (ore.status === 'observed') { + const described = describeOreState(ore.state) + if (described.severity === 'damage') { + p.log.error(described.message) } else { - // Version skew is not damage, and must not be rendered as any ORE - // answer at all: the domain list the poison CHECKs are counted over is - // the PINNED bundle's, so a perfectly healthy fallback install of an - // older EQL classifies as incoherent and would send this operator to - // `install --force` over nothing. Say the true thing instead. - p.log.info( - `ORE operator class: not compared — EQL ${ - ore.installedVersion ?? 'unknown' - } is installed and this CLI pins EQL ${ore.bundleVersion}, so the ORE state cannot be read against the pinned bundle. Run \`${runnerCommand(pm, 'stash eql upgrade')}\`, then check status again.`, - ) + p.log.info(described.message) } - } catch (error) { - // Advisory, not a gate: a status run that could not read one row should - // still report everything else it read. - s.stop('ORE state check failed.') - p.log.warn( - `Could not determine the ORE operator class state: ${ - error instanceof Error ? error.message : String(error) - }`, + } else if (ore.status === 'not-comparable') { + // Version skew is not damage, and must not be rendered as any ORE + // answer at all: the domain list the poison CHECKs are counted over is + // the PINNED bundle's, so a perfectly healthy fallback install of an + // older EQL classifies as incoherent and would send this operator to + // `install --force` over nothing. Say the true thing instead. + p.log.info( + `ORE operator class: not compared — EQL ${ + ore.installedVersion ?? 'unknown' + } is installed and this CLI pins EQL ${ore.bundleVersion}, so the ORE state cannot be read against the pinned bundle. Run \`${runnerCommand(pm, 'stash eql upgrade')}\`, then check status again.`, ) - } finally { - await oreClient.end().catch(() => {}) } } diff --git a/packages/cli/src/commands/db/upgrade.ts b/packages/cli/src/commands/db/upgrade.ts index 74430604f..2573b6f86 100644 --- a/packages/cli/src/commands/db/upgrade.ts +++ b/packages/cli/src/commands/db/upgrade.ts @@ -2,6 +2,7 @@ import * as p from '@clack/prompts' import { detectPackageManager, runnerCommand } from '@/commands/init/utils.js' import { loadStashConfig } from '@/config/index.js' import { EQLInstaller } from '@/installer/index.js' +import { assessEqlInstallation } from '@/installer/installation-state.js' import { reportSupabaseGrantsOutcome } from './grants-report.js' export async function upgradeCommand(options: { @@ -22,8 +23,10 @@ export async function upgradeCommand(options: { const installer = new EQLInstaller({ databaseUrl: config.databaseUrl }) s.start('Checking current EQL v3 installation...') - const installed = await installer.isInstalled() - if (!installed) { + const before = await assessEqlInstallation({ + databaseUrl: config.databaseUrl, + }) + if (before.v3.status === 'absent') { s.stop('EQL v3 is not installed.') p.log.warn( `EQL v3 is not currently installed. Run "${runnerCommand(pm, 'stash eql install')}" first.`, @@ -32,12 +35,12 @@ export async function upgradeCommand(options: { process.exit(1) } - const previousVersion = await installer.getInstalledVersion() - s.stop(`Current version: ${previousVersion ?? 'unknown'}`) + const previousVersion = before.v3.version + s.stop(`Current version: ${previousVersion}`) if (options.dryRun) { p.log.info('Dry run — no changes will be made.') p.note( - `Current version: ${previousVersion ?? 'unknown'}\nWould re-run the pinned EQL v3 install SQL against the database`, + `Current version: ${previousVersion}\nWould re-run the pinned EQL v3 install SQL against the database`, 'Dry Run', ) p.outro('Dry run complete.') @@ -50,9 +53,11 @@ export async function upgradeCommand(options: { if (options.supabase) reportSupabaseGrantsOutcome(result) s.start('Verifying new version...') - const newVersion = await installer.getInstalledVersion() - s.stop(`New version: ${newVersion ?? 'unknown'}`) - if (previousVersion && newVersion && previousVersion === newVersion) { + const after = await assessEqlInstallation({ databaseUrl: config.databaseUrl }) + const newVersion = + after.v3.status === 'installed' ? after.v3.version : 'unknown' + s.stop(`New version: ${newVersion}`) + if (previousVersion === newVersion) { p.log.info('Version unchanged — EQL was already up to date.') } p.outro('Done!') diff --git a/packages/cli/src/commands/eql/verify.ts b/packages/cli/src/commands/eql/verify.ts index f17cfe0da..d26afad27 100644 --- a/packages/cli/src/commands/eql/verify.ts +++ b/packages/cli/src/commands/eql/verify.ts @@ -2,9 +2,9 @@ import * as p from '@clack/prompts' import { emitJsonError, emitJsonEvent } from '@/commands/auth/events.js' import { resolveDiagnosticDatabaseUrl } from '@/commands/db/resolve-diagnostic-url.js' import { detectPackageManager, runnerCommand } from '@/commands/init/utils.js' +import { assessEqlInstallation } from '@/installer/installation-state.js' import { describeOreState } from '@/installer/ore.js' import type { SurfaceFinding, VerifyReport } from '@/installer/verify.js' -import { verifyEqlSurface } from '@/installer/verify.js' /** * `stash eql verify` — assert the installed EQL surface is complete and @@ -46,7 +46,14 @@ export async function verifyCommand( s?.start('Comparing the installed EQL surface with the pinned bundle...') let report: VerifyReport try { - report = await verifyEqlSurface(databaseUrl) + const installation = await assessEqlInstallation({ + databaseUrl, + depth: 'exhaustive', + }) + if (installation.surface.status === 'not-requested') { + throw new Error('EQL surface was not assessed') + } + report = installation.surface.report } catch (error) { const message = error instanceof Error ? error.message : String(error) if (json) { diff --git a/packages/cli/src/installer/index.ts b/packages/cli/src/installer/index.ts index 22c94e176..f6823d84e 100644 --- a/packages/cli/src/installer/index.ts +++ b/packages/cli/src/installer/index.ts @@ -5,15 +5,16 @@ import { EqlReinstallRefusalError, restoreDerivedSearchIndexesAroundEqlReplacement, } from './derived-search-index-restoration.js' -import { SUPPORTED_PGCRYPTO_SCHEMAS } from './eql-bundle.js' import { DEFERRED_GRANTS_HEADER, - EQL_V3_INTERNAL_SCHEMA_NAME, - EQL_V3_SCHEMA_NAME, SUPABASE_DEFAULT_PRIVILEGES_SQL_V3, SUPABASE_IMMEDIATE_GRANTS_SQL_V3, SUPABASE_PERMISSIONS_SQL_V3, } from './grants.js' +import { + assessEqlInstallation, + type PreflightResult, +} from './installation-state.js' import { loadVerifiedEqlBundle } from './verify.js' export { @@ -37,8 +38,6 @@ export { /** EQL generations recognised by read-only installation diagnostics. */ export type EqlVersion = 2 | 3 -const EQL_V2_SCHEMA_NAME = 'eql_v2' - /** * The pinned EQL v3 install SQL, verified against the resolved release's * `installSqlSha256` before it is handed to anything that executes or emits it. @@ -67,48 +66,7 @@ export function supabaseGrantsFor(): string { * fine; the installer defers the owner-scoped Supabase default-privilege * statements instead (see {@link InstallResult.deferredGrantsSql}). */ -export interface PreflightResult { - currentUser: string - isSuperuser: boolean - /** - * Whether `current_user` can run `ALTER DEFAULT PRIVILEGES FOR ROLE - * postgres`. `null` when the database has no `postgres` role at all. - */ - memberOfPostgres: boolean | null - hasDatabaseCreate: boolean - hasPublicCreate: boolean - pgcryptoInstalled: boolean - /** - * The schema `pgcrypto` lives in, or `null` when not installed. The pinned - * bundle accepts `extensions` and `public` (its functions' search_path) and - * ABORTS for any other schema — so an unsupported placement blocks even a - * superuser. - */ - pgcryptoSchema: string | null - eqlV3SchemaPresent: boolean - eqlV3InternalSchemaPresent: boolean - /** - * Whether `current_user` may drop the existing `eql_v3` / `eql_v3_internal` - * schemas (owner, member of the owning role, or superuser). `null` when the - * schema is absent. Matters because a reinstall begins with - * `DROP SCHEMA ... CASCADE`. - */ - canDropEqlV3Schema: boolean | null - canDropEqlV3InternalSchema: boolean | null - /** - * Whether this role can create the ORE btree operator class the `_ord_ore` - * domains need (#891). `null` when the probe could not answer. - * - * Never blocks: the bundle skips the class and installs its loud-failure - * fallback instead, which is a supported configuration. It is reported so - * the trade is known before a schema is written, not after a query fails. - * See {@link probeOperatorClassCreate} for why this is probed rather than - * inferred from `isSuperuser`. - */ - canCreateOperatorClass: boolean | null - missing: string[] - ok: boolean -} +export type { PreflightResult } from './installation-state.js' /** * The legacy permission-check shape. @@ -135,91 +93,6 @@ export interface InstallResult { deferredGrantsSql: string | null } -/** - * One query answering every preflight question. Two guard patterns are - * load-bearing: `pg_has_role` raises on a nonexistent role name (not every - * database has a `postgres` role), and `has_schema_privilege` raises 3F000 on - * a nonexistent schema (hardened databases drop `public`) — each probe that - * can raise is wrapped so a missing object reads as a capability answer, not - * a query failure. The scalar subqueries against `pg_namespace` return NULL - * (not an error) when the schema is absent, which maps to the `null` arms of - * {@link PreflightResult}. - */ -const PREFLIGHT_SQL = ` - SELECT - current_user AS role_name, - (SELECT rolsuper FROM pg_roles WHERE rolname = current_user) AS is_superuser, - CASE WHEN EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'postgres') - THEN pg_has_role(current_user, 'postgres', 'MEMBER') - END AS member_of_postgres, - has_database_privilege(current_user, current_database(), 'CREATE') AS has_database_create, - CASE WHEN EXISTS (SELECT 1 FROM pg_namespace WHERE nspname = 'public') - THEN has_schema_privilege(current_user, 'public', 'CREATE') - ELSE false - END AS has_public_create, - EXISTS (SELECT 1 FROM pg_extension WHERE extname = 'pgcrypto') AS pgcrypto_installed, - (SELECT n.nspname FROM pg_extension e JOIN pg_namespace n ON n.oid = e.extnamespace - WHERE e.extname = 'pgcrypto') AS pgcrypto_schema, - EXISTS (SELECT 1 FROM information_schema.schemata WHERE schema_name = '${EQL_V3_SCHEMA_NAME}') AS eql_v3_present, - EXISTS (SELECT 1 FROM information_schema.schemata WHERE schema_name = '${EQL_V3_INTERNAL_SCHEMA_NAME}') AS eql_v3_internal_present, - (SELECT pg_has_role(current_user, n.nspowner, 'MEMBER') FROM pg_namespace n - WHERE n.nspname = '${EQL_V3_SCHEMA_NAME}') AS can_drop_eql_v3, - (SELECT pg_has_role(current_user, n.nspowner, 'MEMBER') FROM pg_namespace n - WHERE n.nspname = '${EQL_V3_INTERNAL_SCHEMA_NAME}') AS can_drop_eql_v3_internal -` - -/** The schemas the pinned bundle accepts `pgcrypto` in (its search_path). */ -/** - * Can this role create the ORE btree operator class? (#891) - * - * Asked of the server rather than inferred, because `rolsuper` is the wrong - * question. `CREATE OPERATOR CLASS` is superuser-gated in stock PostgreSQL, - * but managed platforms differ on whether their admin role clears that gate: - * AWS RDS and Aurora do (with `rolsuper = f`), cloud-hosted Supabase does not. - * Predicting from `rolsuper` would tell an RDS operator their ORE domains are - * unavailable when they work — exactly the blanket claim about "managed - * Postgres" this whole change exists to stop making. - * - * `CREATE OPERATOR FAMILY` shares the privilege gate with `CREATE OPERATOR - * CLASS` and needs no member operators, so it is the cheapest statement that - * tests it. The whole probe runs in a transaction that is always rolled back, - * so preflight stays observably read-only. - * - * Returns `null` when the attempt could not answer the question — a read-only - * replica (`25006`), a statement timeout, no `public` schema to create into. - * Callers must render that as unknown, never as either answer. - */ -async function probeOperatorClassCreate( - client: pg.ClientBase, -): Promise { - // A name no bundle uses, so a probe that somehow escaped its rollback is - // recognisable rather than mistaken for an EQL object. - const probeName = 'public.stash_preflight_opclass_probe' - try { - await client.query('BEGIN') - } catch { - return null - } - try { - await client.query(`CREATE OPERATOR FAMILY ${probeName} USING btree`) - return true - } catch (error) { - // 42501 insufficient_privilege is the gate itself — a real "no". Anything - // else (no CREATE on public, read-only transaction, timeout) is a probe - // that failed to ask the question. - const code = - typeof error === 'object' && error !== null && 'code' in error - ? String((error as { code?: unknown }).code) - : undefined - return code === '42501' ? false : null - } finally { - // Always: on the success path this is what keeps preflight read-only, and - // on the failure path it clears the aborted transaction. A rollback that - // itself fails leaves nothing behind — the connection is closed next. - await client.query('ROLLBACK').catch(() => {}) - } -} - export class EQLInstaller { private readonly databaseUrl: string @@ -228,103 +101,14 @@ export class EQLInstaller { } async preflight(): Promise { - const client = createPgClient(this.databaseUrl) - try { - await client.connect() - } catch (error) { - await client.end().catch(() => {}) - // Already shaped centrally by createPgClient's connect wrapper — the - // message is self-contained; adding framing would bury the remedy. - if (error instanceof TlsVerificationError) throw error - const detail = error instanceof Error ? error.message : String(error) - throw new Error(`Failed to connect to database: ${detail}`, { - cause: error, - }) - } - try { - const result = await client.query(PREFLIGHT_SQL) - const row = result.rows[0] ?? {} - const isSuperuser = row.is_superuser === true - const hasDatabaseCreate = row.has_database_create === true - const pgcryptoInstalled = row.pgcrypto_installed === true - const pgcryptoSchema = - typeof row.pgcrypto_schema === 'string' ? row.pgcrypto_schema : null - const asBoolOrNull = (value: unknown) => - typeof value === 'boolean' ? value : null - const canDropEqlV3Schema = asBoolOrNull(row.can_drop_eql_v3) - const canDropEqlV3InternalSchema = asBoolOrNull( - row.can_drop_eql_v3_internal, - ) - const missing: string[] = [] - if (!isSuperuser) { - if (!hasDatabaseCreate) { - missing.push( - 'CREATE on database (required for CREATE SCHEMA and CREATE EXTENSION)', - ) - } - if (row.has_public_create !== true) { - missing.push( - 'CREATE on public schema (required for CREATE DOMAIN public.eql_v3_*)', - ) - } - if (!pgcryptoInstalled && !hasDatabaseCreate) { - missing.push( - 'SUPERUSER or extension owner (required for CREATE EXTENSION pgcrypto)', - ) - } - } - // Not gated on superuser: the bundle itself raises for a pgcrypto - // outside its functions' search_path, whoever runs it. - if ( - pgcryptoInstalled && - pgcryptoSchema !== null && - !SUPPORTED_PGCRYPTO_SCHEMAS.includes(pgcryptoSchema) - ) { - missing.push( - `pgcrypto relocated (it is in schema "${pgcryptoSchema}", which is not on the EQL search_path — the install aborts; fix with: ALTER EXTENSION pgcrypto SET SCHEMA extensions)`, - ) - } - // pg_has_role is true for superusers and for the owner, so this only - // fires for a role that genuinely cannot run the bundle's opening - // DROP SCHEMA ... CASCADE against someone else's install. - if ( - canDropEqlV3Schema === false || - canDropEqlV3InternalSchema === false - ) { - missing.push( - 'ownership of the existing EQL schemas (a reinstall begins with DROP SCHEMA eql_v3 / eql_v3_internal CASCADE, which needs the owner, a member of the owning role, or a superuser)', - ) - } - // After the capability read, so a probe that somehow poisons the session - // cannot affect any of the answers above. - const canCreateOperatorClass = await probeOperatorClassCreate(client) - return { - currentUser: String(row.role_name ?? 'unknown'), - isSuperuser, - memberOfPostgres: asBoolOrNull(row.member_of_postgres), - hasDatabaseCreate, - hasPublicCreate: row.has_public_create === true, - pgcryptoInstalled, - pgcryptoSchema, - eqlV3SchemaPresent: row.eql_v3_present === true, - eqlV3InternalSchemaPresent: row.eql_v3_internal_present === true, - canDropEqlV3Schema, - canDropEqlV3InternalSchema, - canCreateOperatorClass, - missing, - // Deliberately not folded into `missing`: the bundle's ORE fallback - // means an install without the operator class is complete, not - // blocked. - ok: missing.length === 0, - } - } catch (error) { - const detail = error instanceof Error ? error.message : String(error) - throw new Error(`Database preflight query failed: ${detail}`, { - cause: error, - }) - } finally { - await client.end() + const installation = await assessEqlInstallation({ + databaseUrl: this.databaseUrl, + includeCapabilities: true, + }) + if (installation.capabilities.status !== 'assessed') { + throw new Error('Database capabilities were not assessed') } + return installation.capabilities.preflight } /** @@ -344,65 +128,21 @@ export class EQLInstaller { /** Generation-aware read-only detection retained for legacy diagnostics. */ async isInstalled(options?: { eqlVersion?: EqlVersion }): Promise { - const client = createPgClient(this.databaseUrl) - const requiredSchemas = - (options?.eqlVersion ?? 3) === 3 - ? [EQL_V3_SCHEMA_NAME, EQL_V3_INTERNAL_SCHEMA_NAME] - : [EQL_V2_SCHEMA_NAME] - try { - await client.connect() - const result = await client.query( - 'SELECT count(*)::int AS found FROM information_schema.schemata WHERE schema_name = ANY($1)', - [requiredSchemas], - ) - return result.rows[0]?.found === requiredSchemas.length - } catch (error) { - if (error instanceof TlsVerificationError) throw error - const detail = error instanceof Error ? error.message : String(error) - throw new Error(`Failed to connect to database: ${detail}`, { - cause: error, - }) - } finally { - await client.end() - } + const installation = await assessEqlInstallation({ + databaseUrl: this.databaseUrl, + }) + return installation[`v${options?.eqlVersion ?? 3}`].status === 'installed' } /** Read-only version diagnostics for current and legacy installs. */ async getInstalledVersion(options?: { eqlVersion?: EqlVersion }): Promise { - const schemaName = - (options?.eqlVersion ?? 3) === 3 ? EQL_V3_SCHEMA_NAME : EQL_V2_SCHEMA_NAME - const client = createPgClient(this.databaseUrl) - try { - await client.connect() - const schemaResult = await client.query( - 'SELECT schema_name FROM information_schema.schemata WHERE schema_name = $1', - [schemaName], - ) - if (schemaResult.rowCount === null || schemaResult.rowCount === 0) { - return null - } - try { - const versionResult = await client.query( - `SELECT ${schemaName}.version() AS version`, - ) - if (versionResult.rows[0]?.version) { - return String(versionResult.rows[0].version) - } - } catch { - // Older installs may not expose version(). - } - return 'unknown' - } catch (error) { - if (error instanceof TlsVerificationError) throw error - const detail = error instanceof Error ? error.message : String(error) - throw new Error(`Failed to connect to database: ${detail}`, { - cause: error, - }) - } finally { - await client.end() - } + const installation = await assessEqlInstallation({ + databaseUrl: this.databaseUrl, + }) + const generation = installation[`v${options?.eqlVersion ?? 3}`] + return generation.status === 'installed' ? generation.version : null } /** diff --git a/packages/cli/src/installer/installation-state.ts b/packages/cli/src/installer/installation-state.ts new file mode 100644 index 000000000..76d4083bb --- /dev/null +++ b/packages/cli/src/installer/installation-state.ts @@ -0,0 +1,269 @@ +import type pg from 'pg' +import { createPgClient, TlsVerificationError } from '@/db/client.js' +import { SUPPORTED_PGCRYPTO_SCHEMAS } from './eql-bundle.js' +import { EQL_V3_INTERNAL_SCHEMA_NAME, EQL_V3_SCHEMA_NAME } from './grants.js' +import { + bundledExpectedSurface, + diffSurface, + readInstalledSurface, + readOreState, + type VerifyReport, +} from './verify.js' + +export type InstalledEqlGeneration = + | { status: 'absent' } + | { status: 'installed'; version: string | 'unknown' } + +export type AssessedOreState = + | { status: 'absent' } + | { + status: 'not-comparable' + bundleVersion: string + installedVersion: string | null + } + | { + status: 'observed' + state: + | 'indexable' + | 'fallback' + | 'incoherent-mixed' + | 'incoherent-poisoned' + | 'incoherent-unpoisoned' + opclassPresent: boolean + poisonedDomains: number + expectedPoisoned: number + } + +export type AssessedEqlSurface = + | { status: 'not-requested' } + | { status: 'not-comparable'; report: VerifyReport } + | { status: 'complete'; report: VerifyReport } + | { status: 'damaged'; report: VerifyReport } + +export interface EqlInstallationState { + v2: InstalledEqlGeneration + v3: InstalledEqlGeneration + ore: AssessedOreState + surface: AssessedEqlSurface + capabilities: + | { status: 'not-requested' } + | { status: 'assessed'; preflight: PreflightResult } +} + +export interface PreflightResult { + currentUser: string + isSuperuser: boolean + memberOfPostgres: boolean | null + hasDatabaseCreate: boolean + hasPublicCreate: boolean + pgcryptoInstalled: boolean + pgcryptoSchema: string | null + eqlV3SchemaPresent: boolean + eqlV3InternalSchemaPresent: boolean + canDropEqlV3Schema: boolean | null + canDropEqlV3InternalSchema: boolean | null + canCreateOperatorClass: boolean | null + missing: string[] + ok: boolean +} + +const CAPABILITIES_SQL = ` + SELECT current_user AS role_name, + (SELECT rolsuper FROM pg_roles WHERE rolname = current_user) AS is_superuser, + CASE WHEN EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'postgres') THEN pg_has_role(current_user, 'postgres', 'MEMBER') END AS member_of_postgres, + has_database_privilege(current_user, current_database(), 'CREATE') AS has_database_create, + CASE WHEN EXISTS (SELECT 1 FROM pg_namespace WHERE nspname = 'public') THEN has_schema_privilege(current_user, 'public', 'CREATE') ELSE false END AS has_public_create, + EXISTS (SELECT 1 FROM pg_extension WHERE extname = 'pgcrypto') AS pgcrypto_installed, + (SELECT n.nspname FROM pg_extension e JOIN pg_namespace n ON n.oid = e.extnamespace WHERE e.extname = 'pgcrypto') AS pgcrypto_schema, + EXISTS (SELECT 1 FROM information_schema.schemata WHERE schema_name = '${EQL_V3_SCHEMA_NAME}') AS eql_v3_present, + EXISTS (SELECT 1 FROM information_schema.schemata WHERE schema_name = '${EQL_V3_INTERNAL_SCHEMA_NAME}') AS eql_v3_internal_present, + (SELECT pg_has_role(current_user, n.nspowner, 'MEMBER') FROM pg_namespace n WHERE n.nspname = '${EQL_V3_SCHEMA_NAME}') AS can_drop_eql_v3, + (SELECT pg_has_role(current_user, n.nspowner, 'MEMBER') FROM pg_namespace n WHERE n.nspname = '${EQL_V3_INTERNAL_SCHEMA_NAME}') AS can_drop_eql_v3_internal +` + +export async function assessEqlInstallation(options: { + databaseUrl: string + depth?: 'summary' | 'exhaustive' + includeCapabilities?: boolean +}): Promise { + const client = createPgClient(options.databaseUrl) + try { + await client.connect() + } catch (error) { + await client.end().catch(() => {}) + if (error instanceof TlsVerificationError) throw error + const detail = error instanceof Error ? error.message : String(error) + throw new Error(`Failed to connect to database: ${detail}`, { + cause: error, + }) + } + + try { + await client.query('BEGIN READ ONLY') + const presence = await client.query<{ + eql_v2_present: boolean + eql_v3_present: boolean + eql_v3_internal_present: boolean + }>(` + SELECT + to_regnamespace('eql_v2') IS NOT NULL AS eql_v2_present, + to_regnamespace('eql_v3') IS NOT NULL AS eql_v3_present, + to_regnamespace('eql_v3_internal') IS NOT NULL AS eql_v3_internal_present + `) + const row = presence.rows[0] + const v2Present = row?.eql_v2_present === true + const v3Present = + row?.eql_v3_present === true && row.eql_v3_internal_present === true + const v2 = v2Present + ? { status: 'installed' as const, version: await readVersion(client, 2) } + : { status: 'absent' as const } + const v3 = v3Present + ? { status: 'installed' as const, version: await readVersion(client, 3) } + : { status: 'absent' as const } + + const ore = v3Present + ? await assessOre(client) + : { status: 'absent' as const } + let surface: AssessedEqlSurface = { status: 'not-requested' } + if (options.depth === 'exhaustive') { + const expected = bundledExpectedSurface() + const report = diffSurface( + expected, + await readInstalledSurface(client, expected, { + manageTransaction: false, + }), + ) + surface = + report.status === 'version-mismatch' + ? { status: 'not-comparable', report } + : report.ok + ? { status: 'complete', report } + : { status: 'damaged', report } + } + const capabilityRow = options.includeCapabilities + ? ((await client.query(CAPABILITIES_SQL)).rows[0] ?? {}) + : null + await client.query('COMMIT') + const capabilities = capabilityRow + ? { + status: 'assessed' as const, + preflight: buildPreflight( + capabilityRow, + await probeOperatorClassCreate(client), + ), + } + : { status: 'not-requested' as const } + return { v2, v3, ore, surface, capabilities } + } catch (error) { + await client.query('ROLLBACK').catch(() => {}) + throw error + } finally { + await client.end() + } +} + +function buildPreflight( + row: Record, + canCreateOperatorClass: boolean | null, +): PreflightResult { + const asBoolOrNull = (value: unknown) => + typeof value === 'boolean' ? value : null + const isSuperuser = row.is_superuser === true + const hasDatabaseCreate = row.has_database_create === true + const pgcryptoInstalled = row.pgcrypto_installed === true + const pgcryptoSchema = + typeof row.pgcrypto_schema === 'string' ? row.pgcrypto_schema : null + const canDropEqlV3Schema = asBoolOrNull(row.can_drop_eql_v3) + const canDropEqlV3InternalSchema = asBoolOrNull(row.can_drop_eql_v3_internal) + const missing: string[] = [] + if (!isSuperuser) { + if (!hasDatabaseCreate) + missing.push( + 'CREATE on database (required for CREATE SCHEMA and CREATE EXTENSION)', + ) + if (row.has_public_create !== true) + missing.push( + 'CREATE on public schema (required for CREATE DOMAIN public.eql_v3_*)', + ) + if (!pgcryptoInstalled && !hasDatabaseCreate) + missing.push( + 'SUPERUSER or extension owner (required for CREATE EXTENSION pgcrypto)', + ) + } + if ( + pgcryptoInstalled && + pgcryptoSchema !== null && + !SUPPORTED_PGCRYPTO_SCHEMAS.includes(pgcryptoSchema) + ) + missing.push( + `pgcrypto relocated (it is in schema "${pgcryptoSchema}", which is not on the EQL search_path — the install aborts; fix with: ALTER EXTENSION pgcrypto SET SCHEMA extensions)`, + ) + if (canDropEqlV3Schema === false || canDropEqlV3InternalSchema === false) + missing.push( + 'ownership of the existing EQL schemas (a reinstall begins with DROP SCHEMA eql_v3 / eql_v3_internal CASCADE, which needs the owner, a member of the owning role, or a superuser)', + ) + return { + currentUser: String(row.role_name ?? 'unknown'), + isSuperuser, + memberOfPostgres: asBoolOrNull(row.member_of_postgres), + hasDatabaseCreate, + hasPublicCreate: row.has_public_create === true, + pgcryptoInstalled, + pgcryptoSchema, + eqlV3SchemaPresent: row.eql_v3_present === true, + eqlV3InternalSchemaPresent: row.eql_v3_internal_present === true, + canDropEqlV3Schema, + canDropEqlV3InternalSchema, + canCreateOperatorClass, + missing, + ok: missing.length === 0, + } +} + +async function probeOperatorClassCreate( + client: pg.ClientBase, +): Promise { + const probeName = 'public.stash_preflight_opclass_probe' + try { + await client.query('BEGIN') + } catch { + return null + } + try { + await client.query(`CREATE OPERATOR FAMILY ${probeName} USING btree`) + return true + } catch (error) { + const code = + typeof error === 'object' && error !== null && 'code' in error + ? String((error as { code?: unknown }).code) + : undefined + return code === '42501' ? false : null + } finally { + await client.query('ROLLBACK').catch(() => {}) + } +} + +async function readVersion( + client: { + query: (sql: string) => Promise<{ rows: Array<{ version?: unknown }> }> + }, + generation: 2 | 3, +): Promise { + try { + const result = await client.query( + `SELECT eql_v${generation}.version() AS version`, + ) + return result.rows[0]?.version ? String(result.rows[0].version) : 'unknown' + } catch { + return 'unknown' + } +} + +async function assessOre( + client: Parameters[0], +): Promise { + const ore = await readOreState(client) + return ore.comparable + ? { status: 'observed', ...ore } + : { status: 'not-comparable', ...ore } +} diff --git a/packages/cli/src/installer/verify.ts b/packages/cli/src/installer/verify.ts index e9f4dceaf..0e59c97c3 100644 --- a/packages/cli/src/installer/verify.ts +++ b/packages/cli/src/installer/verify.ts @@ -557,6 +557,7 @@ async function readInstalledEqlVersion( export async function readInstalledSurface( client: pg.ClientBase, expected: ExpectedSurface, + options: { manageTransaction?: boolean } = {}, ): Promise { // Sequential on purpose: a single pg.Client serialises concurrent query() // calls anyway (and deprecates them); these are six fast catalogue reads. @@ -573,7 +574,7 @@ export async function readInstalledSurface( // session is untouched (the version() probe below runs after COMMIT and // needs the default path restored — `eql_v3.version` is qualified, but its // body's search_path is its own SET clause either way). - await client.query('BEGIN READ ONLY') + if (options.manageTransaction !== false) await client.query('BEGIN READ ONLY') await client.query(`SET LOCAL search_path = ''`) const schemas = await client.query<{ eql_v3_present: boolean @@ -600,14 +601,13 @@ export async function readInstalledSurface( ore_opclass_present: boolean poisoned_domains: number }>(ORE_STATE_SQL, [expected.oreDomains]) - // Ends the SET LOCAL scope. On a mid-transaction error the caller's - // client.end() discards the aborted transaction with the connection. - await client.query('COMMIT') - const eqlV3SchemaPresent = schemas.rows[0]?.eql_v3_present === true const installedVersion = eqlV3SchemaPresent ? await readInstalledEqlVersion(client) : null + // Ends the SET LOCAL scope. On a mid-transaction error the caller's + // client.end() discards the aborted transaction with the connection. + if (options.manageTransaction !== false) await client.query('COMMIT') const functionSignatures = new Map>() for (const row of functions.rows) { diff --git a/packages/eql/CONTEXT.md b/packages/eql/CONTEXT.md index 72efae290..d5c2298a5 100644 --- a/packages/eql/CONTEXT.md +++ b/packages/eql/CONTEXT.md @@ -29,3 +29,10 @@ _Avoid_: Encrypted data, durable data A customer-owned constraint, policy, view, or other database object whose meaning cannot be safely inferred and recreated by the EQL installer. _Avoid_: Derived search index + +**EQL installation state**: +A consistent observation of installed EQL generations, their versions, the +health of comparable EQL machinery, and the ORE state. When the installed EQL +version differs from the observing tool's pinned bundle, health is not +comparable; version skew is not evidence of damage. +_Avoid_: Installation status, database state From 70407bdf0912e2fd9787fa7b0181c54f34338c9f Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Tue, 1 Sep 2026 14:20:32 +1000 Subject: [PATCH 07/11] refactor(cli): deepen EQL surface verification --- .../cli/src/installer/installation-state.ts | 42 ++++++++++--------- packages/cli/src/installer/verify.ts | 33 +++++++++++++++ 2 files changed, 56 insertions(+), 19 deletions(-) diff --git a/packages/cli/src/installer/installation-state.ts b/packages/cli/src/installer/installation-state.ts index 76d4083bb..649dedc3d 100644 --- a/packages/cli/src/installer/installation-state.ts +++ b/packages/cli/src/installer/installation-state.ts @@ -3,10 +3,8 @@ import { createPgClient, TlsVerificationError } from '@/db/client.js' import { SUPPORTED_PGCRYPTO_SCHEMAS } from './eql-bundle.js' import { EQL_V3_INTERNAL_SCHEMA_NAME, EQL_V3_SCHEMA_NAME } from './grants.js' import { - bundledExpectedSurface, - diffSurface, - readInstalledSurface, - readOreState, + assessEqlSurface, + type OreStateReading, type VerifyReport, } from './verify.js' @@ -121,18 +119,27 @@ export async function assessEqlInstallation(options: { ? { status: 'installed' as const, version: await readVersion(client, 3) } : { status: 'absent' as const } - const ore = v3Present - ? await assessOre(client) - : { status: 'absent' as const } + const verification = v3Present + ? await assessEqlSurface( + client, + options.depth === 'exhaustive' ? 'exhaustive' : 'summary', + ) + : null + const ore = + verification?.depth === 'summary' + ? assessOre(verification.ore) + : verification?.report.ore + ? { status: 'observed' as const, ...verification.report.ore } + : v3Present && verification?.report.status === 'version-mismatch' + ? { + status: 'not-comparable' as const, + bundleVersion: verification.report.bundleVersion, + installedVersion: verification.report.installedVersion, + } + : { status: 'absent' as const } let surface: AssessedEqlSurface = { status: 'not-requested' } - if (options.depth === 'exhaustive') { - const expected = bundledExpectedSurface() - const report = diffSurface( - expected, - await readInstalledSurface(client, expected, { - manageTransaction: false, - }), - ) + if (verification?.depth === 'exhaustive') { + const report = verification.report surface = report.status === 'version-mismatch' ? { status: 'not-comparable', report } @@ -259,10 +266,7 @@ async function readVersion( } } -async function assessOre( - client: Parameters[0], -): Promise { - const ore = await readOreState(client) +function assessOre(ore: OreStateReading): AssessedOreState { return ore.comparable ? { status: 'observed', ...ore } : { status: 'not-comparable', ...ore } diff --git a/packages/cli/src/installer/verify.ts b/packages/cli/src/installer/verify.ts index 0e59c97c3..eb33c662a 100644 --- a/packages/cli/src/installer/verify.ts +++ b/packages/cli/src/installer/verify.ts @@ -963,6 +963,32 @@ export type OreStateReading = installedVersion: string | null } +export type EqlSurfaceAssessment = + | { depth: 'summary'; ore: OreStateReading } + | { depth: 'exhaustive'; report: VerifyReport } + +/** + * The report-oriented verification interface used by installation assessment. + * Parsing, catalogue observation, version gating, ORE classification, and + * surface diffing remain implementation details behind this seam. + * + * The caller owns the connection and transaction so installation presence, + * versions, and this result can describe one database snapshot. + */ +export async function assessEqlSurface( + client: pg.ClientBase, + depth: 'summary' | 'exhaustive', +): Promise { + const expected = bundledExpectedSurface() + if (depth === 'summary') { + return { depth, ore: await readOreStateAgainst(client, expected) } + } + const installed = await readInstalledSurface(client, expected, { + manageTransaction: false, + }) + return { depth, report: diffSurface(expected, installed) } +} + /** * Read just the ORE half of an install — the two catalogue values and the * state they classify to (#891). @@ -985,6 +1011,13 @@ export async function readOreState( client: pg.ClientBase, ): Promise { const expected = bundledExpectedSurface() + return readOreStateAgainst(client, expected) +} + +async function readOreStateAgainst( + client: pg.ClientBase, + expected: ExpectedSurface, +): Promise { const installedVersion = await readInstalledEqlVersion(client) if (installedVersion !== expected.eqlVersion) { return { From fd9e9ce424055aeb96afed07c12ead1aa6b51030 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Tue, 1 Sep 2026 14:33:02 +1000 Subject: [PATCH 08/11] refactor(cli): deepen Supabase EQL access policy --- packages/cli/src/commands/eql/migration.ts | 4 +-- packages/cli/src/installer/grants.ts | 38 ++++++++++++++++++++++ packages/cli/src/installer/index.ts | 31 +++++------------- 3 files changed, 48 insertions(+), 25 deletions(-) diff --git a/packages/cli/src/commands/eql/migration.ts b/packages/cli/src/commands/eql/migration.ts index 0e9cf3e69..8bec47090 100644 --- a/packages/cli/src/commands/eql/migration.ts +++ b/packages/cli/src/commands/eql/migration.ts @@ -27,8 +27,8 @@ import { tryResolveDatabaseUrl, } from '@/config/database-url.js' import { + emitSupabaseEqlAccessMigration, loadBundledEqlSql, - SUPABASE_MIGRATION_GRANTS_SQL_V3, } from '@/installer/index.js' import { messages } from '@/messages.js' @@ -227,7 +227,7 @@ export interface EqlMigrationOptions { export function buildEqlV3MigrationSql(opts: { supabase: boolean }): string { const eqlSql = loadBundledEqlSql() const grants = opts.supabase - ? `\n\n-- Supabase role grants: let anon/authenticated/service_role use the\n-- eql_v3 + eql_v3_internal schemas (required when tables are reached via\n-- PostgREST/RLS; harmless otherwise).\n${SUPABASE_MIGRATION_GRANTS_SQL_V3.trim()}` + ? `\n\n-- Supabase role grants: let anon/authenticated/service_role use the\n-- eql_v3 + eql_v3_internal schemas (required when tables are reached via\n-- PostgREST/RLS; harmless otherwise).\n${emitSupabaseEqlAccessMigration().trim()}` : '' return `${eqlSql.trim()}${grants}\n\n-- CipherStash encryption-migration tracking schema.\n-- Tracks per-column phase + backfill progress for \`stash encrypt\`.\n${MIGRATIONS_SCHEMA_SQL.trim()}\n` } diff --git a/packages/cli/src/installer/grants.ts b/packages/cli/src/installer/grants.ts index 352b57474..0594aab1d 100644 --- a/packages/cli/src/installer/grants.ts +++ b/packages/cli/src/installer/grants.ts @@ -189,3 +189,41 @@ export const SUPABASE_MIGRATION_GRANTS_SQL_V3 = `${SUPABASE_IMMEDIATE_GRANTS_SQL -- runs as a member of \`postgres\` (they cover EQL objects \`postgres\` might -- later create outside stash tooling; stash re-grants on every install). ${SUPABASE_GUARDED_DEFAULT_PRIVILEGES_SQL_V3}` + +export type SupabaseEqlAccessOutcome = + | { status: 'applied' } + | { status: 'applied-with-deferred-defaults'; deferredSql: string } + +interface SqlExecutor { + query(sql: string): Promise<{ rows: Record[] }> +} + +/** + * Apply the Supabase EQL access policy through a PostgreSQL adapter. + * + * Role membership, the immediate/default split, statement ordering, and the + * operator-facing deferred SQL are implementation details of this module. + */ +export async function applySupabaseEqlAccess( + database: SqlExecutor, +): Promise { + const membership = await database.query(` + SELECT CASE WHEN EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'postgres') + THEN pg_has_role(current_user, 'postgres', 'MEMBER') + END AS member_of_postgres + `) + if (membership.rows[0]?.member_of_postgres === true) { + await database.query(SUPABASE_PERMISSIONS_SQL_V3) + return { status: 'applied' } + } + await database.query(SUPABASE_IMMEDIATE_GRANTS_SQL_V3) + return { + status: 'applied-with-deferred-defaults', + deferredSql: DEFERRED_GRANTS_HEADER + SUPABASE_DEFAULT_PRIVILEGES_SQL_V3, + } +} + +/** Emit the same access policy through a migration-file adapter. */ +export function emitSupabaseEqlAccessMigration(): string { + return SUPABASE_MIGRATION_GRANTS_SQL_V3 +} diff --git a/packages/cli/src/installer/index.ts b/packages/cli/src/installer/index.ts index f6823d84e..61619063d 100644 --- a/packages/cli/src/installer/index.ts +++ b/packages/cli/src/installer/index.ts @@ -1,4 +1,3 @@ -import type pg from 'pg' import { createPgClient, TlsVerificationError } from '@/db/client.js' import { EqlReinstallConnectionError, @@ -6,9 +5,7 @@ import { restoreDerivedSearchIndexesAroundEqlReplacement, } from './derived-search-index-restoration.js' import { - DEFERRED_GRANTS_HEADER, - SUPABASE_DEFAULT_PRIVILEGES_SQL_V3, - SUPABASE_IMMEDIATE_GRANTS_SQL_V3, + applySupabaseEqlAccess, SUPABASE_PERMISSIONS_SQL_V3, } from './grants.js' import { @@ -23,9 +20,11 @@ export { } from './eql-bundle.js' export { + applySupabaseEqlAccess, DEFERRED_GRANTS_HEADER, EQL_V3_INTERNAL_SCHEMA_NAME, EQL_V3_SCHEMA_NAME, + emitSupabaseEqlAccessMigration, SUPABASE_DEFAULT_PRIVILEGES_SQL_V3, SUPABASE_GUARDED_DEFAULT_PRIVILEGES_SQL_V3, SUPABASE_IMMEDIATE_GRANTS_SQL_V3, @@ -223,7 +222,11 @@ export class EQLInstaller { }) } try { - return await this.runSupabaseGrants(client) + const outcome = await applySupabaseEqlAccess(client) + return { + deferredGrantsSql: + outcome.status === 'applied' ? null : outcome.deferredSql, + } } catch (error) { const detail = error instanceof Error ? error.message : String(error) throw new Error(`Failed to apply the Supabase role grants: ${detail}`, { @@ -233,22 +236,4 @@ export class EQLInstaller { await client.end() } } - - /** The shared grants phase: full block for members, immediate half + deferred tail otherwise. */ - private async runSupabaseGrants(client: pg.Client): Promise { - const memberResult = await client.query(` - SELECT CASE WHEN EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'postgres') - THEN pg_has_role(current_user, 'postgres', 'MEMBER') - END AS member_of_postgres - `) - if (memberResult.rows[0]?.member_of_postgres === true) { - await client.query(SUPABASE_PERMISSIONS_SQL_V3) - return { deferredGrantsSql: null } - } - await client.query(SUPABASE_IMMEDIATE_GRANTS_SQL_V3) - return { - deferredGrantsSql: - DEFERRED_GRANTS_HEADER + SUPABASE_DEFAULT_PRIVILEGES_SQL_V3, - } - } } From fb7ec98930ef85f16cfaa8c2269298e41c0a489a Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Tue, 1 Sep 2026 14:34:00 +1000 Subject: [PATCH 09/11] refactor(cli): deepen declared schema validation --- packages/cli/src/commands/eql/validate.ts | 90 +++++++++++++++++------ 1 file changed, 69 insertions(+), 21 deletions(-) diff --git a/packages/cli/src/commands/eql/validate.ts b/packages/cli/src/commands/eql/validate.ts index c9ecdbbfc..33d58c717 100644 --- a/packages/cli/src/commands/eql/validate.ts +++ b/packages/cli/src/commands/eql/validate.ts @@ -742,6 +742,43 @@ export async function readObservedState( } } +export interface DeclaredSchemaAssessment { + columns: DeclaredColumn[] + tableCount: number + fidelity: 'complete' | 'config-only' + database: + | { status: 'observed' } + | { + status: 'skipped' + reason: 'not-configured' | 'unreachable' + detail?: string + } + issues: ValidationIssue[] +} + +/** + * Assess the declared encryption schema through one result-oriented interface. + * Declaration normalization, catalogue observation, index parsing, and rule + * classification remain implementation details behind this seam. + */ +export async function assessDeclaredSchema(options: { + encryptConfig: EncryptConfig + schemas?: readonly AnyV3Table[] + databaseUrl?: string +}): Promise { + const columns = options.schemas + ? collectDeclaredColumns(options.schemas) + : collectDeclaredColumnsFromConfig(options.encryptConfig) + const observation = await tryReadObservedState(options.databaseUrl, columns) + return { + columns, + tableCount: new Set(columns.map((column) => column.table)).size, + fidelity: options.schemas ? 'complete' : 'config-only', + database: observation.database, + issues: validateSchemas(columns, observation.observed), + } +} + // --------------------------------------------------------------------------- // Reporting // --------------------------------------------------------------------------- @@ -815,32 +852,37 @@ export async function validateCommand(options: { ) s.stop('Encrypt client loaded.') - const columns = schemas - ? collectDeclaredColumns(schemas) - : collectDeclaredColumnsFromConfig(encryptConfig) + const assessment = await assessDeclaredSchema({ + encryptConfig, + schemas, + databaseUrl: config.databaseUrl, + }) - if (!schemas) { + if (assessment.fidelity === 'config-only') { p.log.warn( 'Your installed @cipherstash/stack does not expose `getSchemas()`, so the concrete EQL domain of each column is unavailable. Domain checks (ORE portability, database drift) were skipped — upgrade @cipherstash/stack to run them.', ) } - const tableCount = new Set(columns.map((column) => column.table)).size p.log.success( - `Schema loaded: ${tableCount} table${tableCount !== 1 ? 's' : ''}, ${columns.length} encrypted column${columns.length !== 1 ? 's' : ''}`, + `Schema loaded: ${assessment.tableCount} table${assessment.tableCount !== 1 ? 's' : ''}, ${assessment.columns.length} encrypted column${assessment.columns.length !== 1 ? 's' : ''}`, ) - const observed = await tryReadObservedState(config.databaseUrl, columns) - - const issues = validateSchemas(columns, observed) + if (assessment.database.status === 'skipped') { + p.log.info( + assessment.database.reason === 'not-configured' + ? 'No database URL resolved — skipping the database checks (drift, ORE availability, functional indexes). Pass --database-url or set DATABASE_URL to run them.' + : `Could not read the database (${assessment.database.detail}) — skipping the database checks (drift, ORE availability, functional indexes). The schema checks below still ran.`, + ) + } - if (issues.length === 0) { + if (assessment.issues.length === 0) { p.outro('No issues found.') return } console.log() // blank line before issues - const hasErrors = reportIssues(issues) + const hasErrors = reportIssues(assessment.issues) if (hasErrors) { process.exit(1) @@ -857,12 +899,12 @@ export async function validateCommand(options: { async function tryReadObservedState( databaseUrl: string | undefined, columns: DeclaredColumn[], -): Promise { +): Promise<{ + observed?: ObservedState + database: DeclaredSchemaAssessment['database'] +}> { if (!databaseUrl) { - p.log.info( - 'No database URL resolved — skipping the database checks (drift, ORE availability, functional indexes). Pass --database-url or set DATABASE_URL to run them.', - ) - return undefined + return { database: { status: 'skipped', reason: 'not-configured' } } } const tables = [...new Set(columns.map((column) => column.table))] @@ -870,13 +912,19 @@ async function tryReadObservedState( try { await client.connect() - return await readObservedState(client, tables) + return { + observed: await readObservedState(client, tables), + database: { status: 'observed' }, + } } catch (error) { const message = error instanceof Error ? error.message : String(error) - p.log.info( - `Could not read the database (${message}) — skipping the database checks (drift, ORE availability, functional indexes). The schema checks below still ran.`, - ) - return undefined + return { + database: { + status: 'skipped', + reason: 'unreachable', + detail: message, + }, + } } finally { await client.end().catch(() => {}) } From 989e7695e43e6691e57c3c7aa3be64b0aa66dc8b Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Tue, 1 Sep 2026 14:41:08 +1000 Subject: [PATCH 10/11] fix(cli): preserve EQL assessment and index state --- packages/cli/src/__tests__/installer.test.ts | 69 ++++---- packages/cli/src/commands/db/status.ts | 3 + .../__tests__/installation-state.test.ts | 153 ++++++++++++++++++ .../__tests__/restoration-scenarios.ts | 19 +++ .../derived-search-index-restoration.ts | 67 +++++++- packages/cli/src/installer/index.ts | 54 ++++--- .../cli/src/installer/installation-state.ts | 36 ++++- packages/cli/src/installer/verify.ts | 28 +++- .../__tests__/wasm-entry-edge-safety.test.ts | 5 + 9 files changed, 365 insertions(+), 69 deletions(-) create mode 100644 packages/cli/src/installer/__tests__/installation-state.test.ts diff --git a/packages/cli/src/__tests__/installer.test.ts b/packages/cli/src/__tests__/installer.test.ts index 4791fea77..ee2141610 100644 --- a/packages/cli/src/__tests__/installer.test.ts +++ b/packages/cli/src/__tests__/installer.test.ts @@ -251,15 +251,9 @@ describe('EQLInstaller', () => { const installer = new EQLInstaller({ databaseUrl: 'postgres://test' }) mockQuery.mockImplementation((sql: string) => { - if (sql.includes("to_regnamespace('eql_v2')")) { + if (sql.includes("to_regnamespace('eql_v3')")) { return Promise.resolve({ - rows: [ - { - eql_v2_present: false, - eql_v3_present: true, - eql_v3_internal_present: true, - }, - ], + rows: [{ installed: true }], rowCount: 1, }) } @@ -272,19 +266,11 @@ describe('EQLInstaller', () => { }) }) await expect(installer.isInstalled()).resolves.toBe(true) + expect(mockQuery).toHaveBeenCalledTimes(1) mockQuery.mockImplementation((sql: string) => { - if (sql.includes("to_regnamespace('eql_v2')")) { - return Promise.resolve({ - rows: [ - { - eql_v2_present: false, - eql_v3_present: true, - eql_v3_internal_present: false, - }, - ], - rowCount: 1, - }) + if (sql.includes("to_regnamespace('eql_v3')")) { + return Promise.resolve({ rows: [{ installed: false }], rowCount: 1 }) } return Promise.resolve({ rows: [], rowCount: 0 }) }) @@ -296,16 +282,7 @@ describe('EQLInstaller', () => { mockEnd.mockResolvedValue(undefined) mockQuery.mockImplementation((sql: string) => { if (sql.includes("to_regnamespace('eql_v2')")) { - return Promise.resolve({ - rows: [ - { - eql_v2_present: true, - eql_v3_present: false, - eql_v3_internal_present: false, - }, - ], - rowCount: 1, - }) + return Promise.resolve({ rows: [{ installed: true }], rowCount: 1 }) } if (sql.includes('eql_v2.version()')) { return Promise.resolve({ rows: [{ version: '2.3.1' }], rowCount: 1 }) @@ -316,11 +293,24 @@ describe('EQLInstaller', () => { const installer = new EQLInstaller({ databaseUrl: 'postgres://test' }) await expect(installer.isInstalled({ eqlVersion: 2 })).resolves.toBe(true) + expect(mockQuery).toHaveBeenCalledTimes(1) expect(mockQuery).toHaveBeenCalledWith( expect.stringContaining("to_regnamespace('eql_v2')"), ) }) + it('reads a legacy installed version with one query', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + mockQuery.mockResolvedValue({ rows: [{ version: '3.0.5' }], rowCount: 1 }) + const { EQLInstaller } = await import('@/installer/index.ts') + const installer = new EQLInstaller({ databaseUrl: 'postgres://test' }) + + await expect(installer.getInstalledVersion()).resolves.toBe('3.0.5') + expect(mockQuery).toHaveBeenCalledTimes(1) + expect(mockQuery).toHaveBeenCalledWith('SELECT eql_v3.version() AS version') + }) + it('installs only the pinned EQL v3 bundle', async () => { mockConnect.mockResolvedValue(undefined) mockQuery.mockResolvedValue({ rows: [], rowCount: 0 }) @@ -386,6 +376,27 @@ describe('EQLInstaller', () => { expect(database.events).toContain('cluster') }) + it('restores catalog state attached to a functional index', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + const database = new RecordingRestorationDatabase( + searchIndexRestorationScenario({ + comment: 'Supports encrypted email equality searches', + commentSql: + "COMMENT ON INDEX app.users_email_idx IS 'Supports encrypted email equality searches'", + }), + ) + mockQuery.mockImplementation(database.query) + const { EQLInstaller } = await import('@/installer/index.ts') + + await new EQLInstaller({ databaseUrl: 'postgres://test' }).install() + + expect(database.events).toContain('comment') + expect(database.events.indexOf('comment')).toBeLessThan( + database.events.indexOf('verify'), + ) + }) + it('captures dependencies before destructive SQL in the protected transaction', async () => { mockConnect.mockResolvedValue(undefined) mockEnd.mockResolvedValue(undefined) diff --git a/packages/cli/src/commands/db/status.ts b/packages/cli/src/commands/db/status.ts index 8bd6ae860..d6adec995 100644 --- a/packages/cli/src/commands/db/status.ts +++ b/packages/cli/src/commands/db/status.ts @@ -26,6 +26,7 @@ export async function statusCommand(options: { databaseUrl?: string } = {}) { installation = await assessEqlInstallation({ databaseUrl: config.databaseUrl, includeCapabilities: true, + includeOre: true, }) } catch (error) { s.stop('Failed.') @@ -116,6 +117,8 @@ export async function statusCommand(options: { databaseUrl?: string } = {}) { ore.installedVersion ?? 'unknown' } is installed and this CLI pins EQL ${ore.bundleVersion}, so the ORE state cannot be read against the pinned bundle. Run \`${runnerCommand(pm, 'stash eql upgrade')}\`, then check status again.`, ) + } else if (ore.status === 'unavailable') { + p.log.warn(`Could not read the ORE operator class state: ${ore.message}`) } } diff --git a/packages/cli/src/installer/__tests__/installation-state.test.ts b/packages/cli/src/installer/__tests__/installation-state.test.ts new file mode 100644 index 000000000..388e95ba9 --- /dev/null +++ b/packages/cli/src/installer/__tests__/installation-state.test.ts @@ -0,0 +1,153 @@ +import { beforeEach, describe, expect, it, vi } from 'vitest' + +const query = vi.fn() +const connect = vi.fn() +const end = vi.fn() + +vi.mock('@/db/client.js', () => ({ + createPgClient: () => ({ query, connect, end }), + TlsVerificationError: class extends Error {}, +})) + +describe('EQL installation state', () => { + beforeEach(() => { + vi.resetAllMocks() + connect.mockResolvedValue(undefined) + end.mockResolvedValue(undefined) + }) + + it('recovers from a missing version function before continuing its snapshot', async () => { + let aborted = false + query.mockImplementation(async (sql: string) => { + if (sql.includes("to_regnamespace('eql_v2')")) { + return { + rows: [ + { + eql_v2_present: true, + eql_v3_present: false, + eql_v3_internal_present: false, + }, + ], + } + } + if (sql.includes('eql_v2.version()')) { + aborted = true + throw Object.assign(new Error('undefined function'), { code: '42883' }) + } + if (sql === 'ROLLBACK TO SAVEPOINT eql_version_probe') { + aborted = false + return { rows: [] } + } + if (aborted) { + throw Object.assign(new Error('transaction is aborted'), { + code: '25P02', + }) + } + return { rows: [] } + }) + + const { assessEqlInstallation } = await import('../installation-state.js') + const state = await assessEqlInstallation({ + databaseUrl: 'postgres://test', + }) + + expect(state.v2).toEqual({ status: 'installed', version: 'unknown' }) + expect(query).toHaveBeenCalledWith( + 'ROLLBACK TO SAVEPOINT eql_version_probe', + ) + expect(query).toHaveBeenCalledWith('COMMIT') + }) + + it('recovers when exhaustive surface observation finds a missing version function', async () => { + let versionReads = 0 + let aborted = false + query.mockImplementation(async (sql: string) => { + if (sql.includes("to_regnamespace('eql_v2')")) { + return { + rows: [ + { + eql_v2_present: false, + eql_v3_present: true, + eql_v3_internal_present: true, + }, + ], + } + } + if (sql.includes('eql_v3.version()')) { + versionReads += 1 + if (versionReads === 1) return { rows: [{ version: '3.0.5' }] } + aborted = true + throw Object.assign(new Error('undefined function'), { code: '42883' }) + } + if (sql === 'ROLLBACK TO SAVEPOINT installed_eql_version_probe') { + aborted = false + return { rows: [] } + } + if (aborted) + throw Object.assign(new Error('transaction is aborted'), { + code: '25P02', + }) + if (sql.includes('pgcrypto_installed')) { + return { + rows: [ + { + eql_v3_present: true, + eql_v3_internal_present: true, + pgcrypto_installed: true, + pgcrypto_schema: 'public', + }, + ], + } + } + if (sql.includes('ore_opclass_present')) { + return { rows: [{ ore_opclass_present: true, poisoned_domains: 0 }] } + } + return { rows: [] } + }) + + const { assessEqlInstallation } = await import('../installation-state.js') + const state = await assessEqlInstallation({ + databaseUrl: 'postgres://test', + depth: 'exhaustive', + }) + + expect(state.surface.status).toBe('damaged') + expect(query).toHaveBeenCalledWith( + 'ROLLBACK TO SAVEPOINT installed_eql_version_probe', + ) + expect(query).toHaveBeenCalledWith('COMMIT') + }) + + it('reports an unavailable advisory ORE observation without failing installation state', async () => { + query.mockImplementation(async (sql: string) => { + if (sql.includes("to_regnamespace('eql_v2')")) { + return { + rows: [ + { + eql_v2_present: false, + eql_v3_present: true, + eql_v3_internal_present: true, + }, + ], + } + } + if (sql.includes('eql_v3.version()')) + return { rows: [{ version: '3.0.5' }] } + if (sql.includes('ore_opclass_present')) + throw new Error('catalog unavailable') + return { rows: [] } + }) + + const { assessEqlInstallation } = await import('../installation-state.js') + const state = await assessEqlInstallation({ + databaseUrl: 'postgres://test', + includeOre: true, + }) + + expect(state.ore).toEqual({ + status: 'unavailable', + message: 'catalog unavailable', + }) + expect(query).toHaveBeenCalledWith('COMMIT') + }) +}) diff --git a/packages/cli/src/installer/__tests__/restoration-scenarios.ts b/packages/cli/src/installer/__tests__/restoration-scenarios.ts index b8615f3ec..5e81bcd74 100644 --- a/packages/cli/src/installer/__tests__/restoration-scenarios.ts +++ b/packages/cli/src/installer/__tests__/restoration-scenarios.ts @@ -6,6 +6,10 @@ export interface SearchIndexRestorationScenario { ready: boolean clustered: boolean clusterSql: string | null + replicaIdentity: boolean + replicaIdentitySql: string | null + comment: string | null + commentSql: string | null } export type RestorationEvent = @@ -16,6 +20,8 @@ export type RestorationEvent = | 'replace' | 'reconstruct' | 'cluster' + | 'replica-identity' + | 'comment' | 'analyze' | 'verify' | 'commit' @@ -33,6 +39,10 @@ export function searchIndexRestorationScenario( ready: true, clustered: false, clusterSql: null, + replicaIdentity: false, + replicaIdentitySql: null, + comment: null, + commentSql: null, ...overrides, } } @@ -128,6 +138,10 @@ function captureRow(scenario: SearchIndexRestorationScenario) { ready: scenario.ready, clustered: scenario.clustered, cluster_sql: scenario.clusterSql, + replica_identity: scenario.replicaIdentity, + replica_identity_sql: scenario.replicaIdentitySql, + comment: scenario.comment, + comment_sql: scenario.commentSql, } } @@ -138,6 +152,8 @@ function verificationRow(scenario: SearchIndexRestorationScenario) { valid: scenario.valid, ready: scenario.ready, clustered: scenario.clustered, + replica_identity: scenario.replicaIdentity, + comment: scenario.comment, } } @@ -152,6 +168,9 @@ function restorationEvent( if (sql.includes('CREATE SCHEMA eql_v3')) return 'replace' if (scenario && sql === scenario.definition) return 'reconstruct' if (scenario?.clusterSql && sql === scenario.clusterSql) return 'cluster' + if (scenario?.replicaIdentitySql && sql === scenario.replicaIdentitySql) + return 'replica-identity' + if (scenario?.commentSql && sql === scenario.commentSql) return 'comment' if (sql.startsWith('ANALYZE ')) return 'analyze' if (sql.includes('stash_eql_verify_rebuilt_indexes')) return 'verify' if (sql === 'COMMIT') return 'commit' diff --git a/packages/cli/src/installer/derived-search-index-restoration.ts b/packages/cli/src/installer/derived-search-index-restoration.ts index 678b10ecf..d47b5bb87 100644 --- a/packages/cli/src/installer/derived-search-index-restoration.ts +++ b/packages/cli/src/installer/derived-search-index-restoration.ts @@ -54,6 +54,10 @@ export interface ReinstallIndex { ready: boolean clustered: boolean clusterSql: string | null + replicaIdentity: boolean + replicaIdentitySql: string | null + comment: string | null + commentSql: string | null } interface DependencyRow { @@ -65,6 +69,10 @@ interface DependencyRow { ready?: unknown clustered?: unknown cluster_sql?: unknown + replica_identity?: unknown + replica_identity_sql?: unknown + comment?: unknown + comment_sql?: unknown } const LIFECYCLE_DEPENDENCIES_SQL = ` @@ -376,6 +384,17 @@ SELECT DISTINCT AND index_class.relkind = 'i' AND index_partition.inhrelid IS NULL THEN index_meta.indisclustered END AS clustered, + CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL THEN index_meta.indisreplident + END AS replica_identity, + CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL + THEN pg_catalog.obj_description(index_class.oid, 'pg_class') + END AS comment, CASE WHEN e.classid = 'pg_catalog.pg_class'::regclass AND index_class.relkind = 'i' @@ -387,7 +406,31 @@ SELECT DISTINCT table_class.relname, index_class.relname ) - END AS cluster_sql + END AS cluster_sql, + CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL + AND index_meta.indisreplident + THEN pg_catalog.format( + 'ALTER TABLE %I.%I REPLICA IDENTITY USING INDEX %I', + table_namespace.nspname, + table_class.relname, + index_class.relname + ) + END AS replica_identity_sql, + CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL + AND pg_catalog.obj_description(index_class.oid, 'pg_class') IS NOT NULL + THEN pg_catalog.format( + 'COMMENT ON INDEX %I.%I IS %L', + index_namespace.nspname, + index_class.relname, + pg_catalog.obj_description(index_class.oid, 'pg_class') + ) + END AS comment_sql FROM external_dependants e LEFT JOIN pg_catalog.pg_class index_class ON e.classid = 'pg_catalog.pg_class'::regclass AND index_class.oid = e.objid @@ -406,6 +449,8 @@ SELECT pg_catalog.format('%I.%I', n.nspname, c.relname) AS identity, i.indisvalid AS valid, i.indisready AS ready, i.indisclustered AS clustered, + i.indisreplident AS replica_identity, + pg_catalog.obj_description(c.oid, 'pg_class') AS comment, pg_catalog.pg_get_indexdef(i.indexrelid) AS definition FROM pg_catalog.pg_index i JOIN pg_catalog.pg_class c ON c.oid = i.indexrelid @@ -475,8 +520,15 @@ async function inspectReinstallDependencies( typeof row.valid !== 'boolean' || typeof row.ready !== 'boolean' || typeof row.clustered !== 'boolean' || + typeof row.replica_identity !== 'boolean' || (row.cluster_sql !== null && typeof row.cluster_sql !== 'string') || - (row.clustered && typeof row.cluster_sql !== 'string') + (row.clustered && typeof row.cluster_sql !== 'string') || + (row.replica_identity_sql !== null && + typeof row.replica_identity_sql !== 'string') || + (row.replica_identity && typeof row.replica_identity_sql !== 'string') || + (row.comment !== null && typeof row.comment !== 'string') || + (row.comment_sql !== null && typeof row.comment_sql !== 'string') || + (typeof row.comment === 'string' && typeof row.comment_sql !== 'string') ) { unsafe.push( String(row.identity ?? 'index with incomplete catalog metadata'), @@ -491,6 +543,10 @@ async function inspectReinstallDependencies( ready: row.ready, clustered: row.clustered, clusterSql: row.cluster_sql, + replicaIdentity: row.replica_identity, + replicaIdentitySql: row.replica_identity_sql, + comment: row.comment, + commentSql: row.comment_sql, }) } if (unsafe.length > 0) { @@ -529,6 +585,9 @@ async function rebuildIndexes( } for (const index of indexes) { if (index.clusterSql !== null) await client.query(index.clusterSql) + if (index.replicaIdentitySql !== null) + await client.query(index.replicaIdentitySql) + if (index.commentSql !== null) await client.query(index.commentSql) } const tableIdentities = new Set(indexes.map((index) => index.tableIdentity)) for (const tableIdentity of tableIdentities) { @@ -562,7 +621,9 @@ async function rebuildIndexes( expected.definition === row.definition && (row.valid === true || expected.valid === false) && (row.ready === true || expected.ready === false) && - row.clustered === expected.clustered + row.clustered === expected.clustered && + row.replica_identity === expected.replicaIdentity && + (row.comment ?? null) === expected.comment ) }) .map((row) => String(row.identity)), diff --git a/packages/cli/src/installer/index.ts b/packages/cli/src/installer/index.ts index 61619063d..9a8ce0294 100644 --- a/packages/cli/src/installer/index.ts +++ b/packages/cli/src/installer/index.ts @@ -37,20 +37,6 @@ export { /** EQL generations recognised by read-only installation diagnostics. */ export type EqlVersion = 2 | 3 -/** - * The pinned EQL v3 install SQL, verified against the resolved release's - * `installSqlSha256` before it is handed to anything that executes or emits it. - * - * This is the CLI's single choke point for the bundle — `install()`, - * `stash eql migration`'s emitter and `bundledExpectedSurface()` all come - * through here — which is why the digest check lives in the wrapper rather than - * at each call site. `readInstallSql()` itself is in the frozen - * `@cipherstash/eql` subtree, published from another repository, so a check - * added there would be dead code for every consumer installing from npm. - * - * @throws if the bundle cannot be read, or if its bytes are not the ones the - * resolved release attests to (see {@link assertBundledEqlSqlDigest}). - */ /** Supabase grants for the sole installable generation, EQL v3. */ export function supabaseGrantsFor(): string { return SUPABASE_PERMISSIONS_SQL_V3 @@ -127,21 +113,43 @@ export class EQLInstaller { /** Generation-aware read-only detection retained for legacy diagnostics. */ async isInstalled(options?: { eqlVersion?: EqlVersion }): Promise { - const installation = await assessEqlInstallation({ - databaseUrl: this.databaseUrl, - }) - return installation[`v${options?.eqlVersion ?? 3}`].status === 'installed' + const generation = options?.eqlVersion ?? 3 + const client = createPgClient(this.databaseUrl) + try { + await client.connect() + const result = await client.query<{ installed: boolean }>( + generation === 2 + ? "SELECT to_regnamespace('eql_v2') IS NOT NULL AS installed" + : "SELECT to_regnamespace('eql_v3') IS NOT NULL AND to_regnamespace('eql_v3_internal') IS NOT NULL AS installed", + ) + return result.rows[0]?.installed === true + } finally { + await client.end() + } } /** Read-only version diagnostics for current and legacy installs. */ async getInstalledVersion(options?: { eqlVersion?: EqlVersion }): Promise { - const installation = await assessEqlInstallation({ - databaseUrl: this.databaseUrl, - }) - const generation = installation[`v${options?.eqlVersion ?? 3}`] - return generation.status === 'installed' ? generation.version : null + const generation = options?.eqlVersion ?? 3 + const client = createPgClient(this.databaseUrl) + try { + await client.connect() + const result = await client.query<{ version: string }>( + `SELECT eql_v${generation}.version() AS version`, + ) + return result.rows[0]?.version ?? null + } catch (error) { + const code = + typeof error === 'object' && error !== null && 'code' in error + ? String((error as { code?: unknown }).code) + : undefined + if (code === '42883' || code === '3F000') return null + throw error + } finally { + await client.end() + } } /** diff --git a/packages/cli/src/installer/installation-state.ts b/packages/cli/src/installer/installation-state.ts index 649dedc3d..ea59d71e1 100644 --- a/packages/cli/src/installer/installation-state.ts +++ b/packages/cli/src/installer/installation-state.ts @@ -14,6 +14,8 @@ export type InstalledEqlGeneration = export type AssessedOreState = | { status: 'absent' } + | { status: 'not-requested' } + | { status: 'unavailable'; message: string } | { status: 'not-comparable' bundleVersion: string @@ -83,6 +85,7 @@ export async function assessEqlInstallation(options: { databaseUrl: string depth?: 'summary' | 'exhaustive' includeCapabilities?: boolean + includeOre?: boolean }): Promise { const client = createPgClient(options.databaseUrl) try { @@ -119,12 +122,24 @@ export async function assessEqlInstallation(options: { ? { status: 'installed' as const, version: await readVersion(client, 3) } : { status: 'absent' as const } - const verification = v3Present - ? await assessEqlSurface( - client, - options.depth === 'exhaustive' ? 'exhaustive' : 'summary', - ) - : null + let verification = null + let unavailableOre: AssessedOreState | null = null + if (v3Present && options.depth === 'exhaustive') { + verification = await assessEqlSurface(client, 'exhaustive') + } else if (v3Present && options.includeOre === true) { + await client.query('SAVEPOINT eql_ore_assessment') + try { + verification = await assessEqlSurface(client, 'summary') + await client.query('RELEASE SAVEPOINT eql_ore_assessment') + } catch (error) { + await client.query('ROLLBACK TO SAVEPOINT eql_ore_assessment') + await client.query('RELEASE SAVEPOINT eql_ore_assessment') + unavailableOre = { + status: 'unavailable', + message: error instanceof Error ? error.message : String(error), + } + } + } const ore = verification?.depth === 'summary' ? assessOre(verification.ore) @@ -136,7 +151,10 @@ export async function assessEqlInstallation(options: { bundleVersion: verification.report.bundleVersion, installedVersion: verification.report.installedVersion, } - : { status: 'absent' as const } + : (unavailableOre ?? + (v3Present + ? { status: 'not-requested' as const } + : { status: 'absent' as const })) let surface: AssessedEqlSurface = { status: 'not-requested' } if (verification?.depth === 'exhaustive') { const report = verification.report @@ -256,12 +274,16 @@ async function readVersion( }, generation: 2 | 3, ): Promise { + await client.query('SAVEPOINT eql_version_probe') try { const result = await client.query( `SELECT eql_v${generation}.version() AS version`, ) + await client.query('RELEASE SAVEPOINT eql_version_probe') return result.rows[0]?.version ? String(result.rows[0].version) : 'unknown' } catch { + await client.query('ROLLBACK TO SAVEPOINT eql_version_probe') + await client.query('RELEASE SAVEPOINT eql_version_probe') return 'unknown' } } diff --git a/packages/cli/src/installer/verify.ts b/packages/cli/src/installer/verify.ts index eb33c662a..a8dc8d4c5 100644 --- a/packages/cli/src/installer/verify.ts +++ b/packages/cli/src/installer/verify.ts @@ -528,17 +528,28 @@ const ORE_STATE_SQL = ` */ async function readInstalledEqlVersion( client: pg.ClientBase, + insideCallerTransaction = false, ): Promise { + if (insideCallerTransaction) { + await client.query('SAVEPOINT installed_eql_version_probe') + } try { const version = await client.query<{ version: string }>( `SELECT ${EQL_V3_SCHEMA_NAME}.version() AS version`, ) + if (insideCallerTransaction) { + await client.query('RELEASE SAVEPOINT installed_eql_version_probe') + } return version.rows[0]?.version ?? null } catch (error) { const code = error !== null && typeof error === 'object' && 'code' in error ? (error as { code?: string }).code : undefined + if (insideCallerTransaction) { + await client.query('ROLLBACK TO SAVEPOINT installed_eql_version_probe') + await client.query('RELEASE SAVEPOINT installed_eql_version_probe') + } if (code === '42883') return null const detail = error instanceof Error ? error.message : String(error) throw new Error( @@ -560,7 +571,7 @@ export async function readInstalledSurface( options: { manageTransaction?: boolean } = {}, ): Promise { // Sequential on purpose: a single pg.Client serialises concurrent query() - // calls anyway (and deprecates them); these are six fast catalogue reads. + // calls anyway (and deprecates them); these are seven fast catalogue reads. // // The read-only transaction exists for `SET LOCAL search_path = ''`: // `format_type` qualifies a name exactly when the type is not visible on @@ -571,9 +582,8 @@ export async function readInstalledSurface( // (`integer`, `text[]`). That is precisely the spelling the bundle parser // produces; without the pin the output would vary with the connection's // search_path. SET LOCAL dies with the transaction, so the caller's - // session is untouched (the version() probe below runs after COMMIT and - // needs the default path restored — `eql_v3.version` is qualified, but its - // body's search_path is its own SET clause either way). + // session is untouched. The qualified version() probe is protected by a + // savepoint because a missing function must not abort the caller's snapshot. if (options.manageTransaction !== false) await client.query('BEGIN READ ONLY') await client.query(`SET LOCAL search_path = ''`) const schemas = await client.query<{ @@ -603,7 +613,7 @@ export async function readInstalledSurface( }>(ORE_STATE_SQL, [expected.oreDomains]) const eqlV3SchemaPresent = schemas.rows[0]?.eql_v3_present === true const installedVersion = eqlV3SchemaPresent - ? await readInstalledEqlVersion(client) + ? await readInstalledEqlVersion(client, true) : null // Ends the SET LOCAL scope. On a mid-transaction error the caller's // client.end() discards the aborted transaction with the connection. @@ -981,7 +991,7 @@ export async function assessEqlSurface( ): Promise { const expected = bundledExpectedSurface() if (depth === 'summary') { - return { depth, ore: await readOreStateAgainst(client, expected) } + return { depth, ore: await readOreStateAgainst(client, expected, true) } } const installed = await readInstalledSurface(client, expected, { manageTransaction: false, @@ -1017,8 +1027,12 @@ export async function readOreState( async function readOreStateAgainst( client: pg.ClientBase, expected: ExpectedSurface, + insideCallerTransaction = false, ): Promise { - const installedVersion = await readInstalledEqlVersion(client) + const installedVersion = await readInstalledEqlVersion( + client, + insideCallerTransaction, + ) if (installedVersion !== expected.eqlVersion) { return { comparable: false, diff --git a/packages/stack-supabase/__tests__/wasm-entry-edge-safety.test.ts b/packages/stack-supabase/__tests__/wasm-entry-edge-safety.test.ts index bf193ed41..7e45b3301 100644 --- a/packages/stack-supabase/__tests__/wasm-entry-edge-safety.test.ts +++ b/packages/stack-supabase/__tests__/wasm-entry-edge-safety.test.ts @@ -75,6 +75,11 @@ function reachableBareSpecifiers(entry: string): string[] { const bare = new Set() const walk = (file: string): void => { if (seen.has(file)) return + if (!existsSync(file)) { + throw new Error( + `Emitted module graph references missing relative file: ${file}`, + ) + } seen.add(file) for (const specifier of specifiers(file)) { if (specifier.startsWith('.')) walk(resolve(dirname(file), specifier)) From 386792876d402f36645c6724a58b7a9f4a371e78 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Wed, 2 Sep 2026 10:33:57 +1000 Subject: [PATCH 11/11] fix(cli): address safe reinstall review feedback --- .changeset/safe-eql-reinstall.md | 51 ++--- .github/workflows/tests.yml | 15 +- ...ta-survives-disposable-schema-reinstall.md | 23 +-- .../2026-08-31-eql-safe-reinstall-design.md | 128 ------------ packages/cli/README.md | 19 +- packages/cli/src/__tests__/installer.test.ts | 188 +++++++++++++++++- .../src/commands/db/__tests__/install.test.ts | 93 +++++++++ .../src/commands/db/__tests__/status.test.ts | 52 +++++ .../src/commands/db/__tests__/upgrade.test.ts | 74 +++++++ packages/cli/src/commands/db/install.ts | 15 +- packages/cli/src/commands/db/status.ts | 9 +- packages/cli/src/commands/db/upgrade.ts | 30 ++- .../__tests__/installation-state.test.ts | 99 ++++++++- .../__tests__/reinstall.live.test.ts | 137 +++++++++---- .../__tests__/restoration-scenarios.ts | 40 +++- .../upgrade-encrypted-indexes.live.test.ts | 21 +- .../installer/__tests__/verify.live.test.ts | 8 +- .../src/installer/__tests__/verify.test.ts | 156 ++++++++++++++- .../derived-search-index-restoration.ts | 188 +++++++++--------- packages/cli/src/installer/index.ts | 39 +++- .../cli/src/installer/installation-state.ts | 24 ++- packages/cli/src/installer/verify.ts | 184 +++++++++-------- packages/cli/vitest.config.ts | 8 +- .../__tests__/wasm-entry-edge-safety.test.ts | 5 - .../__tests__/cli-live-postgres-ci.test.mjs | 51 +++++ .../lint-no-eql-registry-pins.test.mjs | 10 +- scripts/lint-no-eql-registry-pins.mjs | 4 +- skills/stash-cli/SKILL.md | 19 +- turbo.json | 1 + 29 files changed, 1188 insertions(+), 503 deletions(-) delete mode 100644 docs/superpowers/specs/2026-08-31-eql-safe-reinstall-design.md create mode 100644 packages/cli/src/commands/db/__tests__/install.test.ts create mode 100644 packages/cli/src/commands/db/__tests__/status.test.ts create mode 100644 packages/cli/src/commands/db/__tests__/upgrade.test.ts create mode 100644 scripts/__tests__/cli-live-postgres-ci.test.mjs diff --git a/.changeset/safe-eql-reinstall.md b/.changeset/safe-eql-reinstall.md index bcea770c5..8f604fa59 100644 --- a/.changeset/safe-eql-reinstall.md +++ b/.changeset/safe-eql-reinstall.md @@ -1,43 +1,20 @@ --- -"stash": patch +"stash": minor --- Preserve encrypted data and reconstruct functional indexes when reinstalling EQL v3, while refusing unsupported external dependencies before mutation. -`stash eql install` and `stash eql upgrade` replace the disposable `eql_v3` and -`eql_v3_internal` schemas with `DROP SCHEMA … CASCADE`. Encrypted columns and -rows live outside those schemas and are never dropped, but anything depending on -EQL machinery goes with it. The installer now: +`stash eql install` and `stash eql upgrade` now capture dependent functional +indexes before replacing the disposable EQL schemas, then restore and verify +their definitions, clustering, replica-identity role, comments, explicit +statistics targets, and health in the same transaction. A +reconstruction failure rolls the replacement back. +PostgreSQL derives index ownership from the table owner, so reinstall verifies +the resulting owner and rolls back on a mismatch rather than independently +restoring ownership. +Unsupported dependencies—including views, policies, constraints, and +partitioned indexes—are named and refused before mutation. -- Takes a lifecycle lock, captures customer functional indexes that depend on - EQL, replaces the schemas, then rebuilds, analyzes and verifies those indexes - — all inside one transaction, so a rebuild failure restores the previous - installation rather than leaving a silently de-indexed database. -- Refuses before making any change when something it cannot reconstruct (a view, - a policy) depends on EQL, naming each object. -- Refuses before mutation when a functional index sits on a partitioned table. - Such an index cannot be reconstructed from its definition alone: the parent's - definition says `ON ONLY`, its per-partition children are separate objects, - and the parent stays invalid until every child is re-attached. Refusing names - the index instead of failing partway through a replacement. -- Classifies bundle-owned operators and casts independently of the session - `search_path`. Previously these were matched through `format_type()`, which - drops the schema qualification for types visible on the `search_path`, so a - connection with `eql_v3` on its path saw EQL's own operators as - customer-owned and refused on a healthy database. -- Waits up to five minutes for a concurrent EQL lifecycle operation instead of - blocking forever with no output, then fails with a message naming the cause - and the remedy. A queued install still waits and succeeds; only a lock nobody - will release now reports itself. -- Reports a bundle its expected-surface parser cannot model with the parser's - own message, naming the statement, instead of burying it in an install - failure that describes a rollback that never happened. - -Two limits worth knowing. Run a reinstall in a schema-migration maintenance -window: the advisory lock serializes cooperating `stash` commands, but an -ordinary PostgreSQL role cannot block unrelated sessions from creating or -dropping EQL-backed indexes, and an index created between capture and -replacement is dropped without being rebuilt. And these protections live in the -CLI, not in the SQL — a migration generated by `stash eql migration` is the raw -bundle, so re-applying one over a database that already carries EQL functional -indexes drops them with no rebuild. +Reinstall remains a maintenance-window operation: its advisory lock serializes +`stash` lifecycle commands, not unrelated database DDL. Generated EQL migrations +contain the raw bundle and do not include these reinstall protections. diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 0f5e2af5d..27bf45bae 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -52,14 +52,23 @@ jobs: strategy: matrix: node-version: [22, 24] + postgres-version: [16, 17] + exclude: + - node-version: 22 + postgres-version: 17 + - node-version: 24 + postgres-version: 16 # Postgres + EQL for the integration tests. Official EQL image — - # PostgreSQL 17 with EQL pre-installed via /docker-entrypoint-initdb.d. + # PostgreSQL 16 and 17 with EQL pre-installed via + # /docker-entrypoint-initdb.d. Keeping one supported Node line on each + # server version exercises the pre-PG17 attstattarget representation + # without adding a third full test leg. # Pinned to eql-2.3.1 to match the EQL payload format the code emits # (protect-ffi 0.23.x); bump in lockstep with the protect-ffi upgrade. services: postgres: - image: ghcr.io/cipherstash/postgres-eql:17-2.3.1 + image: ghcr.io/cipherstash/postgres-eql:${{ matrix.postgres-version }}-2.3.1 env: POSTGRES_USER: cipherstash POSTGRES_PASSWORD: password @@ -339,7 +348,7 @@ jobs: # suite installs EQL v3 into its own schemas, which coexists # with the image's pre-installed EQL v2 that the stack tests use. # They share that one database, so the CLI vitest config runs them - # serially (the `live` project sets `fileParallelism: false` — + # serially (the `live` project uses a single fork — # verify.live's bundle install opens with DROP SCHEMA … CASCADE, which # races destructively under the other suites in parallel forks). # (`supabase-push.live.test.ts` gates on different env vars and still diff --git a/docs/adr/0001-eql-data-survives-disposable-schema-reinstall.md b/docs/adr/0001-eql-data-survives-disposable-schema-reinstall.md index a9795de01..8e62de2fd 100644 --- a/docs/adr/0001-eql-data-survives-disposable-schema-reinstall.md +++ b/docs/adr/0001-eql-data-survives-disposable-schema-reinstall.md @@ -5,17 +5,14 @@ status: accepted # Keep encrypted data durable and EQL schemas disposable EQL data-bearing domains live in `public` and must survive install, uninstall, -and reinstall, while the `eql_v3` and `eql_v3_internal` schemas remain disposable -and may be dropped with `CASCADE`. Search indexes are derived state: tooling must -capture, rebuild, and verify them around reinstall. Tooling must refuse before -mutation when it finds customer-owned dependencies such as policies, -constraints, or views that it cannot reconstruct safely. This follows the EQL -v2 persistence boundary and deliberately rejects brittle object-by-object -in-place upgrades and permanently versioned implementation schemas. - -Schema replacement, index reconstruction, and verification are one PostgreSQL -transaction. A failed reconstruction therefore restores the previous EQL -schemas and indexes instead of leaving a partially upgraded database. +and reinstall, while the `eql_v3` and `eql_v3_internal` schemas remain +disposable and may be dropped with `CASCADE`. Search indexes are derived state: +tooling must capture, rebuild, and verify them around reinstall. Tooling must +refuse before mutation when it finds customer-owned dependencies such as +policies, constraints, or views that it cannot reconstruct safely. This follows +the EQL v2 persistence boundary and deliberately rejects brittle +object-by-object in-place upgrades and permanently versioned implementation +schemas. ## Consequences @@ -26,7 +23,3 @@ schemas and indexes instead of leaving a partially upgraded database. silently to sequential scans. - Changes that make an existing index definition invalid require operator intervention rather than guessed migration semantics. -- Reinstall requires a schema-migration maintenance window: the advisory lock - serializes EQL lifecycle commands, but ordinary PostgreSQL roles cannot block - arbitrary application DDL globally. Do not create, alter, or drop EQL-backed - indexes while reinstall is running. diff --git a/docs/superpowers/specs/2026-08-31-eql-safe-reinstall-design.md b/docs/superpowers/specs/2026-08-31-eql-safe-reinstall-design.md deleted file mode 100644 index 3ce545ced..000000000 --- a/docs/superpowers/specs/2026-08-31-eql-safe-reinstall-design.md +++ /dev/null @@ -1,128 +0,0 @@ -# EQL safe reinstall — durable data and reconstructed indexes - -Status: proposed -Date: 2026-08-31 -Issues: cipherstash/stack#959, cipherstash/stack#918 -ADR: `docs/adr/0001-eql-data-survives-disposable-schema-reinstall.md` - -## 1. Goal - -Make the existing drop-and-reinstall lifecycle safe without introducing -object-by-object upgrade scripts. Every encrypted application table, column, -domain type, and stored value must survive. Functional search indexes are -captured and rebuilt as derived state. Dependencies that cannot be reconstructed -mechanically stop the operation before the first destructive statement. - -## 2. Persistence boundary - -### Durable - -- Application tables and rows. -- Columns typed with any data-bearing `public.eql_v3_*` domain. -- The bytes stored in those columns. -- The data-bearing public domains themselves. - -### Disposable - -- `eql_v3` and `eql_v3_internal`. -- Query-operand domains, functions, operators, aggregates, and internal term - types owned by those schemas. - -### Reconstructable - -- Functional indexes whose complete definitions can be obtained with - `pg_get_indexdef()`. - -### Fail-closed - -- RLS policies, constraints, views, generated expressions, triggers, and any - other customer-owned object depending on disposable EQL machinery. -- Unknown dependency classes. - -## 3. EQL artifact requirements - -The installer and uninstaller may continue dropping the EQL-owned schemas with -`CASCADE`. They must never explicitly drop a `public.eql_v3_*` data-bearing -domain. Every data-bearing domain must be idempotently retained when it already -exists. - -The SQLx lifecycle suite must discover every installed data-bearing public EQL -domain from PostgreSQL's catalog. For each domain it must create an application -table, insert a real cipherstash-client-generated payload accepted by that -domain, uninstall, reinstall, and prove that the table, column type, row count, -and JSONB value are unchanged. - -## 4. CLI reinstall protocol - -`stash eql upgrade` and force-install use one protocol: - -The protocol runs inside a schema-migration maintenance window. Its advisory -lock serializes cooperating EQL lifecycle commands; it cannot serialize -arbitrary DDL issued by unrelated PostgreSQL sessions without superuser-only -event triggers. Application migrations must not run concurrently. - -1. Acquire an advisory lock preventing concurrent EQL lifecycle operations. -2. Discover every customer-owned object with a dependency path to - `eql_v3` or `eql_v3_internal`. -3. Partition dependencies into reconstructable functional indexes and - fail-closed objects. -4. If any fail-closed or unknown dependency exists, print an inventory and exit - before executing installer SQL. -5. Capture each index's identity and `pg_get_indexdef()` output, including - schema-qualified table and index names. -6. Begin one transaction, execute the shipped installer, and recreate captured - indexes before commit. Use the original definition by default; any - concurrent-rebuild mode must account explicitly for PostgreSQL's transaction - restrictions. -7. `ANALYZE` affected tables. -8. Verify every captured index exists, is valid and ready, and still has the - exact server-rendered definition captured before replacement. Query-level - engagement remains the responsibility of `stash eql validate`, which has - the application schema needed to construct representative predicates. -9. Commit only after verification, then release the advisory lock. - -If installer execution or index reconstruction fails, the transaction rolls -back to the previous schemas and indexes. The command exits non-zero and prints -the exact captured definition. It must never report a successful upgrade while -an index is absent or invalid. - -## 5. Dependency discovery - -Discovery follows `pg_depend` transitively from objects in the two disposable -schemas to customer-owned objects. It must not rely only on `pg_indexes`, because -that misses policies, views, constraints, generated expressions, and indirect -dependencies. - -The classifier is an allowlist: only ordinary functional indexes with a complete -server-rendered definition are automatically reconstructable. Every unrecognised -class is fail-closed. - -Uniqueness, predicates, included columns, tablespaces, -storage parameters, quoting, and non-`public` application schemas require test -coverage before their corresponding index form enters the allowlist. - -## 6. Acceptance criteria - -- The lifecycle test covers every installed data-bearing public EQL domain with - real encrypted fixtures and passes on every supported PostgreSQL version. -- Uninstall and reinstall preserve table OIDs, column identities, domain types, - row counts, and stored JSONB values. -- A reinstall with no external dependencies succeeds normally. -- A reinstall with supported functional indexes rebuilds and validates them. -- Unique, partial, expression, quoted-name, and non-public-schema - index cases are either proven safe or rejected before mutation. -- Partitioned indexes are rejected before mutation. Rebuilding their attachment - tree can exhaust PostgreSQL's default lock table inside the bundle transaction. -- A policy, constraint, view, generated column, trigger, or unknown dependency - aborts before schema drop and appears in the diagnostic inventory. -- Installer failure leaves the previous installation and indexes intact. -- Index reconstruction failure is non-zero, names the index and rolls back the - entire replacement. -- Re-running after a failed reconstruction is safe and deterministic. - -## 7. Explicit non-goals - -- Versioned object-by-object EQL upgrade scripts. -- Immutable per-release implementation schemas. -- Preserving functional index OIDs across reinstall. -- Automatically rewriting customer policies, constraints, or views. diff --git a/packages/cli/README.md b/packages/cli/README.md index 22461ec7f..cf68abf9f 100644 --- a/packages/cli/README.md +++ b/packages/cli/README.md @@ -210,15 +210,12 @@ npx stash eql upgrade [options] | `--dry-run` | Show what would happen without making changes | | `--supabase` | Use Supabase-compatible upgrade | -The install SQL is safe to re-run: encrypted columns and rows live outside the -disposable EQL schemas and are never dropped. Before replacing those schemas, -the CLI takes a database lifecycle lock, captures dependent functional indexes, -then rebuilds and verifies their definitions in the same transaction. Any other external -dependency (for example a policy or view) is reported and the operation refuses -before changing the database. If index reconstruction fails, the transaction -restores the previous EQL schemas and indexes. It never reports a partially -indexed database as successfully upgraded. If EQL is not installed, -the command suggests running `npx stash eql install` instead. +Encrypted columns and rows live outside the disposable EQL schemas. Before +replacing those schemas, the CLI captures dependent functional indexes and then +restores their definitions and supported catalog properties in the same +transaction. It refuses unsupported dependencies before mutation and rolls back +if restoration fails. If EQL is absent, the command suggests +`npx stash eql install` instead. Run upgrade in a schema-migration maintenance window. Its advisory lock prevents overlapping `stash` lifecycle commands, but unrelated sessions must not create, @@ -317,7 +314,9 @@ Reads `databaseUrl` from `stash.config.ts`. Use `eql migration` to add the EQL v3 installation to your migration history instead of applying it directly. The install then ships to every environment through the same migrate step as the rest of your schema. -**The re-run protections are in the CLI, not in the emitted SQL.** `eql install` and `eql upgrade` take the lifecycle lock, capture dependent functional indexes, refuse on unsupported dependants, and rebuild afterwards — all of that lives in the stash installer. A generated migration is the raw bundle, so applying it through drizzle-kit, the Supabase CLI, or any other migration runner performs the `DROP SCHEMA ... CASCADE` with none of those steps. On a first install there is nothing to lose. Re-applying one over a database that already has EQL and functional indexes on EQL expressions drops those indexes and does not rebuild them; use `eql upgrade` for that, or recreate the indexes in the same migration. +**Generated migrations contain the raw EQL bundle, not the CLI's reinstall +protocol.** A first install is safe. To replace an existing installation, use +`eql upgrade` or recreate every dependent object in the same migration. ### Drizzle diff --git a/packages/cli/src/__tests__/installer.test.ts b/packages/cli/src/__tests__/installer.test.ts index ee2141610..fbb5457ec 100644 --- a/packages/cli/src/__tests__/installer.test.ts +++ b/packages/cli/src/__tests__/installer.test.ts @@ -299,16 +299,100 @@ describe('EQLInstaller', () => { ) }) - it('reads a legacy installed version with one query', async () => { + it('reads a legacy installed version', async () => { mockConnect.mockResolvedValue(undefined) mockEnd.mockResolvedValue(undefined) - mockQuery.mockResolvedValue({ rows: [{ version: '3.0.5' }], rowCount: 1 }) + mockQuery + .mockResolvedValueOnce({ rows: [{ installed: true }], rowCount: 1 }) + .mockResolvedValueOnce({ rows: [{ version: '3.0.5' }], rowCount: 1 }) const { EQLInstaller } = await import('@/installer/index.ts') const installer = new EQLInstaller({ databaseUrl: 'postgres://test' }) await expect(installer.getInstalledVersion()).resolves.toBe('3.0.5') - expect(mockQuery).toHaveBeenCalledTimes(1) - expect(mockQuery).toHaveBeenCalledWith('SELECT eql_v3.version() AS version') + }) + + it('reads the version when the schema exists but is hidden from information_schema', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + mockQuery.mockImplementation((sql: string) => { + if (sql.includes('to_regnamespace')) { + return Promise.resolve({ rows: [{ installed: true }], rowCount: 1 }) + } + if (sql.includes('information_schema.schemata')) { + return Promise.resolve({ rows: [], rowCount: 0 }) + } + if (sql.includes('eql_v3.version()')) { + return Promise.resolve({ rows: [{ version: '3.0.5' }], rowCount: 1 }) + } + return Promise.resolve({ rows: [], rowCount: 0 }) + }) + const { EQLInstaller } = await import('@/installer/index.ts') + + await expect( + new EQLInstaller({ + databaseUrl: 'postgres://test', + }).getInstalledVersion(), + ).resolves.toBe('3.0.5') + }) + + it('reports unknown for an installed legacy schema without version()', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + mockQuery + .mockResolvedValueOnce({ rows: [{ installed: true }], rowCount: 1 }) + .mockRejectedValueOnce( + Object.assign(new Error('undefined function'), { code: '42883' }), + ) + const { EQLInstaller } = await import('@/installer/index.ts') + + await expect( + new EQLInstaller({ + databaseUrl: 'postgres://test', + }).getInstalledVersion(), + ).resolves.toBe('unknown') + }) + + it('frames connection failures from legacy installation detection', async () => { + mockConnect.mockRejectedValue(new Error('connection refused')) + mockEnd.mockResolvedValue(undefined) + const { EQLInstaller } = await import('@/installer/index.ts') + + await expect( + new EQLInstaller({ databaseUrl: 'postgres://test' }).isInstalled(), + ).rejects.toThrow('Failed to connect to database: connection refused') + }) + + it('preserves lifecycle lock timeout guidance without rollback narration', async () => { + vi.useFakeTimers() + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + mockQuery.mockImplementation((sql: string) => { + if (sql.includes('pg_try_advisory_xact_lock')) { + return Promise.resolve({ rows: [{ acquired: false }], rowCount: 1 }) + } + return Promise.resolve({ rows: [], rowCount: 0 }) + }) + try { + const { EQLInstaller } = await import('@/installer/index.ts') + const { EqlLifecycleLockTimeoutError } = await import( + '../installer/derived-search-index-restoration.js' + ) + const installing = new EQLInstaller({ + databaseUrl: 'postgres://test', + }).install() + const outcome = installing.catch((error: unknown) => error) + + await vi.advanceTimersByTimeAsync(300_000) + + const error = await outcome + expect(error).toBeInstanceOf(EqlLifecycleLockTimeoutError) + expect(error).not.toHaveProperty( + 'message', + expect.stringMatching(/Failed to install EQL/), + ) + } finally { + vi.useRealTimers() + } }) it('installs only the pinned EQL v3 bundle', async () => { @@ -397,6 +481,28 @@ describe('EQLInstaller', () => { ) }) + it('restores explicit per-column index statistics targets', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + const database = new RecordingRestorationDatabase( + searchIndexRestorationScenario({ + statisticsTargets: [750], + statisticsSql: [ + 'ALTER INDEX app.users_email_idx ALTER COLUMN 1 SET STATISTICS 750', + ], + }), + ) + mockQuery.mockImplementation(database.query) + const { EQLInstaller } = await import('@/installer/index.ts') + + await new EQLInstaller({ databaseUrl: 'postgres://test' }).install() + + expect(database.events).toContain('statistics') + expect(database.events.indexOf('statistics')).toBeLessThan( + database.events.indexOf('verify'), + ) + }) + it('captures dependencies before destructive SQL in the protected transaction', async () => { mockConnect.mockResolvedValue(undefined) mockEnd.mockResolvedValue(undefined) @@ -485,6 +591,29 @@ describe('EQLInstaller', () => { ]) }) + it('refuses before mutation when index catalog metadata is incomplete', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + const database = new RecordingRestorationDatabase(undefined, { + incompleteCaptureMetadata: true, + }) + mockQuery.mockImplementation(database.query) + const { EQLInstaller } = await import('@/installer/index.ts') + + await expect( + new EQLInstaller({ databaseUrl: 'postgres://test' }).install(), + ).rejects.toThrow( + /refused before making changes.*incomplete catalog metadata/s, + ) + expect(database.events).toEqual([ + 'begin', + 'lock', + 'configure', + 'capture', + 'rollback', + ]) + }) + it('rolls back schema replacement when index rebuild fails', async () => { mockConnect.mockResolvedValue(undefined) mockEnd.mockResolvedValue(undefined) @@ -514,6 +643,57 @@ describe('EQLInstaller', () => { ]) }) + it('rolls back schema replacement when rebuilt index verification fails', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + const scenario = searchIndexRestorationScenario() + const database = new RecordingRestorationDatabase(scenario, { + verificationOverrides: { valid: false }, + }) + mockQuery.mockImplementation(database.query) + const { EQLInstaller } = await import('@/installer/index.ts') + + await expect( + new EQLInstaller({ databaseUrl: 'postgres://test' }).install(), + ).rejects.toThrow(/missing, invalid, or changed.*app\.users_email_idx/s) + expect(database.events.slice(-2)).toEqual(['verify', 'rollback']) + expect(database.events).not.toContain('commit') + }) + + it('rolls back when a rebuilt index has a different owner', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + const database = new RecordingRestorationDatabase( + searchIndexRestorationScenario(), + { verificationOverrides: { owner: 'unexpected_owner' } }, + ) + mockQuery.mockImplementation(database.query) + const { EQLInstaller } = await import('@/installer/index.ts') + + await expect( + new EQLInstaller({ databaseUrl: 'postgres://test' }).install(), + ).rejects.toThrow(/missing, invalid, or changed.*app\.users_email_idx/s) + expect(database.events.slice(-2)).toEqual(['verify', 'rollback']) + expect(database.events).not.toContain('commit') + }) + + it('rolls back when a rebuilt index has different statistics targets', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + const database = new RecordingRestorationDatabase( + searchIndexRestorationScenario({ statisticsTargets: [750] }), + { verificationOverrides: { statisticsTargets: [100] } }, + ) + mockQuery.mockImplementation(database.query) + const { EQLInstaller } = await import('@/installer/index.ts') + + await expect( + new EQLInstaller({ databaseUrl: 'postgres://test' }).install(), + ).rejects.toThrow(/missing, invalid, or changed.*app\.users_email_idx/s) + expect(database.events.slice(-2)).toEqual(['verify', 'rollback']) + expect(database.events).not.toContain('commit') + }) + it('grants both EQL v3 schemas to Supabase roles when the role is a member of postgres', async () => { mockConnect.mockResolvedValue(undefined) mockQuery.mockImplementation((sql: string) => { diff --git a/packages/cli/src/commands/db/__tests__/install.test.ts b/packages/cli/src/commands/db/__tests__/install.test.ts new file mode 100644 index 000000000..611048b6b --- /dev/null +++ b/packages/cli/src/commands/db/__tests__/install.test.ts @@ -0,0 +1,93 @@ +import { beforeEach, describe, expect, it, vi } from 'vitest' +import { CliExit } from '@/cli/exit.js' +import { EqlReinstallRefusalError } from '@/installer/derived-search-index-restoration.js' + +const install = vi.fn() +const spinner = { start: vi.fn(), stop: vi.fn() } + +vi.mock('@clack/prompts', () => ({ + intro: vi.fn(), + outro: vi.fn(), + note: vi.fn(), + spinner: () => spinner, + log: { info: vi.fn(), warn: vi.fn(), error: vi.fn() }, +})) +vi.mock('@/commands/init/utils.js', () => ({ + detectPackageManager: () => 'pnpm', + runnerCommand: (_pm: string, command: string) => command, +})) +vi.mock('@/config/database-url.js', () => ({ + resolveDatabaseUrl: ({ databaseUrlFlag }: { databaseUrlFlag?: string }) => + databaseUrlFlag ?? 'postgres://test', +})) +vi.mock('@/config/index.js', () => ({ + findConfigFile: () => null, + loadStashConfig: vi.fn(), +})) +vi.mock('../client-scaffold.js', () => ({ ensureEncryptionClient: vi.fn() })) +vi.mock('../config-scaffold.js', () => ({ offerStashConfig: vi.fn() })) +vi.mock('../grants-report.js', () => ({ + reportSupabaseGrantsOutcome: vi.fn(), +})) +vi.mock('@/installer/index.js', () => ({ + EQLInstaller: class { + install = install + }, +})) +vi.mock('@/installer/installation-state.js', () => ({ + assessEqlInstallation: () => + Promise.resolve({ + v3: { status: 'absent' }, + capabilities: { + status: 'assessed', + preflight: { + ok: true, + currentUser: 'installer', + isSuperuser: true, + memberOfPostgres: false, + missing: [], + }, + }, + }), +})) +vi.mock('../detect.js', () => ({ + detectPrismaNext: () => null, + detectSupabase: () => false, +})) + +describe('installCommand', () => { + beforeEach(() => vi.clearAllMocks()) + + it('renders a reinstall refusal as an expected command failure', async () => { + const refusal = new EqlReinstallRefusalError('reinstall refused') + install.mockRejectedValueOnce(refusal) + + const { installCommand } = await import('../install.js') + await expect( + installCommand({ + databaseUrl: 'postgres://test', + force: true, + scaffoldConfig: 'skip', + }), + ).rejects.toEqual(new CliExit(1)) + + expect(spinner.stop).toHaveBeenLastCalledWith('EQL installation failed.') + expect( + vi.mocked((await import('@clack/prompts')).log.error), + ).toHaveBeenCalledWith('reinstall refused') + }) + + it('preserves an unexpected install error', async () => { + const error = new Error('database disappeared') + install.mockRejectedValueOnce(error) + + const { installCommand } = await import('../install.js') + await expect( + installCommand({ + databaseUrl: 'postgres://test', + force: true, + scaffoldConfig: 'skip', + }), + ).rejects.toBe(error) + }) +}) diff --git a/packages/cli/src/commands/db/__tests__/status.test.ts b/packages/cli/src/commands/db/__tests__/status.test.ts new file mode 100644 index 000000000..b17123dbd --- /dev/null +++ b/packages/cli/src/commands/db/__tests__/status.test.ts @@ -0,0 +1,52 @@ +import { beforeEach, describe, expect, it, vi } from 'vitest' + +const assess = vi.fn() +const logError = vi.fn() +const logInfo = vi.fn() +const spinner = { start: vi.fn(), stop: vi.fn() } + +vi.mock('@clack/prompts', () => ({ + intro: vi.fn(), + outro: vi.fn(), + spinner: () => spinner, + log: { error: logError, info: logInfo, success: vi.fn(), warn: vi.fn() }, +})) +vi.mock('@/commands/init/utils.js', () => ({ + detectPackageManager: () => 'pnpm', + runnerCommand: (_pm: string, command: string) => command, +})) +vi.mock('@/config/index.js', () => ({ + loadStashConfig: () => ({ databaseUrl: 'postgres://test' }), +})) +vi.mock('@/installer/installation-state.js', () => ({ + assessEqlInstallation: assess, +})) + +describe('statusCommand advisory sections', () => { + beforeEach(() => vi.clearAllMocks()) + + it('continues to ORE when the independent permission assessment fails', async () => { + assess + .mockResolvedValueOnce({ + v2: { status: 'absent' }, + v3: { status: 'installed', version: '3.0.5' }, + capabilities: { status: 'not-requested' }, + ore: { + status: 'observed', + state: 'indexable', + opclassPresent: true, + poisonedDomains: 0, + expectedPoisoned: 20, + }, + surface: { status: 'not-requested' }, + }) + .mockRejectedValueOnce(new Error('permission probe failed')) + + const { statusCommand } = await import('../status.js') + await statusCommand() + + expect(assess).toHaveBeenCalledTimes(2) + expect(logError).toHaveBeenCalledWith('permission probe failed') + expect(logInfo).toHaveBeenCalledWith(expect.stringContaining('usable')) + }) +}) diff --git a/packages/cli/src/commands/db/__tests__/upgrade.test.ts b/packages/cli/src/commands/db/__tests__/upgrade.test.ts new file mode 100644 index 000000000..8f0f109b2 --- /dev/null +++ b/packages/cli/src/commands/db/__tests__/upgrade.test.ts @@ -0,0 +1,74 @@ +import { beforeEach, describe, expect, it, vi } from 'vitest' +import { CliExit } from '@/cli/exit.js' +import { EqlReinstallRefusalError } from '@/installer/derived-search-index-restoration.js' + +const assess = vi.fn() +const install = vi.fn() +const logInfo = vi.fn() +const logError = vi.fn() +const spinner = { start: vi.fn(), stop: vi.fn() } + +vi.mock('@clack/prompts', () => ({ + intro: vi.fn(), + outro: vi.fn(), + note: vi.fn(), + spinner: () => spinner, + log: { info: logInfo, warn: vi.fn(), error: logError }, +})) +vi.mock('@/commands/init/utils.js', () => ({ + detectPackageManager: () => 'pnpm', + runnerCommand: (_pm: string, command: string) => command, +})) +vi.mock('@/config/index.js', () => ({ + loadStashConfig: () => ({ databaseUrl: 'postgres://test' }), +})) +vi.mock('@/installer/installation-state.js', () => ({ + assessEqlInstallation: assess, +})) +vi.mock('@/installer/index.js', () => ({ + EQLInstaller: class { + install = install + }, +})) + +describe('upgradeCommand version reporting', () => { + beforeEach(() => vi.clearAllMocks()) + + it('does not call two unknown versions unchanged', async () => { + assess.mockResolvedValue({ + v3: { status: 'installed', version: 'unknown' }, + }) + install.mockResolvedValue({ deferredGrantsSql: null }) + + const { upgradeCommand } = await import('../upgrade.js') + await upgradeCommand({}) + + expect(logInfo).not.toHaveBeenCalledWith( + 'Version unchanged — EQL was already up to date.', + ) + }) + + it('renders a reinstall refusal as an expected command failure', async () => { + assess.mockResolvedValue({ + v3: { status: 'installed', version: '3.0.4' }, + }) + install.mockRejectedValue(new EqlReinstallRefusalError('reinstall refused')) + + const { upgradeCommand } = await import('../upgrade.js') + await expect(upgradeCommand({})).rejects.toEqual(new CliExit(1)) + + expect(spinner.stop).toHaveBeenLastCalledWith('EQL upgrade failed.') + expect(logError).toHaveBeenCalledWith('reinstall refused') + }) + + it('preserves an unexpected upgrade error', async () => { + assess.mockResolvedValue({ + v3: { status: 'installed', version: '3.0.4' }, + }) + const error = new Error('database disappeared') + install.mockRejectedValue(error) + + const { upgradeCommand } = await import('../upgrade.js') + await expect(upgradeCommand({})).rejects.toBe(error) + }) +}) diff --git a/packages/cli/src/commands/db/install.ts b/packages/cli/src/commands/db/install.ts index c0a051fee..5cd965595 100644 --- a/packages/cli/src/commands/db/install.ts +++ b/packages/cli/src/commands/db/install.ts @@ -1,8 +1,10 @@ import { installMigrationsSchema } from '@cipherstash/migrate' import * as p from '@clack/prompts' +import { CliExit } from '@/cli/exit.js' import { resolveDatabaseUrl } from '@/config/database-url.js' import { findConfigFile, loadStashConfig } from '@/config/index.js' import { createPgClient } from '@/db/client.js' +import { EqlReinstallRefusalError } from '@/installer/derived-search-index-restoration.js' import { EQLInstaller } from '@/installer/index.js' import { assessEqlInstallation } from '@/installer/installation-state.js' import { describeOreState } from '@/installer/ore.js' @@ -200,7 +202,18 @@ export async function installCommand( } s.start('Installing EQL v3 extensions (pinned bundle)...') - const installResult = await installer.install({ supabase }) + let installResult: Awaited> + try { + installResult = await installer.install({ supabase }) + } catch (error) { + s.stop('EQL installation failed.') + if (error instanceof EqlReinstallRefusalError) { + p.log.error(error.message) + p.outro('Installation aborted.') + throw new CliExit(1) + } + throw error + } s.stop('EQL extensions installed.') if (supabase) reportSupabaseGrantsOutcome(installResult) diff --git a/packages/cli/src/commands/db/status.ts b/packages/cli/src/commands/db/status.ts index d6adec995..c3eac84c1 100644 --- a/packages/cli/src/commands/db/status.ts +++ b/packages/cli/src/commands/db/status.ts @@ -25,7 +25,6 @@ export async function statusCommand(options: { databaseUrl?: string } = {}) { try { installation = await assessEqlInstallation({ databaseUrl: config.databaseUrl, - includeCapabilities: true, includeOre: true, }) } catch (error) { @@ -66,10 +65,14 @@ export async function statusCommand(options: { databaseUrl?: string } = {}) { s.start('Checking database permissions...') try { - if (installation.capabilities.status !== 'assessed') { + const capabilityAssessment = await assessEqlInstallation({ + databaseUrl: config.databaseUrl, + includeCapabilities: true, + }) + if (capabilityAssessment.capabilities.status !== 'assessed') { throw new Error('Database capabilities were not assessed') } - const permissions = installation.capabilities.preflight + const permissions = capabilityAssessment.capabilities.preflight s.stop('Permissions checked.') if (permissions.ok) { diff --git a/packages/cli/src/commands/db/upgrade.ts b/packages/cli/src/commands/db/upgrade.ts index 2573b6f86..156923e6e 100644 --- a/packages/cli/src/commands/db/upgrade.ts +++ b/packages/cli/src/commands/db/upgrade.ts @@ -1,6 +1,8 @@ import * as p from '@clack/prompts' +import { CliExit } from '@/cli/exit.js' import { detectPackageManager, runnerCommand } from '@/commands/init/utils.js' import { loadStashConfig } from '@/config/index.js' +import { EqlReinstallRefusalError } from '@/installer/derived-search-index-restoration.js' import { EQLInstaller } from '@/installer/index.js' import { assessEqlInstallation } from '@/installer/installation-state.js' import { reportSupabaseGrantsOutcome } from './grants-report.js' @@ -35,12 +37,13 @@ export async function upgradeCommand(options: { process.exit(1) } - const previousVersion = before.v3.version - s.stop(`Current version: ${previousVersion}`) + const previousVersion = + before.v3.version === 'unknown' ? null : before.v3.version + s.stop(`Current version: ${previousVersion ?? 'unknown'}`) if (options.dryRun) { p.log.info('Dry run — no changes will be made.') p.note( - `Current version: ${previousVersion}\nWould re-run the pinned EQL v3 install SQL against the database`, + `Current version: ${previousVersion ?? 'unknown'}\nWould re-run the pinned EQL v3 install SQL against the database`, 'Dry Run', ) p.outro('Dry run complete.') @@ -48,16 +51,29 @@ export async function upgradeCommand(options: { } s.start('Upgrading EQL v3 extensions (pinned bundle)...') - const result = await installer.install({ supabase: options.supabase }) + let result: Awaited> + try { + result = await installer.install({ supabase: options.supabase }) + } catch (error) { + s.stop('EQL upgrade failed.') + if (error instanceof EqlReinstallRefusalError) { + p.log.error(error.message) + p.outro('Upgrade aborted.') + throw new CliExit(1) + } + throw error + } s.stop('EQL extensions upgraded.') if (options.supabase) reportSupabaseGrantsOutcome(result) s.start('Verifying new version...') const after = await assessEqlInstallation({ databaseUrl: config.databaseUrl }) const newVersion = - after.v3.status === 'installed' ? after.v3.version : 'unknown' - s.stop(`New version: ${newVersion}`) - if (previousVersion === newVersion) { + after.v3.status === 'installed' && after.v3.version !== 'unknown' + ? after.v3.version + : null + s.stop(`New version: ${newVersion ?? 'unknown'}`) + if (previousVersion && newVersion && previousVersion === newVersion) { p.log.info('Version unchanged — EQL was already up to date.') } p.outro('Done!') diff --git a/packages/cli/src/installer/__tests__/installation-state.test.ts b/packages/cli/src/installer/__tests__/installation-state.test.ts index 388e95ba9..64975978f 100644 --- a/packages/cli/src/installer/__tests__/installation-state.test.ts +++ b/packages/cli/src/installer/__tests__/installation-state.test.ts @@ -1,3 +1,4 @@ +import { releaseManifest } from '@cipherstash/eql/sql' import { beforeEach, describe, expect, it, vi } from 'vitest' const query = vi.fn() @@ -75,7 +76,8 @@ describe('EQL installation state', () => { } if (sql.includes('eql_v3.version()')) { versionReads += 1 - if (versionReads === 1) return { rows: [{ version: '3.0.5' }] } + if (versionReads === 1) + return { rows: [{ version: releaseManifest.eqlVersion }] } aborted = true throw Object.assign(new Error('undefined function'), { code: '42883' }) } @@ -132,7 +134,7 @@ describe('EQL installation state', () => { } } if (sql.includes('eql_v3.version()')) - return { rows: [{ version: '3.0.5' }] } + return { rows: [{ version: releaseManifest.eqlVersion }] } if (sql.includes('ore_opclass_present')) throw new Error('catalog unavailable') return { rows: [] } @@ -150,4 +152,97 @@ describe('EQL installation state', () => { }) expect(query).toHaveBeenCalledWith('COMMIT') }) + + it('reports one authoritative schema-presence observation across installation and capabilities', async () => { + query.mockImplementation(async (sql: string) => { + if (sql.includes("to_regnamespace('eql_v2')")) { + return { + rows: [ + { + eql_v2_present: false, + eql_v3_present: true, + eql_v3_internal_present: true, + }, + ], + } + } + if (sql.includes('eql_v3.version()')) + return { rows: [{ version: releaseManifest.eqlVersion }] } + if (sql.includes('current_user AS role_name')) { + return { + rows: [ + { + role_name: 'restricted_role', + is_superuser: false, + member_of_postgres: false, + has_database_create: true, + has_public_create: true, + pgcrypto_installed: true, + pgcrypto_schema: 'public', + // information_schema can hide schemas that to_regnamespace sees. + eql_v3_present: false, + eql_v3_internal_present: false, + can_drop_eql_v3: false, + can_drop_eql_v3_internal: false, + }, + ], + } + } + return { rows: [] } + }) + + const { assessEqlInstallation } = await import('../installation-state.js') + const state = await assessEqlInstallation({ + databaseUrl: 'postgres://test', + includeCapabilities: true, + }) + + expect(state.v3.status).toBe('installed') + expect(state.capabilities).toMatchObject({ + status: 'assessed', + preflight: { + eqlV3SchemaPresent: true, + eqlV3InternalSchemaPresent: true, + }, + }) + }) + + it('exhaustively reports a missing EQL installation', async () => { + query.mockImplementation(async (sql: string) => { + if (sql.includes("to_regnamespace('eql_v2')")) { + return { + rows: [ + { + eql_v2_present: false, + eql_v3_present: false, + eql_v3_internal_present: false, + }, + ], + } + } + if (sql.includes('pgcrypto_installed')) { + return { + rows: [ + { + eql_v3_present: false, + eql_v3_internal_present: false, + pgcrypto_installed: false, + pgcrypto_schema: null, + }, + ], + } + } + return { rows: [] } + }) + + const { assessEqlInstallation } = await import('../installation-state.js') + const state = await assessEqlInstallation({ + databaseUrl: 'postgres://test', + depth: 'exhaustive', + }) + + expect(state.surface.status).toBe('damaged') + if (state.surface.status !== 'damaged') return + expect(state.surface.report.status).toBe('not-installed') + }) }) diff --git a/packages/cli/src/installer/__tests__/reinstall.live.test.ts b/packages/cli/src/installer/__tests__/reinstall.live.test.ts index e94242e74..250dc880d 100644 --- a/packages/cli/src/installer/__tests__/reinstall.live.test.ts +++ b/packages/cli/src/installer/__tests__/reinstall.live.test.ts @@ -10,9 +10,11 @@ */ import { afterAll, beforeEach, describe, expect, it } from 'vitest' -import { derivedSearchIndexRestorationTestSeam } from '../derived-search-index-restoration.js' -import { EQLInstaller, loadBundledEqlSql } from '../index.js' -import { parseExpectedSurface } from '../verify.js' +import { + derivedSearchIndexRestorationTestSeam, + EqlLifecycleLockTimeoutError, +} from '../derived-search-index-restoration.js' +import { EQLInstaller } from '../index.js' import { LiveRestorationDatabase, searchIndexRestorationScenario, @@ -183,6 +185,57 @@ describeLive('EQLInstaller safe reinstall — live Postgres', () => { ]) }, 180_000) + it('preserves index ownership and explicit statistics targets', async () => { + await query(` + CREATE INDEX records_encrypted_idx + ON stash_reinstall_test.records (eql_v3.eq_term(encrypted)); + ALTER INDEX stash_reinstall_test.records_encrypted_idx + ALTER COLUMN 1 SET STATISTICS 750; + `) + + await new EQLInstaller({ databaseUrl: DATABASE_URL ?? '' }).install() + + expect( + await query<{ owner: string; statistics_target: number }>(` + SELECT pg_catalog.pg_get_userbyid(index_class.relowner) AS owner, + attribute.attstattarget AS statistics_target + FROM pg_catalog.pg_class index_class + JOIN pg_catalog.pg_attribute attribute + ON attribute.attrelid = index_class.oid AND attribute.attnum = 1 + WHERE index_class.oid = + 'stash_reinstall_test.records_encrypted_idx'::regclass + `), + ).toEqual([{ owner: 'cipherstash', statistics_target: 750 }]) + }, 180_000) + + it('preserves the default pre-Postgres-17 statistics target representation', async () => { + const [{ server_version_num: serverVersion }] = await query<{ + server_version_num: string + }>('SHOW server_version_num') + if (Number(serverVersion) >= 170000) return + + await query(` + CREATE INDEX records_encrypted_idx + ON stash_reinstall_test.records (eql_v3.eq_term(encrypted)); + `) + const statisticsTarget = async () => + query<{ statistics_target: number }>(` + SELECT attribute.attstattarget AS statistics_target + FROM pg_catalog.pg_attribute attribute + WHERE attribute.attrelid = + 'stash_reinstall_test.records_encrypted_idx'::regclass + AND attribute.attnum = 1 + `) + + await expect(statisticsTarget()).resolves.toEqual([ + { statistics_target: -1 }, + ]) + await new EQLInstaller({ databaseUrl: DATABASE_URL ?? '' }).install() + await expect(statisticsTarget()).resolves.toEqual([ + { statistics_target: -1 }, + ]) + }, 180_000) + /** * `format_type()` omits the schema whenever the type is visible on the * current search_path, while the identities parsed out of the bundle always @@ -194,11 +247,10 @@ describeLive('EQLInstaller safe reinstall — live Postgres', () => { * in `format_type()`, not in our SQL. */ it("exempts the bundle's own operators and casts when the EQL schemas are on the search_path", async () => { - const expected = parseExpectedSurface(loadBundledEqlSql()) const rows = await postgres.withEqlSearchPath().dependencyInventory<{ dependency_kind: string identity: string - }>(expected.operators, expected.casts) + }>() const unsafe = rows .filter((row) => row.dependency_kind !== 'index') .map((row) => row.identity) @@ -211,36 +263,51 @@ describeLive('EQLInstaller safe reinstall — live Postgres', () => { }) }, 60_000) - /** - * The negative control for the test above. An empty result proves nothing on - * its own — it is also what an EMPTY dependency graph looks like. Withhold - * exactly the operators the bundle declares with a `pg_catalog` operand - * (`text`, `text[]`, `jsonb`, `jsonpath`, `integer` — the parser spells those - * bare, with no schema) and they must reappear as `unsafe`. That is what - * pins the other half of the rule: qualifying every operand unconditionally - * would spell these `pg_catalog.text` and silently break their exemption in - * the direction this test, not the one above, can see. - */ - it('exempts operators with a bare pg_catalog operand, and only because of the match', async () => { - const expected = parseExpectedSurface(loadBundledEqlSql()) - const hasBareOperand = (identity: string) => - identity - .slice(identity.indexOf('(') + 1, -1) - .split(', ') - .some((operand) => operand !== 'none' && !operand.includes('.')) - const withheld = expected.operators.filter(hasBareOperand) - expect(withheld.length).toBeGreaterThan(0) - + it('exempts installed EQL operators independently of the pinned bundle', async () => { const rows = await postgres .withEqlSearchPath() - .dependencyInventory<{ dependency_kind: string; identity: string }>( - expected.operators.filter((o) => !hasBareOperand(o)), - expected.casts, - ) + .dependencyInventory<{ dependency_kind: string; identity: string }>() - expect(rows.filter((row) => row.dependency_kind !== 'index')).toHaveLength( - withheld.length, - ) + expect(rows.filter((row) => row.dependency_kind !== 'index')).toEqual([]) + }, 60_000) + + it('exempts installed EQL casts independently of the pinned bundle', async () => { + await query(` + CREATE DOMAIN stash_reinstall_test.legacy_encrypted AS jsonb; + CREATE FUNCTION eql_v3.legacy_query_cast(stash_reinstall_test.legacy_encrypted) + RETURNS eql_v3.query_text_eq + LANGUAGE sql IMMUTABLE STRICT + AS 'SELECT jsonb_build_object(''v'', $1)::eql_v3.query_text_eq'; + CREATE CAST ( + stash_reinstall_test.legacy_encrypted AS eql_v3.query_text_eq + ) WITH FUNCTION eql_v3.legacy_query_cast(stash_reinstall_test.legacy_encrypted); + `) + + const rows = await postgres.dependencyInventory<{ + dependency_kind: string + identity: string + }>() + + expect(rows.filter((row) => row.dependency_kind !== 'index')).toEqual([]) + }, 60_000) + + it('keeps a customer operator unsafe when it duplicates an EQL signature', async () => { + await query(`CREATE OPERATOR stash_reinstall_test.= ( + FUNCTION = eql_v3.eq, + LEFTARG = public.eql_v3_text_eq, + RIGHTARG = public.eql_v3_text_eq + )`) + const rows = await postgres.dependencyInventory<{ + dependency_kind: string + identity: string + }>() + const unsafe = rows + .filter((row) => row.dependency_kind !== 'index') + .map((row) => row.identity) + + expect(unsafe).toEqual([ + 'stash_reinstall_test.=(public.eql_v3_text_eq,public.eql_v3_text_eq)', + ]) }, 60_000) it('installs over a connection whose search_path names the EQL schemas', async () => { @@ -335,11 +402,11 @@ describeLive('EQLInstaller safe reinstall — live Postgres', () => { // An explicit short budget: the behaviour under test is "refuses rather // than hangs", which does not depend on how long the wait is, and the - // production default is deliberately a minute. The default's own + // production default is deliberately five minutes. The default's own // property — that an ordinary queued install still succeeds — is the // next test. - await expect(acquireLifecycleLock(blocked, 1_000)).rejects.toThrow( - /another EQL lifecycle operation is in progress/i, + await expect(acquireLifecycleLock(blocked, 1_000)).rejects.toBeInstanceOf( + EqlLifecycleLockTimeoutError, ) await blocked.query('ROLLBACK') await holder.query('COMMIT') diff --git a/packages/cli/src/installer/__tests__/restoration-scenarios.ts b/packages/cli/src/installer/__tests__/restoration-scenarios.ts index 5e81bcd74..bd2cfa930 100644 --- a/packages/cli/src/installer/__tests__/restoration-scenarios.ts +++ b/packages/cli/src/installer/__tests__/restoration-scenarios.ts @@ -10,6 +10,9 @@ export interface SearchIndexRestorationScenario { replicaIdentitySql: string | null comment: string | null commentSql: string | null + owner: string + statisticsTargets: Array + statisticsSql: string[] } export type RestorationEvent = @@ -22,6 +25,7 @@ export type RestorationEvent = | 'cluster' | 'replica-identity' | 'comment' + | 'statistics' | 'analyze' | 'verify' | 'commit' @@ -43,6 +47,9 @@ export function searchIndexRestorationScenario( replicaIdentitySql: null, comment: null, commentSql: null, + owner: 'app_owner', + statisticsTargets: [null], + statisticsSql: [], ...overrides, } } @@ -57,6 +64,8 @@ export class RecordingRestorationDatabase { unsafeIdentity?: string failReconstructionWith?: Error failConfigurationWith?: Error + verificationOverrides?: Partial + incompleteCaptureMetadata?: boolean } = {}, ) {} @@ -71,6 +80,18 @@ export class RecordingRestorationDatabase { throw this.options.failConfigurationWith } if (event === 'capture') { + if (this.options.incompleteCaptureMetadata) { + return { + rows: [ + { + ...captureRow(searchIndexRestorationScenario()), + identity: null, + definition: null, + }, + ], + rowCount: 1, + } + } if (this.options.unsafeIdentity) { return { rows: [ @@ -90,7 +111,15 @@ export class RecordingRestorationDatabase { throw this.options.failReconstructionWith } if (event === 'verify' && this.scenario) { - return { rows: [verificationRow(this.scenario)], rowCount: 1 } + return { + rows: [ + verificationRow({ + ...this.scenario, + ...this.options.verificationOverrides, + }), + ], + rowCount: 1, + } } return { rows: [], rowCount: 0 } } @@ -120,10 +149,9 @@ export class LiveRestorationDatabase { } } - dependencyInventory(operators: string[], casts: string[]): Promise { + dependencyInventory(): Promise { return this.query( derivedSearchIndexRestorationTestSeam.lifecycleDependenciesSql, - [operators, casts], ) } } @@ -142,6 +170,9 @@ function captureRow(scenario: SearchIndexRestorationScenario) { replica_identity_sql: scenario.replicaIdentitySql, comment: scenario.comment, comment_sql: scenario.commentSql, + owner: scenario.owner, + statistics_targets: scenario.statisticsTargets, + statistics_sql: scenario.statisticsSql, } } @@ -154,6 +185,8 @@ function verificationRow(scenario: SearchIndexRestorationScenario) { clustered: scenario.clustered, replica_identity: scenario.replicaIdentity, comment: scenario.comment, + owner: scenario.owner, + statistics_targets: scenario.statisticsTargets, } } @@ -171,6 +204,7 @@ function restorationEvent( if (scenario?.replicaIdentitySql && sql === scenario.replicaIdentitySql) return 'replica-identity' if (scenario?.commentSql && sql === scenario.commentSql) return 'comment' + if (scenario?.statisticsSql.includes(sql)) return 'statistics' if (sql.startsWith('ANALYZE ')) return 'analyze' if (sql.includes('stash_eql_verify_rebuilt_indexes')) return 'verify' if (sql === 'COMMIT') return 'commit' diff --git a/packages/cli/src/installer/__tests__/upgrade-encrypted-indexes.live.test.ts b/packages/cli/src/installer/__tests__/upgrade-encrypted-indexes.live.test.ts index 26cdf39d3..52ac614fe 100644 --- a/packages/cli/src/installer/__tests__/upgrade-encrypted-indexes.live.test.ts +++ b/packages/cli/src/installer/__tests__/upgrade-encrypted-indexes.live.test.ts @@ -5,12 +5,9 @@ */ import { readInstallSql as readBaselineInstallSql } from '@cipherstash/eql-upgrade-baseline/sql' -import { - decryptBulk, - type EncryptConfig, - type EncryptedPayload, - encryptBulk, - encryptQuery, +import type { + EncryptConfig, + EncryptedPayload, newClient, } from '@cipherstash/protect-ffi' import { config as loadEnv } from 'dotenv' @@ -45,9 +42,11 @@ const encryptConfig: EncryptConfig = { describeLive('EQLInstaller upgrade — genuine encrypted indexes', () => { const client = new pg.Client({ connectionString: DATABASE_URL }) + let protectFfi: typeof import('@cipherstash/protect-ffi') let protectClient: Awaited> beforeAll(async () => { + protectFfi = await import('@cipherstash/protect-ffi') await client.connect() await client.query('DROP TABLE IF EXISTS eql_upgrade_records') await client.query(readBaselineInstallSql()) @@ -59,8 +58,8 @@ describeLive('EQLInstaller upgrade — genuine encrypted indexes', () => { ).rows[0].version, ).toBe(BASELINE_VERSION) - protectClient = await newClient({ encryptConfig, eqlVersion: 3 }) - const rows = await encryptBulk(protectClient, { + protectClient = await protectFfi.newClient({ encryptConfig, eqlVersion: 3 }) + const rows = await protectFfi.encryptBulk(protectClient, { plaintexts: [ { plaintext: 'alice@example.com', @@ -104,13 +103,13 @@ describeLive('EQLInstaller upgrade — genuine encrypted indexes', () => { it('upgrades a released installation while preserving usable encrypted indexes', async () => { await new EQLInstaller({ databaseUrl: DATABASE_URL ?? '' }).install() - const emailOperand = await encryptQuery(protectClient, { + const emailOperand = await protectFfi.encryptQuery(protectClient, { plaintext: 'bob@example.com', column: 'email', table: 'eql_upgrade_records', indexType: 'unique', }) - const scoreOperand = await encryptQuery(protectClient, { + const scoreOperand = await protectFfi.encryptQuery(protectClient, { plaintext: 15, column: 'score', table: 'eql_upgrade_records', @@ -126,7 +125,7 @@ describeLive('EQLInstaller upgrade — genuine encrypted indexes', () => { [emailOperand, scoreOperand], ) expect( - await decryptBulk(protectClient, { + await protectFfi.decryptBulk(protectClient, { ciphertexts: result.rows.flatMap(({ email, score }) => [ { ciphertext: email }, { ciphertext: score }, diff --git a/packages/cli/src/installer/__tests__/verify.live.test.ts b/packages/cli/src/installer/__tests__/verify.live.test.ts index cf575f4ad..a8eebe234 100644 --- a/packages/cli/src/installer/__tests__/verify.live.test.ts +++ b/packages/cli/src/installer/__tests__/verify.live.test.ts @@ -20,9 +20,9 @@ import { afterAll, beforeAll, describe, expect, it } from 'vitest' import { EQLInstaller } from '../index.js' import { + assessEqlSurface, bundledExpectedSurface, readInstalledSurface, - readOreState, verifyEqlSurface, } from '../verify.js' @@ -78,7 +78,7 @@ describeLive('verifyEqlSurface — live Postgres', () => { }, 60_000) /** - * `eql status` reads the ORE half through {@link readOreState} rather than + * `eql status` reads the ORE half through the summary assessment rather than * the full surface diff (#891). Both must answer the same question the same * way against the same database — a cheap read that disagreed with `verify` * would be worse than no read at all. @@ -90,7 +90,8 @@ describeLive('verifyEqlSurface — live Postgres', () => { const client = new pg.Client({ connectionString: url }) await client.connect() try { - const ore = await readOreState(client) + await client.query('BEGIN READ ONLY') + const { ore } = await assessEqlSurface(client, 'summary') // The database runs the pinned bundle, so the probe is comparable — // it declines to answer only on a version skew. expect(ore.comparable).toBe(true) @@ -100,6 +101,7 @@ describeLive('verifyEqlSurface — live Postgres', () => { expect(ore.poisonedDomains).toBe(report.ore?.poisonedDomains) expect(ore.expectedPoisoned).toBe(report.ore?.expectedPoisoned) } finally { + await client.query('ROLLBACK').catch(() => undefined) await client.end().catch(() => undefined) } }, 60_000) diff --git a/packages/cli/src/installer/__tests__/verify.test.ts b/packages/cli/src/installer/__tests__/verify.test.ts index a51dcbcd9..4ad79888e 100644 --- a/packages/cli/src/installer/__tests__/verify.test.ts +++ b/packages/cli/src/installer/__tests__/verify.test.ts @@ -2,11 +2,12 @@ import { readInstallSql } from '@cipherstash/eql/sql' import type pg from 'pg' import { describe, expect, it } from 'vitest' import { + assessEqlSurface, diffSurface, type ExpectedSurface, type InstalledSurface, parseExpectedSurface, - readOreState, + readInstalledSurface, } from '../verify.js' /** @@ -387,7 +388,7 @@ describe('diffSurface', () => { }) /** - * A client that answers just the two queries `readOreState` issues, so the + * A client that answers the catalogue queries the summary assessment issues, so the * version gate can be exercised without a database. `undefinedFunction` * spells the 42883 an absent `eql_v3.version()` raises. */ @@ -423,10 +424,10 @@ function fakeOreClient(answers: { return { client: client as unknown as pg.ClientBase, queries } } -describe('readOreState', () => { +describe('assessEqlSurface summary', () => { it('classifies the ORE state when the installed version is the pinned one', async () => { const { client } = fakeOreClient({ version: expected.eqlVersion }) - const reading = await readOreState(client) + const { ore: reading } = await assessEqlSurface(client, 'summary') expect(reading.comparable).toBe(true) if (!reading.comparable) return expect(reading.state).toBe('indexable') @@ -439,7 +440,7 @@ describe('readOreState', () => { opclassPresent: false, poisonedDomains: expected.oreDomains.length, }) - const reading = await readOreState(client) + const { ore: reading } = await assessEqlSurface(client, 'summary') expect(reading.comparable && reading.state).toBe('fallback') }) @@ -454,7 +455,7 @@ describe('readOreState', () => { opclassPresent: false, poisonedDomains: expected.oreDomains.length - 2, }) - const reading = await readOreState(client) + const { ore: reading } = await assessEqlSurface(client, 'summary') expect(reading.comparable).toBe(false) if (reading.comparable) return expect(reading.installedVersion).toBe('3.0.0') @@ -467,9 +468,150 @@ describe('readOreState', () => { it('reports a missing version() as not comparable', async () => { const { client } = fakeOreClient({ version: null }) - const reading = await readOreState(client) + const { ore: reading } = await assessEqlSurface(client, 'summary') expect(reading.comparable).toBe(false) if (reading.comparable) return expect(reading.installedVersion).toBeNull() }) }) + +describe('readInstalledSurface in a caller transaction', () => { + it('restores transaction-local settings before returning', async () => { + const queries: string[] = [] + const client = { + async query(sql: string) { + queries.push(sql) + if (sql.includes('pgcrypto_installed')) { + return { + rows: [ + { + eql_v3_present: false, + eql_v3_internal_present: false, + pgcrypto_installed: false, + pgcrypto_schema: null, + }, + ], + } + } + if (sql.includes('ore_opclass_present')) { + return { rows: [{ ore_opclass_present: false, poisoned_domains: 0 }] } + } + return { rows: [] } + }, + } + + await readInstalledSurface(client as unknown as pg.ClientBase, expected, { + manageTransaction: false, + }) + + expect(queries[0]).toBe('SAVEPOINT installed_eql_surface_read') + expect(queries.at(-2)).toBe( + 'ROLLBACK TO SAVEPOINT installed_eql_surface_read', + ) + expect(queries.at(-1)).toBe('RELEASE SAVEPOINT installed_eql_surface_read') + }) + + it('preserves the version-read error when probe cleanup also fails', async () => { + const client = { + async query(sql: string) { + if (sql.includes('pgcrypto_installed')) { + return { + rows: [ + { + eql_v3_present: true, + eql_v3_internal_present: true, + pgcrypto_installed: true, + pgcrypto_schema: 'public', + }, + ], + } + } + if (sql.includes('ore_opclass_present')) { + return { rows: [{ ore_opclass_present: true, poisoned_domains: 0 }] } + } + if (sql.includes('eql_v3.version()')) { + throw Object.assign(new Error('permission denied'), { code: '42501' }) + } + if (sql === 'ROLLBACK TO SAVEPOINT installed_eql_version_probe') { + throw new Error('cleanup failed') + } + return { rows: [] } + }, + } + + await expect( + readInstalledSurface(client as unknown as pg.ClientBase, expected, { + manageTransaction: false, + }), + ).rejects.toThrow('Could not read eql_v3.version(): permission denied') + }) + + it('preserves result-construction errors after restoring the caller transaction', async () => { + const queries: string[] = [] + const client = { + async query(sql: string) { + queries.push(sql) + if (sql.includes('pgcrypto_installed')) { + return { + rows: [ + { + eql_v3_present: false, + eql_v3_internal_present: false, + pgcrypto_installed: false, + pgcrypto_schema: null, + }, + ], + } + } + if (sql.includes('FROM pg_catalog.pg_proc')) { + return { rows: [{ name: null, signature: '' }] } + } + if (sql.includes('ore_opclass_present')) { + return { rows: [{ ore_opclass_present: false, poisoned_domains: 0 }] } + } + if ( + sql === 'ROLLBACK TO SAVEPOINT installed_eql_surface_read' && + queries.filter((query) => query === sql).length > 1 + ) { + throw new Error('savepoint no longer exists') + } + return { rows: [] } + }, + } + + await expect( + readInstalledSurface(client as unknown as pg.ClientBase, expected, { + manageTransaction: false, + }), + ).rejects.toThrow(/null|toLowerCase/) + + expect( + queries.filter( + (query) => query === 'ROLLBACK TO SAVEPOINT installed_eql_surface_read', + ), + ).toHaveLength(1) + }) + + it('preserves a surface-read error when caller-savepoint cleanup also fails', async () => { + const primary = new Error('catalog read failed') + const queries: string[] = [] + const client = { + async query(sql: string) { + queries.push(sql) + if (sql.includes('FROM pg_catalog.pg_proc')) throw primary + if (sql === 'ROLLBACK TO SAVEPOINT installed_eql_surface_read') { + throw new Error('rollback failed') + } + return { rows: [] } + }, + } + + await expect( + readInstalledSurface(client as unknown as pg.ClientBase, expected, { + manageTransaction: false, + }), + ).rejects.toBe(primary) + + expect(queries).toContain('RELEASE SAVEPOINT installed_eql_surface_read') + }) +}) diff --git a/packages/cli/src/installer/derived-search-index-restoration.ts b/packages/cli/src/installer/derived-search-index-restoration.ts index d47b5bb87..96f2d1fa8 100644 --- a/packages/cli/src/installer/derived-search-index-restoration.ts +++ b/packages/cli/src/installer/derived-search-index-restoration.ts @@ -30,6 +30,8 @@ export class EqlReinstallRefusalError extends Error {} export class EqlReinstallConnectionError extends Error {} +export class EqlLifecycleLockTimeoutError extends Error {} + export class DerivedSearchIndexReconstructionError extends Error {} export class DerivedSearchIndexVerificationError extends Error {} @@ -42,8 +44,6 @@ export interface RestorationSummary { interface RestoreAroundEqlReplacementOptions { databaseUrl: string bundledSql: string - bundleOperators: string[] - bundleCasts: string[] } export interface ReinstallIndex { @@ -58,6 +58,9 @@ export interface ReinstallIndex { replicaIdentitySql: string | null comment: string | null commentSql: string | null + owner: string + statisticsTargets: Array + statisticsSql: string[] } interface DependencyRow { @@ -73,50 +76,14 @@ interface DependencyRow { replica_identity_sql?: unknown comment?: unknown comment_sql?: unknown + owner?: unknown + statistics_targets?: unknown + statistics_sql?: unknown } const LIFECYCLE_DEPENDENCIES_SQL = ` /* stash_eql_lifecycle_dependencies */ WITH RECURSIVE -/* - * A type's name spelled the way \`parseExpectedSurface\` spells it, so the - * operator and cast identities below can be compared against the ones parsed - * out of the bundle. - * - * The rule is NOT \`format_type()\`. That function omits the schema whenever the - * type is visible on the current search_path, while the parser always emits the - * qualification the bundle wrote (\`eql_v3.query_text_eq\`, - * \`eql_v3_internal.ore_block_256\`). On a connection whose search_path names - * eql_v3 — routine on a provisioned database, so an application can call - * \`eq_term()\` unqualified — \`format_type()\` answers a bare \`query_text_eq\`, - * every bundle-owned operator and cast misses its ownership exemption, and the - * installer refuses on a healthy database listing EQL's own operators as - * customer-owned. (#918) - * - * It is not unconditional qualification either: the bundle declares operands in - * \`pg_catalog\` (\`text\`, \`text[]\`, \`jsonb\`, \`jsonpath\`, \`integer\`) and writes - * them bare, so \`pg_catalog.text\` would miss in the other direction. Hence: - * \`format_type()\` for pg_catalog — it also spells the SQL-standard multi-word - * names (\`double precision\`) and array suffixes the parser's alias map targets - * — and an explicit qualification for everything else. - */ -type_identity(oid, identity) AS ( - SELECT t.oid, - CASE - WHEN tn.nspname = 'pg_catalog' - THEN pg_catalog.format_type(t.oid, NULL) - -- An array of a non-catalog type: pg_type calls it \`_eql_v3_text_eq\`, - -- the bundle would write \`public.eql_v3_text_eq[]\`. - WHEN t.typcategory = 'A' AND et.oid IS NOT NULL - THEN pg_catalog.format('%I.%I[]', en.nspname, et.typname) - ELSE pg_catalog.format('%I.%I', tn.nspname, t.typname) - END - FROM pg_catalog.pg_type t - JOIN pg_catalog.pg_namespace tn ON tn.oid = t.typnamespace - LEFT JOIN pg_catalog.pg_type et - ON et.oid = t.typelem AND t.typcategory = 'A' - LEFT JOIN pg_catalog.pg_namespace en ON en.oid = et.typnamespace -), eql_roots(classid, objid, objsubid) AS ( SELECT 'pg_catalog.pg_namespace'::regclass, n.oid, 0 FROM pg_catalog.pg_namespace n @@ -269,55 +236,39 @@ external_dependants AS ( AND n.nspname IN ('eql_v3', 'eql_v3_internal') ) ) - -- Only exact operator identities parsed from the pinned bundle are owned by - -- EQL. A customer may legally give an EQL function a different operator - -- name, so implementation namespace alone is not an ownership marker. + -- EQL installs its operators in public and implements them with functions in + -- a disposable EQL schema. Classify the installed catalog, not the incoming + -- bundle: an upgrade may legitimately remove an old operator. AND NOT ( d.classid = 'pg_catalog.pg_operator'::regclass AND EXISTS ( SELECT 1 FROM pg_catalog.pg_operator operator - -- Prefix operators carry oprleft = 0, which matches no pg_type row; the - -- parser writes that operand as \`none\`. - LEFT JOIN type_identity left_type ON left_type.oid = operator.oprleft - LEFT JOIN type_identity right_type ON right_type.oid = operator.oprright + JOIN pg_catalog.pg_namespace operator_namespace + ON operator_namespace.oid = operator.oprnamespace + JOIN pg_catalog.pg_proc implementation + ON implementation.oid = operator.oprcode + JOIN pg_catalog.pg_namespace implementation_namespace + ON implementation_namespace.oid = implementation.pronamespace WHERE operator.oid = d.objid - AND pg_catalog.lower(operator.oprname) || ' (' || - COALESCE(left_type.identity, 'none') || ', ' || - COALESCE(right_type.identity, 'none') || ')' - = ANY($1::text[]) - -- Compared raw, NOT through lower(). Postgres operator names are drawn - -- from +-*/<>=~!@#%^&|? and cannot contain a letter, so lower() is a - -- no-op on the value -- but it is not a no-op on the plan: it makes - -- pg_operator_oprname_l_r_n_index (oprname, oprleft, oprright, - -- oprnamespace) unusable and turns this into a sequential scan. The - -- planner evaluates this count as a filter over the whole of - -- pg_operator before operator.oid = d.objid narrows anything, so - -- with lower() the guard is a self-join of pg_operator against itself: - -- ~3,900 rows squared, ~15M comparisons and ~9GB of buffer traffic on - -- a database with EQL installed, which is most of this query's cost. - -- Do not reintroduce it. - AND ( - SELECT pg_catalog.count(*) - FROM pg_catalog.pg_operator candidate - WHERE candidate.oprname = operator.oprname - AND candidate.oprleft = operator.oprleft - AND candidate.oprright = operator.oprright - ) = 1 + AND operator_namespace.nspname = 'public' + AND implementation_namespace.nspname IN ('eql_v3', 'eql_v3_internal') ) ) - -- Likewise, these public-data-domain <-> query-domain casts are declarations - -- in the bundle, not application objects. + -- Likewise, EQL's casts are implemented by functions in the disposable EQL + -- schemas. Classify the installed catalog rather than the incoming bundle: + -- an upgrade may legitimately remove an old cast. AND NOT ( d.classid = 'pg_catalog.pg_cast'::regclass AND EXISTS ( SELECT 1 FROM pg_catalog.pg_cast cast_row - JOIN type_identity source_type ON source_type.oid = cast_row.castsource - JOIN type_identity target_type ON target_type.oid = cast_row.casttarget + JOIN pg_catalog.pg_proc implementation + ON implementation.oid = cast_row.castfunc + JOIN pg_catalog.pg_namespace implementation_namespace + ON implementation_namespace.oid = implementation.pronamespace WHERE cast_row.oid = d.objid - AND source_type.identity || ' AS ' || target_type.identity - = ANY($2::text[]) + AND implementation_namespace.nspname IN ('eql_v3', 'eql_v3_internal') ) ) -- pg_amop/pg_amproc have no namespace of their own; their owning family does. @@ -431,6 +382,40 @@ SELECT DISTINCT pg_catalog.obj_description(index_class.oid, 'pg_class') ) END AS comment_sql + , pg_catalog.pg_get_userbyid(index_class.relowner) AS owner + , CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL + THEN ARRAY( + SELECT attribute.attstattarget + FROM pg_catalog.pg_attribute attribute + WHERE attribute.attrelid = index_class.oid + AND attribute.attnum > 0 + AND NOT attribute.attisdropped + ORDER BY attribute.attnum + ) + END AS statistics_targets + , CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL + THEN ARRAY( + SELECT pg_catalog.format( + 'ALTER INDEX %I.%I ALTER COLUMN %s SET STATISTICS %s', + index_namespace.nspname, + index_class.relname, + attribute.attnum, + attribute.attstattarget + ) + FROM pg_catalog.pg_attribute attribute + WHERE attribute.attrelid = index_class.oid + AND attribute.attnum > 0 + AND NOT attribute.attisdropped + AND attribute.attstattarget >= 0 + ORDER BY attribute.attnum + ) + END AS statistics_sql FROM external_dependants e LEFT JOIN pg_catalog.pg_class index_class ON e.classid = 'pg_catalog.pg_class'::regclass AND index_class.oid = e.objid @@ -451,6 +436,15 @@ SELECT pg_catalog.format('%I.%I', n.nspname, c.relname) AS identity, i.indisclustered AS clustered, i.indisreplident AS replica_identity, pg_catalog.obj_description(c.oid, 'pg_class') AS comment, + pg_catalog.pg_get_userbyid(c.relowner) AS owner, + ARRAY( + SELECT attribute.attstattarget + FROM pg_catalog.pg_attribute attribute + WHERE attribute.attrelid = c.oid + AND attribute.attnum > 0 + AND NOT attribute.attisdropped + ORDER BY attribute.attnum + ) AS statistics_targets, pg_catalog.pg_get_indexdef(i.indexrelid) AS definition FROM pg_catalog.pg_index i JOIN pg_catalog.pg_class c ON c.oid = i.indexrelid @@ -487,7 +481,7 @@ async function acquireLifecycleLock( return } if (Date.now() >= deadline) { - throw new Error( + throw new EqlLifecycleLockTimeoutError( `Another EQL lifecycle operation is in progress on this database — it has held the installer's advisory lock for more than ${Math.round(waitMs / 1000)} seconds. Nothing was changed. Wait for the other \`stash eql install\`/\`eql upgrade\` to finish and re-run. If no other command is running, an earlier one may have died holding the lock: find its session in \`pg_stat_activity\` and close it, then retry.`, ) } @@ -499,13 +493,8 @@ async function acquireLifecycleLock( async function inspectReinstallDependencies( client: pg.ClientBase, - bundleOperators: string[], - bundleCasts: string[], ): Promise { - const result = await client.query(LIFECYCLE_DEPENDENCIES_SQL, [ - bundleOperators, - bundleCasts, - ]) + const result = await client.query(LIFECYCLE_DEPENDENCIES_SQL) const unsafe: string[] = [] const indexes: ReinstallIndex[] = [] for (const row of result.rows as DependencyRow[]) { @@ -528,7 +517,15 @@ async function inspectReinstallDependencies( (row.replica_identity && typeof row.replica_identity_sql !== 'string') || (row.comment !== null && typeof row.comment !== 'string') || (row.comment_sql !== null && typeof row.comment_sql !== 'string') || - (typeof row.comment === 'string' && typeof row.comment_sql !== 'string') + (typeof row.comment === 'string' && + typeof row.comment_sql !== 'string') || + typeof row.owner !== 'string' || + !Array.isArray(row.statistics_targets) || + !row.statistics_targets.every( + (target) => target === null || typeof target === 'number', + ) || + !Array.isArray(row.statistics_sql) || + !row.statistics_sql.every((sql) => typeof sql === 'string') ) { unsafe.push( String(row.identity ?? 'index with incomplete catalog metadata'), @@ -547,6 +544,9 @@ async function inspectReinstallDependencies( replicaIdentitySql: row.replica_identity_sql, comment: row.comment, commentSql: row.comment_sql, + owner: row.owner, + statisticsTargets: row.statistics_targets, + statisticsSql: row.statistics_sql, }) } if (unsafe.length > 0) { @@ -588,6 +588,9 @@ async function rebuildIndexes( if (index.replicaIdentitySql !== null) await client.query(index.replicaIdentitySql) if (index.commentSql !== null) await client.query(index.commentSql) + for (const statisticsSql of index.statisticsSql) { + await client.query(statisticsSql) + } } const tableIdentities = new Set(indexes.map((index) => index.tableIdentity)) for (const tableIdentity of tableIdentities) { @@ -623,7 +626,14 @@ async function rebuildIndexes( (row.ready === true || expected.ready === false) && row.clustered === expected.clustered && row.replica_identity === expected.replicaIdentity && - (row.comment ?? null) === expected.comment + (row.comment ?? null) === expected.comment && + row.owner === expected.owner && + Array.isArray(row.statistics_targets) && + row.statistics_targets.length === expected.statisticsTargets.length && + row.statistics_targets.every( + (target: unknown, position: number) => + target === expected.statisticsTargets[position], + ) ) }) .map((row) => String(row.identity)), @@ -649,8 +659,6 @@ async function rebuildIndexes( export async function restoreDerivedSearchIndexesAroundEqlReplacement({ databaseUrl, bundledSql, - bundleOperators, - bundleCasts, }: RestoreAroundEqlReplacementOptions): Promise { const client = createPgClient(databaseUrl) try { @@ -672,11 +680,7 @@ export async function restoreDerivedSearchIndexesAroundEqlReplacement({ try { await acquireLifecycleLock(client) await client.query('SET LOCAL jit = off') - const indexes = await inspectReinstallDependencies( - client, - bundleOperators, - bundleCasts, - ) + const indexes = await inspectReinstallDependencies(client) await client.query(bundledSql) const summary = await rebuildIndexes(client, indexes) await client.query('COMMIT') diff --git a/packages/cli/src/installer/index.ts b/packages/cli/src/installer/index.ts index 9a8ce0294..8113bde2b 100644 --- a/packages/cli/src/installer/index.ts +++ b/packages/cli/src/installer/index.ts @@ -1,5 +1,6 @@ import { createPgClient, TlsVerificationError } from '@/db/client.js' import { + EqlLifecycleLockTimeoutError, EqlReinstallConnectionError, EqlReinstallRefusalError, restoreDerivedSearchIndexesAroundEqlReplacement, @@ -123,6 +124,12 @@ export class EQLInstaller { : "SELECT to_regnamespace('eql_v3') IS NOT NULL AND to_regnamespace('eql_v3_internal') IS NOT NULL AS installed", ) return result.rows[0]?.installed === true + } catch (error) { + if (error instanceof TlsVerificationError) throw error + const detail = error instanceof Error ? error.message : String(error) + throw new Error(`Failed to connect to database: ${detail}`, { + cause: error, + }) } finally { await client.end() } @@ -136,17 +143,28 @@ export class EQLInstaller { const client = createPgClient(this.databaseUrl) try { await client.connect() - const result = await client.query<{ version: string }>( - `SELECT eql_v${generation}.version() AS version`, + const schemaName = `eql_v${generation}` + const schema = await client.query<{ installed: boolean }>( + 'SELECT to_regnamespace($1) IS NOT NULL AS installed', + [schemaName], ) - return result.rows[0]?.version ?? null + if (schema.rows[0]?.installed !== true) return null + try { + const result = await client.query<{ version: string }>( + `SELECT ${schemaName}.version() AS version`, + ) + return result.rows[0]?.version + ? String(result.rows[0].version) + : 'unknown' + } catch { + return 'unknown' + } } catch (error) { - const code = - typeof error === 'object' && error !== null && 'code' in error - ? String((error as { code?: unknown }).code) - : undefined - if (code === '42883' || code === '3F000') return null - throw error + if (error instanceof TlsVerificationError) throw error + const detail = error instanceof Error ? error.message : String(error) + throw new Error(`Failed to connect to database: ${detail}`, { + cause: error, + }) } finally { await client.end() } @@ -178,12 +196,11 @@ export class EQLInstaller { await restoreDerivedSearchIndexesAroundEqlReplacement({ databaseUrl: this.databaseUrl, bundledSql: bundle.sql, - bundleOperators: bundle.expectedSurface.operators, - bundleCasts: bundle.expectedSurface.casts, }) } catch (error) { if ( error instanceof TlsVerificationError || + error instanceof EqlLifecycleLockTimeoutError || error instanceof EqlReinstallConnectionError || error instanceof EqlReinstallRefusalError ) { diff --git a/packages/cli/src/installer/installation-state.ts b/packages/cli/src/installer/installation-state.ts index ea59d71e1..c4a0d9f53 100644 --- a/packages/cli/src/installer/installation-state.ts +++ b/packages/cli/src/installer/installation-state.ts @@ -2,6 +2,7 @@ import type pg from 'pg' import { createPgClient, TlsVerificationError } from '@/db/client.js' import { SUPPORTED_PGCRYPTO_SCHEMAS } from './eql-bundle.js' import { EQL_V3_INTERNAL_SCHEMA_NAME, EQL_V3_SCHEMA_NAME } from './grants.js' +import type { OreSurfaceState } from './ore.js' import { assessEqlSurface, type OreStateReading, @@ -23,12 +24,7 @@ export type AssessedOreState = } | { status: 'observed' - state: - | 'indexable' - | 'fallback' - | 'incoherent-mixed' - | 'incoherent-poisoned' - | 'incoherent-unpoisoned' + state: OreSurfaceState opclassPresent: boolean poisonedDomains: number expectedPoisoned: number @@ -75,8 +71,6 @@ const CAPABILITIES_SQL = ` CASE WHEN EXISTS (SELECT 1 FROM pg_namespace WHERE nspname = 'public') THEN has_schema_privilege(current_user, 'public', 'CREATE') ELSE false END AS has_public_create, EXISTS (SELECT 1 FROM pg_extension WHERE extname = 'pgcrypto') AS pgcrypto_installed, (SELECT n.nspname FROM pg_extension e JOIN pg_namespace n ON n.oid = e.extnamespace WHERE e.extname = 'pgcrypto') AS pgcrypto_schema, - EXISTS (SELECT 1 FROM information_schema.schemata WHERE schema_name = '${EQL_V3_SCHEMA_NAME}') AS eql_v3_present, - EXISTS (SELECT 1 FROM information_schema.schemata WHERE schema_name = '${EQL_V3_INTERNAL_SCHEMA_NAME}') AS eql_v3_internal_present, (SELECT pg_has_role(current_user, n.nspowner, 'MEMBER') FROM pg_namespace n WHERE n.nspname = '${EQL_V3_SCHEMA_NAME}') AS can_drop_eql_v3, (SELECT pg_has_role(current_user, n.nspowner, 'MEMBER') FROM pg_namespace n WHERE n.nspname = '${EQL_V3_INTERNAL_SCHEMA_NAME}') AS can_drop_eql_v3_internal ` @@ -124,7 +118,7 @@ export async function assessEqlInstallation(options: { let verification = null let unavailableOre: AssessedOreState | null = null - if (v3Present && options.depth === 'exhaustive') { + if (options.depth === 'exhaustive') { verification = await assessEqlSurface(client, 'exhaustive') } else if (v3Present && options.includeOre === true) { await client.query('SAVEPOINT eql_ore_assessment') @@ -175,6 +169,10 @@ export async function assessEqlInstallation(options: { preflight: buildPreflight( capabilityRow, await probeOperatorClassCreate(client), + { + eqlV3SchemaPresent: row?.eql_v3_present === true, + eqlV3InternalSchemaPresent: row?.eql_v3_internal_present === true, + }, ), } : { status: 'not-requested' as const } @@ -190,6 +188,10 @@ export async function assessEqlInstallation(options: { function buildPreflight( row: Record, canCreateOperatorClass: boolean | null, + presence: { + eqlV3SchemaPresent: boolean + eqlV3InternalSchemaPresent: boolean + }, ): PreflightResult { const asBoolOrNull = (value: unknown) => typeof value === 'boolean' ? value : null @@ -235,8 +237,8 @@ function buildPreflight( hasPublicCreate: row.has_public_create === true, pgcryptoInstalled, pgcryptoSchema, - eqlV3SchemaPresent: row.eql_v3_present === true, - eqlV3InternalSchemaPresent: row.eql_v3_internal_present === true, + eqlV3SchemaPresent: presence.eqlV3SchemaPresent, + eqlV3InternalSchemaPresent: presence.eqlV3InternalSchemaPresent, canDropEqlV3Schema, canDropEqlV3InternalSchema, canCreateOperatorClass, diff --git a/packages/cli/src/installer/verify.ts b/packages/cli/src/installer/verify.ts index a8dc8d4c5..37ab7a784 100644 --- a/packages/cli/src/installer/verify.ts +++ b/packages/cli/src/installer/verify.ts @@ -547,8 +547,12 @@ async function readInstalledEqlVersion( ? (error as { code?: string }).code : undefined if (insideCallerTransaction) { - await client.query('ROLLBACK TO SAVEPOINT installed_eql_version_probe') - await client.query('RELEASE SAVEPOINT installed_eql_version_probe') + await client + .query('ROLLBACK TO SAVEPOINT installed_eql_version_probe') + .catch(() => {}) + await client + .query('RELEASE SAVEPOINT installed_eql_version_probe') + .catch(() => {}) } if (code === '42883') return null const detail = error instanceof Error ? error.message : String(error) @@ -584,74 +588,93 @@ export async function readInstalledSurface( // search_path. SET LOCAL dies with the transaction, so the caller's // session is untouched. The qualified version() probe is protected by a // savepoint because a missing function must not abort the caller's snapshot. - if (options.manageTransaction !== false) await client.query('BEGIN READ ONLY') - await client.query(`SET LOCAL search_path = ''`) - const schemas = await client.query<{ - eql_v3_present: boolean - eql_v3_internal_present: boolean - pgcrypto_installed: boolean - pgcrypto_schema: string | null - }>(SCHEMAS_SQL) - const types = await client.query<{ name: string }>(TYPES_SQL, [ - [...expected.domains, ...expected.types], - ]) - const functions = await client.query<{ name: string; signature: string }>( - FUNCTION_SIGNATURES_SQL, - [[...expected.functions.keys()]], - ) - const operators = await client.query<{ - name: string - leftarg: string - rightarg: string - }>(OPERATORS_SQL) - const casts = await client.query<{ source: string; target: string }>( - CASTS_SQL, - ) - const ore = await client.query<{ - ore_opclass_present: boolean - poisoned_domains: number - }>(ORE_STATE_SQL, [expected.oreDomains]) - const eqlV3SchemaPresent = schemas.rows[0]?.eql_v3_present === true - const installedVersion = eqlV3SchemaPresent - ? await readInstalledEqlVersion(client, true) - : null - // Ends the SET LOCAL scope. On a mid-transaction error the caller's - // client.end() discards the aborted transaction with the connection. - if (options.manageTransaction !== false) await client.query('COMMIT') - - const functionSignatures = new Map>() - for (const row of functions.rows) { - const name = row.name.toLowerCase() - const existing = functionSignatures.get(name) ?? new Set() - existing.add(row.signature.toLowerCase()) - functionSignatures.set(name, existing) - } + const manageTransaction = options.manageTransaction !== false + if (manageTransaction) await client.query('BEGIN READ ONLY') + else await client.query('SAVEPOINT installed_eql_surface_read') + try { + await client.query(`SET LOCAL search_path = ''`) + const schemas = await client.query<{ + eql_v3_present: boolean + eql_v3_internal_present: boolean + pgcrypto_installed: boolean + pgcrypto_schema: string | null + }>(SCHEMAS_SQL) + const types = await client.query<{ name: string }>(TYPES_SQL, [ + [...expected.domains, ...expected.types], + ]) + const functions = await client.query<{ name: string; signature: string }>( + FUNCTION_SIGNATURES_SQL, + [[...expected.functions.keys()]], + ) + const operators = await client.query<{ + name: string + leftarg: string + rightarg: string + }>(OPERATORS_SQL) + const casts = await client.query<{ source: string; target: string }>( + CASTS_SQL, + ) + const ore = await client.query<{ + ore_opclass_present: boolean + poisoned_domains: number + }>(ORE_STATE_SQL, [expected.oreDomains]) + const eqlV3SchemaPresent = schemas.rows[0]?.eql_v3_present === true + const installedVersion = eqlV3SchemaPresent + ? await readInstalledEqlVersion(client, true) + : null + const functionSignatures = new Map>() + for (const row of functions.rows) { + const name = row.name.toLowerCase() + const existing = functionSignatures.get(name) ?? new Set() + existing.add(row.signature.toLowerCase()) + functionSignatures.set(name, existing) + } - return { - eqlV3SchemaPresent, - eqlV3InternalSchemaPresent: - schemas.rows[0]?.eql_v3_internal_present === true, - pgcryptoInstalled: schemas.rows[0]?.pgcrypto_installed === true, - pgcryptoSchema: - typeof schemas.rows[0]?.pgcrypto_schema === 'string' - ? schemas.rows[0].pgcrypto_schema - : null, - installedVersion, - presentTypes: new Set(types.rows.map((row) => row.name.toLowerCase())), - functionSignatures, - presentOperators: new Set( - operators.rows.map( - (row) => - `${row.name.toLowerCase()} (${row.leftarg.toLowerCase()}, ${row.rightarg.toLowerCase()})`, + const installedSurface: InstalledSurface = { + eqlV3SchemaPresent, + eqlV3InternalSchemaPresent: + schemas.rows[0]?.eql_v3_internal_present === true, + pgcryptoInstalled: schemas.rows[0]?.pgcrypto_installed === true, + pgcryptoSchema: + typeof schemas.rows[0]?.pgcrypto_schema === 'string' + ? schemas.rows[0].pgcrypto_schema + : null, + installedVersion, + presentTypes: new Set(types.rows.map((row) => row.name.toLowerCase())), + functionSignatures, + presentOperators: new Set( + operators.rows.map( + (row) => + `${row.name.toLowerCase()} (${row.leftarg.toLowerCase()}, ${row.rightarg.toLowerCase()})`, + ), ), - ), - presentCasts: new Set( - casts.rows.map( - (row) => `${row.source.toLowerCase()} AS ${row.target.toLowerCase()}`, + presentCasts: new Set( + casts.rows.map( + (row) => `${row.source.toLowerCase()} AS ${row.target.toLowerCase()}`, + ), ), - ), - oreOpclassPresent: ore.rows[0]?.ore_opclass_present === true, - poisonedDomains: ore.rows[0]?.poisoned_domains ?? 0, + oreOpclassPresent: ore.rows[0]?.ore_opclass_present === true, + poisonedDomains: ore.rows[0]?.poisoned_domains ?? 0, + } + // Keep the savepoint alive until every fallible conversion above has + // completed. Otherwise the catch path attempts to clean up a savepoint + // already released and can replace the original error with 25P01. + if (manageTransaction) await client.query('COMMIT') + else { + await client.query('ROLLBACK TO SAVEPOINT installed_eql_surface_read') + await client.query('RELEASE SAVEPOINT installed_eql_surface_read') + } + return installedSurface + } catch (error) { + if (!manageTransaction) { + await client + .query('ROLLBACK TO SAVEPOINT installed_eql_surface_read') + .catch(() => {}) + await client + .query('RELEASE SAVEPOINT installed_eql_surface_read') + .catch(() => {}) + } + throw error } } @@ -953,7 +976,7 @@ export async function verifyEqlSurface( } /** - * The ORE half of an install as {@link readOreState} could read it. + * The ORE half of an install returned by a summary surface assessment. * * `comparable: false` means the installed EQL is not the pinned one, so there * is no honest ORE answer to give — not that anything is wrong. Callers must @@ -999,31 +1022,6 @@ export async function assessEqlSurface( return { depth, report: diffSurface(expected, installed) } } -/** - * Read just the ORE half of an install — the two catalogue values and the - * state they classify to (#891). - * - * `eql status` wants the ORE answer and nothing else. Routing it through - * {@link verifyEqlSurface} would work but would read the whole 3,000-operator - * surface to render one row. - * - * It still needs {@link diffSurface}'s version gate, though, because the ORE - * state is NOT a pure catalogue fact: `expectedPoisoned` is the pinned - * bundle's ORE-domain count, and {@link ORE_STATE_SQL} counts poisoned domains - * only among that same pinned list. So a healthy fallback install of a - * DIFFERENT EQL — the ordinary "CLI upgraded, database not yet" case — poisons - * ITS domains, of which the pinned list sees only some, and - * {@link classifyOreState} reads the shortfall as `incoherent-unpoisoned` - * damage. Reporting a version skew as `comparable: false` is what stops - * `eql status` telling that operator to reinstall `--force` over nothing. - */ -export async function readOreState( - client: pg.ClientBase, -): Promise { - const expected = bundledExpectedSurface() - return readOreStateAgainst(client, expected) -} - async function readOreStateAgainst( client: pg.ClientBase, expected: ExpectedSurface, diff --git a/packages/cli/vitest.config.ts b/packages/cli/vitest.config.ts index c65ccc12f..eae72f539 100644 --- a/packages/cli/vitest.config.ts +++ b/packages/cli/vitest.config.ts @@ -5,13 +5,13 @@ export default defineConfig({ test: { globals: true, exclude: ['**/node_modules/**', '**/dist/**', 'tests/e2e/**'], - // Two projects so ONLY the live suites are serialised. Four of them gate + // Two projects so ONLY the live suites are serialised. Several of them gate // on STASH_TEST_DATABASE_URL and share one database and one // eql_v3/eql_v3_internal pair — and verify.live's beforeAll installs the // full bundle, which opens with `DROP SCHEMA … CASCADE`, destroying the // schemas (and their ACLs/OIDs) under a concurrently running - // guarded-grants.live. Run in parallel forks they race; run serially each - // suite sees the database state its comments already assume. The unit + // guarded-grants.live. Run in parallel forks they race; one fork lets each + // suite see the database state its comments already assume. The unit // project keeps default file parallelism — serialising all ~1300 tests // for the sake of four files is the `packages/migrate` fix at the wrong // scale. @@ -33,7 +33,7 @@ export default defineConfig({ test: { name: 'live', include: ['src/**/*.live.test.ts'], - fileParallelism: false, + poolOptions: { forks: { singleFork: true } }, }, }, ], diff --git a/packages/stack-supabase/__tests__/wasm-entry-edge-safety.test.ts b/packages/stack-supabase/__tests__/wasm-entry-edge-safety.test.ts index 7e45b3301..bf193ed41 100644 --- a/packages/stack-supabase/__tests__/wasm-entry-edge-safety.test.ts +++ b/packages/stack-supabase/__tests__/wasm-entry-edge-safety.test.ts @@ -75,11 +75,6 @@ function reachableBareSpecifiers(entry: string): string[] { const bare = new Set() const walk = (file: string): void => { if (seen.has(file)) return - if (!existsSync(file)) { - throw new Error( - `Emitted module graph references missing relative file: ${file}`, - ) - } seen.add(file) for (const specifier of specifiers(file)) { if (specifier.startsWith('.')) walk(resolve(dirname(file), specifier)) diff --git a/scripts/__tests__/cli-live-postgres-ci.test.mjs b/scripts/__tests__/cli-live-postgres-ci.test.mjs new file mode 100644 index 000000000..644cf0355 --- /dev/null +++ b/scripts/__tests__/cli-live-postgres-ci.test.mjs @@ -0,0 +1,51 @@ +import { join } from 'node:path' +import { describe, expect, it } from 'vitest' +import { readJsonc } from './lib/read-jsonc.mjs' +import { REPO_ROOT } from './lib/repo-root.mjs' +import { readWorkflow } from './lib/workflows.mjs' + +describe('CLI live-Postgres CI contract', () => { + it('forwards the live database URL through Turbo test tasks', () => { + const turbo = readJsonc(join(REPO_ROOT, 'turbo.json')) + + expect(turbo.tasks.test.env).toContain('STASH_TEST_DATABASE_URL') + }) + + it('supplies the live database URL from the test workflow', () => { + const workflow = readWorkflow('.github/workflows/tests.yml') + const runTests = workflow.jobs['run-tests'].steps.find( + (step) => step.name === 'Run tests', + ) + + expect(runTests.env.STASH_TEST_DATABASE_URL).toMatch( + /^postgres:\/\/[^/]+\/cipherstash$/, + ) + }) + + it('runs the live reinstall suite against both pre-17 and current Postgres catalogs', () => { + const workflow = readWorkflow('.github/workflows/tests.yml') + const runTests = workflow.jobs['run-tests'] + const versions = runTests.strategy.matrix['postgres-version'] + + expect(versions).toEqual([16, 17]) + expect(runTests.strategy.matrix['node-version']).toEqual([22, 24]) + expect(runTests.strategy.matrix.exclude).toEqual([ + { 'node-version': 22, 'postgres-version': 17 }, + { 'node-version': 24, 'postgres-version': 16 }, + ]) + expect(runTests.services.postgres.image).toContain( + '$' + '{{ matrix.postgres-version }}', + ) + }) + + it('serializes live suites that share the EQL schemas', async () => { + const config = await import( + '../../packages/cli/vitest.config.ts?cli-live-ci-contract' + ) + const live = config.default.test.projects.find( + (project) => project.test.name === 'live', + ) + + expect(live.test.poolOptions.forks.singleFork).toBe(true) + }) +}) diff --git a/scripts/__tests__/lint-no-eql-registry-pins.test.mjs b/scripts/__tests__/lint-no-eql-registry-pins.test.mjs index 04796b8bb..9b581fc01 100644 --- a/scripts/__tests__/lint-no-eql-registry-pins.test.mjs +++ b/scripts/__tests__/lint-no-eql-registry-pins.test.mjs @@ -79,14 +79,12 @@ const cargoForms = (body) => cargoDeclarations('Cargo.toml', body).map((d) => d.form) describe('the tree it actually guards', () => { - it('passes: every EQL dependency resolves in-tree, with nothing exempt', () => { + it('passes with only the immutable CLI upgrade baseline exempt', () => { const { exitCode, output } = run() expect(output).toContain('resolves in-tree') - // No `(N exempt: …)` suffix. The exemption list is empty as of CIP-3744 - // and the success line reports what it excused, so this is the assertion - // that the tree needs no standing permission at all — not merely that the - // one it had is still described accurately. - expect(output).not.toContain('exempt') + expect(output).toContain( + '(1 exempt: packages/cli/package.json :: @cipherstash/eql-upgrade-baseline)', + ) expect(exitCode).toBe(0) }) diff --git a/scripts/lint-no-eql-registry-pins.mjs b/scripts/lint-no-eql-registry-pins.mjs index 62408b632..0bdc21174 100644 --- a/scripts/lint-no-eql-registry-pins.mjs +++ b/scripts/lint-no-eql-registry-pins.mjs @@ -643,9 +643,7 @@ export function lint({ } = {}) { const { declarations, sources: read } = scanTree(root) const ids = declarations.map(declarationId) - const registryPinned = declarations - .filter((d) => !d.inTree) - .map(exemptionId) + const registryPinned = declarations.filter((d) => !d.inTree).map(exemptionId) return { declarations, ids, diff --git a/skills/stash-cli/SKILL.md b/skills/stash-cli/SKILL.md index c40726cdd..d9a3a0567 100644 --- a/skills/stash-cli/SKILL.md +++ b/skills/stash-cli/SKILL.md @@ -419,7 +419,9 @@ Run it whenever query-time behaviour looks inconsistent with a "successful" inst Generates an **EQL v3 install migration**, instead of running SQL directly against the database (`eql install`). Migration-first is the preferred path: the install lands in your migration history and ships to every environment through the same migrate step as the rest of your schema. On Supabase it is the *only* durable path — `supabase db reset` replays the migrations directory, so a direct install is wiped by the next reset. v3 only — there is no `--eql-version` here. -**The re-run protections do not travel with the file.** The lifecycle lock, index capture, refusal on unsupported dependants, and rebuild all live in the `eql install`/`eql upgrade` code path, not in the bundle SQL. A generated migration is the raw bundle, so a migration runner applying it does the `DROP SCHEMA ... CASCADE` unprotected. That is fine on a first install. Re-applying it over a database that already carries EQL functional indexes drops them with no rebuild — reach for `eql upgrade` there, or recreate the indexes in the same migration. +**Generated migrations contain the raw bundle, not the reinstall protocol.** Use +one for a first install. For replacement, use `eql upgrade` or recreate every +dependent object in the same migration; see `eql upgrade` below. ```bash stash eql migration --drizzle # Drizzle custom migration in drizzle/ @@ -497,14 +499,13 @@ An applied migration carrying a statement the sweep would have skipped anyway #### `eql upgrade` -The install SQL is safe to re-run: encrypted columns and rows live outside the -disposable EQL schemas. `upgrade` serializes the lifecycle with a database -advisory lock, captures functional-index definitions, replaces the schemas, -then rebuilds, analyzes, and verifies those indexes in the same transaction. Unsupported external -dependencies (including policies and views) make it refuse before mutation. A -rebuild failure rolls back the schema replacement and restores the prior indexes; -never describe a failed reconstruction as a successful upgrade. `upgrade` is -v3-only and accepts `--supabase`, `--dry-run`, and `--database-url`. +Encrypted columns and rows live outside the disposable EQL schemas. `upgrade` +captures dependent functional indexes, replaces the schemas, then restores and +verifies the indexes in one transaction. Unsupported dependencies are refused +before mutation; restoration failure rolls back the replacement. Completion +means every definition and restorable catalog property matches, with no +validity/readiness regression. `upgrade` is v3-only and accepts `--supabase`, +`--dry-run`, and `--database-url`. Run it in a schema-migration maintenance window. The advisory lock serializes other `stash` lifecycle commands, not arbitrary DDL from unrelated sessions; diff --git a/turbo.json b/turbo.json index a2448e541..ac38cf670 100644 --- a/turbo.json +++ b/turbo.json @@ -18,6 +18,7 @@ "test": { "dependsOn": ["^build"], "inputs": ["$TURBO_DEFAULT$", ".env*"], + "env": ["STASH_TEST_DATABASE_URL"], "cache": false }, // `packages/bench`'s "build" is `tsc --noEmit` — a typecheck, not a bundle —