diff --git a/.changeset/observer-row-container-sharing.md b/.changeset/observer-row-container-sharing.md new file mode 100644 index 0000000000..13480762a8 --- /dev/null +++ b/.changeset/observer-row-container-sharing.md @@ -0,0 +1,5 @@ +--- +'@tanstack/db': patch +--- + +Preserve live-query observer data arrays and lazy state Maps across status-only and health-only snapshot updates. Keep value and ordering changes visible while retaining point-in-time snapshots. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 6e5ba7182e..00fdfe8e35 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -1502,6 +1502,20 @@ The separate `live-query-observer-snapshot-oracle.property.test.ts` owner checks unchanged getter identity with and without a completed DbClient query. It runs ordered/single-result and attached/detached histories through ordinary and server getters. It does not model provisional hydration replacement or host scheduling. +Its `live-query-observer.row-container-sharing` lane also checks the maintainer's +2026-11-01 metadata-sharing contract: loading-to-ready and health-only changes +keep data/Map references, while value writes replace them and retained snapshots +keep their old rows. Fixed and random histories cross attached/detached, +granular/wholesale, zero through three rows, and data-first/state-first reads. +Each begins with readiness before health reports; only writes change the +independent key/version model. The original fresh-container implementation and +same-cardinality reuse both fail the shared comparison. Existing hydration and +observer/layout witnesses retain their scopes. This core-only lane adds no new +framework scheduling promise. The independent Angular PR projects observer +row-state identity before copying its Map; combined integration evidence is +recorded separately from either branch's standalone validation. +The [row-container evidence](oracle-reviews/2026-11-01-observer-row-containers.md) +records RED/GREEN, declared reach, mutant rejection, size, and limits. Focused snapshot checks protect lazy state, one admission walk for an unchanged revision, explicit preload after a loading capture, JSON-compatible non-streaming transport, and pending-stream ownership. The observer history diff --git a/docs/contributing/oracle-reviews/2026-11-01-observer-row-containers.md b/docs/contributing/oracle-reviews/2026-11-01-observer-row-containers.md new file mode 100644 index 0000000000..c93e5c2aa0 --- /dev/null +++ b/docs/contributing/oracle-reviews/2026-11-01-observer-row-containers.md @@ -0,0 +1,83 @@ +# Observer row-container sharing evidence + +This independent PR starts from upstream main at +`15d2d2d328db6f5a4ea0f1a48528c6c5137a1716`. It contains no Angular source or +framework-version change. Observer source SHA-256: +`abb037706e51c05174cb61170afc314f4ac804f611d8da4bc6b7edc6bc20e120`. +The primary owner is `live-query-observer-snapshot-oracle.property.test.ts`. + +- **Law:** metadata-only status/health changes replace the full snapshot while + retaining its data array and lazy state Map. Writes still replace containers, + and retained snapshots keep their capture-time rows. The maintainer approved + this optimization on 2026-11-01; LiveQuerySnapshot documents the new guarantee. +- **Reach:** `live-query-observer.row-container-sharing` executes 12 shared + examples and 40 generated histories per fixed/random campaign. Both campaigns + reach status-only and health-only transitions in 52 histories each. The fixed + campaign reaches attached/detached 26/26, granular/wholesale 31/21, empty rows + in 11 histories, and same-key value writes in 40. Read order varies between + data-first and state-first. Every history becomes ready before health reports. +- **RED:** from `packages/db`, run + `../../node_modules/.bin/vitest --run --coverage.enabled=false tests/live-query-observer-snapshot-oracle.property.test.ts -t 'metadata snapshots'` + with unchanged production. Both campaigns reject a fresh empty array at the + loading-to-ready transition; fixed seed 1101, shrink path `0:0:0:0`. + A first fixture attempted health reporting before readiness and failed setup; + the grammar was corrected with a legal readiness prefix before recording RED. +- **GREEN:** the observer ordinary/history/hydration/retained-snapshot suites + pass 100 tests. The unchanged Angular adapter also passes 89 tests with one + existing TODO against this observer. ESLint, oracle lint, focused TypeScript, + private-member mangle validation, and package build pass on the independent + branch with Node 24.8.0 and pnpm 11.1.0. The documentation snippet check reports + no new failures across 804 examples. Full-workspace pnpm test is not claimed. +- **Mutant:** reusing containers solely by row count fails on two writes to key + `a`: the second value stays stale despite unchanged membership (seed 1101, + path `0:0:1:2`). The shared model's row assertion rejects it. The final code is + restored and the same oracle passes. Earlier point-in-time calibration also + rejects a getter that reads current Collection rows instead of captured rows. +- **Limits:** the new lane covers synchronous direct Collections, zero through + three string-keyed rows, both observer modes and attach states. Existing + ordered/findOne histories and hydration tests protect neighboring behavior; + order-only changes retain the existing observer layout witnesses. No new + framework, browser-rendering, SSR-scheduler, or external-provider claim. + +## Implementation and release impact + +One bounded row cache contains captured entries, their array, and a lazy Map. +Full snapshots capture that cache rather than forwarding through old snapshots, +so metadata updates neither rebuild row containers nor retain an unbounded chain +of full snapshots. Unchanged entry recaptures preserve identity. When a row +revision is unchanged, entry comparison still notices layout-only changes and +supports the existing revisionless-input path. Value-only writes cannot hide +behind layout identity. The private-member mangle cache includes the new field. + +The size comparison reports db +28 gzip bytes and -21 brotli bytes against the +same upstream production source. The one row cache is necessary to preserve a +lazy Map without forwarding chains; the entry-update branch was simplified +alongside it. Tests and this evidence are development-only. A patch Changeset +releases the shared observer optimization independently of the Angular PR. + +The Angular branch separately projects row-state identity before copying its +Map. Both changes together were tested through Angular's generated metadata +receiving histories before the split. That integration evidence does not turn +this core-only PR into a framework contract: Angular's independent PR owns its +receiving behavior and can merge in either order. + +## Oracle rubric assessment + +| Requirement | Evidence or limit | +| --- | --- | +| ORC-001 | Maintainer-approved structural sharing and the LiveQuerySnapshot contract; limits above. | +| ORC-002 | The existing key/version model changes only on writes; metadata references are compared through a metamorphic relation. | +| ORC-003 | Contract prose, modelRows, grammar, real observer driver, and checkRead remain visible in the existing owner. | +| ORC-004 | Shared examples reconstruct metadata, empty rows, and same-key updates. Removing mode, attachment, cardinality, read order, or action kinds loses a distinct cache/laziness path. Health reports before readiness are excluded by the prefix. | +| ORC-005 | Actual observer getSnapshot before/after each metadata or write receipt, followed by retained-snapshot reads. | +| ORC-006 | Unchanged production and the same-cardinality mutant fail at the intended reference/value checks. | +| ORC-007 | Fixed seed 1101 and unseeded campaigns use identical examples, grammar, budget, and checker; shared replay variables select the property and shrink path. | +| ORC-008 | No production cache state is mirrored; the model retains only key/version rows. | +| ORC-009 | Snapshot explicitly projects data/state across direct and virtual query rows; glossary terms are preserved. | +| ORC-010 | finishHistory preserves primary and cleanup failures. | +| ORC-011 | No additional named shared semantic fault; retained old snapshots independently reject in-place Map mutation. | +| ORC-012 | This record identifies the production source and independent branch boundary. | +| ORC-013 | Metadata transitions require sharing, while same-key value writes require replacement. | +| ORC-014 | Core-only synchronous Collection boundary; no unverified host/provider transfer is claimed. | +| ORC-015 | Eight declared reach states, including metadata transitions and value-only changes. | +| ORC-016 | Fixed and generated histories use the same model and comparison; ordinary examples remain in the ordinary test file. | diff --git a/packages/db/mangle-cache.json b/packages/db/mangle-cache.json index b636738f25..05a9977eb0 100644 --- a/packages/db/mangle-cache.json +++ b/packages/db/mangle-cache.json @@ -435,5 +435,6 @@ "awaitsSiblingReplays": "gN", "grownPrefixRows": "gZ", "ownPrefixEnd": "he", - "sourceListenerCleanups": "cb" + "sourceListenerCleanups": "cb", + "cachedRows": "dB" } diff --git a/packages/db/src/live-query-observer.ts b/packages/db/src/live-query-observer.ts index b778c62cf3..475e4f8c94 100644 --- a/packages/db/src/live-query-observer.ts +++ b/packages/db/src/live-query-observer.ts @@ -19,6 +19,7 @@ import type { ChangeMessage, CollectionStatus, HealthNode } from './types.js' * query changes, so `useSyncExternalStore`-style consumers can compare by * reference. Each snapshot owns a captured view of `state`/`data`, so reading * an older snapshot cannot expose rows from a later revision. + * Metadata-only changes replace the snapshot while retaining its row containers. */ export interface LiveQuerySnapshot< T extends object, @@ -168,6 +169,13 @@ class LiveQueryObserverImpl< // Whether the cached snapshot was built for the hydration read. private cachedServerView = false private cachedSnapshot: LiveQuerySnapshot = DISABLED_SNAPSHOT + private cachedRows: + | { + entries: Array<[TKey, T]> + data: Array + readonly state: Map + } + | undefined private layoutRevision = 0 private lastLayoutKeys: Array | undefined private deliveredLayoutRevision: number | undefined @@ -257,7 +265,19 @@ class LiveQueryObserverImpl< if (this.snapshotDirty) { const entries = this.getVisibleEntries(collection) - const data = entries.map(([, value]) => value) + // Capture each row revision once. Sharing this value across metadata + // snapshots keeps rows lazy and avoids retaining a chain of old snapshots. + let state: Map | undefined + const rows = + this.cachedRows?.entries === entries + ? this.cachedRows + : (this.cachedRows = { + entries, + data: entries.map(([, value]) => value), + get state() { + return (state ??= new Map(entries)) + }, + }) const singleResult = isSingleResultCollection(collection) const liveStatus = this.visibleStatus ?? collection.status // The seed stands in only for a result that is still coming, so an @@ -297,13 +317,11 @@ class LiveQueryObserverImpl< const health = this.publishedHealth ? this.publishedHealth.tree : (collection as { health?: HealthNode }).health - // Most consumers read only `data`; build the keyed view on first read. - let state: Map | undefined this.cachedSnapshot = { get state() { - return (state ??= new Map(entries)) + return rows.state }, - data: singleResult ? data[0] : data, + data: singleResult ? rows.data[0] : rows.data, collection: getPublicCollection(collection), layoutRevision: this.layoutRevision, status, @@ -390,12 +408,12 @@ class LiveQueryObserverImpl< revision: number | undefined, ): void { const changed = - revision !== undefined - ? this.cachedEntries === undefined || - revision !== this.cachedCollectionRevision - : !this.entriesEqual(this.cachedEntries, entries) + (revision !== undefined && revision !== this.cachedCollectionRevision) || + !this.entriesEqual(this.cachedEntries, entries) - this.cachedEntries = entries + // Keep identity on unchanged reattachment. Comparing entries when the row + // revision is equal also detects layout-only moves and revisionless inputs. + if (changed) this.cachedEntries = entries this.cachedCollectionRevision = revision if (changed) this.snapshotDirty = true } diff --git a/packages/db/tests/live-query-observer-snapshot-oracle.property.test.ts b/packages/db/tests/live-query-observer-snapshot-oracle.property.test.ts index 48afa97e66..988e45d8a2 100644 --- a/packages/db/tests/live-query-observer-snapshot-oracle.property.test.ts +++ b/packages/db/tests/live-query-observer-snapshot-oracle.property.test.ts @@ -6,10 +6,16 @@ import { getLiveQueryHash } from '../src/live-query-options.js' import { localOnlyCollectionOptions } from '../src/local-only.js' import { createLiveQueryObserver } from '../src/live-query-observer.js' import { Query, createLiveQueryCollection } from '../src/query/index.js' -import { oraclePropertyOptions, oracleRuns } from './oracle-config.js' +import { + oraclePropertyOptions, + oracleReplayTest, + oracleRuns, +} from './oracle-config.js' import { declareOracle } from './oracle-declaration.js' import { createOracleReach } from './oracle-reach.js' -import type { HealthNode } from '../src/types.js' +import { finishHistory } from './observer-history-cleanup.js' +import type { HealthNode, SyncConfig } from '../src/types.js' +import type { LiveQuerySnapshot } from '../src/live-query-observer.js' /** * # Is a retained live-query snapshot point-in-time? @@ -62,7 +68,9 @@ type Command = | { type: `delete`; id: string } | { type: `capture` } type Read = { snapshot: number; dataFirst: boolean } -type Snapshot = ReturnType[`observer`][`getSnapshot`]> +// This projection observes row containers, whether a driver uses a source +// Collection directly or query rows that also carry virtual metadata. +type Snapshot = Pick, `data` | `state`> type Captured = { snapshot: Snapshot rows: Array @@ -638,3 +646,199 @@ describe(`a snapshot shows the current health tree`, () => { ]) }) }) + +/** + * Metadata changes replace the full snapshot but retain its row containers. + * Authority: maintainer decision 2026-11-01, preserving point-in-time rows while + * preventing status/health changes from invalidating row-only consumers. + * A value write must still create fresh containers, even with the same keys. + * + * This extends the retained-snapshot owner with a direct Collection boundary: + * Every history first marks the source ready without writing rows; its suffix + * may then report health legally. These metadata actions do not write rows. The independent model + * is modelRows' key/version map; only `write` changes it. At every action the + * driver compares row values and container identities, then checks all retained + * snapshots again to reject a mutable shared Map. Attached/detached and both + * observer modes cross distinct cache paths. Empty and nonempty rows test lazy + * Map materialization. Query shapes and hydration handoff retain their existing + * owners above; this lane does not claim provider or framework scheduling. + */ +const containerHistory = fc.record({ + mode: fc.constantFrom(`granular` as const, `wholesale` as const), + attached: fc.boolean(), + count: fc.integer({ min: 0, max: 3 }), + dataFirst: fc.boolean(), + commands: fc.array(fc.constantFrom(`report`, `healthy`, `ready`, `write`), { + minLength: 1, + maxLength: 12, + }), +}) +type ContainerHistory = + typeof containerHistory extends fc.Arbitrary ? T : never +type ContainerReach = + | `attached` + | `detached` + | `granular` + | `wholesale` + | `empty` + | `health-only` + | `status-only` + | `value-only` + +async function checkContainerHistory( + scenario: ContainerHistory, + reach: ReturnType>, +): Promise { + const model = new Map( + Array.from( + { length: scenario.count }, + (_, index) => [String.fromCharCode(97 + index), 0] as const, + ), + ) + let controls: Parameters[`sync`]>[0] | undefined + const source = createCollection({ + getKey: (row) => row.id, + gcTime: 0, + startSync: true, + sync: { + sync: (params) => { + controls = params + params.begin() + for (const row of modelRows(model)) + params.write({ type: `insert`, value: row }) + params.commit() + }, + }, + }) + const observer = createLiveQueryObserver(source, { mode: scenario.mode }) + const stop = scenario.attached ? observer.subscribe(() => {}) : undefined + const retained: Array = [] + reach.reach(scenario.attached ? `attached` : `detached`) + reach.reach(scenario.mode) + if (model.size === 0) reach.reach(`empty`) + let failure: { error: unknown } | undefined + try { + let before = observer.getSnapshot() + let beforeRead = checkRead( + `ordered`, + { snapshot: before, rows: modelRows(model) }, + scenario.dataFirst, + `initial`, + ) + retained.push({ snapshot: before, rows: modelRows(model) }) + for (const [index, step] of [ + `ready` as const, + ...scenario.commands, + ].entries()) { + const input = controls! + if (step === `write`) { + const version = (model.get(`a`) ?? -1) + 1 + input.begin() + input.write({ + type: model.has(`a`) ? `update` : `insert`, + value: { id: `a`, version }, + }) + input.commit() + model.set(`a`, version) + } else if (step === `report`) + input.reportError(new Error(`problem ${index}`)) + else if (step === `healthy`) input.reportHealthy() + else input.markReady() + const after = observer.getSnapshot() + const read = checkRead( + `ordered`, + { snapshot: after, rows: modelRows(model) }, + scenario.dataFirst, + `step ${index}`, + ) + if (step === `write`) { + expect(read.data, `value writes replace data`).not.toBe(beforeRead.data) + expect(read.state, `value writes replace state`).not.toBe( + beforeRead.state, + ) + if (beforeRead.state?.has(`a`)) reach.reach(`value-only`) + } else { + expect(read.data, `metadata keeps data`).toBe(beforeRead.data) + expect(read.state, `metadata keeps state`).toBe(beforeRead.state) + if (after.status !== before.status) reach.reach(`status-only`) + if (after.health !== before.health) reach.reach(`health-only`) + if (after.status !== before.status || after.health !== before.health) + expect(after).not.toBe(before) + } + retained.push({ snapshot: after, rows: modelRows(model) }) + before = after + beforeRead = read + } + for (const [index, captured] of retained.entries()) + checkRead(`ordered`, captured, !scenario.dataFirst, `retained ${index}`) + } catch (error) { + failure = { error } + } + await finishHistory( + [() => stop?.(), () => observer.dispose(), () => source.cleanup()], + failure, + ) +} + +declareOracle({ check: [checkContainerHistory], grammar: [containerHistory] }) + +describe(`metadata snapshots share immutable row containers`, () => { + const examples: Array<[ContainerHistory]> = ( + [`granular`, `wholesale`] as const + ).flatMap((mode) => + [false, true].flatMap((attached) => + [0, 1, 3].map( + (count) => + [ + { + mode, + attached, + count, + dataFirst: count % 2 === 0, + commands: [`report`, `ready`, `write`, `healthy`, `write`], + }, + ] as [ContainerHistory], + ), + ), + ) + for (const campaign of [`fixed`, `random`] as const) { + oracleReplayTest( + it, + ...(campaign === `fixed` + ? [] + : [`live-query-observer.row-container-sharing`]), + )(`reuses rows across metadata changes (${campaign})`, async () => { + const reach = createOracleReach( + `live-query-observer.row-container-sharing`, + [ + `attached`, + `detached`, + `granular`, + `wholesale`, + `empty`, + `health-only`, + `status-only`, + `value-only`, + ], + ) + await fc.assert( + fc.asyncProperty(containerHistory, async (scenario) => { + reach.beginHistory() + await checkContainerHistory(scenario, reach) + }), + { + ...oraclePropertyOptions( + 40, + `live-query-observer.row-container-sharing`, + { examples }, + ), + ...(campaign === `fixed` ? { seed: 1101 } : {}), + }, + ) + reach.assertReached() + process.stdout.write( + `observer row-container sharing ${campaign}: ${JSON.stringify(Object.fromEntries(reach.counts()))}\n`, + ) + }) + } +}) diff --git a/packages/db/tests/oracle-config.ts b/packages/db/tests/oracle-config.ts index f851ae34e8..5c79a8d3ae 100644 --- a/packages/db/tests/oracle-config.ts +++ b/packages/db/tests/oracle-config.ts @@ -28,6 +28,7 @@ const staticOracleProperties = [ `live-query-observer.partial-attach`, `live-query-observer.release-failure`, `live-query-observer.hydration-error-timing`, + `live-query-observer.row-container-sharing`, `subscription-lifecycle.notification-error`, `live-query.start-handoff`, `react.descriptor-error-surface`,