Skip to content

Avoid replaying bus history for catalog-only schema changes - #2629

Closed
trissim wants to merge 4 commits into
next/commander-control-planefrom
next/schema-projection-invalidation
Closed

trissim wants to merge 4 commits into
next/commander-control-planefrom
next/schema-projection-invalidation

Conversation

@trissim

@trissim trissim commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

An index-only change made schema reconciliation reset every bus projection and decode the whole history. On the retained 19,189-record bus, that cost 39,827 milliseconds. The same reconciliation now takes 24.845 milliseconds and writes zero rows, preserving the existing source and projection contents.

Invalidated safety premise: a combined producer-behavior and index change is a regression. The old composed carry rebuilt projections; this branch can skip them because it mistakes zero source-row writes for unchanged producing behavior. Keep this branch draft, unaccepted, and out of the current install. The 24.845-millisecond measurement is valid only for the unchanged producer used in that replay.

Open correctness gap: installed decoder/projector changes with identical DDL can silently retain stale projections. No production caller currently guarantees a behavior-change reset. Planned reconciliation checks only DDL, recorded build identity has no projection-invalidation consumer, and direct explicit reset callers are tests. The measured catalog-only improvement is real; automatic freshness across behavior changes remains unfinished in this schema work.

ComposedSchemas derives preceding row writes from the existing SQLite connection's total_changes counter and passes that count through the shared carry contract. DerivedTables owns the decision: rebuild after preceding row writes or a changed projection table shape; otherwise derive the index, trigger, or view through the existing catalog mechanism. TableSchema compares table DDL through its existing declarations and stored-schema readers. Durable carry and explicit reset_schema keep their existing behavior. No new store, version stamp, cache, schema-name switch, or dependency registry is introduced.

The source relation is bounded and explicit: BusSchema composes append-only durable history followed by its projections; ordinary append updates both in one transaction. Nullable-field carry rewrites source rows through TypedTable.rebuild, which records writes in SQLite. Storage renames and rowless catalog changes preserve decoded row values. PrivateCoordinatorSchema composes two durable families and has no derived reset. The only direct carry callers are the top-level reconciliation and composed-family traversal. Explicit reset_schema use outside carry is test code; there is no wired production behavior-change reset.

Validation:

  • Four focused tests passed in 0.41 seconds. The real-bus control covers index repair on both durable and projection tables without row writes, source-row carry requiring replay despite unchanged projection DDL, changed projection tables requiring replay without source writes, and unconditional explicit content repair.
  • Two existing installer CLI cases passed in 16.46 seconds: held reconciliation names every rewritten store and private reconciliation restores missing native tables while retaining registered participants. These exercise both composed schemas through the actual source CLI under a maintenance hold.
  • One final retained-bus replay compared the same missing Addressed index against the earlier 39.83-second baseline. New time: 24.845 milliseconds; total_changes delta: zero. All 19,189 bus/address rows and every projection count remain unchanged; integrity and foreign-key checks pass; repeating reconciliation is a no-op. The original saved database was opened read-only.
  • NRA source census: 956 Core/hooks/tests/tools modules, 5,364 original classes, 5,174 projections, 190 unprojected declarations, zero parse omissions. Current executable-owner labels use NRA indexed-node IDs. Invalid owner labels in two earlier supplemental receipts were removed; source positions/text and counts remain valid. AST evidence is lexical/MRO evidence, not a proof of dynamic binding.

Commands, results, source evidence, and the before/after replay are retained under /home/ts/.cache/agent-scratch/commander-control-plane/: schema-source-before.json, schema-source-after.json, schema-carry-controls-corrected.txt, schema-carry-cli.txt, schema-carry-replay.json, query-reconciled-replay.json.

This is stacked on heartbeat repair #2624 at 477a44a. That accepted head is unchanged. This branch does not install or merge either change; installed paired/runtime qualification remains with the integration owner.

Assumed: carry uses the existing family declaration order. SQLite's row count witnesses actual preceding writes, not arbitrary projection dependencies. Decoder/projector behavior changes with identical stored DDL have an open invalidation defect. An explicit reset was previously described as a contract, but no production owner guarantees it; that claim is withdrawn. This branch remains draft while the schema owner and installer owner resolve that gap.

Batch scope checked: accepted #2624 to this branch changes only schema carry, tests, and plan prose. Installed Core d9ce9cd to root integration 883223e includes the SQLite bus/DDL conversion. No actual unchanged-DDL producer-change stale-output case was identified in those checked revisions. That scoped result does not justify a generic safe-skip claim. The schema owner is tracing the stored response-policy value and its actual role owner; no additional replay or version mechanism was introduced.

@trissim trissim left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not accepted for installation. The measured deletion is real: the copied index-only migration avoids the old full-history replay, and the focused schema/data controls pass. I independently replayed the metadata/source-conversion/projection-shape control plus held-schema and no-install-on-read checks: 3 passed in 0.37s. But DerivedTables.carry_schema now treats zero preceding row writes plus unchanged table DDL as enough to retain projections. That does not establish unchanged producing behavior. Concrete counterexample: Message.projections stores the result of role.response_policy; change that implementation to another already-declared policy while adding an index in the same release. The previous composed carry refreshed those policies; the new skip leaves the old result. No production explicit reset handles that case: planned-reconcile only checks DDL, equal-DDL reconciliation returns early, and the explicit reset in these controls is a test. The same worker retains the structural dependency investigation. No such unchanged-DDL stale-output failure has been demonstrated in the current incoming batch; the generic API counterexample remains valid. Keep this draft outside the current paired installation while its missing ownership relation is repaired. Do not replace that relation with a manual reset obligation, dependency roster or guessed version stamp.

@trissim

trissim commented Oct 23, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by replacement PR #2636, which rebuilds derived projections from declaration-derived producer identity. Closing now per Tristan’s explicit branch-hygiene instruction. The DDL-only skip premise was invalidated; its published branch and evidence remain preserved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant