Skip to content

Rebuild bus projections from one joined cursor with carried claim holders - #2666

Merged
2 commits merged into
mainfrom
fix/projection-stream
Oct 23, 2026
Merged

2 commits merged into
mainfrom
fix/projection-stream

Conversation

@trissim

@trissim trissim commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Defect

BusProjectionTables.create_schema rebuilds every bus projection whenever the Core commit changes. Two parts of that replay did needless work:

  1. Per-row reads. A header stream, then kind.read_from(db, seq) issued one point SELECT per capability table for every record, each through the strict decode boundary.
  2. Claim rereads. Every replayed claim called HeldResources.claims_for. That reselected and fully decoded the current holders' Claimed rows, which the same in-order pass had already decoded.

Fix

Commit 1: one joined cursor.

  • BusCapability.history(db) reads every record in sequence order from one cursor: the history header LEFT JOINed to each capability table on seq. The table set is derived from TypedTable.members_with(CapabilityRow). Each stored row is decoded once, at its column offset.
  • read_from and history share one assembly step, BusCapability.assembled.
  • TypedTable.qualified() and decode_at() let one select join tables that share column names and decode each table's slice.

Commit 2: carried claim holders.

  • HeldResources.project and apply take a ClaimHolders member.
  • StoredHolders reads the holders of one live append from their table. That read is required: each append is its own transaction.
  • ReplayedHolders, owned by the create_schema pass, carries the owners it already projected and never rereads them.
  • apply stays the one owner of which resources leave, and hands that set to the holders.

Deleted

  • The per-row BusRow.stream + header.kind.read_from(db, header.seq) replay in create_schema.
  • The duplicated record assembly in read_from.
  • The claims_for reread during the rebuild.
  • Claimed.replay_claim, folded into project_into.

Not changed

  • ProjectionProducer.records_for: the commit stamp is kept. Stamp derived projections and index answer eligibility at append #2636 replaced per-class producer identities with it so that derived rows can't survive a change to the code that produces them while the SQL stays equal. Those producers are selected at runtime by role, type and reply scope (messages.py:413-414, bus_publication.py:57, read_basis.py:866), so a per-declaration stamp would need a hand-written producer list.
  • No cache and no second reader.

Measurements

All runs were niced, on backup-API copies of the live bus.sqlite3. Each deleted every projection table, replayed all records and rolled back.

Path Records Seconds
Per-row read_from (main) 19,448 46.5
Joined cursor (commit 1) 19,448 37.7
Joined cursor + stored holders (commit 1 shape) 19,457 41.5
Joined cursor + carried holders (commit 2) 19,457 34.5
  • The two 19,457-record runs come from one process on one copy, and every projection table's full contents are identical.
  • The 19,448-record runs compared row counts and decoded records, and both were equal.
  • Where the time goes (cProfile; the run took 109 s under the profiler):
    • All projection inserts together take about 6 s. received_source, at 96,786 rows, is 2.7 s of that.
    • The rest is decoding records (FieldCodec) and replaying claim transitions.

No pytest was run here; the integrator tests this once on integration. tests/test_bus_resource_claims.py:188 now replays through ReplayedHolders and compares against the live projection.

🤖 Generated with Claude Code

@trissim
trissim marked this pull request as ready for review October 23, 2026 07:59
Copilot AI balanced review requested due to automatic review settings October 23, 2026 07:59

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

debt-src at 5d5290c, every finding:

grew GodClassExcess:src/agent_comms/typed_table.py::TypedTable +15
grew LintFindings:src/agent_comms/bus_tables.py +5
grew LintFindings:src/agent_comms/typed_table.py +2
grew TypeErrors:src/agent_comms/bus_tables.py +2

@trissim
trissim force-pushed the fix/projection-stream branch from 3aef4be to 0dcf1f3 Compare October 23, 2026 08:01
trissim and others added 2 commits October 23, 2026 04:13
BusProjectionTables.create_schema replayed history with one header stream
plus one point SELECT per capability table per record (read_from), each
through the strict decode boundary. On a copy of the live bus (19,448
records) that replay took 46.5 s; the joined cursor takes 37.7 s with
identical projection rows. Writes are not the remaining cost: a profile
puts all projection inserts at about 6 s. The rest is decoding records
and replaying claim transitions.

BusCapability.history reads every record in sequence order from one
cursor: the history header LEFT JOINed to each capability table (members
derived from TypedTable.members_with(CapabilityRow)), each stored row
decoded once at its column offset. read_from and history share one
assembly step (BusCapability.assembled). The per-row replay loop in
create_schema is deleted; the projection stamp is unchanged.

TypedTable gains qualified() and decode_at() so a select joining tables
that share column names decodes each table's slice by declaration order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every replayed claim called HeldResources.claims_for, which reselected
and fully decoded the current holders' Claimed rows that the same
in-order pass had already decoded.

HeldResources.project and apply now take a ClaimHolders member:
StoredHolders reads the holders of one live append from their table
(the read a separate append transaction needs); ReplayedHolders, owned
by create_schema's pass, carries the owners it already projected and
never rereads them. apply stays the one owner of which resources leave
and hands that set to the holders. The Claimed.replay_claim split is
folded into project_into.

On a copy of the live bus (19,457 records), the same joined-cursor pass
with stored holders took 41.5 s and with carried holders 34.5 s; every
projection table's contents are identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@trissim
trissim force-pushed the fix/projection-stream branch from 0dcf1f3 to ddda841 Compare October 23, 2026 08:13
@trissim trissim changed the title Rebuild bus projections from one joined history cursor Rebuild bus projections from one joined cursor with carried claim holders Oct 23, 2026
@trissim trissim closed this pull request by merging all changes into main in 27043e0 Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants