Skip to content

Detect missing commander heartbeats from request and message owners - #2624

Closed
trissim wants to merge 4 commits into
integration/owner-requests-current-busfrom
next/commander-control-plane
Closed

trissim wants to merge 4 commits into
integration/owner-requests-current-busfrom
next/commander-control-plane

Conversation

@trissim

@trissim trissim commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

A newer commander reply could hide a stale heartbeat because HeartbeatSilent first selected the newest row of a two-way conversation and only then checked its sender. A service that never sent its first beat also stayed healthy forever. This change reports both failures from the existing message and request owners.

CommanderHeartbeat.requested_by owns live-holder eligibility and the oldest open OwnerRequest. Both service selection and the shared HeartbeatSilent scan use it. HeartbeatNotStarted carries the actual request identity; MissedHeartbeat carries the actual bus sequence. New requests, registration, replies, and renames cannot erase older service debt. The existing 180-second grace begins at the later of the open request and latest beat.

BusReader.latest_addressed selects at most one candidate per exact sender/target alias pair through TypedTable.select, chooses the newest sequence, and decodes that message under the same read snapshot. Addressed declares the sender/target/sequence index; registry provenance supplies aliases. The initially introduced DirectedDMDisplayScope is deleted and ordinary DM scopes retain their original implementation. There is no second timestamp, ping store, scheduler, or participant projection.

This is stacked on #2621, branch integration/owner-requests-current-bus at ca156d2. Review only this branch's changes.

Validation:

  • Eighteen focused tests passed in 7.54 seconds: the two heartbeat cases, the existing bounded-source query case, owner requests, message pages, and channel display. These exercise real copied-bus publications, service/commander renames, newest selection across aliases, later reverse replies, other commanders, and no prior service beat.
  • Replayed the actual BusReader query on a copy of the retained 19,189-record bus. Missing service with aliases: 45,088 to 76 SQLite instructions; existing newest beat: 22,642 to 137. All exact-pair reads use the declared covering index and preserve the selected result. Instrumented elapsed samples were 5.843 to 0.194 milliseconds for absence and 4.057 to 0.782 milliseconds for the existing beat; these are query measurements, not end-to-end tool latency evidence.
  • The index alone was rejected after an alias counterexample made SQLite choose the old inbox scan. The final exact-pair query removes that scan.
  • Existing BusSchema.reconcile_schema adds the declared index on the copied file, preserves all bus sequences and both 19,189 row counts, passes integrity/foreign-key checks, and changes nothing on a second run. Copy authorization uses WriterOwnedFile; installed maintenance qualification remains with the merger.
  • NRA before/after source evidence retains every original declaration and unsupported projection. Final Core/hooks/tests/tools census: 956 modules, 5,364 original classes, 5,174 class projections, 190 unprojected declarations, zero parse omissions. Lexical/MRO evidence is not a proof of dynamic runtime binding.

Source evidence and replay receipts: /home/ts/.cache/agent-scratch/commander-control-plane/ (source-before.json, source-before-never-beat.json, index-source-before.json, index-source-after.json, bounded-liveness-tests.txt, query-reconciled-replay.json).

Remaining: installed commander full-session/parity acceptance and the broader control-plane phase. Plan phase declarations are untouched. Reconciliation currently rebuilds every bus projection even for this index-only delta, costing 39.83 seconds in the replay; that existing schema/installer defect is reported to the assistant commander for implementation ownership. This query fix is not presented as an install-latency optimization.

Slop receipt: IDEN-7. Required fact: the latest actual heartbeat to this commander. Former site: HeartbeatSilent.violations selected a two-way conversation row before testing sender. Derivation: registry-owned aliases feed the bus owner's exact directed lookup, using the Addressed declaration's index. No display row now stands in for service liveness.

Assumed: existing open OwnerRequest.created_at owns when current service is owed; current live-holder eligibility and the 180-second grace are preserved. No live installation or full control-plane completion is claimed.

@trissim
trissim force-pushed the next/commander-control-plane branch from 7289a28 to 83071a0 Compare October 23, 2026 03:09
@trissim trissim changed the title Keep commander replies from hiding a silent heartbeat Detect missing commander heartbeats from request and message owners Oct 23, 2026
@trissim
trissim changed the base branch from integration/bus-current-main to integration/owner-requests-current-bus October 23, 2026 03:30

@trissim trissim left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Source checkpoint accepted against its explicit P421 base. The old scan could call the heartbeat healthy forever if the service never spoke, and a newer commander reply could hide a stale beat. Service eligibility and the oldest unpaid start now derive from the existing OwnerRequest owner; the two failure members carry a real request identity or real beat sequence. BusReader.latest_addressed uses existing typed exact-address selection under one snapshot, with Addressed owning its compound index and registry provenance supplying aliases. The introduced display-scope abstraction is gone. I read the full final source, changed behavior checks and retained alias counterexample/reconciliation receipts; independently replayed newest-across-aliases plus stale/missing-first-beat and rename cases: 3 passed in 1.57s. Query work is bounded by alias pairs, not inbox history. Instrumented milliseconds are query evidence, not whole-tool latency. The discovered roughly forty-second projection rebuild for an index-only change has a named follow-through owner: this same worker, on a separate schema branch coordinated with current shared-file owners. Installed session/parity and the broader control-plane phase remain open.

@trissim

trissim commented Oct 23, 2026

Copy link
Copy Markdown
Contributor Author

Closing as superseded: all 4 commits of this branch are in #2630 by patch id (git cherry origin/next/response-policy-owner origin/next/commander-control-plane shows none missing). Review and land #2630.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant