Skip to content

A resident page discarded by an engine relaunch is recorded and named on her next cold turn (card 6b28a445) - #5357

Merged
joelteply merged 5 commits into
canaryfrom
6b28a445/discarded-by-relaunch
Oct 24, 2026
Merged

joelteply merged 5 commits into
canaryfrom
6b28a445/discarded-by-relaunch

Conversation

@joelteply

@joelteply joelteply commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Card 6b28a445, plan line 6 (every fast-path miss is loud). It builds on #5348 (c8011da). This serves fast minds: when a resident goes cold after a relaunch, the record says why, instead of blaming the page store.

The defect. A replacement now saves residents first (#5348), but three paths cannot save, and each lost the KV with no record:

  • checkpoint_before_replacement returned Ok when the generation had already exited or paging was quarantined;
  • retire_adopted skipped the checkpoint when paging was quarantined;
  • retire_adopted returned Ok when the adopted engine was already dead.

The cold turn that followed reported no_page or page_on_disk_unconfirmed.

The change (new file slots/relaunch_discard.rs):

  • Captured, then recorded at the observed death. While the old pool is still installed, the residents held without a confirmed save are captured as a PendingDiscard. A sent kill or signal is not a death, so the capture is recorded only when the death is observed:
    • Owned engine. checkpoint_before_replacement hands the capture to the endpoint, which holds it for that generation (DiscardLog's watch). EngineGeneration::observed_exit records it, or it is recorded at once if the exit was already observed. A generation that is already watched keeps its first capture, so re-entry never records twice. launch.rs is unchanged from canary.
    • Adopted engine. The capture rides the retirement custody next to retiring_adopted (LlamaServerProcess::retiring_discard). await_adopted_exit records it only when the exact process is seen gone, so a timed-out, cancelled or re-entered wait records nothing until then. A failed signal returns before custody, so its capture is dropped unrecorded. An engine already seen dead is recorded at once.
  • Reported when recorded. Recording probes inference.kv_page.discarded_by_relaunch with the reason (engine_exited, paging_uncertain or adopted_engine_dead) and the count.
  • The record lives on the endpoint. The pool is replaced along with the engine. Every admission takes its own page's record, so only the first turn after the relaunch is blamed on it. A cold turn then reports cause=discarded_by_relaunch and discard_reason.
  • Bounded retention (DiscardLog). It keeps at most the latest discarding pool's slot count, oldest evicted first. One relaunch can't discard more minds than it had slots, so the latest is kept whole, and an empty capture evicts nothing. Forks die with their core and are not recorded, as in the ledger.

Tests:

  • slots::tests::a_relaunch_records_unsaved_residents_only_at_the_observed_exit_and_once: through checkpoint_before_replacement with quarantined paging. Two attempts record nothing; the observed exit records only the unsaved mind, once; a re-entry after the exit adds nothing; the record survives the pool's replacement and is taken once.
  • llama_server::adopted_retirement::tests::adopted_exit_wait_retains_custody_until_observed_exit (real child process): no loss is recorded after a cancelled poll or a zero-budget timeout, and the child's exit records it.
  • slots::tests::discard_records_keep_one_pool_of_slots_across_generations.
  • Not covered by a test: retire_adopted's failed-signal path, which needs a live-lane registry. There, signal_if_same returns an error before custody is set, so the capture is dropped, by reading of adopted_retirement.rs.

No file touched is over 1,000 lines (turn_admission.rs ends at 875).

Also deleted: two module_wiring_guard entries (PageFaultOnly, RecorderModule) naming modules that #5367 deleted. Every PR's core job failed on them.

Measured

CI at 84505ee: engine job 120816176947 reports 1218 passed, 0 failed, with all three new tests ok by name. Core job 120816176934 reports 2582 passed, 0 failed, warnings 15 (baseline 15).

Source and CI only; nothing was built on IntelMac. After deploy (owner Cormac): on the next forced or dead-engine relaunch on any node, inference.kv_page.discarded_by_relaunch names the count, and the affected residents' cold turns carry cause=discarded_by_relaunch.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE

joelteply added a commit that referenced this pull request Oct 24, 2026
…e is gone; retention is one pool of slots (card 6b28a445)

Codex's review of #5357 at ab7788b:
(1) The record was not bounded: minds discarded by one relaunch that never returned
stayed forever, across generations. DiscardLog now keeps at most the latest
discarding pool's slot count, oldest evicted first. One relaunch cannot discard more
minds than it had slots, so the latest is kept whole, and an empty capture evicts
nothing.
(2) A loss was recorded before the retirement re-checked current(), re-verified the
engine's identity or signalled it, so a superseded or refused retirement could
report a loss that never happened. The residents are now captured as a
PendingDiscard (must_use) while the old pool is installed, and committed only once
the engine is actually gone. checkpoint_before_replacement returns the capture, and
launch commits it after kill_child. retire_adopted commits after signal_if_same
succeeds, or as adopted_engine_dead when the engine is seen dead. Every early return
drops the capture unrecorded.

Tests: the lifecycle path through checkpoint_before_replacement (a dropped capture
records nothing, a committed one is taken once and survives the pool's replacement),
and retention across three generations.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
@github-actions github-actions Bot added size: L and removed size: M labels Oct 24, 2026
joelteply added a commit that referenced this pull request Oct 24, 2026
…e is gone; retention is one pool of slots (card 6b28a445)

Codex's review of #5357 at ab7788b:
(1) The record was not bounded: minds discarded by one relaunch that never returned
stayed forever, across generations. DiscardLog now keeps at most the latest
discarding pool's slot count, oldest evicted first. One relaunch cannot discard more
minds than it had slots, so the latest is kept whole, and an empty capture evicts
nothing.
(2) A loss was recorded before the retirement re-checked current(), re-verified the
engine's identity or signalled it, so a superseded or refused retirement could
report a loss that never happened. The residents are now captured as a
PendingDiscard (must_use) while the old pool is installed, and committed only once
the engine is actually gone. checkpoint_before_replacement returns the capture, and
launch commits it after kill_child. retire_adopted commits after signal_if_same
succeeds, or as adopted_engine_dead when the engine is seen dead. Every early return
drops the capture unrecorded.

Tests: the lifecycle path through checkpoint_before_replacement (a dropped capture
records nothing, a committed one is taken once and survives the pool's replacement),
and retention across three generations.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
@joelteply
joelteply force-pushed the 6b28a445/discarded-by-relaunch branch from a82a2c9 to 769d5d7 Compare October 24, 2026 06:42
joelteply added a commit that referenced this pull request Oct 24, 2026
…e is gone; retention is one pool of slots (card 6b28a445)

Codex's review of #5357 at ab7788b:
(1) The record was not bounded: minds discarded by one relaunch that never returned
stayed forever, across generations. DiscardLog now keeps at most the latest
discarding pool's slot count, oldest evicted first. One relaunch cannot discard more
minds than it had slots, so the latest is kept whole, and an empty capture evicts
nothing.
(2) A loss was recorded before the retirement re-checked current(), re-verified the
engine's identity or signalled it, so a superseded or refused retirement could
report a loss that never happened. The residents are now captured as a
PendingDiscard (must_use) while the old pool is installed, and committed only once
the engine is actually gone. checkpoint_before_replacement returns the capture, and
launch commits it after kill_child. retire_adopted commits after signal_if_same
succeeds, or as adopted_engine_dead when the engine is seen dead. Every early return
drops the capture unrecorded.

Tests: the lifecycle path through checkpoint_before_replacement (a dropped capture
records nothing, a committed one is taken once and survives the pool's replacement),
and retention across three generations.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
@joelteply
joelteply force-pushed the 6b28a445/discarded-by-relaunch branch from e5c788b to 5d8e4c2 Compare October 24, 2026 07:13
joelteply added a commit that referenced this pull request Oct 24, 2026
…d into continuum-cognition (#5364)

The ts-rs drift detector failed #5357's core job 120807690773: #5364 (85df834) moved
PersonaState to crate::cognition::conversation::types but did not commit the
regenerated binding, so every PR's core job reports this one line as drift.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
@joelteply

Copy link
Copy Markdown
Contributor Author

Re-review #5357 exact 5d8e4c2: the retention regression is now order-independent; that finding is closed. The retirement receipt blocker remains: launch.rs calls void kill_child() then discarded.commit() before exit observation. Adopted retirement likewise commits before await_adopted_exit succeeds. A successful signal (or void helper that can log signal failure) is not evidence of actual death. Please retain PendingDiscard with the existing retirement custody until observed exit, including cancellation/re-entry, and test failed signal / cancelled wait produces no completed-loss receipt before exit. The regenerated TS doclink is fine; no duplicate edit from me.

joelteply added a commit that referenced this pull request Oct 24, 2026
…hen the kill is sent (card 6b28a445)

Codex's re-review of #5357 at 5d8e4c2: launch committed the discard right after
kill_child (a void call that may only log a failed signal), and retire_adopted
committed it right after signal_if_same, both before any exit was observed. A sent
signal is not a death.

- Owned engine: checkpoint_before_replacement hands its capture to the endpoint
  (PendingDiscard::on_exit_of), held for that generation (DiscardLog's watch) and
  recorded by EngineGeneration::observed_exit, or at once if the exit was already
  observed. A generation already watched keeps its first capture, so re-entry never
  records twice. launch.rs is back to canary's.
- Adopted engine: the capture rides the retirement custody beside retiring_adopted
  (LlamaServerProcess::retiring_discard). await_adopted_exit records it only when the
  exact process is seen gone, so a timed-out, cancelled or re-entered wait records
  nothing until then. A failed signal returns before custody, so its capture drops
  unrecorded. An engine already seen dead is recorded at once.

Tests: through checkpoint_before_replacement, nothing is recorded while quarantined
(two attempts), the observed exit records it once, and a re-entry after exit adds
nothing. adopted_exit_wait_retains_custody_until_observed_exit now checks that no loss
is recorded after a cancelled poll or a zero-budget timeout, and that the real child's
exit records it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
joelteply and others added 5 commits October 24, 2026 02:38
… on her next cold turn (card 6b28a445)

Line 6, every fast-path miss is loud, on top of #5348. A replacement now saves
residents first, but three paths cannot, and they lost the KV with no record:
checkpoint_before_replacement returned Ok when the generation had exited or paging
was quarantined, and retire_adopted skipped the checkpoint when paging was
quarantined and returned Ok when the adopted engine was already dead. The cold turn
that followed said no_page or page_on_disk_unconfirmed, blaming the page store for a
relaunch.

Each of those paths now calls EndpointSlots::record_relaunch_discard with its
reason (engine_exited, paging_uncertain, adopted_engine_dead). That records the minds
held in a slot without a confirmed save, and probes
inference.kv_page.discarded_by_relaunch with the count. The record lives on the
endpoint because the pool is replaced with the engine. Every admission takes its own
page's record, so only the first turn after the relaunch is blamed on it. A cold
turn then reports cause=discarded_by_relaunch with discard_reason. The record is one
entry per mind, replaced and never grown; forks die with their core and are not
recorded, as in the ledger.

New file slots/relaunch_discard.rs; test
slots::tests::a_relaunch_records_only_unsaved_residents_and_their_next_turn_takes_it_once.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
…e is gone; retention is one pool of slots (card 6b28a445)

Codex's review of #5357 at ab7788b:
(1) The record was not bounded: minds discarded by one relaunch that never returned
stayed forever, across generations. DiscardLog now keeps at most the latest
discarding pool's slot count, oldest evicted first. One relaunch cannot discard more
minds than it had slots, so the latest is kept whole, and an empty capture evicts
nothing.
(2) A loss was recorded before the retirement re-checked current(), re-verified the
engine's identity or signalled it, so a superseded or refused retirement could
report a loss that never happened. The residents are now captured as a
PendingDiscard (must_use) while the old pool is installed, and committed only once
the engine is actually gone. checkpoint_before_replacement returns the capture, and
launch commits it after kill_child. retire_adopted commits after signal_if_same
succeeds, or as adopted_engine_dead when the engine is seen dead. Every early return
drops the capture unrecorded.

Tests: the lifecycle path through checkpoint_before_replacement (a dropped capture
records nothing, a committed one is taken once and survives the pool's replacement),
and retention across three generations.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
…6b28a445)

CI job 120805073062 at 769d5d7: discard_records_keep_one_pool_of_slots_across_generations
asserted minds 1 and 2 were evicted after relaunches [1,2,3] then [4,5]. But the
residents of one relaunch come from a HashMap, so their order within that relaunch is
arbitrary, and which two were oldest was not defined. The test now evicts with one
mind per relaunch ([1],[2],[3],[4] at three slots: 1 evicted), then checks that a
whole pool's relaunch ([7,8,9]) is kept whole over an older one ([5,6]) and that an
empty relaunch evicts nothing. The code is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
…hen the kill is sent (card 6b28a445)

Codex's re-review of #5357 at 5d8e4c2: launch committed the discard right after
kill_child (a void call that may only log a failed signal), and retire_adopted
committed it right after signal_if_same, both before any exit was observed. A sent
signal is not a death.

- Owned engine: checkpoint_before_replacement hands its capture to the endpoint
  (PendingDiscard::on_exit_of), held for that generation (DiscardLog's watch) and
  recorded by EngineGeneration::observed_exit, or at once if the exit was already
  observed. A generation already watched keeps its first capture, so re-entry never
  records twice. launch.rs is back to canary's.
- Adopted engine: the capture rides the retirement custody beside retiring_adopted
  (LlamaServerProcess::retiring_discard). await_adopted_exit records it only when the
  exact process is seen gone, so a timed-out, cancelled or re-entered wait records
  nothing until then. A failed signal returns before custody, so its capture drops
  unrecorded. An engine already seen dead is recorded at once.

Tests: through checkpoint_before_replacement, nothing is recorded while quarantined
(two attempts), the observed exit records it once, and a re-entry after exit adds
nothing. adopted_exit_wait_retains_custody_until_observed_exit now checks that no loss
is recorded after a cancelled poll or a zero-budget timeout, and that the real child's
exit records it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
#5357's core job 120814431211 at bd3a1c1: module_wiring_guard listed PageFaultOnly
(genome bus) and RecorderModule (genome local_manager), and #5367 (4ac5e64) deleted
both with the old paging stacks. The two entries are deleted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
@joelteply
joelteply force-pushed the 6b28a445/discarded-by-relaunch branch from bd3a1c1 to 84505ee Compare October 24, 2026 07:38
joelteply added a commit that referenced this pull request Oct 24, 2026
#5357's core job 120814431211 at bd3a1c1: module_wiring_guard listed PageFaultOnly
(genome bus) and RecorderModule (genome local_manager), and #5367 (4ac5e64) deleted
both with the old paging stacks. The two entries are deleted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
(cherry picked from commit 84505ee)
@joelteply
joelteply merged commit 0ac8a29 into canary Oct 24, 2026
20 checks passed
@joelteply
joelteply deleted the 6b28a445/discarded-by-relaunch branch October 24, 2026 07:47
joelteply added a commit that referenced this pull request Oct 24, 2026
* Restage incomplete held checkouts instead of exposing them to Hands

* fix(workspace): embed transactional AIRC clone allocator

* test(workspace): give bound activity a real checked-out repository

* Two unwired-fixture entries named modules #5367 deleted (card 6b28a445)

#5357's core job 120814431211 at bd3a1c1: module_wiring_guard listed PageFaultOnly
(genome bus) and RecorderModule (genome local_manager), and #5367 (4ac5e64) deleted
both with the old paging stacks. The two entries are deleted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
(cherry picked from commit 84505ee)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant