Repository navigation
A resident page discarded by an engine relaunch is recorded and named on her next cold turn (card 6b28a445) - #5357
Merged
Merged
Conversation
joelteply
added a commit
that referenced
this pull request
Oct 24, 2026
…e is gone; retention is one pool of slots (card 6b28a445) Codex's review of #5357 at ab7788b: (1) The record was not bounded: minds discarded by one relaunch that never returned stayed forever, across generations. DiscardLog now keeps at most the latest discarding pool's slot count, oldest evicted first. One relaunch cannot discard more minds than it had slots, so the latest is kept whole, and an empty capture evicts nothing. (2) A loss was recorded before the retirement re-checked current(), re-verified the engine's identity or signalled it, so a superseded or refused retirement could report a loss that never happened. The residents are now captured as a PendingDiscard (must_use) while the old pool is installed, and committed only once the engine is actually gone. checkpoint_before_replacement returns the capture, and launch commits it after kill_child. retire_adopted commits after signal_if_same succeeds, or as adopted_engine_dead when the engine is seen dead. Every early return drops the capture unrecorded. Tests: the lifecycle path through checkpoint_before_replacement (a dropped capture records nothing, a committed one is taken once and survives the pool's replacement), and retention across three generations. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
joelteply
added a commit
that referenced
this pull request
Oct 24, 2026
…e is gone; retention is one pool of slots (card 6b28a445) Codex's review of #5357 at ab7788b: (1) The record was not bounded: minds discarded by one relaunch that never returned stayed forever, across generations. DiscardLog now keeps at most the latest discarding pool's slot count, oldest evicted first. One relaunch cannot discard more minds than it had slots, so the latest is kept whole, and an empty capture evicts nothing. (2) A loss was recorded before the retirement re-checked current(), re-verified the engine's identity or signalled it, so a superseded or refused retirement could report a loss that never happened. The residents are now captured as a PendingDiscard (must_use) while the old pool is installed, and committed only once the engine is actually gone. checkpoint_before_replacement returns the capture, and launch commits it after kill_child. retire_adopted commits after signal_if_same succeeds, or as adopted_engine_dead when the engine is seen dead. Every early return drops the capture unrecorded. Tests: the lifecycle path through checkpoint_before_replacement (a dropped capture records nothing, a committed one is taken once and survives the pool's replacement), and retention across three generations. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
joelteply
force-pushed
the
6b28a445/discarded-by-relaunch
branch
from
October 24, 2026 06:42
a82a2c9 to
769d5d7
Compare
joelteply
added a commit
that referenced
this pull request
Oct 24, 2026
…e is gone; retention is one pool of slots (card 6b28a445) Codex's review of #5357 at ab7788b: (1) The record was not bounded: minds discarded by one relaunch that never returned stayed forever, across generations. DiscardLog now keeps at most the latest discarding pool's slot count, oldest evicted first. One relaunch cannot discard more minds than it had slots, so the latest is kept whole, and an empty capture evicts nothing. (2) A loss was recorded before the retirement re-checked current(), re-verified the engine's identity or signalled it, so a superseded or refused retirement could report a loss that never happened. The residents are now captured as a PendingDiscard (must_use) while the old pool is installed, and committed only once the engine is actually gone. checkpoint_before_replacement returns the capture, and launch commits it after kill_child. retire_adopted commits after signal_if_same succeeds, or as adopted_engine_dead when the engine is seen dead. Every early return drops the capture unrecorded. Tests: the lifecycle path through checkpoint_before_replacement (a dropped capture records nothing, a committed one is taken once and survives the pool's replacement), and retention across three generations. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
joelteply
force-pushed
the
6b28a445/discarded-by-relaunch
branch
from
October 24, 2026 07:13
e5c788b to
5d8e4c2
Compare
joelteply
added a commit
that referenced
this pull request
Oct 24, 2026
…d into continuum-cognition (#5364) The ts-rs drift detector failed #5357's core job 120807690773: #5364 (85df834) moved PersonaState to crate::cognition::conversation::types but did not commit the regenerated binding, so every PR's core job reports this one line as drift. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
Contributor
Author
|
Re-review #5357 exact 5d8e4c2: the retention regression is now order-independent; that finding is closed. The retirement receipt blocker remains: launch.rs calls void kill_child() then discarded.commit() before exit observation. Adopted retirement likewise commits before await_adopted_exit succeeds. A successful signal (or void helper that can log signal failure) is not evidence of actual death. Please retain PendingDiscard with the existing retirement custody until observed exit, including cancellation/re-entry, and test failed signal / cancelled wait produces no completed-loss receipt before exit. The regenerated TS doclink is fine; no duplicate edit from me. |
joelteply
added a commit
that referenced
this pull request
Oct 24, 2026
…hen the kill is sent (card 6b28a445) Codex's re-review of #5357 at 5d8e4c2: launch committed the discard right after kill_child (a void call that may only log a failed signal), and retire_adopted committed it right after signal_if_same, both before any exit was observed. A sent signal is not a death. - Owned engine: checkpoint_before_replacement hands its capture to the endpoint (PendingDiscard::on_exit_of), held for that generation (DiscardLog's watch) and recorded by EngineGeneration::observed_exit, or at once if the exit was already observed. A generation already watched keeps its first capture, so re-entry never records twice. launch.rs is back to canary's. - Adopted engine: the capture rides the retirement custody beside retiring_adopted (LlamaServerProcess::retiring_discard). await_adopted_exit records it only when the exact process is seen gone, so a timed-out, cancelled or re-entered wait records nothing until then. A failed signal returns before custody, so its capture drops unrecorded. An engine already seen dead is recorded at once. Tests: through checkpoint_before_replacement, nothing is recorded while quarantined (two attempts), the observed exit records it once, and a re-entry after exit adds nothing. adopted_exit_wait_retains_custody_until_observed_exit now checks that no loss is recorded after a cancelled poll or a zero-budget timeout, and that the real child's exit records it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
… on her next cold turn (card 6b28a445) Line 6, every fast-path miss is loud, on top of #5348. A replacement now saves residents first, but three paths cannot, and they lost the KV with no record: checkpoint_before_replacement returned Ok when the generation had exited or paging was quarantined, and retire_adopted skipped the checkpoint when paging was quarantined and returned Ok when the adopted engine was already dead. The cold turn that followed said no_page or page_on_disk_unconfirmed, blaming the page store for a relaunch. Each of those paths now calls EndpointSlots::record_relaunch_discard with its reason (engine_exited, paging_uncertain, adopted_engine_dead). That records the minds held in a slot without a confirmed save, and probes inference.kv_page.discarded_by_relaunch with the count. The record lives on the endpoint because the pool is replaced with the engine. Every admission takes its own page's record, so only the first turn after the relaunch is blamed on it. A cold turn then reports cause=discarded_by_relaunch with discard_reason. The record is one entry per mind, replaced and never grown; forks die with their core and are not recorded, as in the ledger. New file slots/relaunch_discard.rs; test slots::tests::a_relaunch_records_only_unsaved_residents_and_their_next_turn_takes_it_once. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
…e is gone; retention is one pool of slots (card 6b28a445) Codex's review of #5357 at ab7788b: (1) The record was not bounded: minds discarded by one relaunch that never returned stayed forever, across generations. DiscardLog now keeps at most the latest discarding pool's slot count, oldest evicted first. One relaunch cannot discard more minds than it had slots, so the latest is kept whole, and an empty capture evicts nothing. (2) A loss was recorded before the retirement re-checked current(), re-verified the engine's identity or signalled it, so a superseded or refused retirement could report a loss that never happened. The residents are now captured as a PendingDiscard (must_use) while the old pool is installed, and committed only once the engine is actually gone. checkpoint_before_replacement returns the capture, and launch commits it after kill_child. retire_adopted commits after signal_if_same succeeds, or as adopted_engine_dead when the engine is seen dead. Every early return drops the capture unrecorded. Tests: the lifecycle path through checkpoint_before_replacement (a dropped capture records nothing, a committed one is taken once and survives the pool's replacement), and retention across three generations. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
…6b28a445) CI job 120805073062 at 769d5d7: discard_records_keep_one_pool_of_slots_across_generations asserted minds 1 and 2 were evicted after relaunches [1,2,3] then [4,5]. But the residents of one relaunch come from a HashMap, so their order within that relaunch is arbitrary, and which two were oldest was not defined. The test now evicts with one mind per relaunch ([1],[2],[3],[4] at three slots: 1 evicted), then checks that a whole pool's relaunch ([7,8,9]) is kept whole over an older one ([5,6]) and that an empty relaunch evicts nothing. The code is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
…hen the kill is sent (card 6b28a445) Codex's re-review of #5357 at 5d8e4c2: launch committed the discard right after kill_child (a void call that may only log a failed signal), and retire_adopted committed it right after signal_if_same, both before any exit was observed. A sent signal is not a death. - Owned engine: checkpoint_before_replacement hands its capture to the endpoint (PendingDiscard::on_exit_of), held for that generation (DiscardLog's watch) and recorded by EngineGeneration::observed_exit, or at once if the exit was already observed. A generation already watched keeps its first capture, so re-entry never records twice. launch.rs is back to canary's. - Adopted engine: the capture rides the retirement custody beside retiring_adopted (LlamaServerProcess::retiring_discard). await_adopted_exit records it only when the exact process is seen gone, so a timed-out, cancelled or re-entered wait records nothing until then. A failed signal returns before custody, so its capture drops unrecorded. An engine already seen dead is recorded at once. Tests: through checkpoint_before_replacement, nothing is recorded while quarantined (two attempts), the observed exit records it once, and a re-entry after exit adds nothing. adopted_exit_wait_retains_custody_until_observed_exit now checks that no loss is recorded after a cancelled poll or a zero-budget timeout, and that the real child's exit records it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
#5357's core job 120814431211 at bd3a1c1: module_wiring_guard listed PageFaultOnly (genome bus) and RecorderModule (genome local_manager), and #5367 (4ac5e64) deleted both with the old paging stacks. The two entries are deleted. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE
joelteply
force-pushed
the
6b28a445/discarded-by-relaunch
branch
from
October 24, 2026 07:38
bd3a1c1 to
84505ee
Compare
joelteply
added a commit
that referenced
this pull request
Oct 24, 2026
#5357's core job 120814431211 at bd3a1c1: module_wiring_guard listed PageFaultOnly (genome bus) and RecorderModule (genome local_manager), and #5367 (4ac5e64) deleted both with the old paging stacks. The two entries are deleted. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE (cherry picked from commit 84505ee)
joelteply
added a commit
that referenced
this pull request
Oct 24, 2026
* Restage incomplete held checkouts instead of exposing them to Hands * fix(workspace): embed transactional AIRC clone allocator * test(workspace): give bound activity a real checked-out repository * Two unwired-fixture entries named modules #5367 deleted (card 6b28a445) #5357's core job 120814431211 at bd3a1c1: module_wiring_guard listed PageFaultOnly (genome bus) and RecorderModule (genome local_manager), and #5367 (4ac5e64) deleted both with the old paging stacks. The two entries are deleted. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE (cherry picked from commit 84505ee) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Card 6b28a445, plan line 6 (every fast-path miss is loud). It builds on #5348 (c8011da). This serves fast minds: when a resident goes cold after a relaunch, the record says why, instead of blaming the page store.
The defect. A replacement now saves residents first (#5348), but three paths cannot save, and each lost the KV with no record:
checkpoint_before_replacementreturned Ok when the generation had already exited or paging was quarantined;retire_adoptedskipped the checkpoint when paging was quarantined;retire_adoptedreturned Ok when the adopted engine was already dead.The cold turn that followed reported
no_pageorpage_on_disk_unconfirmed.The change (new file
slots/relaunch_discard.rs):PendingDiscard. A sent kill or signal is not a death, so the capture is recorded only when the death is observed:checkpoint_before_replacementhands the capture to the endpoint, which holds it for that generation (DiscardLog's watch).EngineGeneration::observed_exitrecords it, or it is recorded at once if the exit was already observed. A generation that is already watched keeps its first capture, so re-entry never records twice.launch.rsis unchanged from canary.retiring_adopted(LlamaServerProcess::retiring_discard).await_adopted_exitrecords it only when the exact process is seen gone, so a timed-out, cancelled or re-entered wait records nothing until then. A failed signal returns before custody, so its capture is dropped unrecorded. An engine already seen dead is recorded at once.inference.kv_page.discarded_by_relaunchwith the reason (engine_exited,paging_uncertainoradopted_engine_dead) and the count.cause=discarded_by_relaunchanddiscard_reason.DiscardLog). It keeps at most the latest discarding pool's slot count, oldest evicted first. One relaunch can't discard more minds than it had slots, so the latest is kept whole, and an empty capture evicts nothing. Forks die with their core and are not recorded, as in the ledger.Tests:
slots::tests::a_relaunch_records_unsaved_residents_only_at_the_observed_exit_and_once: throughcheckpoint_before_replacementwith quarantined paging. Two attempts record nothing; the observed exit records only the unsaved mind, once; a re-entry after the exit adds nothing; the record survives the pool's replacement and is taken once.llama_server::adopted_retirement::tests::adopted_exit_wait_retains_custody_until_observed_exit(real child process): no loss is recorded after a cancelled poll or a zero-budget timeout, and the child's exit records it.slots::tests::discard_records_keep_one_pool_of_slots_across_generations.retire_adopted's failed-signal path, which needs a live-lane registry. There,signal_if_samereturns an error before custody is set, so the capture is dropped, by reading ofadopted_retirement.rs.No file touched is over 1,000 lines (turn_admission.rs ends at 875).
Also deleted: two
module_wiring_guardentries (PageFaultOnly,RecorderModule) naming modules that #5367 deleted. Every PR's core job failed on them.Measured
CI at 84505ee: engine job 120816176947 reports 1218 passed, 0 failed, with all three new tests ok by name. Core job 120816176934 reports 2582 passed, 0 failed, warnings 15 (baseline 15).
Source and CI only; nothing was built on IntelMac. After deploy (owner Cormac): on the next forced or dead-engine relaunch on any node,
inference.kv_page.discarded_by_relaunchnames the count, and the affected residents' cold turns carrycause=discarded_by_relaunch.🤖 Generated with Claude Code
https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE