Skip to content

[SL3.3b] Recover interrupted CLI launches under the sandbox lock - #1446

Merged
Pedro Henrique Penna (ppenna) merged 6 commits into
feature/nvx-refactorfrom
private/simple-1108-02e4-sl33b-20261110
Nov 11, 2026
Merged

Pedro Henrique Penna (ppenna) merged 6 commits into
feature/nvx-refactorfrom
private/simple-1108-02e4-sl33b-20261110

Conversation

@esaurez

@esaurez Enrique Saurez (esaurez) commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Scope and source

SL3.3b of #1108: recover an interrupted, uncommitted CLI launch under
the sandbox lock without signaling an unverified process or discarding
indeterminate evidence.

Current qualified head 971103c782175dee4963005cdb7d65f4c871fc59, tree
11329dc1c5faa5fd739b54f5e9f8a531de9ae52f, continues the captured published
head dd1623f81fed740fdb1d1d326f09f46cb5f438aa by normal fast-forward only.
The base remains feature/nvx-refactor; the own-delta fork is
db39a5d609eb6863f96a8c33f5e3fcaa71339e33.

The inherited db39a5d6..dd1623f8 binary/full-index patch still has SHA256
feafd8869217e9f70bf82abebed5614b30a2c0aeb992b0fe462575d8644b3c67.
The original consumer 8921c45162e38761f1dc5beb51042ee2e7218803, historical
input join b134e702dc97feb620dfa642d0dac1fa46c33246, their retained refs,
and their earlier handoffs remain intact. No old input aggregate or producer
commits are replayed. This update does not rewrite or retarget #1458, #1459
or #1460, require their adoption, or change native stack membership.

Review dispositions

  • 4531275326:
    one absolute caller recovery deadline now reaches both held-owner lookups
    and both platform probes. Exhaustion is rejected before probing; Linux
    backlog sleeps and Windows busy-pipe waits use only the remaining budget.
    Windows sub-millisecond budgets return TimedOut, never the zero
    server-default wait sentinel. Timeout remains an error and retains the
    journal and capability; it is not evidence that the VM is absent.
    Unrelated legacy probe callers keep their five-second per-call policy.
  • 4531275389:
    the shared Windows test helper disconnects only ERROR_NO_DATA from a
    closed pre-accept probe, then makes at most one reaccept. Other errors stay
    strict. The deterministic test closes its real probe before releasing
    acceptance, then exchanges data with the actual client.

Held-process, boot/PID/start-time and log-claim verification are preserved,
including the caller-capped one-second transient-claim recheck. Every other
byte of launch/recovery/termination/record/cleanup code is unchanged apart
from passing the deadline to endpoint_owner. Journal-last cleanup and
committed-runtime exclusion remain.

The independent review's introduced test-only findings were also resolved:
the duplicated-transport test keeps its original strict raw
ConnectNamedPipe/ERROR_PIPE_CONNECTED check, and the standalone stale
fixture has no server thread to abandon. Its exact-invoked owned subprocess
uses a non-inheritable kill-on-close job, an owned child handle and finite
ready/accept/exit waits. Actual connect failure, timeout and panic paths
preserve the primary cause and verify exit through a duplicated process
handle, not a reused PID.

No public API, persistent production lifecycle, schema, general retry
framework, guest or native-source change.

Measured size

Conservative added/changed nonblank production code; deletions and test
lines are separate. Comments and test files do not consume the core budget.

Range Core added/changed Core deleted Test added/changed Test deleted Counted files
This correction, dd1623f8..971103c7 66 16 500 17 7
Inherited, db39a5d6..dd1623f8 115 4 365 7 3
Whole PR, db39a5d6..971103c7 180 19 862 21 9

The correction estimate was 100 core lines, with a 200-line stop; the
original whole-PR stop remains 600. The exact nine-counted-file exception
was explicitly approved. The seven correction files are the direct
launch/liveness caller chain, the Linux and Windows bounded primitives and
their reexports, and the unsupported-platform declaration. Keeping the
legacy wrapper avoids changing unrelated callers or introducing a public
abstraction. There are 17 total source/test files in the whole PR.

Existing test changes

The final inventory reports eight modified existing assets, none retired
or relocated:

  • accept_control: the single stale closed-client disconnect/reaccept
    above; unexpected errors still fail.
  • an_interruption_keeps_the_journal_and_capability_without_publishing_runtime:
    the inherited change checks the same evidence and duplicate-launch
    refusal before reacquiring the lock, because reacquisition now recovers.
  • replacing_the_log_or_sandbox_never_proves_the_launch_exited,
    the_spawned_process_inherits_the_preparations_claim, and
    an_endpoint_owner_without_the_claimed_log_is_rejected_without_signaling_it:
    supply the bounded deadline without weakening their ownership assertions.
  • endpoint_ownership_is_rechecked_only_after_the_server_is_held:
    retains held-before-recheck assertions and also checks that both probes
    receive exactly the same absolute deadline.
  • missing_owners_and_lookup_or_hold_errors_are_not_exit_proofs:
    callback signatures receive the deadline; the missing-owner and original
    error-source assertions remain.
  • a_duplicated_transport_keeps_the_verified_connection_without_reconnecting:
    calls ConnectNamedPipe directly to preserve the strict existing
    connection check rather than permitting a replacement connection.

New ordinary cases cover exhausted first/second lookups, real Linux full
backlog and Windows busy-pipe budgets, retained recovery evidence/no signal,
the forced stale-probe race, and owned-fixture cleanup on connection error,
timeout and panic. The inventory's 45 added parsed assets include helpers,
types and methods, not 45 executed tests. The new intentionally ignored
stale_probe_accept_helper is explicitly invoked by exact name with
--ignored as owned subprocess infrastructure; the ordinary race and
failure-path tests are not ignored.

Exact-head software qualification

Both approved hosts checked the exact current head/tree above with real
private NVX and OpenVMM Git metadata and task-private mutable caches/targets.
scripts/nvx.py verify passed before Cargo, and Windows Cargo ran in the
verified MSVC environment.

  • Linux: 78 passed, 0 failed, 3 intentional helper entries ignored.
  • Windows: 96 passed, 0 failed, 4 intentional helper entries ignored.
    The ordinary stale-race and failure/timeout/panic teardown tests executed;
    their child helper was actually invoked, and held process handles proved
    cleanup.
  • On both: locked Rust 1.95.0 all-target/all-feature compilation; the five
    scoped Clippy configurations (all features, host library, no-default
    library, CLI, testing); Rust 1.89.0 MSRV all-target/all-feature checking;
    protocol-only and host runtime-adapter compilation on 1.95.0 and 1.89.0;
    NVX/adapter formatting, test layout, correction/whole-PR test inventories,
    diff checks and clean NVX/native source checks.

Independent native Opus 5.5/MAX/long code-and-tests review approved this
exact head after the introduced fixture findings were resolved. Native
configuration and actual provider usage were verified separately. This was
a read-only logical review, not test execution or formal GitHub approval.
Earlier dd1623f8, eaf42142, d9e315ea and 9c864b6a results remain tied
to their historical tuples, not substituted for the current passes.

OpenVMM remains 14dc4116ec7e57567633c4b7b07892aca3a58522; kernel and guest
inputs are unchanged. No actual VM/hardware matrix, OpenVMM/kernel/guest
rebuild, macOS target check, complete CI verdict or formal GitHub approval
is claimed. This PR remains Draft with its existing labels and feature
base. No Ready pulse, manual CI trigger, parent/child retarget, stack
mutation, merge, auto-merge or issue closure is performed. A GitHub Copilot
request is a separate step after public head/body readback; request
admission is not a review verdict.

@esaurez Enrique Saurez (esaurez) added the tests-modified Changes existing tests, test data, or test wiring label Nov 10, 2026
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

This pull request changes existing tests

It modifies or removes 8 existing test asset(s), listed below. Review each with care: a change to a test can hide a regression that the test caught. See doc/contribute.md#changes-to-existing-tests.

Test code

Change What Before After
modified helper fn accept_control in host::platform::windows::endpoint::tests nvx-rs/src/host/platform/windows/endpoint/tests.rs:27 nvx-rs/src/host/platform/windows/endpoint/tests.rs:34
modified test fn an_interruption_keeps_the_journal_and_capability_without_publishing_runtime in host::cli::store::tests::launch nvx-rs/src/host/cli/store/tests/launch.rs:166 nvx-rs/src/host/cli/store/tests/launch.rs:166
modified test fn replacing_the_log_or_sandbox_never_proves_the_launch_exited in host::cli::store::tests::liveness nvx-rs/src/host/cli/store/tests/liveness.rs:79 nvx-rs/src/host/cli/store/tests/liveness.rs:79
modified test fn the_spawned_process_inherits_the_preparations_claim in host::cli::store::tests::liveness nvx-rs/src/host/cli/store/tests/liveness.rs:103 nvx-rs/src/host/cli/store/tests/liveness.rs:107
modified test fn an_endpoint_owner_without_the_claimed_log_is_rejected_without_signaling_it in host::cli::store::tests::liveness nvx-rs/src/host/cli/store/tests/liveness.rs:153 nvx-rs/src/host/cli/store/tests/liveness.rs:159
modified test fn endpoint_ownership_is_rechecked_only_after_the_server_is_held in host::cli::store::tests::liveness nvx-rs/src/host/cli/store/tests/liveness.rs:199 nvx-rs/src/host/cli/store/tests/liveness.rs:212
modified test fn missing_owners_and_lookup_or_hold_errors_are_not_exit_proofs in host::cli::store::tests::liveness nvx-rs/src/host/cli/store/tests/liveness.rs:235 nvx-rs/src/host/cli/store/tests/liveness.rs:252
modified test fn a_duplicated_transport_keeps_the_verified_connection_without_reconnecting in host::platform::windows::endpoint::tests nvx-rs/src/host/platform/windows/endpoint/tests.rs:40 nvx-rs/src/host/platform/windows/endpoint/tests.rs:382

Relocated unchanged: 0; added: 45. Compared the merge base db39a5d609eb with 971103c78217.

Enrique Saurez and others added 2 commits November 10, 2026 13:01
Use verified endpoint and log claims for unrecorded launches and boot/PID/start identities for recorded processes. Require confirmed exit before removing capability and journal; retain indeterminate evidence and its refusal cause. Keep committed-runtime recovery separate and bounded claim rechecks intact.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: eb3cc7e0-f51e-45ed-b911-c7b214740e05
Copilot-Session: 6aaf63ff-0a40-44dd-bc07-f43344fb11e5
Port the required single-instance named-pipe disconnect helper and CLI-gated test re-exports. Repeated endpoint-owner verification must reconnect after each finished client; no production API or assertion changes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: eb3cc7e0-f51e-45ed-b911-c7b214740e05
Copilot-Session: 6aaf63ff-0a40-44dd-bc07-f43344fb11e5
@esaurez
Enrique Saurez (esaurez) force-pushed the private/simple-1108-02e4-sl33b-20261110 branch from 8921c45 to dd1623f Compare November 10, 2026 21:26
@esaurez
Enrique Saurez (esaurez) changed the base branch from private/simple-1108-02e4-sl33b-inputs-20261110 to feature/nvx-refactor November 10, 2026 21:26
@esaurez
Enrique Saurez (esaurez) requested a balanced review from Copilot November 10, 2026 22:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Recovery ownership probes can exceed the caller’s deadline and hold the lock for seconds, while an unhandled Windows pipe race makes recovery tests scheduling-dependent.

2 open issues
What changed in this PR

This PR adds interrupted-launch recovery to the CLI-owned sandbox store for SL3.3b of #1108. Recovery runs under the sandbox lock before further mutation, using existing process and log ownership checks while excluding committed runtimes.

Changes:

  • Add verified process termination and journal-last cleanup, retaining evidence when recovery cannot establish exit.
  • Invoke recovery during lock acquisition.
  • Add recovery tests and Windows named-pipe fixture support; adapt the existing interruption test.
File Description
nvx-rs/​src/​host/​platform/​windows/​tests.rs Re-export the CLI-gated pipe disconnect helper.
nvx-rs/​src/​host/​platform/​windows/​endpoint/​tests.rs Add a named-pipe disconnect helper for repeated probes.
nvx-rs/​src/​host/​platform/​tests.rs Expose the Windows helper to CLI tests.
nvx-rs/​src/​host/​cli/​store/​tests/​recovery.rs Cover recovery, identity checks, evidence retention, and cleanup.
nvx-rs/​src/​host/​cli/​store/​tests/​launch.rs Check interrupted-launch evidence before lock reacquisition triggers recovery.
nvx-rs/​src/​host/​cli/​store/​tests.rs Register the recovery test module.
nvx-rs/​src/​host/​cli/​store/​mod.rs Document recovery during lock acquisition.
nvx-rs/​src/​host/​cli/​store/​locked.rs Run recovery before returning the acquired lock.
nvx-rs/​src/​host/​cli/​store/​launch.rs Implement interrupted-launch recovery and verified termination.

Review effort: Balanced

Comment thread nvx-rs/src/host/cli/store/launch.rs Resolved
Enrique Saurez and others added 4 commits November 10, 2026 14:57
Share one recovery deadline across held endpoint-owner checks and platform retry
waits, retaining evidence on timeout without changing legacy five-second callers.
Reconnect only stale closed Windows test-pipe clients and cover both timeout and
connect/drop-before-accept behavior with focused software tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: daabb81a-bdcc-435a-b767-7c672945cccc
Keep the bounded primitive available to legacy platform wrappers, but expose its
new recovery caller only with CLI. Import it directly in Linux platform tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: daabb81a-bdcc-435a-b767-7c672945cccc
Keep the duplicate transport test's pre-existing connection check strict rather
than hiding a lost client behind recovery's stale-probe accept behavior. Limit
stale acceptance to one disconnect/reaccept and avoid scoped-join hangs when
the deterministic stale-probe test's real client fails.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: daabb81a-bdcc-435a-b767-7c672945cccc
Replace the standalone accept thread with an explicitly invoked owned test
subprocess. A non-inheritable kill-on-close job and held child handle bound
cleanup on connection errors, timeouts and panic without abandoning a thread.
Check the original client cause and a signaled held process handle after
teardown, before and after releasing the accept protocol.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: daabb81a-bdcc-435a-b767-7c672945cccc

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the sandbox’s interrupted-launch recovery changes.

2 open issues

Review effort: Balanced

@ppenna

Copy link
Copy Markdown
Contributor

Review-resolution verification for head 971103c782175dee4963005cdb7d65f4c871fc59:

  • Both review findings are already fixed in the published history. Added commit-backed replies and resolved the two threads after verifying their fixes are contained in this head.
  • CI run 42931852341 completed successfully: 37 successful checks, 27 skipped, no failures or pending checks. The Required status check job passed.
  • Inspected the exact-head Linux and Windows Rust job logs: exhausted ownership-probe budgets, busy-endpoint timeout/evidence retention/no signaling, Linux full-backlog deadlines, Windows busy-pipe deadlines, stale-probe acceptance, and owned-fixture failure/timeout/panic cleanup regressions all passed on their applicable platforms.
  • Local python scripts\check_test_layout.py: passed.
  • Local git diff --check: passed; worktree is clean and matches the published head.
  • Local python scripts\nvx.py verify: blocked because this session's OpenVMM submodule is not initialized. No local Cargo build or runtime test is claimed; platform validation above is from CI.

No additional source changes, commits, pushes, retargeting, or merge were needed. The PR remains ready for review.

@ppenna
Pedro Henrique Penna (ppenna) merged commit 34d5c66 into feature/nvx-refactor Nov 11, 2026
88 checks passed
@ppenna
Pedro Henrique Penna (ppenna) deleted the private/simple-1108-02e4-sl33b-20261110 branch November 11, 2026 00:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feat-nvx-refactor tests-modified Changes existing tests, test data, or test wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants