Repository navigation
[SL3.3b] Recover interrupted CLI launches under the sandbox lock - #1446
Pedro Henrique Penna (ppenna) merged 6 commits into
Conversation
This pull request changes existing testsIt modifies or removes 8 existing test asset(s), listed below. Review each with care: a change to a test can hide a regression that the test caught. See doc/contribute.md#changes-to-existing-tests. Test code
Relocated unchanged: 0; added: 45. Compared the merge base |
Use verified endpoint and log claims for unrecorded launches and boot/PID/start identities for recorded processes. Require confirmed exit before removing capability and journal; retain indeterminate evidence and its refusal cause. Keep committed-runtime recovery separate and bounded claim rechecks intact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3cc7e0-f51e-45ed-b911-c7b214740e05 Copilot-Session: 6aaf63ff-0a40-44dd-bc07-f43344fb11e5
Port the required single-instance named-pipe disconnect helper and CLI-gated test re-exports. Repeated endpoint-owner verification must reconnect after each finished client; no production API or assertion changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3cc7e0-f51e-45ed-b911-c7b214740e05 Copilot-Session: 6aaf63ff-0a40-44dd-bc07-f43344fb11e5
8921c45 to
dd1623f
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Recovery ownership probes can exceed the caller’s deadline and hold the lock for seconds, while an unhandled Windows pipe race makes recovery tests scheduling-dependent.
2 open issues
What changed in this PR
This PR adds interrupted-launch recovery to the CLI-owned sandbox store for SL3.3b of #1108. Recovery runs under the sandbox lock before further mutation, using existing process and log ownership checks while excluding committed runtimes.
Changes:
- Add verified process termination and journal-last cleanup, retaining evidence when recovery cannot establish exit.
- Invoke recovery during lock acquisition.
- Add recovery tests and Windows named-pipe fixture support; adapt the existing interruption test.
| File | Description |
|---|---|
nvx-rs/src/host/platform/windows/tests.rs |
Re-export the CLI-gated pipe disconnect helper. |
nvx-rs/src/host/platform/windows/endpoint/tests.rs |
Add a named-pipe disconnect helper for repeated probes. |
nvx-rs/src/host/platform/tests.rs |
Expose the Windows helper to CLI tests. |
nvx-rs/src/host/cli/store/tests/recovery.rs |
Cover recovery, identity checks, evidence retention, and cleanup. |
nvx-rs/src/host/cli/store/tests/launch.rs |
Check interrupted-launch evidence before lock reacquisition triggers recovery. |
nvx-rs/src/host/cli/store/tests.rs |
Register the recovery test module. |
nvx-rs/src/host/cli/store/mod.rs |
Document recovery during lock acquisition. |
nvx-rs/src/host/cli/store/locked.rs |
Run recovery before returning the acquired lock. |
nvx-rs/src/host/cli/store/launch.rs |
Implement interrupted-launch recovery and verified termination. |
Review effort: Balanced
Share one recovery deadline across held endpoint-owner checks and platform retry waits, retaining evidence on timeout without changing legacy five-second callers. Reconnect only stale closed Windows test-pipe clients and cover both timeout and connect/drop-before-accept behavior with focused software tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: daabb81a-bdcc-435a-b767-7c672945cccc
Keep the bounded primitive available to legacy platform wrappers, but expose its new recovery caller only with CLI. Import it directly in Linux platform tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: daabb81a-bdcc-435a-b767-7c672945cccc
Keep the duplicate transport test's pre-existing connection check strict rather than hiding a lost client behind recovery's stale-probe accept behavior. Limit stale acceptance to one disconnect/reaccept and avoid scoped-join hangs when the deterministic stale-probe test's real client fails. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: daabb81a-bdcc-435a-b767-7c672945cccc
Replace the standalone accept thread with an explicitly invoked owned test subprocess. A non-inheritable kill-on-close job and held child handle bound cleanup on connection errors, timeouts and panic without abandoning a thread. Check the original client cause and a signaled held process handle after teardown, before and after releasing the accept protocol. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: daabb81a-bdcc-435a-b767-7c672945cccc
|
Review-resolution verification for head
No additional source changes, commits, pushes, retargeting, or merge were needed. The PR remains ready for review. |
34d5c66
into
feature/nvx-refactor

Scope and source
SL3.3b of #1108: recover an interrupted, uncommitted CLI launch under
the sandbox lock without signaling an unverified process or discarding
indeterminate evidence.
Current qualified head
971103c782175dee4963005cdb7d65f4c871fc59, tree11329dc1c5faa5fd739b54f5e9f8a531de9ae52f, continues the captured publishedhead
dd1623f81fed740fdb1d1d326f09f46cb5f438aaby normal fast-forward only.The base remains
feature/nvx-refactor; the own-delta fork isdb39a5d609eb6863f96a8c33f5e3fcaa71339e33.The inherited
db39a5d6..dd1623f8binary/full-index patch still has SHA256feafd8869217e9f70bf82abebed5614b30a2c0aeb992b0fe462575d8644b3c67.The original consumer
8921c45162e38761f1dc5beb51042ee2e7218803, historicalinput join
b134e702dc97feb620dfa642d0dac1fa46c33246, their retained refs,and their earlier handoffs remain intact. No old input aggregate or producer
commits are replayed. This update does not rewrite or retarget #1458, #1459
or #1460, require their adoption, or change native stack membership.
Review dispositions
one absolute caller recovery deadline now reaches both held-owner lookups
and both platform probes. Exhaustion is rejected before probing; Linux
backlog sleeps and Windows busy-pipe waits use only the remaining budget.
Windows sub-millisecond budgets return
TimedOut, never the zeroserver-default wait sentinel. Timeout remains an error and retains the
journal and capability; it is not evidence that the VM is absent.
Unrelated legacy probe callers keep their five-second per-call policy.
the shared Windows test helper disconnects only
ERROR_NO_DATAfrom aclosed pre-accept probe, then makes at most one reaccept. Other errors stay
strict. The deterministic test closes its real probe before releasing
acceptance, then exchanges data with the actual client.
Held-process, boot/PID/start-time and log-claim verification are preserved,
including the caller-capped one-second transient-claim recheck. Every other
byte of launch/recovery/termination/record/cleanup code is unchanged apart
from passing the deadline to
endpoint_owner. Journal-last cleanup andcommitted-runtime exclusion remain.
The independent review's introduced test-only findings were also resolved:
the duplicated-transport test keeps its original strict raw
ConnectNamedPipe/ERROR_PIPE_CONNECTEDcheck, and the standalone stalefixture has no server thread to abandon. Its exact-invoked owned subprocess
uses a non-inheritable kill-on-close job, an owned child handle and finite
ready/accept/exit waits. Actual connect failure, timeout and panic paths
preserve the primary cause and verify exit through a duplicated process
handle, not a reused PID.
No public API, persistent production lifecycle, schema, general retry
framework, guest or native-source change.
Measured size
Conservative added/changed nonblank production code; deletions and test
lines are separate. Comments and test files do not consume the core budget.
dd1623f8..971103c7db39a5d6..dd1623f8db39a5d6..971103c7The correction estimate was 100 core lines, with a 200-line stop; the
original whole-PR stop remains 600. The exact nine-counted-file exception
was explicitly approved. The seven correction files are the direct
launch/liveness caller chain, the Linux and Windows bounded primitives and
their reexports, and the unsupported-platform declaration. Keeping the
legacy wrapper avoids changing unrelated callers or introducing a public
abstraction. There are 17 total source/test files in the whole PR.
Existing test changes
The final inventory reports eight modified existing assets, none retired
or relocated:
accept_control: the single stale closed-client disconnect/reacceptabove; unexpected errors still fail.
an_interruption_keeps_the_journal_and_capability_without_publishing_runtime:the inherited change checks the same evidence and duplicate-launch
refusal before reacquiring the lock, because reacquisition now recovers.
replacing_the_log_or_sandbox_never_proves_the_launch_exited,the_spawned_process_inherits_the_preparations_claim, andan_endpoint_owner_without_the_claimed_log_is_rejected_without_signaling_it:supply the bounded deadline without weakening their ownership assertions.
endpoint_ownership_is_rechecked_only_after_the_server_is_held:retains held-before-recheck assertions and also checks that both probes
receive exactly the same absolute deadline.
missing_owners_and_lookup_or_hold_errors_are_not_exit_proofs:callback signatures receive the deadline; the missing-owner and original
error-source assertions remain.
a_duplicated_transport_keeps_the_verified_connection_without_reconnecting:calls
ConnectNamedPipedirectly to preserve the strict existingconnection check rather than permitting a replacement connection.
New ordinary cases cover exhausted first/second lookups, real Linux full
backlog and Windows busy-pipe budgets, retained recovery evidence/no signal,
the forced stale-probe race, and owned-fixture cleanup on connection error,
timeout and panic. The inventory's 45 added parsed assets include helpers,
types and methods, not 45 executed tests. The new intentionally ignored
stale_probe_accept_helperis explicitly invoked by exact name with--ignoredas owned subprocess infrastructure; the ordinary race andfailure-path tests are not ignored.
Exact-head software qualification
Both approved hosts checked the exact current head/tree above with real
private NVX and OpenVMM Git metadata and task-private mutable caches/targets.
scripts/nvx.py verifypassed before Cargo, and Windows Cargo ran in theverified MSVC environment.
The ordinary stale-race and failure/timeout/panic teardown tests executed;
their child helper was actually invoked, and held process handles proved
cleanup.
scoped Clippy configurations (all features, host library, no-default
library, CLI, testing); Rust 1.89.0 MSRV all-target/all-feature checking;
protocol-only and host runtime-adapter compilation on 1.95.0 and 1.89.0;
NVX/adapter formatting, test layout, correction/whole-PR test inventories,
diff checks and clean NVX/native source checks.
Independent native Opus 5.5/MAX/long code-and-tests review approved this
exact head after the introduced fixture findings were resolved. Native
configuration and actual provider usage were verified separately. This was
a read-only logical review, not test execution or formal GitHub approval.
Earlier
dd1623f8,eaf42142,d9e315eaand9c864b6aresults remain tiedto their historical tuples, not substituted for the current passes.
OpenVMM remains
14dc4116ec7e57567633c4b7b07892aca3a58522; kernel and guestinputs are unchanged. No actual VM/hardware matrix, OpenVMM/kernel/guest
rebuild, macOS target check, complete CI verdict or formal GitHub approval
is claimed. This PR remains Draft with its existing labels and feature
base. No Ready pulse, manual CI trigger, parent/child retarget, stack
mutation, merge, auto-merge or issue closure is performed. A GitHub Copilot
request is a separate step after public head/body readback; request
admission is not a review verdict.