Skip to content

feat(deploy): add an opt-in PVM profile for Agent DX - #104

Open
hky1999 wants to merge 41 commits into
inclusionAI:developfrom
hky1999:feat/pvm-candidate-adx
Open

hky1999 wants to merge 41 commits into
inclusionAI:developfrom
hky1999:feat/pvm-candidate-adx

Conversation

@hky1999

@hky1999 hky1999 commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

Add an explicit PVM node profile for running AKernel sandboxes inside a Linux VM without exposed VMX/SVM. Image builds select the pinned PVM guest bundle; Standalone and Helm select the PVM backend; the public SDK forwards runtime="firecracker-pvm" through Agent DX.

This head includes current develop@cc9ff612, retaining its updated ADX/Execd release inputs and Redis lifecycle tests. The sandboxd gitlink is resolved to d72f3a5f435c656771b120456a88c9ba4a6885bb, which includes current sandboxd main and all PVM fixes. .gitmodules uses the canonical inclusionAI/sandboxd URL, and fetching that exact SHA from the canonical repository was verified.

Depends on inclusionAI/sandboxd#77. Merge sandboxd first; if its accepted SHA changes through squash/rebase, update this gitlink and qualify the resulting input before merging this PR.

The profile pins the promoted original Firecracker v1.16.1-akernel.4 bytes, the independent strict-TSC guest source, and the paired OOT host input with OSPKE and NX worker recovery fixes. Deployment documentation states the CPU, stock-kernel, module-pairing and boot prerequisites. runsc remains the default runtime.

Validation on exact head b7c62a3dae0b2179ade84068d6efda594fb0e45e:

  • Deployment script checks, 84 Standalone tests, build-helper tests, SDK 298 unit + 35 benchmark tests, ruff and mypy passed locally.
  • Complete Agent DX isolated L1 CI: all Standalone groups, OCI/Nydus checkpoint/reload, network/storage tests, and both pressure workloads passed with no skipped test coverage and empty final inventories.
  • Complete sandboxd PVM CI passed on the exact submodule SHA.

The dedicated CI runs in a bounded disposable Intel/Ubuntu L1 with VMX/SVM hidden. It validates the full resulting image and pinned guest payload. Other host ABIs, AMD, heterogeneous migration and a real Kubernetes rollout remain outside this qualification.

hky1999 added 30 commits October 7, 2026 16:03
Select a distinct PVM runtime in standalone and Helm configurations,
validate the guest bundle before node preparation, and accept only
complete checksum-pinned candidate bundle overrides during image builds.

Document dedicated host prerequisites, backend and TSC snapshot gates,
rollback, and the validation required before promoting release pins.

Signed-off-by: hky1999 <hukeyang.hky@antgroup.com>
Exercise firecracker-pvm alongside the existing free-form runtime case
through Sandbox normalization and the YuanRong backend adapter.

Signed-off-by: hky1999 <hukeyang.hky@antgroup.com>
Use the reviewed PVM runtime and complete E2E entry points. Document that
backend modules remain fixed for each daemon lifetime and require draining
VMMs and restarting sandboxd when the host backend changes.

The feature gitlink depends on the sandboxd PVM PR. Replace it with the
merged upstream revision if squash merging changes the commit identity.
Keep production runtime bundle pins unchanged during candidate validation.

Signed-off-by: hky1999 <hukeyang.hky@antgroup.com>
Pin sandboxd to the reviewed PVM backend revision that rejects requests
whose guest-agent deadline has already expired. Include startup phase
timings and bounded console diagnostics for follow-up qualification.

Signed-off-by: hky1999 <hukeyang.hky@antgroup.com>
Include legacy IPv4 and IPv6 filter tables and connmark support in the
resolved PVM host configuration. The standalone startup script requires
these modules before creating its node container.

Document installing the full matching module tree. Rebuild and boot the
isolated PVM host, load all 14 standalone prerequisites, and start the
complete node and Traefik deployment successfully.

Signed-off-by: hky1999 <hukeyang.hky@antgroup.com>
Exercise network policy replacement with image-backed sandboxes while
checking that SDK command execution remains available. Allow integration
runs to use a private HTTP target and preinstalled curl so isolated PVM
nodes can qualify OCI and Nydus roots without package mirror access.

Document runtime selection, immutable qualification images, and the RAFS
v5 requirement. The integration suite passed all eight cases separately
against OCI and Nydus roots on the isolated PVM standalone node.

Signed-off-by: hky1999 <hukeyang.hky@antgroup.com>
Document the host networking and guest MPK configuration requirements,
and include the separately applied nested DEBUGCTL reference patch.
Explain its hardware-backend state requirements and the distinction
between prior microbenchmarks and the unpatched SDK qualification.

Advance the sandboxd gitlink to include its deployment documentation;
the runtime implementation remains the previously validated revision.

Signed-off-by: hky1999 <hukeyang.hky@antgroup.com>
Pin the conflict-resolved sandboxd revision that retains the PVM backend
and incorporates current main DNS and OOM/deletion fixes. The merged
sandboxd source passes format checks, vet and its complete Go test suite.

Earlier isolated PVM execution evidence remains tied to its recorded
implementation revision; release candidates still need exact-byte tests.

Signed-off-by: hky1999 <hukeyang.hky@antgroup.com>
Advance the sandboxd gitlink to keep its optional live ABI probe compatible
with unprivileged CI runners. The production PVM implementation is
unchanged from the conflict-resolved source pinned previously.

Signed-off-by: hky1999 <hukeyang.hky@antgroup.com>
Pin the reviewed OOT host-module source separately from the PVM guest
source and remove the obsolete custom-host configuration. Document
build/load/rollback without rebuilding the host kernel, including L1
hosts that expose no VMX/SVM. Keep the optional DEBUGCTL change scoped
to rebuilding OOT modules and advance sandboxd to its matching guidance.

Signed-off-by: hky1999 <hukeyang.hky@antgroup.com>
Retain architecture-specific artifact pins, build platform checks and
standalone preflight ordering from main. PVM remains an amd64 profile
and consumes sandboxd manifest pins rather than caller overrides.

Advance sandboxd to the merged PVM branch so the node image includes
both the backend implementation and the dual-architecture changes.

Signed-off-by: hky1999 <hukeyang.hky@antgroup.com>
Pin the reorganized PVM OOT source and its separate native baseline.
Prepare and verify the Ubuntu target profile before building and loading
the matched modules, and apply the optional DEBUGCTL patch after target
preparation. Keep guest source pins and runtime interfaces unchanged.

Signed-off-by: hky1999 <hukeyang.hky@antgroup.com>
Initialize the backend selector in function-only launcher fixtures, as
the real entrypoint does. Check invalid selectors before data mutation
and preserve direct resolver configuration when selecting PVM.

Signed-off-by: hky1999 <hukeyang.hky@antgroup.com>
Keep the source standalone lifecycle and opt-in build mirrors from main
alongside the PVM profile. SDK, PVM pins, launch/config generation and
node entrypoint bytes remain identical to the qualified runtime head.

Validate the merged build helpers, apt mirror handling, deployment
syntax and standalone lifecycle, including optional shutdown identities.

Signed-off-by: hky1999 <hukeyang.hky@antgroup.com>
Pin the requalified OOT baseline and Ubuntu target profile, and consume
the matching sandboxd fork revision. Document host nopku support, the
unchanged guest TSC boundary and the need for dedicated isolated PVM CI.
Keep guest release pins and the optional DEBUGCTL patch unchanged.

Signed-off-by: hky1999 <hukeyang.hky@antgroup.com>
Link the immutable strict-TSC guest and Firecracker fork source pins,
and update sandboxd to the fork revision documenting that candidate.
Keep the OSPKE-fixed host source and optional DEBUGCTL patch unchanged.

Record boot-parameter precedence, Full/Diff integrity, normal/nopku
host modes, same-frequency TSC-domain changes and OCI/Nydus SDK recovery.
Distinguish manual guest validation from a promoted runtime bundle,
Agent DX E2E and CI or full pressure qualification. Preserve ordinary
KVM releases and all checkpoint compatibility gates.

Signed-off-by: hky1999 <hukeyang.hky@antgroup.com>
Merge Agent DX deployment, SDK and ingress changes while retaining the
opt-in PVM node profile and the fork sandboxd source. Resolve retired
YuanRong configuration without changing ADX's control/data listeners.

Select the independently pinned PVM Actions archive through sandboxd's
verified staging helper. Keep ordinary KVM releases, host-module inputs
and DEBUGCTL defaults unchanged. Update build wiring and deployment docs.

Validate SDK unit tests and build/standalone/syntax helper suites before
running the exact final image inside the bounded disposable PVM L1.

Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
Add an independent fork-only workflow that validates the disposable L1,
builds the image from the locked PVM Actions archive and checks the final
VMM, guest, config and initrd payload. Exercise real ADX control/data
listeners, SDK lifecycle, OCI/Nydus recovery and bounded pressure.

Keep fixture services and module operations inside the owned L1 and
collect evidence without credentials. Make maintained networking/storage
examples honor the selected runtime; retain the runsc-specific distinct
storage-limit case and unchanged ordinary defaults.

Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
Keep Redis and control state on the L1 root disk, with runtime data and
checkpoints bound to the separate XFS disk. Validate and unmount only
owned bind mounts during teardown.

Give allowed and blocked DNS fixtures different IP addresses so the
traffic allowlist test can distinguish its permitted peer. Remove the
owned documentation address after the job.

Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
Select a separate ADX profile for PVM with a 90-second coordinator and
adxlet RPC budget. Keep the ordinary KVM profile and the 15-second guest
startup budget unchanged. Document the qualified nested-L1 deletion and
snapshot timings, and isolate control persistence from snapshot writes.

Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
Retain the 2 GiB SDK resource request and the admission pressure guard.
Exercise the supported dirty-page checkpoint path explicitly. Preserve
the large Full-snapshot timings and rejected reload as an unresolved
resource-pressure limitation, alongside separate Full runtime coverage.

Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
Parameterize only checkpoint-test memory, retaining the 2 GiB default
and the existing functional assertions. The isolated L1 uses 512 MiB and
records that size in qualification evidence.

Remove the ineffective Incremental default: ADX explicitly requests
Full. Preserve the failed 2 GiB reload and missing snapshot-type
selection as limitations, without disabling admission pressure.

Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
The standalone default uses a separate ext4 loop-backed filestore,
which was outside the previous runtime bind. Bind the real filestore and
checkpoint paths to XFS, disable the loop wrapper, and keep sandboxd
metadata and control persistence on ext4. Recheck the original 2 GiB SDK
checkpoint request with explicit storage evidence.

Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
Run the PVM qualification without initializing unrelated runsc handlers.
Preserve the observed mixed-runtime health failure as a limitation.
Continue Nydus and pressure phases after earlier failures while requiring
all phase results, inventories and cleanup checks for a passing job.

Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
Record the successful dedicated PVM runtime and SDK/OCI/Nydus pressure
runs, exact candidate/image/agent digests and cross-TSC-domain results.
Preserve mixed-runtime readiness, explicit Full selection and nested
snapshot-latency limitations. This commit changes documentation only;
the runtime qualification remains bound to source 433ffe5.

Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
…nosis

Advance sandboxd to bounded per-runtime inventory health and detailed
checkpoint timing diagnostics. Record the inherited 20 MiB/s backing I/O
limits, matched KVM/PVM Full restore results and memory-backed controls.

Document successful SDK lifecycle traffic while runsc is degraded, its
admission failure and subsequent recovery. Keep genuine core-storage
health checks, guest startup deadlines and optional DEBUGCTL unchanged.

Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
Include the upstream ADX Environment API and TLS changes and the current
runc lifecycle regressions before rebuilding and qualifying PVM. Resolve
the sandboxd gitlink to the fork revision containing both the upstream
fixes and PVM runtime health isolation. Keep DEBUGCTL optional.

Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
Advance the sandboxd fork input with current parent memory accounting and
independent candidate/release bundle verification. Qualify the canonical
Firecracker develop archive through the complete isolated PVM Agent DX job.

Document the unchanged runtime payloads and keep public release downloads
independent of Actions credentials. Preserve fork-only development, the
ordinary KVM pins and optional DEBUGCTL module policy.

Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
The parent checkout does not remove ignored files within a separately
initialized sandboxd submodule. A prewarmed L1 can therefore retain the
previous candidate archive after the pinned source changes, correctly
failing checksum verification before the node build starts.

Clean that disposable CI submodule's ignored build inputs after checkout.
Preserve strict archive checks and ordinary developer checkout behavior.

Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
Backport the NX recovery-worker retry fix into the host source profile and
advance sandboxd to cancel startup and restore I/O when the VMM exits.
Keep the target adaptation, guest runtime bundle and optional DEBUGCTL
optimization separate, with the new module pair requiring qualification.

Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
Advance sandboxd so a completed successful housekeeping pass retains
admission after slow checkpoint writeback. Genuine local reconciliation
errors still close readiness and backend-specific inventory gates remain
unchanged.

Qualify the fixed source with the existing PVM guest candidate and NX-fixed
host modules through the full isolated Agent DX workflow.

Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
Advance sandboxd to the public v1.16.1-akernel.4 archive while preserving
the qualified candidate digest and original build provenance. Record the
unchanged-byte promotion and credential-free download verification.

Keep ordinary KVM artifacts and the NX-fixed host-module profile unchanged.
Build, qualify and export the final clean PVM node from this revision.

Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
Consume the updated ADX release inputs and upstream lifecycle checks.
Resolve the sandboxd gitlink to the PVM runtime merged with current main,
and restore the canonical submodule URL for upstream review.

Keep the published PVM guest bytes and qualified host-module inputs
unchanged. Requalify this combined source before opening new PRs.

Signed-off-by: 月可 <hukeyang.hky@antgroup.com>

rcyw commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

Standalone review of b7c62a3dae0b2179ade84068d6efda594fb0e45e, with the exact canonical sandboxd gitlink d72f3a5f435c656771b120456a88c9ba4a6885bb and ADX/Execd af7d82f04a52d4604ea7ca4c550f09c6114dee4d.

Independent validation completed on this combination:

  • SDK 298 unit + 35 benchmark tests, lint/types, 84 Standalone tests, and the build/deployment helper checks passed.
  • Built the standard amd64 PVM image with Kata/runc disabled. Checked 28 actual payload files: PVM VMM/kernel/config matched the verified public archive; ADX binaries matched the pinned platform manifest; both the installed Execd and the copy extracted from the actual EROFS runtime rootfs matched the pinned Execd archive. The initrd came from the selected sandboxd source.
  • Used the source SDK with the declared ADX backend and TLS verification against an isolated deployment. runsc command/file/resource/delete controls passed; ordinary hardware KVM correctly refused PVM with no silent fallback. Logical and physical cleanup were checked separately, followed by release of the owned container/network/backing loop/ports.

There was no disposable PVM L1 available locally. Actual PVM boot and checkpoint/reload remain covered by the exact-head author L1 CI, whose downloaded ABI/summary artifacts were checked separately. This does not establish another host ABI or Kubernetes rollout.

For long-term adoption, I would keep the current explicit experimental profile and close two ownership/regression gaps:

  1. Provide an upstream qualification entry point. This job condition and the sandboxd equivalent hard-code the author's fork and feature branch, so upstream dispatch will skip even after merge. A maintainer-controlled dispatch against allowed refs, an owned disposable-L1 runner, and a versioned provisioning/fixture manifest would make this repeatable. Preserve the private Docker socket, identity checks, resource bounds and cleanup gates when moving that ownership upstream.
  2. Make the host profile a maintained release input. The existing CPU/boot/ABI restrictions are already documented. Before broadening the support claim, assign ownership for the OOT KVM source, target-profile/module-pair rebuilds, and security updates, and make qualification compare those committed inputs with the L1 build evidence. Updating the stock kernel does not by itself update the forked OOT KVM source. The concrete missing source-pin check is described in the companion sandboxd review.

The declared sandboxd-first merge order is appropriate: qualify the final accepted sandboxd SHA/gitlink when it changes. Reuse unchanged, version-matched evidence and rerun the affected checks, rather than treating every merge as a reason to repeat the entire previous Standalone/macOS matrix. These follow-ups concern sustainable qualification and support ownership; the local checks did not identify a new blocking runtime-function defect.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants