[WenYuLuo]

AKernel's ADX adapter and probes need the Environment SDK/API exposed by the refreshed platform release. This change migrates the internal integration to adx-environment==0.1.0rc1 and /api/environment/v1 while preserving the public akernel_sdk.Sandbox API. It also exposes TLS certificate verification and applies the policy consistently across SDK and CLI connections.

Related to #97. This PR delivers configurable opt-in verification; the compatibility default remains off. The secure-default recommendation remains tracked in #97.

• Adapt Environment handles, errors and network-policy fields; update resource queries, CLI deletion, benchmark clients and E2E fixtures.
• Pin the AMD64 platform/Execd archives and src/agent-dx to the latest refactor@ce80a189f352f1ee2ebb6a9f30ee218f2ae896d3, the updated v0.1.0rc1 tag and verified ADX Buildkite #161. The same archive bytes are published in the GitCode release. Defaults use immutable OBS URLs with SHA-256 verification.
• Add AKERNEL_VERIFY_TLS and propagate it through control, command/file, command-watch and PTY connections. CLI inventory/deletion and SDK resource queries share the TLS context policy. Verification defaults to off; AKERNEL_VERIFY_TLS=1 enables chain and hostname validation, and SSL_CERT_FILE selects private trust.
• Add global CLI --verify-tls, --no-verify-tls and --ca-file options. --ca-file enables verification; incompatible options and invalid environment values fail explicitly. Document trust configuration and simplify installation/deployment guidance.
• Document the transport trust-source contract: HTTPX explicitly loads a selected SSL_CERT_FILE ahead of SSL_CERT_DIR, rejecting a missing/malformed file. CLI inventory/deletion, resources, PTY and command-watch use Python SSL/OpenSSL default trust paths, which can validate through a usable CA directory despite an invalid file. --ca-file sets the process variable and enables verification; it does not select an exclusive trust store. Certificate-chain and hostname verification remain enabled when opted in. Trust configuration is set before first use.

Validation:

• Trust-source contract follow-up: all 14 real TLS tests and the complete 291 SDK unit tests, 35 benchmark-driver tests, Ruff and mypy (22 files) passed on Linux using the bundled #161 SDK. Missing and malformed CA files were each exercised with a usable hashed CA directory: HTTPX rejected the file at client construction; verified CLI/resources and PTY/command-watch connected through the directory; wrong-host certificates were still rejected. Existing unset/default-unverified tests also passed.

• The 14 real HTTPS/WSS TLS tests also cover configured trusted CA acceptance, untrusted/wrong-host rejection, default and explicit unverified behavior, SDK control/command/file, CLI/resource queries, PTY and command-watch. The current Python 3.10–3.14 CI unit-test jobs passed. Deployment syntax and 82 standalone-helper tests plus 61 build/release/Helm/Terraform contract tests passed on the artifact-pin baseline; this follow-up changes only documentation and TLS regression tests.

• Fresh Linux AMD64 standalone/runsc deployment used the publicly downloaded #161 platform/Execd archives, with archive/manifests/file hashes and installed binaries checked against ce80a189. Actual deployed TLS acceptance passed: untrusted verified access was rejected, while configuring the deployment CA enabled resource queries, commands, files, command-watch and PTY.

• All 18/18 E2E groups passed in one campaign: 13 standalone and five selected full groups, 45 unittest cases plus two script groups, zero skips. Coverage includes admission, concurrent claims, lost create result, create transport disconnect and tenant isolation. Both Redis audits report cluster_clean=true; final held resources and physical instances are zero. The task container was removed and the pre-existing service preserved.

PyPI adx-environment==0.1.0rc1 remains the previously published #160 candidate. The #161 SDK changes only docstrings; runtime AST and package dependency/version metadata are unchanged. This deployment regression installs the exact #161 bundled wheel; no PyPI overwrite or new version was performed.

CI on the prior TLS head passed all ten required jobs, including standalone E2E. The artifact-pin head CI also passed all ten required checks, including standalone E2E. Kubernetes runtime, Kata/Firecracker/runc/GPU acceptance are outside this AKernel deployment run.

Current trust-contract head CI passed all ten required checks, including Python 3.10–3.14 tests and standalone E2E; Docker Hub publication was skipped.

[rcyw]

Review of HEAD 8949111920834f6fc4ce5605952ae425e4e6be95 (rechecked before posting). The ten required CI checks pass, including Standalone E2E; Docker Hub publication is skipped. Please keep this PR unmerged for now while the P2 below is fixed and the #97 acceptance difference is resolved explicitly.

P2 — Explicit CA-file errors are not rejected consistently

Locations: sdk/python/akernel_sdk/_tls.py:23–24 and cli.py:574–578.

--ca-file assigns SSL_CERT_FILE, but the shared stdlib context calls ssl.create_default_context() without an explicit cafile. OpenSSL's default trust-path loading can ignore a nonexistent certificate file and still trust the server through a CA directory. The native SDK's HTTPX client fails immediately under the same configuration.

Observed on Python 3.12.3 / OpenSSL 3.0.13, this PR's editable SDK, adx-environment==0.1.0rc1 and httpx==0.28.1:

AKERNEL_VERIFY_TLS=1
SSL_CERT_FILE=<nonexistent file>
SSL_CERT_DIR=<hashed directory containing the test server's CA>

CLI GET: HTTP 200; check_hostname=True; verify_mode=CERT_REQUIRED
Native EnvironmentClient construction: FileNotFoundError

This is a trust-configuration failure-handling inconsistency, not a fallback to verify=False. A typo in an explicit CA-file selection is masked on CLI inventory/deletion and resource-query paths. The pinned dependency's PTY and command-watch contexts also use parameterless ssl.create_default_context() (source inspection; the HTTP discrepancy above was reproduced).

Reproduce from this HEAD with SDK/dev dependencies and OpenSSL installed. This uses the checked-in loopback TLS fixture and temporary test certificates; it needs no deployment or real credentials:

PYTHONPATH=sdk/python python3 - <<'PY'
import os, shutil, ssl, subprocess
from unittest.mock import patch
from tests.unit.test_tls import TLSHandshakeTest
from akernel_sdk import cli
from adx_environment._transport import EnvironmentClient
from adx_environment.types import ConnectionConfig

TLSHandshakeTest.setUpClass()
test = TLSHandshakeTest()
try:
    ca_dir = test.root / "ca-dir"
    ca_dir.mkdir()
    shutil.copy(test.root / "ca.pem", ca_dir / "trusted.pem")
    subprocess.run(["openssl", "rehash", str(ca_dir)], check=True)
    env = {
        "AKERNEL_VERIFY_TLS": "1",
        "SSL_CERT_FILE": str(test.root / "missing.pem"),
        "SSL_CERT_DIR": str(ca_dir),
        "NO_PROXY": "*",
    }
    with test._https("valid") as port, patch.dict(os.environ, env, clear=True):
        context = cli._create_ssl_context()
        result = cli._make_get_request(
            f"https://127.0.0.1:{port}/api/instances", "test-token", context
        )
        print("CLI:", result["status"], context.check_hostname,
              context.verify_mode == ssl.CERT_REQUIRED)
        try:
            client = EnvironmentClient(connection=ConnectionConfig(
                server_address=f"127.0.0.1:{port}", token="test-token",
                use_tls=True, verify_tls=True,
            ))
        except Exception as error:
            print("SDK:", type(error).__name__)
        else:
            client.close()
            raise AssertionError("Expected configured-file validation to fail")
finally:
    TLSHandshakeTest.doClassCleanups()
PY

Current output: CLI: 200 True True, then SDK: FileNotFoundError.

Suggested repair: validate and explicitly load any selected CA file before network traffic; define CA-file/CA-directory precedence and keep it consistent across control/command/file HTTPX, CLI/resources, PTY and command-watch. For dependency-owned WebSocket contexts, coordinate the equivalent trust-loading change or validate configuration on the AKernel side before creating them. Add missing/malformed CA-file regression cases while another usable trust directory is present; fail as configuration errors instead of silently using the other trust source.

#97 — Default verification remains an acceptance difference

_tls.py:12–16 still defaults to verification off, as documented in the SDK README. This is an explicit compatibility decision and existing behavior, rather than a new regression. It does not satisfy the #97 recommendation that insecure development mode require explicit opt-in.

Local HTTPS/WSS tests passed trusted-CA acceptance and untrusted/wrong-host rejection, including nine additional command/file/CLI DELETE handshake cases. No retry with verification disabled was found. This establishes opt-in verification, not a secure default.

Suggested resolution: enable verification by default with an explicit development opt-out, or explicitly agree to the compatibility exception and retain the remaining secure-default requirement in issue tracking. The current Closes #97 wording should not imply that this acceptance requirement is already met.

#96 — Integrated, but real workload acceptance remains outstanding

The latest PR no longer pins 0a5dea4c: the gitlink and the actual downloaded platform/Execd archive manifests all identify ce80a189f352f1ee2ebb6a9f30ee218f2ae896d3, which contains the deadline/retry fix. Both archive SHA-256 values and every internal manifest file hash were checked:

• Platform: 2639ff62149ce915a173d5b205309d5bcfa8e85c49495bb453be024e791a093d.
• Execd: c169abc604e19f69ffc9d4e9714ed4e4609d0510a92a533ebd0a1263c1af882c.

Execd is identical in both archives. The bundled SDK and installed PyPI candidate have matching runtime Python AST and dependency metadata; the Python source difference is a docstring change. There is no remaining finding that #96 source/binary integration is missing.

This review did not run a PR98 runc/KVM node or macOS guest. Cold large-image import exceeding the former 30-second RPC window, expiry/cancellation, lost-result retries and independently observed physical cleanup remain unverified here. Deletion ACK, Deleted receipts, empty API inventory, retry tests and green runsc CI do not establish that in-flight import has stopped or that processes/netns/mounts/runc state are gone. Previous macOS acceptance on the older SDK/runtime tuple does not transfer to this HEAD.

Keep #96's deployment acceptance tracked: use an exclusively owned runc/KVM environment, verify the original deadline and exact instance identity across retries, then independently audit API/queue, held resources, sandboxd/runc, processes, network namespaces, mounts and runtime-state directories after cancellation/deletion. Upgraded macOS acceptance also needs trusted guest SSH, Darwin/version/build and GUI session checks, real command/file/desktop operations, normal shutdown and repeat launch.

Local review validation: 513 SDK/benchmark/deployment-helper/build/contract tests passed, plus the nine additional TLS handshake cases and CA-error reproduction; Ruff and mypy passed. These tests do not close the real runc/KVM/macOS acceptance boundary. Recommendation: do not merge yet; fix the P2, explicitly settle #97's default-policy acceptance, and keep #96's uncompleted real-workload acceptance visible.

[WenYuLuo]

Thanks for the review of 8949111. The current PR head is e82d8a5d4cd05578e0eadf67234104f3e4c23b84.

CA loading contract. We are retaining the existing transport behavior and documenting the contract explicitly. HTTPX loads a selected SSL_CERT_FILE ahead of SSL_CERT_DIR and rejects missing/malformed files during client construction. CLI inventory/deletion, resource queries, PTY and command-watch use Python SSL/OpenSSL default trust paths, so another usable CA directory can still validate the server when the selected file is invalid. --ca-file sets the process variable and enables verification; it does not select an exclusive trust store. This does not disable chain or hostname checks. The SDK README and agent guidance now spell out these semantics.

Three additional real HTTPS/WSS regressions cover missing and malformed files with an actual hashed CA directory: HTTPX errors, verified CLI/resources and PTY/command-watch acceptance through the directory, and hostname rejection. All 14 TLS tests, 291 SDK unit tests, 35 benchmark tests, Ruff and mypy passed; the current head CI passed all ten checks including standalone E2E.

#97 default policy. As explained in the existing #97 reply, verification remains off to preserve the existing one-click self-signed deployment behavior. This is an explicit compatibility decision; HTTPS encryption alone does not authenticate the server. Users enable chain/hostname validation with AKERNEL_VERIFY_TLS=1 or CLI --verify-tls/--ca-file, and configure private trust using SSL_CERT_FILE. Trusted-CA acceptance and untrusted/wrong-host rejection have regression coverage. The PR now says Related to #97, rather than Closes #97; the secure-default recommendation remains visible in the open issue.

#96 regression results. We ran a task-owned Linux AMD64 standalone node on PR head e82d8a5, with sandboxd 7f16edd2, the actual #161 platform/Execd archives (ce80a189) and verified installed binary hashes. Native runc was enabled, and real /dev/kvm ioctl returned API version 12 inside the sandbox. Coordinator and adxlet RPC settings remained at 30 seconds; no 900-second workaround was used. Unique OCI layer digests and a loopback test registry controlled cold preparation delays.

Scenario	Observation	Result
Cold preparation delayed 45 seconds	Running after 46.19 seconds with a 180-second create budget; real KVM ioctl and 64 KiB binary file roundtrip passed; named deletion and independent cleanup passed.	Passed
Lost final create TCP response	Running after 46.33 seconds; two attempts kept the same request ID/spec, node/generation and one runtime identity. Remaining client timeout decreased from about 210 to 163.72 seconds. KVM/files and cleanup passed.	Passed
Delete during a 70-second cold preparation	Delete was requested after the cold layer GET began. A delete read timeout was retried; ACK arrived about 66.01 seconds after the delete request. Create returned an error, no running instance reappeared, and exact Deleted receipt plus API/queue, Redis/held, netns, mounts, processes, bundle/private storage and runc state cleanup passed.	Passed
Expiry during an 80-second cold preparation	The 61-second create budget failed after 61.39 seconds. Retries retained the same request ID and reduced remaining timeout. Logical/catalog cleanup completed, but a late empty runc state directory remained.	Failed strict physical-cleanup acceptance

The expiry timeline is material: the exact Deleted receipt for default-issue96-expiry-eebb5bedc5d4, generation 6, was first observed about 63.19 seconds after create started. The empty directory /run/sandboxd/runc/sbox-884ad2e6-c9f9-488f-84c1-4eafcae5047c first appeared about 81.85 seconds after start, after the delayed layer work, and was still present at the final observation around 274.53 seconds. Independent inspection confirmed it was empty. runc list --format json returned null while warning container does not exist; API inventory/queue, Redis records/held resources, netns, runtime mounts, runc-shim/QEMU processes, bundles and private storage were empty. This is a residual state directory, not evidence of a running orphan, and the strict no-residual-state assertion was kept.

Result: 3/4 cases passed. The extended cold-create budget and retry integration are verified in this controlled native-runc workload; #96's complete cleanup acceptance remains open because of the expiry residue. We did not remove that directory to make the test pass. Evidence was saved before teardown, then the task-owned test container was removed; the pre-existing service remained running. Two initial harness mistakes (binary-read argument and detached-handle deletion semantics) were corrected and their unsuccessful runs retained separately.

This covers controlled cold OCI download/preparation with native runc and real KVM capability, not the original 36.6 GB private image, QEMU/macOS guest, Darwin/GUI/SSH or repeated desktop launch/shutdown acceptance. Those remain separate from these results.

Follow-up: empty runc state residue fixed. PR #98 is now merged into develop. The fix is in sandboxd #72, and AKernel #99 advances the gitlink on top of that develop merge. Cold image preparation outlived the create context, but runc still launched a shim. Cancellation fences now reject that late startup and cancelled ready markers. After the runc CLI confirms absence, deletion safely removes an empty state directory; nonempty state and symlinks remain protected.

• A fresh native-runc AKernel image was built from AKernel 2f166c2 and sandboxd 7fea1b8, using the unchanged, hash-verified ADX #161 platform/Execd archives. Real Redis, ADX services and sandboxd were running. Coordinator/adxlet RPC settings remained 30 seconds; API Server remained 90 seconds.
• Cold create (45-second delay), lost final TCP reply/retry (45-second delay) and delete during import (70-second delay) passed on the first fixed-source campaign. Successful creations exercised real /dev/kvm ioctl API 12 and a 64 KiB binary file roundtrip; retries retained request/spec and one placement/runtime identity.
• A fresh 80-second expiry layer was replayed with a 61-second create budget. Create failed after 61.39 seconds. All 17 post-Delete-ACK observations contained no runc state directory or runtime process. After the delayed import completed, API/queue, Redis held/catalog, netns, runtime state, bundles/private storage, mounts and audited runc/Execd/QEMU processes were empty. No directory was manually removed.
• The first fixed-source campaign failed an additional assertion requiring every resource to be empty immediately after ACK. Its timeline shows the netns was allocated at create +1 second and remained until the in-flight import returned around +82 seconds; no runc directory or runtime process appeared. That failed assertion/run is retained. The fresh expiry replay verifies no late runc startup plus complete final cleanup. Delete ACK still precedes release of the existing netns by roughly 20–21 seconds; that ADX cold image create hits 30s node timeout; Deleted receipt precedes in-flight runtime cleanup #96 timing observation remains open.
• The task-owned node was removed and its profile stopped; the original service/container identity remained running. This covers controlled cold OCI import and native Linux/runc KVM capability, not the original 36.6 GB private image or a QEMU/macOS guest.

The complete sandboxd runc/unit suite, format and vet pass. sandboxd CI passed 14 checks and AKernel CI passed ten checks, with publication skipped. The four scenario outcomes are composed of the first three successful fixed-source cases and a fresh expiry replay on the same source/image; the first stricter ACK assertion failure is retained as described above. #96 remains open for its timing and full original-workload acceptance. The CA loading contract/default-off policy remains unchanged, and #97 stays related and open.

[rcyw]

Thanks for clarifying the trust-source contract and adding the regressions. I’ve reviewed e82d8a5d and accept the documented behavior for this PR. My earlier merge-blocking recommendation no longer applies; I have approved and squash-merged it.

Please keep #97 open for the remaining secure-default recommendation, and add the planned native-runc regression results to #96, including the tested revisions, timeout configuration, retry identity and independent physical-cleanup evidence.